feat(processing): 领域/投影/终态三步提交与 FlightProjectionPort(ACM2-79)

PIPELINE_LOCK 跨 Redis 写;MSG_EVENT HELD→PENDING;Redis 失败不终态不投递;V3 迁移。

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
windyboy
2026-09-21 15:24:34 +08:00
co-authored by Cursor
parent 891088235c
commit 2c3d6af309
28 changed files with 928 additions and 167 deletions
+1 -1
View File
@@ -73,7 +73,7 @@
仓库根目录提供兼容 Podman Compose 与 Docker Compose 的开发中间件栈 `compose.yaml`,包含:
- **共享信箱 MySQL**`mysql:8.4` LTS,端口 3306,库 `cdairport`):容器启动时自动执行 `deploy/dev/mysql-init/01-mailbox.sql` 创建本地联调所需的 `CMINMSGS``CMINMSGS_HST``COUTMSGS` 模拟表。
- **自有 PostgreSQL**`postgres:17-alpine`,端口 5432,库 `msgx`):容器提供干净数据库,应用启动时由 Flyway(`src/main/resources/db/migration/V1__flight_state_baseline.sql`)自动建自有表。
- **Valkey**`valkey/valkey:8-alpine`,端口 6379):本地兼容服务;应用不依赖它(`application.yml` 无对应配置键,航班状态权威在自有 PG)。Redis 航班快照`GET /all/flights` 是目标设计`G-REDIS-PROJECTION`,当前实现不写入也不读取
- **Valkey**`valkey/valkey:8-alpine`,端口 6379):本地兼容服务;应用仍不连接它(航班状态权威在自有 PG)。处理侧的投影写入时序已就位,但 `msgx.redis.enabled` 默认关闭、写入是空操作,真实客户端`GET /all/flights` 仍未接通`G-REDIS-PROJECTION`)。
- **Kafka**`apache/kafka:3.8.0` KRaft 单节点,端口 9092):listener `PLAINTEXT://localhost:9092``default.replication.factor=1`,已预配幂等生产者与 acks=all 所需的单节点参数。
### 快速启动
+12 -8
View File
@@ -60,13 +60,15 @@ msgexchange-v2 怎么处理报文:记录模型、状态机、事务边界、
→ FAILED(等待退避重试)
→ DEADMALFORMED / PROTOCOL / EXHAUSTED,均需人工处置)
投递:PENDING → SENT
→ PENDING(退避后重试)
→ DEAD(重试耗尽,记录保留作死信)
投递:HELD → PENDING → SENT
→ PENDING(退避后重试)
→ DEAD(重试耗尽,记录保留作死信)
```
`SUCCEEDED`/`SKIPPED`/`DEAD` 是终态,不挡后续;`FAILED` 仍占队头。用尽重试 → `DEAD(EXHAUSTED)`。错误分类与重放白名单见 [reference.md](reference.md)。
`HELD` 是尚未放行的待发事件:领域事务登记,投影写成功后由终态事务转 `PENDING`(「事务边界」)。
## 4. 收报
### 4.1 收报
@@ -116,9 +118,9 @@ processOne(head)
载荷缺失 → DEAD(MALFORMED)
整包协议拒绝 → DEAD(PROTOCOL),不落半包
5. 业务型成功,分三步(INV-3、INV-10):
① 事务提交:航班变更 + 待发事件
① 事务提交:航班变更 + 待发事件(登记为 HELD,投递领不到)
② 写 Redis 航班快照;失败 → 保持未完成,下轮重处理
③ 事务提交:SUCCEEDED + 回填意图
③ 事务提交:SUCCEEDED + 回填意图 + 待发事件放行(HELD → PENDING
6. 结束:主泵不做回填;回填意图已随终态落库,由扫描补写信箱标记
```
@@ -130,15 +132,17 @@ processOne(head)
|---|---|---|---|---|
| 收报入队(`insertIfAbsent`) | 是 | 否 | 否 | 同库事务 |
| 身份首次绑定 | 否 | 否 | 否 | 单语句 + 唯一约束 |
| 业务型领域变更(航班变更 + 待发事件) | 是 | 是 | 是 | 同库事务(`INV-3` |
| 业务型领域变更(航班变更 + 待发事件) | 是 | 是 | 是 | 同库事务(`INV-3`;事件落 `HELD` |
| Redis 航班快照写 | 否 | 是(处理步骤锁跨越本步) | 否 | 外部副作用,不在 PG 事务内;写成功是终态事务的前置(`INV-10` |
| 业务型终态(`SUCCEEDED` + 回填意图) | 是 | 是 | 否 | 同库事务(`INV-3` |
| 业务型终态(`SUCCEEDED` + 回填意图 + 事件放行) | 是 | 是 | 否 | 同库事务(`INV-3` |
| 非业务型终态(`MALFORMED` / `PROTOCOL` / `SKIPPED` / `EXHAUSTED`) | 否 | 否 | 否 | 单语句(终态与回填意图同一条 UPDATE) |
| 航班历史清理的物理删除 | 是 | 是(落实 `US-14` AC4) | 是 | 同库事务:复查判据 + 历史写入成功后删除 |
| 回填(信箱标记 + `BACKFILL_AT`) | 否 | 否 | 否 | 跨库两次单写;幂等可重跑 |
| 人工重放(批量改回 `PENDING`) | 否 | 否 | 否 | 单语句批量;`MessageLifecycleGate` 与回填互斥 |
航班变更与终态不在同一事务,中间夹 Redis 写(`INV-3``INV-10`)。`PIPELINE_LOCK` 跨 Redis 写,与历史清理互斥(`US-14` AC4);无第二写者时不额外串行。
航班变更与终态不在同一事务,中间夹 Redis 写(`INV-3``INV-10`)。三步跑在同一条数据库连接上:进处理步骤先在该连接取会话级咨询锁(`pg_advisory_lock`),两次事务复用这条连接,整步结束才解锁。行锁随提交释放,单靠它盖不住提交之后的 Redis 写,所以第二个写者(历史清理)在自己的事务里也要先取同一把咨询锁,再照旧锁 `PIPELINE_LOCK` 行,两边因此在整段内互斥(`US-14` AC4);无第二写者时不额外串行。
投影失败时领域事务已提交而终态未写,`MSG_EVENT` 行停在 `HELD`:投递只领 `PENDING`Kafka 不会先于处理完成发出(`INV-3`)。下一轮重处理按 `MSG_ID` 先清掉上一轮的 `HELD` 行再重新登记,崩溃重启后既不留悬挂行,也不把同一次变更投两遍。
### 5.4 历史积压
+4 -1
View File
@@ -54,6 +54,8 @@
| `msgx.identity.include-day-boundary` | `false` | 去重时是否把日期算进消息身份;受 `C-3` 约束,不能随意改变 | 默认关闭 |
| `msgx.health.backlog-cache-ttl-ms` | `30000` ms | `/health``/metrics` 共用的未处理消息统计最多缓存多久;`0` 表示每次重算 | 暂定 |
| `msgx.history.history-store-enabled` | `false` | 是否启用历史航班写入;关闭时不删除实时航班 | 默认关闭 |
| `msgx.redis.enabled` | `false` | 是否把航班快照写进 Redis 查询投影;关闭时投影写入是空操作(`G-REDIS-PROJECTION` | 默认关闭 |
| `msgx.redis.flight-key` | `flightInfo` | 航班快照所在的 Redis 哈希键,字段名是 `FLID` | 沿用旧系统 |
| `msgx.history.planned-age-days` | `3` 天 | `US-14` AC2 条件 1:计划时间早于当前超过该天数视为已结束 | `US-14` AC2 |
| `msgx.history.cancelled-hours` | `1` 小时 | `US-14` AC2 条件 2:取消时间早于当前超过该窗口视为已结束 | `US-14` AC2 |
| `msgx.history.diverted-hours` | `1` 小时 | `US-14` AC2 条件 3:备降且计划时间早于当前超过该窗口;备降依据 `FDIV``DDES`/`DDIR`)落地前不参与判定(`Q3`、ACM2-100 | `US-14` AC2 |
@@ -90,7 +92,7 @@
**缺值**:带环境变量占位的必需键(`MSGX_PG_URL``MSGX_MAILBOX_URL``MSGX_KAFKA_SERVERS` 等)缺失时由 DI 解析失败拒绝启动,无需自检代码。
**显式开关(非配置错误)**`msgx.stubs``datasources.default.enabled``mailbox.shared-mysql.enabled``msgx.history.history-store-enabled``msgx.pipeline.autostart`。默认关闭是"未接入"的显式声明,不判违规;只有 `msgx.pipeline.autostart=true` 才要求依赖齐备。Kafka 运行时不可达不是配置错误,归 `/health` 与投递指标告警(`OPS-2`)。
**显式开关(非配置错误)**`msgx.stubs``datasources.default.enabled``mailbox.shared-mysql.enabled``msgx.history.history-store-enabled``msgx.redis.enabled``msgx.pipeline.autostart`。默认关闭是"未接入"的显式声明,不判违规;只有 `msgx.pipeline.autostart=true` 才要求依赖齐备。Kafka 运行时不可达不是配置错误,归 `/health` 与投递指标告警(`OPS-2`)。
## 指标与健康
@@ -121,6 +123,7 @@
| 读取信箱与 HTTP 入口 | `ingress/InboxPoller.kt``ingress/InboxController.kt` |
| XML 解析 | `codec/JacksonXmlCodec.kt``codec/SisWireMapper.kt` |
| 顺序处理与航班变更 | `processing/Pump.kt``processing/DynamicProcessors.kt``processing/ScheduleProcessor.kt` |
| 三步提交与 Redis 投影 | `processing/FlightCommit.kt``processing/FlightProjection.kt``infra/projection/` |
| 写回信箱与发送 Kafka | `processing/BackfillService.kt``delivery/Dispatcher.kt``infra/kafka/KafkaDeliveryPort.kt` |
| 定时任务 | `jobs/JobRunner.kt``jobs/HistorySweepJob.kt``jobs/EventCleanupJob.kt` |
| 数据库读写与手动重试 | `infra/persistence/jdbc/``infra/retry/` |
+1 -1
View File
@@ -153,7 +153,7 @@
| `G-FLOP-UNMAPPED` | [XSD](legacy/unisysaodbsis.xsd) FLOP 字段映射不全 | `US-05` |
| `G-MAFL` | 主航班共享列表未做 | `US-06` AC2 |
| `G-PROC-CLEANUP` | 处理记录到期清理未做 | `US-11` |
| `G-REDIS-PROJECTION` | Redis 快照写入与删除未做 | `US-05``US-06``US-07``US-12` |
| `G-REDIS-PROJECTION` | Redis 快照写入与删除未做:投影端口与三步提交时序已就位,真实 Redis 客户端未接通,日计划覆盖范围的缺席清扫也未做 | `US-05``US-06``US-07``US-12` |
| `G-REF-DATA` | 静态参考数据处理与 admin-api 直读未做 | `US-13` |
| `G-REQ-OPEN-UNIQUE` | 同类型未处理完时不允许再发,未做 | `US-09` |
| `G-REQ-TRACK` | 出站请求跟踪未做 | `US-09` |
@@ -0,0 +1,13 @@
package com.gzzn.omms.msgexchange.config
import io.micronaut.context.annotation.ConfigurationProperties
/** 航班查询投影所在的 Redis,对应 `msgx.redis.*`;键位沿用旧系统(`C-11`、`Q6`)。 */
@ConfigurationProperties("msgx.redis")
class RedisProps {
/** 有没有 Redis 可用。关着的时候投影写是空操作(`G-REDIS-PROJECTION`)。 */
var enabled: Boolean = false
/** 航班快照所在的哈希键,字段名是 `FLID`。 */
var flightKey: String = "flightInfo"
}
@@ -6,8 +6,14 @@ package com.gzzn.omms.msgexchange.domain
*/
enum class EventType { UPSERT, TOMBSTONE }
/** 投递状态:PENDING 待发 → SENT 已发出;发送失败按退避重试,重试次数用尽转 DEAD,留在表里当死信队列。 */
enum class EventStatus { PENDING, SENT, DEAD }
/**
* 投递状态:HELD 暂存 → PENDING 待发 → SENT 已发出;发送失败按退避重试,重试次数用尽转 DEAD,
* 留在表里当死信队列。
*
* HELD 是领域事务登记、还没放行的事件:Redis 投影写成功后由终态事务转成 PENDING,
* 投递只领 PENDING,因此处理完成前不会发 Kafka`INV-3`、`INV-10`)。
*/
enum class EventStatus { HELD, PENDING, SENT, DEAD }
/**
* MSG_EVENT:一张待发事件表(outbox),记录航班状态变更要对外发什么。
@@ -21,6 +27,8 @@ enum class EventStatus { PENDING, SENT, DEAD }
*/
data class MsgEvent(
val eventId: Long? = null,
/** 登记这条事件的消息 ID;暂存事件按它清理与放行(`HELD`)。非管道写入方可留空。 */
val msgId: Long? = null,
val target: String,
val partitionKey: String, // 分区键,恒为 FLID(航班实例 ID)
val eventType: EventType = EventType.UPSERT,
@@ -31,6 +31,14 @@ interface PipelineTransactionManager {
/** 单行锁:进事务后先 `SELECT ... FOR UPDATE`,让并发的状态写事务排队执行。 */
interface PipelineLockRepository {
fun lock()
/**
* 跨事务持锁:块内的多次事务复用同一条连接,锁到块结束才释放。
*
* 「领域提交 → 写 Redis → 终态提交」整段都要互斥,而行锁在提交那一刻就没了,
* 盖不住中间的 Redis 写(docs/implementation.md「事务边界」)。
*/
fun <T> holdAcrossTransactions(block: () -> T): T
}
/**
@@ -172,6 +180,20 @@ data class Backlog(
interface MsgEventRepository {
fun insertAll(events: List<MsgEvent>): List<Long>
/**
* 清掉这条消息上一轮残留的暂存事件(`HELD`)。
*
* 领域事务重跑前调用:投影写失败或进程崩溃时,上一轮登记的事件还停在 `HELD`,
* 不清掉就会在重处理后变成永远没人放行的悬挂行。
*/
fun discardHeld(msgId: Long): Int
/**
* 放行这条消息的暂存事件:`HELD` → `PENDING`,投递这才领得到。
* 由终态事务调用,所以 Redis 投影没写成功之前不会发出去(`INV-3`、`INV-10`)。
*/
fun releaseHeld(msgId: Long): Int
/**
* 领取待发事件;[excludePartitionKeys] 为本轮已暂停的 FLID,排除后避免单航班占满整批
* 导致其他航班饿死(ACM2-95)。
@@ -14,13 +14,36 @@ private val transactionConnections = ThreadLocal.withInitial {
java.util.IdentityHashMap<DataSource, java.sql.Connection>()
}
/**
* 被钉住的连接:块内的事务与单语句都走它。会话级锁(`pg_advisory_lock`)只在自己的会话里成立,
* 所以要跨事务持锁,就得让这几次事务落在同一条连接上。
*/
private val pinnedConnections = ThreadLocal.withInitial {
java.util.IdentityHashMap<DataSource, java.sql.Connection>()
}
internal fun <T> DataSource.withPinnedSession(block: (java.sql.Connection) -> T): T {
val pinned = pinnedConnections.get()
check(pinned[this] == null) { "session already pinned for this data source" }
val conn = this.connection
pinned[this] = conn
try {
return block(conn)
} finally {
pinned.remove(this)
if (pinned.isEmpty()) pinnedConnections.remove()
conn.close()
}
}
internal fun <T> DataSource.withTransaction(block: () -> T): T {
val connections = transactionConnections.get()
val existing = connections[this]
if (existing != null) {
return block()
}
val conn = this.connection
val pinned = pinnedConnections.get()[this]
val conn = pinned ?: this.connection
val oldAutoCommit = conn.autoCommit
conn.autoCommit = false
connections[this] = conn
@@ -42,19 +65,20 @@ internal fun <T> DataSource.withTransaction(block: () -> T): T {
conn.autoCommit = oldAutoCommit
} catch (_: Throwable) {
}
conn.close()
// 钉住的连接由 withPinnedSession 关闭:提前还池会连同会话锁一起丢掉。
if (pinned == null) conn.close()
}
}
internal fun DataSource.obtainConnection(): java.sql.Connection =
transactionConnections.get()[this] ?: this.connection
transactionConnections.get()[this] ?: pinnedConnections.get()[this] ?: this.connection
internal fun java.sql.Connection.releaseIfNotInTransaction(dataSource: DataSource) {
val connections = transactionConnections.get()
if (connections[dataSource] !== this) {
this.close()
if (connections.isEmpty()) transactionConnections.remove()
}
if (connections[dataSource] === this) return
if (pinnedConnections.get()[dataSource] === this) return
this.close()
if (connections.isEmpty()) transactionConnections.remove()
}
internal fun <T> DataSource.query(sql: String, bind: (java.sql.PreparedStatement) -> Unit, map: (ResultSet) -> T): List<T> {
@@ -63,10 +63,40 @@ class JdbcPipelineTransactionManager(
class JdbcPipelineLockRepository(
private val ds: DataSource,
) : PipelineLockRepository {
/** 进事务后的第一步:对锁行 `FOR UPDATE`,让并发的状态写事务排队。 */
/**
* 进事务后的第一步:先在咨询锁上排队,再对锁行 `FOR UPDATE`。
*
* 行锁在提交时就释放,盖不住处理步骤中段的 Redis 写,所以两个写者都要过同一把咨询锁:
* 主泵整段持会话级([holdAcrossTransactions]),历史清理在自己的事务里持事务级。
* 同一会话重复取同一把咨询锁不自阻塞,因此持锁期间内层事务照常通过。
*/
override fun lock() {
ds.queryOne("SELECT pg_advisory_xact_lock(?)", { ps -> ps.setLong(1, PIPELINE_ADVISORY_KEY) }) { 1 }
ds.queryOne("SELECT lock_id FROM pipeline_lock WHERE lock_id = 1 FOR UPDATE", {}) { 1 } ?: error("PIPELINE_LOCK row missing")
}
/** 钉住一条连接并在其上持会话级咨询锁:块内的两次事务复用它,出块才解锁还池。 */
override fun <T> holdAcrossTransactions(block: () -> T): T = ds.withPinnedSession {
advisory("SELECT pg_advisory_lock(?)")
try {
block()
} finally {
// 解锁失败只能记账:连接随后归还连接池,会话锁若残留会挡住后续处理步骤。
runCatching { advisory("SELECT pg_advisory_unlock(?)") }
.onFailure { log.error("pipeline advisory unlock failed", it) }
}
}
private fun advisory(sql: String) {
ds.queryOne(sql, { ps -> ps.setLong(1, PIPELINE_ADVISORY_KEY) }) { 1 }
}
private companion object {
private val log = org.slf4j.LoggerFactory.getLogger(JdbcPipelineLockRepository::class.java)
/** `PIPELINE_LOCK` 的咨询锁键('msgx' 的 ASCII 加序号 1):同库其他应用撞键的概率足够低。 */
private const val PIPELINE_ADVISORY_KEY = 0x6D73_6778_01L
}
}
@Singleton
@@ -379,8 +409,8 @@ class JdbcMsgEventRepository(
private fun insertOne(e: MsgEvent): Long =
ds.updateReturningLong(
"""
INSERT INTO msg_event (target, partition_key, event_type, state_version, payload_json, state, attempts, next_attempt_at, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING event_id
INSERT INTO msg_event (target, partition_key, event_type, state_version, payload_json, state, attempts, next_attempt_at, created_at, msg_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING event_id
""".trimIndent(),
{ ps -> bindEvent(ps, e) },
)
@@ -393,15 +423,16 @@ class JdbcMsgEventRepository(
private fun upsertSchd(e: MsgEvent): Long? =
ds.queryOne(
"""
INSERT INTO msg_event (target, partition_key, event_type, state_version, payload_json, state, attempts, next_attempt_at, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
INSERT INTO msg_event (target, partition_key, event_type, state_version, payload_json, state, attempts, next_attempt_at, created_at, msg_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (target, partition_key) WHERE target = 'KAFKA:schd'
DO UPDATE SET
event_id = EXCLUDED.event_id,
event_type = EXCLUDED.event_type,
state_version = EXCLUDED.state_version,
payload_json = EXCLUDED.payload_json,
state = 'PENDING',
msg_id = EXCLUDED.msg_id,
state = EXCLUDED.state,
attempts = 0,
next_attempt_at = NULL,
error_class = NULL,
@@ -425,8 +456,23 @@ class JdbcMsgEventRepository(
ps.setInt(7, e.attempts)
ps.setTimestamp(8, e.nextAttemptAt?.toSqlTimestamp())
ps.setTimestamp(9, e.createdAt.toSqlTimestamp())
e.msgId?.let { ps.setLong(10, it) } ?: ps.setNull(10, java.sql.Types.BIGINT)
}
/** 重处理前清残:上一轮登记但没放行的事件行删掉,避免留下永远发不出去的悬挂行。 */
override fun discardHeld(msgId: Long): Int =
ds.update(
"DELETE FROM msg_event WHERE msg_id = ? AND state = 'HELD'",
{ ps -> ps.setLong(1, msgId) },
)
/** 放行:投影已写成功,事件转 PENDING 交给投递。 */
override fun releaseHeld(msgId: Long): Int =
ds.update(
"UPDATE msg_event SET state = 'PENDING' WHERE msg_id = ? AND state = 'HELD'",
{ ps -> ps.setLong(1, msgId) },
)
override fun claimBatch(target: String, limit: Int, excludePartitionKeys: Set<String>): List<MsgEvent> {
if (excludePartitionKeys.isEmpty()) {
return ds.query(
@@ -536,6 +582,7 @@ class JdbcMsgEventRepository(
private fun mapEvent(rs: ResultSet) = MsgEvent(
eventId = rs.getLong("event_id"),
msgId = rs.getLong("msg_id").takeIf { !rs.wasNull() },
target = rs.getString("target"),
partitionKey = rs.getString("partition_key"),
eventType = EventType.valueOf(rs.getString("event_type")),
@@ -0,0 +1,49 @@
package com.gzzn.omms.msgexchange.infra.projection
import com.gzzn.omms.msgexchange.config.RedisProps
import com.gzzn.omms.msgexchange.processing.FlightProjectionPort
import com.gzzn.omms.msgexchange.processing.FlightProjectionWrite
import io.micronaut.context.annotation.Requires
import jakarta.inject.Singleton
/**
* Redis 投影适配器骨架:键位与写法在这里定死,**但客户端还没接进来**`G-REDIS-PROJECTION`)。
*
* 写入形态沿用旧系统:一个哈希键(`PARAM:msgx.redis.flight-key`)装全部航班,
* field 是 `FLID`value 是整态 JSON,不设过期;删除航班就删掉这个 field(`INV-8`)。
*
* 开了 `msgx.redis.enabled` 却没有客户端时,写入直接抛异常而不是假装成功:
* 按 `INV-10`,这条消息就停在未完成、下轮重试,不会被当成已处理写回信箱。
*/
@Requires(property = "msgx.stubs", notEquals = "true")
@Requires(property = "msgx.redis.enabled", value = "true")
@Singleton
class RedisFlightProjectionPort(
private val props: RedisProps,
) : FlightProjectionPort {
override fun write(writes: List<FlightProjectionWrite>) {
throw UnsupportedOperationException(
"redis client not wired yet (G-REDIS-PROJECTION); pending writes=${writes.size} key=${props.flightKey}",
)
}
override fun ping(): Boolean = false
}
/**
* 没接 Redis 时的投影出口:什么都不做。
*
* 这不是"投影写成功"的承诺——`INV-10` 在真实客户端接通之前只能空转(`G-REDIS-PROJECTION`)。
* 摆这个 bean 是为了让三步提交的时序在没有 Redis 的环境里也照常跑,而不是让每条航班报文都失败。
*/
@Requires(property = "msgx.stubs", notEquals = "true")
@Requires(property = "msgx.redis.enabled", notEquals = "true")
@Singleton
class NoopFlightProjectionPort : FlightProjectionPort {
private val log = org.slf4j.LoggerFactory.getLogger(NoopFlightProjectionPort::class.java)
override fun write(writes: List<FlightProjectionWrite>) {
log.debug("redis projection disabled, skipping {} write(s) [G-REDIS-PROJECTION]", writes.size)
}
}
@@ -1,6 +1,8 @@
package com.gzzn.omms.msgexchange.infra.stub
import com.gzzn.omms.msgexchange.delivery.DeliveryPort
import com.gzzn.omms.msgexchange.processing.FlightProjectionPort
import com.gzzn.omms.msgexchange.processing.FlightProjectionWrite
import io.micronaut.context.annotation.Requires
import jakarta.inject.Singleton
@@ -32,3 +34,37 @@ class StubDeliveryPort : DeliveryPort {
sent += Sent(topic, key, null)
}
}
/**
* 航班查询投影的内存假实现,只有配置 msgx.stubs=true 时才装配。
*
* [snapshots] 是投影的当前内容(FLID → 整态 JSON),删除会把条目移走,行为对齐 Redis 哈希。
* [failWith] 用来做故障注入:置上以后每次写都抛这个异常,用于验证"投影写不成功就不算处理完成"
* `INV-10`);置回 null 即恢复。
*/
@Requires(property = "msgx.stubs", value = "true")
@Singleton
class StubFlightProjectionPort : FlightProjectionPort {
val snapshots = linkedMapOf<String, String>()
val writes = mutableListOf<FlightProjectionWrite>()
@Volatile
var failWith: RuntimeException? = null
fun clear() {
snapshots.clear(); writes.clear(); failWith = null
}
override fun write(writes: List<FlightProjectionWrite>) {
failWith?.let { throw it }
this.writes += writes
writes.forEach { w ->
when (w) {
is FlightProjectionWrite.Upsert -> snapshots[w.flid] = w.payloadJson
is FlightProjectionWrite.Delete -> snapshots.remove(w.flid)
}
}
}
override fun ping(): Boolean = failWith == null
}
@@ -51,6 +51,8 @@ class StubPipelineTx : PipelineTransactionManager {
@Requires(property = "msgx.stubs", value = "true")
class StubPipelineLock : PipelineLockRepository {
override fun lock() = Unit
override fun <T> holdAcrossTransactions(block: () -> T): T = block()
}
/** 内存版 PROC_STATE:入队幂等,写终态时一并写下回填待办。 */
@@ -261,7 +263,7 @@ class StubMsgEvents : MsgEventRepository {
/**
* 与 JDBC upsert 同口径的单行投影:只进不退;同版本只有 tombstone 能覆盖非 tombstone
* 接受的写代次换新 `EVENT_ID` 并重置 `PENDING`(清空 attempts/next/error)。
* 接受的写代次换新 `EVENT_ID` 并沿用本次写入的状态(清空 attempts/next/error)。
*/
private fun upsertSchd(e: MsgEvent): Long {
val existing = rows.values.firstOrNull { it.target == Targets.KAFKA_SCHD && it.partitionKey == e.partitionKey }
@@ -273,12 +275,26 @@ class StubMsgEvents : MsgEventRepository {
}
val id = ids.incrementAndGet()
rows[id] = e.copy(
eventId = id, state = EventStatus.PENDING, attempts = 0, nextAttemptAt = null,
eventId = id, attempts = 0, nextAttemptAt = null,
errorClass = null, lastError = null, sentAt = null,
)
return id
}
/** 清残:上一轮登记但没放行的事件行删掉(对齐 JDBC 的按 MSG_ID 删 HELD)。 */
override fun discardHeld(msgId: Long): Int {
val ids = rows.values.filter { it.msgId == msgId && it.state == EventStatus.HELD }.map { it.eventId!! }
ids.forEach { rows.remove(it) }
return ids.size
}
/** 放行:HELD → PENDING,投递这才领得到。 */
override fun releaseHeld(msgId: Long): Int {
val held = rows.values.filter { it.msgId == msgId && it.state == EventStatus.HELD }
held.forEach { rows[it.eventId!!] = it.copy(state = EventStatus.PENDING) }
return held.size
}
override fun claimBatch(target: String, limit: Int, excludePartitionKeys: Set<String>): List<MsgEvent> =
rows.values
.filter { it.target == target && it.state == EventStatus.PENDING }
@@ -4,11 +4,10 @@ import com.fasterxml.jackson.databind.ObjectMapper
import com.gzzn.omms.msgexchange.codec.FlopPayload
import com.gzzn.omms.msgexchange.config.OperationDayProps
import com.gzzn.omms.msgexchange.domain.DecodedMessage
import com.gzzn.omms.msgexchange.domain.EventType
import com.gzzn.omms.msgexchange.domain.EventStatus
import com.gzzn.omms.msgexchange.domain.MsgEvent
import com.gzzn.omms.msgexchange.domain.OperationDayCalculator
import com.gzzn.omms.msgexchange.domain.ProcState
import com.gzzn.omms.msgexchange.domain.ProcStatus
import com.gzzn.omms.msgexchange.domain.Targets
import com.gzzn.omms.msgexchange.domain.flight.FlightSnapshot
import com.gzzn.omms.msgexchange.domain.flight.FlightState
@@ -18,9 +17,6 @@ import com.gzzn.omms.msgexchange.domain.flight.ScheduleRecord
import com.gzzn.omms.msgexchange.infra.persistence.FlightStateRepository
import com.gzzn.omms.msgexchange.infra.persistence.MsgEventRepository
import com.gzzn.omms.msgexchange.infra.persistence.PersistOutcome
import com.gzzn.omms.msgexchange.infra.persistence.PipelineLockRepository
import com.gzzn.omms.msgexchange.infra.persistence.PipelineTransactionManager
import com.gzzn.omms.msgexchange.infra.persistence.ProcStateRepository
import jakarta.inject.Singleton
import java.time.Clock
import java.time.Instant
@@ -35,29 +31,22 @@ import java.time.ZoneId
*/
@Singleton
class FlopProcessor(
private val txManager: PipelineTransactionManager,
private val lock: PipelineLockRepository,
private val commit: FlightCommit,
private val flightState: FlightStateRepository,
private val msgEvents: MsgEventRepository,
private val procState: ProcStateRepository,
private val mapper: ObjectMapper,
private val clock: Clock,
) {
fun apply(head: ProcState, msg: DecodedMessage, payload: FlopPayload): ApplyResult = txManager.inTransaction {
lock.lock()
fun apply(head: ProcState, msg: DecodedMessage, payload: FlopPayload): ApplyResult = commit.commit(head) {
// 迟到/未知航班:幂等成功,不创建(创建入口只有 SCHD/ADFT),也没有投影要刷
val current = flightState.loadFullSnapshot(payload.flid)
if (current == null) {
// 迟到/未知航班:幂等成功,不创建(创建入口只有 SCHD/ADFT);终态同事务落库
procState.markTerminal(head.msgId, ProcStatus.SUCCEEDED, now = clock.instant())
return@inTransaction ApplyResult.Succeeded
}
?: return@commit DomainOutcome(ApplyResult.Succeeded)
val change = MergeChange(flid = payload.flid, scalars = payload.scalars, collections = payload.collections)
val next = FlightStateEngine.mergedState(current, change)
flightState.persistFullState(next, msgId = head.msgId, now = clock.instant())
msgEvents.insertAll(eventsFor(next, mapper, clock.instant()))
procState.markTerminal(head.msgId, ProcStatus.SUCCEEDED, now = clock.instant())
ApplyResult.Succeeded
msgEvents.insertAll(eventsFor(next, head.msgId, mapper, clock.instant()))
DomainOutcome(ApplyResult.Succeeded, listOf(projectionOf(next, mapper)))
}
}
@@ -65,43 +54,47 @@ class FlopProcessor(
* 处理 FDEL 删除报文:把在用的航班标记为 DELETED,版本号加一,明细数据保留,
* 并且只发布一次删除事件。
*
* 已经删除过、或航班本来就不存在时,算处理成功但不再动版本、不重复发事件
* 已经删除过、或航班本来就不存在时,算处理成功但不再动版本、不重复发事件
* 但只要航班还在库里且是删除态,投影删除就照发一次——幂等命令换来"上一轮没删干净"能自愈(`INV-8`)。
*/
@Singleton
class FdelProcessor(
private val txManager: PipelineTransactionManager,
private val lock: PipelineLockRepository,
private val commit: FlightCommit,
private val flightState: FlightStateRepository,
private val msgEvents: MsgEventRepository,
private val procState: ProcStateRepository,
private val mapper: ObjectMapper,
private val clock: Clock,
) {
fun apply(head: ProcState, msg: DecodedMessage, payload: FlopPayload): ApplyResult = txManager.inTransaction {
lock.lock()
fun apply(head: ProcState, msg: DecodedMessage, payload: FlopPayload): ApplyResult = commit.commit(head) {
val deleted = flightState.markDeleted(payload.flid, msgId = head.msgId, now = clock.instant())
if (deleted) {
val current = flightState.loadFullSnapshot(payload.flid)
// 只有"在用 → 删除"这一步才发删除通知,而且和状态变更写在同一个事务里
val createdAt = clock.instant()
// 航班根本不存在(迟到或多余的删除报文):算成功,也没有投影要删
val main = flightState.findMainRow(payload.flid)
?: return@commit DomainOutcome(ApplyResult.Succeeded)
// 这一次删掉的,或者上一轮就是本消息删掉的(投影写失败后重处理),都要登记删除通知:
// 重处理开始时已把上一轮没放行的事件清掉,这里不补登就会把这条通知丢了。
if (deleted || (main.state == FlightState.DELETED && main.lastMsgId == head.msgId)) {
msgEvents.insertAll(
listOf(
// C-9:删除通知只走 KAFKA:msgschd 不再发 tombstone
// value 形态("deleted":true 的 JSON)沿用现状,待 Q5 定稿
MsgEvent(
msgId = head.msgId,
target = Targets.KAFKA_MSG,
partitionKey = payload.flid,
stateVersion = current?.stateVersion ?: 0L,
stateVersion = main.stateVersion,
payloadJson = mapper.writeValueAsString(
mapOf("flid" to payload.flid, "stateVersion" to (current?.stateVersion ?: 0L), "deleted" to true),
mapOf("flid" to payload.flid, "stateVersion" to main.stateVersion, "deleted" to true),
),
createdAt = createdAt,
state = EventStatus.HELD,
createdAt = clock.instant(),
),
),
)
}
procState.markTerminal(head.msgId, ProcStatus.SUCCEEDED, now = clock.instant()) // 没删到东西说明是迟到或重复报文,照样算成功
ApplyResult.Succeeded
// INV-8:只要航班是删除态就得从投影里抹掉。重复报文也重删一次——代价是一条幂等命令,
// 换来"上一轮删投影没成功"能自愈。
DomainOutcome(ApplyResult.Succeeded, listOf(FlightProjectionWrite.Delete(payload.flid, main.stateVersion)))
}
}
@@ -114,11 +107,9 @@ class FdelProcessor(
*/
@Singleton
class AdftProcessor(
private val txManager: PipelineTransactionManager,
private val lock: PipelineLockRepository,
private val commit: FlightCommit,
private val flightState: FlightStateRepository,
private val msgEvents: MsgEventRepository,
private val procState: ProcStateRepository,
operationDayProps: OperationDayProps,
private val mapper: ObjectMapper,
private val clock: Clock,
@@ -130,24 +121,27 @@ class AdftProcessor(
fun apply(head: ProcState, msg: DecodedMessage, record: ScheduleRecord): ApplyResult =
try {
txManager.inTransaction {
lock.lock()
commit.commit(head) {
val main = flightState.findMainRow(record.flid)
if (main != null && main.state == FlightState.DELETED) {
// 已删除的航班重新激活:状态改回 ACTIVE、版本号加一,并登记状态事件
var revived: FlightSnapshot? = null
if (flightState.revive(record.flid, msgId = head.msgId, now = clock.instant())) {
val current = flightState.loadFullSnapshot(record.flid)
if (current != null) {
val next = FlightStateEngine.mergedState(current, setOnly(record))
flightState.persistFullState(next, msgId = head.msgId, now = clock.instant())
msgEvents.insertAll(eventsFor(next, mapper, clock.instant()))
msgEvents.insertAll(eventsFor(next, head.msgId, mapper, clock.instant()))
revived = next
}
} else {
// 并发下可能已被别的消息处理掉:不改版本、不重复发事件,但留下痕迹便于对账。
log.warn("adft revive no-op (already active or vanished) msgId={} flid={}", head.msgId, record.flid)
}
procState.markTerminal(head.msgId, ProcStatus.SUCCEEDED, now = clock.instant())
return@inTransaction ApplyResult.Succeeded
return@commit DomainOutcome(
ApplyResult.Succeeded,
listOfNotNull(revived?.let { projectionOf(it, mapper) }),
)
}
val current = flightState.loadFullSnapshot(record.flid)
@@ -173,9 +167,8 @@ class AdftProcessor(
// 若按 INFRA 抛出去,会被当成暂时性故障白白重试到耗尽,并给出误导的错误类别。
throw ProtocolViolation("operation-day guard violated flid=${record.flid}")
}
msgEvents.insertAll(eventsFor(next, mapper, clock.instant()))
procState.markTerminal(head.msgId, ProcStatus.SUCCEEDED, now = clock.instant())
ApplyResult.Succeeded
msgEvents.insertAll(eventsFor(next, head.msgId, mapper, clock.instant()))
DomainOutcome(ApplyResult.Succeeded, listOf(projectionOf(next, mapper)))
}
} catch (e: ProtocolViolation) {
log.error("adft DEAD(PROTOCOL) msgId={} reason={}", head.msgId, e.message)
@@ -201,28 +194,37 @@ class AdftProcessor(
// 共享小工具(处理器层私有约定)
// =====================================================================
/** 航班状态变化后要发的两类事件:KAFKA_SCHD 发完整状态,KAFKA_MSG 发一条变更通知。 */
internal fun eventsFor(next: FlightSnapshot, mapper: ObjectMapper, createdAt: Instant): List<MsgEvent> {
val payload = linkedMapOf<String, Any>(
"flid" to next.flid,
"stateVersion" to next.stateVersion,
"scalars" to next.scalars,
"collections" to next.collections,
)
return listOf(
/** 航班整态载荷:`KAFKA:schd` 事件与 Redis 投影共用这一份形状。 */
internal fun flightPayload(next: FlightSnapshot): Map<String, Any> = linkedMapOf(
"flid" to next.flid,
"stateVersion" to next.stateVersion,
"scalars" to next.scalars,
"collections" to next.collections,
)
/**
* 航班状态变化后要发的两类事件:KAFKA_SCHD 发完整状态,KAFKA_MSG 发一条变更通知。
*
* 两条都以 [EventStatus.HELD] 登记,投影写成功后由终态事务放行(`INV-3`、`INV-10`)。
*/
internal fun eventsFor(next: FlightSnapshot, msgId: Long, mapper: ObjectMapper, createdAt: Instant): List<MsgEvent> =
listOf(
MsgEvent(
msgId = msgId,
target = Targets.KAFKA_SCHD,
partitionKey = next.flid,
stateVersion = next.stateVersion,
payloadJson = mapper.writeValueAsString(payload),
payloadJson = mapper.writeValueAsString(flightPayload(next)),
state = EventStatus.HELD,
createdAt = createdAt,
),
MsgEvent(
msgId = msgId,
target = Targets.KAFKA_MSG,
partitionKey = next.flid,
stateVersion = next.stateVersion,
payloadJson = mapper.writeValueAsString(mapOf("flid" to next.flid, "stateVersion" to next.stateVersion)),
state = EventStatus.HELD,
createdAt = createdAt,
),
)
}
@@ -0,0 +1,67 @@
package com.gzzn.omms.msgexchange.processing
import com.gzzn.omms.msgexchange.domain.ProcState
import com.gzzn.omms.msgexchange.domain.ProcStatus
import com.gzzn.omms.msgexchange.infra.persistence.MsgEventRepository
import com.gzzn.omms.msgexchange.infra.persistence.PipelineLockRepository
import com.gzzn.omms.msgexchange.infra.persistence.PipelineTransactionManager
import com.gzzn.omms.msgexchange.infra.persistence.ProcStateRepository
import jakarta.inject.Singleton
import java.time.Clock
/**
* 领域事务的产物:交给调用方的结果,加上这次变更要刷进查询投影的航班。
* 没有航班变更(迟到报文、空日计划)时投影为空,这一步直接跳过。
*/
data class DomainOutcome<T>(val value: T, val projections: List<FlightProjectionWrite> = emptyList())
/**
* 业务型成功的三步提交(`INV-3`、`INV-10`,见 docs/implementation.md「事务边界」):
*
* 1. 领域事务:航班变更与待发事件一起提交,事件落 `HELD`(投递领不到);
* 2. 写 Redis 投影:失败就抛出去,这条消息保持未完成,下轮重处理;
* 3. 终态事务:放行待发事件,写 `SUCCEEDED` 与回填意图。
*
* 三步整段持 `PIPELINE_LOCK`:行锁提交即释放,盖不住中间的投影写,所以由
* [PipelineLockRepository.holdAcrossTransactions] 在一条连接上从头持到尾。
*
* 投影失败时第一步已经提交,**不回滚**(`US-03` AC3):重处理按当前完整态重写投影,
* 上一轮的 `HELD` 事件先清后补,不会重复投递。
*/
@Singleton
class FlightCommit(
private val txManager: PipelineTransactionManager,
private val lock: PipelineLockRepository,
private val procState: ProcStateRepository,
private val msgEvents: MsgEventRepository,
private val projection: FlightProjectionPort,
private val clock: Clock,
) {
fun <T> commit(head: ProcState, domain: () -> DomainOutcome<T>): T = lock.holdAcrossTransactions {
val outcome = txManager.inTransaction {
lock.lock()
msgEvents.discardHeld(head.msgId) // 清掉上一轮没放行的事件,再登记本轮的
domain()
}
writeProjection(head, outcome.projections)
txManager.inTransaction {
lock.lock()
msgEvents.releaseHeld(head.msgId)
procState.markTerminal(head.msgId, ProcStatus.SUCCEEDED, now = clock.instant())
}
outcome.value
}
private fun writeProjection(head: ProcState, writes: List<FlightProjectionWrite>) {
if (writes.isEmpty()) return
try {
projection.write(writes)
} catch (e: InterruptedException) {
throw e // 停机信号不归到这条消息头上
} catch (e: Exception) {
throw FlightProjectionFailure(head.msgId, e)
}
}
}
@@ -0,0 +1,47 @@
package com.gzzn.omms.msgexchange.processing
import com.fasterxml.jackson.databind.ObjectMapper
import com.gzzn.omms.msgexchange.domain.flight.FlightSnapshot
/**
* 一次投影写:把某个航班的整态写进查询投影,或者把它从投影里删掉。
* 带上版本号,适配层据此做"只进不退"的覆盖判断。
*/
sealed interface FlightProjectionWrite {
val flid: String
val stateVersion: Long
data class Upsert(
override val flid: String,
override val stateVersion: Long,
val payloadJson: String,
) : FlightProjectionWrite
data class Delete(override val flid: String, override val stateVersion: Long) : FlightProjectionWrite
}
/**
* 航班查询投影(Redis)的出口。投影是 PG 当前态的副本,只读不权威(`INV-5`、`INV-11`)。
*
* 写成功这条消息才算处理完成(`INV-10`):**写不成功必须抛异常**,不能吞掉——
* 调用方据此保持未完成状态、下轮整条重处理(`US-05` AC4)。
*/
interface FlightProjectionPort {
/** 一批一起写:同一条消息产生的写要么整批成功,要么抛异常整批重来。 */
fun write(writes: List<FlightProjectionWrite>)
/** 给健康检查用的连通性探测。 */
fun ping(): Boolean = true
}
/** 投影写失败:包一层好让日志和错误分类看得出是投影这一步挂的,处理侧按 `INFRA` 重试。 */
class FlightProjectionFailure(msgId: Long, cause: Throwable) :
RuntimeException("flight projection write failed msgId=$msgId: ${cause.message ?: cause.javaClass.simpleName}", cause)
/** 投影载荷与 `KAFKA:schd` 整态同一份 JSON:投影和通知同源,对账时不用比两种形状(`C-11`)。 */
internal fun projectionOf(snapshot: FlightSnapshot, mapper: ObjectMapper): FlightProjectionWrite.Upsert =
FlightProjectionWrite.Upsert(
flid = snapshot.flid,
stateVersion = snapshot.stateVersion,
payloadJson = mapper.writeValueAsString(flightPayload(snapshot)),
)
@@ -106,7 +106,8 @@ class Pump(
/**
* 处理一条消息:读原文 → 解码 → 绑定业务身份 → 分派给对应处理器。
*
* 业务数据、终态与回填意图由各处理器在自己的事务里写入(终态与回填意图是同一条 UPDATE)。
* 业务数据、Redis 投影、终态与回填意图由各处理器按三步提交写入([FlightCommit]):
* 领域事务提交后才写投影,投影成功后才记终态与回填意图(同一条 UPDATE)。
* **这里不做信箱回填**:主泵是 FIFO 关键路径,跨库写会把它绑在共享 MySQL 的可用性上。
* 回填由 `JobRunner` 定时的 `BackfillService.sweep` 驱动(调度周期不等于完成时限)。
*
@@ -6,33 +6,26 @@ import com.gzzn.omms.msgexchange.config.OperationDayProps
import com.gzzn.omms.msgexchange.domain.DecodedMessage
import com.gzzn.omms.msgexchange.domain.MsgEvent
import com.gzzn.omms.msgexchange.domain.ProcState
import com.gzzn.omms.msgexchange.domain.ProcStatus
import com.gzzn.omms.msgexchange.domain.SnapshotFlag
import com.gzzn.omms.msgexchange.domain.SnapshotLogEntry
import com.gzzn.omms.msgexchange.domain.SnapshotResult
import com.gzzn.omms.msgexchange.domain.Targets
import com.gzzn.omms.msgexchange.domain.OperationDayCalculator
import com.gzzn.omms.msgexchange.domain.flight.FlightSnapshot
import com.gzzn.omms.msgexchange.domain.flight.FlightState
import com.gzzn.omms.msgexchange.domain.flight.FlightStateEngine
import com.gzzn.omms.msgexchange.domain.flight.ScheduleRecord
import com.gzzn.omms.msgexchange.domain.flight.SnapshotValidation
import com.gzzn.omms.msgexchange.infra.persistence.FlightStateRepository
import com.gzzn.omms.msgexchange.infra.persistence.MsgEventRepository
import com.gzzn.omms.msgexchange.infra.persistence.PersistOutcome
import com.gzzn.omms.msgexchange.infra.persistence.PipelineLockRepository
import com.gzzn.omms.msgexchange.infra.persistence.PipelineTransactionManager
import com.gzzn.omms.msgexchange.infra.persistence.ProcStateRepository
import com.gzzn.omms.msgexchange.infra.persistence.SnapshotLogRepository
import jakarta.inject.Singleton
import java.time.Clock
import java.time.Instant
import java.time.LocalDate
import java.time.ZoneId
/**
* 处理器告诉调用方这一条消息处理成了什么。
* 无论哪种结果,终态和回填待办都由处理器在自己的事务里写好
* 业务型成功的终态和回填待办已由三步提交写好([FlightCommit]),调用方不必再补
*/
sealed interface ApplyResult {
/** 业务处理成功(包含"重复写入但结果一致"这种幂等成功)。 */
@@ -52,15 +45,14 @@ class ProtocolViolation(message: String) : RuntimeException(message)
* 处理 SCHD 日计划报文(DNLD 和 RESP 走同一条路):把报文里的航班记录合并进航班当前态。
*
* 顺序是:整包校验 → 加锁 → 核对每条航班的运营日 → 逐条合并写入 → 登记待发事件 →
* 写终态和回填待办。除了校验,后面所有步骤都在同一个事务里,任何一步失败整包回滚,
* 不会留下写了一半的数据。
* 刷投影 → 写终态和回填待办。领域这一段在同一个事务里,任何一步失败整包回滚,
* 不会留下写了一半的数据;投影与终态按三步提交各走一步([FlightCommit]
*
* 报文里没提到的航班不会被删除——日计划只负责写它带来的那部分。
*/
@Singleton
class ScheduleProcessor(
private val txManager: PipelineTransactionManager,
private val lock: PipelineLockRepository,
private val commit: FlightCommit,
private val procState: ProcStateRepository,
private val flightState: FlightStateRepository,
private val msgEvents: MsgEventRepository,
@@ -98,21 +90,17 @@ class ScheduleProcessor(
val ok = validation as SnapshotValidation.Ok
if (ok.perRecordDay.isEmpty()) {
// 报文合法但没有记录:不写航班,但终态与回填意图仍在锁事务内一起提交
txManager.inTransaction {
lock.lock()
procState.markTerminal(head.msgId, ProcStatus.SUCCEEDED, now = clock.instant())
}
// 报文合法但没有记录:不写航班、没有投影要刷,终态与回填意图仍走三步提交的最后一步
commit.commit(head) { DomainOutcome(Unit) }
logSnapshot(head, body, SnapshotResult.COMMITTED, upserted = 0, setOf(SnapshotFlag.EMPTY), started)
return ApplyResult.Succeeded
}
val flags = linkedSetOf<SnapshotFlag>()
return try {
// 以下都在同一个事务里:加锁 → 校验运营日 → 逐条合并写入 → 登记事件与回填待办
val upserted = txManager.inTransaction {
lock.lock()
// 领域这一段在同一个事务里:加锁 → 校验运营日 → 逐条合并写入 → 登记事件
// 投影与终态由 FlightCommit 接着走后两步
val upserted = commit.commit(head) {
// 一次批量查出这些航班现有的运营日,逐个比对(避免逐条查询)
val mains = flightState.findMainRows(ok.perRecordDay.keys)
ok.perRecordDay.forEach { (flid, day) ->
@@ -126,6 +114,7 @@ class ScheduleProcessor(
var written = 0
val events = mutableListOf<MsgEvent>()
val projections = mutableListOf<FlightProjectionWrite>()
// 一次建索引,避免在航班级循环里反复线性扫描(大日计划下是 O(N²))。
val recordsByFlid = body.records.associateBy { it.flid }
ok.perRecordDay.forEach { (flid, day) ->
@@ -145,12 +134,11 @@ class ScheduleProcessor(
throw ProtocolViolation("operation-day guard violated flid=$flid")
else -> written++
}
events += snapshotEvents(next)
events += eventsFor(next, head.msgId, mapper, clock.instant())
projections += projectionOf(next, mapper)
}
if (events.isNotEmpty()) msgEvents.insertAll(events)
// 终态与回填待办跟业务数据同事务提交:要么全成,要么全回滚
procState.markTerminal(head.msgId, ProcStatus.SUCCEEDED, now = clock.instant())
written
DomainOutcome(written, projections)
}
logSnapshot(head, body, SnapshotResult.COMMITTED, upserted, flags, started)
ApplyResult.Succeeded
@@ -160,34 +148,6 @@ class ScheduleProcessor(
}
}
/** 每次航班状态变化登记两个事件:KAFKA_SCHD 发完整状态,KAFKA_MSG 发一条变更通知。 */
private fun snapshotEvents(next: FlightSnapshot): List<MsgEvent> {
val createdAt = clock.instant()
val payload = linkedMapOf<String, Any>(
"flid" to next.flid,
"stateVersion" to next.stateVersion,
"scalars" to next.scalars,
"collections" to next.collections,
)
val notify = mapper.writeValueAsString(mapOf("flid" to next.flid, "stateVersion" to next.stateVersion))
return listOf(
MsgEvent(
target = Targets.KAFKA_SCHD,
partitionKey = next.flid,
stateVersion = next.stateVersion,
payloadJson = mapper.writeValueAsString(payload),
createdAt = createdAt,
),
MsgEvent(
target = Targets.KAFKA_MSG,
partitionKey = next.flid,
stateVersion = next.stateVersion,
payloadJson = notify,
createdAt = createdAt,
),
)
}
/**
* 写一条处理留痕(仅供排查和统计,不参与业务判断)。放在事务外做,
* 写失败也只记日志,不会连累处理结果。scope 是这份报文覆盖的运营日范围。
+3
View File
@@ -35,6 +35,9 @@ msgx:
arrived-hours: 1 # 条件 5:实际到港早于当前超过 1 小时
snap-log-retention-days: 90
history-store-enabled: false # 未接通历史存储时 HISTORY_SWEEP 删 0 条
redis: # 航班查询投影(C-11);真实客户端未接通(G-REDIS-PROJECTION
enabled: false # 关闭时投影写是空操作;开启但无客户端时写入直接失败,不伪装成功
flight-key: flightInfo # 航班快照哈希键,field = FLID(沿用旧系统)
micronaut:
application:
@@ -0,0 +1,16 @@
-- =====================================================================
-- MSG_EVENT 暂存放行:新增 MSG_ID 归属列
-- ---------------------------------------------------------------------
-- 领域变更与待发事件同一次提交(INV-3),但 Redis 投影写成功之前不能发 KafkaINV-10)。
-- 事件因此先以 STATE='HELD' 落库,终态事务按 MSG_ID 放行成 'PENDING';投递只领 'PENDING'。
-- 归属列也是重处理的清残依据:崩溃或投影失败后重跑,先删同一 MSG_ID 的 HELD 行再重新登记,
-- 既不留悬挂行,也不把同一次变更投两遍(docs/implementation.md「事务边界」)。
-- 只作用于自有 PostgreSQL,共享 MySQL 信箱不受影响(C-2)。
-- =====================================================================
ALTER TABLE msg_event ADD COLUMN msg_id BIGINT;
COMMENT ON COLUMN msg_event.msg_id IS '登记该事件的信箱消息 ID;HELD 行按它清理与放行,非管道写入方可为空';
-- 暂存行按消息清理/放行,行数远小于全表,故只索引 HELD。
CREATE INDEX idx_evt_held ON msg_event (msg_id) WHERE state = 'HELD';
@@ -21,7 +21,9 @@ Flyway 配置**PG 路径使用 `classpath:db/migration`,两者互不混用
- `INSERT ... ON CONFLICT` 改 11g MERGEOPERATION_DAY 不可变条件,implementation「航班身份与运营日」)。
- 空串按 NULL 的语义回归:显式清空的 presence 信息不得被 11g 空串语义吞掉
(增量报文未携带不清空见 `US-04` AC2,日计划清除语义见 `C-6`)。
- 11g 无部分索引:`uq_req_open` / `uq_schd_event` / `idx_proc_backfill_due` 三个带 `WHERE`
的索引必须换成等价的函数索引或冗余列方案,不能照搬 PG 定义。
- 11g 无部分索引:`uq_req_open` / `uq_schd_event` / `idx_proc_backfill_due` / `idx_evt_held`
四个带 `WHERE` 的索引必须换成等价的函数索引或冗余列方案,不能照搬 PG 定义。
- 11g 无 `pg_advisory_lock`:跨事务持 `PIPELINE_LOCK` 的会话锁须另找等价物
`DBMS_LOCK` 或独占连接上的长事务),语义见 `docs/implementation.md`「事务边界」。
版本号与 PG location 各自独立推进,禁止复用版本号语义。
@@ -11,10 +11,10 @@ import java.sql.DriverManager
/**
* 在真实 PostgreSQL 上跑一遍迁移链,确认结果符合预期:`V1__flight_state_baseline.sql`
* 基线加 `V2__flight_chute_class_type_rename.sql` 更名依序执行成功,该建的表和 PIPELINE_LOCK
* 单行种子都在,回填事实落在基线里,`INBOX_CURSOR`、`BACKFILL_TODO`、
* `idx_evt_flid`、`PROC_STATE` 的处理开始时间列都不复存在;FLIGHT_CHUTE 的类字段列
* 已由 V2 更名为 CCLS/CTYPSIS 口径)。
* 基线加 `V2__flight_chute_class_type_rename.sql` 更名、`V3__msg_event_hold.sql` 事件归属列
* 依序执行成功,该建的表和 PIPELINE_LOCK 单行种子都在,回填事实落在基线里,`INBOX_CURSOR`、
* `BACKFILL_TODO`、`idx_evt_flid`、`PROC_STATE` 的处理开始时间列都不复存在;FLIGHT_CHUTE
* 的类字段列已由 V2 更名为 CCLS/CTYPSIS 口径)。
*
* 没有可用的 PostgreSQL 时跳过(不假装通过)。
*/
@@ -51,11 +51,13 @@ class FlywayMigrationTest {
while (rs.next()) {
records.add(Triple(rs.getString("version"), rs.getString("script"), rs.getBoolean("success")))
}
assertEquals(2, records.size, "迁移链应为 V1 基线 + V2 更名")
assertEquals(3, records.size, "迁移链应为 V1 基线 + V2 更名 + V3 事件归属三")
assertEquals("1", records[0].first)
assertEquals("V1__flight_state_baseline.sql", records[0].second)
assertEquals("2", records[1].first)
assertEquals("V2__flight_chute_class_type_rename.sql", records[1].second)
assertEquals("3", records[2].first)
assertEquals("V3__msg_event_hold.sql", records[2].second)
assertTrue(records.all { it.third })
}
@@ -183,6 +185,21 @@ class FlywayMigrationTest {
assertEquals(0, rs.getInt(1), "基线必须不含已无消费者的 idx_evt_flid")
}
// V3:事件归属列与暂存行的部分索引(HELD 按 MSG_ID 清理与放行)
stmt.executeQuery(
"SELECT count(*) FROM information_schema.columns WHERE table_name = 'msg_event' " +
"AND column_name = 'msg_id'",
).use { rs ->
assertTrue(rs.next())
assertEquals(1, rs.getInt(1), "V3 必须加上 MSG_EVENT.MSG_ID")
}
stmt.executeQuery(
"SELECT indexdef FROM pg_indexes WHERE tablename = 'msg_event' AND indexname = 'idx_evt_held'",
).use { rs ->
assertTrue(rs.next(), "V3 必须建立暂存行索引 idx_evt_held")
assertTrue(rs.getString(1).contains("WHERE"), "idx_evt_held 只索引 HELD 行")
}
// 基线:投递确认时间 SENT_AT 是保留期判定的唯一基准
stmt.executeQuery(
"SELECT count(*) FROM information_schema.columns WHERE table_name = 'msg_event' " +
@@ -69,6 +69,34 @@ class JdbcMsgEventUpsertPgTest {
assertNull(row.lastError)
}
@Test
fun `held events are invisible to delivery until released and stale holds are discarded`() {
val repo = JdbcMsgEventRepository(dataSource(), Clock.systemUTC())
val key = "PG-" + UUID.randomUUID().toString().take(8)
val msgId = System.nanoTime()
repo.insertAll(listOf(held(msgId, key, Targets.KAFKA_MSG, 1), held(msgId, key, Targets.KAFKA_SCHD, 1)))
assertTrue(repo.claimBatch(Targets.KAFKA_MSG, 1000).none { it.partitionKey == key }, "HELD 不进投递批")
assertTrue(pendingSchd(repo, key).isEmpty(), "HELD 的整态行也领不到")
// 重处理:上一轮的暂存行先清掉,再重新登记
assertEquals(2, repo.discardHeld(msgId))
repo.insertAll(listOf(held(msgId, key, Targets.KAFKA_MSG, 2), held(msgId, key, Targets.KAFKA_SCHD, 2)))
assertEquals(2, repo.releaseHeld(msgId))
assertEquals(1, repo.claimBatch(Targets.KAFKA_MSG, 1000).count { it.partitionKey == key })
assertEquals(2, pendingSchd(repo, key).single().stateVersion)
assertEquals(0, repo.releaseHeld(msgId), "已放行的行不会被重复放行")
}
private fun held(msgId: Long, key: String, target: String, version: Long) =
MsgEvent(
msgId = msgId, target = target, partitionKey = key, stateVersion = version,
payloadJson = """{"v":$version}""", state = com.gzzn.omms.msgexchange.domain.EventStatus.HELD,
createdAt = now,
)
private fun pendingSchd(repo: JdbcMsgEventRepository, key: String): List<MsgEvent> =
repo.mergePendingSchd(now, 1000).filter { it.partitionKey == key }
@@ -0,0 +1,110 @@
package com.gzzn.omms.msgexchange.infra.persistence.jdbc
import com.gzzn.omms.msgexchange.support.PgTestSupport
import com.zaxxer.hikari.HikariConfig
import com.zaxxer.hikari.HikariDataSource
import org.flywaydb.core.Flyway
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Assertions.assertFalse
import org.junit.jupiter.api.Assertions.assertThrows
import org.junit.jupiter.api.Assertions.assertTrue
import org.junit.jupiter.api.Assumptions.assumeTrue
import org.junit.jupiter.api.Test
import java.util.UUID
import java.util.concurrent.CountDownLatch
import java.util.concurrent.TimeUnit
import javax.sql.DataSource
/**
* 在真实 PostgreSQL 上验证跨事务持锁:`holdAcrossTransactions` 把连接钉住并持会话级咨询锁,
* 块内的多次事务各自独立提交,另一个写者要等整段结束才拿得到锁
* docs/implementation.md「事务边界」)。拿不到 PG 时跳过,不假装通过。
*/
class JdbcPipelineLockPgTest {
@Test
fun `each transaction inside one hold commits on its own`() {
val ds = dataSource()
val lock = JdbcPipelineLockRepository(ds)
val tx = JdbcPipelineTransactionManager(ds)
val key = "LOCK-" + UUID.randomUUID().toString().take(8)
// 第一段提交后中途失败:模拟"领域已提交、投影写挂了"
assertThrows(IllegalStateException::class.java) {
lock.holdAcrossTransactions {
tx.inTransaction {
lock.lock()
insertHeldEvent(ds, key)
}
error("projection-down")
}
}
assertEquals(1, countEvents(ds, key), "已提交的第一段不因后续失败而回滚")
}
@Test
fun `a second writer waits until the whole hold is released`() {
val ds = dataSource()
val lock = JdbcPipelineLockRepository(ds)
val tx = JdbcPipelineTransactionManager(ds)
val holding = CountDownLatch(1)
val release = CountDownLatch(1)
val secondAcquired = CountDownLatch(1)
val holder = Thread.ofPlatform().daemon(true).start {
lock.holdAcrossTransactions {
tx.inTransaction { lock.lock() } // 第一段事务提交,行锁随之释放
holding.countDown()
release.await()
}
}
val contender = Thread.ofPlatform().daemon(true).unstarted {
tx.inTransaction {
lock.lock()
secondAcquired.countDown()
}
}
try {
assertTrue(holding.await(10, TimeUnit.SECONDS), "持锁方必须先进入")
contender.start()
assertFalse(secondAcquired.await(500, TimeUnit.MILLISECONDS), "行锁已释放,但整段仍由咨询锁挡着")
release.countDown()
assertTrue(secondAcquired.await(10, TimeUnit.SECONDS), "整段结束后第二个写者应立刻拿到锁")
} finally {
release.countDown()
holder.join(5_000)
contender.join(5_000)
}
}
private fun insertHeldEvent(ds: DataSource, key: String) {
ds.update(
"INSERT INTO msg_event (target, partition_key, event_type, state_version, payload_json, state, attempts, created_at, msg_id) " +
"VALUES ('KAFKA:msg', ?, 'UPSERT', 1, '{}', 'HELD', 0, now(), 1)",
{ ps -> ps.setString(1, key) },
)
}
private fun countEvents(ds: DataSource, key: String): Int =
ds.queryOne("SELECT count(*) AS n FROM msg_event WHERE partition_key = ?", { ps -> ps.setString(1, key) }) {
it.getInt("n")
} ?: 0
private fun dataSource(): DataSource {
assumeTrue(PgTestSupport.canConnect(), PgTestSupport.skipMessage())
Flyway.configure()
.dataSource(PgTestSupport.jdbcUrl, PgTestSupport.user, PgTestSupport.password)
.locations("classpath:db/migration")
.load()
.migrate()
return HikariDataSource(
HikariConfig().apply {
jdbcUrl = PgTestSupport.jdbcUrl
username = PgTestSupport.user
password = PgTestSupport.password
maximumPoolSize = 4
},
)
}
}
@@ -63,8 +63,9 @@ class FdelAndAdftProcessorTest {
val proc = StubProcState()
proc.insertIfAbsent(msgId, null)
val result = FdelProcessor(StubPipelineTx(), StubPipelineLock(), f, events, proc, ObjectMapper(), java.time.Clock.systemUTC())
.apply(head(), msg(), FlopPayload("121"))
val result = FdelProcessor(
testCommit(procState = proc, msgEvents = events), f, events, ObjectMapper(), java.time.Clock.systemUTC(),
).apply(head(), msg(), FlopPayload("121"))
assertEquals(ApplyResult.Succeeded, result)
assertEquals(FlightState.DELETED, f.findMainRow("121")!!.state)
@@ -84,11 +85,15 @@ class FdelAndAdftProcessorTest {
fun `repeated FDEL is idempotent without version bump or duplicate event`() {
val f = flights()
val events = StubMsgEvents()
val proc = FdelProcessor(StubPipelineTx(), StubPipelineLock(), f, events, StubProcState(), ObjectMapper(), java.time.Clock.systemUTC())
val proc = FdelProcessor(
testCommit(msgEvents = events), f, events, ObjectMapper(), java.time.Clock.systemUTC(),
)
proc.apply(head(), msg(), FlopPayload("121"))
val version = f.findMainRow("121")!!.stateVersion
val result = proc.apply(head(), msg(), FlopPayload("121"))
// 第二份删除报文是另一条消息(同一条消息到终态后不会再进主泵)
val later = ProcState(msgId + 1, ProcStatus.PENDING, updatedAt = java.time.Instant.EPOCH)
val result = proc.apply(later, msg(), FlopPayload("121"))
assertEquals(ApplyResult.Succeeded, result) // 重复的删除报文算成功,但不再动数据
assertEquals(version, f.findMainRow("121")!!.stateVersion)
@@ -100,8 +105,9 @@ class FdelAndAdftProcessorTest {
val f = StubFlightState()
val events = StubMsgEvents()
val result = FdelProcessor(StubPipelineTx(), StubPipelineLock(), f, events, StubProcState(), ObjectMapper(), java.time.Clock.systemUTC())
.apply(head(), msg(), FlopPayload("999"))
val result = FdelProcessor(
testCommit(msgEvents = events), f, events, ObjectMapper(), java.time.Clock.systemUTC(),
).apply(head(), msg(), FlopPayload("999"))
assertEquals(ApplyResult.Succeeded, result) // 航班不存在(迟到或多余)也算成功
assertEquals(0, events.rows.size)
@@ -113,7 +119,7 @@ class FdelAndAdftProcessorTest {
f.markDeleted("121", msgId = 2, now = java.time.Instant.now())
val events = StubMsgEvents()
val adft = AdftProcessor(
StubPipelineTx(), StubPipelineLock(), f, events, StubProcState(),
testCommit(msgEvents = events), f, events,
OperationDayProps().apply { zone = "Asia/Shanghai" }, ObjectMapper(), java.time.Clock.systemUTC(),
)
val record = com.gzzn.omms.msgexchange.domain.flight.ScheduleRecord(
@@ -134,7 +140,7 @@ class FdelAndAdftProcessorTest {
val f = StubFlightState()
val events = StubMsgEvents()
val adft = AdftProcessor(
StubPipelineTx(), StubPipelineLock(), f, events, StubProcState(),
testCommit(msgEvents = events), f, events,
OperationDayProps().apply { zone = "Asia/Shanghai" }, ObjectMapper(), java.time.Clock.systemUTC(),
)
@@ -160,7 +166,7 @@ class FdelAndAdftProcessorTest {
msgId = 1, now = java.time.Instant.now(),
)
val adft = AdftProcessor(
StubPipelineTx(), StubPipelineLock(), f, StubMsgEvents(), StubProcState(),
testCommit(), f, StubMsgEvents(),
OperationDayProps().apply { zone = "Asia/Shanghai" }, ObjectMapper(), java.time.Clock.systemUTC(),
)
@@ -0,0 +1,263 @@
package com.gzzn.omms.msgexchange.processing
import com.fasterxml.jackson.databind.ObjectMapper
import com.gzzn.omms.msgexchange.MutableClock
import com.gzzn.omms.msgexchange.codec.DecodeResult
import com.gzzn.omms.msgexchange.codec.FlopPayload
import com.gzzn.omms.msgexchange.codec.XmlCodec
import com.gzzn.omms.msgexchange.config.OperationDayProps
import com.gzzn.omms.msgexchange.config.PipelineProps
import com.gzzn.omms.msgexchange.delivery.Dispatcher
import com.gzzn.omms.msgexchange.domain.DecodedMessage
import com.gzzn.omms.msgexchange.domain.ErrorClass
import com.gzzn.omms.msgexchange.domain.EventStatus
import com.gzzn.omms.msgexchange.domain.MetaFields
import com.gzzn.omms.msgexchange.domain.MsgKind
import com.gzzn.omms.msgexchange.domain.ProcState
import com.gzzn.omms.msgexchange.domain.ProcStatus
import com.gzzn.omms.msgexchange.domain.Targets
import com.gzzn.omms.msgexchange.domain.flight.FlightSnapshot
import com.gzzn.omms.msgexchange.domain.flight.FlightState
import com.gzzn.omms.msgexchange.infra.metrics.PipelineCounters
import com.gzzn.omms.msgexchange.infra.persistence.PipelineLockRepository
import com.gzzn.omms.msgexchange.infra.persistence.PipelineTransactionManager
import com.gzzn.omms.msgexchange.infra.retry.FailureScheduler
import com.gzzn.omms.msgexchange.infra.retry.ProcFailure
import com.gzzn.omms.msgexchange.infra.stub.StubDeliveryPort
import com.gzzn.omms.msgexchange.infra.stub.StubFlightProjectionPort
import com.gzzn.omms.msgexchange.infra.stub.StubFlightState
import com.gzzn.omms.msgexchange.infra.stub.StubInbox
import com.gzzn.omms.msgexchange.infra.stub.StubMsgEvents
import com.gzzn.omms.msgexchange.infra.stub.StubProcState
import com.gzzn.omms.msgexchange.infra.stub.StubSnapshotLog
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Assertions.assertFalse
import org.junit.jupiter.api.Assertions.assertNotNull
import org.junit.jupiter.api.Assertions.assertNull
import org.junit.jupiter.api.Assertions.assertThrows
import org.junit.jupiter.api.Assertions.assertTrue
import org.junit.jupiter.api.Test
import java.time.Duration
import java.time.LocalDate
/**
* 三步提交(`INV-3`、`INV-10`)的故障注入:
* 投影写不成功时,已提交的航班数据不撤销,但这条消息不进终态、不记回填意图,
* 待发事件停在 `HELD` 发不出去;恢复后重处理只留一次效果。
*/
class FlightCommitTest {
private val clock = MutableClock(MutableClock.BASE)
private val mapper = ObjectMapper()
private val msgId = 21L
private val flid = "121"
private fun head() = ProcState(msgId, ProcStatus.PENDING, updatedAt = MutableClock.BASE)
/** 处理器不读报文体以外的内容,几个用例共用这一条 FDEL 形状的解码结果。 */
private fun msg() = DecodedMessage(
meta = MetaFields("AODB", "FDEL", "", 9L, 1L),
kind = MsgKind.Fdel,
rawXml = "<MSG/>",
body = FlopPayload(flid),
)
/** 一套内存装置:航班 121 已在库、版本 1。 */
private inner class Fixture {
val flights = StubFlightState()
val events = StubMsgEvents()
val procState = StubProcState()
val projection = StubFlightProjectionPort()
init {
flights.persistFullState(
FlightSnapshot(flid, LocalDate.of(2026, 12, 15), FlightState.ACTIVE, 1, mapOf("FLNO" to "CA001"), emptyMap()),
msgId = 1,
now = MutableClock.BASE,
)
procState.insertIfAbsent(msgId, MutableClock.BASE)
}
/** 每次新建处理器:等价于重启后由新进程接着处理同一条消息。 */
fun flop() = FlopProcessor(commit(), flights, events, mapper, clock)
fun fdel() = FdelProcessor(commit(), flights, events, mapper, clock)
private fun commit() = testCommit(
procState = procState, msgEvents = events, projection = projection, clock = clock,
)
}
@Test
fun `projection failure keeps committed flight data but leaves the message unfinished`() {
val f = Fixture()
f.projection.failWith = IllegalStateException("redis-down")
assertThrows(FlightProjectionFailure::class.java) { f.flop().apply(head(), msg(), FlopPayload(flid)) }
// 已提交的 PG 不因投影失败而撤销(US-03 AC3)
assertEquals(2L, f.flights.findMainRow(flid)!!.stateVersion)
// 不记终态、不记回填意图(INV-10)
val row = f.procState.find(msgId)!!
assertEquals(ProcStatus.PENDING, row.state)
assertNull(row.backfillNextAt)
// 事件登记了但发不出去:投递只领 PENDING
assertTrue(f.events.rows.values.all { it.state == EventStatus.HELD })
assertTrue(f.events.claimBatch(Targets.KAFKA_MSG, 10).isEmpty())
assertTrue(f.events.mergePendingSchd(clock.instant(), 10).isEmpty())
}
@Test
fun `reprocessing after recovery releases each event exactly once`() {
val f = Fixture()
f.projection.failWith = IllegalStateException("redis-down")
assertThrows(FlightProjectionFailure::class.java) { f.flop().apply(head(), msg(), FlopPayload(flid)) }
f.projection.failWith = null
val result = f.flop().apply(head(), msg(), FlopPayload(flid)) // 新处理器 = 重启后重处理
assertEquals(ApplyResult.Succeeded, result)
val row = f.procState.find(msgId)!!
assertEquals(ProcStatus.SUCCEEDED, row.state)
assertNotNull(row.backfillNextAt)
// 上一轮的暂存行先清后补:通知不重复,整态仍是每个 FLID 一行
assertEquals(1, f.events.rows.values.count { it.target == Targets.KAFKA_MSG })
assertEquals(1, f.events.rows.values.count { it.target == Targets.KAFKA_SCHD })
assertTrue(f.events.rows.values.all { it.state == EventStatus.PENDING })
assertEquals(3L, f.flights.findMainRow(flid)!!.stateVersion) // 领域再跑一次,版本推进
assertTrue(f.projection.snapshots.containsKey(flid))
}
@Test
fun `FDEL removes the flight from the projection`() {
val f = Fixture()
f.projection.snapshots[flid] = """{"flid":"121"}"""
val result = f.fdel().apply(head(), msg(), FlopPayload(flid))
assertEquals(ApplyResult.Succeeded, result)
assertFalse(f.projection.snapshots.containsKey(flid)) // INV-8:标删即从投影移除
assertTrue(f.projection.writes.single() is FlightProjectionWrite.Delete)
assertEquals(ProcStatus.SUCCEEDED, f.procState.find(msgId)!!.state)
}
@Test
fun `projection failure on FDEL keeps the deletion committed and the projection entry untouched`() {
val f = Fixture()
f.projection.snapshots[flid] = """{"flid":"121"}"""
f.projection.failWith = IllegalStateException("redis-down")
assertThrows(FlightProjectionFailure::class.java) { f.fdel().apply(head(), msg(), FlopPayload(flid)) }
assertEquals(FlightState.DELETED, f.flights.findMainRow(flid)!!.state)
assertTrue(f.projection.snapshots.containsKey(flid)) // 投影还在,下轮重删
assertEquals(ProcStatus.PENDING, f.procState.find(msgId)!!.state)
f.projection.failWith = null
f.fdel().apply(head(), msg(), FlopPayload(flid))
assertFalse(f.projection.snapshots.containsKey(flid))
assertEquals(ProcStatus.SUCCEEDED, f.procState.find(msgId)!!.state)
// 重复的 FDEL 不再推进版本,也不重复登记删除通知
assertEquals(1, f.events.rows.size)
}
@Test
fun `lock is held from domain commit through projection write to terminal transaction`() {
val trace = mutableListOf<String>()
val flights = StubFlightState()
flights.persistFullState(
FlightSnapshot(flid, LocalDate.of(2026, 12, 15), FlightState.ACTIVE, 1, mapOf("FLNO" to "CA001"), emptyMap()),
msgId = 1, now = MutableClock.BASE,
)
val events = StubMsgEvents()
val commit = FlightCommit(
txManager = TracingTx(trace),
lock = TracingLock(trace),
procState = StubProcState(),
msgEvents = events,
projection = TracingProjection(trace),
clock = clock,
)
FlopProcessor(commit, flights, events, mapper, clock).apply(head(), msg(), FlopPayload(flid))
assertEquals(
listOf("hold", "tx-begin", "lock", "tx-end", "projection", "tx-begin", "lock", "tx-end", "release"),
trace,
)
}
@Test
fun `main pump records a retryable failure and delivers nothing when the projection is down`() {
val f = Fixture()
f.projection.failWith = IllegalStateException("redis-down")
val inbox = StubInbox()
inbox.raws[msgId] = "<RAW/>"
val props = PipelineProps()
val opDay = OperationDayProps().apply { zone = "Asia/Shanghai"; cutoffHour = 0 }
val commit = testCommit(procState = f.procState, msgEvents = f.events, projection = f.projection, clock = clock)
val processor = MessageProcessor(
inbox = inbox,
procState = f.procState,
codec = codecReturning(msg()),
scheduleProcessor = ScheduleProcessor(commit, f.procState, f.flights, f.events, StubSnapshotLog(), opDay, mapper, clock),
flopProcessor = FlopProcessor(commit, f.flights, f.events, mapper, clock),
fdelProcessor = FdelProcessor(commit, f.flights, f.events, mapper, clock),
adftProcessor = AdftProcessor(commit, f.flights, f.events, opDay, mapper, clock),
procFailure = ProcFailure(f.procState, FailureScheduler(props, clock)),
props = props,
clock = clock,
operationDayProps = opDay,
counters = PipelineCounters(),
)
processor.processOne(head())
val row = f.procState.find(msgId)!!
assertEquals(ProcStatus.FAILED, row.state) // 可重试失败,不是终态
assertEquals(ErrorClass.INFRA, row.errorClass)
assertNull(row.backfillNextAt) // 没有回填意图,信箱标记不会被写回
val port = StubDeliveryPort()
val deliveryProps = PipelineProps().apply { pipeline.pollInterval = Duration.ofMillis(1) }
Dispatcher(f.events, port, deliveryProps, FailureScheduler(deliveryProps, clock), PipelineCounters(), mapper).tick()
assertTrue(port.sent.isEmpty()) // 处理完成前不发 KafkaINV-3
}
private fun codecReturning(decoded: DecodedMessage) = object : XmlCodec {
override fun decode(rawXml: String): DecodeResult = DecodeResult.Ok(decoded)
override fun encodeRqrd(kind: String, rangeJson: String): String = ""
}
private class TracingTx(private val trace: MutableList<String>) : PipelineTransactionManager {
override fun <T> inTransaction(block: () -> T): T {
trace += "tx-begin"
val result = block()
trace += "tx-end"
return result
}
}
private class TracingLock(private val trace: MutableList<String>) : PipelineLockRepository {
override fun lock() {
trace += "lock"
}
override fun <T> holdAcrossTransactions(block: () -> T): T {
trace += "hold"
try {
return block()
} finally {
trace += "release"
}
}
}
private class TracingProjection(private val trace: MutableList<String>) : FlightProjectionPort {
override fun write(writes: List<FlightProjectionWrite>) {
trace += "projection"
}
}
}
@@ -50,11 +50,6 @@ class IgnoreBranchTest {
override fun encodeRqrd(kind: String, rangeJson: String): String = ""
}
private val noopTx = object : PipelineTransactionManager {
override fun <T> inTransaction(block: () -> T): T = block()
}
private val noopLock = object : PipelineLockRepository { override fun lock() = Unit }
private fun processor(
proc: StubProcState = StubProcState(),
inbox: StubInbox = StubInbox(),
@@ -68,14 +63,15 @@ class IgnoreBranchTest {
val events = StubMsgEvents()
val log = StubSnapshotLog()
val procFailure = ProcFailure(proc, FailureScheduler(props, clock))
val commit = testCommit(procState = proc, msgEvents = events, clock = clock)
return MessageProcessor(
inbox = inbox,
procState = proc,
codec = codec,
scheduleProcessor = ScheduleProcessor(noopTx, noopLock, proc, flights, events, log, opDay, ObjectMapper(), clock),
flopProcessor = FlopProcessor(noopTx, noopLock, flights, events, proc, ObjectMapper(), clock),
fdelProcessor = FdelProcessor(noopTx, noopLock, flights, events, proc, ObjectMapper(), clock),
adftProcessor = AdftProcessor(noopTx, noopLock, flights, events, proc, opDay, ObjectMapper(), clock),
scheduleProcessor = ScheduleProcessor(commit, proc, flights, events, log, opDay, ObjectMapper(), clock),
flopProcessor = FlopProcessor(commit, flights, events, ObjectMapper(), clock),
fdelProcessor = FdelProcessor(commit, flights, events, ObjectMapper(), clock),
adftProcessor = AdftProcessor(commit, flights, events, opDay, ObjectMapper(), clock),
procFailure = procFailure,
props = props,
clock = clock,
@@ -15,10 +15,7 @@ import com.gzzn.omms.msgexchange.domain.flight.FlightState
import com.gzzn.omms.msgexchange.infra.persistence.FlightStateRepository
import com.gzzn.omms.msgexchange.infra.persistence.MsgEventRepository
import com.gzzn.omms.msgexchange.infra.persistence.PersistOutcome
import com.gzzn.omms.msgexchange.infra.persistence.PipelineLockRepository
import com.gzzn.omms.msgexchange.infra.persistence.PipelineTransactionManager
import com.gzzn.omms.msgexchange.infra.persistence.ProcStateRepository
import com.gzzn.omms.msgexchange.infra.persistence.SnapshotLogRepository
import com.gzzn.omms.msgexchange.infra.stub.StubFlightState
import com.gzzn.omms.msgexchange.infra.stub.StubMsgEvents
import com.gzzn.omms.msgexchange.infra.stub.StubProcState
@@ -75,8 +72,7 @@ class ScheduleProcessorTest {
override fun <T> inTransaction(block: () -> T): T = block()
}
return ScheduleProcessor(
txManager = txRunner,
lock = object : PipelineLockRepository { override fun lock() = Unit },
commit = testCommit(procState = proc, msgEvents = events, txManager = txRunner),
procState = proc,
flightState = flights,
msgEvents = events,
@@ -0,0 +1,25 @@
package com.gzzn.omms.msgexchange.processing
import com.gzzn.omms.msgexchange.infra.persistence.MsgEventRepository
import com.gzzn.omms.msgexchange.infra.persistence.PipelineLockRepository
import com.gzzn.omms.msgexchange.infra.persistence.PipelineTransactionManager
import com.gzzn.omms.msgexchange.infra.persistence.ProcStateRepository
import com.gzzn.omms.msgexchange.infra.stub.StubFlightProjectionPort
import com.gzzn.omms.msgexchange.infra.stub.StubMsgEvents
import com.gzzn.omms.msgexchange.infra.stub.StubPipelineLock
import com.gzzn.omms.msgexchange.infra.stub.StubPipelineTx
import com.gzzn.omms.msgexchange.infra.stub.StubProcState
import java.time.Clock
/**
* 处理器用例共用的三步提交装配:默认全内存、投影不失败。
* 事件仓储要和处理器用同一个实例,否则放行(`HELD` → `PENDING`)落不到处理器写的那些行上。
*/
internal fun testCommit(
procState: ProcStateRepository = StubProcState(),
msgEvents: MsgEventRepository = StubMsgEvents(),
projection: FlightProjectionPort = StubFlightProjectionPort(),
lock: PipelineLockRepository = StubPipelineLock(),
txManager: PipelineTransactionManager = StubPipelineTx(),
clock: Clock = Clock.systemUTC(),
) = FlightCommit(txManager, lock, procState, msgEvents, projection, clock)