go-caatsm/docs/reliability.md

## Reliability and Fault Handling

This document summarises how the service handles failures, provides resilience, and avoids data loss or duplication.

### Dead-Letter Queue (DLQ) and Poison Messages

Configuration is defined under `dlq`:

```toml
[dlq]
enabled = true
subject = "caatsm.dlq"
```

- When `dlq.enabled` is `true` and `dlq.subject` is non-empty, **permanent** failures are routed to the DLQ subject.  
- A permanent failure is indicated by wrapping an error with `app.Permanent` and is treated as a **poison message**.

Behaviour:

1. The NATS consumer calls `processor.Handle`.  
2. If an error is returned and `app.IsPermanent(err)` is true:
   - The original NATS message is copied into a DLQ payload with metadata:
     - `transport_msg_id` (NATS message ID)  
     - `subject`, `stream`, `consumer`  
     - `nats_sequence`, `deliveries`  
     - `error` (stringified cause)  
     - `received_at` (DLQ event time)  
     - `body` (raw message body)
   - The payload is published to `dlq.subject` using JetStream.  
   - The original message is **ACKed**, so it will not be redelivered.

The DLQ subject should be consumed by an offline repair/analysis tool or operational dashboard that can:

- Inspect poison messages.  
- Decide whether to fix and re-publish, or discard with justification.  
- Track DLQ volume over time for alerting (e.g. via `caatsm_dlq_messages_total`
  and `caatsm_dlq_publish_failures_total` metrics).

### Transient Errors and Backoff

Transient errors (not marked permanent) result in:

- Negative acknowledgements with delay (`NakWithDelay`) according to `nats.consumer_rules.backoff`.  
- A retry streak counter inside the consumer:
  - Each transient error increases `consecutiveProcessErrors`.  
  - After 10 or more consecutive errors, the consumer applies an additional **sleep**:
    - `backoff = min(consecutive_errors * 100ms, 5s)`.  
    - A warning log with the sleep duration and error count is emitted.

This combination provides **backpressure** when downstream systems (especially the DB) are in trouble, slowing down consumption instead of aggressively retrying.

### Persistence and Idempotency

The primary persistence path is `Repository.InsertOne` into `aviation.telegrams`. To avoid applying the same business event multiple times, a **minimal idempotency check** is implemented:

- If both `message_id` and `date_time` are non-empty:
  - `InsertOne` first calls `messageExists(message_id, date_time)`.  
  - If a row already exists, the insert is **skipped** and an informational log is written.  
  - Otherwise, the insert proceeds.

This makes repeated delivery of the same telegram (same `message_id`/`date_time`) safe from a business perspective, even if JetStream redelivers messages or upstream replays.

For higher guarantees in production environments, you may:

- Add a unique index on `(message_id, date_time)` at the DB level, and treat any conflict as a duplicate.  
- Extend the idempotency key with additional fields (e.g. originator, category) if required by the business model.

### DB Degradation and Backpressure

Database write failures in `Repository.InsertOne` and related methods are treated as **transient** by default:

- Errors propagate back to the NATS consumer.  
- The consumer issues a NAK (with delay) and increases the transient error counter.  
- When errors persist, the added sleep in the consumer reduces message throughput and gives the DB time to recover.

DB health also feeds into readiness:

- The monitoring server hits `pgxpool.Pool.Ping` on `/readyz` and `/healthz`.  
- If the DB is not reachable, the endpoints return `503`, signalling to orchestrators that this instance should be drained from traffic.

Together, this yields:

- **Backpressure** via reduced consumption rate and NATS-level backoff.  
- **Degradation signalling** via health probes for external systems to act upon.

### Retry and Max Deliver

JetStream consumer configuration (via `nats.consumer_rules`) controls:

- `max_deliver` – maximum number of redeliveries before JetStream gives up.  
- `ack_wait` – how long JetStream waits for an ACK before considering the message pending.  
- `backoff` – per-attempt delays for `NakWithDelay`.

Recommended pattern:

- Keep `max_deliver` modest (e.g. 5).  
- Use a backoff array such as `[5s, 30s, 2m]`.  
- Treat messages that still fail after `max_deliver` as candidates for DLQ, via the permanent error/poison message path where applicable.

### JetStream Availability and Auto-Recovery (Dev vs Prod)

- When the JetStream API is temporarily unavailable (for example, NATS has just
  restarted and returns `ErrNoResponders`), the consumer uses an exponential
  backoff when retrying `Fetch` calls (roughly `1s, 2s, 4s, ...` up to
  around `30s`) to avoid log spam while allowing the system to recover.
- At startup the receiver always calls `StreamManager.EnsureStream` and
  `ConsumerManager.EnsureConsumer`. If the JetStream account allows it,
  missing streams are created with the configured retention limits
  (`max_msgs`, `max_bytes`, `max_age`, discard/storage policy, replicas) before
  the durable consumer is created. This keeps dev/test clusters self-healing
  after `docker compose down -v` and removes the race where a consumer was
  created without its stream.
- When the JetStream account lacks permissions to create protected resources
  (a common production posture), the same code path fails fast with a clear
  error message so operators know they must provision the stream out-of-band.
- On the publishing side, JetStream `ErrNoResponders` and similar errors are
  treated as temporary by the processor:
  - Such errors cause the consumer to NAK messages and rely on the configured
    backoff for retries.
  - Permanent configuration/permission errors remain mapped to permanent
    failures and follow the DLQ + ACK flow.

### Alerts and Dashboards

Prometheus alert suggestions:

- High failure rate:
  - `rate(caatsm_messages_total{result!="ok"}[5m])` above a small threshold.  
  - `rate(caatsm_retries_total[5m])` above a threshold.

- DLQ growth:
  - Alerts on DLQ stream message count, using NATS/JetStream exporter metrics.

- Readiness / health:
  - Alert when `/readyz` fails or when DB/NATS checks start failing consistently.

Dashboards should combine:

- Message rates, error rates, and DLQ rates.  
- NATS consumer statistics (pending, redelivered, ack_pending).  
- DB health indicators (latency, error counts, connection usage).