## Observability

### Metrics

The service exposes Prometheus metrics via the monitoring HTTP server (default `:2112`):

- `caatsm_messages_total{stream,consumer,result}`  
  Total number of messages handled by the receiver, labelled by stream/consumer and result (`ok`, `fail`, `permanent_fail`, `retry`).

- `caatsm_handle_latency_seconds{stream,consumer}`  
  End-to-end handling latency from NATS receive to handler completion.

- `caatsm_retries_total{stream,consumer,reason}`  
  Number of retries (NAKs) issued by the consumer, labelled by reason (e.g. `processor_error`).

- `caatsm_js_api_calls_total{operation}`  
  JetStream API calls performed by the service.

- `caatsm_db_queries_total{operation,result}`  
  Database operations grouped by operation (`insert_one`, `insert_batch`, `insert_raw`) and result (`ok`, `error`).

- `caatsm_db_query_latency_seconds{operation}`  
  DB operation latency.

Additional OTEL metrics are emitted via the configured OTEL endpoint, including:

- `caatsm_messages_processed_total`  
- `caatsm_parse_duration_ms`  
- `caatsm_publish_failures_total`  
- `caatsm_nats_consumer_ack_pending`  
- `caatsm_nats_consumer_redelivered`  
- `caatsm_nats_consumer_pending`  
- `caatsm_nats_consumer_delivered`

These metrics are intended to be scraped by Prometheus (either directly or via the OTEL collector) and visualised in Grafana dashboards. Recommended dashboard panels include:

- Per-stream/consumer message rate and error rate.  
- Handling latency P50/P95/P99.  
- NATS consumer backlog and redelivery counts.  
- DB query rates and latencies.

#### CAATSM – Receiver Overview Dashboard

The `caatsm-overview` Grafana dashboard (provisioned from `configs/grafana-dashboards.dev/caatsm-overview.json`) focuses on the CAATSM receiver service and surfaces:

- **Message throughput by result** – derived from `caatsm_messages_total{result}`.  
- **Per stream/consumer rates** – `caatsm_messages_total{stream,consumer}`.  
- **End-to-end handle latency** – P50/P95/P99 from `caatsm_handle_latency_seconds_bucket`.  
- **DB query rate and latency** – from `caatsm_db_queries_total` and `caatsm_db_query_latency_seconds_bucket`.  
- **Retry and permanent failure rates** – from `caatsm_retries_total` and `caatsm_messages_total{result="permanent_fail"}`.  
- **Publish failures** – from `caatsm_publish_failures_total`.

### Health and Readiness

The monitoring server exposes:

- `/healthz` – basic liveness and dependency check.  
- `/readyz` – readiness endpoint with the same logic as `/healthz`, intended for load balancers / orchestrators.

Checks performed:

- PostgreSQL: `pgxpool.Pool.Ping` with configurable timeout (`monitoring.health_timeout`).  
- NATS: connection status must be `CONNECTED`.

A non-200 response indicates the service is not healthy/ready and should be removed from traffic.

### Tracing

Tracing is configured via the `telemetry` section:

- `telemetry.enabled` – enables OTEL exporters.  
- `telemetry.endpoint` – OTLP HTTP endpoint (e.g. `localhost:4318`).  
- `telemetry.insecure` – disables TLS for local/dev.

Key spans:

- `caatsm/nats`  
  - `Consumer.processMessage`
- `caatsm/app`  
  - `MessageProcessor.Handle`  
  - `Publisher.Publish`
- `caatsm/postgres`  
  - `Repository.InsertOne`  
  - `Repository.InsertBatch`  
  - `Repository.InsertRaw`

Important attributes:

- `nats.subject`, `nats.msg_id`, `nats.js.stream_seq`, `nats.js.consumer_seq`  
- `telegram.message_id`, `telegram.category`, `telegram.status`  
- `db.table`, `db.inserted`

### Structured Logging Contract

Logging is done with Zap. The `internal/observability/logging` package standardises fields via `MessageFields`:

- `service` – logical component (`caatsm-consumer`, `caatsm-processor` etc.).  
- `transport_msg_id` – NATS/envelope message ID (derived from `Nats-Msg-Id` or JetStream sequence).  
- `telegram_message_id` – business telegram message ID from the payload.  
- `category` – telegram category (ARR, DEP, FPL, etc.).  
- `stream`, `consumer`, `subject` – JetStream context.  
- `nats_sequence` – JetStream stream sequence, when available.  
- `request_id`, `trace_id` – correlation identifiers.  
- `error_type` – high-level classification:
  - `business` – payload/validation/domain issues; not suitable for retry.
  - `transient` – network/DB/NATS glitches that may succeed on retry.
  - `fatal` – programming errors, schema mismatches, or configuration issues requiring operator attention.

Handler and consumer logs should always be emitted through `WithMessageContext` to ensure these fields are present where applicable.