✨ Implement dead-letter queue (DLQ) functionality for handling permanent failures in message processing. Update configuration to enable DLQ and specify the subject for routing failed messages. Enhance observability by adding metrics for message handling, retries, and database operations. Introduce structured logging for better traceability of message processing events.
This commit is contained in:
windyboy
@@ -0,0 +1,111 @@
|
||||
## Observability
|
||||
|
||||
### Metrics
|
||||
|
||||
The service exposes Prometheus metrics via the monitoring HTTP server (default `:2112`):
|
||||
|
||||
- `caatsm_messages_total{stream,consumer,result}`
|
||||
Total number of messages handled by the receiver, labelled by stream/consumer and result (`ok`, `fail`, `permanent_fail`, `retry`).
|
||||
|
||||
- `caatsm_handle_latency_seconds{stream,consumer}`
|
||||
End-to-end handling latency from NATS receive to handler completion.
|
||||
|
||||
- `caatsm_retries_total{stream,consumer,reason}`
|
||||
Number of retries (NAKs) issued by the consumer, labelled by reason (e.g. `processor_error`).
|
||||
|
||||
- `caatsm_js_api_calls_total{operation}`
|
||||
JetStream API calls performed by the service.
|
||||
|
||||
- `caatsm_db_queries_total{operation,result}`
|
||||
Database operations grouped by operation (`insert_one`, `insert_batch`, `insert_raw`) and result (`ok`, `error`).
|
||||
|
||||
- `caatsm_db_query_latency_seconds{operation}`
|
||||
DB operation latency.
|
||||
|
||||
Additional OTEL metrics are emitted via the configured OTEL endpoint, including:
|
||||
|
||||
- `caatsm_messages_processed_total`
|
||||
- `caatsm_parse_duration_ms`
|
||||
- `caatsm_publish_failures_total`
|
||||
- `caatsm_nats_consumer_ack_pending`
|
||||
- `caatsm_nats_consumer_redelivered`
|
||||
- `caatsm_nats_consumer_pending`
|
||||
- `caatsm_nats_consumer_delivered`
|
||||
|
||||
These metrics are intended to be scraped by Prometheus (either directly or via the OTEL collector) and visualised in Grafana dashboards. Recommended dashboard panels include:
|
||||
|
||||
- Per-stream/consumer message rate and error rate.
|
||||
- Handling latency P50/P95/P99.
|
||||
- NATS consumer backlog and redelivery counts.
|
||||
- DB query rates and latencies.
|
||||
|
||||
#### CAATSM – Receiver Overview Dashboard
|
||||
|
||||
The `caatsm-overview` Grafana dashboard (provisioned from `configs/grafana-dashboards.dev/caatsm-overview.json`) focuses on the CAATSM receiver service and surfaces:
|
||||
|
||||
- **Message throughput by result** – derived from `caatsm_messages_total{result}`.
|
||||
- **Per stream/consumer rates** – `caatsm_messages_total{stream,consumer}`.
|
||||
- **End-to-end handle latency** – P50/P95/P99 from `caatsm_handle_latency_seconds_bucket`.
|
||||
- **DB query rate and latency** – from `caatsm_db_queries_total` and `caatsm_db_query_latency_seconds_bucket`.
|
||||
- **Retry and permanent failure rates** – from `caatsm_retries_total` and `caatsm_messages_total{result="permanent_fail"}`.
|
||||
- **Publish failures** – from `caatsm_publish_failures_total`.
|
||||
|
||||
### Health and Readiness
|
||||
|
||||
The monitoring server exposes:
|
||||
|
||||
- `/healthz` – basic liveness and dependency check.
|
||||
- `/readyz` – readiness endpoint with the same logic as `/healthz`, intended for load balancers / orchestrators.
|
||||
|
||||
Checks performed:
|
||||
|
||||
- PostgreSQL: `pgxpool.Pool.Ping` with configurable timeout (`monitoring.health_timeout`).
|
||||
- NATS: connection status must be `CONNECTED`.
|
||||
|
||||
A non-200 response indicates the service is not healthy/ready and should be removed from traffic.
|
||||
|
||||
### Tracing
|
||||
|
||||
Tracing is configured via the `telemetry` section:
|
||||
|
||||
- `telemetry.enabled` – enables OTEL exporters.
|
||||
- `telemetry.endpoint` – OTLP HTTP endpoint (e.g. `localhost:4318`).
|
||||
- `telemetry.insecure` – disables TLS for local/dev.
|
||||
|
||||
Key spans:
|
||||
|
||||
- `caatsm/nats`
|
||||
- `Consumer.processMessage`
|
||||
- `caatsm/app`
|
||||
- `MessageProcessor.Handle`
|
||||
- `Publisher.Publish`
|
||||
- `caatsm/postgres`
|
||||
- `Repository.InsertOne`
|
||||
- `Repository.InsertBatch`
|
||||
- `Repository.InsertRaw`
|
||||
|
||||
Important attributes:
|
||||
|
||||
- `nats.subject`, `nats.msg_id`, `nats.js.stream_seq`, `nats.js.consumer_seq`
|
||||
- `telegram.message_id`, `telegram.category`, `telegram.status`
|
||||
- `db.table`, `db.inserted`
|
||||
|
||||
### Structured Logging Contract
|
||||
|
||||
Logging is done with Zap. The `internal/observability/logging` package standardises fields via `MessageFields`:
|
||||
|
||||
- `service` – logical component (`caatsm-consumer`, `caatsm-processor` etc.).
|
||||
- `transport_msg_id` – NATS/envelope message ID (derived from `Nats-Msg-Id` or JetStream sequence).
|
||||
- `telegram_message_id` – business telegram message ID from the payload.
|
||||
- `category` – telegram category (ARR, DEP, FPL, etc.).
|
||||
- `stream`, `consumer`, `subject` – JetStream context.
|
||||
- `nats_sequence` – JetStream stream sequence, when available.
|
||||
- `request_id`, `trace_id` – correlation identifiers.
|
||||
- `error_type` – high-level classification:
|
||||
- `business` – payload/validation/domain issues; not suitable for retry.
|
||||
- `transient` – network/DB/NATS glitches that may succeed on retry.
|
||||
- `fatal` – programming errors, schema mismatches, or configuration issues requiring operator attention.
|
||||
|
||||
Handler and consumer logs should always be emitted through `WithMessageContext` to ensure these fields are present where applicable.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user