✨ Revise observability features by introducing new health and liveness endpoints (/livez and /readyz), updating Prometheus metrics to track NATS consumer pending messages, and enhancing telemetry integration for better monitoring. Update documentation to reflect these changes and ensure consistency in metric naming conventions.

2025-11-16 13:14:46 +08:00
parent 3d8572a69f
commit e27328378a
17 changed files with 582 additions and 154 deletions
@@ -70,13 +70,13 @@ spec:
              mountPath: /etc/caatsm
          livenessProbe:
            httpGet:
-              path: /healthz
+              path: /livez
              port: monitoring
            initialDelaySeconds: 10
            periodSeconds: 15
          readinessProbe:
            httpGet:
-              path: /healthz
+              path: /readyz
              port: monitoring
            initialDelaySeconds: 5
            periodSeconds: 15
@@ -105,5 +105,5 @@ spec:
      protocol: TCP
 ```

-Point Prometheus at the service above (or annotate it if you use `prometheus-operator`). The `/healthz` probe doubles as a readiness check and quickly surfaces upstream connectivity issues.
+Point Prometheus at the service above (or annotate it if you use `prometheus-operator`). The `/readyz` probe surfaces upstream connectivity issues, while `/livez` is used solely for liveness.

@@ -58,7 +58,7 @@ sudo systemctl enable --now caatsm
 ## 4. Observability Hooks

 - Expose `monitoring.addr = ":2112"` (default) and add firewall rules so Prometheus can scrape `http://host:2112/metrics`.
- systemd watchdogs can use `curl -sf http://127.0.0.1:2112/healthz`.
+- systemd watchdogs can use `curl -sf http://127.0.0.1:2112/livez` for liveness and `curl -sf http://127.0.0.1:2112/readyz` for readiness.

 With these three files (binary, config, env) the service becomes repeatable and easy to operate.

@@ -120,6 +120,7 @@ Services:
  - Persists data in `grafana-data`, provisions datasources via `configs/grafana-datasources.dev.yml`, and listens on <http://localhost:3000> (login `admin` / `admin`)
  - Automatically loads dashboards from `configs/grafana-dashboards.dev/`, including OpenTelemetry Collector and NATS/JetStream overviews (find them under the **Dev Observability** folder)
  - The OpenTelemetry dashboard also charts the CAATSM-specific metrics `caatsm_messages_processed_total`, `caatsm_publish_failures_total`, and `caatsm_parse_duration_ms` (percentiles) so you can track throughput and parsing latency.
+  - Note: `caatsm_parse_duration_ms` has been renamed to `caatsm_parse_duration_seconds` to align with Prometheus `_seconds` conventions.

 ### Customizing Collections & Dashboards

@@ -28,10 +28,13 @@ The service exposes Prometheus metrics via the monitoring HTTP server (default `
 - `caatsm_dlq_publish_failures_total{stream,consumer}`  
  Count of failures when attempting to publish messages to the DLQ.

+- `caatsm_nats_consumer_pending_messages{stream,consumer}`  
+  Current pending message count for each JetStream consumer (useful for lag/backlog alerts).
+
 Additional OTEL metrics are emitted via the configured OTEL endpoint, including:

 - `caatsm_messages_processed_total`  
- `caatsm_parse_duration_ms`  
+- `caatsm_parse_duration_seconds`  
 - `caatsm_publish_failures_total`  
 - `caatsm_nats_consumer_ack_pending`  
 - `caatsm_nats_consumer_redelivered`  
@@ -133,15 +136,33 @@ To validate that the dashboard is receiving data:

 The monitoring server exposes:

- `/healthz` – basic liveness and dependency check.  
- `/readyz` – readiness endpoint with the same logic as `/healthz`, intended for load balancers / orchestrators.
+- `/livez` – lightweight liveness endpoint that reports process/build information without checking dependencies.  
+- `/healthz` – backward-compatible health endpoint used by existing deploys; currently shares logic with `/readyz`.  
+- `/readyz` – readiness endpoint that checks critical dependencies and should be used by load balancers / orchestrators.

 Checks performed:

- PostgreSQL: `pgxpool.Pool.Ping` with configurable timeout (`monitoring.health_timeout`).  
- NATS: connection status must be `CONNECTED`.
+- PostgreSQL: `pgxpool.Pool.Ping` with configurable timeout (`monitoring.health_timeout`), reporting `status` and `latency_ms`.  
+- NATS: connection status must be `CONNECTED`; otherwise the dependency is marked as unavailable.

-A non-200 response indicates the service is not healthy/ready and should be removed from traffic.
+Responses include build metadata and a dependency map, for example:
+
+```json
+{
+  "status": "ok",
+  "build": {
+    "version": "v0.4.3",
+    "rev": "abc1234",
+    "built_at": "2025-11-16T08:35:00Z"
+  },
+  "dependencies": {
+    "postgres": {"status": "ok", "latency_ms": 4},
+    "nats": {"status": "CONNECTED"}
+  }
+}
+```
+
+A non-2xx response indicates the service is not healthy/ready and should be removed from traffic.

 ### Tracing

@@ -167,7 +188,7 @@ The receiver reports two complementary sets of metrics:
  Implemented using `otel.Meter` in the NATS consumer and app processor,
  including:
  - `caatsm_messages_processed_total`
-  - `caatsm_parse_duration_ms`
+  - `caatsm_parse_duration_seconds`
  - `caatsm_publish_failures_total`
  - `caatsm_nats_consumer_ack_pending`
  - `caatsm_nats_consumer_redelivered`