windy/go-caatsm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

✨ Enhance observability and error handling in NATS integration. Introduce comprehensive OpenTelemetry support with environment-based sampling and semantic attributes for tracing and metrics. Implement an advisory dead-letter queue (DLQ) handler for managing message delivery failures. Update NATS consumer to utilize structured logging and improve error handling strategies. Refactor configuration files for OpenTelemetry collector in both development and production environments, ensuring robust telemetry integration. Enhance documentation to reflect new features and best practices for observability.

Browse Source
This commit is contained in:
windyboy
2025-11-18 11:47:35 +08:00
parent 5d05237283
commit 7f44b5389d
31 changed files with 2546 additions and 476 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/nats.md
+323
View File
@@ -0,0 +1,323 @@
# NATS Integration Architecture
## Overview
The NATS integration provides a robust, production-ready message processing system built on Clean Architecture principles. It supports both JetStream (persistent) and Core NATS (fire-and-forget) modes with comprehensive error handling, observability, and resilience features.
## Architecture
### Clean Architecture Layers
```
┌─────────────────────────────────────┐
│ Port Interfaces │
│ (Publisher, Consumer contracts) │
├─────────────────────────────────────┤
│ Application Layer │
│ (Message processing logic) │
├─────────────────────────────────────┤
│ Infrastructure Layer │
│ (NATS implementation details) │
│ │
│ ┌─────────────────────────────┐ │
│ │ Consumer │ │
│ │ ┌─────────────────────┐ │ │
│ │ │ MessageFetcher │ │ │
│ │ │ MessageProcessor │ │ │
│ │ │ ErrorHandler │ │ │
│ │ │ DLQHandler │ │ │
│ │ └─────────────────────┘ │ │
│ └─────────────────────────────┘ │
│ │
│ ┌─────────────────────────────┐ │
│ │ Publisher │ │
│ │ ┌─────────────────────┐ │ │
│ │ │ MessageSerializer │ │ │
│ │ │ HeaderEnricher │ │ │
│ │ └─────────────────────┘ │ │
│ └─────────────────────────────┘ │
└─────────────────────────────────────┘
```
## Core Components
### Consumer
The consumer handles message consumption with the following features:
#### Modes
- **JetStream Mode**: Persistent, durable message processing with acknowledgments
- **Core Mode**: Fire-and-forget message processing for simple use cases
#### Key Features
- **Batch Processing**: Configurable batch sizes and timeouts for efficient processing
- **Backpressure**: Automatic backpressure when processing errors accumulate
- **Dead Letter Queue (DLQ)**: Automatic routing of failed messages to DLQ
- **Advisory DLQ**: Handles messages that exceed MaxDeliver limits
- **Self-Healing**: Automatic recreation of missing streams/consumers in dev environments
- **Graceful Shutdown**: Proper cleanup and draining of connections
#### Configuration
```toml
[NATS]
Mode = "jetstream" # or "core"
Stream = "TELEGRAM"
Consumer = "telegram-consumer"
[NATS.ConsumerRules]
AckWait = "30s"
MaxDeliver = 3
MaxAckPending = 1000
DeliverPolicy = "all"
ReplayPolicy = "instant"
Backoff = ["1s", "2s", "5s", "10s"]
[DLQ]
Enabled = true
Subject = "caatsm.dlq"
[App]
BatchSize = 50
BatchTimeout = "2s"
MonitorInterval = "30s"
```
### Publisher
The publisher handles message publishing with deduplication and observability.
#### Features
- **Message Deduplication**: Automatic UUID-based deduplication headers
- **JetStream/Core Mode Support**: Adapts based on available JetStream context
- **Structured Logging**: Comprehensive logging of publish operations
- **Error Classification**: Distinguishes between transient and permanent errors
### Error Handling
#### Error Types
- **Transient Errors**: Network issues, temporary unavailability (retried with backoff)
- **Permanent Errors**: Message format issues, business logic failures (routed to DLQ)
- **Resource Errors**: Missing streams/consumers (auto-recovered in dev, fail in prod)
#### Recovery Strategies
- **Exponential Backoff**: Configurable backoff for transient failures
- **Circuit Breaker Pattern**: Prevents cascade failures
- **Resource Recreation**: Automatic recreation of missing JetStream resources
- **Graceful Degradation**: Continues processing other messages when one fails
### Dead Letter Queue (DLQ)
#### Features
- **Rich Metadata**: Includes original message, error details, delivery attempts
- **Stream Validation**: Validates DLQ stream exists at startup
- **Advisory Processing**: Handles MaxDeliver exhaustion automatically
- **Operational Visibility**: Comprehensive logging and metrics
#### DLQ Message Format
```json
{
"transport_msg_id": "uuid",
"subject": "original.subject",
"stream": "TELEGRAM",
"consumer": "telegram-consumer",
"nats_sequence": 12345,
"deliveries": 3,
"error": "processing failed: invalid format",
"received_at": "2024-01-01T12:00:00Z",
"body": "original message data"
}
```
## Observability
### Metrics
- **Consumer Metrics**: ack_pending, redelivered, pending, delivered counts
- **Processing Metrics**: batch size, processing duration, error rates
- **DLQ Metrics**: messages routed to DLQ, publish failures
- **Connection Metrics**: connection health, reconnection events
### Tracing
- **End-to-End Tracing**: Request correlation through trace IDs
- **Span Attributes**: Consumer name, stream name, batch size, error details
- **Context Propagation**: Trace context passed through processing pipeline
### Logging
- **Structured Logs**: JSON format with correlation IDs
- **Log Levels**: Debug, Info, Warn, Error with appropriate detail levels
- **Operational Context**: Includes consumer, stream, and message metadata
## Resilience Patterns
### Backpressure
- **Error Accumulation**: Tracks consecutive processing errors
- **Adaptive Delay**: Increases delay based on error frequency
- **Circuit Breaking**: Stops processing when errors exceed threshold
### Connection Management
- **Auto-Reconnection**: Built-in NATS reconnection logic
- **Graceful Shutdown**: Proper draining with timeouts
- **Resource Cleanup**: Ensures subscriptions and connections are closed
### Self-Healing
- **Development Mode**: Auto-creates missing streams/consumers
- **Production Mode**: Fails fast on configuration issues
- **Recovery Logic**: Attempts to recreate resources on errors
## Configuration
### Environment Variables
```bash
CAATSM_NATS_URL=nats://localhost:4222
CAATSM_NATS_MODE=jetstream
CAATSM_DLQ_ENABLED=true
CAATSM_DLQ_SUBJECT=caatsm.dlq
```
### Runtime Configuration
- **Hot Reload**: Configuration changes applied without restart
- **Validation**: Comprehensive validation at startup
- **Defaults**: Sensible defaults for all configuration options
## Testing Strategy
### Unit Tests
- **Pure Functions**: Configuration normalization, policy mapping
- **Mock Dependencies**: NATS connections, JetStream contexts
- **Table-Driven Tests**: Comprehensive coverage of edge cases
### Integration Tests
- **Real NATS**: Testcontainers with actual NATS server
- **End-to-End**: Complete message processing pipelines
- **Failure Scenarios**: Network failures, resource unavailability
### Test Categories
- **Happy Path**: Normal operation scenarios
- **Error Recovery**: Various failure and recovery scenarios
- **Performance**: Load testing and resource usage
- **Configuration**: Different configuration combinations
## Usage Examples
### Basic Consumer Setup
```go
consumer, err := natsinfra.ProvideConsumer(
natsConn,
jetStream,
messageProcessor,
config,
telemetryRecorder,
logger,
)
if err != nil {
return err
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
return consumer.Start(ctx)
```
### Publishing Messages
```go
publisher, err := natsinfra.ProvidePublisher(
jetStream,
natsConn,
config,
logger,
)
if err != nil {
return err
}
err = publisher.Publish(&dto.ParsedTelegram{
Uuid: uuid.NewString(),
Data: telegramData,
})
```
### Custom Error Handling
```go
type CustomProcessor struct {
// implementation
}
func (p *CustomProcessor) ProcessMessage(ctx context.Context, msg *nats.Msg) error {
// Business logic here
if shouldRetry := someCondition(); shouldRetry {
return app.NewTransientError("temporary failure")
}
if isInvalid := validateMessage(msg); isInvalid {
return app.NewPermanentError("invalid message format")
}
return nil
}
```
## Performance Considerations
### Optimization Strategies
- **Batch Processing**: Reduces per-message overhead
- **Connection Pooling**: Reuses connections efficiently
- **Memory Management**: Proper buffer sizing and cleanup
- **Concurrent Processing**: Parallel message processing within batches
### Monitoring Points
- **Throughput**: Messages processed per second
- **Latency**: End-to-end processing time
- **Resource Usage**: Memory, CPU, and network utilization
- **Error Rates**: Percentage of failed messages
## Operational Guide
### Deployment
1. **Configuration**: Set appropriate timeouts and limits
2. **Resource Provisioning**: Ensure sufficient NATS cluster capacity
3. **Monitoring Setup**: Configure alerts and dashboards
4. **DLQ Monitoring**: Set up DLQ message processing
### Troubleshooting
- **High Latency**: Check batch sizes and processing logic
- **Message Loss**: Verify consumer acks and DLQ configuration
- **Connection Issues**: Check NATS cluster health and network connectivity
- **Resource Exhaustion**: Monitor memory usage and connection counts
### Maintenance
- **Stream Cleanup**: Periodically clean up old streams
- **Consumer Recreation**: Recreate consumers for configuration changes
- **Performance Tuning**: Adjust batch sizes based on load patterns
- **Version Upgrades**: Test compatibility with NATS server versions
## Security Considerations
### Authentication
- **NATS Auth**: Use NATS built-in authentication mechanisms
- **TLS**: Enable TLS for encrypted communication
- **Token Auth**: Use NATS tokens for service authentication
### Authorization
- **Subject Permissions**: Restrict publish/subscribe permissions
- **Stream Access**: Control access to specific streams
- **DLQ Security**: Secure DLQ access to prevent data leakage
### Data Protection
- **Message Encryption**: Encrypt sensitive message data
- **Audit Logging**: Log all message operations for compliance
- **PII Handling**: Avoid logging sensitive information
## Future Enhancements
### Planned Features
- **Consumer Groups**: Horizontal scaling with multiple consumers
- **Message Filtering**: Subject-based and header-based filtering
- **Priority Queues**: High-priority message processing
- **Rate Limiting**: Per-consumer and per-subject rate limits
- **Message Transformation**: In-flight message modification
- **Multi-Region**: Cross-region message replication
### Extensibility Points
- **Custom Serializers**: Pluggable message serialization
- **Middleware**: Request/response middleware support
- **Hooks**: Pre/post processing hooks
- **Metrics Backends**: Support for additional metrics systems
- **Storage Backends**: Alternative storage for DLQ messages
docs/observability.md
+62 -43
View File
@@ -166,60 +166,79 @@ A non-2xx response indicates the service is not healthy/ready and should be remo
### Tracing
The application implements production-ready OpenTelemetry tracing with comprehensive span coverage and semantic attributes.
#### Configuration
Tracing is configured via the `telemetry` section:
- `telemetry.enabled` – enables OTEL exporters.
- `telemetry.endpoint` – OTLP HTTP endpoint (e.g. `localhost:4318`).
- `telemetry.insecure` – disables TLS for local/dev.
- `telemetry.enabled` – enables OTEL exporters (default: `false` in dev, `true` in prod)
- `telemetry.endpoint` – OTLP HTTP endpoint (e.g. `localhost:4318` for dev, `otel-collector.company.com:4318` for prod)
- `telemetry.insecure` – disables TLS for local/dev (default: `true` in dev, `false` in prod)
#### OTEL vs Prometheus metrics
#### Sampling Strategy
The receiver reports two complementary sets of metrics:
Environment-based sampling ensures cost-effective production monitoring:
- **Prometheus metrics via `/metrics`**
Implemented in `internal/infra/metrics`, covering:
- End-to-end message handling (`caatsm_messages_total`,
`caatsm_handle_latency_seconds`, `caatsm_retries_total`)
- DB activity (`caatsm_db_queries_total`,
`caatsm_db_query_latency_seconds`)
- Legacy per-telegram metrics
- **Production**: 1% sampling (cost-effective, maintains observability)
- **Staging**: 10% sampling (balanced observability for testing)
- **Development/Test**: 100% sampling (full debugging coverage)
- **OpenTelemetry metrics via OTLP**
Implemented using `otel.Meter` in the NATS consumer and app processor,
including:
- `caatsm_messages_processed_total`
- `caatsm_parse_duration_seconds`
- `caatsm_publish_failures_total`
- `caatsm_nats_consumer_ack_pending`
- `caatsm_nats_consumer_redelivered`
- `caatsm_nats_consumer_pending`
- `caatsm_nats_consumer_delivered`
#### Resource Attributes
Prometheus only sees the metrics exposed on `/metrics`. OTEL metrics are
exported to the configured OTEL collector (`telemetry.endpoint`) via OTLP and
are, by default, forwarded to Jaeger (traces) and logs (metrics) according to
`configs/otel-collector.dev.yaml`. If you want OTEL metrics to appear in
Prometheus as well, you can extend the collector configuration with a
`prometheus` or `prometheusremotewrite` exporter and add a corresponding
scrape or remote-write configuration.
All spans include comprehensive resource metadata:
Key spans:
- `caatsm/nats`
- `Consumer.processMessage`
- `caatsm/app`
- `MessageProcessor.Handle`
- `Publisher.Publish`
- `caatsm/postgres`
- `Repository.InsertOne`
- `Repository.InsertBatch`
- `Repository.InsertRaw`
#### Key Spans with Semantic Attributes
Important attributes:
**NATS Consumer (`caatsm/nats`)**:
- `Consumer.processMessage`
- `messaging.system: nats`
- `messaging.operation: receive`
- `messaging.destination: <subject>`
- `messaging.consumer.id: <consumer-name>`
- `caatsm.stream: <stream-name>`
- `nats.subject`, `nats.msg_id`, `nats.js.stream_seq`, `nats.js.consumer_seq`
- `telegram.message_id`, `telegram.category`, `telegram.status`
- `db.table`, `db.inserted`
**Application Processor (`caatsm/app`)**:
- `MessageProcessor.Handle`
- `messaging.system: nats`
- `messaging.operation: receive`
- `messaging.message_id: <msg-id>`
- `caatsm.component: processor`
- `caatsm.message.category: <ARR|DEP|FPL|etc>`
**Database Operations (`caatsm/postgres`)**:
- `Repository.InsertOne`, `Repository.InsertBatch`, `Repository.InsertRaw`
- `db.system: postgresql`
- `db.operation: insert`
- `db.name: aviation`
- `db.table: telegrams`
- `caatsm.message.id: <telegram-id>`
#### OTEL vs Prometheus Metrics
The receiver reports complementary metrics through both systems:
**Prometheus metrics via `/metrics`** (operational focus):
- End-to-end message handling (`caatsm_messages_total`, `caatsm_handle_latency_seconds`, `caatsm_retries_total`)
- DB activity (`caatsm_db_queries_total`, `caatsm_db_query_latency_seconds`)
- NATS consumer metrics (`caatsm_nats_consumer_pending_messages`)
- DLQ operations (`caatsm_dlq_messages_total`, `caatsm_dlq_publish_failures_total`)
**OpenTelemetry metrics via OTLP** (business focus):
- Message processing results (`caatsm_messages_processed_total`)
- Parse performance (`caatsm_parse_duration_seconds`)
- Publish reliability (`caatsm_publish_failures_total`)
- NATS consumer health metrics (ack pending, redelivered, delivered counts)
#### Collector Integration
OTEL metrics and traces are exported to the configured collector:
- **Development**: `configs/otel-collector.dev.yaml` (batching, resource processing, retry logic)
- **Production**: `configs/otel-collector.prod.yaml` (TLS, authentication, high availability)
To integrate OTEL metrics with Prometheus, extend the collector configuration with a `prometheusremotewrite` exporter.
### Structured Logging Contract
docs/otel-best-practices.md
+353
View File
@@ -0,0 +1,353 @@
# OpenTelemetry Best Practices
This document outlines the OpenTelemetry implementation and best practices for the CAATSM receiver service.
## Architecture Overview
The CAATSM receiver implements a dual-telemetry approach:
1. **OpenTelemetry (OTEL)**: Business metrics and distributed tracing
2. **Prometheus**: Operational metrics and alerting
## OTEL Implementation
### SDK Configuration
The application uses a production-ready OTEL SDK setup with:
- **Environment-based sampling**: Cost-effective production monitoring
- **Comprehensive resource attributes**: Service identification and metadata
- **Optimized batching**: Efficient export with retry logic
- **TLS security**: Configurable secure connections
### Sampling Strategy
```go
Production: 1% // Cost-effective, maintains observability
Staging: 10% // Balanced observability for testing
Dev/Test: 100% // Full debugging coverage
```
### Resource Attributes
All telemetry includes standardized resource metadata:
```yaml
# Service identification
service.name: caatsm
service.version: dev
service.namespace: airport
service.component: receiver
# Environment context
deployment.environment: prod|staging|dev
# Build information
build.commit: <git-hash>
build.built_at: <timestamp>
# Telemetry configuration
telemetry.endpoint: <collector-url>
telemetry.insecure: true|false
```
## Span Semantics
### Messaging Spans
**NATS Consumer Operations**:
```yaml
Span: Consumer.processMessage
Attributes:
messaging.system: nats
messaging.operation: receive
messaging.destination: telegram.serial
messaging.consumer.id: telegram-consumer
caatsm.stream: TELEGRAM
```
**Application Processing**:
```yaml
Span: MessageProcessor.Handle
Attributes:
messaging.system: nats
messaging.operation: receive
messaging.message_id: <nats-msg-id>
caatsm.component: processor
caatsm.message.category: ARR|DEP|FPL|etc
```
### Database Spans
**Repository Operations**:
```yaml
Span: Repository.InsertOne|InsertBatch|InsertRaw
Attributes:
db.system: postgresql
db.operation: insert
db.name: aviation
db.table: telegrams
caatsm.message.id: <telegram-id>
```
## Metrics Strategy
### OTEL Metrics (Business Focus)
- `caatsm_messages_processed_total{message.status, message.category}`
- `caatsm_publish_failures_total{message.category}`
- `caatsm_parse_duration_seconds{message.status, message.category}`
### Prometheus Metrics (Operational Focus)
- `caatsm_messages_total{stream, consumer, result}`
- `caatsm_handle_latency_seconds{stream, consumer}`
- `caatsm_db_queries_total{operation, result}`
- `caatsm_nats_consumer_pending_messages{stream, consumer}`
## Collector Configuration
### Development Setup
```yaml
# configs/otel-collector.dev.yaml
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
max_request_body_size: 20971520
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
send_batch_size: 1024
timeout: 1s
resource:
attributes:
- key: service.instance.id
value: "${POD_NAME}"
action: upsert
exporters:
logging:
sampling_initial: 10
sampling_thereafter: 100
otlphttp/jaeger:
endpoint: http://jaeger:4318
sending_queue:
queue_size: 10000
retry_on_failure:
enabled: true
prometheus:
endpoint: "0.0.0.0:8889"
service:
pipelines:
traces:
receivers: [otlp]
processors: [resource, batch]
exporters: [logging, otlphttp/jaeger]
metrics:
receivers: [otlp]
processors: [resource, batch]
exporters: [logging, prometheus]
```
### Production Setup
```yaml
# configs/otel-collector.prod.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
tls:
cert_file: /etc/ssl/certs/otel-collector.crt
key_file: /etc/ssl/private/otel-collector.key
auth:
authenticator: bearer_token
http:
endpoint: 0.0.0.0:4318
tls:
cert_file: /etc/ssl/certs/otel-collector.crt
key_file: /etc/ssl/private/otel-collector.key
auth:
authenticator: bearer_token
extensions:
health_check:
endpoint: 0.0.0.0:13133
pprof:
endpoint: :1888
zpages:
endpoint: :55679
exporters:
otlphttp/jaeger:
endpoint: https://jaeger.company.com:4318
headers:
authorization: "Bearer ${JAEGER_API_TOKEN}"
tls:
insecure: false
sending_queue:
queue_size: 10000
retry_on_failure:
enabled: true
prometheusremotewrite:
endpoint: https://prometheus.company.com/api/v1/write
headers:
authorization: "Bearer ${PROMETHEUS_API_TOKEN}"
tls:
insecure: false
sending_queue:
queue_size: 10000
retry_on_failure:
enabled: true
service:
extensions: [health_check, pprof, zpages]
pipelines:
traces:
receivers: [otlp]
processors: [resource, batch]
exporters: [otlphttp/jaeger]
metrics:
receivers: [otlp]
processors: [resource, batch]
exporters: [prometheusremotewrite]
```
## Configuration Examples
### Development Configuration
```toml
[telemetry]
enabled = true
endpoint = "localhost:4318"
insecure = true
```
### Production Configuration
```toml
[telemetry]
enabled = true
endpoint = "otel-collector.company.com:4318"
insecure = false
```
### CLI Overrides
```bash
# Enable telemetry
./bin/receiver listen --telemetry-enabled
# Custom endpoint
./bin/receiver listen --telemetry-endpoint https://otel-collector.prod:4318
# Insecure for development
./bin/receiver listen --telemetry-insecure
```
## Monitoring and Debugging
### Health Checks
```bash
# Collector health
curl http://otel-collector:13133
# Application metrics
curl http://localhost:2112/metrics
# OTEL collector metrics
curl http://otel-collector:8888/metrics
```
### Tracing Verification
```bash
# Jaeger UI
open http://localhost:16686
# Search for caatsm traces
Service: caatsm
Operation: Consumer.processMessage OR MessageProcessor.Handle
```
### Metrics Verification
```bash
# Prometheus queries
caatsm_messages_processed_total
caatsm_parse_duration_seconds
rate(caatsm_messages_total[5m])
```
## Best Practices
### 1. Sampling Strategy
- Use environment-appropriate sampling rates
- Monitor sampling effectiveness
- Adjust based on cost and observability needs
### 2. Resource Attributes
- Include comprehensive service metadata
- Use semantic conventions
- Add custom attributes for business context
### 3. Span Attributes
- Follow OpenTelemetry semantic conventions
- Include relevant business context
- Avoid high-cardinality attributes
### 4. Error Handling
- Always record errors on spans
- Set appropriate span status
- Include error context in attributes
### 5. Performance
- Use batching to reduce export overhead
- Configure appropriate queue sizes
- Monitor exporter performance
## Troubleshooting
### Common Issues
1. **No traces in Jaeger**
- Check collector logs: `docker logs otel-collector`
- Verify endpoint configuration
- Check network connectivity
2. **High sampling rate**
- Adjust sampling configuration
- Monitor cost impact
- Consider head-based sampling
3. **Missing metrics**
- Verify collector pipeline configuration
- Check Prometheus remote write configuration
- Validate metric names and labels
4. **Performance impact**
- Review sampling rates
- Check batch configuration
- Monitor exporter queue sizes
### Debug Commands
```bash
# View collector configuration
docker exec otel-collector cat /etc/otel/config.yaml
# Check collector metrics
curl -s http://otel-collector:8888/metrics | grep otel
# View application telemetry logs
./bin/receiver listen --log-level=debug 2>&1 | grep -i telemetry
```