✨ Enhance observability and error handling in NATS integration. Introduce comprehensive OpenTelemetry support with environment-based sampling and semantic attributes for tracing and metrics. Implement an advisory dead-letter queue (DLQ) handler for managing message delivery failures. Update NATS consumer to utilize structured logging and improve error handling strategies. Refactor configuration files for OpenTelemetry collector in both development and production environments, ensuring robust telemetry integration. Enhance documentation to reflect new features and best practices for observability.
This commit is contained in:
windyboy
@@ -0,0 +1,353 @@
|
||||
# OpenTelemetry Best Practices
|
||||
|
||||
This document outlines the OpenTelemetry implementation and best practices for the CAATSM receiver service.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
The CAATSM receiver implements a dual-telemetry approach:
|
||||
|
||||
1. **OpenTelemetry (OTEL)**: Business metrics and distributed tracing
|
||||
2. **Prometheus**: Operational metrics and alerting
|
||||
|
||||
## OTEL Implementation
|
||||
|
||||
### SDK Configuration
|
||||
|
||||
The application uses a production-ready OTEL SDK setup with:
|
||||
|
||||
- **Environment-based sampling**: Cost-effective production monitoring
|
||||
- **Comprehensive resource attributes**: Service identification and metadata
|
||||
- **Optimized batching**: Efficient export with retry logic
|
||||
- **TLS security**: Configurable secure connections
|
||||
|
||||
### Sampling Strategy
|
||||
|
||||
```go
|
||||
Production: 1% // Cost-effective, maintains observability
|
||||
Staging: 10% // Balanced observability for testing
|
||||
Dev/Test: 100% // Full debugging coverage
|
||||
```
|
||||
|
||||
### Resource Attributes
|
||||
|
||||
All telemetry includes standardized resource metadata:
|
||||
|
||||
```yaml
|
||||
# Service identification
|
||||
service.name: caatsm
|
||||
service.version: dev
|
||||
service.namespace: airport
|
||||
service.component: receiver
|
||||
|
||||
# Environment context
|
||||
deployment.environment: prod|staging|dev
|
||||
|
||||
# Build information
|
||||
build.commit: <git-hash>
|
||||
build.built_at: <timestamp>
|
||||
|
||||
# Telemetry configuration
|
||||
telemetry.endpoint: <collector-url>
|
||||
telemetry.insecure: true|false
|
||||
```
|
||||
|
||||
## Span Semantics
|
||||
|
||||
### Messaging Spans
|
||||
|
||||
**NATS Consumer Operations**:
|
||||
```yaml
|
||||
Span: Consumer.processMessage
|
||||
Attributes:
|
||||
messaging.system: nats
|
||||
messaging.operation: receive
|
||||
messaging.destination: telegram.serial
|
||||
messaging.consumer.id: telegram-consumer
|
||||
caatsm.stream: TELEGRAM
|
||||
```
|
||||
|
||||
**Application Processing**:
|
||||
```yaml
|
||||
Span: MessageProcessor.Handle
|
||||
Attributes:
|
||||
messaging.system: nats
|
||||
messaging.operation: receive
|
||||
messaging.message_id: <nats-msg-id>
|
||||
caatsm.component: processor
|
||||
caatsm.message.category: ARR|DEP|FPL|etc
|
||||
```
|
||||
|
||||
### Database Spans
|
||||
|
||||
**Repository Operations**:
|
||||
```yaml
|
||||
Span: Repository.InsertOne|InsertBatch|InsertRaw
|
||||
Attributes:
|
||||
db.system: postgresql
|
||||
db.operation: insert
|
||||
db.name: aviation
|
||||
db.table: telegrams
|
||||
caatsm.message.id: <telegram-id>
|
||||
```
|
||||
|
||||
## Metrics Strategy
|
||||
|
||||
### OTEL Metrics (Business Focus)
|
||||
|
||||
- `caatsm_messages_processed_total{message.status, message.category}`
|
||||
- `caatsm_publish_failures_total{message.category}`
|
||||
- `caatsm_parse_duration_seconds{message.status, message.category}`
|
||||
|
||||
### Prometheus Metrics (Operational Focus)
|
||||
|
||||
- `caatsm_messages_total{stream, consumer, result}`
|
||||
- `caatsm_handle_latency_seconds{stream, consumer}`
|
||||
- `caatsm_db_queries_total{operation, result}`
|
||||
- `caatsm_nats_consumer_pending_messages{stream, consumer}`
|
||||
|
||||
## Collector Configuration
|
||||
|
||||
### Development Setup
|
||||
|
||||
```yaml
|
||||
# configs/otel-collector.dev.yaml
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
max_request_body_size: 20971520
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
|
||||
processors:
|
||||
batch:
|
||||
send_batch_size: 1024
|
||||
timeout: 1s
|
||||
resource:
|
||||
attributes:
|
||||
- key: service.instance.id
|
||||
value: "${POD_NAME}"
|
||||
action: upsert
|
||||
|
||||
exporters:
|
||||
logging:
|
||||
sampling_initial: 10
|
||||
sampling_thereafter: 100
|
||||
otlphttp/jaeger:
|
||||
endpoint: http://jaeger:4318
|
||||
sending_queue:
|
||||
queue_size: 10000
|
||||
retry_on_failure:
|
||||
enabled: true
|
||||
prometheus:
|
||||
endpoint: "0.0.0.0:8889"
|
||||
|
||||
service:
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [resource, batch]
|
||||
exporters: [logging, otlphttp/jaeger]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [resource, batch]
|
||||
exporters: [logging, prometheus]
|
||||
```
|
||||
|
||||
### Production Setup
|
||||
|
||||
```yaml
|
||||
# configs/otel-collector.prod.yaml
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
tls:
|
||||
cert_file: /etc/ssl/certs/otel-collector.crt
|
||||
key_file: /etc/ssl/private/otel-collector.key
|
||||
auth:
|
||||
authenticator: bearer_token
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
tls:
|
||||
cert_file: /etc/ssl/certs/otel-collector.crt
|
||||
key_file: /etc/ssl/private/otel-collector.key
|
||||
auth:
|
||||
authenticator: bearer_token
|
||||
|
||||
extensions:
|
||||
health_check:
|
||||
endpoint: 0.0.0.0:13133
|
||||
pprof:
|
||||
endpoint: :1888
|
||||
zpages:
|
||||
endpoint: :55679
|
||||
|
||||
exporters:
|
||||
otlphttp/jaeger:
|
||||
endpoint: https://jaeger.company.com:4318
|
||||
headers:
|
||||
authorization: "Bearer ${JAEGER_API_TOKEN}"
|
||||
tls:
|
||||
insecure: false
|
||||
sending_queue:
|
||||
queue_size: 10000
|
||||
retry_on_failure:
|
||||
enabled: true
|
||||
prometheusremotewrite:
|
||||
endpoint: https://prometheus.company.com/api/v1/write
|
||||
headers:
|
||||
authorization: "Bearer ${PROMETHEUS_API_TOKEN}"
|
||||
tls:
|
||||
insecure: false
|
||||
sending_queue:
|
||||
queue_size: 10000
|
||||
retry_on_failure:
|
||||
enabled: true
|
||||
|
||||
service:
|
||||
extensions: [health_check, pprof, zpages]
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [resource, batch]
|
||||
exporters: [otlphttp/jaeger]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [resource, batch]
|
||||
exporters: [prometheusremotewrite]
|
||||
```
|
||||
|
||||
## Configuration Examples
|
||||
|
||||
### Development Configuration
|
||||
|
||||
```toml
|
||||
[telemetry]
|
||||
enabled = true
|
||||
endpoint = "localhost:4318"
|
||||
insecure = true
|
||||
```
|
||||
|
||||
### Production Configuration
|
||||
|
||||
```toml
|
||||
[telemetry]
|
||||
enabled = true
|
||||
endpoint = "otel-collector.company.com:4318"
|
||||
insecure = false
|
||||
```
|
||||
|
||||
### CLI Overrides
|
||||
|
||||
```bash
|
||||
# Enable telemetry
|
||||
./bin/receiver listen --telemetry-enabled
|
||||
|
||||
# Custom endpoint
|
||||
./bin/receiver listen --telemetry-endpoint https://otel-collector.prod:4318
|
||||
|
||||
# Insecure for development
|
||||
./bin/receiver listen --telemetry-insecure
|
||||
```
|
||||
|
||||
## Monitoring and Debugging
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Collector health
|
||||
curl http://otel-collector:13133
|
||||
|
||||
# Application metrics
|
||||
curl http://localhost:2112/metrics
|
||||
|
||||
# OTEL collector metrics
|
||||
curl http://otel-collector:8888/metrics
|
||||
```
|
||||
|
||||
### Tracing Verification
|
||||
|
||||
```bash
|
||||
# Jaeger UI
|
||||
open http://localhost:16686
|
||||
|
||||
# Search for caatsm traces
|
||||
Service: caatsm
|
||||
Operation: Consumer.processMessage OR MessageProcessor.Handle
|
||||
```
|
||||
|
||||
### Metrics Verification
|
||||
|
||||
```bash
|
||||
# Prometheus queries
|
||||
caatsm_messages_processed_total
|
||||
caatsm_parse_duration_seconds
|
||||
rate(caatsm_messages_total[5m])
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Sampling Strategy
|
||||
- Use environment-appropriate sampling rates
|
||||
- Monitor sampling effectiveness
|
||||
- Adjust based on cost and observability needs
|
||||
|
||||
### 2. Resource Attributes
|
||||
- Include comprehensive service metadata
|
||||
- Use semantic conventions
|
||||
- Add custom attributes for business context
|
||||
|
||||
### 3. Span Attributes
|
||||
- Follow OpenTelemetry semantic conventions
|
||||
- Include relevant business context
|
||||
- Avoid high-cardinality attributes
|
||||
|
||||
### 4. Error Handling
|
||||
- Always record errors on spans
|
||||
- Set appropriate span status
|
||||
- Include error context in attributes
|
||||
|
||||
### 5. Performance
|
||||
- Use batching to reduce export overhead
|
||||
- Configure appropriate queue sizes
|
||||
- Monitor exporter performance
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **No traces in Jaeger**
|
||||
- Check collector logs: `docker logs otel-collector`
|
||||
- Verify endpoint configuration
|
||||
- Check network connectivity
|
||||
|
||||
2. **High sampling rate**
|
||||
- Adjust sampling configuration
|
||||
- Monitor cost impact
|
||||
- Consider head-based sampling
|
||||
|
||||
3. **Missing metrics**
|
||||
- Verify collector pipeline configuration
|
||||
- Check Prometheus remote write configuration
|
||||
- Validate metric names and labels
|
||||
|
||||
4. **Performance impact**
|
||||
- Review sampling rates
|
||||
- Check batch configuration
|
||||
- Monitor exporter queue sizes
|
||||
|
||||
### Debug Commands
|
||||
|
||||
```bash
|
||||
# View collector configuration
|
||||
docker exec otel-collector cat /etc/otel/config.yaml
|
||||
|
||||
# Check collector metrics
|
||||
curl -s http://otel-collector:8888/metrics | grep otel
|
||||
|
||||
# View application telemetry logs
|
||||
./bin/receiver listen --log-level=debug 2>&1 | grep -i telemetry
|
||||
```
|
||||
Reference in New Issue
Block a user