windy/go-caatsm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

✨ Add repository guidelines and enhance documentation for project structure, build commands, coding standards, and testing practices. Introduce AGENTS.md for contributor guidance, update README.md to reference new guidelines, and improve configuration documentation for NATS modes. Update Makefile and Taskfile with clearer run commands and requirements for development and production modes. Add production deployment guide and improve logging configuration for better observability.

Browse Source
This commit is contained in:
windyboy
2025-11-17 16:25:23 +08:00
parent 67abd66fa3
commit 704c7b80f6
34 changed files with 2976 additions and 782 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/dev-guide.md
+59 -11
View File
@@ -10,7 +10,11 @@ Spin up PostgreSQL/TimescaleDB and NATS JetStream in the background:
docker compose -f docker-compose.dev.yml up -d postgres nats nats-box
```
> Development mode defaults to `nats.mode = "core"`, so the processor consumes directly from the configured subject (`subscription.topic`). **However, the publisher always targets JetStream for deduplicated fan-out, so the provided Taskfile (and most examples below) override the mode to `jetstream`.** If you truly need core mode, set `CAATSM_NATS_MODE=core` manually and ensure any publishers use core subjects.
> **NATS Mode Selection:** The application supports two consumption modes:
> - **Core NATS mode** (default for development): Simple pub/sub without persistence or retry mechanisms. Recommended for local development and testing where message loss is acceptable. Fast and lightweight.
> - **JetStream mode**: Provides message persistence, ACK/NAK, automatic retries, and DLQ support. Recommended for production environments and integration testing.
>
> Development mode defaults to `nats.mode = "core"` in `config.dev.toml`. To use JetStream in development, set `CAATSM_NATS_MODE=jetstream` or change the config file. **Note:** The publisher always targets JetStream for deduplicated fan-out, so if you use Core mode for consumption, ensure your publishers align with your messaging strategy. See the README.md "NATS Mode Selection" section for a detailed comparison.
- `postgres` seeds the `aviation` schema using `internal/infra/postgres/telegrams.ddl` and exposes port `5432`.
- `nats` enables JetStream with client port `4222` and monitoring/UI on `8222`.
@@ -55,7 +59,7 @@ docker compose -f docker-compose.dev.yml up -d postgres nats nats-box
The `Taskfile.yml` includes helper targets that wrap the commands above:
- `task up` – starts PostgreSQL, NATS (JetStream, toolbox, and Prometheus exporter), and the observability stack (OpenTelemetry Collector, Jaeger, Prometheus, Grafana) using Docker Compose.
- `task dev-run` – ensures `task up` has run, exports the necessary `CAATSM_*` environment variables (including `CAATSM_NATS_MODE=jetstream`), and executes `go run ./cmd/main listen` with telemetry enabled.
- `task dev-run` – ensures `task up` has run, exports the necessary `CAATSM_*` environment variables (defaults to `CAATSM_NATS_MODE=core` for development), and executes `go run ./cmd/main listen` with telemetry enabled.
- `task down` – stops the entire stack and removes containers/volumes.
Use these tasks if you prefer a one-command workflow instead of invoking `docker compose` and environment exports manually.
@@ -64,10 +68,29 @@ Use these tasks if you prefer a one-command workflow instead of invoking `docker
Use the helper CLI in `cmd/seed-telegrams` to push realistic payloads onto NATS (mirrors the fixtures in `internal/adapter/parser/aviation_parser_test.go`):
### Publishing to JetStream (Recommended)
When using JetStream mode, publish messages to the JetStream stream:
```bash
# Insert rows into aviation.telegrams_raw and publish to NATS simultaneously
# Publish to JetStream stream (messages are persisted)
GO_ENV=dev go run ./cmd/seed-telegrams \
--nats-url nats://127.0.0.1:4222 \
--jetstream \
--stream TELEGRAM \
--js-subject telegram.serial \
--count 20 \
--category mixed \
--status random
```
### Publishing to Core NATS
For Core NATS mode, use standard publish:
```bash
# Publish to Core NATS (no persistence)
GO_ENV=dev go run ./cmd/seed-telegrams \
--postgres-url postgres://caatsm:caatsm@localhost:5432/aviation?sslmode=disable \
--nats-url nats://127.0.0.1:4222 \
--subject telegram.serial \
--count 20 \
@@ -75,13 +98,29 @@ GO_ENV=dev go run ./cmd/seed-telegrams \
--status random
```
- `--postgres-url` controls database insertion (omit to skip DB writes); metadata lands in `aviation.telegrams_raw.metadata`.
- `--dry-run` prints telegrams without touching NATS/Postgres.
- `--category` chooses ARR/DEP/CNL/DLA/FPL or `mixed`.
- `--status` controls stored/published status (`parsed|header_error|body_error|publish_error|repository_error|random`).
- `--no-nats` disables publishing; `--jetstream`, `--stream`, `--js-subject` toggle JetStream publishing.
- Inspect deliveries with `docker compose exec nats-box nats sub 'telegram.>'`.
- When running in core mode (default), the seeder publishes via standard `nc.Publish` and sets `Nats-Msg-Id` headers so the processor can derive message IDs.
### Common Options
- `--postgres-url`: Insert rows into `aviation.telegrams_raw` (omit to skip DB writes)
- `--dry-run`: Print telegrams without publishing to NATS/Postgres
- `--category`: Choose message type (`ARR|DEP|CNL|DLA|FPL|mixed`)
- `--status`: Control stored/published status (`parsed|header_error|body_error|publish_error|repository_error|random`)
- `--no-nats`: Disable publishing to NATS
- `--jetstream`: Enable JetStream publishing (requires `--stream` and `--js-subject`)
- `--stream`: JetStream stream name (default: `TELEGRAM`)
- `--js-subject`: Subject within the JetStream stream
### Inspecting Messages
```bash
# View messages in JetStream stream
docker compose exec nats-box nats stream view TELEGRAM
# Subscribe to messages (Core NATS or JetStream)
docker compose exec nats-box nats sub 'telegram.>'
# View consumer status and pending messages
docker compose exec nats-box nats consumer info TELEGRAM telegram-consumer
```
The main processor keeps consuming `subscription.topic` (defaults to `telegram.>`). Use the seeder to simulate parser failures, publish errors, or replay raw telegrams directly from the database.
@@ -96,6 +135,15 @@ The main processor keeps consuming `subscription.topic` (defaults to `telegram.>
2. **Run the processor with telemetry enabled**
```bash
# Using Core NATS mode (default for development)
CAATSM_TELEMETRY_ENABLED=true \
CAATSM_TELEMETRY_ENDPOINT=localhost:4318 \
CAATSM_TELEMETRY_INSECURE=true \
GO_ENV=dev \
CAATSM_POSTGRES_URL=postgres://caatsm:caatsm@localhost:5432/aviation?sslmode=disable \
go run ./cmd/main listen
# Or use JetStream mode for integration testing
CAATSM_TELEMETRY_ENABLED=true \
CAATSM_TELEMETRY_ENDPOINT=localhost:4318 \
CAATSM_TELEMETRY_INSECURE=true \
docs/prod-guide.md
+458
View File
@@ -0,0 +1,458 @@
# Production Deployment Guide
This guide covers deploying and running the CAATSM application in production environments.
## Overview
Production deployments require:
- **JetStream mode** (mandatory) - for message reliability and persistence
- **Manual Stream/Consumer creation** - not auto-created in production
- **Proper configuration** - using `configs/config.prod.toml`
- **SSL/TLS connections** - for secure database and telemetry connections
- **High availability** - stream replicas set to 3+ for HA
## Prerequisites
Before deploying to production:
1. **Build the binary:**
```bash
make build
# Binary will be at bin/receiver
```
2. **NATS JetStream cluster** - must be running and accessible
3. **PostgreSQL/TimescaleDB** - database must be accessible with SSL
4. **OpenTelemetry collector** (optional) - for observability
5. **Prometheus** (optional) - for metrics scraping
## Configuration
### 1. Production Configuration File
The repository includes a production configuration template at `configs/config.prod.toml`. Copy and customize it:
```bash
cp configs/config.prod.toml configs/config.prod.toml.local
# Edit config.prod.toml.local with your production values
```
**Key configuration sections:**
```toml
[nats]
url = "nats://nats.prod:4222"
mode = "jetstream" # Production MUST use JetStream
stream = "TELEGRAM"
consumer = "telegram-consumer"
[nats.stream_limits]
max_msgs = 1000000 # Adjust based on requirements
max_bytes = 1073741824 # 1GB
max_age = "168h" # 7 days
discard = "old"
storage = "file"
replicas = 3 # Use 3+ for HA in production
[nats.consumer_rules]
max_deliver = 5
ack_wait = "30s"
max_ack_pending = 1024
deliver_policy = "new" # Start from new messages in production
replay_policy = "instant"
backoff = ["5s", "30s", "2m", "5m"]
[subscription]
topic = "telegram.serial"
queue_group = "tele-queue"
[postgres]
url = "postgres://user:password@db.prod:5432/aviation?sslmode=require"
max_conns = 20
min_conns = 5
[app]
batch_size = 100
batch_timeout = "2s"
monitor_interval = "30s"
[log]
level = "info"
format = "json"
output = ["stdout"]
[telemetry]
enabled = true
endpoint = "otel-collector.prod:4318"
insecure = false # Use TLS in production
[monitoring]
disabled = false
addr = ":2112"
enable_metrics = true
enable_health = true
[dlq]
enabled = true
subject = "caatsm.dlq"
```
### 2. Environment Variables
Alternatively, you can use environment variables instead of a config file:
```bash
export GO_ENV=prod
export CAATSM_NATS_URL=nats://nats.prod:4222
export CAATSM_NATS_MODE=jetstream
export CAATSM_POSTGRES_URL=postgres://user:pass@db:5432/aviation?sslmode=require
export CAATSM_LOG_LEVEL=info
export CAATSM_TELEMETRY_ENABLED=true
export CAATSM_TELEMETRY_ENDPOINT=otel-collector.prod:4318
export CAATSM_TELEMETRY_INSECURE=false
```
## JetStream Setup
### Pre-create Stream and Consumer
**IMPORTANT:** The application does NOT auto-create streams/consumers in production. You must create them manually before starting the application.
#### Using NATS CLI
```bash
# Create stream
nats stream add TELEGRAM \
--subjects "telegram.serial,telegram.json" \
--storage file \
--replicas 3 \
--max-msgs 1000000 \
--max-bytes 1GB \
--max-age 7d \
--discard old
# Create consumer
nats consumer add TELEGRAM telegram-consumer \
--filter "telegram.serial" \
--ack explicit \
--deliver new \
--max-deliver 5 \
--ack-wait 30s \
--max-pending 1024
```
#### Using NATS Management API
You can also create streams/consumers programmatically using the NATS management API or configuration files.
### Verify Setup
```bash
# Check stream exists
nats stream info TELEGRAM
# Check consumer exists
nats consumer info TELEGRAM telegram-consumer
# Test connection
nats pub telegram.serial "ZCZC TEST 150631..."
```
## Running the Application
### Using Make
```bash
make run-prod # GO_ENV=prod (requires config.prod.toml)
```
### Using Task
```bash
task run-prod # GO_ENV=prod (requires config.prod.toml)
```
### Direct Execution
```bash
# Using binary with config file
GO_ENV=prod ./bin/receiver listen
# Or with environment variables (no config file needed)
GO_ENV=prod \
CAATSM_NATS_URL=nats://nats.prod:4222 \
CAATSM_NATS_MODE=jetstream \
CAATSM_POSTGRES_URL=postgres://user:pass@db:5432/aviation?sslmode=require \
./bin/receiver listen
```
## Production Checklist
Before deploying to production, verify:
- ✅ `nats.mode = "jetstream"` in config (mandatory)
- ✅ Stream and Consumer created manually
- ✅ Stream replicas set to 3+ for high availability
- ✅ PostgreSQL connection configured with SSL (`sslmode=require`)
- ✅ Log level set to `info` or `warn` (not `debug`)
- ✅ Log format set to `json` for log aggregation
- ✅ Telemetry endpoint configured (if using observability)
- ✅ Telemetry TLS enabled (`insecure = false`)
- ✅ Monitoring endpoints exposed for Prometheus scraping
- ✅ DLQ enabled for poison message handling
- ✅ Appropriate retention limits configured (max_msgs, max_bytes, max_age)
- ✅ Connection pool sizes appropriate for workload
- ✅ Batch sizes tuned for throughput
## Monitoring and Observability
### Health Endpoints
The application exposes health and readiness endpoints:
- `GET /livez` - Liveness endpoint (process health)
- `GET /readyz` - Readiness endpoint (checks PostgreSQL and NATS)
- `GET /healthz` - Alias for `/readyz`
- `GET /metrics` - Prometheus metrics
Configure Prometheus to scrape metrics:
```yaml
scrape_configs:
- job_name: 'caatsm'
static_configs:
- targets: ['caatsm:2112']
```
### Key Metrics
Monitor these metrics in production:
- `caatsm_messages_total{stream,consumer,result}` - Message throughput and results
- `caatsm_handle_latency_seconds_bucket` - Processing latency
- `caatsm_retries_total{stream,consumer,reason}` - Retry counts
- `caatsm_db_queries_total{operation,result}` - Database activity
- `caatsm_dlq_messages_total` - Dead-letter queue messages
- `caatsm_nats_consumer_pending_messages` - Consumer backlog/lag
### Logging
Production logs are in JSON format for easy parsing by log aggregation systems:
```json
{
"level": "info",
"ts": 1234567890.123,
"caller": "nats/consumer.go:123",
"msg": "Started consuming messages",
"subject": "telegram.serial",
"consumer": "telegram-consumer",
"stream": "TELEGRAM"
}
```
## High Availability
### Multiple Instances
Run multiple instances of the application for high availability:
- All instances use the same durable consumer name
- JetStream distributes messages across instances
- Each instance independently fetches messages
- If an instance fails, others continue processing
### Stream Replication
Configure stream with 3+ replicas for HA:
```toml
[nats.stream_limits]
replicas = 3 # Minimum 3 for HA, 5 for better distribution
```
### Database Connection Pooling
Configure appropriate connection pool sizes:
```toml
[postgres]
max_conns = 20 # Adjust based on number of instances
min_conns = 5
```
## Troubleshooting
### Stream Not Found
**Error:** `stream TELEGRAM not found`
**Solution:** Create the stream manually before starting the application (see "JetStream Setup" above).
### Consumer Not Found
**Error:** `consumer telegram-consumer not found in stream TELEGRAM`
**Solution:** Create the consumer manually before starting the application (see "JetStream Setup" above).
### Messages Not Being Consumed
**Symptoms:** High pending count, no messages processed
**Check:**
1. Verify consumer exists: `nats consumer info TELEGRAM telegram-consumer`
2. Check pending messages: `nats consumer next TELEGRAM telegram-consumer`
3. Verify application is running and connected
4. Check logs for errors
**Solutions:**
- Increase `batch_size` if processing is slow
- Add more consumer instances
- Check for processing errors in logs
### High Pending Count
**Symptoms:** Consumer has many pending messages
**Solutions:**
- Increase `batch_size` in config
- Add more application instances
- Check processing latency
- Verify database performance
### Messages Being Redelivered
**Symptoms:** Same messages processed multiple times
**Check:**
- Processing logs for errors
- `ack_wait` timeout may be too short
- Processing may be taking longer than `ack_wait`
**Solutions:**
- Increase `ack_wait` if processing takes longer
- Fix processing errors
- Check database connection and performance
### Connection Issues
**NATS Connection:**
- Verify NATS server is accessible
- Check network connectivity
- Verify NATS URL in config
**PostgreSQL Connection:**
- Verify database is accessible
- Check SSL certificate configuration
- Verify connection string format
- Check firewall rules
## Deployment Options
### Systemd Deployment
See `docs/deploy-systemd.md` for a complete systemd service deployment example.
### Kubernetes Deployment
See `docs/deploy-k8s.md` for Kubernetes deployment with ConfigMap/Secret and health probes.
## Performance Tuning
### Batch Processing
Adjust batch size based on message size and processing time:
```toml
[app]
batch_size = 100 # Increase for higher throughput
batch_timeout = "2s" # Adjust based on latency requirements
```
### Connection Pools
Tune database connection pool:
```toml
[postgres]
max_conns = 20 # Total connections across all instances
min_conns = 5 # Keep-alive connections
```
### Stream Retention
Configure retention based on requirements:
```toml
[nats.stream_limits]
max_msgs = 1000000 # Maximum messages
max_bytes = 1073741824 # Maximum size (1GB)
max_age = "168h" # Maximum age (7 days)
```
### Consumer Settings
Tune consumer for your workload:
```toml
[nats.consumer_rules]
max_ack_pending = 1024 # Increase for higher throughput
ack_wait = "30s" # Adjust based on processing time
backoff = ["5s", "30s", "2m", "5m"] # Retry delays
```
## Security Considerations
1. **Use SSL/TLS** for all connections:
- PostgreSQL: `sslmode=require`
- Telemetry: `insecure = false`
2. **Secure secrets** - Use environment variables or secret management:
- Database passwords
- NATS credentials
- API keys
3. **Network security**:
- Use private networks for internal services
- Restrict access to monitoring endpoints
- Use firewall rules appropriately
4. **Logging** - Avoid logging sensitive data:
- Don't log message payloads in production
- Use appropriate log levels
## Backup and Recovery
### Database Backups
Ensure regular backups of PostgreSQL/TimescaleDB:
- Use pg_dump or TimescaleDB backup tools
- Test restore procedures regularly
### JetStream State
JetStream state is stored in NATS:
- Ensure NATS cluster has proper backup procedures
- Stream data is replicated across cluster nodes
- Test disaster recovery procedures
### Message Replay
If needed, messages can be replayed from JetStream:
```bash
# Replay from a specific sequence
./bin/receiver listen --replay-from seq:12345
# Replay from a specific time
./bin/receiver listen --replay-from time:2024-11-15T08:00:00Z
```
## Support
For issues or questions:
- Check logs: `journalctl -u caatsm` (systemd) or container logs
- Review metrics in Prometheus/Grafana
- Check health endpoints: `curl http://localhost:2112/readyz`
- Consult deployment-specific documentation