🔧 Update Go version in go.mod and enhance build process with versioning information. Modify Makefile and Taskfile to inject build metadata (version, commit, build time) into the binary. Improve README with instructions for custom version builds and document new build info features. Add benchmarks for message parsing and processing to improve performance testing capabilities.
This commit is contained in:
windyboy
@@ -0,0 +1,299 @@
|
||||
# Database Migrations Guide
|
||||
|
||||
This document describes the database schema management and migration strategy for the CAATSM application.
|
||||
|
||||
## Current Approach
|
||||
|
||||
The application currently uses DDL (Data Definition Language) files for schema management:
|
||||
|
||||
- **Schema file**: `internal/infra/postgres/telegrams.ddl`
|
||||
- **Manual execution**: Schema changes are applied manually using `psql` or similar tools
|
||||
- **Version control**: DDL files are version-controlled in the repository
|
||||
|
||||
### Current Schema Structure
|
||||
|
||||
The application uses TimescaleDB (PostgreSQL extension) with the following key components:
|
||||
|
||||
- **Schema**: `aviation`
|
||||
- **Main table**: `aviation.telegrams` (hypertable for time-series data)
|
||||
- **Raw table**: `aviation.telegrams_raw` (for unparsed/failed messages)
|
||||
- **Indexes**: Multiple indexes on key fields for query performance
|
||||
|
||||
## Recommended Migration Tools
|
||||
|
||||
For production deployments, we recommend using a dedicated migration tool for better schema management:
|
||||
|
||||
### Option 1: golang-migrate (Recommended)
|
||||
|
||||
[golang-migrate](https://github.com/golang-migrate/migrate) is a popular Go-based migration tool with excellent PostgreSQL support.
|
||||
|
||||
**Installation:**
|
||||
```bash
|
||||
go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
|
||||
```
|
||||
|
||||
**Setup:**
|
||||
1. Create migrations directory:
|
||||
```bash
|
||||
mkdir -p migrations
|
||||
```
|
||||
|
||||
2. Create initial migration from existing schema:
|
||||
```bash
|
||||
migrate create -ext sql -dir migrations -seq initial_schema
|
||||
```
|
||||
|
||||
3. Copy DDL content to migration files:
|
||||
- `migrations/000001_initial_schema.up.sql` - Create schema
|
||||
- `migrations/000001_initial_schema.down.sql` - Drop schema
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Apply all migrations
|
||||
migrate -path migrations -database "postgres://user:pass@localhost:5432/aviation?sslmode=disable" up
|
||||
|
||||
# Rollback last migration
|
||||
migrate -path migrations -database "postgres://user:pass@localhost:5432/aviation?sslmode=disable" down 1
|
||||
|
||||
# Check migration version
|
||||
migrate -path migrations -database "postgres://user:pass@localhost:5432/aviation?sslmode=disable" version
|
||||
```
|
||||
|
||||
### Option 2: migrate (by golang-migrate, different package)
|
||||
|
||||
Similar to golang-migrate but distributed as a separate package.
|
||||
|
||||
### Option 3: Custom Migration Scripts
|
||||
|
||||
For simple deployments, you can create custom migration scripts that:
|
||||
- Check current schema version
|
||||
- Apply migrations sequentially
|
||||
- Track migration state in a `schema_migrations` table
|
||||
|
||||
## Migration Workflow
|
||||
|
||||
### Development
|
||||
|
||||
1. **Create migration file:**
|
||||
```bash
|
||||
migrate create -ext sql -dir migrations -seq add_new_column
|
||||
```
|
||||
|
||||
2. **Write up migration** (`migrations/XXXXXX_add_new_column.up.sql`):
|
||||
```sql
|
||||
ALTER TABLE aviation.telegrams
|
||||
ADD COLUMN new_field TEXT;
|
||||
|
||||
CREATE INDEX idx_telegrams_new_field ON aviation.telegrams (new_field);
|
||||
```
|
||||
|
||||
3. **Write down migration** (`migrations/XXXXXX_add_new_column.down.sql`):
|
||||
```sql
|
||||
DROP INDEX IF EXISTS idx_telegrams_new_field;
|
||||
ALTER TABLE aviation.telegrams
|
||||
DROP COLUMN IF EXISTS new_field;
|
||||
```
|
||||
|
||||
4. **Test migration:**
|
||||
```bash
|
||||
# Apply
|
||||
migrate -path migrations -database "$DATABASE_URL" up
|
||||
|
||||
# Rollback
|
||||
migrate -path migrations -database "$DATABASE_URL" down 1
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
1. **Backup database** before applying migrations:
|
||||
```bash
|
||||
pg_dump -U postgres -d aviation > backup_$(date +%Y%m%d_%H%M%S).sql
|
||||
```
|
||||
|
||||
2. **Test migration on staging** environment first
|
||||
|
||||
3. **Apply migration** during maintenance window:
|
||||
```bash
|
||||
migrate -path migrations -database "$DATABASE_URL" up
|
||||
```
|
||||
|
||||
4. **Verify migration** success:
|
||||
```bash
|
||||
migrate -path migrations -database "$DATABASE_URL" version
|
||||
```
|
||||
|
||||
5. **Monitor application** for any issues
|
||||
|
||||
## Schema Evolution Best Practices
|
||||
|
||||
### 1. Backward Compatibility
|
||||
|
||||
- **Additive changes** (new columns, indexes) are generally safe
|
||||
- **Removing columns** requires application code changes first
|
||||
- **Changing column types** requires careful planning and data migration
|
||||
|
||||
### 2. TimescaleDB Considerations
|
||||
|
||||
- **Hypertables**: Be careful when modifying hypertable structure
|
||||
- **Retention policies**: Consider impact on existing data
|
||||
- **Compression**: Test compression policies with schema changes
|
||||
|
||||
### 3. Index Management
|
||||
|
||||
- **Create indexes concurrently** in production to avoid locking:
|
||||
```sql
|
||||
CREATE INDEX CONCURRENTLY idx_telegrams_new_field ON aviation.telegrams (new_field);
|
||||
```
|
||||
|
||||
- **Drop unused indexes** to improve write performance
|
||||
|
||||
### 4. Data Migrations
|
||||
|
||||
For data transformations, use separate migration steps:
|
||||
|
||||
1. **Add new column** (nullable)
|
||||
2. **Backfill data** in application or migration script
|
||||
3. **Add constraints** (NOT NULL, etc.) after backfill
|
||||
4. **Remove old column** in separate migration
|
||||
|
||||
### 5. Rollback Procedures
|
||||
|
||||
Always provide rollback migrations:
|
||||
|
||||
- **Test rollback** on staging before production
|
||||
- **Document rollback steps** in migration comments
|
||||
- **Consider data loss** implications of rollbacks
|
||||
|
||||
## Example Migration
|
||||
|
||||
### Adding a New Index
|
||||
|
||||
**Up migration:**
|
||||
```sql
|
||||
-- Add index for querying by category and date
|
||||
CREATE INDEX CONCURRENTLY idx_telegrams_category_date
|
||||
ON aviation.telegrams (category, received_at DESC);
|
||||
```
|
||||
|
||||
**Down migration:**
|
||||
```sql
|
||||
-- Remove index
|
||||
DROP INDEX IF EXISTS idx_telegrams_category_date;
|
||||
```
|
||||
|
||||
### Adding a New Column
|
||||
|
||||
**Up migration:**
|
||||
```sql
|
||||
-- Add processing_metadata column for additional metadata
|
||||
ALTER TABLE aviation.telegrams
|
||||
ADD COLUMN processing_metadata JSONB;
|
||||
|
||||
-- Add index for JSONB queries
|
||||
CREATE INDEX CONCURRENTLY idx_telegrams_processing_metadata_gin
|
||||
ON aviation.telegrams USING GIN (processing_metadata);
|
||||
```
|
||||
|
||||
**Down migration:**
|
||||
```sql
|
||||
-- Remove index and column
|
||||
DROP INDEX IF EXISTS idx_telegrams_processing_metadata_gin;
|
||||
ALTER TABLE aviation.telegrams
|
||||
DROP COLUMN IF EXISTS processing_metadata;
|
||||
```
|
||||
|
||||
## Migration State Management
|
||||
|
||||
### Schema Version Tracking
|
||||
|
||||
Migration tools typically use a `schema_migrations` table to track applied migrations:
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version BIGINT NOT NULL PRIMARY KEY,
|
||||
dirty BOOLEAN NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
### Checking Migration Status
|
||||
|
||||
```bash
|
||||
# Check current version
|
||||
migrate -path migrations -database "$DATABASE_URL" version
|
||||
|
||||
# Check for pending migrations
|
||||
migrate -path migrations -database "$DATABASE_URL" up
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### Automated Migration Testing
|
||||
|
||||
Add migration tests to CI pipeline:
|
||||
|
||||
```yaml
|
||||
# Example GitHub Actions workflow
|
||||
- name: Test migrations
|
||||
run: |
|
||||
# Start test database
|
||||
docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=test postgres:15
|
||||
|
||||
# Wait for database
|
||||
sleep 5
|
||||
|
||||
# Apply migrations
|
||||
migrate -path migrations -database "postgres://postgres:test@localhost:5432/test?sslmode=disable" up
|
||||
|
||||
# Verify schema
|
||||
psql "postgres://postgres:test@localhost:5432/test?sslmode=disable" -c "\d aviation.telegrams"
|
||||
```
|
||||
|
||||
### Deployment Automation
|
||||
|
||||
For production deployments, integrate migrations into deployment process:
|
||||
|
||||
1. **Pre-deployment**: Backup database
|
||||
2. **Deployment**: Apply migrations
|
||||
3. **Post-deployment**: Verify migration success
|
||||
4. **Rollback**: If migration fails, rollback application and database
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Migration Failures
|
||||
|
||||
**Common issues:**
|
||||
- **Lock conflicts**: Use `CONCURRENTLY` for index creation
|
||||
- **Timeout errors**: Increase migration timeout for large tables
|
||||
- **Dirty state**: Manually fix `schema_migrations` table if migration fails mid-way
|
||||
|
||||
**Recovery:**
|
||||
```sql
|
||||
-- Check migration state
|
||||
SELECT * FROM schema_migrations;
|
||||
|
||||
-- Fix dirty state (if needed)
|
||||
UPDATE schema_migrations SET dirty = false WHERE version = X;
|
||||
```
|
||||
|
||||
### Performance Considerations
|
||||
|
||||
- **Large tables**: Test migrations on production-sized data
|
||||
- **Downtime**: Plan for maintenance windows for major schema changes
|
||||
- **Replication lag**: Consider impact on read replicas
|
||||
|
||||
## Future Improvements
|
||||
|
||||
Consider implementing:
|
||||
|
||||
1. **Automated migration testing** in CI/CD
|
||||
2. **Migration rollback automation** in deployment pipeline
|
||||
3. **Schema validation** before applying migrations
|
||||
4. **Migration dry-run** mode for testing
|
||||
5. **Migration status monitoring** and alerting
|
||||
|
||||
## References
|
||||
|
||||
- [golang-migrate Documentation](https://github.com/golang-migrate/migrate)
|
||||
- [TimescaleDB Best Practices](https://docs.timescale.com/timescaledb/latest/how-to-guides/migrate-data/)
|
||||
- [PostgreSQL Migration Guide](https://www.postgresql.org/docs/current/ddl-alter.html)
|
||||
|
||||
+699
@@ -4,17 +4,45 @@
|
||||
|
||||
The NATS integration provides a robust, production-ready message processing system built on Clean Architecture principles. It supports both JetStream (persistent) and Core NATS (fire-and-forget) modes with comprehensive error handling, observability, and resilience features.
|
||||
|
||||
### Key Concepts
|
||||
|
||||
1. **Consumer**: Pulls messages from NATS JetStream in batches, processes them, and handles ACKs/NAKs
|
||||
2. **Publisher**: Publishes messages to NATS with automatic deduplication via UUID headers
|
||||
3. **Batch Processing**: Fetches multiple messages at once (configurable size) for efficiency
|
||||
4. **Error Classification**: Distinguishes between transient (retry) and permanent (DLQ) errors
|
||||
5. **Dead Letter Queue (DLQ)**: Routes failed messages to a separate queue for analysis
|
||||
6. **Backpressure**: Automatically slows down processing when errors accumulate
|
||||
7. **Self-Healing**: Automatically recreates missing streams/consumers in development
|
||||
8. **Observability**: Built-in metrics, tracing, and structured logging
|
||||
|
||||
### Quick Start Flow
|
||||
|
||||
```
|
||||
1. Configure NATS connection and consumer settings
|
||||
2. Create Consumer with dependencies (processor, logger, telemetry)
|
||||
3. Start Consumer - begins fetching and processing messages
|
||||
4. Messages flow: Fetch → Process → ACK/NAK/DLQ
|
||||
5. Errors handled automatically with retries and backoff
|
||||
6. Graceful shutdown on context cancellation
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
### Clean Architecture Layers
|
||||
|
||||
The NATS integration follows Clean Architecture principles, separating concerns into distinct layers:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ Port Interfaces │
|
||||
│ (Publisher, Consumer contracts) │
|
||||
│ - Define contracts, not impl │
|
||||
│ - Enable dependency inversion │
|
||||
├─────────────────────────────────────┤
|
||||
│ Application Layer │
|
||||
│ (Message processing logic) │
|
||||
│ - Business logic │
|
||||
│ - Use case orchestration │
|
||||
├─────────────────────────────────────┤
|
||||
│ Infrastructure Layer │
|
||||
│ (NATS implementation details) │
|
||||
@@ -39,6 +67,337 @@ The NATS integration provides a robust, production-ready message processing syst
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Component Interaction Diagram
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ Publisher │
|
||||
│ │
|
||||
│ 1. Serialize │
|
||||
│ 2. Add UUID │
|
||||
│ 3. Publish │
|
||||
└──────┬───────┘
|
||||
│
|
||||
│ Publish to Subject
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ NATS JetStream │
|
||||
│ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ Stream │ │
|
||||
│ │ (TELEGRAM) │ │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
│ ┌──────▼───────┐ │
|
||||
│ │ Consumer │ │
|
||||
│ │ (Pull Sub) │ │
|
||||
│ └──────┬───────┘ │
|
||||
└─────────┼───────────────────────────┘
|
||||
│
|
||||
│ Fetch Batch
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ Consumer │
|
||||
│ │
|
||||
│ ┌──────────────────────────────┐ │
|
||||
│ │ MessageFetcher │ │
|
||||
│ │ - FetchBatch() │ │
|
||||
│ │ - HandleFetchError() │ │
|
||||
│ └──────────┬───────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────▼───────────────────┐ │
|
||||
│ │ MessageProcessor │ │
|
||||
│ │ - ProcessBatch() │ │
|
||||
│ │ - ProcessSingleMessage() │ │
|
||||
│ └──────────┬───────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────▼───────────────────┐ │
|
||||
│ │ ErrorHandler │ │
|
||||
│ │ - Classify errors │ │
|
||||
│ │ - Apply backpressure │ │
|
||||
│ └──────────┬───────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────▼───────────────────┐ │
|
||||
│ │ DLQHandler │ │
|
||||
│ │ - RouteToDLQ() │ │
|
||||
│ │ - AdvisoryDLQHandler │ │
|
||||
│ └──────────────────────────────┘ │
|
||||
└─────────────────────────────────────┘
|
||||
│
|
||||
│ ACK/NAK
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ Application Processor │
|
||||
│ (Business Logic) │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Architecture Principles
|
||||
|
||||
1. **Dependency Inversion**: High-level modules (Consumer, Publisher) depend on abstractions (interfaces), not concrete implementations
|
||||
2. **Separation of Concerns**: Each component has a single responsibility:
|
||||
- `MessageFetcher`: Handles message retrieval
|
||||
- `MessageProcessor`: Handles message processing logic
|
||||
- `ErrorHandler`: Handles error classification and recovery
|
||||
- `DLQHandler`: Handles dead letter queue routing
|
||||
3. **Testability**: All components can be mocked and tested independently
|
||||
4. **Extensibility**: New implementations can be added without modifying existing code
|
||||
|
||||
## Logic Flow
|
||||
|
||||
### Consumer Processing Flow
|
||||
|
||||
The consumer follows a well-defined processing loop with error handling at each stage:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Consumer Start │
|
||||
│ 1. Initialize components (Fetcher, Processor, DLQ) │
|
||||
│ 2. Create/validate JetStream resources │
|
||||
│ 3. Start advisory DLQ handler (if enabled) │
|
||||
│ 4. Start metrics collection goroutine │
|
||||
└──────────────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Main Loop │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────┐ │
|
||||
│ │ Step 1: Check Context │ │
|
||||
│ │ - If cancelled, exit gracefully │ │
|
||||
│ └──────────────────┬───────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────▼───────────────────────────────────┐ │
|
||||
│ │ Step 2: Fetch Batch │ │
|
||||
│ │ - Fetch up to BatchSize messages │ │
|
||||
│ │ - Wait up to BatchTimeout │ │
|
||||
│ │ - Handle fetch errors with recovery │ │
|
||||
│ └──────────────────┬───────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌───────────┴───────────┐ │
|
||||
│ │ │ │
|
||||
│ Success Error │
|
||||
│ │ │ │
|
||||
│ │ ┌────────▼────────┐ │
|
||||
│ │ │ Handle Error │ │
|
||||
│ │ │ - Classify type │ │
|
||||
│ │ │ - Apply backoff │ │
|
||||
│ │ │ - Recover if dev│ │
|
||||
│ │ └────────┬────────┘ │
|
||||
│ │ │ │
|
||||
│ │ ┌────────▼────────┐ │
|
||||
│ │ │ Continue? │ │
|
||||
│ │ └────────┬────────┘ │
|
||||
│ │ │ │
|
||||
│ │ Yes │ No │
|
||||
│ │ │ │ │
|
||||
│ │ └───┬────┴───┐ │
|
||||
│ │ │ │ │
|
||||
│ │ Continue Exit │
|
||||
│ │ │ │
|
||||
│ └──────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────▼───────────────────────────────────┐ │
|
||||
│ │ Step 3: Process Batch │ │
|
||||
│ │ - For each message in batch: │ │
|
||||
│ │ * Check context │ │
|
||||
│ │ * Extract message ID │ │
|
||||
│ │ * Create tracing span │ │
|
||||
│ │ * Call processor.Handle() │ │
|
||||
│ │ * Handle result (ACK/NAK/DLQ) │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌───────────┴───────────┐ │
|
||||
│ │ │ │
|
||||
│ Success Error │
|
||||
│ │ │ │
|
||||
│ │ ┌────────▼────────┐ │
|
||||
│ │ │ Classify Error │ │
|
||||
│ │ └────────┬────────┘ │
|
||||
│ │ │ │
|
||||
│ │ ┌─────────────┴─────────────┐ │
|
||||
│ │ │ │ │
|
||||
│ │ Permanent Transient │
|
||||
│ │ │ │ │
|
||||
│ │ ┌──────▼──────┐ ┌────────▼──────┐ │
|
||||
│ │ │ Route to DLQ│ │ NAK with delay│ │
|
||||
│ │ │ ACK message │ │ Apply backpres│ │
|
||||
│ │ └──────┬──────┘ └────────┬──────┘ │
|
||||
│ │ │ │ │
|
||||
│ └─────────┴───────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────▼───────────────────────────────────┐ │
|
||||
│ │ Step 4: Reset Error Streak (if successful) │ │
|
||||
│ └──────────────────┬───────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ └─────────── Loop ─────────────────────┘
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Message Processing Logic
|
||||
|
||||
#### Single Message Processing Flow
|
||||
|
||||
```
|
||||
Message Received
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Extract Message ID │
|
||||
│ - Check header │
|
||||
│ - Fallback to meta │
|
||||
│ - Generate if none │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Create Trace Span │
|
||||
│ - Add attributes │
|
||||
│ - Propagate context │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Process Message │
|
||||
│ - Call processor │
|
||||
│ - Business logic │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
┌──────┴──────┐
|
||||
│ │
|
||||
Success Error
|
||||
│ │
|
||||
│ ┌──────▼──────────┐
|
||||
│ │ Classify Error │
|
||||
│ └──────┬──────────┘
|
||||
│ │
|
||||
│ ┌──────┴──────┐
|
||||
│ │ │
|
||||
│ Permanent Transient
|
||||
│ │ │
|
||||
│ ┌───▼───┐ ┌────▼────┐
|
||||
│ │ DLQ │ │ NAK │
|
||||
│ │ ACK │ │ Backoff │
|
||||
│ └───┬───┘ └────┬────┘
|
||||
│ │ │
|
||||
└──────┴─────────────┘
|
||||
│
|
||||
▼
|
||||
End Processing
|
||||
```
|
||||
|
||||
#### Error Handling Logic
|
||||
|
||||
```
|
||||
Error Occurred
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Is Permanent Error? │
|
||||
│ - app.IsPermanent() │
|
||||
└──────┬──────────────┘
|
||||
│
|
||||
┌───┴───┐
|
||||
│ │
|
||||
Yes No
|
||||
│ │
|
||||
│ ┌───▼──────────────────────┐
|
||||
│ │ Increment Error Streak │
|
||||
│ └───┬──────────────────────┘
|
||||
│ │
|
||||
│ ┌───▼──────────────────────┐
|
||||
│ │ Streak >= Threshold? │
|
||||
│ │ (default: 10 errors) │
|
||||
│ └───┬──────────────────────┘
|
||||
│ │
|
||||
│ ┌───┴───┐
|
||||
│ │ │
|
||||
│ Yes No
|
||||
│ │ │
|
||||
│ │ ┌───▼──────────────┐
|
||||
│ │ │ NAK with delay │
|
||||
│ │ │ - Use backoff │
|
||||
│ │ │ - Request retry │
|
||||
│ │ └──────────────────┘
|
||||
│ │
|
||||
│ ▼
|
||||
│ ┌──────────────────────┐
|
||||
│ │ Apply Backpressure │
|
||||
│ │ - Sleep: errors*100ms│
|
||||
│ │ - Max: 5 seconds │
|
||||
│ └───┬──────────────────┘
|
||||
│ │
|
||||
│ ▼
|
||||
│ ┌──────────────────────┐
|
||||
│ │ NAK with delay │
|
||||
│ └──────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Route to DLQ │
|
||||
│ - Enrich metadata │
|
||||
│ - Publish to DLQ │
|
||||
│ - ACK original msg │
|
||||
└──────────────────────┘
|
||||
```
|
||||
|
||||
### Publisher Logic Flow
|
||||
|
||||
```
|
||||
Publish Request
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Validate Topic │
|
||||
│ - Check config │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Serialize Message │
|
||||
│ - JSON marshal │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Extract/Generate ID │
|
||||
│ - From message.Uuid │
|
||||
│ - Or generate UUID │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Set Header │
|
||||
│ - Nats-Msg-Id │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Publish to NATS │
|
||||
│ - js.PublishMsg() │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
┌──────┴──────┐
|
||||
│ │
|
||||
Success Error
|
||||
│ │
|
||||
│ ┌──────▼──────────┐
|
||||
│ │ Classify Error │
|
||||
│ └──────┬──────────┘
|
||||
│ │
|
||||
│ ┌──────┴──────┐
|
||||
│ │ │
|
||||
│ Transient Permanent
|
||||
│ │ │
|
||||
│ ┌───▼───┐ ┌────▼────┐
|
||||
│ │ Retry │ │ Fail │
|
||||
│ │ Later │ │ Fast │
|
||||
│ └───────┘ └─────────┘
|
||||
│
|
||||
▼
|
||||
Success
|
||||
```
|
||||
|
||||
## Core Components
|
||||
|
||||
### Consumer
|
||||
@@ -57,6 +416,38 @@ The consumer handles message consumption with the following features:
|
||||
- **Self-Healing**: Automatic recreation of missing streams/consumers in dev environments
|
||||
- **Graceful Shutdown**: Proper cleanup and draining of connections
|
||||
|
||||
#### Component Logic
|
||||
|
||||
**MessageFetcher (`defaultMessageFetcher`)**
|
||||
- Fetches batches of messages using `sub.Fetch(batchSize, MaxWait(timeout))`
|
||||
- Handles fetch errors with exponential backoff
|
||||
- Recovers subscriptions when connection issues occur
|
||||
- Context-aware: respects cancellation signals
|
||||
|
||||
**MessageProcessor (`defaultBatchProcessor`)**
|
||||
- Processes messages sequentially within a batch
|
||||
- Extracts message IDs (header → metadata → generated)
|
||||
- Creates OpenTelemetry spans for tracing
|
||||
- Calls application processor for business logic
|
||||
- Handles ACK/NAK based on processing results
|
||||
|
||||
**ErrorHandler**
|
||||
- Classifies errors as transient or permanent using `app.IsPermanent()`
|
||||
- Tracks consecutive error streaks
|
||||
- Applies backpressure when streak exceeds threshold (default: 10)
|
||||
- Calculates backoff delays for retries
|
||||
|
||||
**DLQHandler (`defaultDLQHandler`)**
|
||||
- Routes permanent errors to DLQ with enriched metadata
|
||||
- Validates DLQ stream exists at startup
|
||||
- Publishes DLQ messages with error context
|
||||
|
||||
**AdvisoryDLQHandler**
|
||||
- Subscribes to JetStream advisory events: `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.*`
|
||||
- Handles messages that exhaust MaxDeliver attempts
|
||||
- Retrieves original message from stream using `GetMsg()`
|
||||
- Routes to DLQ with advisory metadata
|
||||
|
||||
#### Configuration
|
||||
```toml
|
||||
[NATS]
|
||||
@@ -92,6 +483,26 @@ The publisher handles message publishing with deduplication and observability.
|
||||
- **Structured Logging**: Comprehensive logging of publish operations
|
||||
- **Error Classification**: Distinguishes between transient and permanent errors
|
||||
|
||||
#### Component Logic
|
||||
|
||||
**Publishing Flow**
|
||||
1. **Validation**: Checks that publisher topic is configured
|
||||
2. **Serialization**: Marshals message to JSON using `json.Marshal()`
|
||||
3. **ID Extraction**: Extracts UUID from message (if `ParsedTelegram` type) or generates new UUID
|
||||
4. **Header Attachment**: Sets `Nats-Msg-Id` header for deduplication
|
||||
5. **Publishing**: Calls `js.PublishMsg()` to publish to JetStream
|
||||
6. **Error Handling**: Classifies errors as transient (`ErrNoResponders`) or permanent
|
||||
|
||||
**Deduplication Strategy**
|
||||
- Uses `Nats-Msg-Id` header for JetStream deduplication
|
||||
- Extracts UUID from `ParsedTelegram.Uuid` field if available
|
||||
- Falls back to generating new UUID if not present
|
||||
- JetStream uses this header to prevent duplicate message processing
|
||||
|
||||
**Error Classification**
|
||||
- **Transient**: `nats.ErrNoResponders` - JetStream temporarily unavailable, should retry
|
||||
- **Permanent**: Other errors - configuration issues, should fail fast
|
||||
|
||||
### Error Handling
|
||||
|
||||
#### Error Types
|
||||
@@ -196,6 +607,294 @@ CAATSM_DLQ_SUBJECT=caatsm.dlq
|
||||
- **Performance**: Load testing and resource usage
|
||||
- **Configuration**: Different configuration combinations
|
||||
|
||||
## Simple Examples
|
||||
|
||||
### Example 1: Complete Consumer Setup and Start
|
||||
|
||||
This example shows how to set up and start a consumer from scratch:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"caatsm/internal/infra/config"
|
||||
"caatsm/internal/infra/nats"
|
||||
"caatsm/internal/app"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// 1. Load configuration
|
||||
cfg := &config.Config{
|
||||
NATS: config.NATSConfig{
|
||||
URL: "nats://localhost:4222",
|
||||
Mode: "jetstream",
|
||||
Stream: "TELEGRAM",
|
||||
Consumer: "telegram-consumer",
|
||||
ConsumerRules: config.ConsumerRules{
|
||||
AckWait: 30 * time.Second,
|
||||
MaxDeliver: 3,
|
||||
Backoff: []time.Duration{1*time.Second, 2*time.Second, 5*time.Second},
|
||||
},
|
||||
},
|
||||
App: config.AppConfig{
|
||||
BatchSize: 50,
|
||||
BatchTimeout: 2 * time.Second,
|
||||
},
|
||||
DLQ: config.DLQConfig{
|
||||
Enabled: true,
|
||||
Subject: "caatsm.dlq",
|
||||
},
|
||||
}
|
||||
|
||||
// 2. Create NATS connection
|
||||
nc, _ := nats.Connect(cfg.NATS.URL)
|
||||
defer nc.Close()
|
||||
|
||||
// 3. Get JetStream context
|
||||
js, _ := nc.JetStream()
|
||||
|
||||
// 4. Create message processor (your business logic)
|
||||
processor := app.NewMessageProcessor(/* dependencies */)
|
||||
|
||||
// 5. Create logger
|
||||
logger, _ := zap.NewProduction()
|
||||
|
||||
// 6. Create telemetry recorder
|
||||
telemetry := /* your telemetry implementation */
|
||||
|
||||
// 7. Create consumer
|
||||
consumer, err := natsinfra.ProvideConsumer(
|
||||
nc,
|
||||
js,
|
||||
processor,
|
||||
cfg,
|
||||
telemetry,
|
||||
logger,
|
||||
)
|
||||
if err != nil {
|
||||
logger.Fatal("Failed to create consumer", zap.Error(err))
|
||||
}
|
||||
|
||||
// 8. Start consumer with context
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
// Handle graceful shutdown
|
||||
go func() {
|
||||
// Wait for interrupt signal
|
||||
<-ctx.Done()
|
||||
shutdownCtx, _ := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
consumer.Shutdown(shutdownCtx)
|
||||
}()
|
||||
|
||||
// 9. Start consuming (blocks until context cancelled)
|
||||
if err := consumer.Start(ctx); err != nil {
|
||||
logger.Error("Consumer stopped", zap.Error(err))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example 2: Publishing a Message
|
||||
|
||||
Simple example of publishing a message:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"caatsm/internal/adapter/dto"
|
||||
"caatsm/internal/infra/nats"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
func publishMessage(publisher port.Publisher) error {
|
||||
// Create message with UUID
|
||||
message := &dto.ParsedTelegram{
|
||||
Uuid: uuid.NewString(), // Used for deduplication
|
||||
Data: []byte("telegram message data"),
|
||||
// ... other fields
|
||||
}
|
||||
|
||||
// Publish - automatically handles:
|
||||
// - JSON serialization
|
||||
// - UUID header attachment
|
||||
// - Error classification
|
||||
if err := publisher.Publish(message); err != nil {
|
||||
return fmt.Errorf("failed to publish: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
### Example 3: Message Processing Flow
|
||||
|
||||
Step-by-step what happens when a message is processed:
|
||||
|
||||
```go
|
||||
// Step 1: Consumer fetches batch of messages
|
||||
msgs, err := subscription.Fetch(50, nats.MaxWait(2*time.Second))
|
||||
// Result: Up to 50 messages, or timeout after 2 seconds
|
||||
|
||||
// Step 2: For each message in batch
|
||||
for _, msg := range msgs {
|
||||
// Step 2a: Extract message ID
|
||||
msgID := msg.Header.Get("Nats-Msg-Id")
|
||||
if msgID == "" {
|
||||
// Fallback: use JetStream sequence
|
||||
meta, _ := msg.Metadata()
|
||||
msgID = fmt.Sprintf("js-%d", meta.Sequence.Stream)
|
||||
}
|
||||
|
||||
// Step 2b: Create tracing span
|
||||
ctx, span := tracer.Start(ctx, "process.message")
|
||||
span.SetAttributes(
|
||||
attribute.String("messaging.system", "nats"),
|
||||
attribute.String("messaging.destination.name", msg.Subject),
|
||||
)
|
||||
|
||||
// Step 2c: Process message (your business logic)
|
||||
err := processor.Handle(ctx, msg.Data, msgID)
|
||||
|
||||
// Step 2d: Handle result
|
||||
if err != nil {
|
||||
if app.IsPermanent(err) {
|
||||
// Permanent error: route to DLQ and ACK
|
||||
dlqHandler.RouteToDLQ(ctx, msg, err)
|
||||
msg.Ack()
|
||||
} else {
|
||||
// Transient error: NAK with backoff
|
||||
msg.NakWithDelay(calculateBackoff(msg))
|
||||
}
|
||||
} else {
|
||||
// Success: ACK message
|
||||
msg.Ack()
|
||||
}
|
||||
|
||||
span.End()
|
||||
}
|
||||
```
|
||||
|
||||
### Example 4: Error Handling Scenarios
|
||||
|
||||
Different error scenarios and how they're handled:
|
||||
|
||||
```go
|
||||
// Scenario 1: Transient Error (Network Issue)
|
||||
func processMessage(msg *nats.Msg) error {
|
||||
// Simulate network error
|
||||
if networkDown {
|
||||
return app.NewTransientError("network unavailable")
|
||||
}
|
||||
// Result: Message is NAK'd, will be redelivered with backoff
|
||||
}
|
||||
|
||||
// Scenario 2: Permanent Error (Invalid Format)
|
||||
func processMessage(msg *nats.Msg) error {
|
||||
var data MyStruct
|
||||
if err := json.Unmarshal(msg.Data, &data); err != nil {
|
||||
return app.NewPermanentError("invalid JSON format")
|
||||
}
|
||||
// Result: Message routed to DLQ, original message ACK'd
|
||||
}
|
||||
|
||||
// Scenario 3: Backpressure Trigger
|
||||
// When 10+ consecutive errors occur:
|
||||
// - Processing pauses
|
||||
// - Sleep duration = min(consecutiveErrors * 100ms, 5s)
|
||||
// - Prevents overwhelming the system
|
||||
|
||||
// Scenario 4: MaxDeliver Exhausted
|
||||
// When message fails MaxDeliver times (default: 3):
|
||||
// - JetStream publishes advisory event
|
||||
// - AdvisoryDLQHandler catches event
|
||||
// - Retrieves original message
|
||||
// - Routes to DLQ with metadata
|
||||
```
|
||||
|
||||
### Example 5: DLQ Message Structure
|
||||
|
||||
What a DLQ message looks like:
|
||||
|
||||
```json
|
||||
{
|
||||
"transport_msg_id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"subject": "telegram.orders.12345",
|
||||
"stream": "TELEGRAM",
|
||||
"consumer": "telegram-consumer",
|
||||
"nats_sequence": 12345,
|
||||
"deliveries": 3,
|
||||
"error": "permanent error: invalid message format",
|
||||
"received_at": "2024-01-15T10:30:00Z",
|
||||
"body": "{\"order_id\":123,\"invalid\":\"data\"}",
|
||||
"advisory_source": false
|
||||
}
|
||||
```
|
||||
|
||||
### Example 6: Configuration Examples
|
||||
|
||||
Different configuration scenarios:
|
||||
|
||||
```toml
|
||||
# Example 1: High Throughput Configuration
|
||||
[NATS]
|
||||
Mode = "jetstream"
|
||||
Stream = "TELEGRAM"
|
||||
Consumer = "telegram-consumer"
|
||||
|
||||
[NATS.ConsumerRules]
|
||||
AckWait = "60s"
|
||||
MaxDeliver = 5
|
||||
MaxAckPending = 5000
|
||||
Backoff = ["1s", "2s", "5s", "10s", "30s"]
|
||||
|
||||
[App]
|
||||
BatchSize = 100 # Larger batches
|
||||
BatchTimeout = "5s" # Longer timeout
|
||||
|
||||
# Example 2: Low Latency Configuration
|
||||
[App]
|
||||
BatchSize = 10 # Smaller batches
|
||||
BatchTimeout = "500ms" # Shorter timeout
|
||||
|
||||
# Example 3: Development Mode (Self-Healing)
|
||||
[NATS]
|
||||
Mode = "jetstream"
|
||||
# Missing streams/consumers auto-created
|
||||
|
||||
# Example 4: Production Mode (Fail Fast)
|
||||
[NATS]
|
||||
Mode = "jetstream"
|
||||
# Missing streams/consumers cause startup failure
|
||||
```
|
||||
|
||||
### Example 7: Observability Integration
|
||||
|
||||
How to monitor the consumer:
|
||||
|
||||
```go
|
||||
// Metrics are automatically collected:
|
||||
// - ack_pending: Messages waiting for ACK
|
||||
// - redelivered: Messages being redelivered
|
||||
// - pending: Messages in stream
|
||||
// - delivered: Total messages delivered
|
||||
|
||||
// Tracing spans are created for:
|
||||
// - Each message processing
|
||||
// - DLQ routing
|
||||
// - Error handling
|
||||
|
||||
// Logs include:
|
||||
// - Message processing events
|
||||
// - Error details with context
|
||||
// - DLQ routing events
|
||||
// - Connection status changes
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Consumer Setup
|
||||
|
||||
@@ -0,0 +1,396 @@
|
||||
# Performance Tuning Guide
|
||||
|
||||
This document provides guidelines for optimizing the performance of the CAATSM application.
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
Key performance indicators to monitor:
|
||||
|
||||
- **Message throughput**: Messages processed per second
|
||||
- **Latency**: End-to-end processing time (NATS receive → DB insert → publish)
|
||||
- **Database query time**: Time spent on database operations
|
||||
- **Memory usage**: Application memory consumption
|
||||
- **CPU usage**: CPU utilization
|
||||
- **Connection pool utilization**: Database and NATS connection usage
|
||||
|
||||
## Batch Processing Configuration
|
||||
|
||||
### Current Implementation
|
||||
|
||||
The application processes messages in batches for efficiency:
|
||||
|
||||
```toml
|
||||
[app]
|
||||
batch_size = 50 # Number of messages per batch
|
||||
batch_timeout = "2s" # Maximum wait time for a batch
|
||||
monitor_interval = "30s" # Consumer metrics reporting interval
|
||||
```
|
||||
|
||||
### Tuning Guidelines
|
||||
|
||||
**Batch Size:**
|
||||
- **Small batches (10-50)**: Lower latency, higher overhead
|
||||
- **Medium batches (50-200)**: Balanced latency and throughput
|
||||
- **Large batches (200-1000)**: Higher throughput, higher latency
|
||||
|
||||
**Recommendations:**
|
||||
- **Development**: 10-50 messages (faster feedback)
|
||||
- **Production (low latency)**: 50-100 messages
|
||||
- **Production (high throughput)**: 200-500 messages
|
||||
|
||||
**Batch Timeout:**
|
||||
- **Low latency**: 500ms-1s (process quickly even with small batches)
|
||||
- **Balanced**: 2-5s (good balance)
|
||||
- **High throughput**: 5-10s (wait for larger batches)
|
||||
|
||||
### Example Configuration
|
||||
|
||||
```toml
|
||||
# High-throughput production configuration
|
||||
[app]
|
||||
batch_size = 200
|
||||
batch_timeout = "5s"
|
||||
monitor_interval = "30s"
|
||||
|
||||
# Low-latency production configuration
|
||||
[app]
|
||||
batch_size = 50
|
||||
batch_timeout = "1s"
|
||||
monitor_interval = "30s"
|
||||
```
|
||||
|
||||
## Database Connection Pooling
|
||||
|
||||
### Configuration
|
||||
|
||||
```toml
|
||||
[postgres]
|
||||
url = "postgres://user:pass@db:5432/aviation?sslmode=require"
|
||||
max_conns = 20 # Maximum connections in pool
|
||||
min_conns = 5 # Minimum connections in pool
|
||||
```
|
||||
|
||||
### Tuning Guidelines
|
||||
|
||||
**Connection Pool Size:**
|
||||
- **Formula**: `max_conns = (expected_concurrent_requests * avg_query_time) / target_latency`
|
||||
- **Minimum**: 2-5 connections (small deployments)
|
||||
- **Recommended**: 10-20 connections (medium deployments)
|
||||
- **Maximum**: 50-100 connections (high-throughput deployments)
|
||||
|
||||
**Considerations:**
|
||||
- Each connection consumes memory (~2-5MB)
|
||||
- PostgreSQL has a maximum connection limit (default: 100)
|
||||
- Too many connections can degrade performance
|
||||
- Use connection pooler (PgBouncer) for high concurrency
|
||||
|
||||
### Example Configurations
|
||||
|
||||
```toml
|
||||
# Small deployment (single instance)
|
||||
[postgres]
|
||||
max_conns = 10
|
||||
min_conns = 2
|
||||
|
||||
# Medium deployment (2-3 instances)
|
||||
[postgres]
|
||||
max_conns = 20
|
||||
min_conns = 5
|
||||
|
||||
# Large deployment (5+ instances, use PgBouncer)
|
||||
[postgres]
|
||||
max_conns = 10 # Per instance
|
||||
min_conns = 2
|
||||
# Use PgBouncer with pool_mode=transaction
|
||||
```
|
||||
|
||||
### Connection Pool Monitoring
|
||||
|
||||
Monitor connection pool metrics:
|
||||
- Active connections
|
||||
- Idle connections
|
||||
- Connection wait time
|
||||
- Connection errors
|
||||
|
||||
## Database Indexing Strategy
|
||||
|
||||
### Current Indexes
|
||||
|
||||
The application creates indexes on key fields:
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_telegrams_message_id ON aviation.telegrams (message_id);
|
||||
CREATE INDEX idx_telegrams_date_time ON aviation.telegrams (date_time);
|
||||
CREATE INDEX idx_telegrams_priority_indicator ON aviation.telegrams (priority_indicator);
|
||||
CREATE INDEX idx_telegrams_primary_address ON aviation.telegrams (primary_address);
|
||||
CREATE INDEX idx_telegrams_received_at ON aviation.telegrams (received_at);
|
||||
CREATE INDEX idx_telegrams_uuid ON aviation.telegrams (uuid);
|
||||
```
|
||||
|
||||
### Index Optimization
|
||||
|
||||
**Query Patterns:**
|
||||
- **Time-range queries**: Index on `received_at` (already exists)
|
||||
- **Message lookup**: Index on `message_id` (already exists)
|
||||
- **Category filtering**: Consider index on `category` if frequently queried
|
||||
- **Composite indexes**: For multi-column queries
|
||||
|
||||
**Example Composite Index:**
|
||||
```sql
|
||||
-- For queries filtering by category and date range
|
||||
CREATE INDEX idx_telegrams_category_received_at
|
||||
ON aviation.telegrams (category, received_at DESC);
|
||||
```
|
||||
|
||||
### Index Maintenance
|
||||
|
||||
- **Monitor index usage**: Use `pg_stat_user_indexes` to identify unused indexes
|
||||
- **Rebuild indexes**: Periodically rebuild indexes to reduce bloat
|
||||
- **Concurrent creation**: Use `CREATE INDEX CONCURRENTLY` in production
|
||||
|
||||
## NATS JetStream Performance Tuning
|
||||
|
||||
### Stream Configuration
|
||||
|
||||
```toml
|
||||
[nats.stream_limits]
|
||||
max_msgs = 1000000 # Maximum messages in stream
|
||||
max_bytes = 1073741824 # Maximum size (1GB)
|
||||
max_age = "168h" # Retention period (7 days)
|
||||
discard = "old" # Discard policy
|
||||
storage = "file" # Storage type (file or memory)
|
||||
replicas = 3 # Number of replicas
|
||||
```
|
||||
|
||||
### Tuning Guidelines
|
||||
|
||||
**Storage Type:**
|
||||
- **File storage**: Persistent, slower (recommended for production)
|
||||
- **Memory storage**: Faster, ephemeral (suitable for high-throughput temporary streams)
|
||||
|
||||
**Replicas:**
|
||||
- **Single node**: 1 replica (development)
|
||||
- **Production**: 3+ replicas (high availability)
|
||||
|
||||
**Retention:**
|
||||
- **Short retention**: Lower storage, faster cleanup
|
||||
- **Long retention**: More storage, replay capability
|
||||
|
||||
### Consumer Configuration
|
||||
|
||||
```toml
|
||||
[nats.consumer_rules]
|
||||
max_deliver = 5 # Maximum redelivery attempts
|
||||
ack_wait = "30s" # ACK wait time
|
||||
max_ack_pending = 1024 # Maximum unacknowledged messages
|
||||
deliver_policy = "new" # Delivery policy
|
||||
backoff = ["5s", "30s", "2m"] # Retry delays
|
||||
```
|
||||
|
||||
**Tuning:**
|
||||
- **ack_wait**: Set based on processing time (processing_time * 2-3)
|
||||
- **max_ack_pending**: Increase for high-throughput (1024-4096)
|
||||
- **backoff**: Adjust based on failure patterns
|
||||
|
||||
## Memory Optimization
|
||||
|
||||
### Garbage Collection Tuning
|
||||
|
||||
Set Go GC environment variables for production:
|
||||
|
||||
```bash
|
||||
# Balanced GC (default)
|
||||
export GOGC=100
|
||||
|
||||
# Aggressive GC (lower memory, higher CPU)
|
||||
export GOGC=50
|
||||
|
||||
# Conservative GC (higher memory, lower CPU)
|
||||
export GOGC=200
|
||||
```
|
||||
|
||||
### Memory Profiling
|
||||
|
||||
Use `pprof` to identify memory issues:
|
||||
|
||||
```bash
|
||||
# Enable memory profiling
|
||||
go tool pprof http://localhost:2112/debug/pprof/heap
|
||||
|
||||
# Generate memory profile
|
||||
go tool pprof -alloc_space http://localhost:2112/debug/pprof/heap
|
||||
```
|
||||
|
||||
## CPU Optimization
|
||||
|
||||
### Goroutine Management
|
||||
|
||||
- **Limit goroutines**: Use worker pools for concurrent processing
|
||||
- **Context cancellation**: Properly cancel goroutines to prevent leaks
|
||||
- **Monitor goroutine count**: Use `runtime.NumGoroutine()`
|
||||
|
||||
### CPU Profiling
|
||||
|
||||
```bash
|
||||
# Enable CPU profiling
|
||||
go tool pprof http://localhost:2112/debug/pprof/profile
|
||||
|
||||
# 30-second CPU profile
|
||||
go tool pprof http://localhost:2112/debug/pprof/profile?seconds=30
|
||||
```
|
||||
|
||||
## Monitoring and Profiling
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
Key metrics to monitor:
|
||||
|
||||
- `caatsm_messages_total`: Message throughput
|
||||
- `caatsm_handle_latency_seconds`: Processing latency
|
||||
- `caatsm_db_query_latency_seconds`: Database query time
|
||||
- `caatsm_nats_consumer_pending_messages`: Consumer lag
|
||||
|
||||
### Grafana Dashboards
|
||||
|
||||
Create dashboards for:
|
||||
- Message throughput over time
|
||||
- Latency percentiles (P50, P95, P99)
|
||||
- Error rates
|
||||
- Resource utilization (CPU, memory, connections)
|
||||
|
||||
### Profiling Endpoints
|
||||
|
||||
The application exposes profiling endpoints (if enabled):
|
||||
|
||||
```bash
|
||||
# Heap profile
|
||||
curl http://localhost:2112/debug/pprof/heap > heap.prof
|
||||
|
||||
# CPU profile
|
||||
curl http://localhost:2112/debug/pprof/profile?seconds=30 > cpu.prof
|
||||
|
||||
# Goroutine profile
|
||||
curl http://localhost:2112/debug/pprof/goroutine > goroutine.prof
|
||||
```
|
||||
|
||||
## Performance Testing
|
||||
|
||||
### Load Testing
|
||||
|
||||
Use tools like `k6`, `wrk`, or `vegeta` for load testing:
|
||||
|
||||
```bash
|
||||
# Example: Generate load with seed-telegrams
|
||||
go run ./cmd/seed-telegrams \
|
||||
--count=10000 \
|
||||
--mode=burst \
|
||||
--category=mixed
|
||||
```
|
||||
|
||||
### Benchmark Tests
|
||||
|
||||
Run built-in benchmarks:
|
||||
|
||||
```bash
|
||||
# Run all benchmarks
|
||||
go test -bench=. -benchmem ./...
|
||||
|
||||
# Run specific benchmark
|
||||
go test -bench=BenchmarkParser -benchmem ./internal/adapter/parser
|
||||
```
|
||||
|
||||
### Performance Baselines
|
||||
|
||||
Establish performance baselines:
|
||||
- **Throughput**: Messages per second
|
||||
- **Latency**: P50, P95, P99 percentiles
|
||||
- **Resource usage**: CPU, memory, connections
|
||||
|
||||
## Optimization Checklist
|
||||
|
||||
### Application Level
|
||||
|
||||
- [ ] Optimize batch size for workload
|
||||
- [ ] Tune connection pool sizes
|
||||
- [ ] Review and optimize database queries
|
||||
- [ ] Add missing indexes for query patterns
|
||||
- [ ] Enable query result caching where appropriate
|
||||
|
||||
### Infrastructure Level
|
||||
|
||||
- [ ] Use connection pooler (PgBouncer) for high concurrency
|
||||
- [ ] Configure database connection limits appropriately
|
||||
- [ ] Use read replicas for query-heavy workloads
|
||||
- [ ] Optimize NATS JetStream stream configuration
|
||||
- [ ] Scale horizontally (multiple instances)
|
||||
|
||||
### Monitoring
|
||||
|
||||
- [ ] Set up performance dashboards
|
||||
- [ ] Configure alerts for performance degradation
|
||||
- [ ] Regular performance profiling
|
||||
- [ ] Monitor resource utilization
|
||||
- [ ] Track performance trends over time
|
||||
|
||||
## Troubleshooting Performance Issues
|
||||
|
||||
### High Latency
|
||||
|
||||
**Symptoms:**
|
||||
- Slow message processing
|
||||
- High P95/P99 latencies
|
||||
|
||||
**Investigation:**
|
||||
1. Check database query times
|
||||
2. Review NATS consumer lag
|
||||
3. Profile CPU and memory usage
|
||||
4. Check for connection pool exhaustion
|
||||
|
||||
**Solutions:**
|
||||
- Optimize slow database queries
|
||||
- Increase batch size
|
||||
- Add database indexes
|
||||
- Scale horizontally
|
||||
|
||||
### Low Throughput
|
||||
|
||||
**Symptoms:**
|
||||
- Low messages per second
|
||||
- High CPU usage
|
||||
|
||||
**Investigation:**
|
||||
1. Check for bottlenecks (DB, NATS, CPU)
|
||||
2. Review batch processing configuration
|
||||
3. Profile application code
|
||||
|
||||
**Solutions:**
|
||||
- Increase batch size
|
||||
- Optimize hot code paths
|
||||
- Scale horizontally
|
||||
- Use connection pooling
|
||||
|
||||
### High Memory Usage
|
||||
|
||||
**Symptoms:**
|
||||
- Memory leaks
|
||||
- High memory consumption
|
||||
|
||||
**Investigation:**
|
||||
1. Heap profiling
|
||||
2. Check for goroutine leaks
|
||||
3. Review batch sizes
|
||||
|
||||
**Solutions:**
|
||||
- Fix memory leaks
|
||||
- Reduce batch sizes
|
||||
- Tune GC settings
|
||||
- Limit concurrent operations
|
||||
|
||||
## References
|
||||
|
||||
- [Go Performance Best Practices](https://go.dev/doc/effective_go#performance)
|
||||
- [PostgreSQL Performance Tuning](https://www.postgresql.org/docs/current/performance-tips.html)
|
||||
- [TimescaleDB Performance Tuning](https://docs.timescale.com/timescaledb/latest/how-to-guides/performance/)
|
||||
- [NATS JetStream Performance](https://docs.nats.io/nats-concepts/jetstream/performance)
|
||||
- [Go Profiling Guide](https://go.dev/blog/pprof)
|
||||
|
||||
@@ -421,6 +421,146 @@ backoff = ["5s", "30s", "2m", "5m"] # Retry delays
|
||||
- Don't log message payloads in production
|
||||
- Use appropriate log levels
|
||||
|
||||
## NATS Authentication
|
||||
|
||||
The application supports multiple NATS authentication methods for secure connections. Configure authentication in the `[nats.auth]` section of your production configuration.
|
||||
|
||||
### Authentication Methods
|
||||
|
||||
Only one authentication method can be used at a time. Choose the method that best fits your infrastructure:
|
||||
|
||||
#### 1. Token Authentication
|
||||
|
||||
Simple token-based authentication suitable for service-to-service communication:
|
||||
|
||||
```toml
|
||||
[nats.auth]
|
||||
token = "your-nats-token-here"
|
||||
tls_enabled = true
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Simple service-to-service authentication
|
||||
- Single token shared across services
|
||||
- Quick setup for development/staging
|
||||
|
||||
**Security considerations:**
|
||||
- Tokens should be rotated regularly
|
||||
- Store tokens securely (use secret management)
|
||||
- Use TLS to encrypt token transmission
|
||||
|
||||
#### 2. Credentials File (Recommended)
|
||||
|
||||
NATS credentials file authentication provides fine-grained access control:
|
||||
|
||||
```toml
|
||||
[nats.auth]
|
||||
credentials_file = "/etc/caatsm/nats.creds"
|
||||
tls_enabled = true
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Production environments requiring fine-grained permissions
|
||||
- Multiple services with different access levels
|
||||
- Integration with NATS account system
|
||||
|
||||
**Setup:**
|
||||
1. Generate credentials file using NATS CLI:
|
||||
```bash
|
||||
nats account creds -n caatsm-service > /etc/caatsm/nats.creds
|
||||
```
|
||||
2. Ensure the file is readable by the application user
|
||||
3. Set appropriate file permissions (e.g., `chmod 600 /etc/caatsm/nats.creds`)
|
||||
|
||||
#### 3. User/Password Authentication
|
||||
|
||||
Traditional username/password authentication:
|
||||
|
||||
```toml
|
||||
[nats.auth]
|
||||
user = "caatsm-service"
|
||||
password = "secure-password-here"
|
||||
tls_enabled = true
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Legacy NATS server configurations
|
||||
- Simple authentication requirements
|
||||
- Integration with existing user management systems
|
||||
|
||||
**Security considerations:**
|
||||
- Use strong, unique passwords
|
||||
- Store passwords securely (use secret management)
|
||||
- Rotate passwords regularly
|
||||
|
||||
### TLS Configuration
|
||||
|
||||
TLS encryption is **required** for production deployments. Configure TLS in the `[nats.auth]` section:
|
||||
|
||||
```toml
|
||||
[nats.auth]
|
||||
credentials_file = "/etc/caatsm/nats.creds"
|
||||
tls_enabled = true
|
||||
tls_cert_file = "/etc/caatsm/tls/client.crt" # Optional: client certificate
|
||||
tls_key_file = "/etc/caatsm/tls/client.key" # Optional: client private key
|
||||
tls_ca_file = "/etc/caatsm/tls/ca.crt" # Optional: CA certificate for server verification
|
||||
```
|
||||
|
||||
**TLS Options:**
|
||||
- `tls_enabled`: Enable TLS encryption (required for production)
|
||||
- `tls_cert_file`: Client certificate file path (for mutual TLS)
|
||||
- `tls_key_file`: Client private key file path (for mutual TLS)
|
||||
- `tls_ca_file`: CA certificate file for server certificate verification
|
||||
|
||||
**Note:** If `tls_ca_file` is not specified, the system's default CA certificates are used. For production, it's recommended to specify a CA file for explicit certificate validation. The application will load and use the CA certificate file for server verification when provided.
|
||||
|
||||
### Environment Variable Configuration
|
||||
|
||||
You can also configure authentication via environment variables:
|
||||
|
||||
```bash
|
||||
# Token authentication
|
||||
export CAATSM_NATS_AUTH_TOKEN="your-token"
|
||||
|
||||
# Credentials file
|
||||
export CAATSM_NATS_AUTH_CREDENTIALS_FILE="/etc/caatsm/nats.creds"
|
||||
|
||||
# User/Password
|
||||
export CAATSM_NATS_AUTH_USER="caatsm-service"
|
||||
export CAATSM_NATS_AUTH_PASSWORD="secure-password"
|
||||
|
||||
# TLS
|
||||
export CAATSM_NATS_AUTH_TLS_ENABLED="true"
|
||||
export CAATSM_NATS_AUTH_TLS_CERT_FILE="/etc/caatsm/tls/client.crt"
|
||||
export CAATSM_NATS_AUTH_TLS_KEY_FILE="/etc/caatsm/tls/client.key"
|
||||
export CAATSM_NATS_AUTH_TLS_CA_FILE="/etc/caatsm/tls/ca.crt"
|
||||
```
|
||||
|
||||
### Testing Authentication
|
||||
|
||||
After configuring authentication, verify the connection:
|
||||
|
||||
```bash
|
||||
# Test connection with authentication
|
||||
./bin/receiver listen --nats-url nats://nats.prod:4222
|
||||
|
||||
# Check logs for authentication success
|
||||
# Look for: "NATS reconnected" or connection errors
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
**Connection failures:**
|
||||
- Verify authentication credentials are correct
|
||||
- Check NATS server logs for authentication errors
|
||||
- Ensure TLS certificates are valid and accessible
|
||||
- Verify file permissions on credentials/certificate files
|
||||
|
||||
**Common errors:**
|
||||
- `authentication failed`: Check token/credentials/user-password
|
||||
- `tls: bad certificate`: Verify TLS certificate configuration
|
||||
- `permission denied`: Check file permissions on credentials/certificate files
|
||||
|
||||
## Backup and Recovery
|
||||
|
||||
### Database Backups
|
||||
|
||||
@@ -0,0 +1,427 @@
|
||||
# Secret Management Guide
|
||||
|
||||
This document describes best practices for managing secrets and sensitive configuration in the CAATSM application.
|
||||
|
||||
## Current Approach
|
||||
|
||||
The application currently supports secrets via environment variables with the `CAATSM_` prefix:
|
||||
|
||||
```bash
|
||||
export CAATSM_POSTGRES_URL="postgres://user:password@localhost:5432/aviation"
|
||||
export CAATSM_NATS_AUTH_TOKEN="your-token"
|
||||
export CAATSM_NATS_AUTH_PASSWORD="secure-password"
|
||||
```
|
||||
|
||||
**Security considerations:**
|
||||
- Environment variables are visible to all processes on the system
|
||||
- Secrets may be logged in process lists or shell history
|
||||
- No automatic rotation or expiration
|
||||
- Manual management required
|
||||
|
||||
## Recommended Secret Management Systems
|
||||
|
||||
For production deployments, use a dedicated secret management system:
|
||||
|
||||
### Option 1: HashiCorp Vault (Recommended)
|
||||
|
||||
[Vault](https://www.vaultproject.io/) provides secure secret storage with dynamic secrets, encryption, and access control.
|
||||
|
||||
#### Setup
|
||||
|
||||
1. **Install Vault:**
|
||||
```bash
|
||||
# Download and install Vault
|
||||
wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_linux_amd64.zip
|
||||
unzip vault_1.15.0_linux_amd64.zip
|
||||
sudo mv vault /usr/local/bin/
|
||||
```
|
||||
|
||||
2. **Start Vault (dev mode for testing):**
|
||||
```bash
|
||||
vault server -dev
|
||||
```
|
||||
|
||||
3. **Store secrets:**
|
||||
```bash
|
||||
export VAULT_ADDR='http://127.0.0.1:8200'
|
||||
vault kv put secret/caatsm \
|
||||
postgres_url="postgres://user:pass@db:5432/aviation" \
|
||||
nats_token="your-token" \
|
||||
nats_password="secure-password"
|
||||
```
|
||||
|
||||
#### Integration
|
||||
|
||||
Create a wrapper script or init container to fetch secrets from Vault:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# fetch-secrets.sh
|
||||
|
||||
export VAULT_ADDR="${VAULT_ADDR:-http://vault:8200}"
|
||||
export VAULT_TOKEN="${VAULT_TOKEN}"
|
||||
|
||||
# Fetch secrets from Vault
|
||||
vault kv get -format=json secret/caatsm | jq -r '.data.data | to_entries | .[] | "export CAATSM_\(.key | ascii_upcase | gsub("-"; "_"))=\(.value)"' > /tmp/secrets.env
|
||||
|
||||
# Source secrets
|
||||
source /tmp/secrets.env
|
||||
|
||||
# Start application
|
||||
exec ./bin/receiver listen
|
||||
```
|
||||
|
||||
#### Kubernetes Integration
|
||||
|
||||
Use Vault Agent Sidecar or Vault Secrets Operator:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: caatsm-receiver
|
||||
spec:
|
||||
containers:
|
||||
- name: vault-agent
|
||||
image: vault:latest
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
vault agent -config=/vault/config/agent.hcl
|
||||
- name: caatsm-receiver
|
||||
image: caatsm/receiver:latest
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: caatsm-secrets
|
||||
```
|
||||
|
||||
### Option 2: AWS Secrets Manager
|
||||
|
||||
For AWS deployments, use [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) for centralized secret management.
|
||||
|
||||
#### Setup
|
||||
|
||||
1. **Store secrets:**
|
||||
```bash
|
||||
aws secretsmanager create-secret \
|
||||
--name caatsm/production \
|
||||
--secret-string '{
|
||||
"postgres_url": "postgres://user:pass@db:5432/aviation",
|
||||
"nats_token": "your-token",
|
||||
"nats_password": "secure-password"
|
||||
}'
|
||||
```
|
||||
|
||||
2. **Retrieve secrets:**
|
||||
```bash
|
||||
aws secretsmanager get-secret-value \
|
||||
--secret-id caatsm/production \
|
||||
--query SecretString \
|
||||
--output text | jq -r 'to_entries | .[] | "export CAATSM_\(.key | ascii_upcase | gsub("-"; "_"))=\(.value)"'
|
||||
```
|
||||
|
||||
#### Integration
|
||||
|
||||
Use AWS SDK or CLI in init container:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# fetch-aws-secrets.sh
|
||||
|
||||
SECRET_JSON=$(aws secretsmanager get-secret-value \
|
||||
--secret-id caatsm/production \
|
||||
--query SecretString \
|
||||
--output text)
|
||||
|
||||
echo "$SECRET_JSON" | jq -r 'to_entries | .[] | "export CAATSM_\(.key | ascii_upcase | gsub("-"; "_"))=\(.value)"' > /tmp/secrets.env
|
||||
|
||||
source /tmp/secrets.env
|
||||
exec ./bin/receiver listen
|
||||
```
|
||||
|
||||
### Option 3: Kubernetes Secrets
|
||||
|
||||
For Kubernetes deployments, use [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/).
|
||||
|
||||
#### Setup
|
||||
|
||||
1. **Create secret:**
|
||||
```bash
|
||||
kubectl create secret generic caatsm-secrets \
|
||||
--from-literal=postgres-url="postgres://user:pass@db:5432/aviation" \
|
||||
--from-literal=nats-token="your-token" \
|
||||
--from-literal=nats-password="secure-password"
|
||||
```
|
||||
|
||||
2. **Use in deployment:**
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: caatsm-receiver
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: receiver
|
||||
image: caatsm/receiver:latest
|
||||
env:
|
||||
- name: CAATSM_POSTGRES_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: caatsm-secrets
|
||||
key: postgres-url
|
||||
- name: CAATSM_NATS_AUTH_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: caatsm-secrets
|
||||
key: nats-token
|
||||
```
|
||||
|
||||
#### Best Practices
|
||||
|
||||
- **Encrypt at rest**: Enable encryption for etcd (Kubernetes backend)
|
||||
- **RBAC**: Restrict access to secrets using Role-Based Access Control
|
||||
- **External Secrets Operator**: Use [External Secrets Operator](https://external-secrets.io/) for integration with external secret stores
|
||||
|
||||
### Option 4: Docker Secrets
|
||||
|
||||
For Docker Swarm deployments, use [Docker Secrets](https://docs.docker.com/engine/swarm/secrets/).
|
||||
|
||||
#### Setup
|
||||
|
||||
1. **Create secret:**
|
||||
```bash
|
||||
echo "your-secret-value" | docker secret create caatsm_nats_token -
|
||||
```
|
||||
|
||||
2. **Use in service:**
|
||||
```yaml
|
||||
version: '3.8'
|
||||
services:
|
||||
receiver:
|
||||
image: caatsm/receiver:latest
|
||||
secrets:
|
||||
- caatsm_nats_token
|
||||
environment:
|
||||
- CAATSM_NATS_AUTH_TOKEN_FILE=/run/secrets/caatsm_nats_token
|
||||
```
|
||||
|
||||
## Secret Rotation
|
||||
|
||||
### Manual Rotation
|
||||
|
||||
1. **Update secret** in secret management system
|
||||
2. **Restart application** to pick up new secret
|
||||
3. **Verify** application is working correctly
|
||||
4. **Remove old secret** after verification
|
||||
|
||||
### Automated Rotation
|
||||
|
||||
For AWS Secrets Manager, enable automatic rotation:
|
||||
|
||||
```bash
|
||||
aws secretsmanager rotate-secret \
|
||||
--secret-id caatsm/production \
|
||||
--rotation-lambda-arn arn:aws:lambda:region:account:function:rotate-secret
|
||||
```
|
||||
|
||||
For Vault, use dynamic secrets or scheduled rotation policies.
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Principle of Least Privilege
|
||||
|
||||
- **Minimal access**: Grant only necessary permissions
|
||||
- **Service accounts**: Use dedicated service accounts for applications
|
||||
- **Secret scoping**: Limit secrets to specific services/environments
|
||||
|
||||
### 2. Encryption
|
||||
|
||||
- **Encryption at rest**: Ensure secrets are encrypted in storage
|
||||
- **Encryption in transit**: Use TLS for secret retrieval
|
||||
- **Key management**: Use proper key management (HSM, KMS, etc.)
|
||||
|
||||
### 3. Audit and Monitoring
|
||||
|
||||
- **Audit logs**: Enable audit logging for secret access
|
||||
- **Monitoring**: Monitor secret access patterns
|
||||
- **Alerts**: Set up alerts for unusual access patterns
|
||||
|
||||
### 4. Secret Lifecycle
|
||||
|
||||
- **Rotation**: Rotate secrets regularly (e.g., every 90 days)
|
||||
- **Expiration**: Set expiration dates for secrets
|
||||
- **Revocation**: Have a process for revoking compromised secrets
|
||||
|
||||
### 5. Development vs Production
|
||||
|
||||
- **Separate stores**: Use different secret stores for dev/staging/prod
|
||||
- **No production secrets in code**: Never commit production secrets
|
||||
- **Local development**: Use local secret files or dev vault instance
|
||||
|
||||
## Configuration Examples
|
||||
|
||||
### Environment Variables (Current)
|
||||
|
||||
```bash
|
||||
# Development
|
||||
export CAATSM_POSTGRES_URL="postgres://user:pass@localhost:5432/aviation?sslmode=disable"
|
||||
export CAATSM_NATS_URL="nats://localhost:4222"
|
||||
export CAATSM_NATS_AUTH_TOKEN="dev-token"
|
||||
|
||||
# Production (via secret management)
|
||||
# Secrets loaded from Vault/AWS/K8s before application start
|
||||
```
|
||||
|
||||
### Configuration File (Not Recommended for Secrets)
|
||||
|
||||
```toml
|
||||
# config.prod.toml
|
||||
# DO NOT store secrets in config files
|
||||
# Use environment variables or secret management instead
|
||||
|
||||
[postgres]
|
||||
# URL should come from CAATSM_POSTGRES_URL env var
|
||||
url = "" # Empty, will be overridden by env var
|
||||
|
||||
[nats.auth]
|
||||
# Token should come from CAATSM_NATS_AUTH_TOKEN env var
|
||||
token = "" # Empty, will be overridden by env var
|
||||
```
|
||||
|
||||
## Secret Injection Patterns
|
||||
|
||||
### Pattern 1: Init Container (Kubernetes)
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: caatsm-receiver
|
||||
spec:
|
||||
initContainers:
|
||||
- name: fetch-secrets
|
||||
image: vault:latest
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
vault kv get -format=json secret/caatsm | \
|
||||
jq -r '.data.data | to_entries | .[] | "\(.key | ascii_upcase | gsub("-"; "_"))=\(.value)"' > \
|
||||
/shared/secrets.env
|
||||
volumeMounts:
|
||||
- name: shared-secrets
|
||||
mountPath: /shared
|
||||
containers:
|
||||
- name: receiver
|
||||
image: caatsm/receiver:latest
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: caatsm-config
|
||||
env:
|
||||
- name: CAATSM_SECRETS_FILE
|
||||
value: /shared/secrets.env
|
||||
volumeMounts:
|
||||
- name: shared-secrets
|
||||
mountPath: /shared
|
||||
volumes:
|
||||
- name: shared-secrets
|
||||
emptyDir: {}
|
||||
```
|
||||
|
||||
### Pattern 2: Sidecar Container
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: caatsm-receiver
|
||||
spec:
|
||||
containers:
|
||||
- name: vault-agent
|
||||
image: vault:latest
|
||||
command: ["vault", "agent", "-config=/vault/config/agent.hcl"]
|
||||
volumeMounts:
|
||||
- name: vault-config
|
||||
mountPath: /vault/config
|
||||
- name: receiver
|
||||
image: caatsm/receiver:latest
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: caatsm-secrets
|
||||
volumes:
|
||||
- name: vault-config
|
||||
configMap:
|
||||
name: vault-agent-config
|
||||
```
|
||||
|
||||
### Pattern 3: Application-Level Integration
|
||||
|
||||
For applications that need to fetch secrets at runtime:
|
||||
|
||||
```go
|
||||
// Example: Fetch secrets from Vault at startup
|
||||
func loadSecretsFromVault() error {
|
||||
client, err := vault.NewClient(vault.DefaultConfig())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
secret, err := client.Logical().Read("secret/data/caatsm")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Set environment variables
|
||||
for k, v := range secret.Data["data"].(map[string]interface{}) {
|
||||
os.Setenv("CAATSM_"+strings.ToUpper(k), v.(string))
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Secret Not Found
|
||||
|
||||
**Symptoms:**
|
||||
- Application fails to start
|
||||
- Connection errors to database/NATS
|
||||
|
||||
**Solutions:**
|
||||
- Verify secret exists in secret store
|
||||
- Check secret name/path is correct
|
||||
- Verify application has permissions to access secret
|
||||
- Check secret format (JSON, plain text, etc.)
|
||||
|
||||
### Secret Access Denied
|
||||
|
||||
**Symptoms:**
|
||||
- Authentication errors when fetching secrets
|
||||
- Permission denied errors
|
||||
|
||||
**Solutions:**
|
||||
- Verify IAM roles/service accounts have correct permissions
|
||||
- Check Vault policies or AWS IAM policies
|
||||
- Verify authentication tokens/credentials are valid
|
||||
|
||||
### Secret Rotation Issues
|
||||
|
||||
**Symptoms:**
|
||||
- Application fails after secret rotation
|
||||
- Connection errors after rotation
|
||||
|
||||
**Solutions:**
|
||||
- Implement graceful secret reloading
|
||||
- Use connection pooling with automatic reconnection
|
||||
- Test rotation process in staging first
|
||||
|
||||
## References
|
||||
|
||||
- [HashiCorp Vault Documentation](https://www.vaultproject.io/docs)
|
||||
- [AWS Secrets Manager Documentation](https://docs.aws.amazon.com/secretsmanager/)
|
||||
- [Kubernetes Secrets Documentation](https://kubernetes.io/docs/concepts/configuration/secret/)
|
||||
- [External Secrets Operator](https://external-secrets.io/)
|
||||
- [12-Factor App: Config](https://12factor.net/config)
|
||||
|
||||
Reference in New Issue
Block a user