# NATS Integration ## Overview The NATS integration provides a streamlined, production-ready message processing system focused on essential functionality. It supports JetStream persistent messaging with basic error handling, TLS security, and observability. ### Key Concepts 1. **Consumer**: Pulls messages from NATS JetStream in batches and processes them 2. **Publisher**: Publishes messages to NATS with deduplication 3. **Batch Processing**: Fetches multiple messages for efficiency 4. **Error Classification**: Distinguishes transient vs permanent errors 5. **Dead Letter Queue (DLQ)**: Routes permanent errors to DLQ 6. **TLS Support**: Secure connections with client certificates 7. **Basic Monitoring**: Essential metrics and logging ### Quick Start Flow ``` 1. Configure NATS connection and consumer settings 2. Create Consumer with dependencies (processor, logger, telemetry) 3. Start Consumer - begins fetching and processing messages 4. Messages flow: Fetch → Process → ACK/NAK/DLQ 5. Errors handled automatically with retries and backoff 6. Graceful shutdown on context cancellation ``` ## Architecture ### Architecture The NATS integration follows simplified Clean Architecture with focused components: ``` ┌─────────────────────────────────────┐ │ Application Layer │ │ (Business logic & processing) │ ├─────────────────────────────────────┤ │ Infrastructure Layer │ │ (NATS implementation) │ │ │ │ ┌─────────────────────────────┐ │ │ │ Consumer │ │ │ │ ┌─────────────────────┐ │ │ │ │ │ MessageFetcher │ │ │ │ │ │ MessageProcessor │ │ │ │ │ │ DLQHandler │ │ │ │ │ └─────────────────────┘ │ │ │ └─────────────────────────────┘ │ │ │ │ ┌─────────────────────────────┐ │ │ │ Publisher │ │ │ └─────────────────────────────┘ │ └─────────────────────────────────────┘ ``` ### Component Interaction ``` ┌──────────────┐ │ Publisher │ │ │ │ 1. Serialize │ │ 2. Publish │ └──────┬───────┘ │ │ Publish to Subject ▼ ┌─────────────────────────────────────┐ │ NATS JetStream │ │ │ │ ┌──────────────┐ │ │ │ Stream │ │ │ │ (TELEGRAM) │ │ │ └──────┬───────┘ │ │ │ │ │ ┌──────▼───────┐ │ │ │ Consumer │ │ │ │ (Pull Sub) │ │ │ └──────┬───────┘ │ └─────────┼───────────────────────────┘ │ │ Fetch Batch ▼ ┌─────────────────────────────────────┐ │ Consumer │ │ │ │ ┌──────────────────────────────┐ │ │ │ MessageFetcher │ │ │ │ - FetchBatch() │ │ │ │ - HandleFetchError() │ │ │ └──────────┬───────────────────┘ │ │ │ │ │ ┌──────────▼───────────────────┐ │ │ │ MessageProcessor │ │ │ │ - ProcessBatch() │ │ │ │ - ProcessMessage() │ │ │ └──────────┬───────────────────┘ │ │ │ │ │ ┌──────────▼───────────────────┐ │ │ │ DLQHandler │ │ │ │ - RouteToDLQ() │ │ │ └──────────────────────────────┘ │ └─────────────────────────────────────┘ │ │ ACK/NAK ▼ ┌─────────────────────────────────────┐ │ Application Processor │ │ (Business Logic) │ └─────────────────────────────────────┘ ``` ### Architecture Principles 1. **Dependency Inversion**: High-level modules (Consumer, Publisher) depend on abstractions (interfaces), not concrete implementations 2. **Separation of Concerns**: Each component has a single responsibility: - `MessageFetcher`: Handles message retrieval - `MessageProcessor`: Handles message processing logic - `DLQHandler`: Handles dead letter queue routing 3. **Testability**: All components can be mocked and tested independently 4. **Extensibility**: New implementations can be added without modifying existing code ## Logic Flow ### Consumer Processing Flow The consumer follows a simplified processing loop: ``` ┌─────────────────────────────────────────────────────────────┐ │ Consumer Start │ │ 1. Initialize components (Fetcher, Processor, DLQ) │ │ 2. Start main processing loop │ └──────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Main Loop │ │ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Step 1: Check Context │ │ │ │ - If cancelled, exit gracefully │ │ │ └──────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌──────────────────▼───────────────────────────────────┐ │ │ │ Step 2: Fetch Batch │ │ │ │ - Fetch up to BatchSize messages │ │ │ │ - Wait up to BatchTimeout │ │ │ │ - Handle fetch errors with backoff │ │ │ └──────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌───────────┴───────────┐ │ │ │ │ │ │ Success Error │ │ │ │ │ │ │ ┌────────▼────────┐ │ │ │ │ Apply Backoff │ │ │ │ │ Continue Loop │ │ │ │ └─────────────────┘ │ │ │ │ │ └──────────────────┬──────────────────────────────┘ │ │ │ ┌─────────────────────────▼─────────────────────────────┐ │ │ │ Step 3: Process Batch │ │ │ │ - For each message in batch: │ │ │ │ * Extract message ID │ │ │ │ * Call processor.Handle() │ │ │ │ * Handle result (ACK/NAK/DLQ) │ │ │ └──────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌───────────┴───────────┐ │ │ │ │ │ │ Success Error │ │ │ │ │ │ │ ┌────────▼────────┐ │ │ │ │ Classify Error │ │ │ │ └────────┬────────┘ │ │ │ │ │ │ │ ┌─────────────┴─────────────┐ │ │ │ │ │ │ │ │ Permanent Transient │ │ │ │ │ │ │ │ ┌──────▼──────┐ ┌────────▼──────┐ │ │ │ │ Route to DLQ│ │ NAK with delay│ │ │ │ │ ACK message │ └────────────────┘ │ │ │ └──────┬──────┘ │ │ │ │ │ │ └─────────┴───────────────────────────────────────┘ │ │ │ └─────────── Loop ─────────────────────┘ └─────────────────────────────────────────────────────────────┘ ``` ┌─────────────────────────────────────────────────────────────┐ │ Consumer Start │ │ 1. Initialize components (Fetcher, Processor, DLQ) │ │ 2. Create/validate JetStream resources │ │ 3. Start advisory DLQ handler (if enabled) │ │ 4. Start metrics collection goroutine │ └──────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Main Loop │ │ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │ Step 1: Check Context │ │ │ │ - If cancelled, exit gracefully │ │ │ └──────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌──────────────────▼───────────────────────────────────┐ │ │ │ Step 2: Fetch Batch │ │ │ │ - Fetch up to BatchSize messages │ │ │ │ - Wait up to BatchTimeout │ │ │ │ - Handle fetch errors with recovery │ │ │ └──────────────────┬───────────────────────────────────┘ │ │ │ │ │ ┌───────────┴───────────┐ │ │ │ │ │ │ Success Error │ │ │ │ │ │ │ ┌────────▼────────┐ │ │ │ │ Handle Error │ │ │ │ │ - Classify type │ │ │ │ │ - Apply backoff │ │ │ │ │ - Recover if dev│ │ │ │ └────────┬────────┘ │ │ │ │ │ │ │ ┌────────▼────────┐ │ │ │ │ Continue? │ │ │ │ └────────┬────────┘ │ │ │ │ │ │ │ Yes │ No │ │ │ │ │ │ │ │ └───┬────┴───┐ │ │ │ │ │ │ │ │ Continue Exit │ │ │ │ │ │ └──────────────────┘ │ │ │ │ │ ┌──────────────────▼───────────────────────────────────┐ │ │ │ Step 3: Process Batch │ │ │ │ - For each message in batch: │ │ │ │ * Check context │ │ │ │ * Extract message ID │ │ │ │ * Create tracing span │ │ │ │ * Call processor.Handle() │ │ │ │ * Handle result (ACK/NAK/DLQ) │ │ │ └──────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────┴───────────┐ │ │ │ │ │ │ Success Error │ │ │ │ │ │ │ ┌────────▼────────┐ │ │ │ │ Classify Error │ │ │ │ └────────┬────────┘ │ │ │ │ │ │ │ ┌─────────────┴─────────────┐ │ │ │ │ │ │ │ │ Permanent Transient │ │ │ │ │ │ │ │ ┌──────▼──────┐ ┌────────▼──────┐ │ │ │ │ Route to DLQ│ │ NAK with delay│ │ │ │ │ ACK message │ │ Apply backpres│ │ │ │ └──────┬──────┘ └────────┬──────┘ │ │ │ │ │ │ │ └─────────┴───────────────────────────┘ │ │ │ │ │ ┌──────────────────▼───────────────────────────────────┐ │ │ │ Step 4: Reset Error Streak (if successful) │ │ │ └──────────────────┬───────────────────────────────────┘ │ │ │ │ │ └─────────── Loop ─────────────────────┘ └─────────────────────────────────────────────────────────────┘ ``` ### Message Processing Logic #### Single Message Processing Flow ``` Message Received │ ▼ ┌─────────────────────┐ │ Extract Message ID │ │ - Check header │ │ - Fallback to meta │ │ - Generate if none │ └──────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ Create Trace Span │ │ - Add attributes │ │ - Propagate context │ └──────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ Process Message │ │ - Call processor │ │ - Business logic │ └──────────┬──────────┘ │ ┌──────┴──────┐ │ │ Success Error │ │ │ ┌──────▼──────────┐ │ │ Classify Error │ │ └──────┬──────────┘ │ │ │ ┌──────┴──────┐ │ │ │ │ Permanent Transient │ │ │ │ ┌───▼───┐ ┌────▼────┐ │ │ DLQ │ │ NAK │ │ │ ACK │ │ Backoff │ │ └───┬───┘ └────┬────┘ │ │ │ └──────┴─────────────┘ │ ▼ End Processing ``` #### Error Handling Logic ``` Error Occurred │ ▼ ┌─────────────────────┐ │ Is Permanent Error? │ │ - app.IsPermanent() │ └──────┬──────────────┘ │ ┌───┴───┐ │ │ Yes No │ │ │ ┌───▼──────────────────────┐ │ │ Increment Error Streak │ │ └───┬──────────────────────┘ │ │ │ ┌───▼──────────────────────┐ │ │ Streak >= Threshold? │ │ │ (default: 10 errors) │ │ └───┬──────────────────────┘ │ │ │ ┌───┴───┐ │ │ │ │ Yes No │ │ │ │ │ ┌───▼──────────────┐ │ │ │ NAK with delay │ │ │ │ - Use backoff │ │ │ │ - Request retry │ │ │ └──────────────────┘ │ │ │ ▼ │ ┌──────────────────────┐ │ │ Apply Backpressure │ │ │ - Sleep: errors*100ms│ │ │ - Max: 5 seconds │ │ └───┬──────────────────┘ │ │ │ ▼ │ ┌──────────────────────┐ │ │ NAK with delay │ │ └──────────────────────┘ │ ▼ ┌──────────────────────┐ │ Route to DLQ │ │ - Enrich metadata │ │ - Publish to DLQ │ │ - ACK original msg │ └──────────────────────┘ ``` ### Publisher Logic Flow ``` Publish Request │ ▼ ┌─────────────────────┐ │ Validate Topic │ │ - Check config │ └──────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ Serialize Message │ │ - JSON marshal │ └──────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ Extract/Generate ID │ │ - From message.Uuid │ │ - Or generate UUID │ └──────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ Set Header │ │ - Nats-Msg-Id │ └──────────┬──────────┘ │ ▼ ┌─────────────────────┐ │ Publish to NATS │ │ - js.PublishMsg() │ └──────────┬──────────┘ │ ┌──────┴──────┐ │ │ Success Error │ │ │ ┌──────▼──────────┐ │ │ Classify Error │ │ └──────┬──────────┘ │ │ │ ┌──────┴──────┐ │ │ │ │ Transient Permanent │ │ │ │ ┌───▼───┐ ┌────▼────┐ │ │ Retry │ │ Fail │ │ │ Later │ │ Fast │ │ └───────┘ └─────────┘ │ ▼ Success ``` ## Core Components ### Consumer The consumer handles message consumption with the following features: #### Modes - **JetStream Mode**: Persistent, durable message processing with acknowledgments - **Core Mode**: Fire-and-forget message processing for simple use cases #### Key Features - **Batch Processing**: Configurable batch sizes for efficient processing - **Error Handling**: Distinguishes transient vs permanent errors - **Dead Letter Queue (DLQ)**: Routes permanent errors to DLQ - **TLS Support**: Secure connections with client certificates - **Basic Monitoring**: Essential metrics collection #### Component Logic **MessageFetcher (`defaultMessageFetcher`)** - Fetches batches of messages using `sub.Fetch(batchSize, MaxWait(timeout))` - Handles fetch errors with simple exponential backoff - Context-aware: respects cancellation signals **MessageProcessor (`defaultBatchProcessor`)** - Processes messages sequentially within a batch - Extracts message IDs (header → metadata → generated) - Calls application processor for business logic - Handles ACK/NAK based on processing results **DLQHandler (`defaultDLQHandler`)** - Routes permanent errors to DLQ with basic metadata - Validates DLQ stream exists at startup - Publishes DLQ messages with error context #### Configuration ```toml [NATS] URL = "nats://localhost:4222" Stream = "TELEGRAM" Consumer = "telegram-consumer" [NATS.Auth] Token = "your-token" # Optional token authentication TLSEnabled = true # Enable TLS TLSCertFile = "/path/to/client.crt" # Client certificate TLSKeyFile = "/path/to/client.key" # Client private key TLSCAFile = "/path/to/ca.crt" # CA certificate [NATS.ConsumerRules] AckWait = "30s" MaxDeliver = 3 MaxAckPending = 1000 [DLQ] Enabled = true Subject = "caatsm.dlq" [App] BatchSize = 50 BatchTimeout = "2s" ``` ### Publisher The publisher handles message publishing with deduplication and observability. #### Features - **Message Deduplication**: Automatic UUID-based deduplication headers - **JetStream/Core Mode Support**: Adapts based on available JetStream context - **Structured Logging**: Comprehensive logging of publish operations - **Error Classification**: Distinguishes between transient and permanent errors #### Component Logic **Publishing Flow** 1. **Validation**: Checks that publisher topic is configured 2. **Serialization**: Marshals message to JSON using `json.Marshal()` 3. **ID Extraction**: Extracts UUID from message (if `ParsedTelegram` type) or generates new UUID 4. **Header Attachment**: Sets `Nats-Msg-Id` header for deduplication 5. **Publishing**: Calls `js.PublishMsg()` to publish to JetStream 6. **Error Handling**: Classifies errors as transient (`ErrNoResponders`) or permanent **Deduplication Strategy** - Uses `Nats-Msg-Id` header for JetStream deduplication - Extracts UUID from `ParsedTelegram.Uuid` field if available - Falls back to generating new UUID if not present - JetStream uses this header to prevent duplicate message processing **Error Classification** - **Transient**: `nats.ErrNoResponders` - JetStream temporarily unavailable, should retry - **Permanent**: Other errors - configuration issues, should fail fast ### Error Handling #### Error Types - **Transient Errors**: Network issues, temporary unavailability (retried with backoff) - **Permanent Errors**: Message format issues, business logic failures (routed to DLQ) #### Recovery Strategies - **Simple Backoff**: Exponential backoff for transient failures - **Graceful Degradation**: Continues processing other messages when one fails ### Dead Letter Queue (DLQ) #### Features - **Rich Metadata**: Includes original message, error details, delivery attempts - **Stream Validation**: Validates DLQ stream exists at startup - **Advisory Processing**: Handles MaxDeliver exhaustion automatically - **Operational Visibility**: Comprehensive logging and metrics #### DLQ Message Format ```json { "transport_msg_id": "uuid", "subject": "original.subject", "stream": "TELEGRAM", "consumer": "telegram-consumer", "nats_sequence": 12345, "deliveries": 3, "error": "processing failed: invalid format", "received_at": "2024-01-01T12:00:00Z", "body": "original message data" } ``` ## Observability ### Metrics - **Consumer Metrics**: ack_pending, redelivered, pending, delivered counts - **Processing Metrics**: batch size, processing duration, error rates - **DLQ Metrics**: messages routed to DLQ, publish failures - **Connection Metrics**: connection health, reconnection events ### Tracing - **End-to-End Tracing**: Request correlation through trace IDs - **Span Attributes**: Consumer name, stream name, batch size, error details - **Context Propagation**: Trace context passed through processing pipeline ### Logging - **Structured Logs**: JSON format with correlation IDs - **Log Levels**: Debug, Info, Warn, Error with appropriate detail levels - **Operational Context**: Includes consumer, stream, and message metadata ## Resilience Patterns ### Backpressure - **Error Accumulation**: Tracks consecutive processing errors - **Adaptive Delay**: Increases delay based on error frequency - **Circuit Breaking**: Stops processing when errors exceed threshold ### Connection Management - **Auto-Reconnection**: Built-in NATS reconnection logic - **Graceful Shutdown**: Proper draining with timeouts - **Resource Cleanup**: Ensures subscriptions and connections are closed ## Configuration ### Environment Variables ```bash CAATSM_NATS_URL=nats://localhost:4222 CAATSM_NATS_TOKEN=your-token # Optional CAATSM_DLQ_ENABLED=true CAATSM_DLQ_SUBJECT=caatsm.dlq ``` ### TLS Configuration For production deployments with TLS: ```toml [NATS.Auth] TLSEnabled = true TLSCertFile = "/etc/ssl/certs/client.crt" TLSKeyFile = "/etc/ssl/private/client.key" TLSCAFile = "/etc/ssl/certs/ca.crt" ``` ### Runtime Configuration - **Validation**: Basic validation at startup - **Defaults**: Sensible defaults for essential options ## Testing Strategy ### Unit Tests - **Pure Functions**: Configuration normalization, policy mapping - **Mock Dependencies**: NATS connections, JetStream contexts - **Table-Driven Tests**: Comprehensive coverage of edge cases ### Integration Tests - **Real NATS**: Testcontainers with actual NATS server - **End-to-End**: Complete message processing pipelines - **Failure Scenarios**: Network failures, resource unavailability ### Test Categories - **Happy Path**: Normal operation scenarios - **Error Handling**: Basic error scenarios - **Configuration**: Configuration validation ## Simple Examples ### Example 1: Complete Consumer Setup and Start This example shows how to set up and start a consumer from scratch: ```go package main import ( "context" "time" "caatsm/internal/infra/config" "caatsm/internal/infra/nats" "caatsm/internal/app" "go.uber.org/zap" ) func main() { // 1. Load configuration cfg := &config.Config{ NATS: config.NATSConfig{ URL: "nats://localhost:4222", Stream: "TELEGRAM", Consumer: "telegram-consumer", Auth: config.NATSAuthConfig{ Token: "your-token", // Optional }, ConsumerRules: config.ConsumerRules{ AckWait: 30 * time.Second, MaxDeliver: 3, MaxAckPending: 1000, }, }, App: config.AppConfig{ BatchSize: 50, BatchTimeout: 2 * time.Second, }, DLQ: config.DLQConfig{ Enabled: true, Subject: "caatsm.dlq", }, } // 2. Create NATS connection nc, err := nats.ProvideNATSConn(cfg, zap.NewNop()) if err != nil { panic(err) } defer nc.Close() // 3. Get JetStream context js, err := nats.ProvideJetStream(nc, cfg, zap.NewNop()) if err != nil { panic(err) } // 4. Create message processor (your business logic) processor := app.NewMessageProcessor(/* dependencies */) // 5. Create logger logger := zap.NewNop() // 6. Create telemetry recorder telemetry := /* your telemetry implementation */ // 7. Create consumer consumer, err := nats.ProvideConsumer( nc, js, processor, cfg, telemetry, logger, ) if err != nil { panic(err) } // 8. Start consumer with context ctx, cancel := context.WithCancel(context.Background()) defer cancel() // 9. Start consuming (blocks until context cancelled) if err := consumer.Start(ctx); err != nil { logger.Error("Consumer stopped", zap.Error(err)) } } ``` ### Example 2: Publishing a Message Simple example of publishing a message: ```go package main import ( "caatsm/internal/adapter/dto" "caatsm/internal/infra/nats" "github.com/google/uuid" ) func publishMessage(publisher port.Publisher) error { // Create message with UUID message := &dto.ParsedTelegram{ Uuid: uuid.NewString(), // Used for deduplication Data: []byte("telegram message data"), // ... other fields } // Publish - automatically handles: // - JSON serialization // - UUID header attachment // - Error classification if err := publisher.Publish(message); err != nil { return fmt.Errorf("failed to publish: %w", err) } return nil } ``` ### Example 3: Message Processing Flow Step-by-step what happens when a message is processed: ```go // Step 1: Consumer fetches batch of messages msgs, err := subscription.Fetch(50, nats.MaxWait(2*time.Second)) // Result: Up to 50 messages, or timeout after 2 seconds // Step 2: For each message in batch for _, msg := range msgs { // Step 2a: Extract message ID msgID := msg.Header.Get("Nats-Msg-Id") if msgID == "" { // Fallback: use JetStream sequence meta, _ := msg.Metadata() msgID = fmt.Sprintf("js-%d", meta.Sequence.Stream) } // Step 2b: Create tracing span ctx, span := tracer.Start(ctx, "process.message") span.SetAttributes( attribute.String("messaging.system", "nats"), attribute.String("messaging.destination.name", msg.Subject), ) // Step 2c: Process message (your business logic) err := processor.Handle(ctx, msg.Data, msgID) // Step 2d: Handle result if err != nil { if app.IsPermanent(err) { // Permanent error: route to DLQ and ACK dlqHandler.RouteToDLQ(ctx, msg, err) msg.Ack() } else { // Transient error: NAK with backoff msg.NakWithDelay(calculateBackoff(msg)) } } else { // Success: ACK message msg.Ack() } span.End() } ``` ### Example 4: Error Handling Scenarios Different error scenarios and how they're handled: ```go // Scenario 1: Transient Error (Network Issue) func processMessage(msg *nats.Msg) error { // Simulate network error if networkDown { return app.NewTransientError("network unavailable") } // Result: Message is NAK'd, will be redelivered with backoff } // Scenario 2: Permanent Error (Invalid Format) func processMessage(msg *nats.Msg) error { var data MyStruct if err := json.Unmarshal(msg.Data, &data); err != nil { return app.NewPermanentError("invalid JSON format") } // Result: Message routed to DLQ, original message ACK'd } // Scenario 3: Backpressure Trigger // When 10+ consecutive errors occur: // - Processing pauses // - Sleep duration = min(consecutiveErrors * 100ms, 5s) // - Prevents overwhelming the system // Scenario 4: MaxDeliver Exhausted // When message fails MaxDeliver times (default: 3): // - Message is not automatically handled // - Consider monitoring JetStream consumer info for failed deliveries ``` ### Example 5: DLQ Message Structure What a DLQ message looks like: ```json { "transport_msg_id": "550e8400-e29b-41d4-a716-446655440000", "subject": "telegram.orders.12345", "stream": "TELEGRAM", "consumer": "telegram-consumer", "nats_sequence": 12345, "deliveries": 3, "error": "permanent error: invalid message format", "received_at": "2024-01-15T10:30:00Z", "body": "{\"order_id\":123,\"invalid\":\"data\"}", "advisory_source": false } ``` ### Example 6: Configuration Examples Basic configuration with TLS: ```toml [NATS] URL = "nats://secure.nats.server:4222" Stream = "TELEGRAM" Consumer = "telegram-consumer" [NATS.Auth] TLSEnabled = true TLSCertFile = "/etc/ssl/certs/client.crt" TLSKeyFile = "/etc/ssl/private/client.key" TLSCAFile = "/etc/ssl/certs/ca.crt" [NATS.ConsumerRules] AckWait = "30s" MaxDeliver = 3 MaxAckPending = 1000 [DLQ] Enabled = true Subject = "caatsm.dlq" [App] BatchSize = 50 BatchTimeout = "2s" ``` Development configuration: ```toml [NATS] URL = "nats://localhost:4222" Stream = "TELEGRAM" Consumer = "telegram-consumer" [DLQ] Enabled = true Subject = "caatsm.dlq" ``` ### Example 7: Observability Integration How to monitor the consumer: ```go // Metrics are automatically collected: // - ack_pending: Messages waiting for ACK // - redelivered: Messages being redelivered // - pending: Messages in stream // - delivered: Total messages delivered // Tracing spans are created for: // - Each message processing // - DLQ routing // - Error handling // Logs include: // - Message processing events // - Error details with context // - DLQ routing events // - Connection status changes ``` ## Usage Examples ### Basic Consumer Setup ```go consumer, err := natsinfra.ProvideConsumer( natsConn, jetStream, messageProcessor, config, telemetryRecorder, logger, ) if err != nil { return err } ctx, cancel := context.WithCancel(context.Background()) defer cancel() return consumer.Start(ctx) ``` ### Publishing Messages ```go publisher, err := natsinfra.ProvidePublisher( jetStream, natsConn, config, logger, ) if err != nil { return err } err = publisher.Publish(&dto.ParsedTelegram{ Uuid: uuid.NewString(), Data: telegramData, }) ``` ### Custom Error Handling ```go type CustomProcessor struct { // implementation } func (p *CustomProcessor) ProcessMessage(ctx context.Context, msg *nats.Msg) error { // Business logic here if shouldRetry := someCondition(); shouldRetry { return app.NewTransientError("temporary failure") } if isInvalid := validateMessage(msg); isInvalid { return app.NewPermanentError("invalid message format") } return nil } ``` ## Performance Considerations ### Optimization Strategies - **Batch Processing**: Reduces per-message overhead - **Connection Pooling**: Reuses connections efficiently - **Memory Management**: Proper buffer sizing and cleanup - **Concurrent Processing**: Parallel message processing within batches ### Monitoring Points - **Throughput**: Messages processed per second - **Latency**: End-to-end processing time - **Resource Usage**: Memory, CPU, and network utilization - **Error Rates**: Percentage of failed messages ## Operational Guide ### Deployment 1. **Configuration**: Set appropriate timeouts and limits 2. **Resource Provisioning**: Ensure sufficient NATS cluster capacity 3. **Monitoring Setup**: Configure alerts and dashboards 4. **DLQ Monitoring**: Set up DLQ message processing ### Troubleshooting - **High Latency**: Check batch sizes and processing logic - **Message Loss**: Verify consumer acks and DLQ configuration - **Connection Issues**: Check NATS cluster health and network connectivity - **Resource Exhaustion**: Monitor memory usage and connection counts ### Maintenance - **Stream Cleanup**: Periodically clean up old streams - **Consumer Recreation**: Recreate consumers for configuration changes - **Performance Tuning**: Adjust batch sizes based on load patterns - **Version Upgrades**: Test compatibility with NATS server versions ## Security Considerations ### Authentication - **Token Auth**: Use NATS tokens for simple authentication - **TLS**: Enable TLS with client certificates for secure communication ### TLS Configuration ```toml [NATS.Auth] TLSEnabled = true TLSCertFile = "/path/to/client.crt" TLSKeyFile = "/path/to/client.key" TLSCAFile = "/path/to/ca.crt" ``` ### Data Protection - **TLS Encryption**: All communication is encrypted - **Basic Logging**: Avoid logging sensitive message content ## Future Enhancements ### Future Enhancements - **Additional Auth Methods**: Support for more authentication mechanisms if needed - **Advanced Monitoring**: Enhanced metrics and tracing if required - **Performance Tuning**: Batch size and timeout optimizations