Compare commits
10
Commits
6eff35b56f
...
c398e51779
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c398e51779 | ||
|
|
c0a66cf845 | ||
|
|
ba82b9206a | ||
|
|
b82b707a25 | ||
|
|
dbde6524b3 | ||
|
|
d0fd461e38 | ||
|
|
c1808c0523 | ||
|
|
cf8e196248 | ||
|
|
9a5f8a7d72 | ||
|
|
7b6f6383ad |
@@ -10,6 +10,8 @@
|
||||
|
||||
## Code Style & Architecture
|
||||
- **Structure**: Clean Architecture (`cmd/`, `internal/{domain,app,adapter,infra}`, `pkg/`).
|
||||
- **Parsers**: Composite parser pattern with specialized sub-parsers (aviation, weather).
|
||||
- **Domain**: Core domain types include aviation telegrams and weather reports.
|
||||
- **Formatting**: Run `go fmt ./...` and `goimports` before committing.
|
||||
- **Naming**: `CamelCase` (exported), `camelCase` (private). Package names match dirs.
|
||||
- **Errors**: Wrap with context (`fmt.Errorf("...: %w", err)`). Use `errors.Is`.
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
**go-caatsm** is a Civil Aviation Authority Telegram Message Processor - a high-performance, production-ready message processing system for aviation telegrams. It uses Clean Architecture, NATS JetStream for reliable message streaming, and PostgreSQL/TimescaleDB for persistence.
|
||||
|
||||
The system processes ICAO-format aviation telegrams (ARR, DEP, CNL, DLA, FPL), parsing raw messages from NATS JetStream, storing them in PostgreSQL, and republishing structured JSON to downstream consumers.
|
||||
|
||||
## Essential Commands
|
||||
|
||||
### Build & Run
|
||||
```bash
|
||||
# Build binary (outputs to bin/receiver)
|
||||
make build # or: task build
|
||||
VERSION=v1.0.0 make build # with custom version
|
||||
|
||||
# Development (JetStream mode, auto-creates stream/consumer)
|
||||
make run-dev # or: task run-dev
|
||||
make run-local # or: task run-local (go run, honors GO_ENV)
|
||||
|
||||
# Production (requires config.prod.toml, manual stream/consumer setup)
|
||||
make run-prod # or: task run-prod
|
||||
```
|
||||
|
||||
### Testing
|
||||
```bash
|
||||
# Unit tests (Ginkgo, verbose)
|
||||
make test # or: task test
|
||||
ginkgo -r -v ./cmd ./internal
|
||||
|
||||
# Run single test by description
|
||||
ginkgo -r -v --focus "Test Description" ./path/to/package
|
||||
|
||||
# Integration tests (requires Docker)
|
||||
make test-int # or: task test-int
|
||||
|
||||
# All tests
|
||||
make test-all # or: task test-all
|
||||
|
||||
# Coverage report (target >80%)
|
||||
make coverage # or: task coverage
|
||||
# Opens coverage/coverage.html
|
||||
```
|
||||
|
||||
### Code Quality
|
||||
```bash
|
||||
# Lint (golangci-lint)
|
||||
make lint # or: task lint
|
||||
|
||||
# Format code
|
||||
make fmt # or: task fmt
|
||||
go fmt ./...
|
||||
|
||||
# Regenerate dependency injection (after modifying pkg/di/wire.go)
|
||||
make wire # or: task wire
|
||||
wire ./pkg/di
|
||||
```
|
||||
|
||||
### Development Tools
|
||||
```bash
|
||||
# Install Ginkgo tooling
|
||||
task install-test
|
||||
go install github.com/onsi/ginkgo/v2/ginkgo@latest
|
||||
|
||||
# Generate sample telegrams (burst mode)
|
||||
make seed COUNT=100 CATEGORY=mixed STATUS=random # or: task seed
|
||||
|
||||
# Continuous slow seeding (useful for monitoring)
|
||||
make seed-slow INTERVAL_MIN=2s INTERVAL_MAX=5s # or: task seed-slow
|
||||
# Press Ctrl-C to stop
|
||||
|
||||
# Start dev infrastructure (Docker Compose)
|
||||
task up # Postgres, NATS, observability stack
|
||||
task dev-run # Start infra + run receiver locally
|
||||
task down # Stop and remove containers
|
||||
```
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### Clean Architecture Layers
|
||||
|
||||
```
|
||||
internal/
|
||||
├── domain/ # Pure business entities (no dependencies)
|
||||
│ └── aviation.go # ARR, DEP, CNL, DLA, FPL, Weather domain models
|
||||
├── port/ # Interface contracts (Repository, Publisher)
|
||||
│ ├── repository.go
|
||||
│ └── publisher.go
|
||||
├── app/ # Application logic (orchestration)
|
||||
│ └── processor.go # MessageProcessor - main processing pipeline
|
||||
├── adapter/ # Interface implementations & data transformations
|
||||
│ ├── parser/ # Telegram parsers (composite pattern)
|
||||
│ │ ├── aviation/ # Aviation telegrams (ARR, DEP, CNL, DLA, FPL)
|
||||
│ │ ├── weather/ # Weather reports (METAR, SPECI, TAF)
|
||||
│ │ └── schedule/ # Flight schedule messages
|
||||
│ ├── mapper/ # Domain ↔ DTO transformations
|
||||
│ └── dto/ # Data Transfer Objects (ParsedTelegram, MessageStatus)
|
||||
└── infra/ # Infrastructure concerns
|
||||
├── nats/ # NATS JetStream consumer & publisher
|
||||
├── postgres/ # PostgreSQL repository (pgx, batch inserts)
|
||||
├── config/ # Koanf configuration loading
|
||||
├── log/ # Zap structured logging
|
||||
├── telemetry/ # OpenTelemetry traces/metrics
|
||||
├── metrics/ # Prometheus metrics
|
||||
├── monitoring/ # Health/metrics HTTP server
|
||||
└── buildinfo/ # Build metadata
|
||||
```
|
||||
|
||||
**Dependency Flow**: `infra` → `adapter` → `app` → `domain` ← `port`
|
||||
The domain layer has zero external dependencies. All layers depend on interfaces (ports), not concrete implementations.
|
||||
|
||||
### Key Components
|
||||
|
||||
1. **NATS Consumer** (`internal/infra/nats/consumer.go`)
|
||||
- Pulls messages from JetStream in batches (default: 50)
|
||||
- ACK/NAK handling with automatic retries
|
||||
- Routes to DLQ after max delivery attempts (default: 3)
|
||||
- Monitors consumer lag and pending messages
|
||||
|
||||
2. **MessageProcessor** (`internal/app/processor.go`)
|
||||
- Orchestrates: Parse → Persist → Publish
|
||||
- Error categorization: `parsed`, `header_error`, `body_error`, `publish_error`, `repository_error`
|
||||
- Parser failures are permanent (ACK'd, no retry)
|
||||
- Repository failures are transient (NAK'd, retry with backoff)
|
||||
|
||||
3. **Parser** (`internal/adapter/parser/`)
|
||||
- Composite parser architecture with specialized sub-parsers
|
||||
- **Aviation Parser** (`internal/adapter/parser/aviation/`) - ICAO telegram formats (ARR, DEP, CNL, DLA, FPL)
|
||||
- **Weather Parser** (`internal/adapter/parser/weather/`) - Weather reports (METAR, SPECI, TAF)
|
||||
- **Schedule Parser** (`internal/adapter/parser/schedule/`) - Flight schedule messages
|
||||
- Uses pattern matching, tokenization, and lexical analysis
|
||||
- Returns structured domain models or error status
|
||||
|
||||
4. **Repository** (`internal/infra/postgres/repository.go`)
|
||||
- PostgreSQL persistence via pgx connection pool
|
||||
- Batch insert support via `COPY FROM`
|
||||
- Stores parsed telegrams in `aviation.telegrams` table
|
||||
|
||||
5. **Monitoring Server** (`internal/infra/monitoring/server.go`)
|
||||
- `GET /livez` - Liveness (process info, no dependencies checked)
|
||||
- `GET /readyz` - Readiness (pings Postgres & NATS, 503 on failure)
|
||||
- `GET /metrics` - Prometheus metrics endpoint
|
||||
- Default address: `:2112`
|
||||
|
||||
### Configuration
|
||||
|
||||
Uses Koanf for config loading from TOML files + environment variables:
|
||||
- Config file: `configs/config.{GO_ENV}.toml` (GO_ENV defaults to `dev`)
|
||||
- Env overrides: `CAATSM_` prefix (e.g., `CAATSM_NATS_URL`, `CAATSM_POSTGRES_URL`)
|
||||
- Key settings:
|
||||
- `nats.mode`: Must be `"jetstream"` (only supported mode)
|
||||
- `nats.stream`: JetStream stream name (default: `TELEGRAM`)
|
||||
- `nats.consumer`: Durable consumer name (default: `telegram-consumer`)
|
||||
- `nats.consumer_rules.max_deliver`: Max retry attempts (default: 3)
|
||||
- `dlq.enabled`: Enable Dead-Letter Queue for poison messages
|
||||
- `app.batch_size`: JetStream pull batch size (default: 50)
|
||||
- `monitoring.addr`: Metrics/health server address (default: `:2112`)
|
||||
|
||||
### Parser Architecture
|
||||
|
||||
The system uses a **composite parser pattern** with specialized sub-parsers:
|
||||
|
||||
1. **Composite Parser** (`internal/adapter/parser/composite.go`)
|
||||
- Orchestrates multiple specialized parsers
|
||||
- Routes messages to appropriate parser based on content classification
|
||||
- Falls back gracefully if primary parser fails
|
||||
|
||||
2. **Aviation Parser** (`internal/adapter/parser/aviation/`)
|
||||
- Pattern-based parsing using registry of message type patterns
|
||||
- Tokenizer for breaking down telegram structure
|
||||
- Handles ARR, DEP, CNL, DLA, FPL message types
|
||||
- Extracts flight details, aircraft info, timestamps, airports
|
||||
|
||||
3. **Weather Parser** (`internal/adapter/parser/weather/`)
|
||||
- Lexer-based parsing for weather reports
|
||||
- Classifier to identify METAR, SPECI, or TAF format
|
||||
- Pattern matching for weather elements (wind, visibility, clouds, etc.)
|
||||
- Normalizes weather data into structured format
|
||||
|
||||
4. **Schedule Parser** (`internal/adapter/parser/schedule/`)
|
||||
- Extracts flight schedule information
|
||||
- Pattern matching for schedule-specific fields
|
||||
- Handles recurring flight patterns and time ranges
|
||||
|
||||
Each parser implements the `Parser` interface from `internal/port/parser.go`, enabling easy extension and testing.
|
||||
|
||||
### Dependency Injection
|
||||
|
||||
Uses Google Wire for compile-time DI:
|
||||
- Wire spec: `pkg/di/wire.go`
|
||||
- Generated code: `pkg/di/wire_gen.go` (NEVER edit manually)
|
||||
- To add a new dependency:
|
||||
1. Create a `ProvideXxx` function in the appropriate package
|
||||
2. Add it to the `runtimeSet` in `pkg/di/wire.go`
|
||||
3. Run `make wire` or `task wire`
|
||||
|
||||
### Message Processing Flow
|
||||
|
||||
```
|
||||
NATS JetStream → Consumer.Fetch(batch) → MessageProcessor.Handle(msg) →
|
||||
├─ Parser.Parse(raw) → domain model or error status
|
||||
├─ Repository.Insert(parsed) → Postgres aviation.telegrams
|
||||
├─ Publisher.Publish(json) → NATS output topic
|
||||
└─ ACK (success) | NAK (transient failure) | ACK (permanent failure)
|
||||
```
|
||||
|
||||
**Error Handling**:
|
||||
- **Parser failures** (invalid format): Store raw + status in DB, ACK message (permanent)
|
||||
- **Repository failures** (DB down): NAK message, JetStream redelivers (transient)
|
||||
- **Publisher failures** (downstream topic): Store raw + error status, ACK message (permanent, can replay from DB)
|
||||
- After `max_deliver` attempts: Route to DLQ if enabled, else discard
|
||||
|
||||
### Observability
|
||||
|
||||
Three observability surfaces:
|
||||
|
||||
1. **Prometheus Metrics** (`/metrics`)
|
||||
- `caatsm_messages_total{stream,consumer,result}` - Throughput & results
|
||||
- `caatsm_handle_latency_seconds_bucket{stream,consumer}` - Processing latency
|
||||
- `caatsm_retries_total{stream,consumer,reason}` - JetStream retries
|
||||
- `caatsm_db_queries_total{operation,result}` - DB activity
|
||||
- `caatsm_nats_consumer_pending_messages{stream,consumer}` - Consumer lag
|
||||
- `caatsm_dlq_messages_total{stream,consumer}` - DLQ routing
|
||||
|
||||
2. **OpenTelemetry** (traces + metrics)
|
||||
- Environment-based sampling: Production (1%), Staging (10%), Dev/Test (100%)
|
||||
- Semantic attributes: `messaging.system`, `db.system`, `caatsm.message.category`
|
||||
- Configure via `[telemetry]` block (endpoint, enabled, insecure)
|
||||
- Export to OTLP/HTTP collector (default: `localhost:4318`)
|
||||
|
||||
3. **Health Endpoints**
|
||||
- `GET /livez` - Process liveness (no dependency checks)
|
||||
- `GET /readyz` - Readiness (checks Postgres & NATS, timeout: 2s)
|
||||
- Both return JSON with build metadata + dependency status
|
||||
|
||||
### Testing Philosophy
|
||||
|
||||
- **Unit tests**: Ginkgo BDD style, table-driven, mock interfaces
|
||||
- Location: Alongside code (`*_test.go`)
|
||||
- Run with `make test` or `ginkgo -r -v ./cmd ./internal`
|
||||
- Target: >80% coverage
|
||||
|
||||
- **Integration tests**: Docker-based (testcontainers), full E2E flow
|
||||
- Location: `test/integration/`
|
||||
- Run with `make test-int` (requires Docker)
|
||||
- Spins up real Postgres + NATS JetStream
|
||||
|
||||
- **Benchmarks**: Performance-critical paths (parser, mapper, processor)
|
||||
- Run with `go test -bench=BenchmarkXxx -benchmem ./path/to/pkg`
|
||||
|
||||
## Important Development Notes
|
||||
|
||||
### Code Style (from .cursor/rules/do.mdc)
|
||||
- **Clean Architecture**: Strict layer separation, dependency inversion
|
||||
- **Interface-driven**: All public functions interact with interfaces, not concrete types
|
||||
- **Error handling**: Always wrap errors with context (`fmt.Errorf("context: %w", err)`)
|
||||
- **Context propagation**: Pass `context.Context` everywhere (deadlines, cancellations, tracing)
|
||||
- **No global state**: Use constructor functions with DI
|
||||
- **Resource cleanup**: Use `defer` for closing resources
|
||||
- **Security**: Input validation, secure defaults, retries with exponential backoff
|
||||
- **Observability**: Trace all service boundaries (HTTP, NATS, DB), structured logs (JSON)
|
||||
|
||||
### Never Edit Generated Files
|
||||
- `pkg/di/wire_gen.go` - Wire-generated DI code
|
||||
- Any `*_gen.go` files - Code generation output
|
||||
|
||||
### Common Gotchas
|
||||
|
||||
1. **JetStream-only mode**: The system only supports JetStream mode (`nats.mode = "jetstream"`). Do not attempt to use Core NATS mode.
|
||||
|
||||
2. **Production setup**: In production (`GO_ENV=prod`):
|
||||
- Stream and Consumer must be created manually (not auto-created)
|
||||
- Requires `configs/config.prod.toml` or `CAATSM_*` env vars
|
||||
- Enable TLS for NATS/Postgres connections
|
||||
|
||||
3. **Parsing failures are permanent**: Parser errors (invalid format) are ACK'd and stored with error status. They do NOT trigger JetStream retries.
|
||||
|
||||
4. **Repository failures are transient**: DB connection failures trigger NAK, causing JetStream to redeliver the message.
|
||||
|
||||
5. **Batch processing**: Default batch size is 50 messages. Tune `app.batch_size` for your workload. Larger batches improve throughput but increase latency.
|
||||
|
||||
6. **Consumer lag**: Monitor `caatsm_nats_consumer_pending_messages` metric. High values indicate slow processing or insufficient consumer instances.
|
||||
|
||||
7. **DLQ routing**: Enable DLQ (`dlq.enabled = true`) to capture poison messages after max delivery attempts. Inspect DLQ subject (`caatsm.dlq`) for failed messages.
|
||||
|
||||
## Additional Documentation
|
||||
|
||||
- `README.md` - Comprehensive user guide (setup, configuration, deployment)
|
||||
- `AGENTS.md` - Quick reference for AI assistants (commands, style guide)
|
||||
- `docs/prod-guide.md` - Production deployment guide
|
||||
- `docs/nats.md` - NATS/JetStream configuration details
|
||||
- `docs/observability.md` - Observability setup & metrics
|
||||
- `docs/migrations.md` - Database migration strategy
|
||||
- `docs/performance.md` - Performance tuning guidelines
|
||||
@@ -0,0 +1,94 @@
|
||||
# go-caatsm (Civil Aviation Authority Telegram Message Processor)
|
||||
|
||||
## Project Overview
|
||||
|
||||
`go-caatsm` is a high-performance Go application designed to process aviation telegrams (like FPL, ARR, DEP) and weather reports (METAR, SPECI, TAF) from NATS JetStream, parse them, persist them to PostgreSQL/TimescaleDB, and republish the parsed results. It follows Clean Architecture principles to ensure modularity and testability.
|
||||
|
||||
## Architecture
|
||||
|
||||
The project is structured using Clean Architecture:
|
||||
|
||||
* **`cmd/`**: Application entry points. `cmd/main` is the primary service, `cmd/seed-telegrams` is a utility for generating test data.
|
||||
* **`internal/domain/`**: Core business logic and types (e.g., `aviation.go`, `fpl.go`, `weather/`). Pure Go, no dependencies on outer layers.
|
||||
* **`internal/app/`**: Application business rules (use cases). `MessageProcessor` orchestrates the flow between ports.
|
||||
* **`internal/adapter/`**: Adapters for external interfaces.
|
||||
* `parser/`: Logic to parse raw telegram text into domain objects.
|
||||
* `aviation/`: Aviation telegram parser (ARR, DEP, CNL, DLA, FPL)
|
||||
* `weather/`: Weather report parser (METAR, SPECI, TAF)
|
||||
* `composite.go`: Composite parser that routes messages to appropriate parser
|
||||
* `validator/`: AFTN protocol validation.
|
||||
* `dto/`: Data Transfer Objects.
|
||||
* **`internal/infra/`**: Infrastructure implementations.
|
||||
* `nats/`: NATS JetStream consumer and publisher.
|
||||
* `postgres/`: Database repository using `pgx`.
|
||||
* `config/`, `log/`, `telemetry/`, `monitoring/`: Cross-cutting concerns.
|
||||
* **`internal/port/`**: Interfaces defining the contracts for repositories, publishers, and parsers.
|
||||
* **`pkg/di/`**: Dependency Injection using Google Wire.
|
||||
|
||||
## Tech Stack
|
||||
|
||||
* **Language:** Go 1.24+
|
||||
* **Messaging:** NATS JetStream
|
||||
* **Database:** PostgreSQL (with TimescaleDB extension for time-series data)
|
||||
* **Observability:** OpenTelemetry (OTLP), Prometheus, Jaeger, Grafana, Zap Logger
|
||||
* **CLI:** `urfave/cli`
|
||||
* **DI:** Google Wire
|
||||
* **Testing:** Ginkgo (BDD), Gomega, Testcontainers (integration tests)
|
||||
|
||||
## Key Commands (Taskfile)
|
||||
|
||||
The project uses `Taskfile.yml` for managing common tasks.
|
||||
|
||||
* **Build:** `task build` (Output: `bin/receiver`)
|
||||
* **Run (Dev):** `task run-dev` (Connects to local Docker stack)
|
||||
* **Run (Prod):** `task run-prod`
|
||||
* **Test (Unit):** `task test`
|
||||
* **Test (Integration):** `task test-int` (Requires Docker)
|
||||
* **Lint:** `task lint`
|
||||
* **Start Infrastructure:** `task up` (Starts Postgres, NATS, Observability stack)
|
||||
* **Stop Infrastructure:** `task down`
|
||||
* **Seed Data:** `task seed` (Injects sample telegrams into NATS)
|
||||
|
||||
## Configuration
|
||||
|
||||
Configuration is managed via TOML files in `configs/` and environment variables.
|
||||
* `configs/config.dev.toml`: Default for development (`GO_ENV=dev`).
|
||||
* `configs/config.prod.toml`: Production settings (`GO_ENV=prod`).
|
||||
* Environment Variables: Prefix `CAATSM_` (e.g., `CAATSM_NATS_URL`, `CAATSM_POSTGRES_URL`).
|
||||
|
||||
## Development Workflow
|
||||
|
||||
1. **Start Infrastructure:**
|
||||
```bash
|
||||
task up
|
||||
```
|
||||
2. **Run Service Locally:**
|
||||
```bash
|
||||
task run-dev
|
||||
```
|
||||
3. **Generate Traffic:**
|
||||
```bash
|
||||
task seed
|
||||
# OR for continuous traffic
|
||||
task seed-slow
|
||||
```
|
||||
4. **Observe:**
|
||||
* Grafana: http://localhost:3000 (admin/admin)
|
||||
* Jaeger: http://localhost:16686
|
||||
* Prometheus: http://localhost:9090
|
||||
|
||||
## Key Files & Directories
|
||||
|
||||
* `cmd/main/main.go`: Application entry point. Sets up config, DI, and starts the listener.
|
||||
* `internal/app/processor.go`: `MessageProcessor` - The core orchestration logic.
|
||||
* `internal/adapter/parser/`: Contains parsers for aviation telegrams and weather reports (composite pattern).
|
||||
* `internal/infra/nats/consumer.go`: JetStream consumer implementation.
|
||||
* `internal/infra/postgres/telegrams.ddl`: Database schema.
|
||||
* `docs/`: Extensive documentation (Architecture, NATS, Dev Guide).
|
||||
|
||||
## Notes for AI Agent
|
||||
|
||||
* **Conventions:** Follow existing patterns in `internal/`. Use `internal/port` for interfaces.
|
||||
* **Testing:** New features must include Ginkgo tests. Integration tests should be added for infrastructure components.
|
||||
* **DI:** If adding new components, update `pkg/di/wire.go` and run `task wire` (or `task generate`).
|
||||
* **Safety:** Always check `go.mod` before adding imports.
|
||||
@@ -118,3 +118,12 @@ health_timeout = "2s"
|
||||
enabled = true # Set to true when switching to JetStream mode
|
||||
# subject: NATS subject where failed messages will be published for manual inspection
|
||||
subject = "caatsm.dlq"
|
||||
|
||||
[aftn]
|
||||
# AFTN Protocol Validation and Monitoring
|
||||
# validation_enabled: Enable AFTN protocol validation for telegrams (disabled by default for safe rollout)
|
||||
validation_enabled = false
|
||||
# message_gap_threshold: Duration after which serial reader is considered stalled (no messages received)
|
||||
message_gap_threshold = "2m"
|
||||
# enable_sequence_gap_detection: Monitor for missing sequence numbers in telegram stream
|
||||
enable_sequence_gap_detection = true
|
||||
|
||||
@@ -0,0 +1,863 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "Health status of the serial reader. 1 = healthy (messages flowing), 0 = stalled (no messages received for > 2 minutes)",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"mappings": [
|
||||
{
|
||||
"options": {
|
||||
"0": {
|
||||
"color": "red",
|
||||
"index": 1,
|
||||
"text": "STALLED"
|
||||
},
|
||||
"1": {
|
||||
"color": "green",
|
||||
"index": 0,
|
||||
"text": "HEALTHY"
|
||||
}
|
||||
},
|
||||
"type": "value"
|
||||
}
|
||||
],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 6,
|
||||
"w": 6,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"text": {},
|
||||
"textMode": "value_and_name",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "10.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "caatsm_serial_reader_healthy{stream=\"$stream\", consumer=\"$consumer\"}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Serial Reader Health",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "Time in seconds since the last message was received from the serial reader",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 60
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 120
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 6,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"text": {},
|
||||
"textMode": "value_and_name",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "10.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "caatsm_message_gap_seconds{stream=\"$stream\", consumer=\"$consumer\"}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Message Gap (seconds)",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "Total number of AFTN validation errors in the last 5 minutes",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 20,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 6,
|
||||
"w": 6,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"last"
|
||||
],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(caatsm_aftn_validation_errors_total[5m])) by (error_type)",
|
||||
"legendFormat": "{{error_type}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "AFTN Validation Errors by Type (5m rate)",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "Percentage of messages failing AFTN validation",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"mappings": [],
|
||||
"max": 100,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 1
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 5
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percent"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 6,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"displayMode": "gradient",
|
||||
"maxVizHeight": 300,
|
||||
"minVizHeight": 10,
|
||||
"minVizWidth": 0,
|
||||
"namePlacement": "auto",
|
||||
"orientation": "horizontal",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showUnfilled": true,
|
||||
"sizing": "auto",
|
||||
"text": {},
|
||||
"valueMode": "color"
|
||||
},
|
||||
"pluginVersion": "10.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "100 * (sum(rate(caatsm_aftn_validation_errors_total[5m])) / sum(rate(caatsm_processed_total[5m])))",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "AFTN Error Rate %",
|
||||
"type": "bargauge"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "Time series showing message gap evolution over time",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "Seconds",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "line"
|
||||
}
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 120
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 6
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "caatsm_message_gap_seconds{stream=\"$stream\", consumer=\"$consumer\"}",
|
||||
"legendFormat": "{{stream}}/{{consumer}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Message Gap Over Time",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "Rate of sequence gaps detected (missing message sequence numbers)",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "Gaps/sec",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 20,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "stepAfter",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 6
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"sum"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(caatsm_message_sequence_gap_total{stream=\"$stream\", consumer=\"$consumer\"}[5m])",
|
||||
"legendFormat": "{{stream}}/{{consumer}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Sequence Gap Rate (5m)",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "Number of pending messages in the JetStream consumer queue",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "Messages",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "line"
|
||||
}
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 1000
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 5000
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 14
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "caatsm_nats_consumer_pending_messages{stream=\"$stream\", consumer=\"$consumer\"}",
|
||||
"legendFormat": "{{stream}}/{{consumer}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Consumer Pending Messages",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "Message processing throughput by status",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "Messages/sec",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 20,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "normal"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byRegexp",
|
||||
"options": ".*error.*"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "red",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byRegexp",
|
||||
"options": ".*parsed.*"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "green",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 14
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(caatsm_processed_total[5m])) by (status)",
|
||||
"legendFormat": "{{status}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Message Processing Rate by Status",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"schemaVersion": 39,
|
||||
"tags": [
|
||||
"caatsm",
|
||||
"aftn",
|
||||
"aviation",
|
||||
"telegram"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "Prometheus",
|
||||
"value": "Prometheus"
|
||||
},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"label": "Datasource",
|
||||
"multi": false,
|
||||
"name": "DS_PROMETHEUS",
|
||||
"options": [],
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"type": "datasource"
|
||||
},
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "TELEGRAM",
|
||||
"value": "TELEGRAM"
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"definition": "label_values(caatsm_nats_consumer_pending_messages, stream)",
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"label": "Stream",
|
||||
"multi": false,
|
||||
"name": "stream",
|
||||
"options": [],
|
||||
"query": {
|
||||
"qryType": 1,
|
||||
"query": "label_values(caatsm_nats_consumer_pending_messages, stream)",
|
||||
"refId": "PrometheusVariableQueryEditor-VariableQuery"
|
||||
},
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": 0,
|
||||
"type": "query"
|
||||
},
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "telegram-consumer",
|
||||
"value": "telegram-consumer"
|
||||
},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"definition": "label_values(caatsm_nats_consumer_pending_messages{stream=\"$stream\"}, consumer)",
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"label": "Consumer",
|
||||
"multi": false,
|
||||
"name": "consumer",
|
||||
"options": [],
|
||||
"query": {
|
||||
"qryType": 1,
|
||||
"query": "label_values(caatsm_nats_consumer_pending_messages{stream=\"$stream\"}, consumer)",
|
||||
"refId": "PrometheusVariableQueryEditor-VariableQuery"
|
||||
},
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": 0,
|
||||
"type": "query"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "CAATSM AFTN Health & Validation",
|
||||
"uid": "caatsm-aftn-health",
|
||||
"version": 1,
|
||||
"weekStart": ""
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
# Prometheus Alert Rules for CAATSM AFTN Telegram Processor
|
||||
#
|
||||
# Installation:
|
||||
# 1. Copy this file to your Prometheus server's rules directory
|
||||
# 2. Add to prometheus.yml:
|
||||
# rule_files:
|
||||
# - "prometheus-alerts.yml"
|
||||
# 3. Reload Prometheus configuration
|
||||
#
|
||||
# Alert Severity Levels:
|
||||
# - critical: Immediate action required (pages on-call)
|
||||
# - warning: Investigation needed (notify team channel)
|
||||
|
||||
groups:
|
||||
- name: caatsm_aftn_health
|
||||
interval: 30s
|
||||
rules:
|
||||
# Critical: Serial reader has stopped publishing messages
|
||||
- alert: SerialReaderStalled
|
||||
expr: caatsm_serial_reader_healthy == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
component: serial_reader
|
||||
annotations:
|
||||
summary: "Serial reader stalled for {{ $labels.stream }}/{{ $labels.consumer }}"
|
||||
description: |
|
||||
No messages have been received from the serial reader for more than 2 minutes.
|
||||
This indicates the serial port reader may have crashed or the hardware connection is broken.
|
||||
|
||||
Current gap: {{ with query "caatsm_message_gap_seconds{stream=\"" }}{{ . | first | value | humanizeDuration }}{{ end }}
|
||||
Stream: {{ $labels.stream }}
|
||||
Consumer: {{ $labels.consumer }}
|
||||
|
||||
ACTION REQUIRED:
|
||||
1. Check serial reader process status
|
||||
2. Verify serial port connection
|
||||
3. Check hardware status
|
||||
4. Review serial reader logs
|
||||
|
||||
# Warning: Message gap is growing but not yet critical
|
||||
- alert: HighMessageGap
|
||||
expr: caatsm_message_gap_seconds > 60 and caatsm_serial_reader_healthy == 1
|
||||
for: 1m
|
||||
labels:
|
||||
severity: warning
|
||||
component: serial_reader
|
||||
annotations:
|
||||
summary: "High message gap detected: {{ $labels.stream }}/{{ $labels.consumer }}"
|
||||
description: |
|
||||
Message gap is {{ $value }}s but still below critical threshold.
|
||||
This may indicate slow message processing or reduced incoming message rate.
|
||||
|
||||
Stream: {{ $labels.stream }}
|
||||
Consumer: {{ $labels.consumer }}
|
||||
|
||||
# Warning: Sequence gaps detected (missing messages)
|
||||
- alert: MessageSequenceGaps
|
||||
expr: rate(caatsm_message_sequence_gap_total[5m]) > 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: warning
|
||||
component: serial_reader
|
||||
annotations:
|
||||
summary: "Message sequence gaps detected: {{ $labels.stream }}/{{ $labels.consumer }}"
|
||||
description: |
|
||||
Missing message sequence numbers detected at {{ $value | humanize }} gaps/sec.
|
||||
This indicates messages are being lost or skipped in the stream.
|
||||
|
||||
Stream: {{ $labels.stream }}
|
||||
Consumer: {{ $labels.consumer }}
|
||||
Rate: {{ $value | humanize }} gaps/sec
|
||||
|
||||
Possible causes:
|
||||
- Serial reader buffer overflow
|
||||
- Network packet loss (if messages forwarded over network)
|
||||
- Stream retention limits exceeded
|
||||
- Consumer processing too slow
|
||||
|
||||
# Warning: High AFTN validation error rate
|
||||
- alert: HighAFTNValidationErrorRate
|
||||
expr: |
|
||||
(
|
||||
sum(rate(caatsm_aftn_validation_errors_total[5m])) by (stream, consumer)
|
||||
/
|
||||
sum(rate(caatsm_processed_total[5m])) by (stream, consumer)
|
||||
) > 0.05
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
component: aftn_validator
|
||||
annotations:
|
||||
summary: "High AFTN validation error rate: {{ $value | humanizePercentage }}"
|
||||
description: |
|
||||
More than 5% of incoming telegrams are failing AFTN protocol validation.
|
||||
Current error rate: {{ $value | humanizePercentage }}
|
||||
|
||||
This may indicate:
|
||||
- Upstream system sending malformed telegrams
|
||||
- Serial port data corruption
|
||||
- Configuration mismatch
|
||||
|
||||
Check DLQ for error details and patterns.
|
||||
|
||||
# Warning: Consumer lag is growing
|
||||
- alert: ConsumerLagGrowing
|
||||
expr: |
|
||||
deriv(caatsm_nats_consumer_pending_messages[5m]) > 10
|
||||
for: 3m
|
||||
labels:
|
||||
severity: warning
|
||||
component: consumer
|
||||
annotations:
|
||||
summary: "Consumer lag growing: {{ $labels.stream }}/{{ $labels.consumer }}"
|
||||
description: |
|
||||
Consumer pending messages is growing at {{ $value | humanize }} msgs/sec.
|
||||
Current pending: {{ with query "caatsm_nats_consumer_pending_messages" }}{{ . | first | value }}{{ end }}
|
||||
|
||||
This indicates the consumer cannot keep up with incoming message rate.
|
||||
|
||||
Stream: {{ $labels.stream }}
|
||||
Consumer: {{ $labels.consumer }}
|
||||
|
||||
# Critical: Consumer critically behind
|
||||
- alert: ConsumerCriticallyBehind
|
||||
expr: caatsm_nats_consumer_pending_messages > 5000
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
component: consumer
|
||||
annotations:
|
||||
summary: "Consumer critically behind: {{ $value }} pending messages"
|
||||
description: |
|
||||
Consumer has {{ $value }} pending messages - critically behind.
|
||||
This will cause message processing delays and may trigger stream retention limits.
|
||||
|
||||
Stream: {{ $labels.stream }}
|
||||
Consumer: {{ $labels.consumer }}
|
||||
Pending: {{ $value }}
|
||||
|
||||
ACTION REQUIRED:
|
||||
1. Check processor performance and errors
|
||||
2. Check database connection and performance
|
||||
3. Consider scaling consumers horizontally
|
||||
4. Review stream retention settings
|
||||
|
||||
# Warning: High processing failure rate
|
||||
- alert: HighProcessingFailureRate
|
||||
expr: |
|
||||
(
|
||||
sum(rate(caatsm_messages_total{result="fail"}[5m])) by (stream, consumer)
|
||||
/
|
||||
sum(rate(caatsm_messages_total[5m])) by (stream, consumer)
|
||||
) > 0.10
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
component: processor
|
||||
annotations:
|
||||
summary: "High processing failure rate: {{ $value | humanizePercentage }}"
|
||||
description: |
|
||||
More than 10% of messages are failing to process.
|
||||
Current failure rate: {{ $value | humanizePercentage }}
|
||||
|
||||
Stream: {{ $labels.stream }}
|
||||
Consumer: {{ $labels.consumer }}
|
||||
|
||||
Check application logs for error details.
|
||||
|
||||
# Warning: High publish failure rate
|
||||
- alert: HighPublishFailureRate
|
||||
expr: |
|
||||
sum(rate(caatsm_publish_failures_total[5m])) by (category)
|
||||
/
|
||||
sum(rate(caatsm_processed_total[5m])) by (category) > 0.05
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
component: publisher
|
||||
annotations:
|
||||
summary: "High publish failure rate for {{ $labels.category }}: {{ $value | humanizePercentage }}"
|
||||
description: |
|
||||
More than 5% of {{ $labels.category }} messages failing to publish.
|
||||
Current failure rate: {{ $value | humanizePercentage }}
|
||||
|
||||
Category: {{ $labels.category }}
|
||||
|
||||
Check NATS JetStream connectivity and publisher logs.
|
||||
|
||||
# Warning: DLQ publish failures (messages lost)
|
||||
- alert: DLQPublishFailures
|
||||
expr: rate(caatsm_dlq_publish_failures_total[5m]) > 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: warning
|
||||
component: dlq
|
||||
annotations:
|
||||
summary: "DLQ publish failures detected"
|
||||
description: |
|
||||
Failed messages cannot be published to DLQ - messages may be lost!
|
||||
Failure rate: {{ $value | humanize }} msgs/sec
|
||||
|
||||
Stream: {{ $labels.stream }}
|
||||
Consumer: {{ $labels.consumer }}
|
||||
|
||||
Check DLQ subject configuration and NATS JetStream health.
|
||||
|
||||
- name: caatsm_aftn_validation_details
|
||||
interval: 1m
|
||||
rules:
|
||||
# Recording rule: AFTN error rate by type
|
||||
- record: caatsm:aftn_validation_error_rate:5m
|
||||
expr: |
|
||||
rate(caatsm_aftn_validation_errors_total[5m])
|
||||
|
||||
# Recording rule: Total processing rate
|
||||
- record: caatsm:processing_rate:5m
|
||||
expr: |
|
||||
sum(rate(caatsm_processed_total[5m])) by (stream, consumer, status)
|
||||
|
||||
# Recording rule: Average message gap
|
||||
- record: caatsm:message_gap_seconds:avg
|
||||
expr: |
|
||||
avg(caatsm_message_gap_seconds) by (stream, consumer)
|
||||
+1
-1
@@ -58,7 +58,7 @@ Use these tasks if you prefer a one-command workflow instead of invoking `docker
|
||||
|
||||
## Publishing Sample Telegrams
|
||||
|
||||
Use the helper CLI in `cmd/seed-telegrams` to push realistic payloads onto NATS (mirrors the fixtures in `internal/adapter/parser/aviation_parser_test.go`):
|
||||
Use the helper CLI in `cmd/seed-telegrams` to push realistic payloads onto NATS. The tool supports both aviation telegrams (FPL, ARR, DEP, etc.) and weather reports (METAR, SPECI, TAF).
|
||||
|
||||
### Publishing to JetStream (Recommended)
|
||||
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
# Weather Parser Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
The weather parser module provides parsing capabilities for aviation weather reports including METAR, SPECI, and TAF messages. It is integrated into the system using a composite parser pattern that routes weather reports to the weather parser while maintaining backward compatibility with existing aviation telegram parsing.
|
||||
|
||||
## Architecture
|
||||
|
||||
The weather parser follows Clean Architecture principles:
|
||||
|
||||
- **Domain Layer** (`internal/domain/weather/`): Core domain types and interfaces
|
||||
- **Port Layer** (`internal/port/weather_parser.go`): Parser interface definition
|
||||
- **Adapter Layer** (`internal/adapter/parser/weather/`): Parser implementation
|
||||
- **Composite Parser** (`internal/adapter/parser/composite.go`): Routes messages to appropriate parser
|
||||
|
||||
## Supported Report Types
|
||||
|
||||
### METAR (Aviation Routine Weather Report)
|
||||
Standard hourly weather observations from airports.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 Q1013=
|
||||
```
|
||||
|
||||
### SPECI (Aviation Selected Special Weather Report)
|
||||
Special weather observations issued when conditions change significantly.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
SPECI KORD 251215Z 27015G25KT 5SM -RA BKN030 OVC050 20/18 A2992=
|
||||
```
|
||||
|
||||
### TAF (Terminal Aerodrome Forecast)
|
||||
Forecast weather conditions for airports, typically valid for 24-30 hours.
|
||||
|
||||
**Example:**
|
||||
```
|
||||
TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020 FM251800 36015KT 10SM SCT030=
|
||||
```
|
||||
|
||||
## Parsed Elements
|
||||
|
||||
### Core Elements
|
||||
|
||||
- **Station**: 4-letter ICAO airport code
|
||||
- **Time**: Issue/observation time (DDHHmmZ format)
|
||||
- **Wind**: Direction, speed, gusts, variable conditions
|
||||
- **Visibility**: Distance, unit (meters or statute miles), directional visibility
|
||||
- **Clouds**: Type (FEW/SCT/BKN/OVC/VV), altitude, modifiers (CB/TCU)
|
||||
- **Temperature/Dewpoint**: Temperature in Celsius
|
||||
- **Altimeter**: Pressure setting (QNH in hPa or A in inHg)
|
||||
- **Weather Phenomena**: Intensity, descriptors, weather codes
|
||||
|
||||
### TAF-Specific Elements
|
||||
|
||||
- **Validity Period**: Forecast valid from/to times
|
||||
- **Periods**: Main forecast, FM (from), TEMPO (temporary), BECMG (becoming)
|
||||
- **Probability**: PROB30, PROB40 for uncertain conditions
|
||||
|
||||
## Error Handling
|
||||
|
||||
The parser uses a lenient approach:
|
||||
|
||||
- **Unrecognized tokens**: Recorded in `warnings` array, parsing continues
|
||||
- **Missing required fields**: Returns appropriate domain errors
|
||||
- **Invalid format**: Returns `ErrInvalidFormat`
|
||||
|
||||
This ensures that partial parsing is possible even when some elements are not recognized.
|
||||
|
||||
## Usage
|
||||
|
||||
The weather parser is automatically integrated via the composite parser. No special configuration is required.
|
||||
|
||||
### Message Flow
|
||||
|
||||
1. Raw message received
|
||||
2. Composite parser checks if message is a weather report
|
||||
3. If weather report: parsed by weather parser
|
||||
4. If not: parsed by aviation parser (existing behavior)
|
||||
5. Parsed result stored in `telegrams` table with `category` = "METAR"/"SPECI"/"TAF"
|
||||
6. Structured data stored in `body_data` JSONB field
|
||||
|
||||
### Database Storage
|
||||
|
||||
Weather reports are stored in the existing `telegrams` table:
|
||||
|
||||
- `category`: "METAR", "SPECI", or "TAF"
|
||||
- `body_data`: JSONB containing structured weather data
|
||||
- `content`: Original raw text
|
||||
- `message_id`: Generated as `{station}-{issue_time}`
|
||||
|
||||
## Testing
|
||||
|
||||
Test files are located in `internal/adapter/parser/weather/`:
|
||||
|
||||
- `classifier_test.go`: Tests report type classification
|
||||
- `metar_parser_test.go`: Tests METAR/SPECI parsing
|
||||
- `taf_parser_test.go`: Tests TAF parsing
|
||||
- `composite_test.go`: Tests composite parser routing
|
||||
|
||||
Run tests:
|
||||
```bash
|
||||
go test ./internal/adapter/parser/weather/... -v
|
||||
```
|
||||
|
||||
## Limitations and Future Enhancements
|
||||
|
||||
Current implementation covers core METAR/TAF elements. Future enhancements may include:
|
||||
|
||||
- Runway Visual Range (RVR) parsing
|
||||
- More comprehensive weather phenomenon codes
|
||||
- Enhanced TAF period parsing
|
||||
- Additional METAR modifiers
|
||||
- Station metadata integration
|
||||
|
||||
## References
|
||||
|
||||
- [ICAO Annex 3: Meteorological Service for International Air Navigation](https://www.icao.int/safety/meteorology/pages/annex-3.aspx)
|
||||
- [WMO Manual on Codes](https://library.wmo.int/index.php?lvl=notice_display&id=13617)
|
||||
|
||||
@@ -12,7 +12,7 @@ require (
|
||||
github.com/knadh/koanf/v2 v2.3.0
|
||||
github.com/nats-io/nats.go v1.47.0
|
||||
github.com/onsi/ginkgo/v2 v2.27.2
|
||||
github.com/onsi/gomega v1.38.2
|
||||
github.com/onsi/gomega v1.38.3
|
||||
github.com/prometheus/client_golang v1.23.2
|
||||
github.com/stretchr/testify v1.11.1
|
||||
github.com/testcontainers/testcontainers-go v0.30.0
|
||||
|
||||
@@ -14,6 +14,7 @@ const (
|
||||
MessageStatusParsed MessageStatus = "parsed"
|
||||
MessageStatusHeaderError MessageStatus = "header_error"
|
||||
MessageStatusBodyError MessageStatus = "body_error"
|
||||
MessageStatusAFTNError MessageStatus = "aftn_error"
|
||||
)
|
||||
|
||||
// ParsedTelegram holds the parsed data from an aviation message.
|
||||
|
||||
+149
-177
@@ -1,46 +1,14 @@
|
||||
package parser
|
||||
package aviation
|
||||
|
||||
import (
|
||||
"caatsm/internal/adapter/dto"
|
||||
"caatsm/internal/domain"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
const (
|
||||
SSR = "ssr"
|
||||
DepartureCode = "dep"
|
||||
DepartureTime = "dep_time"
|
||||
ArrivalCode = "arr"
|
||||
ArrivalTime = "arr_time"
|
||||
DestinationCode = "dest"
|
||||
OtherInfo = "other"
|
||||
|
||||
ReferenceData = "reference_data"
|
||||
CategorySurveillance = "surve"
|
||||
Indicator = "indicator"
|
||||
Other = "other"
|
||||
AircraftID = "aircraft"
|
||||
Surveillance = "surve"
|
||||
Speed = "speed"
|
||||
Level = "level"
|
||||
Route = "route"
|
||||
EstimatedTime = "estt"
|
||||
AlternateAirport = "alter"
|
||||
PBN = "pbn"
|
||||
NavigationEquipment = "nav"
|
||||
EstimatedElapsedTime = "eet"
|
||||
SELCALCode = "sel"
|
||||
PerformanceCategory = "per"
|
||||
RerouteInformation = "rif"
|
||||
Remarks = "remark"
|
||||
)
|
||||
|
||||
var (
|
||||
@@ -61,7 +29,6 @@ var (
|
||||
type BodyParser struct {
|
||||
body string
|
||||
bodyPatterns map[string]BodyConfig
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func NewBodyParser(body string) *BodyParser {
|
||||
@@ -71,39 +38,41 @@ func NewBodyParser(body string) *BodyParser {
|
||||
}
|
||||
}
|
||||
|
||||
// GetBodyPatterns returns the body patterns map.
|
||||
// The bodyPatterns map is a reference to the package-level bodyPatterns,
|
||||
// which is initialized once at startup and never modified, making it safe
|
||||
// for concurrent reads without synchronization.
|
||||
func (parser *BodyParser) GetBodyPatterns() map[string]BodyConfig {
|
||||
parser.mu.Lock()
|
||||
defer parser.mu.Unlock()
|
||||
copied := make(map[string]BodyConfig, len(parser.bodyPatterns))
|
||||
for k, v := range parser.bodyPatterns {
|
||||
copied[k] = v
|
||||
}
|
||||
return copied
|
||||
}
|
||||
|
||||
func (parser *BodyParser) SetBodyPatterns(patterns map[string]BodyConfig) {
|
||||
parser.mu.Lock()
|
||||
defer parser.mu.Unlock()
|
||||
parser.bodyPatterns = patterns
|
||||
return parser.bodyPatterns
|
||||
}
|
||||
|
||||
func (parser *BodyParser) Parse() (string, interface{}, error) {
|
||||
parser.mu.Lock()
|
||||
defer parser.mu.Unlock()
|
||||
|
||||
parser.body = strings.TrimSpace(parser.body)
|
||||
category := findCategory(parser.body)
|
||||
if category == "" {
|
||||
return "", nil, fmt.Errorf("no category found in body text")
|
||||
}
|
||||
|
||||
if patternConfig, exists := parser.bodyPatterns[category]; exists && patternConfig.Patterns != nil {
|
||||
for _, p := range patternConfig.Patterns {
|
||||
if data := extract(parser.body, p.Expression); data != nil {
|
||||
return parser.createBodyData(data)
|
||||
patternConfig, exists := parser.bodyPatterns[category]
|
||||
if !exists || patternConfig.Patterns == nil {
|
||||
return "", nil, fmt.Errorf("no matching pattern found for category: %s", category)
|
||||
}
|
||||
|
||||
ctx := ParseContext{
|
||||
Body: parser.body,
|
||||
Tokens: Tokenizer{}.Tokenize(parser.body),
|
||||
}
|
||||
|
||||
for _, p := range patternConfig.Patterns {
|
||||
if data := extract(parser.body, p.Expression); data != nil {
|
||||
parsed, err := parseCategory(category, ctx, data)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
return category, parsed, nil
|
||||
}
|
||||
}
|
||||
|
||||
return "", nil, fmt.Errorf("no matching pattern found for body: %s", parser.body)
|
||||
}
|
||||
|
||||
@@ -119,7 +88,12 @@ func findCategory(body string) string {
|
||||
}
|
||||
|
||||
func extract(data string, exp *regexp.Regexp) map[string]string {
|
||||
match := exp.FindStringSubmatch(data)
|
||||
// Use timeout protection to prevent ReDoS attacks
|
||||
match, err := MatchWithTimeout(exp, data, DefaultRegexTimeout)
|
||||
if err != nil {
|
||||
// Timeout occurred - return nil to indicate no match
|
||||
return nil
|
||||
}
|
||||
if len(match) > 0 {
|
||||
return extractData(match, exp)
|
||||
}
|
||||
@@ -136,73 +110,6 @@ func extractData(match []string, re *regexp.Regexp) map[string]string {
|
||||
return data
|
||||
}
|
||||
|
||||
func (parser *BodyParser) createBodyData(data map[string]string) (string, interface{}, error) {
|
||||
switch category := data["category"]; category {
|
||||
case CategoryArrival:
|
||||
return category, &domain.ARR{
|
||||
Category: data[Category],
|
||||
AircraftID: data[FlightNumber],
|
||||
SSRModeAndCode: data[SSR],
|
||||
DepartureAirport: data[DepartureCode],
|
||||
ArrivalAirport: data[ArrivalCode],
|
||||
ArrivalTime: data[ArrivalTime],
|
||||
}, nil
|
||||
case CategoryDeparture:
|
||||
return category, &domain.DEP{
|
||||
Category: data[Category],
|
||||
AircraftID: data[FlightNumber],
|
||||
SSRModeAndCode: data[SSR],
|
||||
DepartureAirport: data[DepartureCode],
|
||||
DepartureTime: data[DepartureTime],
|
||||
Destination: data[ArrivalCode],
|
||||
}, nil
|
||||
case CategoryCancellation:
|
||||
return category, &domain.CNL{
|
||||
Category: data[Category],
|
||||
AircraftID: data[FlightNumber],
|
||||
DepartureAirport: data[DepartureCode],
|
||||
DestinationAirport: data[ArrivalCode],
|
||||
}, nil
|
||||
case CategoryDelay:
|
||||
return category, &domain.DLA{
|
||||
Category: data[Category],
|
||||
AircraftID: data[FlightNumber],
|
||||
DepartureAirport: data[DepartureCode],
|
||||
NewDepartureTime: data[DepartureTime],
|
||||
ArrivalAirport: data[ArrivalCode],
|
||||
ArrivalTime: data[ArrivalTime],
|
||||
}, nil
|
||||
case CategoryFlightPlan:
|
||||
otherData := parseOther(data[OtherInfo])
|
||||
return category, &domain.FPL{
|
||||
Category: data[Category],
|
||||
FlightNumber: data[FlightNumber],
|
||||
ReferenceData: data[ReferenceData],
|
||||
AircraftID: data[AircraftID],
|
||||
SSRModeAndCode: data[Surveillance],
|
||||
FlightRulesAndType: data[Indicator],
|
||||
CruisingSpeedAndLevel: data[Speed] + data[Level],
|
||||
DepartureAirport: data[DepartureCode],
|
||||
DepartureTime: data[DepartureTime],
|
||||
Route: data[Route],
|
||||
DestinationAndTotalTime: data[DestinationCode] + data[EstimatedTime],
|
||||
AlternateAirport: data[AlternateAirport],
|
||||
OtherInfo: data[OtherInfo],
|
||||
Register: otherData[Register],
|
||||
EstimatedArrivalTime: data[EstimatedTime],
|
||||
PBN: otherData[PBN],
|
||||
NavigationEquipment: otherData[NavigationEquipment],
|
||||
EstimatedElapsedTime: otherData[EstimatedElapsedTime],
|
||||
SELCALCode: otherData[SELCALCode],
|
||||
PerformanceCategory: otherData[PerformanceCategory],
|
||||
RerouteInformation: otherData[RerouteInformation],
|
||||
Remarks: otherData[Remarks],
|
||||
}, nil
|
||||
default:
|
||||
return category, nil, fmt.Errorf("invalid message type: %s", category)
|
||||
}
|
||||
}
|
||||
|
||||
func headerToParsedTelegram(header Header) dto.ParsedTelegram {
|
||||
return dto.ParsedTelegram{
|
||||
MessageID: header.MessageID,
|
||||
@@ -220,7 +127,50 @@ func headerToParsedTelegram(header Header) dto.ParsedTelegram {
|
||||
}
|
||||
}
|
||||
|
||||
// Parse parses a raw ICAO aviation telegram and returns a ParsedTelegram with parsing status.
|
||||
//
|
||||
// IMPORTANT ERROR HANDLING PATTERN:
|
||||
// This function intentionally returns both a non-nil ParsedTelegram AND an error when parsing fails.
|
||||
// This design decision allows the caller to persist failed parse attempts with error details to the
|
||||
// database for audit and compliance purposes. This pattern is specific to the aviation parser's
|
||||
// error handling strategy where parser failures are permanent (ACK'd, not retried) and must be
|
||||
// stored for regulatory compliance and troubleshooting.
|
||||
//
|
||||
// Error Handling Strategy:
|
||||
// - Input validation failure: Returns ParsedTelegram with MessageStatusHeaderError + ErrHeaderParse
|
||||
// - Header parse failure: Returns ParsedTelegram with MessageStatusHeaderError + ErrHeaderParse
|
||||
// - Body parse failure: Returns ParsedTelegram with MessageStatusBodyError + ErrBodyParse
|
||||
// - Success: Returns ParsedTelegram with MessageStatusParsed + nil error
|
||||
//
|
||||
// The returned ParsedTelegram is ALWAYS non-nil and safe to use, even when error is non-nil.
|
||||
// Callers should check both the error and the ParsedTelegram.Status field to determine the outcome.
|
||||
//
|
||||
// Security:
|
||||
// - Input size validation prevents DoS attacks (max 1800 chars per AFTN standard)
|
||||
// - Regex timeout protection prevents ReDoS attacks (100ms timeout)
|
||||
//
|
||||
// Example usage:
|
||||
//
|
||||
// parsed, err := Parse(rawTelegram)
|
||||
// if err != nil {
|
||||
// // Parse failed, but parsed contains error details for storage
|
||||
// repository.InsertRaw(parsed) // Store for audit
|
||||
// return Permanent(err) // Don't retry
|
||||
// }
|
||||
// // Parse succeeded
|
||||
// repository.Insert(parsed)
|
||||
// publisher.Publish(parsed.BodyData)
|
||||
func Parse(rawText string) (*dto.ParsedTelegram, error) {
|
||||
// Validate input size to prevent DoS attacks
|
||||
if err := ValidateInputSize(rawText); err != nil {
|
||||
msg := dto.NewParsedTelegram()
|
||||
msg.Content = rawText
|
||||
msg.Comments = err.Error()
|
||||
msg.ErrorReason = err.Error()
|
||||
msg.Status = dto.MessageStatusHeaderError
|
||||
return msg, fmt.Errorf("%w: %w", ErrHeaderParse, err)
|
||||
}
|
||||
|
||||
header, err := ParseHeader(rawText)
|
||||
if err != nil {
|
||||
msg := dto.NewParsedTelegram()
|
||||
@@ -283,14 +233,18 @@ type Header struct {
|
||||
ParsedAt time.Time
|
||||
}
|
||||
|
||||
// ParseHeader parses the header portion of an ICAO telegram.
|
||||
// It extracts message metadata (ID, datetime, addresses, originator) and separates
|
||||
// the body content for subsequent parsing.
|
||||
//
|
||||
// Returns Header struct with parsed fields and the raw body content.
|
||||
// On error, returns Header with Content field populated for audit purposes.
|
||||
func ParseHeader(fullMessage string) (Header, error) {
|
||||
log := zap.S()
|
||||
cleaned := cleanMessage(fullMessage)
|
||||
lines := strings.Split(cleaned, "\n")
|
||||
|
||||
if len(lines) < 3 {
|
||||
log.Warnf("invalid message format: %s", fullMessage)
|
||||
return Header{Content: fullMessage}, fmt.Errorf("invalid message format: %s", fullMessage)
|
||||
if len(lines) < MinHeaderLines {
|
||||
return Header{Content: fullMessage}, fmt.Errorf("invalid message format: expected at least %d lines, got %d", MinHeaderLines, len(lines))
|
||||
}
|
||||
|
||||
_, messageID, dateTime, err := parseStartIndicator(lines[0])
|
||||
@@ -317,86 +271,104 @@ func ParseHeader(fullMessage string) (Header, error) {
|
||||
|
||||
func parseStartIndicator(line string) (string, string, string, error) {
|
||||
parts := strings.Fields(line)
|
||||
if len(parts) >= 3 && strings.HasPrefix(parts[0], StartIndicatorPrefix) {
|
||||
if len(parts) >= MinStartIndicatorParts && strings.HasPrefix(parts[0], StartIndicatorPrefix) {
|
||||
return parts[0], parts[1], parts[2], nil
|
||||
}
|
||||
zap.S().Warnf("invalid start indicator line format: %s", line)
|
||||
return "", "", "", fmt.Errorf("invalid start indicator line format: %s", line)
|
||||
}
|
||||
|
||||
func parsePriorityAndPrimary(line string) (string, string) {
|
||||
parts := strings.Fields(line)
|
||||
if len(parts) >= 2 {
|
||||
if len(parts) >= MinPriorityLineParts {
|
||||
return parts[0], parts[1]
|
||||
}
|
||||
zap.S().Warnf("invalid priority and primary address line format: %s", line)
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// isOriginatorLine checks if a dot-prefixed line matches the originator format (.CODE DATETIME).
|
||||
// Returns the originator code, datetime, and whether it's a valid match.
|
||||
func isOriginatorLine(line string) (originator, dateTime string, isMatch bool) {
|
||||
if !strings.HasPrefix(line, ".") {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
parts := strings.Fields(line[1:])
|
||||
if len(parts) < MinOriginatorParts {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
if isAllUppercaseLetters(parts[0]) && isAllDigits(parts[1]) {
|
||||
return parts[0], parts[1], true
|
||||
}
|
||||
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// isBodyStartLine checks if a line indicates the start of the message body.
|
||||
func isBodyStartLine(line string) bool {
|
||||
return strings.HasPrefix(line, BeginPartMarker) || strings.HasPrefix(line, "(")
|
||||
}
|
||||
|
||||
func parseRemainingLines(lines []string) (string, string, string, string) {
|
||||
var (
|
||||
secondaryAddresses string
|
||||
secondaryAddresses strings.Builder
|
||||
originator string
|
||||
originatorDateTime string
|
||||
bodyAndFooter strings.Builder
|
||||
headerEnded bool
|
||||
body strings.Builder
|
||||
inBody bool
|
||||
)
|
||||
|
||||
for _, line := range lines {
|
||||
line = strings.TrimSpace(line)
|
||||
if headerEnded {
|
||||
bodyAndFooter.WriteString(line + "\n")
|
||||
} else {
|
||||
switch {
|
||||
case line == EndHeaderMarker:
|
||||
case strings.HasPrefix(line, "."):
|
||||
// Validate if dot-prefixed line matches originator format: .ORIGINATOR_CODE YYMMDD
|
||||
// Originator code should be uppercase letters, date/time should be digits
|
||||
originatorInfo := strings.Fields(line[1:])
|
||||
if len(originatorInfo) >= 2 {
|
||||
// Check if first token is all uppercase letters and second is all digits
|
||||
firstToken := originatorInfo[0]
|
||||
secondToken := originatorInfo[1]
|
||||
if isAllUppercaseLetters(firstToken) && isAllDigits(secondToken) {
|
||||
originator = firstToken
|
||||
originatorDateTime = secondToken
|
||||
headerEnded = true
|
||||
} else {
|
||||
// Doesn't match originator format, treat as body content
|
||||
headerEnded = true
|
||||
bodyAndFooter.WriteString(line + "\n")
|
||||
}
|
||||
} else {
|
||||
// Not enough tokens for originator format, treat as body content
|
||||
headerEnded = true
|
||||
bodyAndFooter.WriteString(line + "\n")
|
||||
}
|
||||
case strings.HasPrefix(line, BeginPartMarker) || strings.HasPrefix(line, "("):
|
||||
headerEnded = true
|
||||
if strings.Index(line, "NNNN") > 0 {
|
||||
break
|
||||
}
|
||||
bodyAndFooter.WriteString(line + "\n")
|
||||
default:
|
||||
if o1, o2 := getOriginator(line); o1 != "" {
|
||||
originatorDateTime = o1
|
||||
originator = o2
|
||||
} else {
|
||||
secondaryAddresses = secondaryAddresses + " " + line
|
||||
}
|
||||
}
|
||||
|
||||
// Skip empty lines and single dots
|
||||
if line == "" || line == EndHeaderMarker {
|
||||
continue
|
||||
}
|
||||
|
||||
// Once in body, collect all remaining lines
|
||||
if inBody {
|
||||
body.WriteString(line + "\n")
|
||||
continue
|
||||
}
|
||||
|
||||
// Check for originator line (.CODE DATETIME)
|
||||
if orig, dt, isOrig := isOriginatorLine(line); isOrig {
|
||||
originator = orig
|
||||
originatorDateTime = dt
|
||||
inBody = true
|
||||
continue
|
||||
}
|
||||
|
||||
// Check for body start markers
|
||||
if isBodyStartLine(line) {
|
||||
inBody = true
|
||||
// Skip lines containing NNNN (end marker)
|
||||
if !strings.Contains(line, "NNNN") {
|
||||
body.WriteString(line + "\n")
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// Try to parse as originator using regex (fallback)
|
||||
if dt, orig := getOriginator(line); orig != "" {
|
||||
originatorDateTime = dt
|
||||
originator = orig
|
||||
continue
|
||||
}
|
||||
|
||||
// Otherwise, treat as secondary address
|
||||
secondaryAddresses.WriteString(" " + line)
|
||||
}
|
||||
|
||||
return secondaryAddresses, originator, originatorDateTime, bodyAndFooter.String()
|
||||
return secondaryAddresses.String(), originator, originatorDateTime, body.String()
|
||||
}
|
||||
|
||||
func getOriginator(line string) (string, string) {
|
||||
match := originator.FindStringSubmatch(line)
|
||||
if len(match) >= 3 {
|
||||
if len(match) >= MinOriginatorMatchGroups {
|
||||
return match[1], match[2]
|
||||
}
|
||||
zap.S().Warnf("invalid originator line format: %s", line)
|
||||
return "", ""
|
||||
}
|
||||
|
||||
+1
-2
@@ -1,4 +1,4 @@
|
||||
package parser
|
||||
package aviation
|
||||
|
||||
import (
|
||||
"testing"
|
||||
@@ -134,4 +134,3 @@ func BenchmarkParseMixed(b *testing.B) {
|
||||
_, _ = Parse(msg)
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
package parser
|
||||
package aviation
|
||||
|
||||
import (
|
||||
"caatsm/internal/domain"
|
||||
@@ -0,0 +1,159 @@
|
||||
package aviation
|
||||
|
||||
import "regexp"
|
||||
|
||||
// String constants
|
||||
const (
|
||||
StartIndicatorPrefix = "ZCZC"
|
||||
EndHeaderMarker = "."
|
||||
BeginPartMarker = "BEGIN PART"
|
||||
|
||||
Category = "category"
|
||||
CategoryArrival = "ARR"
|
||||
CategoryDeparture = "DEP"
|
||||
CategoryCancellation = "CNL"
|
||||
CategoryDelay = "DLA"
|
||||
CategoryFlightPlan = "FPL"
|
||||
|
||||
FlightNumber = "number"
|
||||
Register = "reg"
|
||||
|
||||
SSR = "ssr"
|
||||
DepartureCode = "dep"
|
||||
DepartureTime = "dep_time"
|
||||
ArrivalCode = "arr"
|
||||
ArrivalTime = "arr_time"
|
||||
DestinationCode = "dest"
|
||||
OtherInfo = "other"
|
||||
|
||||
ReferenceData = "reference_data"
|
||||
CategorySurveillance = "surve"
|
||||
Indicator = "indicator"
|
||||
Other = "other"
|
||||
AircraftID = "aircraft"
|
||||
Surveillance = "surve"
|
||||
Speed = "speed"
|
||||
Level = "level"
|
||||
Route = "route"
|
||||
EstimatedTime = "estt"
|
||||
AlternateAirport = "alter"
|
||||
PBN = "pbn"
|
||||
NavigationEquipment = "nav"
|
||||
EstimatedElapsedTime = "eet"
|
||||
SELCALCode = "sel"
|
||||
PerformanceCategory = "per"
|
||||
RerouteInformation = "rif"
|
||||
Remarks = "remark"
|
||||
)
|
||||
|
||||
// Parser configuration constants
|
||||
const (
|
||||
// MinHeaderLines is the minimum number of lines required for a valid telegram header
|
||||
MinHeaderLines = 3
|
||||
|
||||
// MinStartIndicatorParts is the minimum number of parts in the start indicator line (ZCZC MessageID DateTime)
|
||||
MinStartIndicatorParts = 3
|
||||
|
||||
// MinPriorityLineParts is the minimum number of parts in the priority line (Priority PrimaryAddress)
|
||||
MinPriorityLineParts = 2
|
||||
|
||||
// MinOriginatorParts is the minimum number of parts in an originator line (.CODE DATETIME)
|
||||
MinOriginatorParts = 2
|
||||
|
||||
// MinOriginatorMatchGroups is the minimum number of regex match groups for originator pattern
|
||||
MinOriginatorMatchGroups = 3
|
||||
)
|
||||
|
||||
// Regular expression patterns for ICAO telegram body parsing.
|
||||
// These patterns match specific message types defined in ICAO standards.
|
||||
const (
|
||||
// ArrPatternString matches ARR (Arrival) messages.
|
||||
// Format: (ARR-FLIGHTNUM[/SSR]-DEPICAO-ARRICAOTIME)
|
||||
// Example: (ARR-CES5470/A1234-ZBTJ-ZSHC1614)
|
||||
// Capture groups:
|
||||
// - category: Message type (ARR)
|
||||
// - number: Flight number (alphanumeric, e.g., CES5470)
|
||||
// - ssr: SSR mode and code (optional, after /, e.g., A1234)
|
||||
// - dep: Departure airport (4-letter ICAO code, e.g., ZBTJ)
|
||||
// - arr: Arrival airport (4-letter ICAO code, e.g., ZSHC)
|
||||
// - arr_time: Arrival time (4 digits HHMM, e.g., 1614)
|
||||
ArrPatternString = `^\((?P<category>[A-Z]{3})-(?P<number>[A-Z0-9]+)(\/?(?P<ssr>[A-Z0-9]+))?-(?P<dep>[A-Z]{4})-(?P<arr>[A-Z]{4})(?P<arr_time>\d{4})\)$`
|
||||
|
||||
// DepPatternString matches DEP (Departure) messages.
|
||||
// Format: (DEP-FLIGHTNUM[/SSR]-DEPICAOTIME-ARRICAO)
|
||||
// Example: (DEP-CYZ9017/A5633-ZBTJ1638-ZSPD)
|
||||
// Capture groups:
|
||||
// - category: Message type (DEP)
|
||||
// - number: Flight number (alphanumeric, e.g., CYZ9017)
|
||||
// - ssr: SSR mode and code (optional, after /, e.g., A5633)
|
||||
// - dep: Departure airport (4-letter ICAO code, e.g., ZBTJ)
|
||||
// - dep_time: Departure time (4 digits HHMM, e.g., 1638)
|
||||
// - arr: Destination airport (4-letter ICAO code, e.g., ZSPD)
|
||||
DepPatternString = `^\((?P<category>[A-Z]{3})-(?P<number>[A-Z0-9]+)(\/(?P<ssr>[A-Z0-9]+))?-(?P<dep>[A-Z]{4})(?P<dep_time>\d{4})-(?P<arr>[A-Z]{4})\)$`
|
||||
|
||||
// FplPatternString matches FPL (Flight Plan) messages.
|
||||
// This is the most complex pattern, matching ICAO Doc 4444 Field Type 15 format.
|
||||
// Format spans multiple lines with specific field ordering per ICAO standards.
|
||||
// Example: (FPL-CCA1532-IS\n-A332/H\n-SDE3FGHIJ4J5M1RWY/LB101\n-ZSSS2035\n-K0859S1040 PIAKS G330...\n-ZBAA0153 ZBYN\n-PBN/A1B2... RMK/TCAS EQUIPPED)
|
||||
// Capture groups:
|
||||
// - category: Message type (FPL)
|
||||
// - number: Flight number (e.g., CCA1532)
|
||||
// - indicator: Flight rules and type (2 letters, e.g., IS)
|
||||
// - aircraft: Aircraft type and wake turbulence (e.g., A332/H)
|
||||
// - surve: Surveillance equipment codes
|
||||
// - dep: Departure airport (4-letter ICAO)
|
||||
// - dep_time: Departure time (4 digits HHMM)
|
||||
// - speed: Cruising speed (e.g., K0859)
|
||||
// - level: Flight level (e.g., S1040)
|
||||
// - route: Flight route (can span multiple lines)
|
||||
// - dest: Destination airport (4-letter ICAO)
|
||||
// - estt: Estimated elapsed time (4 digits)
|
||||
// - alter: Alternate airports (space-separated ICAO codes)
|
||||
// - other: Other information fields (PBN, NAV, REG, EET, SEL, PER, RIF, RMK)
|
||||
FplPatternString = `\((?P<category>[A-Z]{3})-(?P<number>[A-Z]+\d+)-(?P<indicator>[A-Z]{2})\n-(?P<aircraft>[A-Z]+\d+\/?[A-Z]?)\n?-(?P<surve>.*)\n?-(?P<dep>[A-Z]{4})(?P<dep_time>\d{4})\n?-(?P<speed>[A-Z]+\d+)(?P<level>[A-Z0-9]+)\s+(?P<route>(.|\n)+)\n-(?P<dest>[A-Z]{4})(?P<estt>\d{4})\s?(?P<alter>(\s[A-Z]{4})+)\n?-([A-Z]{3}\/(?:[A-Z]{4}\d{4}\s?)+)?(?P<other>(?m)[A-Z]{3}\/(.|\n)*)\)$`
|
||||
|
||||
// CnlPatternString matches CNL (Cancellation) messages.
|
||||
// Format: (CNL-FLIGHTNUM-[DEPICAO]-ARRICAO)
|
||||
// Example: (CNL-YZR7979-ZSPD-ZBTJ)
|
||||
// Capture groups:
|
||||
// - category: Message type (CNL)
|
||||
// - number: Flight number (alphanumeric, e.g., YZR7979)
|
||||
// - dep: Departure airport (4-letter ICAO, optional)
|
||||
// - arr: Destination airport (4-letter ICAO)
|
||||
CnlPatternString = `^\((?P<category>[A-Z]{3})-(?P<number>\w+\d+)-?(?P<dep>[A-Z]{4})?-?(?<arr>[A-Z]{4})\)$`
|
||||
|
||||
// DlaPatternString matches DLA (Delay) messages.
|
||||
// Format: (DLA-FLIGHTNUM-DEPICAO[TIME]-ARRICAO[TIME])
|
||||
// Example: (DLA-CSN3133-ZGGG0110-ZBTJ)
|
||||
// Capture groups:
|
||||
// - category: Message type (DLA)
|
||||
// - number: Flight number (alphanumeric, e.g., CSN3133)
|
||||
// - dep: Departure airport (4-letter ICAO)
|
||||
// - dep_time: New departure time (4 digits HHMM, optional)
|
||||
// - arr: Arrival airport (4-letter ICAO)
|
||||
// - arr_time: Arrival time (4 digits HHMM, optional)
|
||||
DlaPatternString = `^\((?P<category>[A-Z]{3})-(?P<number>\w+\d+)-?(?P<dep>[A-Z]{4})(?P<dep_time>\d{4})?-?(?<arr>[A-Z]{4})(?<arr_time>\d{4})?\)$`
|
||||
)
|
||||
|
||||
// Compiled regular expressions
|
||||
var (
|
||||
ArrPatternExpression = regexp.MustCompile(ArrPatternString)
|
||||
DepPatternExpression = regexp.MustCompile(DepPatternString)
|
||||
FplPatternExpression = regexp.MustCompile(FplPatternString)
|
||||
CnlPatternExpression = regexp.MustCompile(CnlPatternString)
|
||||
DlaPatternExpression = regexp.MustCompile(DlaPatternString)
|
||||
BodyTypePattern = regexp.MustCompile(`^\(([A-Z]{3})(.*\n?)+\)$`)
|
||||
|
||||
categoryRegex = regexp.MustCompile(`\((?P<category>[A-Z]+)-`)
|
||||
emptyLineRemove = regexp.MustCompile(`(?m)^\s*$`)
|
||||
bodyOnly = regexp.MustCompile(`(.|\n)?(ZCZC(.|\n)*)NNNN(.|\n)?$`)
|
||||
originator = regexp.MustCompile(`(?P<originatorDateTime>[0-9]+)\s(?P<originator>[A-Z]+)`)
|
||||
navPattern = regexp.MustCompile(`(?m)NAV\/(?P<nav>\w+)`)
|
||||
remarkPattern = regexp.MustCompile(`(?s)RMK\/(?P<remark>.*)`)
|
||||
selPattern = regexp.MustCompile(`(?m)SEL\/(?P<sel>\w+)`)
|
||||
regPattern = regexp.MustCompile(`(?m)REG\/(?P<reg>[A-Z0-9]+)`)
|
||||
pbnPattern = regexp.MustCompile(`(?m)PBN\/(?P<pbn>[A-Z0-9]+)`)
|
||||
eetPattern = regexp.MustCompile(`(?s)(-?EET\/(?P<eet>(?:[A-Z]{4}\d{4}\s*)+))`)
|
||||
performancePattern = regexp.MustCompile(`(?s)-?PER\/(?P<per>\w)`)
|
||||
reroutePattern = regexp.MustCompile(`(?m)RIF\/(?P<rif>.*)[A-Z]{3}\/`)
|
||||
)
|
||||
@@ -0,0 +1,88 @@
|
||||
// Package aviation provides parsing capabilities for ICAO aviation telegrams.
|
||||
//
|
||||
// This package implements a robust parser for ICAO-format aviation messages
|
||||
// following AFTN (Aeronautical Fixed Telecommunication Network) standards.
|
||||
// It supports multiple message types used in civil aviation operations.
|
||||
//
|
||||
// # Supported Message Types
|
||||
//
|
||||
// The parser handles five primary message categories:
|
||||
//
|
||||
// - ARR (Arrival): Aircraft arrival notifications with departure/arrival airports and times
|
||||
// - DEP (Departure): Aircraft departure notifications with departure/destination airports
|
||||
// - CNL (Cancellation): Flight cancellation messages
|
||||
// - DLA (Delay): Flight delay notifications with updated times
|
||||
// - FPL (Flight Plan): Complete flight plan messages per ICAO Doc 4444
|
||||
//
|
||||
// # Architecture
|
||||
//
|
||||
// The parser uses a registry-based architecture with specialized parsers for each
|
||||
// message category. The main components are:
|
||||
//
|
||||
// - Parse(): Entry point for parsing complete telegrams (header + body)
|
||||
// - ParseHeader(): Extracts header metadata (addresses, originator, timestamps)
|
||||
// - BodyParser: Routes body content to category-specific parsers
|
||||
// - CategoryParser: Interface implemented by each message type parser
|
||||
//
|
||||
// # Security Features
|
||||
//
|
||||
// The parser includes multiple security protections:
|
||||
//
|
||||
// - Input size validation (max 1800 chars per AFTN standard)
|
||||
// - ReDoS protection with 100ms regex timeout
|
||||
// - Field validation to prevent nil pointer dereferences
|
||||
// - Error message sanitization to prevent data leakage
|
||||
//
|
||||
// # Usage Example
|
||||
//
|
||||
// rawTelegram := `ZCZC ABC123 261530
|
||||
// FF ZBBBZPZX
|
||||
// 261530 ZBBBYMYX
|
||||
// (ARR-CES5470/A1234-ZBTJ-ZSHC1614)
|
||||
// NNNN`
|
||||
//
|
||||
// parsed, err := aviation.Parse(rawTelegram)
|
||||
// if err != nil {
|
||||
// // Parse failed - check parsed.Status for error type
|
||||
// log.Printf("Parse error: %v, status: %s", err, parsed.Status)
|
||||
// return
|
||||
// }
|
||||
//
|
||||
// // Parse succeeded - access structured data
|
||||
// if arr, ok := parsed.BodyData.(*domain.ARR); ok {
|
||||
// fmt.Printf("Flight %s arrived at %s\n", arr.AircraftID, arr.ArrivalAirport)
|
||||
// }
|
||||
//
|
||||
// # Error Handling
|
||||
//
|
||||
// The Parse() function follows a unique error handling pattern: it ALWAYS returns
|
||||
// a non-nil ParsedTelegram, even when an error occurs. This allows callers to
|
||||
// persist failed parse attempts with error details for audit and compliance.
|
||||
//
|
||||
// Error categories:
|
||||
//
|
||||
// - MessageStatusHeaderError: Invalid header format or size validation failure
|
||||
// - MessageStatusBodyError: Invalid body format or unsupported message type
|
||||
// - MessageStatusParsed: Successful parse
|
||||
//
|
||||
// Parser failures are considered permanent (should be ACK'd in message queue systems).
|
||||
// The returned ParsedTelegram contains error details in the ErrorReason field.
|
||||
//
|
||||
// # Performance
|
||||
//
|
||||
// The parser is designed for high-throughput message processing:
|
||||
//
|
||||
// - Zero-allocation tokenization where possible
|
||||
// - Compiled regex patterns (initialized once at startup)
|
||||
// - No global state or locks (thread-safe by design)
|
||||
// - Batch-friendly (no shared mutable state between Parse() calls)
|
||||
//
|
||||
// # Standards Compliance
|
||||
//
|
||||
// This implementation follows:
|
||||
//
|
||||
// - ICAO Doc 4444 (PANS-ATM) for flight plan format
|
||||
// - ICAO Annex 10 for AFTN message structure
|
||||
// - AFTN size limits (1800 characters maximum)
|
||||
//
|
||||
package aviation
|
||||
@@ -0,0 +1,48 @@
|
||||
package aviation
|
||||
|
||||
import "regexp"
|
||||
|
||||
// BodyConfig represents the configuration for parsing message bodies.
|
||||
type BodyConfig struct {
|
||||
Patterns []PatternConfig
|
||||
}
|
||||
|
||||
// PatternConfig represents the configuration for a specific pattern.
|
||||
type PatternConfig struct {
|
||||
Pattern string
|
||||
Comments string
|
||||
Expression *regexp.Regexp
|
||||
}
|
||||
|
||||
var (
|
||||
bodyPatterns = buildBodyPatterns()
|
||||
)
|
||||
|
||||
// FindPatterns finds the matching body configuration based on the message body.
|
||||
func FindPatterns(messageBody string) *BodyConfig {
|
||||
if match := BodyTypePattern.FindStringSubmatch(messageBody); len(match) > 1 {
|
||||
name := match[1]
|
||||
if bodyConfig, found := bodyPatterns[name]; found {
|
||||
return &bodyConfig
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ParseBody parses the message body and returns the extracted values.
|
||||
func ParseBody(messageBody string) map[string]string {
|
||||
if body := FindPatterns(messageBody); body != nil {
|
||||
for _, pattern := range body.Patterns {
|
||||
if matches := pattern.Expression.FindStringSubmatch(messageBody); matches != nil {
|
||||
result := make(map[string]string)
|
||||
for i, name := range pattern.Expression.SubexpNames() {
|
||||
if i != 0 && name != "" {
|
||||
result[name] = matches[i]
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
package parser
|
||||
package aviation
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
@@ -0,0 +1,16 @@
|
||||
package aviation
|
||||
|
||||
import "caatsm/internal/adapter/dto"
|
||||
|
||||
// AviationParser implements the Parser interface for aviation telegrams.
|
||||
type AviationParser struct{}
|
||||
|
||||
// NewParser creates a new aviation parser instance.
|
||||
func NewParser() *AviationParser {
|
||||
return &AviationParser{}
|
||||
}
|
||||
|
||||
// Parse parses a raw message string and returns a ParsedTelegram.
|
||||
func (p *AviationParser) Parse(rawText string) (*dto.ParsedTelegram, error) {
|
||||
return Parse(rawText)
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package aviation
|
||||
|
||||
import (
|
||||
"context"
|
||||
"regexp"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// DefaultRegexTimeout is the maximum time allowed for regex matching operations.
|
||||
// This prevents ReDoS (Regular Expression Denial of Service) attacks from
|
||||
// maliciously crafted inputs that cause catastrophic backtracking.
|
||||
DefaultRegexTimeout = 100 * time.Millisecond
|
||||
)
|
||||
|
||||
// MatchWithTimeout executes a regex match with timeout protection.
|
||||
// It runs the regex matching in a goroutine and returns an error if the
|
||||
// operation exceeds the specified timeout duration.
|
||||
//
|
||||
// This is critical for preventing ReDoS attacks where complex patterns
|
||||
// (especially the FPL pattern with nested quantifiers) could hang indefinitely
|
||||
// on malicious input.
|
||||
//
|
||||
// Parameters:
|
||||
// - re: The compiled regular expression to match
|
||||
// - input: The input string to match against
|
||||
// - timeout: Maximum duration allowed for the match operation
|
||||
//
|
||||
// Returns:
|
||||
// - []string: The match result (same format as regexp.FindStringSubmatch)
|
||||
// - error: ValidationError if timeout occurs, nil otherwise
|
||||
func MatchWithTimeout(re *regexp.Regexp, input string, timeout time.Duration) ([]string, error) {
|
||||
type result struct {
|
||||
match []string
|
||||
}
|
||||
|
||||
resultChan := make(chan result, 1)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), timeout)
|
||||
defer cancel()
|
||||
|
||||
// Run regex matching in a goroutine
|
||||
go func() {
|
||||
match := re.FindStringSubmatch(input)
|
||||
resultChan <- result{match: match}
|
||||
}()
|
||||
|
||||
// Wait for either result or timeout
|
||||
select {
|
||||
case res := <-resultChan:
|
||||
return res.match, nil
|
||||
case <-ctx.Done():
|
||||
return nil, &ValidationError{
|
||||
Field: "regex_timeout",
|
||||
Message: "regex matching exceeded timeout",
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
package aviation
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"time"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("Regex Timeout", func() {
|
||||
|
||||
Describe("MatchWithTimeout", func() {
|
||||
Context("with simple pattern and normal input", func() {
|
||||
It("should match successfully within timeout", func() {
|
||||
re := regexp.MustCompile(`^(\w+)-(\w+)$`)
|
||||
input := "ARR-CES5470"
|
||||
|
||||
match, err := MatchWithTimeout(re, input, DefaultRegexTimeout)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(match).To(HaveLen(3))
|
||||
Expect(match[0]).To(Equal("ARR-CES5470"))
|
||||
Expect(match[1]).To(Equal("ARR"))
|
||||
Expect(match[2]).To(Equal("CES5470"))
|
||||
})
|
||||
})
|
||||
|
||||
Context("with pattern that doesn't match", func() {
|
||||
It("should return nil match without error", func() {
|
||||
re := regexp.MustCompile(`^(\w+)-(\w+)$`)
|
||||
input := "INVALID FORMAT"
|
||||
|
||||
match, err := MatchWithTimeout(re, input, DefaultRegexTimeout)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(match).To(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with complex ARR pattern", func() {
|
||||
It("should match ARR message within timeout", func() {
|
||||
input := "(ARR-CES5470/A1234-ZBTJ-ZSHC1614)"
|
||||
|
||||
match, err := MatchWithTimeout(ArrPatternExpression, input, DefaultRegexTimeout)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(match).ToNot(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with complex DEP pattern", func() {
|
||||
It("should match DEP message within timeout", func() {
|
||||
input := "(DEP-CYZ9017/A5633-ZBTJ1638-ZSPD)"
|
||||
|
||||
match, err := MatchWithTimeout(DepPatternExpression, input, DefaultRegexTimeout)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(match).ToNot(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with very short timeout", func() {
|
||||
It("should timeout on complex pattern", func() {
|
||||
// Use a very short timeout to force timeout
|
||||
veryShortTimeout := 1 * time.Nanosecond
|
||||
re := regexp.MustCompile(`^(.+)+$`)
|
||||
input := "aaaaaaaaaaaaaaaaaaaaaaaaaaaa!"
|
||||
|
||||
match, err := MatchWithTimeout(re, input, veryShortTimeout)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(match).To(BeNil())
|
||||
|
||||
valErr, ok := err.(*ValidationError)
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(valErr.Field).To(Equal("regex_timeout"))
|
||||
Expect(valErr.Message).To(ContainSubstring("exceeded timeout"))
|
||||
})
|
||||
})
|
||||
|
||||
Context("with empty input", func() {
|
||||
It("should handle empty input gracefully", func() {
|
||||
re := regexp.MustCompile(`^(\w+)$`)
|
||||
input := ""
|
||||
|
||||
match, err := MatchWithTimeout(re, input, DefaultRegexTimeout)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(match).To(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with named capture groups", func() {
|
||||
It("should preserve named groups in match result", func() {
|
||||
re := regexp.MustCompile(`^(?P<category>\w+)-(?P<number>\w+)$`)
|
||||
input := "ARR-CES5470"
|
||||
|
||||
match, err := MatchWithTimeout(re, input, DefaultRegexTimeout)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(match).To(HaveLen(3))
|
||||
Expect(match[0]).To(Equal("ARR-CES5470"))
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,317 @@
|
||||
package aviation
|
||||
|
||||
import (
|
||||
"caatsm/internal/domain"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// ParseContext carries the raw body and tokens for field parsers.
|
||||
type ParseContext struct {
|
||||
Body string
|
||||
Tokens []Token
|
||||
}
|
||||
|
||||
// CategoryParser maps a category to patterns and output parsing.
|
||||
type CategoryParser interface {
|
||||
Category() string
|
||||
Patterns() []PatternConfig
|
||||
Parse(ctx ParseContext, data map[string]string) (interface{}, error)
|
||||
}
|
||||
|
||||
type arrParser struct{}
|
||||
|
||||
func (arrParser) Category() string { return CategoryArrival }
|
||||
|
||||
func (arrParser) Patterns() []PatternConfig {
|
||||
return []PatternConfig{
|
||||
{
|
||||
Pattern: ArrPatternString,
|
||||
Comments: "Pattern for ARR message",
|
||||
Expression: ArrPatternExpression,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (arrParser) Parse(_ ParseContext, data map[string]string) (interface{}, error) {
|
||||
// Validate and extract required fields
|
||||
category, err := GetRequiredField(data, Category)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
aircraftID, err := GetRequiredField(data, FlightNumber)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
depAirport, err := GetRequiredField(data, DepartureCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
arrAirport, err := GetRequiredField(data, ArrivalCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
arrTime, err := GetRequiredField(data, ArrivalTime)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &domain.ARR{
|
||||
Category: category,
|
||||
AircraftID: aircraftID,
|
||||
SSRModeAndCode: GetOptionalField(data, SSR),
|
||||
DepartureAirport: depAirport,
|
||||
ArrivalAirport: arrAirport,
|
||||
ArrivalTime: arrTime,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type depParser struct{}
|
||||
|
||||
func (depParser) Category() string { return CategoryDeparture }
|
||||
|
||||
func (depParser) Patterns() []PatternConfig {
|
||||
return []PatternConfig{
|
||||
{
|
||||
Pattern: DepPatternString,
|
||||
Comments: "Pattern for DEP message",
|
||||
Expression: DepPatternExpression,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (depParser) Parse(_ ParseContext, data map[string]string) (interface{}, error) {
|
||||
// Validate and extract required fields
|
||||
category, err := GetRequiredField(data, Category)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
aircraftID, err := GetRequiredField(data, FlightNumber)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
depAirport, err := GetRequiredField(data, DepartureCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
depTime, err := GetRequiredField(data, DepartureTime)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
destination, err := GetRequiredField(data, ArrivalCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &domain.DEP{
|
||||
Category: category,
|
||||
AircraftID: aircraftID,
|
||||
SSRModeAndCode: GetOptionalField(data, SSR),
|
||||
DepartureAirport: depAirport,
|
||||
DepartureTime: depTime,
|
||||
Destination: destination,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type cnlParser struct{}
|
||||
|
||||
func (cnlParser) Category() string { return CategoryCancellation }
|
||||
|
||||
func (cnlParser) Patterns() []PatternConfig {
|
||||
return []PatternConfig{
|
||||
{
|
||||
Pattern: CnlPatternString,
|
||||
Comments: "Pattern for CNL message",
|
||||
Expression: CnlPatternExpression,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (cnlParser) Parse(_ ParseContext, data map[string]string) (interface{}, error) {
|
||||
// Validate and extract required fields
|
||||
category, err := GetRequiredField(data, Category)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
aircraftID, err := GetRequiredField(data, FlightNumber)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
depAirport, err := GetRequiredField(data, DepartureCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
destAirport, err := GetRequiredField(data, ArrivalCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &domain.CNL{
|
||||
Category: category,
|
||||
AircraftID: aircraftID,
|
||||
DepartureAirport: depAirport,
|
||||
DestinationAirport: destAirport,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type dlaParser struct{}
|
||||
|
||||
func (dlaParser) Category() string { return CategoryDelay }
|
||||
|
||||
func (dlaParser) Patterns() []PatternConfig {
|
||||
return []PatternConfig{
|
||||
{
|
||||
Pattern: DlaPatternString,
|
||||
Comments: "Pattern for DLA message",
|
||||
Expression: DlaPatternExpression,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (dlaParser) Parse(_ ParseContext, data map[string]string) (interface{}, error) {
|
||||
// Validate and extract required fields
|
||||
category, err := GetRequiredField(data, Category)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
aircraftID, err := GetRequiredField(data, FlightNumber)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
depAirport, err := GetRequiredField(data, DepartureCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
arrAirport, err := GetRequiredField(data, ArrivalCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &domain.DLA{
|
||||
Category: category,
|
||||
AircraftID: aircraftID,
|
||||
DepartureAirport: depAirport,
|
||||
NewDepartureTime: GetOptionalField(data, DepartureTime),
|
||||
ArrivalAirport: arrAirport,
|
||||
ArrivalTime: GetOptionalField(data, ArrivalTime),
|
||||
}, nil
|
||||
}
|
||||
|
||||
type fplParser struct{}
|
||||
|
||||
func (fplParser) Category() string { return CategoryFlightPlan }
|
||||
|
||||
func (fplParser) Patterns() []PatternConfig {
|
||||
return []PatternConfig{
|
||||
{
|
||||
Pattern: FplPatternString,
|
||||
Comments: "Pattern for FPL message",
|
||||
Expression: FplPatternExpression,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (fplParser) Parse(_ ParseContext, data map[string]string) (interface{}, error) {
|
||||
// Validate and extract required fields
|
||||
category, err := GetRequiredField(data, Category)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
flightNumber, err := GetRequiredField(data, FlightNumber)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
aircraftID, err := GetRequiredField(data, AircraftID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
indicator, err := GetRequiredField(data, Indicator)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
speed, err := GetRequiredField(data, Speed)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
level, err := GetRequiredField(data, Level)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
depAirport, err := GetRequiredField(data, DepartureCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
depTime, err := GetRequiredField(data, DepartureTime)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
route, err := GetRequiredField(data, Route)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
destCode, err := GetRequiredField(data, DestinationCode)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
estTime, err := GetRequiredField(data, EstimatedTime)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Parse optional "other" fields
|
||||
otherInfo := GetOptionalField(data, OtherInfo)
|
||||
otherData := parseOther(otherInfo)
|
||||
|
||||
return &domain.FPL{
|
||||
Category: category,
|
||||
FlightNumber: flightNumber,
|
||||
ReferenceData: GetOptionalField(data, ReferenceData),
|
||||
AircraftID: aircraftID,
|
||||
SSRModeAndCode: GetOptionalField(data, Surveillance),
|
||||
FlightRulesAndType: indicator,
|
||||
CruisingSpeedAndLevel: speed + level,
|
||||
DepartureAirport: depAirport,
|
||||
DepartureTime: depTime,
|
||||
Route: route,
|
||||
DestinationAndTotalTime: destCode + estTime,
|
||||
AlternateAirport: GetOptionalField(data, AlternateAirport),
|
||||
OtherInfo: otherInfo,
|
||||
Register: otherData[Register],
|
||||
EstimatedArrivalTime: estTime,
|
||||
PBN: otherData[PBN],
|
||||
NavigationEquipment: otherData[NavigationEquipment],
|
||||
EstimatedElapsedTime: otherData[EstimatedElapsedTime],
|
||||
SELCALCode: otherData[SELCALCode],
|
||||
PerformanceCategory: otherData[PerformanceCategory],
|
||||
RerouteInformation: otherData[RerouteInformation],
|
||||
Remarks: otherData[Remarks],
|
||||
}, nil
|
||||
}
|
||||
|
||||
var categoryRegistry = map[string]CategoryParser{
|
||||
CategoryArrival: arrParser{},
|
||||
CategoryDeparture: depParser{},
|
||||
CategoryCancellation: cnlParser{},
|
||||
CategoryDelay: dlaParser{},
|
||||
CategoryFlightPlan: fplParser{},
|
||||
}
|
||||
|
||||
func lookupCategoryParser(category string) (CategoryParser, bool) {
|
||||
parser, ok := categoryRegistry[category]
|
||||
return parser, ok
|
||||
}
|
||||
|
||||
func buildBodyPatterns() map[string]BodyConfig {
|
||||
patterns := make(map[string]BodyConfig, len(categoryRegistry))
|
||||
for category, parser := range categoryRegistry {
|
||||
patterns[category] = BodyConfig{Patterns: parser.Patterns()}
|
||||
}
|
||||
return patterns
|
||||
}
|
||||
|
||||
func parseCategory(category string, ctx ParseContext, data map[string]string) (interface{}, error) {
|
||||
parser, ok := lookupCategoryParser(category)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("invalid message type: %s", category)
|
||||
}
|
||||
return parser.Parse(ctx, data)
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
package aviation
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestAviation(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Aviation Parser Suite")
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
package aviation
|
||||
|
||||
import "strings"
|
||||
|
||||
// Token represents a lexeme in the body with its byte offsets.
|
||||
type Token struct {
|
||||
Text string
|
||||
Start int
|
||||
End int
|
||||
}
|
||||
|
||||
// Tokenizer splits text into tokens using a whitespace set.
|
||||
// Whitespace characters split tokens but are not emitted.
|
||||
// All other characters (including '/') are included in tokens.
|
||||
type Tokenizer struct {
|
||||
Whitespace string
|
||||
}
|
||||
|
||||
// Tokenize tokenizes input and returns tokens with byte offsets.
|
||||
func (t Tokenizer) Tokenize(input string) []Token {
|
||||
if t.Whitespace == "" {
|
||||
t.Whitespace = " \n\t\r"
|
||||
}
|
||||
|
||||
var tokens []Token
|
||||
start := -1
|
||||
|
||||
for idx, r := range input {
|
||||
if strings.ContainsRune(t.Whitespace, r) {
|
||||
if start != -1 {
|
||||
tokens = append(tokens, Token{
|
||||
Text: input[start:idx],
|
||||
Start: start,
|
||||
End: idx,
|
||||
})
|
||||
start = -1
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
if start == -1 {
|
||||
start = idx
|
||||
}
|
||||
}
|
||||
|
||||
if start != -1 {
|
||||
tokens = append(tokens, Token{
|
||||
Text: input[start:],
|
||||
Start: start,
|
||||
End: len(input),
|
||||
})
|
||||
}
|
||||
|
||||
return tokens
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package aviation
|
||||
|
||||
import (
|
||||
"caatsm/internal/domain"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTokenizerDefaultWhitespace(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := "A B\nC\tD\rE"
|
||||
tokens := Tokenizer{}.Tokenize(input)
|
||||
|
||||
expected := []Token{
|
||||
{Text: "A", Start: 0, End: 1},
|
||||
{Text: "B", Start: 2, End: 3},
|
||||
{Text: "C", Start: 4, End: 5},
|
||||
{Text: "D", Start: 6, End: 7},
|
||||
{Text: "E", Start: 8, End: 9},
|
||||
}
|
||||
|
||||
if len(tokens) != len(expected) {
|
||||
t.Fatalf("expected %d tokens, got %d", len(expected), len(tokens))
|
||||
}
|
||||
|
||||
for i, token := range tokens {
|
||||
if token != expected[i] {
|
||||
t.Fatalf("token %d mismatch: got %#v, expected %#v", i, token, expected[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTokenizerSlashWhitespace(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// When slash is in whitespace, it splits tokens but is not emitted
|
||||
input := "A/B C"
|
||||
tokens := Tokenizer{Whitespace: " \n\t\r/"}.Tokenize(input)
|
||||
|
||||
expected := []Token{
|
||||
{Text: "A", Start: 0, End: 1},
|
||||
{Text: "B", Start: 2, End: 3},
|
||||
{Text: "C", Start: 4, End: 5},
|
||||
}
|
||||
|
||||
if len(tokens) != len(expected) {
|
||||
t.Fatalf("expected %d tokens, got %d", len(expected), len(tokens))
|
||||
}
|
||||
|
||||
for i, token := range tokens {
|
||||
if token != expected[i] {
|
||||
t.Fatalf("token %d mismatch: got %#v, expected %#v", i, token, expected[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildBodyPatternsIncludesCategories(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
patterns := buildBodyPatterns()
|
||||
for _, category := range []string{
|
||||
CategoryArrival,
|
||||
CategoryDeparture,
|
||||
CategoryCancellation,
|
||||
CategoryDelay,
|
||||
CategoryFlightPlan,
|
||||
} {
|
||||
config, ok := patterns[category]
|
||||
if !ok {
|
||||
t.Fatalf("expected category %s in body patterns", category)
|
||||
}
|
||||
if len(config.Patterns) == 0 || config.Patterns[0].Expression == nil {
|
||||
t.Fatalf("expected pattern expression for category %s", category)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCategoryInvalid(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := parseCategory("XYZ", ParseContext{}, map[string]string{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error for invalid category")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseCategoryFPL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
body := `(FPL-CCA1532-IS
|
||||
-A332/H
|
||||
-SDE3FGHIJ4J5M1RWY/LB101
|
||||
-ZSSS2035
|
||||
-K0859S1040 PIAKS G330 PIMOL A539 BTO W82 DOGAR
|
||||
-ZBAA0153 ZBYN
|
||||
-PBN/A1B2B3B4B5D1L1 NAV/ABAS REG/B6513 EET/ZBPE0112 SEL/KMAL PER/C RIF/FRT N640 ZBYN RMK/TCAS EQUIPPED)`
|
||||
|
||||
data := extract(body, FplPatternExpression)
|
||||
if data == nil {
|
||||
t.Fatal("expected FPL pattern to match")
|
||||
}
|
||||
|
||||
parsed, err := parseCategory(CategoryFlightPlan, ParseContext{
|
||||
Body: body,
|
||||
Tokens: Tokenizer{}.Tokenize(body),
|
||||
}, data)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected parse error: %v", err)
|
||||
}
|
||||
|
||||
fpl, ok := parsed.(*domain.FPL)
|
||||
if !ok {
|
||||
t.Fatalf("expected *domain.FPL, got %T", parsed)
|
||||
}
|
||||
|
||||
if fpl.FlightNumber != "CCA1532" {
|
||||
t.Fatalf("expected flight number CCA1532, got %s", fpl.FlightNumber)
|
||||
}
|
||||
if fpl.PBN != "A1B2B3B4B5D1L1" {
|
||||
t.Fatalf("expected PBN A1B2B3B4B5D1L1, got %s", fpl.PBN)
|
||||
}
|
||||
if fpl.RerouteInformation != "FRT N640 ZBYN" {
|
||||
t.Fatalf("expected reroute information, got %s", fpl.RerouteInformation)
|
||||
}
|
||||
if fpl.Remarks != "TCAS EQUIPPED" {
|
||||
t.Fatalf("expected remarks TCAS EQUIPPED, got %s", fpl.Remarks)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
package aviation
|
||||
|
||||
import "fmt"
|
||||
|
||||
// Validation limits based on ICAO and AFTN standards
|
||||
const (
|
||||
// MaxTelegramSize is the maximum size for an AFTN telegram (ICAO standard)
|
||||
MaxTelegramSize = 1800
|
||||
|
||||
// MaxHeaderLines is the maximum number of lines allowed in the header section
|
||||
MaxHeaderLines = 20
|
||||
|
||||
// MaxBodySize is the maximum size for the telegram body
|
||||
MaxBodySize = 1500
|
||||
|
||||
// MaxTokenCount is the maximum number of tokens allowed to prevent tokenizer abuse
|
||||
MaxTokenCount = 500
|
||||
)
|
||||
|
||||
// ValidationError represents a validation failure with field context.
|
||||
type ValidationError struct {
|
||||
Field string
|
||||
Message string
|
||||
}
|
||||
|
||||
// Error implements the error interface.
|
||||
func (e *ValidationError) Error() string {
|
||||
return fmt.Sprintf("validation error [%s]: %s", e.Field, e.Message)
|
||||
}
|
||||
|
||||
// ValidateInputSize checks if the input telegram is within acceptable size limits.
|
||||
// Returns ValidationError if the input is empty or exceeds MaxTelegramSize.
|
||||
func ValidateInputSize(rawText string) error {
|
||||
if len(rawText) == 0 {
|
||||
return &ValidationError{
|
||||
Field: "input",
|
||||
Message: "empty input",
|
||||
}
|
||||
}
|
||||
|
||||
if len(rawText) > MaxTelegramSize {
|
||||
return &ValidationError{
|
||||
Field: "input",
|
||||
Message: fmt.Sprintf("input exceeds maximum size of %d characters (got %d)", MaxTelegramSize, len(rawText)),
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// ValidateBodySize checks if the body content is within acceptable size limits.
|
||||
// Returns ValidationError if the body exceeds MaxBodySize.
|
||||
func ValidateBodySize(body string) error {
|
||||
if len(body) > MaxBodySize {
|
||||
return &ValidationError{
|
||||
Field: "body",
|
||||
Message: fmt.Sprintf("body exceeds maximum size of %d characters (got %d)", MaxBodySize, len(body)),
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ValidateTokenCount checks if the token count is within reasonable limits.
|
||||
// Returns ValidationError if token count exceeds MaxTokenCount.
|
||||
func ValidateTokenCount(tokens []Token) error {
|
||||
if len(tokens) > MaxTokenCount {
|
||||
return &ValidationError{
|
||||
Field: "tokens",
|
||||
Message: fmt.Sprintf("token count exceeds maximum of %d (got %d)", MaxTokenCount, len(tokens)),
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetRequiredField safely extracts a required field from parsed data.
|
||||
// Returns ValidationError if the field doesn't exist or is empty.
|
||||
func GetRequiredField(data map[string]string, field string) (string, error) {
|
||||
value, exists := data[field]
|
||||
if !exists {
|
||||
return "", &ValidationError{
|
||||
Field: field,
|
||||
Message: fmt.Sprintf("required field '%s' not found in parsed data", field),
|
||||
}
|
||||
}
|
||||
|
||||
if value == "" {
|
||||
return "", &ValidationError{
|
||||
Field: field,
|
||||
Message: fmt.Sprintf("required field '%s' is empty", field),
|
||||
}
|
||||
}
|
||||
|
||||
return value, nil
|
||||
}
|
||||
|
||||
// GetOptionalField safely extracts an optional field from parsed data.
|
||||
// Returns empty string if the field doesn't exist.
|
||||
func GetOptionalField(data map[string]string, field string) string {
|
||||
value, exists := data[field]
|
||||
if !exists {
|
||||
return ""
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
// SanitizeErrorForClient removes sensitive information from error messages
|
||||
// before exposing them to external clients. This prevents leaking:
|
||||
// - Raw telegram content (may contain sensitive flight data)
|
||||
// - Internal implementation details
|
||||
// - System paths or configuration
|
||||
//
|
||||
// The function preserves error type and general context while removing
|
||||
// specific content that could be sensitive.
|
||||
func SanitizeErrorForClient(err error) string {
|
||||
if err == nil {
|
||||
return ""
|
||||
}
|
||||
|
||||
// Get the error message string
|
||||
errMsg := err.Error()
|
||||
|
||||
// Truncate long error messages that might contain sensitive content
|
||||
// This applies to all error types, including ValidationError
|
||||
if len(errMsg) > 200 {
|
||||
return errMsg[:200] + "..."
|
||||
}
|
||||
|
||||
return errMsg
|
||||
}
|
||||
@@ -0,0 +1,268 @@
|
||||
package aviation
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var _ = Describe("Validation", func() {
|
||||
|
||||
Describe("ValidateInputSize", func() {
|
||||
Context("with empty input", func() {
|
||||
It("should return validation error", func() {
|
||||
err := ValidateInputSize("")
|
||||
Expect(err).To(HaveOccurred())
|
||||
|
||||
valErr, ok := err.(*ValidationError)
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(valErr.Field).To(Equal("input"))
|
||||
Expect(valErr.Message).To(ContainSubstring("empty input"))
|
||||
})
|
||||
})
|
||||
|
||||
Context("with valid input size", func() {
|
||||
It("should accept input under limit", func() {
|
||||
input := strings.Repeat("A", 1000)
|
||||
err := ValidateInputSize(input)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("should accept input at exact limit", func() {
|
||||
input := strings.Repeat("A", MaxTelegramSize)
|
||||
err := ValidateInputSize(input)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with oversized input", func() {
|
||||
It("should reject input exceeding limit", func() {
|
||||
input := strings.Repeat("A", MaxTelegramSize+1)
|
||||
err := ValidateInputSize(input)
|
||||
Expect(err).To(HaveOccurred())
|
||||
|
||||
valErr, ok := err.(*ValidationError)
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(valErr.Field).To(Equal("input"))
|
||||
Expect(valErr.Message).To(ContainSubstring("exceeds maximum size"))
|
||||
Expect(valErr.Message).To(ContainSubstring("1800"))
|
||||
})
|
||||
|
||||
It("should reject very large input", func() {
|
||||
input := strings.Repeat("A", 10000)
|
||||
err := ValidateInputSize(input)
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("ValidateBodySize", func() {
|
||||
Context("with valid body size", func() {
|
||||
It("should accept body under limit", func() {
|
||||
body := strings.Repeat("B", 1000)
|
||||
err := ValidateBodySize(body)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("should accept body at exact limit", func() {
|
||||
body := strings.Repeat("B", MaxBodySize)
|
||||
err := ValidateBodySize(body)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("should accept empty body", func() {
|
||||
err := ValidateBodySize("")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with oversized body", func() {
|
||||
It("should reject body exceeding limit", func() {
|
||||
body := strings.Repeat("B", MaxBodySize+1)
|
||||
err := ValidateBodySize(body)
|
||||
Expect(err).To(HaveOccurred())
|
||||
|
||||
valErr, ok := err.(*ValidationError)
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(valErr.Field).To(Equal("body"))
|
||||
Expect(valErr.Message).To(ContainSubstring("exceeds maximum size"))
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("ValidateTokenCount", func() {
|
||||
Context("with valid token count", func() {
|
||||
It("should accept empty token list", func() {
|
||||
tokens := []Token{}
|
||||
err := ValidateTokenCount(tokens)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("should accept token count under limit", func() {
|
||||
tokens := make([]Token, 100)
|
||||
err := ValidateTokenCount(tokens)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
It("should accept token count at exact limit", func() {
|
||||
tokens := make([]Token, MaxTokenCount)
|
||||
err := ValidateTokenCount(tokens)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with excessive token count", func() {
|
||||
It("should reject token count exceeding limit", func() {
|
||||
tokens := make([]Token, MaxTokenCount+1)
|
||||
err := ValidateTokenCount(tokens)
|
||||
Expect(err).To(HaveOccurred())
|
||||
|
||||
valErr, ok := err.(*ValidationError)
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(valErr.Field).To(Equal("tokens"))
|
||||
Expect(valErr.Message).To(ContainSubstring("exceeds maximum"))
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("GetRequiredField", func() {
|
||||
Context("with existing non-empty field", func() {
|
||||
It("should return the field value", func() {
|
||||
data := map[string]string{
|
||||
"category": "ARR",
|
||||
"number": "CES5470",
|
||||
}
|
||||
|
||||
value, err := GetRequiredField(data, "category")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(value).To(Equal("ARR"))
|
||||
})
|
||||
})
|
||||
|
||||
Context("with missing field", func() {
|
||||
It("should return validation error", func() {
|
||||
data := map[string]string{
|
||||
"category": "ARR",
|
||||
}
|
||||
|
||||
value, err := GetRequiredField(data, "number")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(value).To(Equal(""))
|
||||
|
||||
valErr, ok := err.(*ValidationError)
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(valErr.Field).To(Equal("number"))
|
||||
Expect(valErr.Message).To(ContainSubstring("not found"))
|
||||
})
|
||||
})
|
||||
|
||||
Context("with empty field value", func() {
|
||||
It("should return validation error", func() {
|
||||
data := map[string]string{
|
||||
"category": "",
|
||||
}
|
||||
|
||||
value, err := GetRequiredField(data, "category")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(value).To(Equal(""))
|
||||
|
||||
valErr, ok := err.(*ValidationError)
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(valErr.Field).To(Equal("category"))
|
||||
Expect(valErr.Message).To(ContainSubstring("is empty"))
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("GetOptionalField", func() {
|
||||
Context("with existing field", func() {
|
||||
It("should return the field value", func() {
|
||||
data := map[string]string{
|
||||
"ssr": "A1234",
|
||||
}
|
||||
|
||||
value := GetOptionalField(data, "ssr")
|
||||
Expect(value).To(Equal("A1234"))
|
||||
})
|
||||
|
||||
It("should return empty string for empty value", func() {
|
||||
data := map[string]string{
|
||||
"ssr": "",
|
||||
}
|
||||
|
||||
value := GetOptionalField(data, "ssr")
|
||||
Expect(value).To(Equal(""))
|
||||
})
|
||||
})
|
||||
|
||||
Context("with missing field", func() {
|
||||
It("should return empty string", func() {
|
||||
data := map[string]string{
|
||||
"category": "ARR",
|
||||
}
|
||||
|
||||
value := GetOptionalField(data, "ssr")
|
||||
Expect(value).To(Equal(""))
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("ValidationError", func() {
|
||||
It("should format error message correctly", func() {
|
||||
err := &ValidationError{
|
||||
Field: "test_field",
|
||||
Message: "test message",
|
||||
}
|
||||
|
||||
Expect(err.Error()).To(Equal("validation error [test_field]: test message"))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("SanitizeErrorForClient", func() {
|
||||
Context("with nil error", func() {
|
||||
It("should return empty string", func() {
|
||||
result := SanitizeErrorForClient(nil)
|
||||
Expect(result).To(Equal(""))
|
||||
})
|
||||
})
|
||||
|
||||
Context("with ValidationError", func() {
|
||||
It("should return the validation error message", func() {
|
||||
err := &ValidationError{
|
||||
Field: "input",
|
||||
Message: "empty input",
|
||||
}
|
||||
|
||||
result := SanitizeErrorForClient(err)
|
||||
Expect(result).To(Equal("validation error [input]: empty input"))
|
||||
})
|
||||
})
|
||||
|
||||
Context("with short error message", func() {
|
||||
It("should return the error message as-is", func() {
|
||||
err := &ValidationError{
|
||||
Field: "category",
|
||||
Message: "invalid format",
|
||||
}
|
||||
|
||||
result := SanitizeErrorForClient(err)
|
||||
Expect(result).To(ContainSubstring("invalid format"))
|
||||
})
|
||||
})
|
||||
|
||||
Context("with long error message containing sensitive data", func() {
|
||||
It("should truncate the message to prevent data leakage", func() {
|
||||
// Create a long error message that might contain sensitive telegram content
|
||||
sensitiveData := strings.Repeat("SENSITIVE_FLIGHT_DATA ", 20)
|
||||
err := &ValidationError{
|
||||
Field: "body",
|
||||
Message: "invalid telegram format: " + sensitiveData,
|
||||
}
|
||||
|
||||
result := SanitizeErrorForClient(err)
|
||||
// Should be truncated to 200 chars + "..."
|
||||
Expect(len(result)).To(BeNumerically("<=", 203))
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,69 @@
|
||||
package parser
|
||||
|
||||
import (
|
||||
"caatsm/internal/adapter/dto"
|
||||
"caatsm/internal/domain/weather"
|
||||
"caatsm/internal/port"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// CompositeParser combines multiple parsers (weather and aviation)
|
||||
type CompositeParser struct {
|
||||
aviationParser Parser
|
||||
weatherParser port.WeatherParser
|
||||
}
|
||||
|
||||
// NewCompositeParser creates a new composite parser
|
||||
func NewCompositeParser(aviation Parser, weather port.WeatherParser) *CompositeParser {
|
||||
return &CompositeParser{
|
||||
aviationParser: aviation,
|
||||
weatherParser: weather,
|
||||
}
|
||||
}
|
||||
|
||||
// Parse attempts to parse using multiple parsers
|
||||
func (p *CompositeParser) Parse(rawText string) (*dto.ParsedTelegram, error) {
|
||||
// 1. Try weather parser first
|
||||
if p.weatherParser != nil && p.weatherParser.CanParse(rawText) {
|
||||
wMsg, err := p.weatherParser.Parse(rawText)
|
||||
if err == nil {
|
||||
return p.weatherToTelegram(wMsg, rawText), nil
|
||||
}
|
||||
// If parsing fails, continue to aviation parser as fallback
|
||||
}
|
||||
|
||||
// 2. Try aviation parser (existing logic)
|
||||
return p.aviationParser.Parse(rawText)
|
||||
}
|
||||
|
||||
// weatherToTelegram converts a WeatherMessage to ParsedTelegram
|
||||
func (p *CompositeParser) weatherToTelegram(wMsg weather.WeatherMessage, raw string) *dto.ParsedTelegram {
|
||||
parsed := dto.NewParsedTelegram()
|
||||
parsed.Content = raw
|
||||
parsed.Body = raw
|
||||
parsed.Category = string(wMsg.Type())
|
||||
parsed.BodyData = wMsg
|
||||
parsed.Parsed = true
|
||||
parsed.Status = dto.MessageStatusParsed
|
||||
parsed.Uuid = uuid.New().String()
|
||||
parsed.ReceivedAt = time.Now()
|
||||
parsed.ParsedAt = time.Now()
|
||||
|
||||
// Extract basic information from weather message
|
||||
switch msg := wMsg.(type) {
|
||||
case *weather.Metar:
|
||||
issueTime := msg.IssueTime()
|
||||
parsed.MessageID = fmt.Sprintf("%s-%s", msg.Station(), issueTime.Format("20060102150405"))
|
||||
parsed.DateTime = issueTime.Format("060102150405")
|
||||
case *weather.Taf:
|
||||
issueTime := msg.IssueTime()
|
||||
parsed.MessageID = fmt.Sprintf("%s-%s", msg.Station(), issueTime.Format("20060102150405"))
|
||||
parsed.DateTime = issueTime.Format("060102150405")
|
||||
}
|
||||
|
||||
return parsed
|
||||
}
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
package parser
|
||||
|
||||
import (
|
||||
"caatsm/internal/adapter/dto"
|
||||
"caatsm/internal/adapter/parser/aviation"
|
||||
weatherparser "caatsm/internal/adapter/parser/weather"
|
||||
"caatsm/internal/port"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("CompositeParser", func() {
|
||||
var composite *CompositeParser
|
||||
var aviationParser Parser
|
||||
var weatherParser port.WeatherParser
|
||||
|
||||
BeforeEach(func() {
|
||||
aviationParser = aviation.NewParser()
|
||||
weatherParser = weatherparser.NewWeatherParser()
|
||||
composite = NewCompositeParser(aviationParser, weatherParser)
|
||||
})
|
||||
|
||||
Describe("Parse", func() {
|
||||
It("should route METAR to weather parser", func() {
|
||||
raw := "METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 Q1013="
|
||||
parsed, err := composite.Parse(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(parsed).ToNot(BeNil())
|
||||
Expect(parsed.Category).To(Equal("METAR"))
|
||||
Expect(parsed.Parsed).To(BeTrue())
|
||||
Expect(parsed.Status).To(Equal(dto.MessageStatusParsed))
|
||||
})
|
||||
|
||||
It("should route SPECI to weather parser", func() {
|
||||
raw := "SPECI KORD 251215Z 27015G25KT 5SM -RA BKN030 OVC050 20/18 A2992="
|
||||
parsed, err := composite.Parse(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(parsed).ToNot(BeNil())
|
||||
Expect(parsed.Category).To(Equal("SPECI"))
|
||||
Expect(parsed.Parsed).To(BeTrue())
|
||||
})
|
||||
|
||||
It("should route TAF to weather parser", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020="
|
||||
parsed, err := composite.Parse(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(parsed).ToNot(BeNil())
|
||||
Expect(parsed.Category).To(Equal("TAF"))
|
||||
Expect(parsed.Parsed).To(BeTrue())
|
||||
})
|
||||
|
||||
It("should route aviation messages to aviation parser", func() {
|
||||
raw := `ZCZC TMQ2617 142150
|
||||
GG ZBTJZPZX
|
||||
150551 ZBTJUOBK
|
||||
(FPL-OKA2861-IS
|
||||
-MA60/M-SHID/C
|
||||
-ZBTJ0030
|
||||
-K0420S0450 CG J1 FZ
|
||||
-ZSYT0100 ZSQD ZYTL
|
||||
-DOF/241215 EET/ZPKM0012 REG/B00FA PER/C)
|
||||
NNNN`
|
||||
parsed, err := composite.Parse(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(parsed).ToNot(BeNil())
|
||||
Expect(parsed.Category).To(Equal("FPL"))
|
||||
Expect(parsed.Parsed).To(BeTrue())
|
||||
})
|
||||
|
||||
It("should handle weather reports without ending =", func() {
|
||||
raw := "METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 Q1013"
|
||||
// Should fall back to aviation parser
|
||||
parsed, err := composite.Parse(raw)
|
||||
// May fail or succeed depending on aviation parser
|
||||
_ = parsed
|
||||
_ = err
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -1,73 +0,0 @@
|
||||
package parser
|
||||
|
||||
import "regexp"
|
||||
|
||||
// String constants
|
||||
const (
|
||||
StartIndicatorPrefix = "ZCZC"
|
||||
EndHeaderMarker = "."
|
||||
BeginPartMarker = "BEGIN PART"
|
||||
|
||||
Category = "category"
|
||||
CategoryArrival = "ARR"
|
||||
CategoryDeparture = "DEP"
|
||||
CategoryCancellation = "CNL"
|
||||
CategoryDelay = "DLA"
|
||||
CategoryFlightPlan = "FPL"
|
||||
|
||||
CANCELLED = "CNL"
|
||||
AirportCode = "airport"
|
||||
Date = "date"
|
||||
Task = "task"
|
||||
Index = "idx"
|
||||
FlightNumber = "number"
|
||||
Register = "reg"
|
||||
)
|
||||
|
||||
// Regular expression patterns
|
||||
const (
|
||||
AllDigitsPattern = `^(?P<dep_time>\d+)$`
|
||||
IndexPattern = `^(?P<idx>\(?L?[0-9]+\)?:?\.?)$`
|
||||
DatePattern = `^(?P<date>\d{2}\w{3})$`
|
||||
TaskPattern = `(?P<task>[A-Z]\/[A-Z])$`
|
||||
WaypointPattern = `^(SI:)?(?P<arr_time>\d{4}(\(\d{2}[A-Z]{3}\))?)?\/?(?P<airport>[A-Z]{3})\/?(?P<dep_time>\d{4}(\(\d{2}[A-Z]{3}\))?)?$`
|
||||
FlightNumberPattern = `^(?P<number>[0-9A-Z][0-9A-Z]\d{3,5}(\/\d+)*)$`
|
||||
RegisterPattern = `^(?P<reg>B\d{4})$`
|
||||
|
||||
ArrPatternString = `^\((?P<category>[A-Z]{3})-(?P<number>[A-Z0-9]+)(\/?(?P<ssr>[A-Z0-9]+))?-(?P<dep>[A-Z]{4})-(?P<arr>[A-Z]{4})(?P<arr_time>\d{4})\)$`
|
||||
DepPatternString = `^\((?P<category>[A-Z]{3})-(?P<number>[A-Z0-9]+)(\/(?P<ssr>[A-Z0-9]+))?-(?P<dep>[A-Z]{4})(?P<dep_time>\d{4})-(?P<arr>[A-Z]{4})\)$`
|
||||
FplPatternString = `\((?P<category>[A-Z]{3})-(?P<number>[A-Z]+\d+)-(?P<indicator>[A-Z]{2})\n-(?P<aircraft>[A-Z]+\d+\/?[A-Z]?)\n?-(?P<surve>.*)\n?-(?P<dep>[A-Z]{4})(?P<dep_time>\d{4})\n?-(?P<speed>[A-Z]+\d+)(?P<level>[A-Z0-9]+)\s+(?P<route>(.|\n)+)\n-(?P<dest>[A-Z]{4})(?P<estt>\d{4})\s?(?P<alter>(\s[A-Z]{4})+)\n?-([A-Z]{3}\/(?:[A-Z]{4}\d{4}\s?)+)?(?P<other>(?m)[A-Z]{3}\/(.|\n)*)\)$`
|
||||
CnlPatternString = `^\((?P<category>[A-Z]{3})-(?P<number>\w+\d+)-?(?P<dep>[A-Z]{4})?-?(?<arr>[A-Z]{4})\)$`
|
||||
DlaPatternString = `^\((?P<category>[A-Z]{3})-(?P<number>\w+\d+)-?(?P<dep>[A-Z]{4})(?P<dep_time>\d{4})?-?(?<arr>[A-Z]{4})(?<arr_time>\d{4})?\)$`
|
||||
)
|
||||
|
||||
// Compiled regular expressions
|
||||
var (
|
||||
AllDigitsExpression = regexp.MustCompile(AllDigitsPattern)
|
||||
IndexExpression = regexp.MustCompile(IndexPattern)
|
||||
TaskExpression = regexp.MustCompile(TaskPattern)
|
||||
DateExpression = regexp.MustCompile(DatePattern)
|
||||
WaypointExpression = regexp.MustCompile(WaypointPattern)
|
||||
FlightNumberExpression = regexp.MustCompile(FlightNumberPattern)
|
||||
RegisterExpression = regexp.MustCompile(RegisterPattern)
|
||||
ArrPatternExpression = regexp.MustCompile(ArrPatternString)
|
||||
DepPatternExpression = regexp.MustCompile(DepPatternString)
|
||||
FplPatternExpression = regexp.MustCompile(FplPatternString)
|
||||
CnlPatternExpression = regexp.MustCompile(CnlPatternString)
|
||||
DlaPatternExpression = regexp.MustCompile(DlaPatternString)
|
||||
BodyTypePattern = regexp.MustCompile(`^\(([A-Z]{3})(.*\n?)+\)$`)
|
||||
|
||||
categoryRegex = regexp.MustCompile(`\((?P<category>[A-Z]+)-`)
|
||||
emptyLineRemove = regexp.MustCompile(`(?m)^\s*$`)
|
||||
bodyOnly = regexp.MustCompile(`(.|\n)?(ZCZC(.|\n)*)NNNN(.|\n)?$`)
|
||||
originator = regexp.MustCompile(`(?P<originatorDateTime>[0-9]+)\s(?P<originator>[A-Z]+)`)
|
||||
navPattern = regexp.MustCompile(`(?m)NAV\/(?P<nav>\w+)`)
|
||||
remarkPattern = regexp.MustCompile(`(?s)RMK\/(?P<remark>.*)`)
|
||||
selPattern = regexp.MustCompile(`(?m)SEL\/(?P<sel>\w+)`)
|
||||
regPattern = regexp.MustCompile(`(?m)REG\/(?P<reg>[A-Z0-9]+)`)
|
||||
pbnPattern = regexp.MustCompile(`(?m)PBN\/(?P<pbn>[A-Z0-9]+)`)
|
||||
eetPattern = regexp.MustCompile(`(?s)(-?EET\/(?P<eet>(?:[A-Z]{4}\d{4}\s*)+))`)
|
||||
performancePattern = regexp.MustCompile(`(?s)-?PER\/(?P<per>\w)`)
|
||||
reroutePattern = regexp.MustCompile(`(?m)RIF\/(?P<rif>.*)[A-Z]{3}\/`)
|
||||
cancelledPattern = regexp.MustCompile(`\bCNL\b`)
|
||||
)
|
||||
@@ -1,17 +1,12 @@
|
||||
package parser
|
||||
|
||||
import "caatsm/internal/adapter/dto"
|
||||
import (
|
||||
"caatsm/internal/adapter/parser/aviation"
|
||||
"caatsm/internal/port"
|
||||
)
|
||||
|
||||
// AviationParser implements the Parser interface
|
||||
type AviationParser struct{}
|
||||
|
||||
// Parse parses a raw message string and returns a ParsedTelegram
|
||||
func (p *AviationParser) Parse(rawText string) (*dto.ParsedTelegram, error) {
|
||||
return Parse(rawText)
|
||||
// ProvideParser creates a composite parser instance that combines weather and aviation parsers.
|
||||
func ProvideParser(weatherParser port.WeatherParser) Parser {
|
||||
aviationParser := aviation.NewParser()
|
||||
return NewCompositeParser(aviationParser, weatherParser)
|
||||
}
|
||||
|
||||
// ProvideParser creates a parser instance
|
||||
func ProvideParser() Parser {
|
||||
return &AviationParser{}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package schedule
|
||||
|
||||
import "regexp"
|
||||
|
||||
// String constants
|
||||
const (
|
||||
AirportCode = "airport"
|
||||
Date = "date"
|
||||
Task = "task"
|
||||
Index = "idx"
|
||||
FlightNumber = "number"
|
||||
Register = "reg"
|
||||
ArrivalTime = "arr_time"
|
||||
DepartureTime = "dep_time"
|
||||
)
|
||||
|
||||
// Regular expression patterns
|
||||
const (
|
||||
AllDigitsPattern = `^(?P<dep_time>\d+)$`
|
||||
IndexPattern = `^(?P<idx>\(?L?[0-9]+\)?:?\.?)$`
|
||||
DatePattern = `^(?P<date>\d{2}\w{3})$`
|
||||
TaskPattern = `(?P<task>[A-Z]\/[A-Z])$`
|
||||
WaypointPattern = `^(SI:)?(?P<arr_time>\d{4}(\(\d{2}[A-Z]{3}\))?)?\/?(?P<airport>[A-Z]{3})\/?(?P<dep_time>\d{4}(\(\d{2}[A-Z]{3}\))?)?$`
|
||||
FlightNumberPattern = `^(?P<number>[0-9A-Z][0-9A-Z]\d{3,5}(\/\d+)*)$`
|
||||
RegisterPattern = `^(?P<reg>B\d{4})$`
|
||||
)
|
||||
|
||||
// Compiled regular expressions
|
||||
var (
|
||||
AllDigitsExpression = regexp.MustCompile(AllDigitsPattern)
|
||||
IndexExpression = regexp.MustCompile(IndexPattern)
|
||||
TaskExpression = regexp.MustCompile(TaskPattern)
|
||||
DateExpression = regexp.MustCompile(DatePattern)
|
||||
WaypointExpression = regexp.MustCompile(WaypointPattern)
|
||||
FlightNumberExpression = regexp.MustCompile(FlightNumberPattern)
|
||||
RegisterExpression = regexp.MustCompile(RegisterPattern)
|
||||
cancelledPattern = regexp.MustCompile(`\bCNL\b`)
|
||||
)
|
||||
@@ -0,0 +1,24 @@
|
||||
package schedule
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func extract(data string, exp *regexp.Regexp) map[string]string {
|
||||
match := exp.FindStringSubmatch(data)
|
||||
if len(match) > 0 {
|
||||
return extractData(match, exp)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func extractData(match []string, re *regexp.Regexp) map[string]string {
|
||||
data := make(map[string]string)
|
||||
for i, name := range re.SubexpNames() {
|
||||
if i != 0 && name != "" {
|
||||
data[name] = strings.TrimSpace(match[i])
|
||||
}
|
||||
}
|
||||
return data
|
||||
}
|
||||
@@ -1,20 +1,6 @@
|
||||
package parser
|
||||
package schedule
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
)
|
||||
|
||||
// BodyConfig represents the configuration for parsing message bodies.
|
||||
type BodyConfig struct {
|
||||
Patterns []PatternConfig
|
||||
}
|
||||
|
||||
// PatternConfig represents the configuration for a specific pattern.
|
||||
type PatternConfig struct {
|
||||
Pattern string
|
||||
Comments string
|
||||
Expression *regexp.Regexp
|
||||
}
|
||||
import "regexp"
|
||||
|
||||
// LineParser represents a line parser configuration.
|
||||
type LineParser struct {
|
||||
@@ -25,61 +11,11 @@ type LineParser struct {
|
||||
}
|
||||
|
||||
var (
|
||||
bodyPatterns = map[string]BodyConfig{}
|
||||
parserMap = map[string]*regexp.Regexp{}
|
||||
parserDef = &[]LineParser{}
|
||||
parserMap = map[string]*regexp.Regexp{}
|
||||
parserDef = &[]LineParser{}
|
||||
)
|
||||
|
||||
func init() {
|
||||
// Initialize body patterns.
|
||||
bodyPatterns = map[string]BodyConfig{
|
||||
"ARR": {
|
||||
Patterns: []PatternConfig{
|
||||
{
|
||||
Pattern: ArrPatternString,
|
||||
Comments: "Pattern for ARR message",
|
||||
Expression: ArrPatternExpression,
|
||||
},
|
||||
},
|
||||
},
|
||||
"DEP": {
|
||||
Patterns: []PatternConfig{
|
||||
{
|
||||
Pattern: DepPatternString,
|
||||
Comments: "Pattern for DEP message",
|
||||
Expression: DepPatternExpression,
|
||||
},
|
||||
},
|
||||
},
|
||||
"FPL": {
|
||||
Patterns: []PatternConfig{
|
||||
{
|
||||
Pattern: FplPatternString,
|
||||
Comments: "Pattern for FPL message",
|
||||
Expression: FplPatternExpression,
|
||||
},
|
||||
},
|
||||
},
|
||||
"CNL": {
|
||||
Patterns: []PatternConfig{
|
||||
{
|
||||
Pattern: CnlPatternString,
|
||||
Comments: "Pattern for CNL message",
|
||||
Expression: CnlPatternExpression,
|
||||
},
|
||||
},
|
||||
},
|
||||
"DLA": {
|
||||
Patterns: []PatternConfig{
|
||||
{
|
||||
Pattern: DlaPatternString,
|
||||
Comments: "Pattern for DLA message",
|
||||
Expression: DlaPatternExpression,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
// Initialize parser map.
|
||||
parserMap = map[string]*regexp.Regexp{
|
||||
Index: IndexExpression,
|
||||
@@ -296,32 +232,3 @@ func init() {
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// FindPatterns finds the matching body configuration based on the message body.
|
||||
func FindPatterns(messageBody string) *BodyConfig {
|
||||
if match := BodyTypePattern.FindStringSubmatch(messageBody); len(match) > 1 {
|
||||
name := match[1]
|
||||
if bodyConfig, found := bodyPatterns[name]; found {
|
||||
return &bodyConfig
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ParseBody parses the message body and returns the extracted values.
|
||||
func ParseBody(messageBody string) map[string]string {
|
||||
if body := FindPatterns(messageBody); body != nil {
|
||||
for _, pattern := range body.Patterns {
|
||||
if matches := pattern.Expression.FindStringSubmatch(messageBody); matches != nil {
|
||||
result := make(map[string]string)
|
||||
for i, name := range pattern.Expression.SubexpNames() {
|
||||
if i != 0 && name != "" {
|
||||
result[name] = matches[i]
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package parser
|
||||
package schedule
|
||||
|
||||
import (
|
||||
"caatsm/internal/domain"
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
package parser
|
||||
package schedule
|
||||
|
||||
import (
|
||||
"strings"
|
||||
@@ -0,0 +1,44 @@
|
||||
package weather
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var (
|
||||
// metarPattern matches METAR or SPECI at the start followed by station code
|
||||
metarPattern = regexp.MustCompile(`^(METAR|SPECI)\s+[A-Z0-9]{4}`)
|
||||
// tafPattern matches TAF at the start followed by station code
|
||||
tafPattern = regexp.MustCompile(`^TAF\s+[A-Z0-9]{4}`)
|
||||
)
|
||||
|
||||
// Classify identifies the weather report type from raw text
|
||||
// Returns the report type (METAR, SPECI, or TAF) and true if it's a weather report
|
||||
func Classify(raw string) (string, bool) {
|
||||
raw = strings.TrimSpace(raw)
|
||||
if raw == "" {
|
||||
return "", false
|
||||
}
|
||||
|
||||
// Check for METAR/SPECI
|
||||
if metarPattern.MatchString(raw) {
|
||||
if strings.HasPrefix(raw, "SPECI") {
|
||||
return "SPECI", true
|
||||
}
|
||||
return "METAR", true
|
||||
}
|
||||
|
||||
// Check for TAF
|
||||
if tafPattern.MatchString(raw) {
|
||||
return "TAF", true
|
||||
}
|
||||
|
||||
return "", false
|
||||
}
|
||||
|
||||
// HasValidEnding checks if the report ends with '='
|
||||
func HasValidEnding(raw string) bool {
|
||||
raw = strings.TrimSpace(raw)
|
||||
return strings.HasSuffix(raw, "=")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
package weather
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("Classifier", func() {
|
||||
Describe("Classify", func() {
|
||||
It("should identify METAR reports", func() {
|
||||
reportType, ok := Classify("METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 Q1013=")
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(reportType).To(Equal("METAR"))
|
||||
})
|
||||
|
||||
It("should identify SPECI reports", func() {
|
||||
reportType, ok := Classify("SPECI KORD 251215Z 27015G25KT 5SM -RA BKN030 OVC050 20/18 A2992=")
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(reportType).To(Equal("SPECI"))
|
||||
})
|
||||
|
||||
It("should identify TAF reports", func() {
|
||||
reportType, ok := Classify("TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020=")
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(reportType).To(Equal("TAF"))
|
||||
})
|
||||
|
||||
It("should return false for non-weather reports", func() {
|
||||
_, ok := Classify("(FPL-JAE7433-IS")
|
||||
Expect(ok).To(BeFalse())
|
||||
})
|
||||
|
||||
It("should return false for empty string", func() {
|
||||
_, ok := Classify("")
|
||||
Expect(ok).To(BeFalse())
|
||||
})
|
||||
})
|
||||
|
||||
Describe("HasValidEnding", func() {
|
||||
It("should return true for reports ending with =", func() {
|
||||
Expect(HasValidEnding("METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 Q1013=")).To(BeTrue())
|
||||
})
|
||||
|
||||
It("should return false for reports without =", func() {
|
||||
Expect(HasValidEnding("METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 Q1013")).To(BeFalse())
|
||||
})
|
||||
|
||||
It("should handle whitespace", func() {
|
||||
Expect(HasValidEnding("METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 Q1013= ")).To(BeTrue())
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
package weather
|
||||
|
||||
import (
|
||||
"caatsm/internal/domain/weather"
|
||||
"fmt"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
func parseWindAndVisibility(tokens []string) (int, *weather.Wind, *weather.Visibility) {
|
||||
pos := 0
|
||||
var wind *weather.Wind
|
||||
var visibility *weather.Visibility
|
||||
|
||||
if pos < len(tokens) {
|
||||
if parsed := parseWind(tokens[pos]); parsed != nil {
|
||||
wind = parsed
|
||||
pos++
|
||||
|
||||
if pos < len(tokens) {
|
||||
if match := variableWindPattern.FindStringSubmatch(tokens[pos]); match != nil {
|
||||
from, _ := strconv.Atoi(match[1])
|
||||
to, _ := strconv.Atoi(match[2])
|
||||
wind.Variable = true
|
||||
wind.VariableFrom = from
|
||||
wind.VariableTo = to
|
||||
pos++
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if pos < len(tokens) {
|
||||
if parsed := parseVisibility(tokens[pos]); parsed != nil {
|
||||
visibility = parsed
|
||||
pos++
|
||||
}
|
||||
}
|
||||
|
||||
return pos, wind, visibility
|
||||
}
|
||||
|
||||
func parsePhenomena(tokens []string) (int, []weather.Phenomenon) {
|
||||
pos := 0
|
||||
var phenomena []weather.Phenomenon
|
||||
|
||||
for pos < len(tokens) {
|
||||
if match := phenomenonPattern.FindStringSubmatch(tokens[pos]); match != nil {
|
||||
phenomena = append(phenomena, weather.Phenomenon{
|
||||
Intensity: match[1],
|
||||
Descriptor: match[2],
|
||||
Weather: match[3],
|
||||
})
|
||||
pos++
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
return pos, phenomena
|
||||
}
|
||||
|
||||
func parseClouds(tokens []string) (int, []weather.Cloud) {
|
||||
pos := 0
|
||||
var clouds []weather.Cloud
|
||||
|
||||
if pos < len(tokens) {
|
||||
if skyClearPattern.MatchString(tokens[pos]) {
|
||||
return 1, clouds
|
||||
}
|
||||
}
|
||||
|
||||
for pos < len(tokens) {
|
||||
if match := cloudPattern.FindStringSubmatch(tokens[pos]); match != nil {
|
||||
alt, _ := strconv.Atoi(match[2])
|
||||
clouds = append(clouds, weather.Cloud{
|
||||
Type: match[1],
|
||||
Altitude: alt * 100,
|
||||
Modifier: match[3],
|
||||
})
|
||||
pos++
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
return pos, clouds
|
||||
}
|
||||
|
||||
func appendPeriodWarnings(warnings *[]string, tokens []string) {
|
||||
if warnings == nil {
|
||||
return
|
||||
}
|
||||
for _, token := range tokens {
|
||||
*warnings = append(*warnings, fmt.Sprintf("unrecognized token in period: %s", token))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package weather
|
||||
|
||||
import "strings"
|
||||
|
||||
// Tokenize splits the raw text into tokens by whitespace
|
||||
func Tokenize(raw string) []string {
|
||||
raw = strings.TrimSpace(raw)
|
||||
// Remove trailing '=' if present
|
||||
if strings.HasSuffix(raw, "=") {
|
||||
raw = raw[:len(raw)-1]
|
||||
raw = strings.TrimSpace(raw)
|
||||
}
|
||||
|
||||
tokens := strings.Fields(raw)
|
||||
return tokens
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package weather
|
||||
|
||||
import (
|
||||
"caatsm/internal/domain/weather"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// parseMetar parses a METAR or SPECI report
|
||||
func parseMetar(raw string, reportType string) (*weather.Metar, error) {
|
||||
tokens := Tokenize(raw)
|
||||
if len(tokens) < 3 {
|
||||
return nil, weather.ErrInvalidFormat
|
||||
}
|
||||
|
||||
metar, pos, err := parseMetarHeader(tokens, raw, reportType)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Parse runway visual range (RVR) - skip for now, add to warnings
|
||||
for pos < len(tokens) && strings.HasPrefix(tokens[pos], "R") {
|
||||
metar.Warnings = append(metar.Warnings, fmt.Sprintf("RVR not parsed: %s", tokens[pos]))
|
||||
pos++
|
||||
}
|
||||
|
||||
phenomConsumed, phenomena := parsePhenomena(tokens[pos:])
|
||||
metar.Phenomena = append(metar.Phenomena, phenomena...)
|
||||
pos += phenomConsumed
|
||||
|
||||
cloudConsumed, clouds := parseClouds(tokens[pos:])
|
||||
metar.Clouds = append(metar.Clouds, clouds...)
|
||||
pos += cloudConsumed
|
||||
|
||||
// Parse temperature/dewpoint
|
||||
if pos < len(tokens) {
|
||||
if match := tempPattern.FindStringSubmatch(tokens[pos]); match != nil {
|
||||
tempVal, _ := strconv.ParseFloat(match[2], 64)
|
||||
if match[1] == "M" {
|
||||
tempVal = -tempVal
|
||||
}
|
||||
|
||||
dewVal, _ := strconv.ParseFloat(match[4], 64)
|
||||
if match[3] == "M" {
|
||||
dewVal = -dewVal
|
||||
}
|
||||
|
||||
metar.Temperature = &weather.Temperature{
|
||||
Value: tempVal,
|
||||
Unit: "C",
|
||||
}
|
||||
metar.Dewpoint = &weather.Temperature{
|
||||
Value: dewVal,
|
||||
Unit: "C",
|
||||
}
|
||||
pos++
|
||||
}
|
||||
}
|
||||
|
||||
// Parse altimeter
|
||||
if pos < len(tokens) {
|
||||
if match := altimeterPattern.FindStringSubmatch(tokens[pos]); match != nil {
|
||||
value, _ := strconv.ParseFloat(match[2], 64)
|
||||
unit := match[1]
|
||||
|
||||
switch unit {
|
||||
case "Q":
|
||||
// QNH in hPa
|
||||
metar.Altimeter = &weather.Altimeter{
|
||||
Value: value,
|
||||
Unit: "QNH",
|
||||
}
|
||||
case "A":
|
||||
// Altimeter in inHg
|
||||
metar.Altimeter = &weather.Altimeter{
|
||||
Value: value / 100.0, // A2992 means 29.92 inHg
|
||||
Unit: "A",
|
||||
}
|
||||
}
|
||||
pos++
|
||||
}
|
||||
}
|
||||
|
||||
// Parse remarks (everything after RMK)
|
||||
remarksStart := -1
|
||||
for i := pos; i < len(tokens); i++ {
|
||||
if strings.HasPrefix(strings.ToUpper(tokens[i]), "RMK") {
|
||||
remarksStart = i
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if remarksStart >= 0 {
|
||||
metar.Remarks = strings.Join(tokens[remarksStart:], " ")
|
||||
pos = len(tokens) // Skip remaining tokens
|
||||
}
|
||||
|
||||
// Collect any remaining unrecognized tokens as warnings
|
||||
for pos < len(tokens) {
|
||||
metar.Warnings = append(metar.Warnings, fmt.Sprintf("unrecognized token: %s", tokens[pos]))
|
||||
pos++
|
||||
}
|
||||
|
||||
return metar, nil
|
||||
}
|
||||
|
||||
func parseMetarHeader(tokens []string, raw string, reportType string) (*weather.Metar, int, error) {
|
||||
metar := &weather.Metar{
|
||||
ReportType: weather.ReportType(reportType),
|
||||
RawTextVal: raw,
|
||||
Warnings: []string{},
|
||||
Clouds: []weather.Cloud{},
|
||||
Phenomena: []weather.Phenomenon{},
|
||||
}
|
||||
|
||||
pos := 0
|
||||
if pos >= len(tokens) {
|
||||
return nil, 0, weather.ErrMissingStation
|
||||
}
|
||||
pos++
|
||||
|
||||
if pos >= len(tokens) {
|
||||
return nil, 0, weather.ErrMissingStation
|
||||
}
|
||||
metar.StationID = tokens[pos]
|
||||
pos++
|
||||
|
||||
if pos >= len(tokens) {
|
||||
return nil, 0, weather.ErrMissingTime
|
||||
}
|
||||
issueTime, err := ParseTime(tokens[pos])
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("failed to parse issue time: %w", err)
|
||||
}
|
||||
metar.IssueTimeVal = issueTime
|
||||
metar.ObsTime = issueTime
|
||||
pos++
|
||||
|
||||
if pos < len(tokens) {
|
||||
if match := modifierPattern.FindStringSubmatch(tokens[pos]); match != nil {
|
||||
metar.Modifier = match[1]
|
||||
pos++
|
||||
}
|
||||
}
|
||||
|
||||
consumed, wind, visibility := parseWindAndVisibility(tokens[pos:])
|
||||
metar.Wind = wind
|
||||
metar.Visibility = visibility
|
||||
pos += consumed
|
||||
|
||||
return metar, pos, nil
|
||||
}
|
||||
|
||||
// parseWind parses wind information
|
||||
func parseWind(token string) *weather.Wind {
|
||||
match := windPattern.FindStringSubmatch(token)
|
||||
if len(match) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
wind := &weather.Wind{}
|
||||
|
||||
// Parse direction
|
||||
if match[1] == "VRB" {
|
||||
wind.Variable = true
|
||||
wind.Direction = 0
|
||||
} else {
|
||||
dir, _ := strconv.Atoi(match[1])
|
||||
wind.Direction = dir
|
||||
}
|
||||
|
||||
// Parse speed
|
||||
speed, _ := strconv.Atoi(match[2])
|
||||
wind.Speed = speed
|
||||
|
||||
// Parse gust
|
||||
if match[3] != "" {
|
||||
gust, _ := strconv.Atoi(match[4])
|
||||
wind.Gust = gust
|
||||
}
|
||||
|
||||
// Parse unit
|
||||
wind.Unit = match[5]
|
||||
if wind.Unit == "" {
|
||||
wind.Unit = "KT" // Default to knots
|
||||
}
|
||||
|
||||
return wind
|
||||
}
|
||||
|
||||
// parseVisibility parses visibility information
|
||||
func parseVisibility(token string) *weather.Visibility {
|
||||
// Try directional visibility first
|
||||
if match := directionalVisibilityPattern.FindStringSubmatch(token); match != nil {
|
||||
dist, _ := strconv.ParseFloat(match[1], 64)
|
||||
return &weather.Visibility{
|
||||
Distance: dist,
|
||||
Unit: "M",
|
||||
Direction: match[2],
|
||||
}
|
||||
}
|
||||
|
||||
// Try standard visibility pattern
|
||||
match := visibilityPattern.FindStringSubmatch(token)
|
||||
if len(match) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
vis := &weather.Visibility{
|
||||
Modifier: match[1],
|
||||
Unit: match[3],
|
||||
}
|
||||
|
||||
// Parse distance
|
||||
distStr := match[2]
|
||||
if strings.Contains(distStr, "/") {
|
||||
// Fractional visibility (e.g., "1/4SM")
|
||||
dist, err := ParseFraction(distStr)
|
||||
if err == nil {
|
||||
vis.Distance = dist
|
||||
} else {
|
||||
return nil
|
||||
}
|
||||
} else {
|
||||
dist, err := strconv.ParseFloat(distStr, 64)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
vis.Distance = dist
|
||||
}
|
||||
|
||||
// Default unit
|
||||
if vis.Unit == "" {
|
||||
if vis.Distance >= 10 {
|
||||
vis.Unit = "M" // Meters (e.g., 9999)
|
||||
} else {
|
||||
vis.Unit = "SM" // Statute miles
|
||||
}
|
||||
}
|
||||
|
||||
return vis
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
package weather
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("METAR Parser", func() {
|
||||
Describe("parseMetar", func() {
|
||||
It("should parse a standard METAR", func() {
|
||||
raw := "METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 Q1013="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(metar).ToNot(BeNil())
|
||||
Expect(metar.StationID).To(Equal("KJFK"))
|
||||
Expect(metar.Wind).ToNot(BeNil())
|
||||
Expect(metar.Wind.Direction).To(Equal(350))
|
||||
Expect(metar.Wind.Speed).To(Equal(12))
|
||||
Expect(metar.Visibility).ToNot(BeNil())
|
||||
Expect(metar.Visibility.Distance).To(Equal(10.0))
|
||||
Expect(metar.Visibility.Unit).To(Equal("SM"))
|
||||
Expect(len(metar.Clouds)).To(Equal(1))
|
||||
Expect(metar.Clouds[0].Type).To(Equal("FEW"))
|
||||
Expect(metar.Temperature).ToNot(BeNil())
|
||||
Expect(metar.Temperature.Value).To(Equal(25.0))
|
||||
Expect(metar.Dewpoint).ToNot(BeNil())
|
||||
Expect(metar.Dewpoint.Value).To(Equal(18.0))
|
||||
Expect(metar.Altimeter).ToNot(BeNil())
|
||||
Expect(metar.Altimeter.Value).To(Equal(1013.0))
|
||||
})
|
||||
|
||||
It("should parse METAR with variable wind", func() {
|
||||
raw := "METAR KORD 251200Z VRB05KT 10SM CLR 20/15 Q1013="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(metar.Wind).ToNot(BeNil())
|
||||
Expect(metar.Wind.Variable).To(BeTrue())
|
||||
})
|
||||
|
||||
It("should parse METAR with gust", func() {
|
||||
raw := "METAR KJFK 251200Z 27015G25KT 10SM FEW020 25/18 Q1013="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(metar.Wind).ToNot(BeNil())
|
||||
Expect(metar.Wind.Gust).To(Equal(25))
|
||||
})
|
||||
|
||||
It("should parse METAR with AUTO modifier", func() {
|
||||
raw := "METAR KJFK 251200Z AUTO 35012KT 10SM FEW020 25/18 Q1013="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(metar.Modifier).To(Equal("AUTO"))
|
||||
})
|
||||
|
||||
It("should parse METAR with COR modifier", func() {
|
||||
raw := "METAR KJFK 251200Z COR 35012KT 10SM FEW020 25/18 Q1013="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(metar.Modifier).To(Equal("COR"))
|
||||
})
|
||||
|
||||
It("should parse METAR with weather phenomena", func() {
|
||||
raw := "METAR KJFK 251200Z 35012KT 5SM -RA BKN030 OVC050 20/18 Q1013="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(len(metar.Phenomena)).To(BeNumerically(">", 0))
|
||||
Expect(metar.Phenomena[0].Intensity).To(Equal("-"))
|
||||
Expect(metar.Phenomena[0].Weather).To(Equal("RA"))
|
||||
})
|
||||
|
||||
It("should parse METAR with multiple clouds", func() {
|
||||
raw := "METAR KJFK 251200Z 35012KT 10SM FEW020 SCT030 BKN100 25/18 Q1013="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(len(metar.Clouds)).To(Equal(3))
|
||||
})
|
||||
|
||||
It("should parse METAR with CB clouds", func() {
|
||||
raw := "METAR KJFK 251200Z 35012KT 10SM SCT030CB 25/18 Q1013="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(len(metar.Clouds)).To(Equal(1))
|
||||
Expect(metar.Clouds[0].Modifier).To(Equal("CB"))
|
||||
})
|
||||
|
||||
It("should parse METAR with altimeter in inHg", func() {
|
||||
raw := "METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 A2992="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(metar.Altimeter).ToNot(BeNil())
|
||||
Expect(metar.Altimeter.Unit).To(Equal("A"))
|
||||
Expect(metar.Altimeter.Value).To(Equal(29.92))
|
||||
})
|
||||
|
||||
It("should parse METAR with negative temperature", func() {
|
||||
raw := "METAR KJFK 251200Z 35012KT 10SM FEW020 M05/M10 Q1013="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(metar.Temperature).ToNot(BeNil())
|
||||
Expect(metar.Temperature.Value).To(Equal(-5.0))
|
||||
Expect(metar.Dewpoint).ToNot(BeNil())
|
||||
Expect(metar.Dewpoint.Value).To(Equal(-10.0))
|
||||
})
|
||||
|
||||
It("should parse METAR with remarks", func() {
|
||||
raw := "METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 Q1013 RMK TEST REMARKS="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(metar.Remarks).To(ContainSubstring("RMK"))
|
||||
})
|
||||
|
||||
It("should handle unrecognized tokens as warnings", func() {
|
||||
raw := "METAR KJFK 251200Z 35012KT 10SM FEW020 25/18 Q1013 UNKNOWN TOKEN="
|
||||
metar, err := parseMetar(raw, "METAR")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(len(metar.Warnings)).To(BeNumerically(">", 0))
|
||||
})
|
||||
|
||||
It("should parse SPECI reports", func() {
|
||||
raw := "SPECI KORD 251215Z 27015G25KT 5SM -RA BKN030 OVC050 20/18 A2992="
|
||||
metar, err := parseMetar(raw, "SPECI")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(string(metar.ReportType)).To(Equal("SPECI"))
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
package weather
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ConvertKTToMPS converts knots to meters per second
|
||||
func ConvertKTToMPS(knots int) int {
|
||||
// 1 knot = 0.514444 m/s
|
||||
return int(float64(knots) * 0.514444)
|
||||
}
|
||||
|
||||
// ConvertMPSToKT converts meters per second to knots
|
||||
func ConvertMPSToKT(mps int) int {
|
||||
// 1 m/s = 1.94384 knots
|
||||
return int(float64(mps) * 1.94384)
|
||||
}
|
||||
|
||||
// ConvertSMToMeters converts statute miles to meters
|
||||
func ConvertSMToMeters(sm float64) float64 {
|
||||
// 1 SM = 1609.34 meters
|
||||
return sm * 1609.34
|
||||
}
|
||||
|
||||
// ConvertMetersToSM converts meters to statute miles
|
||||
func ConvertMetersToSM(meters float64) float64 {
|
||||
// 1 meter = 0.000621371 SM
|
||||
return meters * 0.000621371
|
||||
}
|
||||
|
||||
// ConvertInHgToHPa converts inches of mercury to hectopascals
|
||||
func ConvertInHgToHPa(inHg float64) float64 {
|
||||
// 1 inHg = 33.8639 hPa
|
||||
return inHg * 33.8639
|
||||
}
|
||||
|
||||
// ConvertHPatoInHg converts hectopascals to inches of mercury
|
||||
func ConvertHPatoInHg(hPa float64) float64 {
|
||||
// 1 hPa = 0.0295299 inHg
|
||||
return hPa * 0.0295299
|
||||
}
|
||||
|
||||
// ParseTime parses a time string in DDHHmmZ format to time.Time
|
||||
// Uses the current year/month as reference
|
||||
func ParseTime(timeStr string) (time.Time, error) {
|
||||
match := timePattern.FindStringSubmatch(timeStr)
|
||||
if len(match) == 0 {
|
||||
return time.Time{}, fmt.Errorf("invalid time format: %s", timeStr)
|
||||
}
|
||||
|
||||
day, err := strconv.Atoi(match[1])
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("invalid day: %w", err)
|
||||
}
|
||||
|
||||
hour, err := strconv.Atoi(match[2])
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("invalid hour: %w", err)
|
||||
}
|
||||
|
||||
min, err := strconv.Atoi(match[3])
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("invalid minute: %w", err)
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
// Use current year and month, but adjust if day is in the future (likely next month)
|
||||
t := time.Date(now.Year(), now.Month(), day, hour, min, 0, 0, time.UTC)
|
||||
|
||||
// If the day is significantly in the past (more than 15 days), assume next month
|
||||
if day < now.Day()-15 {
|
||||
t = t.AddDate(0, 1, 0)
|
||||
}
|
||||
|
||||
return t, nil
|
||||
}
|
||||
|
||||
// ParseTAFValidity parses TAF validity period in DDHH/DDHH format
|
||||
func ParseTAFValidity(validityStr string, issueTime time.Time) (time.Time, time.Time, error) {
|
||||
match := tafValidityPattern.FindStringSubmatch(validityStr)
|
||||
if len(match) == 0 {
|
||||
return time.Time{}, time.Time{}, fmt.Errorf("invalid TAF validity format: %s", validityStr)
|
||||
}
|
||||
|
||||
fromDay, _ := strconv.Atoi(match[1])
|
||||
fromHour, _ := strconv.Atoi(match[2])
|
||||
toDay, _ := strconv.Atoi(match[3])
|
||||
toHour, _ := strconv.Atoi(match[4])
|
||||
|
||||
year := issueTime.Year()
|
||||
month := issueTime.Month()
|
||||
|
||||
fromTime := time.Date(year, month, fromDay, fromHour, 0, 0, 0, time.UTC)
|
||||
toTime := time.Date(year, month, toDay, toHour, 0, 0, 0, time.UTC)
|
||||
|
||||
// If toDay is less than fromDay, assume next month
|
||||
if toDay < fromDay {
|
||||
toTime = toTime.AddDate(0, 1, 0)
|
||||
}
|
||||
|
||||
return fromTime, toTime, nil
|
||||
}
|
||||
|
||||
// ParseFraction parses a fraction string like "1/4" or "1 1/2"
|
||||
func ParseFraction(fracStr string) (float64, error) {
|
||||
fracStr = strings.TrimSpace(fracStr)
|
||||
|
||||
// Handle whole number with fraction: "1 1/2"
|
||||
if strings.Contains(fracStr, " ") {
|
||||
parts := strings.Fields(fracStr)
|
||||
if len(parts) != 2 {
|
||||
return 0, fmt.Errorf("invalid fraction format: %s", fracStr)
|
||||
}
|
||||
|
||||
whole, err := strconv.ParseFloat(parts[0], 64)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("invalid whole number: %w", err)
|
||||
}
|
||||
|
||||
frac, err := parseSimpleFraction(parts[1])
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return whole + frac, nil
|
||||
}
|
||||
|
||||
// Handle simple fraction: "1/4"
|
||||
return parseSimpleFraction(fracStr)
|
||||
}
|
||||
|
||||
func parseSimpleFraction(fracStr string) (float64, error) {
|
||||
parts := strings.Split(fracStr, "/")
|
||||
if len(parts) != 2 {
|
||||
return 0, fmt.Errorf("invalid fraction format: %s", fracStr)
|
||||
}
|
||||
|
||||
numerator, err := strconv.ParseFloat(parts[0], 64)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("invalid numerator: %w", err)
|
||||
}
|
||||
|
||||
denominator, err := strconv.ParseFloat(parts[1], 64)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("invalid denominator: %w", err)
|
||||
}
|
||||
|
||||
if denominator == 0 {
|
||||
return 0, fmt.Errorf("division by zero")
|
||||
}
|
||||
|
||||
return numerator / denominator, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package weather
|
||||
|
||||
import "regexp"
|
||||
|
||||
var (
|
||||
// Wind patterns: 35012KT, VRB05KT, 27015G25KT, 00000KT
|
||||
windPattern = regexp.MustCompile(`^(?P<dir>\d{3}|VRB)(?P<speed>\d{2,3})(G(?P<gust>\d{2,3}))?(?P<unit>KT|MPS)$`)
|
||||
|
||||
// Variable wind: 180V240 (variable from 180 to 240 degrees)
|
||||
variableWindPattern = regexp.MustCompile(`^(?P<from>\d{3})V(?P<to>\d{3})$`)
|
||||
|
||||
// Visibility patterns: 9999, 10SM, M1/4SM, 1 1/2SM, 1500
|
||||
visibilityPattern = regexp.MustCompile(`^(?P<modifier>[MP\+\-]?)(?P<dist>\d+(?:\s*\d+/\d+)?)(?P<unit>SM|M)?$`)
|
||||
|
||||
// Directional visibility: 2000NE (visibility in a specific direction)
|
||||
directionalVisibilityPattern = regexp.MustCompile(`^(?P<dist>\d{4})(?P<dir>[NSEW]{1,2})$`)
|
||||
|
||||
// Cloud patterns: FEW020, SCT030CB, BKN100, OVC200, VV010, SKC, CLR, NSC
|
||||
cloudPattern = regexp.MustCompile(`^(?P<type>FEW|SCT|BKN|OVC|VV)(?P<alt>\d{3})(?P<modifier>CB|TCU)?$`)
|
||||
|
||||
// Special cloud codes: SKC (sky clear), CLR (clear), NSC (no significant clouds)
|
||||
skyClearPattern = regexp.MustCompile(`^(SKC|CLR|NSC)$`)
|
||||
|
||||
// Temperature/Dewpoint: 25/18, M05/M10, XX/XX
|
||||
tempPattern = regexp.MustCompile(`^(?P<temp_mod>M?)(?P<temp>\d{2})/(?P<dew_mod>M?)(?P<dew>\d{2})$`)
|
||||
|
||||
// Altimeter patterns: Q1013 (hPa), A2992 (inHg)
|
||||
altimeterPattern = regexp.MustCompile(`^(?P<unit>[QA])(?P<value>\d{4})$`)
|
||||
|
||||
// Weather phenomenon patterns: -RA, +SN, TSRA, FZFG, BR, FG, etc.
|
||||
// Intensity: -, +, or empty
|
||||
// Descriptors: MI, BC, PR, DR, BL, SH, TS, FZ, DZ, RA, SN, SG, IC, PL, GR, GS, UP, BR, FG, FU, VA, DU, SA, HZ, PY, PO, SQ, FC, SS, DS
|
||||
// Weather: DZ, RA, SN, SG, IC, PL, GR, GS, UP, BR, FG, FU, VA, DU, SA, HZ, PY, PO, SQ, FC, SS, DS
|
||||
phenomenonPattern = regexp.MustCompile(`^(?P<intensity>[\+\-])?(?P<descriptor>MI|BC|PR|DR|BL|SH|TS|FZ|DZ|RA|SN|SG|IC|PL|GR|GS|UP|BR|FG|FU|VA|DU|SA|HZ|PY|PO|SQ|FC|SS|DS)?(?P<weather>DZ|RA|SN|SG|IC|PL|GR|GS|UP|BR|FG|FU|VA|DU|SA|HZ|PY|PO|SQ|FC|SS|DS)+$`)
|
||||
|
||||
// Time pattern: 251200Z (DDHHmmZ format)
|
||||
timePattern = regexp.MustCompile(`^(?P<day>\d{2})(?P<hour>\d{2})(?P<min>\d{2})Z$`)
|
||||
|
||||
// TAF validity period: 2512/2612 (DDHH/DDHH format)
|
||||
tafValidityPattern = regexp.MustCompile(`^(?P<from_day>\d{2})(?P<from_hour>\d{2})/(?P<to_day>\d{2})(?P<to_hour>\d{2})$`)
|
||||
|
||||
// TAF period markers: FM251200, TEMPO2512/2515, BECMG2512/2515
|
||||
tafFMPattern = regexp.MustCompile(`^FM(?P<day>\d{2})(?P<hour>\d{2})(?P<min>\d{2})$`)
|
||||
tafTEMPOPattern = regexp.MustCompile(`^TEMPO(?P<from_day>\d{2})(?P<from_hour>\d{2})/(?P<to_day>\d{2})(?P<to_hour>\d{2})$`)
|
||||
tafBECMGPattern = regexp.MustCompile(`^BECMG(?P<from_day>\d{2})(?P<from_hour>\d{2})/(?P<to_day>\d{2})(?P<to_hour>\d{2})$`)
|
||||
|
||||
// Modifiers: AUTO, COR, NIL, etc.
|
||||
modifierPattern = regexp.MustCompile(`^(AUTO|COR|NIL)$`)
|
||||
)
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
package weather
|
||||
|
||||
import (
|
||||
"caatsm/internal/domain/weather"
|
||||
"caatsm/internal/port"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// WeatherParserImpl implements the WeatherParser interface
|
||||
type WeatherParserImpl struct{}
|
||||
|
||||
// NewWeatherParser creates a new weather parser instance
|
||||
func NewWeatherParser() port.WeatherParser {
|
||||
return &WeatherParserImpl{}
|
||||
}
|
||||
|
||||
// CanParse determines if the raw string can be parsed as a weather report
|
||||
func (p *WeatherParserImpl) CanParse(raw string) bool {
|
||||
reportType, ok := Classify(raw)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
|
||||
// Check for valid ending
|
||||
if !HasValidEnding(raw) {
|
||||
return false
|
||||
}
|
||||
|
||||
_ = reportType // Suppress unused variable warning
|
||||
return true
|
||||
}
|
||||
|
||||
// Parse parses a raw weather report string and returns a WeatherMessage
|
||||
func (p *WeatherParserImpl) Parse(raw string) (weather.WeatherMessage, error) {
|
||||
reportType, ok := Classify(raw)
|
||||
if !ok {
|
||||
return nil, weather.ErrInvalidFormat
|
||||
}
|
||||
|
||||
switch reportType {
|
||||
case "METAR", "SPECI":
|
||||
return parseMetar(raw, reportType)
|
||||
case "TAF":
|
||||
return parseTaf(raw)
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported report type: %s", reportType)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
package weather
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestWeather(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Weather Parser Suite")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
package weather
|
||||
|
||||
import (
|
||||
"caatsm/internal/domain/weather"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// parseTaf parses a TAF report
|
||||
func parseTaf(raw string) (*weather.Taf, error) {
|
||||
tokens := Tokenize(raw)
|
||||
if len(tokens) < 4 {
|
||||
return nil, weather.ErrInvalidFormat
|
||||
}
|
||||
|
||||
taf, pos, err := parseTafHeader(tokens, raw)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Parse main forecast period (before any FM/TEMPO/BECMG)
|
||||
mainPeriod := weather.TafPeriod{
|
||||
Type: "MAIN",
|
||||
ValidFrom: taf.ValidFrom,
|
||||
ValidTo: taf.ValidTo,
|
||||
}
|
||||
|
||||
// Find first special section
|
||||
firstSpecialIdx := len(tokens)
|
||||
for i := pos; i < len(tokens); i++ {
|
||||
upperToken := strings.ToUpper(tokens[i])
|
||||
if strings.HasPrefix(upperToken, "FM") ||
|
||||
strings.HasPrefix(upperToken, "TEMPO") ||
|
||||
strings.HasPrefix(upperToken, "BECMG") ||
|
||||
strings.HasPrefix(upperToken, "PROB") ||
|
||||
strings.HasPrefix(upperToken, "RMK") {
|
||||
firstSpecialIdx = i
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Parse main period tokens
|
||||
if firstSpecialIdx > pos {
|
||||
if firstSpecialIdx < len(tokens) {
|
||||
upperToken := strings.ToUpper(tokens[firstSpecialIdx])
|
||||
if strings.HasPrefix(upperToken, "FM") {
|
||||
if match := tafFMPattern.FindStringSubmatch(tokens[firstSpecialIdx]); match != nil {
|
||||
day, _ := strconv.Atoi(match[1])
|
||||
hour, _ := strconv.Atoi(match[2])
|
||||
min, _ := strconv.Atoi(match[3])
|
||||
fmStart := resolveDayTime(taf.ValidFrom, day, hour, min)
|
||||
if fmStart.Before(mainPeriod.ValidTo) {
|
||||
mainPeriod.ValidTo = fmStart
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
mainTokens := tokens[pos:firstSpecialIdx]
|
||||
parsePeriodElements(mainTokens, &mainPeriod, &taf.Warnings)
|
||||
taf.Periods = append(taf.Periods, mainPeriod)
|
||||
pos = firstSpecialIdx
|
||||
}
|
||||
|
||||
// Parse special sections (FM, TEMPO, BECMG)
|
||||
var pendingProb int
|
||||
for pos < len(tokens) {
|
||||
upperToken := strings.ToUpper(tokens[pos])
|
||||
|
||||
switch {
|
||||
case strings.HasPrefix(upperToken, "PROB"):
|
||||
// PROB30 or PROB40 - probability for next period
|
||||
probStr := strings.TrimPrefix(upperToken, "PROB")
|
||||
pendingProb, _ = strconv.Atoi(probStr)
|
||||
pos++
|
||||
|
||||
case strings.HasPrefix(upperToken, "FM"):
|
||||
period, newPos, err := parseFMPeriod(tokens, pos, taf.ValidFrom, taf.ValidTo, &taf.Warnings)
|
||||
if err != nil {
|
||||
taf.Warnings = append(taf.Warnings, fmt.Sprintf("failed to parse FM period: %v", err))
|
||||
pos++
|
||||
continue
|
||||
}
|
||||
if pendingProb > 0 {
|
||||
period.Probability = pendingProb
|
||||
pendingProb = 0
|
||||
}
|
||||
taf.Periods = append(taf.Periods, period)
|
||||
pos = newPos
|
||||
|
||||
case strings.HasPrefix(upperToken, "TEMPO"):
|
||||
period, newPos, err := parseTEMPOPeriod(tokens, pos, taf.ValidFrom, &taf.Warnings)
|
||||
if err != nil {
|
||||
taf.Warnings = append(taf.Warnings, fmt.Sprintf("failed to parse TEMPO period: %v", err))
|
||||
pos++
|
||||
continue
|
||||
}
|
||||
if pendingProb > 0 {
|
||||
period.Probability = pendingProb
|
||||
pendingProb = 0
|
||||
}
|
||||
taf.Periods = append(taf.Periods, period)
|
||||
pos = newPos
|
||||
|
||||
case strings.HasPrefix(upperToken, "BECMG"):
|
||||
period, newPos, err := parseBECMGPeriod(tokens, pos, taf.ValidFrom, &taf.Warnings)
|
||||
if err != nil {
|
||||
taf.Warnings = append(taf.Warnings, fmt.Sprintf("failed to parse BECMG period: %v", err))
|
||||
pos++
|
||||
continue
|
||||
}
|
||||
if pendingProb > 0 {
|
||||
period.Probability = pendingProb
|
||||
pendingProb = 0
|
||||
}
|
||||
taf.Periods = append(taf.Periods, period)
|
||||
pos = newPos
|
||||
|
||||
case strings.HasPrefix(upperToken, "RMK"):
|
||||
// Remarks section - include RMK token and all following tokens
|
||||
taf.Remarks = strings.Join(tokens[pos:], " ")
|
||||
// Set pos to exit the loop
|
||||
pos = len(tokens)
|
||||
|
||||
default:
|
||||
// Unrecognized token
|
||||
taf.Warnings = append(taf.Warnings, fmt.Sprintf("unrecognized token: %s", tokens[pos]))
|
||||
pos++
|
||||
}
|
||||
}
|
||||
|
||||
return taf, nil
|
||||
}
|
||||
|
||||
func parseTafHeader(tokens []string, raw string) (*weather.Taf, int, error) {
|
||||
taf := &weather.Taf{
|
||||
ReportType: weather.ReportTypeTAF,
|
||||
RawTextVal: raw,
|
||||
Warnings: []string{},
|
||||
Periods: []weather.TafPeriod{},
|
||||
}
|
||||
|
||||
pos := 0
|
||||
if pos >= len(tokens) {
|
||||
return nil, 0, weather.ErrMissingStation
|
||||
}
|
||||
pos++
|
||||
|
||||
if pos >= len(tokens) {
|
||||
return nil, 0, weather.ErrMissingStation
|
||||
}
|
||||
taf.StationID = tokens[pos]
|
||||
pos++
|
||||
|
||||
if pos >= len(tokens) {
|
||||
return nil, 0, weather.ErrMissingTime
|
||||
}
|
||||
issueTime, err := ParseTime(tokens[pos])
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("failed to parse issue time: %w", err)
|
||||
}
|
||||
taf.IssueTimeVal = issueTime
|
||||
pos++
|
||||
|
||||
if pos >= len(tokens) {
|
||||
return nil, 0, fmt.Errorf("missing validity period")
|
||||
}
|
||||
validFrom, validTo, err := ParseTAFValidity(tokens[pos], taf.IssueTimeVal)
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("failed to parse validity period: %w", err)
|
||||
}
|
||||
taf.ValidFrom = validFrom
|
||||
taf.ValidTo = validTo
|
||||
pos++
|
||||
|
||||
return taf, pos, nil
|
||||
}
|
||||
|
||||
func resolveDayTime(base time.Time, day, hour, min int) time.Time {
|
||||
t := time.Date(base.Year(), base.Month(), day, hour, min, 0, 0, time.UTC)
|
||||
if day < base.Day() {
|
||||
t = t.AddDate(0, 1, 0)
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// parseFMPeriod parses an FM (from) period
|
||||
func parseFMPeriod(tokens []string, startPos int, validityFrom, validityTo time.Time, warnings *[]string) (weather.TafPeriod, int, error) {
|
||||
period := weather.TafPeriod{
|
||||
Type: "FM",
|
||||
}
|
||||
|
||||
match := tafFMPattern.FindStringSubmatch(tokens[startPos])
|
||||
if len(match) == 0 {
|
||||
return period, startPos + 1, fmt.Errorf("invalid FM format")
|
||||
}
|
||||
|
||||
day, _ := strconv.Atoi(match[1])
|
||||
hour, _ := strconv.Atoi(match[2])
|
||||
min, _ := strconv.Atoi(match[3])
|
||||
|
||||
period.ValidFrom = resolveDayTime(validityFrom, day, hour, min)
|
||||
period.ValidTo = validityTo
|
||||
|
||||
// Find end of this period (next FM, TEMPO, BECMG, or end)
|
||||
endPos := len(tokens)
|
||||
for i := startPos + 1; i < len(tokens); i++ {
|
||||
upperToken := strings.ToUpper(tokens[i])
|
||||
if strings.HasPrefix(upperToken, "FM") ||
|
||||
strings.HasPrefix(upperToken, "TEMPO") ||
|
||||
strings.HasPrefix(upperToken, "BECMG") ||
|
||||
strings.HasPrefix(upperToken, "RMK") {
|
||||
endPos = i
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Parse period elements
|
||||
periodTokens := tokens[startPos+1 : endPos]
|
||||
parsePeriodElements(periodTokens, &period, warnings)
|
||||
|
||||
// Set valid_to to start of next period or end of validity
|
||||
if endPos < len(tokens) {
|
||||
upperToken := strings.ToUpper(tokens[endPos])
|
||||
if strings.HasPrefix(upperToken, "FM") {
|
||||
// Next period starts here
|
||||
if match := tafFMPattern.FindStringSubmatch(tokens[endPos]); match != nil {
|
||||
nextDay, _ := strconv.Atoi(match[1])
|
||||
nextHour, _ := strconv.Atoi(match[2])
|
||||
nextMin, _ := strconv.Atoi(match[3])
|
||||
nextStart := resolveDayTime(period.ValidFrom, nextDay, nextHour, nextMin)
|
||||
if nextStart.Before(period.ValidTo) {
|
||||
period.ValidTo = nextStart
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return period, endPos, nil
|
||||
}
|
||||
|
||||
// parseTEMPOPeriod parses a TEMPO (temporary) period
|
||||
func parseTEMPOPeriod(tokens []string, startPos int, validityFrom time.Time, warnings *[]string) (weather.TafPeriod, int, error) {
|
||||
period := weather.TafPeriod{
|
||||
Type: "TEMPO",
|
||||
}
|
||||
|
||||
match := tafTEMPOPattern.FindStringSubmatch(tokens[startPos])
|
||||
if len(match) == 0 {
|
||||
return period, startPos + 1, fmt.Errorf("invalid TEMPO format")
|
||||
}
|
||||
|
||||
fromDay, _ := strconv.Atoi(match[1])
|
||||
fromHour, _ := strconv.Atoi(match[2])
|
||||
toDay, _ := strconv.Atoi(match[3])
|
||||
toHour, _ := strconv.Atoi(match[4])
|
||||
|
||||
period.ValidFrom = resolveDayTime(validityFrom, fromDay, fromHour, 0)
|
||||
period.ValidTo = resolveDayTime(period.ValidFrom, toDay, toHour, 0)
|
||||
|
||||
// Find end of this period
|
||||
endPos := startPos + 1
|
||||
for endPos < len(tokens) {
|
||||
upperToken := strings.ToUpper(tokens[endPos])
|
||||
if strings.HasPrefix(upperToken, "FM") ||
|
||||
strings.HasPrefix(upperToken, "TEMPO") ||
|
||||
strings.HasPrefix(upperToken, "BECMG") ||
|
||||
strings.HasPrefix(upperToken, "RMK") {
|
||||
break
|
||||
}
|
||||
endPos++
|
||||
}
|
||||
|
||||
// Parse period elements
|
||||
periodTokens := tokens[startPos+1 : endPos]
|
||||
parsePeriodElements(periodTokens, &period, warnings)
|
||||
|
||||
return period, endPos, nil
|
||||
}
|
||||
|
||||
// parseBECMGPeriod parses a BECMG (becoming) period
|
||||
func parseBECMGPeriod(tokens []string, startPos int, validityFrom time.Time, warnings *[]string) (weather.TafPeriod, int, error) {
|
||||
period := weather.TafPeriod{
|
||||
Type: "BECMG",
|
||||
}
|
||||
|
||||
match := tafBECMGPattern.FindStringSubmatch(tokens[startPos])
|
||||
if len(match) == 0 {
|
||||
return period, startPos + 1, fmt.Errorf("invalid BECMG format")
|
||||
}
|
||||
|
||||
fromDay, _ := strconv.Atoi(match[1])
|
||||
fromHour, _ := strconv.Atoi(match[2])
|
||||
toDay, _ := strconv.Atoi(match[3])
|
||||
toHour, _ := strconv.Atoi(match[4])
|
||||
|
||||
period.ValidFrom = resolveDayTime(validityFrom, fromDay, fromHour, 0)
|
||||
period.ValidTo = resolveDayTime(period.ValidFrom, toDay, toHour, 0)
|
||||
|
||||
// Find end of this period
|
||||
endPos := startPos + 1
|
||||
for endPos < len(tokens) {
|
||||
upperToken := strings.ToUpper(tokens[endPos])
|
||||
if strings.HasPrefix(upperToken, "FM") ||
|
||||
strings.HasPrefix(upperToken, "TEMPO") ||
|
||||
strings.HasPrefix(upperToken, "BECMG") ||
|
||||
strings.HasPrefix(upperToken, "RMK") {
|
||||
break
|
||||
}
|
||||
endPos++
|
||||
}
|
||||
|
||||
// Parse period elements
|
||||
periodTokens := tokens[startPos+1 : endPos]
|
||||
parsePeriodElements(periodTokens, &period, warnings)
|
||||
|
||||
return period, endPos, nil
|
||||
}
|
||||
|
||||
// parsePeriodElements parses common elements (wind, visibility, clouds, phenomena) for a TAF period
|
||||
func parsePeriodElements(tokens []string, period *weather.TafPeriod, warnings *[]string) {
|
||||
pos, wind, visibility := parseWindAndVisibility(tokens)
|
||||
period.Wind = wind
|
||||
period.Visibility = visibility
|
||||
|
||||
phenomConsumed, phenomena := parsePhenomena(tokens[pos:])
|
||||
period.Phenomena = append(period.Phenomena, phenomena...)
|
||||
pos += phenomConsumed
|
||||
|
||||
cloudConsumed, clouds := parseClouds(tokens[pos:])
|
||||
period.Clouds = append(period.Clouds, clouds...)
|
||||
pos += cloudConsumed
|
||||
|
||||
appendPeriodWarnings(warnings, tokens[pos:])
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
package weather
|
||||
|
||||
import (
|
||||
domainweather "caatsm/internal/domain/weather"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("TAF Parser", func() {
|
||||
Describe("parseTaf", func() {
|
||||
It("should parse a simple TAF", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(taf).ToNot(BeNil())
|
||||
Expect(taf.StationID).To(Equal("KJFK"))
|
||||
Expect(len(taf.Periods)).To(BeNumerically(">", 0))
|
||||
})
|
||||
|
||||
It("should parse TAF with FM period", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020 FM251800 36015KT 10SM SCT030="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(len(taf.Periods)).To(BeNumerically(">=", 2))
|
||||
Expect(taf.Periods[1].Type).To(Equal("FM"))
|
||||
})
|
||||
|
||||
It("should set final FM validTo to the TAF validity end", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020 FM251800 36015KT 10SM SCT030="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
var fmPeriod *domainweather.TafPeriod
|
||||
for i := range taf.Periods {
|
||||
if taf.Periods[i].Type == "FM" {
|
||||
fmPeriod = &taf.Periods[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
Expect(fmPeriod).ToNot(BeNil())
|
||||
Expect(fmPeriod.ValidTo).To(BeTemporally("==", taf.ValidTo))
|
||||
})
|
||||
|
||||
It("should truncate main period at the first FM", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020 FM251800 36015KT 10SM SCT030="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
var fmPeriod *domainweather.TafPeriod
|
||||
for i := range taf.Periods {
|
||||
if taf.Periods[i].Type == "FM" {
|
||||
fmPeriod = &taf.Periods[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
Expect(fmPeriod).ToNot(BeNil())
|
||||
Expect(taf.Periods[0].Type).To(Equal("MAIN"))
|
||||
Expect(taf.Periods[0].ValidTo).To(BeTemporally("==", fmPeriod.ValidFrom))
|
||||
})
|
||||
|
||||
It("should roll FM into next month when day precedes validity start", func() {
|
||||
raw := "TAF KJFK 301200Z 3012/0112 35012KT 10SM FEW020 FM010600 36015KT 10SM SCT030="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
var fmPeriod *domainweather.TafPeriod
|
||||
for i := range taf.Periods {
|
||||
if taf.Periods[i].Type == "FM" {
|
||||
fmPeriod = &taf.Periods[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
Expect(fmPeriod).ToNot(BeNil())
|
||||
Expect(fmPeriod.ValidFrom).To(BeTemporally(">", taf.ValidFrom))
|
||||
})
|
||||
|
||||
It("should parse TAF with TEMPO period", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020 TEMPO2512/2515 27015G25KT 5SM -RA="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(len(taf.Periods)).To(BeNumerically(">=", 1))
|
||||
// Find TEMPO period
|
||||
found := false
|
||||
for _, period := range taf.Periods {
|
||||
if period.Type == "TEMPO" {
|
||||
found = true
|
||||
Expect(period.Wind).ToNot(BeNil())
|
||||
Expect(len(period.Phenomena)).To(BeNumerically(">", 0))
|
||||
break
|
||||
}
|
||||
}
|
||||
Expect(found).To(BeTrue())
|
||||
})
|
||||
|
||||
It("should parse TAF with BECMG period", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020 BECMG2512/2515 36015KT="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
// Find BECMG period
|
||||
found := false
|
||||
for _, period := range taf.Periods {
|
||||
if period.Type == "BECMG" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
Expect(found).To(BeTrue())
|
||||
})
|
||||
|
||||
It("should roll TEMPO into next month when day precedes validity start", func() {
|
||||
raw := "TAF KJFK 301200Z 3012/0112 35012KT 10SM FEW020 TEMPO0102/0106 5SM -RA="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
var tempoPeriod *domainweather.TafPeriod
|
||||
for i := range taf.Periods {
|
||||
if taf.Periods[i].Type == "TEMPO" {
|
||||
tempoPeriod = &taf.Periods[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
Expect(tempoPeriod).ToNot(BeNil())
|
||||
Expect(tempoPeriod.ValidFrom).To(BeTemporally(">", taf.ValidFrom))
|
||||
})
|
||||
|
||||
It("should roll BECMG into next month when day precedes validity start", func() {
|
||||
raw := "TAF KJFK 301200Z 3012/0112 35012KT 10SM FEW020 BECMG0102/0106 36015KT="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
var becmgPeriod *domainweather.TafPeriod
|
||||
for i := range taf.Periods {
|
||||
if taf.Periods[i].Type == "BECMG" {
|
||||
becmgPeriod = &taf.Periods[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
Expect(becmgPeriod).ToNot(BeNil())
|
||||
Expect(becmgPeriod.ValidFrom).To(BeTemporally(">", taf.ValidFrom))
|
||||
})
|
||||
|
||||
It("should parse TAF with PROB", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020 PROB30 TEMPO2512/2515 27015G25KT 5SM -RA="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
// Find period with probability
|
||||
found := false
|
||||
for _, period := range taf.Periods {
|
||||
if period.Probability > 0 {
|
||||
found = true
|
||||
Expect(period.Probability).To(Equal(30))
|
||||
break
|
||||
}
|
||||
}
|
||||
Expect(found).To(BeTrue())
|
||||
})
|
||||
|
||||
It("should parse TAF with multiple periods", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020 FM251800 36015KT 10SM SCT030 TEMPO2520/2602 27015G25KT 5SM -RA="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(len(taf.Periods)).To(BeNumerically(">=", 2))
|
||||
})
|
||||
|
||||
It("should parse TAF with remarks", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020 RMK TEST REMARKS="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(taf.Remarks).To(ContainSubstring("RMK"))
|
||||
})
|
||||
|
||||
It("should handle unrecognized tokens as warnings", func() {
|
||||
raw := "TAF KJFK 251200Z 2512/2612 35012KT 10SM FEW020 UNKNOWN TOKEN="
|
||||
taf, err := parseTaf(raw)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(len(taf.Warnings)).To(BeNumerically(">", 0))
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,176 @@
|
||||
package validator
|
||||
|
||||
import (
|
||||
"caatsm/internal/adapter/dto"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// AFTNError represents an AFTN protocol violation
|
||||
type AFTNError struct {
|
||||
Field string // e.g., "priority_indicator", "icao_address"
|
||||
Value string
|
||||
Message string
|
||||
}
|
||||
|
||||
func (e *AFTNError) Error() string {
|
||||
return fmt.Sprintf("AFTN validation error [%s]: %s (value: %q)", e.Field, e.Message, e.Value)
|
||||
}
|
||||
|
||||
// AFTN field validators
|
||||
var (
|
||||
// Priority indicators: FF (Flash), GG (Immediate), QU (Distress), DD (Delay), SS (Service), KK (Correction)
|
||||
validPriorities = map[string]bool{
|
||||
"FF": true, "GG": true, "QU": true,
|
||||
"DD": true, "SS": true, "KK": true,
|
||||
}
|
||||
|
||||
// ICAO address: 4 uppercase alphanumeric characters
|
||||
icaoAddressPattern = regexp.MustCompile(`^[A-Z0-9]{4}$`)
|
||||
|
||||
// DateTime: DDHHMM (6 digits)
|
||||
dateTimePattern = regexp.MustCompile(`^\d{6}$`)
|
||||
)
|
||||
|
||||
// ValidatePriorityIndicator validates AFTN priority indicator
|
||||
func ValidatePriorityIndicator(priority string) error {
|
||||
priority = strings.TrimSpace(strings.ToUpper(priority))
|
||||
if priority == "" {
|
||||
return nil // Optional field
|
||||
}
|
||||
if !validPriorities[priority] {
|
||||
return &AFTNError{
|
||||
Field: "priority_indicator",
|
||||
Value: priority,
|
||||
Message: "must be one of FF, GG, QU, DD, SS, KK",
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ValidateICAOAddress validates 4-character ICAO address
|
||||
func ValidateICAOAddress(address string) error {
|
||||
address = strings.TrimSpace(strings.ToUpper(address))
|
||||
if address == "" {
|
||||
return nil // Optional field
|
||||
}
|
||||
if !icaoAddressPattern.MatchString(address) {
|
||||
return &AFTNError{
|
||||
Field: "icao_address",
|
||||
Value: address,
|
||||
Message: "must be 4 uppercase alphanumeric characters",
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ValidateDateTime validates DDHHMM format
|
||||
func ValidateDateTime(dt string) error {
|
||||
dt = strings.TrimSpace(dt)
|
||||
if dt == "" {
|
||||
return nil // Optional field
|
||||
}
|
||||
if !dateTimePattern.MatchString(dt) {
|
||||
return &AFTNError{
|
||||
Field: "datetime",
|
||||
Value: dt,
|
||||
Message: "must be 6 digits (DDHHMM format)",
|
||||
}
|
||||
}
|
||||
// Additional semantic validation
|
||||
if len(dt) == 6 {
|
||||
day := dt[0:2]
|
||||
hour := dt[2:4]
|
||||
minute := dt[4:6]
|
||||
// Basic range checks
|
||||
if !isValidRange(day, 1, 31) || !isValidRange(hour, 0, 23) || !isValidRange(minute, 0, 59) {
|
||||
return &AFTNError{
|
||||
Field: "datetime",
|
||||
Value: dt,
|
||||
Message: "invalid date/time ranges (DD:01-31, HH:00-23, MM:00-59)",
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ValidateTelegram validates all AFTN fields in ParsedTelegram
|
||||
func ValidateTelegram(telegram *dto.ParsedTelegram) error {
|
||||
if telegram == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
var errors []error
|
||||
|
||||
// Validate priority indicator
|
||||
if err := ValidatePriorityIndicator(telegram.PriorityIndicator); err != nil {
|
||||
errors = append(errors, err)
|
||||
}
|
||||
|
||||
// Validate primary address (ICAO)
|
||||
if err := ValidateICAOAddress(telegram.PrimaryAddress); err != nil {
|
||||
errors = append(errors, err)
|
||||
}
|
||||
|
||||
// Validate originator (ICAO)
|
||||
if err := ValidateICAOAddress(telegram.Originator); err != nil {
|
||||
errors = append(errors, err)
|
||||
}
|
||||
|
||||
// Validate datetime
|
||||
if err := ValidateDateTime(telegram.DateTime); err != nil {
|
||||
errors = append(errors, err)
|
||||
}
|
||||
|
||||
// Validate originator datetime
|
||||
if err := ValidateDateTime(telegram.OriginatorDateTime); err != nil {
|
||||
errors = append(errors, err)
|
||||
}
|
||||
|
||||
if len(errors) > 0 {
|
||||
return &AFTNValidationErrors{Errors: errors}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// AFTNValidationErrors wraps multiple validation errors
|
||||
type AFTNValidationErrors struct {
|
||||
Errors []error
|
||||
}
|
||||
|
||||
func (e *AFTNValidationErrors) Error() string {
|
||||
messages := make([]string, len(e.Errors))
|
||||
for i, err := range e.Errors {
|
||||
messages[i] = err.Error()
|
||||
}
|
||||
return fmt.Sprintf("AFTN validation failed: %s", strings.Join(messages, "; "))
|
||||
}
|
||||
|
||||
// IsAFTNError checks if error is an AFTN validation error
|
||||
func IsAFTNError(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
_, ok1 := err.(*AFTNError)
|
||||
_, ok2 := err.(*AFTNValidationErrors)
|
||||
return ok1 || ok2
|
||||
}
|
||||
|
||||
// GetAFTNErrorType extracts the error type for metrics labeling
|
||||
func GetAFTNErrorType(err error) string {
|
||||
if aftnErr, ok := err.(*AFTNError); ok {
|
||||
return aftnErr.Field
|
||||
}
|
||||
if _, ok := err.(*AFTNValidationErrors); ok {
|
||||
return "multiple_errors"
|
||||
}
|
||||
return "unknown"
|
||||
}
|
||||
|
||||
// isValidRange checks if a numeric string is within the specified range
|
||||
func isValidRange(s string, min, max int) bool {
|
||||
val, err := strconv.Atoi(s)
|
||||
return err == nil && val >= min && val <= max
|
||||
}
|
||||
@@ -0,0 +1,341 @@
|
||||
package validator_test
|
||||
|
||||
import (
|
||||
"caatsm/internal/adapter/dto"
|
||||
"caatsm/internal/adapter/validator"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("AFTN Validator", func() {
|
||||
Describe("ValidatePriorityIndicator", func() {
|
||||
Context("with valid priority indicators", func() {
|
||||
It("accepts FF (Flash)", func() {
|
||||
err := validator.ValidatePriorityIndicator("FF")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts GG (Immediate)", func() {
|
||||
err := validator.ValidatePriorityIndicator("GG")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts QU (Distress)", func() {
|
||||
err := validator.ValidatePriorityIndicator("QU")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts DD (Delay)", func() {
|
||||
err := validator.ValidatePriorityIndicator("DD")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts SS (Service)", func() {
|
||||
err := validator.ValidatePriorityIndicator("SS")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts KK (Correction)", func() {
|
||||
err := validator.ValidatePriorityIndicator("KK")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts lowercase with trimming", func() {
|
||||
err := validator.ValidatePriorityIndicator(" ff ")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts empty string (optional field)", func() {
|
||||
err := validator.ValidatePriorityIndicator("")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with invalid priority indicators", func() {
|
||||
It("rejects invalid code XX", func() {
|
||||
err := validator.ValidatePriorityIndicator("XX")
|
||||
Expect(err).ToNot(BeNil())
|
||||
Expect(validator.IsAFTNError(err)).To(BeTrue())
|
||||
Expect(validator.GetAFTNErrorType(err)).To(Equal("priority_indicator"))
|
||||
})
|
||||
|
||||
It("rejects single character", func() {
|
||||
err := validator.ValidatePriorityIndicator("F")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
|
||||
It("rejects three characters", func() {
|
||||
err := validator.ValidatePriorityIndicator("FFF")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("ValidateICAOAddress", func() {
|
||||
Context("with valid ICAO addresses", func() {
|
||||
It("accepts ZBTJ (Beijing)", func() {
|
||||
err := validator.ValidateICAOAddress("ZBTJ")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts KLAX (Los Angeles)", func() {
|
||||
err := validator.ValidateICAOAddress("KLAX")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts ZGGG (Guangzhou)", func() {
|
||||
err := validator.ValidateICAOAddress("ZGGG")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts alphanumeric codes like Z999", func() {
|
||||
err := validator.ValidateICAOAddress("Z999")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts 1ABC", func() {
|
||||
err := validator.ValidateICAOAddress("1ABC")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts lowercase with trimming", func() {
|
||||
err := validator.ValidateICAOAddress(" zbtj ")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts empty string (optional field)", func() {
|
||||
err := validator.ValidateICAOAddress("")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with invalid ICAO addresses", func() {
|
||||
It("rejects too short (3 chars)", func() {
|
||||
err := validator.ValidateICAOAddress("ZBT")
|
||||
Expect(err).ToNot(BeNil())
|
||||
Expect(validator.IsAFTNError(err)).To(BeTrue())
|
||||
Expect(validator.GetAFTNErrorType(err)).To(Equal("icao_address"))
|
||||
})
|
||||
|
||||
It("rejects too long (5 chars)", func() {
|
||||
err := validator.ValidateICAOAddress("ZBTJX")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
|
||||
It("rejects special characters", func() {
|
||||
err := validator.ValidateICAOAddress("ZB-J")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
|
||||
It("rejects spaces", func() {
|
||||
err := validator.ValidateICAOAddress("ZB J")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("ValidateDateTime", func() {
|
||||
Context("with valid datetime values", func() {
|
||||
It("accepts 151430 (15th day, 14:30)", func() {
|
||||
err := validator.ValidateDateTime("151430")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts 010000 (1st day, 00:00)", func() {
|
||||
err := validator.ValidateDateTime("010000")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts 312359 (31st day, 23:59)", func() {
|
||||
err := validator.ValidateDateTime("312359")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts empty string (optional field)", func() {
|
||||
err := validator.ValidateDateTime("")
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with invalid datetime values", func() {
|
||||
It("rejects non-numeric", func() {
|
||||
err := validator.ValidateDateTime("15A430")
|
||||
Expect(err).ToNot(BeNil())
|
||||
Expect(validator.IsAFTNError(err)).To(BeTrue())
|
||||
Expect(validator.GetAFTNErrorType(err)).To(Equal("datetime"))
|
||||
})
|
||||
|
||||
It("rejects too short (5 digits)", func() {
|
||||
err := validator.ValidateDateTime("15143")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
|
||||
It("rejects too long (7 digits)", func() {
|
||||
err := validator.ValidateDateTime("1514301")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
|
||||
It("rejects invalid day (00)", func() {
|
||||
err := validator.ValidateDateTime("001430")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
|
||||
It("rejects invalid day (32)", func() {
|
||||
err := validator.ValidateDateTime("321430")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
|
||||
It("rejects invalid hour (24)", func() {
|
||||
err := validator.ValidateDateTime("152430")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
|
||||
It("rejects invalid minute (60)", func() {
|
||||
err := validator.ValidateDateTime("151460")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
|
||||
It("rejects invalid minute (99)", func() {
|
||||
err := validator.ValidateDateTime("151499")
|
||||
Expect(err).ToNot(BeNil())
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("ValidateTelegram", func() {
|
||||
Context("with valid telegram", func() {
|
||||
It("accepts telegram with all valid fields", func() {
|
||||
telegram := &dto.ParsedTelegram{
|
||||
PriorityIndicator: "FF",
|
||||
PrimaryAddress: "ZBTJ",
|
||||
Originator: "KLAX",
|
||||
DateTime: "151430",
|
||||
OriginatorDateTime: "151425",
|
||||
}
|
||||
err := validator.ValidateTelegram(telegram)
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts telegram with empty optional fields", func() {
|
||||
telegram := &dto.ParsedTelegram{
|
||||
PriorityIndicator: "",
|
||||
PrimaryAddress: "ZBTJ",
|
||||
Originator: "",
|
||||
DateTime: "151430",
|
||||
OriginatorDateTime: "",
|
||||
}
|
||||
err := validator.ValidateTelegram(telegram)
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
|
||||
It("accepts nil telegram", func() {
|
||||
err := validator.ValidateTelegram(nil)
|
||||
Expect(err).To(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Context("with invalid telegram fields", func() {
|
||||
It("reports invalid priority indicator", func() {
|
||||
telegram := &dto.ParsedTelegram{
|
||||
PriorityIndicator: "XX",
|
||||
PrimaryAddress: "ZBTJ",
|
||||
DateTime: "151430",
|
||||
}
|
||||
err := validator.ValidateTelegram(telegram)
|
||||
Expect(err).ToNot(BeNil())
|
||||
Expect(validator.IsAFTNError(err)).To(BeTrue())
|
||||
})
|
||||
|
||||
It("reports invalid primary address", func() {
|
||||
telegram := &dto.ParsedTelegram{
|
||||
PriorityIndicator: "FF",
|
||||
PrimaryAddress: "TOOLONG",
|
||||
DateTime: "151430",
|
||||
}
|
||||
err := validator.ValidateTelegram(telegram)
|
||||
Expect(err).ToNot(BeNil())
|
||||
Expect(validator.IsAFTNError(err)).To(BeTrue())
|
||||
})
|
||||
|
||||
It("reports invalid originator", func() {
|
||||
telegram := &dto.ParsedTelegram{
|
||||
PriorityIndicator: "FF",
|
||||
PrimaryAddress: "ZBTJ",
|
||||
Originator: "KL",
|
||||
DateTime: "151430",
|
||||
}
|
||||
err := validator.ValidateTelegram(telegram)
|
||||
Expect(err).ToNot(BeNil())
|
||||
Expect(validator.IsAFTNError(err)).To(BeTrue())
|
||||
})
|
||||
|
||||
It("reports invalid datetime", func() {
|
||||
telegram := &dto.ParsedTelegram{
|
||||
PriorityIndicator: "FF",
|
||||
PrimaryAddress: "ZBTJ",
|
||||
DateTime: "321430",
|
||||
}
|
||||
err := validator.ValidateTelegram(telegram)
|
||||
Expect(err).ToNot(BeNil())
|
||||
Expect(validator.IsAFTNError(err)).To(BeTrue())
|
||||
})
|
||||
|
||||
It("reports multiple errors", func() {
|
||||
telegram := &dto.ParsedTelegram{
|
||||
PriorityIndicator: "XX",
|
||||
PrimaryAddress: "TOOLONG",
|
||||
Originator: "KL",
|
||||
DateTime: "321430",
|
||||
OriginatorDateTime: "991499",
|
||||
}
|
||||
err := validator.ValidateTelegram(telegram)
|
||||
Expect(err).ToNot(BeNil())
|
||||
Expect(validator.IsAFTNError(err)).To(BeTrue())
|
||||
Expect(validator.GetAFTNErrorType(err)).To(Equal("multiple_errors"))
|
||||
Expect(err.Error()).To(ContainSubstring("AFTN validation failed"))
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
Describe("IsAFTNError", func() {
|
||||
It("returns true for AFTNError", func() {
|
||||
err := validator.ValidatePriorityIndicator("XX")
|
||||
Expect(validator.IsAFTNError(err)).To(BeTrue())
|
||||
})
|
||||
|
||||
It("returns true for AFTNValidationErrors", func() {
|
||||
telegram := &dto.ParsedTelegram{
|
||||
PriorityIndicator: "XX",
|
||||
PrimaryAddress: "TOOLONG",
|
||||
}
|
||||
err := validator.ValidateTelegram(telegram)
|
||||
Expect(validator.IsAFTNError(err)).To(BeTrue())
|
||||
})
|
||||
|
||||
It("returns false for nil error", func() {
|
||||
Expect(validator.IsAFTNError(nil)).To(BeFalse())
|
||||
})
|
||||
})
|
||||
|
||||
Describe("GetAFTNErrorType", func() {
|
||||
It("extracts field name from AFTNError", func() {
|
||||
err := validator.ValidatePriorityIndicator("XX")
|
||||
Expect(validator.GetAFTNErrorType(err)).To(Equal("priority_indicator"))
|
||||
})
|
||||
|
||||
It("returns 'multiple_errors' for AFTNValidationErrors", func() {
|
||||
telegram := &dto.ParsedTelegram{
|
||||
PriorityIndicator: "XX",
|
||||
PrimaryAddress: "TOOLONG",
|
||||
}
|
||||
err := validator.ValidateTelegram(telegram)
|
||||
Expect(validator.GetAFTNErrorType(err)).To(Equal("multiple_errors"))
|
||||
})
|
||||
|
||||
It("returns 'unknown' for non-AFTN errors", func() {
|
||||
errorType := validator.GetAFTNErrorType(nil)
|
||||
Expect(errorType).To(Equal("unknown"))
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,13 @@
|
||||
package validator_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestValidator(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Validator Suite")
|
||||
}
|
||||
@@ -3,6 +3,8 @@ package app
|
||||
import (
|
||||
"caatsm/internal/adapter/dto"
|
||||
"caatsm/internal/adapter/parser"
|
||||
"caatsm/internal/adapter/validator"
|
||||
"caatsm/internal/infra/config"
|
||||
"caatsm/internal/infra/log"
|
||||
"caatsm/internal/infra/telemetry"
|
||||
"caatsm/internal/port"
|
||||
@@ -25,6 +27,7 @@ type MessageProcessor struct {
|
||||
publisher port.Publisher
|
||||
logger *zap.Logger
|
||||
telemetry telemetry.Recorder
|
||||
cfg *config.Config
|
||||
}
|
||||
|
||||
// ProcessingStatus represents the outcome of the processing pipeline
|
||||
@@ -45,6 +48,7 @@ func NewMessageProcessor(
|
||||
publisher port.Publisher,
|
||||
rec telemetry.Recorder,
|
||||
logger *zap.Logger,
|
||||
cfg *config.Config,
|
||||
) *MessageProcessor {
|
||||
return &MessageProcessor{
|
||||
parser: parser,
|
||||
@@ -52,6 +56,7 @@ func NewMessageProcessor(
|
||||
publisher: publisher,
|
||||
logger: logger,
|
||||
telemetry: rec,
|
||||
cfg: cfg,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -138,6 +143,31 @@ func (p *MessageProcessor) Handle(ctx context.Context, raw []byte, msgID string)
|
||||
}
|
||||
parsed.ErrorReason = ""
|
||||
|
||||
// AFTN protocol validation (if enabled)
|
||||
if p.cfg.AFTN.ValidationEnabled {
|
||||
if err := validator.ValidateTelegram(parsed); err != nil {
|
||||
parsed.Status = dto.MessageStatusAFTNError
|
||||
parsed.ErrorReason = err.Error()
|
||||
span.RecordError(err)
|
||||
span.SetStatus(codes.Error, err.Error())
|
||||
span.SetAttributes(
|
||||
attribute.String("aftn.error_type", validator.GetAFTNErrorType(err)),
|
||||
)
|
||||
p.telemetry.RecordAFTNValidationError(ctx, validator.GetAFTNErrorType(err))
|
||||
p.persistRaw(ctx, parsed)
|
||||
msgLogger.With(zap.String("status", string(parsed.Status))).
|
||||
Warn("AFTN validation failed",
|
||||
zap.String("error_type", validator.GetAFTNErrorType(err)),
|
||||
zap.String("content_preview", truncateContent(parsed.Content, 256)),
|
||||
zap.Error(err),
|
||||
)
|
||||
latency := parsed.ParsedAt.Sub(receivedAt)
|
||||
p.telemetry.RecordFailure("aftn_validator")
|
||||
p.telemetry.RecordProcessingResult(ctx, string(parsed.Status), parsed.Category, latency)
|
||||
return Permanent(fmt.Errorf("AFTN validation error: %w", err))
|
||||
}
|
||||
}
|
||||
|
||||
// Log parsing result
|
||||
span.SetAttributes(
|
||||
attribute.String("telegram.status", string(parsed.Status)),
|
||||
|
||||
@@ -3,9 +3,12 @@ package app
|
||||
import (
|
||||
"caatsm/internal/adapter/dto"
|
||||
"caatsm/internal/adapter/parser"
|
||||
"caatsm/internal/adapter/parser/weather"
|
||||
"caatsm/internal/infra/config"
|
||||
"caatsm/internal/infra/telemetry"
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
@@ -59,13 +62,21 @@ NNNN`)
|
||||
|
||||
// createBenchmarkProcessor creates a processor with mocks for benchmarking
|
||||
func createBenchmarkProcessor() *MessageProcessor {
|
||||
aviationParser := parser.ProvideParser()
|
||||
weatherParser := weather.NewWeatherParser()
|
||||
aviationParser := parser.ProvideParser(weatherParser)
|
||||
mockRepo := &mockRepository{}
|
||||
mockPub := &mockPublisher{}
|
||||
logger := zap.NewNop()
|
||||
recorder := telemetry.NewNoop()
|
||||
cfg := &config.Config{
|
||||
AFTN: config.AFTNConfig{
|
||||
ValidationEnabled: false,
|
||||
MessageGapThreshold: 2 * time.Minute,
|
||||
EnableSequenceGapDetection: true,
|
||||
},
|
||||
}
|
||||
|
||||
return NewMessageProcessor(aviationParser, mockRepo, mockPub, recorder, logger)
|
||||
return NewMessageProcessor(aviationParser, mockRepo, mockPub, recorder, logger, cfg)
|
||||
}
|
||||
|
||||
// BenchmarkHandleARR benchmarks processing ARR messages end-to-end
|
||||
@@ -126,15 +137,23 @@ func BenchmarkHandleMixed(b *testing.B) {
|
||||
// BenchmarkHandleParseOnly benchmarks parsing without persistence/publishing
|
||||
// This isolates parser performance
|
||||
func BenchmarkHandleParseOnly(b *testing.B) {
|
||||
aviationParser := parser.ProvideParser()
|
||||
weatherParser := weather.NewWeatherParser()
|
||||
aviationParser := parser.ProvideParser(weatherParser)
|
||||
// Use a repository that does nothing
|
||||
mockRepo := &mockRepository{}
|
||||
// Use a publisher that does nothing
|
||||
mockPub := &mockPublisher{}
|
||||
logger := zap.NewNop()
|
||||
recorder := telemetry.NewNoop()
|
||||
cfg := &config.Config{
|
||||
AFTN: config.AFTNConfig{
|
||||
ValidationEnabled: false,
|
||||
MessageGapThreshold: 2 * time.Minute,
|
||||
EnableSequenceGapDetection: true,
|
||||
},
|
||||
}
|
||||
|
||||
processor := NewMessageProcessor(aviationParser, mockRepo, mockPub, recorder, logger)
|
||||
processor := NewMessageProcessor(aviationParser, mockRepo, mockPub, recorder, logger, cfg)
|
||||
ctx := context.Background()
|
||||
|
||||
b.ResetTimer()
|
||||
|
||||
@@ -6,8 +6,9 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"caatsm/internal/adapter/parser"
|
||||
"caatsm/internal/adapter/dto"
|
||||
"caatsm/internal/adapter/parser"
|
||||
"caatsm/internal/infra/config"
|
||||
"caatsm/internal/infra/telemetry"
|
||||
"caatsm/internal/port"
|
||||
|
||||
@@ -122,7 +123,7 @@ var _ = Describe("MessageProcessor", func() {
|
||||
},
|
||||
err: errors.New("parse failure"),
|
||||
}
|
||||
proc = NewMessageProcessor(parserStub, repo, pub, telemetry.NewNoop(), logger)
|
||||
proc = NewMessageProcessor(parserStub, repo, pub, telemetry.NewNoop(), logger, newTestConfig())
|
||||
|
||||
err := proc.Handle(ctx, []byte("raw"), "msg-6")
|
||||
Expect(err).To(HaveOccurred())
|
||||
@@ -146,7 +147,17 @@ var _ = Describe("MessageProcessor", func() {
|
||||
})
|
||||
|
||||
func newTestProcessor(p parser.Parser, repo port.Repository, pub port.Publisher) *MessageProcessor {
|
||||
return NewMessageProcessor(p, repo, pub, telemetry.NewNoop(), zap.NewNop())
|
||||
return NewMessageProcessor(p, repo, pub, telemetry.NewNoop(), zap.NewNop(), newTestConfig())
|
||||
}
|
||||
|
||||
func newTestConfig() *config.Config {
|
||||
return &config.Config{
|
||||
AFTN: config.AFTNConfig{
|
||||
ValidationEnabled: false, // Disabled by default for tests
|
||||
MessageGapThreshold: 2 * time.Minute,
|
||||
EnableSequenceGapDetection: true,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
type stubParser struct {
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
package weather
|
||||
|
||||
import "time"
|
||||
|
||||
// Wind represents wind information
|
||||
type Wind struct {
|
||||
Direction int `json:"direction"` // Degrees
|
||||
Speed int `json:"speed"` // KT or MPS
|
||||
Gust int `json:"gust,omitempty"` // Gust speed
|
||||
Variable bool `json:"variable,omitempty"` // VRB
|
||||
VariableFrom int `json:"variable_from,omitempty"` // Variable wind from direction
|
||||
VariableTo int `json:"variable_to,omitempty"` // Variable wind to direction
|
||||
Unit string `json:"unit"` // "KT", "MPS"
|
||||
}
|
||||
|
||||
// Visibility represents visibility information
|
||||
type Visibility struct {
|
||||
Distance float64 `json:"distance"` // Meters or statute miles
|
||||
Unit string `json:"unit"` // "M", "SM"
|
||||
Direction string `json:"direction,omitempty"` // Directional visibility
|
||||
Modifier string `json:"modifier,omitempty"` // +, -, M, P
|
||||
}
|
||||
|
||||
// Cloud represents cloud information
|
||||
type Cloud struct {
|
||||
Type string `json:"type"` // FEW, SCT, BKN, OVC, VV
|
||||
Altitude int `json:"altitude"` // Feet
|
||||
Modifier string `json:"modifier,omitempty"` // CB, TCU
|
||||
}
|
||||
|
||||
// Temperature represents temperature or dewpoint
|
||||
type Temperature struct {
|
||||
Value float64 `json:"value"`
|
||||
Unit string `json:"unit"` // "C"
|
||||
}
|
||||
|
||||
// Altimeter represents altimeter setting
|
||||
type Altimeter struct {
|
||||
Value float64 `json:"value"`
|
||||
Unit string `json:"unit"` // "QNH" (hPa), "A" (inHg)
|
||||
}
|
||||
|
||||
// Phenomenon represents weather phenomenon
|
||||
type Phenomenon struct {
|
||||
Intensity string `json:"intensity,omitempty"` // -, +
|
||||
Descriptor string `json:"descriptor,omitempty"` // MI, BC, PR, TS, etc.
|
||||
Weather string `json:"weather"` // RA, SN, FG, etc.
|
||||
}
|
||||
|
||||
// TafPeriod represents a TAF period (FM, TEMPO, BECMG, or main forecast)
|
||||
type TafPeriod struct {
|
||||
Type string `json:"type"` // "FM", "TEMPO", "BECMG", "MAIN"
|
||||
ValidFrom time.Time `json:"valid_from,omitempty"`
|
||||
ValidTo time.Time `json:"valid_to,omitempty"`
|
||||
Wind *Wind `json:"wind,omitempty"`
|
||||
Visibility *Visibility `json:"visibility,omitempty"`
|
||||
Clouds []Cloud `json:"clouds,omitempty"`
|
||||
Phenomena []Phenomenon `json:"phenomena,omitempty"`
|
||||
Probability int `json:"probability,omitempty"` // PROB30, PROB40
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
package weather
|
||||
|
||||
import "errors"
|
||||
|
||||
var (
|
||||
// ErrInvalidFormat indicates an invalid weather report format
|
||||
ErrInvalidFormat = errors.New("invalid weather report format")
|
||||
|
||||
// ErrUnsupportedToken indicates an unsupported token in the report
|
||||
ErrUnsupportedToken = errors.New("unsupported token")
|
||||
|
||||
// ErrMissingStation indicates missing station identifier
|
||||
ErrMissingStation = errors.New("missing station identifier")
|
||||
|
||||
// ErrMissingTime indicates missing time information
|
||||
ErrMissingTime = errors.New("missing time information")
|
||||
)
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
package weather
|
||||
|
||||
import "time"
|
||||
|
||||
// ReportType represents the type of weather report
|
||||
type ReportType string
|
||||
|
||||
const (
|
||||
ReportTypeMETAR ReportType = "METAR"
|
||||
ReportTypeSPECI ReportType = "SPECI"
|
||||
ReportTypeTAF ReportType = "TAF"
|
||||
)
|
||||
|
||||
// WeatherMessage is the common interface for all weather messages
|
||||
type WeatherMessage interface {
|
||||
Type() ReportType
|
||||
Station() string
|
||||
IssueTime() time.Time
|
||||
RawText() string
|
||||
}
|
||||
|
||||
// Metar represents a METAR or SPECI weather report
|
||||
type Metar struct {
|
||||
ReportType ReportType `json:"type"`
|
||||
StationID string `json:"station"`
|
||||
IssueTimeVal time.Time `json:"issue_time"`
|
||||
ObsTime time.Time `json:"obs_time,omitempty"`
|
||||
RawTextVal string `json:"raw_text"`
|
||||
|
||||
// Core elements
|
||||
Wind *Wind `json:"wind,omitempty"`
|
||||
Visibility *Visibility `json:"visibility,omitempty"`
|
||||
Clouds []Cloud `json:"clouds,omitempty"`
|
||||
Temperature *Temperature `json:"temperature,omitempty"`
|
||||
Dewpoint *Temperature `json:"dewpoint,omitempty"`
|
||||
Altimeter *Altimeter `json:"altimeter,omitempty"`
|
||||
Phenomena []Phenomenon `json:"phenomena,omitempty"`
|
||||
|
||||
// Optional fields
|
||||
Modifier string `json:"modifier,omitempty"` // AUTO, COR
|
||||
Remarks string `json:"remarks,omitempty"`
|
||||
Warnings []string `json:"warnings,omitempty"` // Unrecognized tokens
|
||||
}
|
||||
|
||||
// Type returns the report type
|
||||
func (m *Metar) Type() ReportType {
|
||||
return m.ReportType
|
||||
}
|
||||
|
||||
// Station returns the station identifier
|
||||
func (m *Metar) Station() string {
|
||||
return m.StationID
|
||||
}
|
||||
|
||||
// IssueTime returns the issue time
|
||||
func (m *Metar) IssueTime() time.Time {
|
||||
return m.IssueTimeVal
|
||||
}
|
||||
|
||||
// RawText returns the raw text
|
||||
func (m *Metar) RawText() string {
|
||||
return m.RawTextVal
|
||||
}
|
||||
|
||||
// Taf represents a TAF (Terminal Aerodrome Forecast) weather report
|
||||
type Taf struct {
|
||||
ReportType ReportType `json:"type"`
|
||||
StationID string `json:"station"`
|
||||
IssueTimeVal time.Time `json:"issue_time"`
|
||||
ValidFrom time.Time `json:"valid_from"`
|
||||
ValidTo time.Time `json:"valid_to"`
|
||||
RawTextVal string `json:"raw_text"`
|
||||
|
||||
Periods []TafPeriod `json:"periods"` // FM, TEMPO, BECMG segments
|
||||
Remarks string `json:"remarks,omitempty"`
|
||||
Warnings []string `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
// Type returns the report type
|
||||
func (t *Taf) Type() ReportType {
|
||||
return t.ReportType
|
||||
}
|
||||
|
||||
// Station returns the station identifier
|
||||
func (t *Taf) Station() string {
|
||||
return t.StationID
|
||||
}
|
||||
|
||||
// IssueTime returns the issue time
|
||||
func (t *Taf) IssueTime() time.Time {
|
||||
return t.IssueTimeVal
|
||||
}
|
||||
|
||||
// RawText returns the raw text
|
||||
func (t *Taf) RawText() string {
|
||||
return t.RawTextVal
|
||||
}
|
||||
|
||||
@@ -22,6 +22,7 @@ type Config struct {
|
||||
Telemetry TelemetryConfig `koanf:"telemetry"`
|
||||
Monitoring MonitoringConfig `koanf:"monitoring"`
|
||||
DLQ DLQConfig `koanf:"dlq"`
|
||||
AFTN AFTNConfig `koanf:"aftn"`
|
||||
// Legacy fields for backward compatibility during migration
|
||||
Subscription SubscriptionConfig `koanf:"subscription"`
|
||||
Timeouts TimeoutsConfig `koanf:"timeouts"`
|
||||
@@ -146,6 +147,19 @@ type DLQConfig struct {
|
||||
Subject string `koanf:"subject"`
|
||||
}
|
||||
|
||||
// AFTNConfig defines AFTN protocol validation and monitoring settings
|
||||
type AFTNConfig struct {
|
||||
// ValidationEnabled enables AFTN protocol validation
|
||||
ValidationEnabled bool `koanf:"validation_enabled"`
|
||||
|
||||
// MessageGapThreshold is the duration after which the serial reader
|
||||
// is considered stalled (no messages received). Default: 2 minutes.
|
||||
MessageGapThreshold time.Duration `koanf:"message_gap_threshold"`
|
||||
|
||||
// EnableSequenceGapDetection enables monitoring for missing sequence numbers
|
||||
EnableSequenceGapDetection bool `koanf:"enable_sequence_gap_detection"`
|
||||
}
|
||||
|
||||
// MonitoringConfig controls the lightweight HTTP server that exposes health and metrics endpoints.
|
||||
type MonitoringConfig struct {
|
||||
Disabled bool `koanf:"disabled"`
|
||||
@@ -317,6 +331,11 @@ func LoadConfig() (*Config, error) {
|
||||
cfg.Monitoring.HealthTimeout = 2 * time.Second
|
||||
}
|
||||
|
||||
// Set AFTN defaults
|
||||
if cfg.AFTN.MessageGapThreshold == 0 {
|
||||
cfg.AFTN.MessageGapThreshold = 2 * time.Minute
|
||||
}
|
||||
|
||||
// Validate configuration
|
||||
if err := cfg.Validate(); err != nil {
|
||||
return nil, fmt.Errorf("config validation failed: %w", err)
|
||||
@@ -406,6 +425,10 @@ func (c *Config) Validate() error {
|
||||
if c.Monitoring.HealthTimeout < 0 {
|
||||
return fmt.Errorf("monitoring.health_timeout must be >= 0")
|
||||
}
|
||||
// Validate AFTN configuration
|
||||
if c.AFTN.MessageGapThreshold < 0 {
|
||||
return fmt.Errorf("aftn.message_gap_threshold must be >= 0")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"context"
|
||||
"math"
|
||||
"net/http"
|
||||
"strings"
|
||||
@@ -25,10 +26,14 @@ const (
|
||||
MetricJSAPICallsTotal = "caatsm_js_api_calls_total"
|
||||
MetricDBQueriesTotal = "caatsm_db_queries_total"
|
||||
MetricDBQueryLatencySeconds = "caatsm_db_query_latency_seconds"
|
||||
MetricDLQMessagesTotal = "caatsm_dlq_messages_total"
|
||||
MetricDLQPublishFailures = "caatsm_dlq_publish_failures_total"
|
||||
MetricPublishFailuresTotal = "caatsm_publish_failures_total"
|
||||
MetricNATSConsumerPending = "caatsm_nats_consumer_pending_messages"
|
||||
MetricDLQMessagesTotal = "caatsm_dlq_messages_total"
|
||||
MetricDLQPublishFailures = "caatsm_dlq_publish_failures_total"
|
||||
MetricPublishFailuresTotal = "caatsm_publish_failures_total"
|
||||
MetricNATSConsumerPending = "caatsm_nats_consumer_pending_messages"
|
||||
MetricAFTNValidationErrorsTotal = "caatsm_aftn_validation_errors_total"
|
||||
MetricMessageGapSeconds = "caatsm_message_gap_seconds"
|
||||
MetricMessageSequenceGapTotal = "caatsm_message_sequence_gap_total"
|
||||
MetricSerialReaderHealthy = "caatsm_serial_reader_healthy"
|
||||
|
||||
// Common label keys.
|
||||
LabelStatus = "status"
|
||||
@@ -39,6 +44,7 @@ const (
|
||||
LabelResult = "result"
|
||||
LabelReason = "reason"
|
||||
LabelOperation = "operation"
|
||||
LabelErrorType = "error_type"
|
||||
|
||||
// Standard result label values for caatsm_messages_total.
|
||||
ResultOK = "ok"
|
||||
@@ -78,6 +84,12 @@ var (
|
||||
|
||||
// NATS consumer lag metrics.
|
||||
natsConsumerPending *prometheus.GaugeVec
|
||||
|
||||
// AFTN validation and health metrics.
|
||||
aftnValidationErrorsTotal *prometheus.CounterVec
|
||||
messageGapSeconds *prometheus.GaugeVec
|
||||
messageSequenceGapTotal *prometheus.CounterVec
|
||||
serialReaderHealthy *prometheus.GaugeVec
|
||||
)
|
||||
|
||||
func initCollectors() {
|
||||
@@ -154,6 +166,27 @@ func initCollectors() {
|
||||
Help: "Approximate number of pending messages for a JetStream consumer, labelled by stream and consumer.",
|
||||
}, []string{LabelStream, LabelConsumer})
|
||||
|
||||
// AFTN validation and health metrics.
|
||||
aftnValidationErrorsTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: MetricAFTNValidationErrorsTotal,
|
||||
Help: "Total number of AFTN protocol validation errors, labelled by error type.",
|
||||
}, []string{LabelErrorType})
|
||||
|
||||
messageGapSeconds = prometheus.NewGaugeVec(prometheus.GaugeOpts{
|
||||
Name: MetricMessageGapSeconds,
|
||||
Help: "Time in seconds since the last message was received from the serial reader.",
|
||||
}, []string{LabelStream, LabelConsumer})
|
||||
|
||||
messageSequenceGapTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: MetricMessageSequenceGapTotal,
|
||||
Help: "Total number of message sequence gaps detected (missing sequence numbers).",
|
||||
}, []string{LabelStream, LabelConsumer})
|
||||
|
||||
serialReaderHealthy = prometheus.NewGaugeVec(prometheus.GaugeOpts{
|
||||
Name: MetricSerialReaderHealthy,
|
||||
Help: "Serial reader health status: 1 = healthy (messages flowing), 0 = stalled (no messages).",
|
||||
}, []string{LabelStream, LabelConsumer})
|
||||
|
||||
registry.MustRegister(
|
||||
processedCounter,
|
||||
failureCounter,
|
||||
@@ -168,6 +201,10 @@ func initCollectors() {
|
||||
dbQueriesTotal,
|
||||
dbQueryLatency,
|
||||
natsConsumerPending,
|
||||
aftnValidationErrorsTotal,
|
||||
messageGapSeconds,
|
||||
messageSequenceGapTotal,
|
||||
serialReaderHealthy,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -272,6 +309,35 @@ func RecordNATSConsumerPending(stream, consumer string, pending uint64) {
|
||||
natsConsumerPending.WithLabelValues(streamLabel, consumerLabel).Set(float64(pending))
|
||||
}
|
||||
|
||||
// RecordAFTNValidationError increments the AFTN validation error counter for the given error type.
|
||||
func RecordAFTNValidationError(ctx context.Context, errorType string) {
|
||||
ensureCollectors()
|
||||
aftnValidationErrorsTotal.WithLabelValues(labelValue(errorType)).Inc()
|
||||
}
|
||||
|
||||
// RecordMessageGap records the time gap (in seconds) since the last message was received.
|
||||
func RecordMessageGap(stream, consumer string, gapSeconds float64) {
|
||||
ensureCollectors()
|
||||
messageGapSeconds.WithLabelValues(labelValue(stream), labelValue(consumer)).Set(gapSeconds)
|
||||
}
|
||||
|
||||
// RecordSequenceGap increments the sequence gap counter when missing sequence numbers are detected.
|
||||
func RecordSequenceGap(stream, consumer string, gapSize uint64) {
|
||||
ensureCollectors()
|
||||
messageSequenceGapTotal.WithLabelValues(labelValue(stream), labelValue(consumer)).Add(float64(gapSize))
|
||||
}
|
||||
|
||||
// RecordSerialReaderHealth sets the serial reader health status.
|
||||
// healthy=1 means messages are flowing normally, healthy=0 means the reader has stalled.
|
||||
func RecordSerialReaderHealth(stream, consumer string, healthy bool) {
|
||||
ensureCollectors()
|
||||
value := 0.0
|
||||
if healthy {
|
||||
value = 1.0
|
||||
}
|
||||
serialReaderHealthy.WithLabelValues(labelValue(stream), labelValue(consumer)).Set(value)
|
||||
}
|
||||
|
||||
func labelValue(value string) string {
|
||||
value = strings.TrimSpace(value)
|
||||
if value == "" {
|
||||
|
||||
+278
-348
@@ -8,14 +8,17 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
// Consumer handles NATS JetStream message consumption with clean separation of concerns
|
||||
// Consumer handles NATS JetStream message consumption.
|
||||
// It consolidates stream management, fetching, and processing into a single, cohesive unit.
|
||||
type Consumer struct {
|
||||
// Core dependencies
|
||||
conn *nats.Conn
|
||||
@@ -25,35 +28,24 @@ type Consumer struct {
|
||||
logger *zap.Logger
|
||||
telemetry telemetry.Recorder
|
||||
|
||||
// Components
|
||||
monitor *ConsumerMonitor
|
||||
dlqHandler DLQHandler
|
||||
|
||||
// Configuration
|
||||
config consumerConfig
|
||||
|
||||
// Collaborators (injected for testability)
|
||||
fetcher MessageFetcher
|
||||
batchProcessor MessageProcessor
|
||||
dlqHandler DLQHandler
|
||||
|
||||
// Resource managers
|
||||
consumerManager *ConsumerManager
|
||||
streamManager *StreamManager
|
||||
streamName string
|
||||
consumerName string
|
||||
subject string
|
||||
batchSize int
|
||||
batchTimeout time.Duration
|
||||
ackWait time.Duration
|
||||
backoff []time.Duration
|
||||
|
||||
// State
|
||||
consecutiveProcessErrors int
|
||||
consecutiveErrors int
|
||||
}
|
||||
|
||||
// consumerConfig holds normalized consumer configuration values.
|
||||
type consumerConfig struct {
|
||||
subject string
|
||||
consumerName string
|
||||
streamName string
|
||||
dlqSubject string
|
||||
ackWait time.Duration
|
||||
batchSize int
|
||||
batchTimeout time.Duration
|
||||
monitorInterval time.Duration
|
||||
}
|
||||
|
||||
// ProvideConsumer creates a NATS consumer with clean architecture.
|
||||
// ProvideConsumer initializes a NATS consumer, ensuring infrastructure exists.
|
||||
func ProvideConsumer(
|
||||
conn *nats.Conn,
|
||||
js nats.JetStreamContext,
|
||||
@@ -62,364 +54,302 @@ func ProvideConsumer(
|
||||
rec telemetry.Recorder,
|
||||
logger *zap.Logger,
|
||||
) (*Consumer, error) {
|
||||
normCfg := normalizeConsumerConfig(cfg)
|
||||
|
||||
consumer := &Consumer{
|
||||
conn: conn,
|
||||
js: js,
|
||||
processor: processor,
|
||||
cfg: cfg,
|
||||
logger: logger,
|
||||
telemetry: rec,
|
||||
config: *normCfg, // dereference the pointer
|
||||
}
|
||||
consumer.initCollaborators()
|
||||
|
||||
// Initialize the pending messages metric early (set to 0) so it appears in Prometheus
|
||||
// even before the consumer starts. This ensures the metric is always visible.
|
||||
logger.Info("Initializing NATS consumer pending messages metric",
|
||||
zap.String("stream", normCfg.streamName),
|
||||
zap.String("consumer", normCfg.consumerName),
|
||||
zap.Uint64("pending", 0),
|
||||
zap.Bool("js_available", js != nil),
|
||||
)
|
||||
obsmetrics.RecordNATSConsumerPending(normCfg.streamName, normCfg.consumerName, 0)
|
||||
|
||||
// Initialize managers
|
||||
consumer.consumerManager = NewConsumerManager(js, normCfg.streamName, normCfg.consumerName, normCfg.subject, logger)
|
||||
// Use StreamManager with full configuration
|
||||
streamSubjects := []string{normCfg.subject}
|
||||
if publisherSubject := strings.TrimSpace(cfg.Publisher.Topic); publisherSubject != "" {
|
||||
streamSubjects = append(streamSubjects, publisherSubject)
|
||||
}
|
||||
// Add DLQ subject to stream if DLQ is enabled
|
||||
if normCfg.dlqSubject != "" {
|
||||
streamSubjects = append(streamSubjects, normCfg.dlqSubject)
|
||||
}
|
||||
streamSubjects = dedupeSubjects(streamSubjects)
|
||||
consumer.streamManager = NewStreamManager(js, normCfg.streamName, streamSubjects, logger)
|
||||
|
||||
// Update fetcher with managers now that they're initialized
|
||||
if fetcher, ok := consumer.fetcher.(*defaultMessageFetcher); ok {
|
||||
fetcher.consumerManager = consumer.consumerManager
|
||||
fetcher.streamManager = consumer.streamManager
|
||||
// 1. Normalize Configuration
|
||||
c := &Consumer{
|
||||
conn: conn,
|
||||
js: js,
|
||||
processor: processor,
|
||||
cfg: cfg,
|
||||
logger: logger,
|
||||
telemetry: rec,
|
||||
streamName: orDefault(cfg.NATS.Stream, "TELEGRAM"),
|
||||
consumerName: orDefault(cfg.NATS.Consumer, "telegram-consumer"),
|
||||
subject: cfg.EffectiveSubscriptionTopic(),
|
||||
batchSize: cfg.App.BatchSize,
|
||||
batchTimeout: cfg.App.BatchTimeout,
|
||||
ackWait: orDefaultDuration(cfg.NATS.ConsumerRules.AckWait, 30*time.Second),
|
||||
backoff: cfg.NATS.ConsumerRules.Backoff,
|
||||
}
|
||||
|
||||
// Ensure stream exists before creating consumer
|
||||
streamCfg := &StreamConfig{
|
||||
MaxMsgs: cfg.NATS.StreamLimits.MaxMsgs,
|
||||
MaxBytes: cfg.NATS.StreamLimits.MaxBytes,
|
||||
MaxAge: cfg.NATS.StreamLimits.MaxAge,
|
||||
Discard: cfg.NATS.StreamLimits.Discard,
|
||||
Storage: cfg.NATS.StreamLimits.Storage,
|
||||
Replicas: cfg.NATS.StreamLimits.Replicas,
|
||||
if c.batchSize <= 0 {
|
||||
c.batchSize = 50
|
||||
}
|
||||
if err := consumer.streamManager.EnsureStream(streamCfg); err != nil {
|
||||
return nil, fmt.Errorf("failed to ensure stream: %w", err)
|
||||
if c.batchTimeout <= 0 {
|
||||
c.batchTimeout = 2 * time.Second
|
||||
}
|
||||
|
||||
// Create consumer if it doesn't exist
|
||||
consumerConfig := consumer.buildConsumerConfig()
|
||||
if err := consumer.consumerManager.EnsureConsumer(consumerConfig); err != nil {
|
||||
return nil, fmt.Errorf("failed to ensure consumer: %w", err)
|
||||
}
|
||||
// Validate DLQ configuration early so misconfiguration is visible at startup
|
||||
// rather than only when the first poison message appears.
|
||||
if err := consumer.validateDLQ(); err != nil {
|
||||
return nil, fmt.Errorf("DLQ validation failed: %w", err)
|
||||
}
|
||||
|
||||
return consumer, nil
|
||||
}
|
||||
|
||||
// initCollaborators initializes the collaborator components
|
||||
func (c *Consumer) initCollaborators() {
|
||||
c.fetcher = &defaultMessageFetcher{
|
||||
batchSize: c.config.batchSize,
|
||||
batchTimeout: c.config.batchTimeout,
|
||||
logger: c.logger,
|
||||
conn: c.conn,
|
||||
js: c.js,
|
||||
consumerManager: c.consumerManager,
|
||||
streamManager: c.streamManager,
|
||||
config: &c.config,
|
||||
cfg: c.cfg,
|
||||
}
|
||||
|
||||
// Initialize DLQ handler first if needed, so batch processor can reference it
|
||||
if c.config.dlqSubject != "" {
|
||||
// 2. Initialize Components
|
||||
c.monitor = NewConsumerMonitor(logger, cfg, js, c.streamName, c.consumerName, cfg.App.MonitorInterval)
|
||||
if cfg.DLQ.Enabled && cfg.DLQ.Subject != "" {
|
||||
c.dlqHandler = &defaultDLQHandler{
|
||||
js: c.js,
|
||||
dlqSubject: c.config.dlqSubject,
|
||||
streamName: c.config.streamName,
|
||||
consumerName: c.config.consumerName,
|
||||
logger: c.logger,
|
||||
telemetry: c.telemetry,
|
||||
js: js,
|
||||
dlqSubject: cfg.DLQ.Subject,
|
||||
streamName: c.streamName,
|
||||
consumerName: c.consumerName,
|
||||
logger: logger,
|
||||
telemetry: rec,
|
||||
}
|
||||
}
|
||||
|
||||
c.batchProcessor = &defaultBatchProcessor{
|
||||
processor: c.processor,
|
||||
dlqHandler: c.dlqHandler,
|
||||
logger: c.logger,
|
||||
telemetry: c.telemetry,
|
||||
streamName: c.config.streamName,
|
||||
consumerName: c.config.consumerName,
|
||||
backoff: c.cfg.NATS.ConsumerRules.Backoff,
|
||||
consecutiveProcessErrors: &c.consecutiveProcessErrors,
|
||||
// 3. Ensure Infrastructure (Stream & Consumer)
|
||||
if err := c.ensureInfrastructure(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// normalizeConsumerConfig extracts and normalizes consumer configuration from the application config.
|
||||
// This function can be unit-tested without requiring a JetStream context.
|
||||
func normalizeConsumerConfig(cfg *config.Config) *consumerConfig {
|
||||
subject := cfg.EffectiveSubscriptionTopic()
|
||||
|
||||
consumerName := cfg.NATS.Consumer
|
||||
if consumerName == "" {
|
||||
consumerName = "telegram-consumer"
|
||||
}
|
||||
|
||||
streamName := cfg.NATS.Stream
|
||||
if streamName == "" {
|
||||
streamName = "TELEGRAM"
|
||||
}
|
||||
|
||||
// DLQ routing is only meaningful in JetStream mode. Respect dlq.enabled to allow
|
||||
// environments to opt out cleanly even if a subject is configured.
|
||||
dlqSubject := ""
|
||||
if cfg.DLQ.Enabled {
|
||||
dlqSubject = strings.TrimSpace(cfg.DLQ.Subject)
|
||||
}
|
||||
|
||||
ackWait := cfg.NATS.ConsumerRules.AckWait
|
||||
if ackWait == 0 {
|
||||
ackWait = cfg.Timeouts.AckWait
|
||||
}
|
||||
if ackWait == 0 {
|
||||
ackWait = 30 * time.Second
|
||||
}
|
||||
|
||||
batchSize := cfg.App.BatchSize
|
||||
if batchSize == 0 {
|
||||
batchSize = 50
|
||||
}
|
||||
|
||||
batchTimeout := cfg.App.BatchTimeout
|
||||
if batchTimeout == 0 {
|
||||
batchTimeout = 2 * time.Second
|
||||
}
|
||||
|
||||
monitorInterval := cfg.App.MonitorInterval
|
||||
if monitorInterval <= 0 {
|
||||
monitorInterval = 30 * time.Second
|
||||
}
|
||||
|
||||
return &consumerConfig{
|
||||
subject: subject,
|
||||
consumerName: consumerName,
|
||||
streamName: streamName,
|
||||
dlqSubject: dlqSubject,
|
||||
ackWait: ackWait,
|
||||
batchSize: batchSize,
|
||||
batchTimeout: batchTimeout,
|
||||
monitorInterval: monitorInterval,
|
||||
}
|
||||
}
|
||||
|
||||
// buildConsumerConfig builds the NATS consumer configuration
|
||||
func (c *Consumer) buildConsumerConfig() *nats.ConsumerConfig {
|
||||
return &nats.ConsumerConfig{
|
||||
Durable: c.config.consumerName,
|
||||
DeliverPolicy: mapDeliverPolicy(c.cfg.NATS.ConsumerRules.DeliverPolicy),
|
||||
AckPolicy: nats.AckExplicitPolicy,
|
||||
AckWait: c.config.ackWait,
|
||||
ReplayPolicy: mapReplayPolicy(c.cfg.NATS.ConsumerRules.ReplayPolicy),
|
||||
MaxDeliver: c.cfg.NATS.ConsumerRules.MaxDeliver,
|
||||
MaxAckPending: c.cfg.NATS.ConsumerRules.MaxAckPending,
|
||||
FilterSubject: c.config.subject,
|
||||
BackOff: c.cfg.NATS.ConsumerRules.Backoff,
|
||||
}
|
||||
}
|
||||
|
||||
// Start starts consuming messages from JetStream.
|
||||
// Start begins the main consumption loop.
|
||||
func (c *Consumer) Start(ctx context.Context) error {
|
||||
return c.startJetStream(ctx)
|
||||
}
|
||||
|
||||
// RouteToDLQ implements DLQHandler interface
|
||||
func (c *Consumer) RouteToDLQ(ctx context.Context, msg *nats.Msg, cause error) error {
|
||||
if c.dlqHandler != nil {
|
||||
return c.dlqHandler.RouteToDLQ(ctx, msg, cause)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ValidateDLQ implements DLQHandler interface
|
||||
func (c *Consumer) ValidateDLQ() error {
|
||||
if c.dlqHandler != nil {
|
||||
return c.dlqHandler.ValidateDLQ()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// validateDLQ is a helper for internal use (lowercase)
|
||||
func (c *Consumer) validateDLQ() error {
|
||||
return c.ValidateDLQ()
|
||||
}
|
||||
|
||||
// createPullSubscription creates a pull subscription
|
||||
func (c *Consumer) createPullSubscription() (*nats.Subscription, error) {
|
||||
return c.consumerManager.CreatePullSubscription()
|
||||
}
|
||||
|
||||
// startJetStream starts the JetStream consumer loop.
|
||||
func (c *Consumer) startJetStream(ctx context.Context) error {
|
||||
// Create pull subscription
|
||||
sub, err := c.createPullSubscription()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Use a closure that always cleans up the current subscription.
|
||||
// When subscription is replaced in handleFetchError, this will clean up
|
||||
// whatever currentSub points to at shutdown time.
|
||||
var currentSub = sub
|
||||
cleanupSubscriber := func() {
|
||||
if currentSub != nil {
|
||||
if err := currentSub.Unsubscribe(); err != nil {
|
||||
c.logger.Error("Failed to unsubscribe subscription", zap.Error(err))
|
||||
}
|
||||
currentSub = nil
|
||||
}
|
||||
}
|
||||
defer cleanupSubscriber()
|
||||
|
||||
c.logger.Info("Started consuming messages",
|
||||
zap.String("subject", c.config.subject),
|
||||
zap.String("consumer", c.config.consumerName),
|
||||
zap.String("stream", c.config.streamName),
|
||||
c.logger.Info("Starting consumer",
|
||||
zap.String("stream", c.streamName),
|
||||
zap.String("consumer", c.consumerName),
|
||||
zap.String("subject", c.subject),
|
||||
)
|
||||
|
||||
// Record initial pending messages metric immediately
|
||||
// This ensures the metric appears in Prometheus right away
|
||||
if info, err := c.js.ConsumerInfo(c.config.streamName, c.config.consumerName); err == nil {
|
||||
c.logger.Info("Recording initial NATS consumer pending messages metric",
|
||||
zap.String("stream", c.config.streamName),
|
||||
zap.String("consumer", c.config.consumerName),
|
||||
zap.Uint64("pending", info.NumPending),
|
||||
)
|
||||
obsmetrics.RecordNATSConsumerPending(c.config.streamName, c.config.consumerName, info.NumPending)
|
||||
} else {
|
||||
c.logger.Warn("Failed to fetch initial consumer info for pending messages metric",
|
||||
zap.String("stream", c.config.streamName),
|
||||
zap.String("consumer", c.config.consumerName),
|
||||
zap.Error(err),
|
||||
)
|
||||
// Start background monitoring
|
||||
monitorCtx, cancelMonitor := context.WithCancel(ctx)
|
||||
defer cancelMonitor()
|
||||
go c.monitor.Start(monitorCtx)
|
||||
|
||||
// Create subscription
|
||||
sub, err := c.js.PullSubscribe(c.subject, c.consumerName, nats.BindStream(c.streamName))
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to subscribe: %w", err)
|
||||
}
|
||||
defer sub.Unsubscribe()
|
||||
|
||||
// Initial metric recording
|
||||
if info, err := c.js.ConsumerInfo(c.streamName, c.consumerName); err == nil {
|
||||
c.monitor.RecordInitialPending(info.NumPending)
|
||||
}
|
||||
|
||||
statsCtx, statsCancel := context.WithCancel(ctx)
|
||||
defer statsCancel()
|
||||
go c.emitConsumerStats(statsCtx)
|
||||
|
||||
var fetchErrorStreak int
|
||||
|
||||
// Main Loop
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
c.logger.Info("Stopping consumer", zap.Error(ctx.Err()))
|
||||
return ctx.Err()
|
||||
return nil
|
||||
default:
|
||||
}
|
||||
|
||||
// Fetch messages in batch
|
||||
msgs, err := c.fetcher.FetchBatch(ctx, currentSub)
|
||||
msgs, err := sub.Fetch(c.batchSize, nats.MaxWait(c.batchTimeout))
|
||||
if err != nil {
|
||||
// If context was cancelled, return immediately
|
||||
if errors.Is(err, nats.ErrTimeout) {
|
||||
continue // Normal timeout, just retry
|
||||
}
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
c.logger.Info("Stopping consumer due to context cancellation", zap.Error(err))
|
||||
return err
|
||||
}
|
||||
shouldContinue, handleErr := c.fetcher.HandleFetchError(ctx, err, ¤tSub, &fetchErrorStreak)
|
||||
if !shouldContinue {
|
||||
return handleErr
|
||||
return nil
|
||||
}
|
||||
// Log other errors but keep loop alive unless critical
|
||||
c.logger.Warn("Fetch error", zap.Error(err))
|
||||
time.Sleep(100 * time.Millisecond) // Slight backoff
|
||||
continue
|
||||
}
|
||||
|
||||
// Successful fetch -> reset error streak.
|
||||
if fetchErrorStreak > 0 {
|
||||
fetchErrorStreak = 0
|
||||
}
|
||||
|
||||
// Process batch
|
||||
c.batchProcessor.ProcessBatch(ctx, msgs)
|
||||
c.processBatch(ctx, msgs)
|
||||
}
|
||||
}
|
||||
|
||||
// emitConsumerStats periodically emits basic consumer statistics.
|
||||
func (c *Consumer) emitConsumerStats(ctx context.Context) {
|
||||
ticker := time.NewTicker(c.config.monitorInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
// Record initial metric (0) to ensure it appears in Prometheus even before first tick
|
||||
c.logger.Info("Starting NATS consumer stats emission goroutine, recording initial pending metric",
|
||||
zap.String("stream", c.config.streamName),
|
||||
zap.String("consumer", c.config.consumerName),
|
||||
zap.Duration("interval", c.config.monitorInterval),
|
||||
)
|
||||
obsmetrics.RecordNATSConsumerPending(c.config.streamName, c.config.consumerName, 0)
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
info, err := c.js.ConsumerInfo(c.config.streamName, c.config.consumerName)
|
||||
if err != nil {
|
||||
c.logger.Warn("Failed to fetch consumer info for pending messages metric",
|
||||
zap.String("stream", c.config.streamName),
|
||||
zap.String("consumer", c.config.consumerName),
|
||||
zap.Error(err),
|
||||
)
|
||||
continue
|
||||
}
|
||||
// Record pending messages for monitoring
|
||||
c.logger.Debug("Recording NATS consumer pending messages metric",
|
||||
zap.String("stream", c.config.streamName),
|
||||
zap.String("consumer", c.config.consumerName),
|
||||
zap.Uint64("pending", info.NumPending),
|
||||
)
|
||||
obsmetrics.RecordNATSConsumerPending(c.config.streamName, c.config.consumerName, info.NumPending)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Shutdown drains the underlying NATS connection gracefully.
|
||||
// Shutdown gracefully drains the connection.
|
||||
func (c *Consumer) Shutdown(ctx context.Context) error {
|
||||
if c.conn == nil {
|
||||
return nil
|
||||
}
|
||||
c.logger.Info("Draining NATS connection...")
|
||||
return c.conn.Drain()
|
||||
}
|
||||
|
||||
timeout := c.cfg.Timeouts.Close
|
||||
if timeout <= 0 {
|
||||
timeout = 2 * time.Second // Reduced from 10s for faster shutdown
|
||||
}
|
||||
|
||||
closeCtx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
errCh := make(chan error, 1)
|
||||
go func() {
|
||||
errCh <- c.conn.Drain()
|
||||
}()
|
||||
|
||||
select {
|
||||
case err := <-errCh:
|
||||
c.conn.Close()
|
||||
return err
|
||||
case <-closeCtx.Done():
|
||||
c.conn.Close()
|
||||
return fmt.Errorf("nats drain timeout: %w", closeCtx.Err())
|
||||
// processBatch iterates through a batch of messages.
|
||||
func (c *Consumer) processBatch(ctx context.Context, msgs []*nats.Msg) {
|
||||
for _, msg := range msgs {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
default:
|
||||
c.monitor.TrackMessage(msg)
|
||||
c.processMsg(ctx, msg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// processMsg handles a single message: Trace -> App Logic -> Ack/Nak.
|
||||
func (c *Consumer) processMsg(ctx context.Context, msg *nats.Msg) {
|
||||
start := time.Now()
|
||||
ctx, span := otel.Tracer("caatsm/nats").Start(ctx, "Consumer.processMsg")
|
||||
defer span.End()
|
||||
|
||||
msgID := c.resolveMsgID(msg)
|
||||
|
||||
// Add metadata to span/logger
|
||||
span.SetAttributes(
|
||||
attribute.String("messaging.system", "nats"),
|
||||
attribute.String("messaging.message_id", msgID),
|
||||
attribute.String("caatsm.stream", c.streamName),
|
||||
)
|
||||
|
||||
// Execute Application Logic
|
||||
err := c.processor.Handle(ctx, msg.Data, msgID)
|
||||
|
||||
// Handle Result
|
||||
if err != nil {
|
||||
c.handleError(ctx, msg, msgID, err)
|
||||
span.RecordError(err)
|
||||
span.SetStatus(codes.Error, err.Error())
|
||||
c.telemetry.RecordMessageHandled(ctx, c.streamName, c.consumerName, obsmetrics.ResultFail, time.Since(start))
|
||||
} else {
|
||||
// Success
|
||||
if c.consecutiveErrors > 0 {
|
||||
c.consecutiveErrors = 0
|
||||
}
|
||||
if ackErr := msg.Ack(); ackErr != nil {
|
||||
c.logger.Warn("Failed to ACK", zap.String("msg_id", msgID), zap.Error(ackErr))
|
||||
}
|
||||
c.telemetry.RecordMessageHandled(ctx, c.streamName, c.consumerName, "ok", time.Since(start))
|
||||
}
|
||||
}
|
||||
|
||||
// handleError decides whether to Ack (Permanent/DLQ) or Nak (Transient).
|
||||
func (c *Consumer) handleError(ctx context.Context, msg *nats.Msg, msgID string, err error) {
|
||||
isPermanent := app.IsPermanent(err)
|
||||
c.logger.Error("Processing failed",
|
||||
zap.String("msg_id", msgID),
|
||||
zap.Error(err),
|
||||
zap.Bool("permanent", isPermanent),
|
||||
)
|
||||
|
||||
if isPermanent {
|
||||
// Poison message: Route to DLQ -> Ack
|
||||
c.consecutiveErrors = 0
|
||||
if c.dlqHandler != nil {
|
||||
_ = c.dlqHandler.RouteToDLQ(ctx, msg, err) // Logged inside handler
|
||||
}
|
||||
_ = msg.Ack()
|
||||
return
|
||||
}
|
||||
|
||||
// Transient error: Backpressure -> Nak with Backoff
|
||||
c.consecutiveErrors++
|
||||
c.applyBackpressure(ctx)
|
||||
|
||||
_ = c.nakWithBackoff(msg)
|
||||
}
|
||||
|
||||
// nakWithBackoff calculates the appropriate NAK delay based on delivery attempts.
|
||||
func (c *Consumer) nakWithBackoff(msg *nats.Msg) error {
|
||||
if len(c.backoff) == 0 {
|
||||
return msg.Nak()
|
||||
}
|
||||
meta, err := msg.Metadata()
|
||||
if err != nil {
|
||||
return msg.Nak()
|
||||
}
|
||||
|
||||
// attempt is 1-based, index is 0-based
|
||||
attempt := int(meta.NumDelivered)
|
||||
index := attempt - 1
|
||||
if index >= len(c.backoff) {
|
||||
index = len(c.backoff) - 1
|
||||
} else if index < 0 {
|
||||
index = 0
|
||||
}
|
||||
|
||||
return msg.NakWithDelay(c.backoff[index])
|
||||
}
|
||||
|
||||
// applyBackpressure sleeps if error streak is high to protect the system.
|
||||
func (c *Consumer) applyBackpressure(ctx context.Context) {
|
||||
if c.consecutiveErrors < 10 {
|
||||
return
|
||||
}
|
||||
delay := time.Duration(c.consecutiveErrors) * 100 * time.Millisecond
|
||||
if delay > 5*time.Second {
|
||||
delay = 5 * time.Second
|
||||
}
|
||||
|
||||
select {
|
||||
case <-time.After(delay):
|
||||
case <-ctx.Done():
|
||||
}
|
||||
}
|
||||
|
||||
// ensureInfrastructure creates the Stream and Consumer if they don't exist.
|
||||
func (c *Consumer) ensureInfrastructure() error {
|
||||
// 1. Ensure Stream
|
||||
subjects := []string{c.subject}
|
||||
if c.cfg.Publisher.Topic != "" {
|
||||
subjects = append(subjects, c.cfg.Publisher.Topic)
|
||||
}
|
||||
if c.cfg.DLQ.Enabled && c.cfg.DLQ.Subject != "" {
|
||||
subjects = append(subjects, c.cfg.DLQ.Subject)
|
||||
}
|
||||
|
||||
streamCfg := &nats.StreamConfig{
|
||||
Name: c.streamName,
|
||||
Subjects: dedupeSubjects(subjects),
|
||||
Retention: nats.WorkQueuePolicy, // Defaulting to WorkQueue for queues
|
||||
MaxMsgs: c.cfg.NATS.StreamLimits.MaxMsgs,
|
||||
MaxBytes: c.cfg.NATS.StreamLimits.MaxBytes,
|
||||
MaxAge: c.cfg.NATS.StreamLimits.MaxAge,
|
||||
Replicas: c.cfg.NATS.StreamLimits.Replicas,
|
||||
Storage: nats.FileStorage,
|
||||
}
|
||||
if c.cfg.NATS.StreamLimits.Discard == "new" {
|
||||
streamCfg.Discard = nats.DiscardNew
|
||||
}
|
||||
if c.cfg.NATS.StreamLimits.Storage == "memory" {
|
||||
streamCfg.Storage = nats.MemoryStorage
|
||||
}
|
||||
|
||||
// Idempotent add/update
|
||||
if _, err := c.js.AddStream(streamCfg); err != nil {
|
||||
return fmt.Errorf("ensure stream: %w", err)
|
||||
}
|
||||
|
||||
// 2. Ensure Consumer
|
||||
consumerCfg := &nats.ConsumerConfig{
|
||||
Durable: c.consumerName,
|
||||
FilterSubject: c.subject,
|
||||
AckPolicy: nats.AckExplicitPolicy,
|
||||
AckWait: c.ackWait,
|
||||
MaxDeliver: c.cfg.NATS.ConsumerRules.MaxDeliver,
|
||||
MaxAckPending: c.cfg.NATS.ConsumerRules.MaxAckPending,
|
||||
ReplayPolicy: nats.ReplayInstantPolicy,
|
||||
}
|
||||
if c.cfg.NATS.ConsumerRules.ReplayPolicy == "original" {
|
||||
consumerCfg.ReplayPolicy = nats.ReplayOriginalPolicy
|
||||
}
|
||||
|
||||
// Idempotent add/update
|
||||
if _, err := c.js.AddConsumer(c.streamName, consumerCfg); err != nil {
|
||||
return fmt.Errorf("ensure consumer: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveMsgID extracts the ID from headers or metadata.
|
||||
func (c *Consumer) resolveMsgID(msg *nats.Msg) string {
|
||||
if id := msg.Header.Get("Nats-Msg-Id"); id != "" {
|
||||
return id
|
||||
}
|
||||
if meta, err := msg.Metadata(); err == nil {
|
||||
return fmt.Sprintf("js-%d", meta.Sequence.Stream)
|
||||
}
|
||||
return "unknown"
|
||||
}
|
||||
|
||||
// --- Helpers ---
|
||||
|
||||
func orDefault(val, def string) string {
|
||||
if val != "" {
|
||||
return val
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func orDefaultDuration(val, def time.Duration) time.Duration {
|
||||
if val > 0 {
|
||||
return val
|
||||
}
|
||||
return def
|
||||
}
|
||||
@@ -1,70 +0,0 @@
|
||||
package nats
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
// ConsumerManager handles JetStream consumer lifecycle management
|
||||
type ConsumerManager struct {
|
||||
js nats.JetStreamContext
|
||||
streamName string
|
||||
consumerName string
|
||||
subject string
|
||||
logger *zap.Logger
|
||||
}
|
||||
|
||||
// NewConsumerManager creates a new consumer manager
|
||||
func NewConsumerManager(js nats.JetStreamContext, streamName, consumerName, subject string, logger *zap.Logger) *ConsumerManager {
|
||||
return &ConsumerManager{
|
||||
js: js,
|
||||
streamName: streamName,
|
||||
consumerName: consumerName,
|
||||
subject: subject,
|
||||
logger: logger,
|
||||
}
|
||||
}
|
||||
|
||||
// EnsureConsumer creates the consumer if it doesn't exist; if it already exists, it is reused.
|
||||
func (cm *ConsumerManager) EnsureConsumer(config *nats.ConsumerConfig) error {
|
||||
// First check if the consumer already exists to make this initialization idempotent.
|
||||
info, err := cm.js.ConsumerInfo(cm.streamName, cm.consumerName)
|
||||
if err == nil && info != nil {
|
||||
cm.logger.Info("Using existing JetStream consumer",
|
||||
zap.String("consumer", cm.consumerName),
|
||||
zap.String("stream", cm.streamName),
|
||||
zap.String("subject", cm.subject),
|
||||
)
|
||||
return nil
|
||||
}
|
||||
if err != nil && !errors.Is(err, nats.ErrConsumerNotFound) {
|
||||
return fmt.Errorf("failed to fetch consumer info: %w", err)
|
||||
}
|
||||
|
||||
// Consumer does not exist; create it.
|
||||
if _, err := cm.js.AddConsumer(cm.streamName, config); err != nil {
|
||||
return fmt.Errorf("failed to create consumer: %w", err)
|
||||
}
|
||||
|
||||
cm.logger.Info("Created JetStream consumer",
|
||||
zap.String("consumer", cm.consumerName),
|
||||
zap.String("stream", cm.streamName),
|
||||
zap.String("subject", cm.subject),
|
||||
zap.Duration("ack_wait", config.AckWait),
|
||||
)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreatePullSubscription creates a pull subscription with recovery logic
|
||||
func (cm *ConsumerManager) CreatePullSubscription() (*nats.Subscription, error) {
|
||||
return cm.js.PullSubscribe(cm.subject, cm.consumerName, nats.Bind(cm.streamName, cm.consumerName))
|
||||
}
|
||||
|
||||
// CreatePullSubscriptionWithRecovery creates a pull subscription
|
||||
func (cm *ConsumerManager) CreatePullSubscriptionWithRecovery(streamManager *StreamManager, consumerConfig *nats.ConsumerConfig) (*nats.Subscription, error) {
|
||||
return cm.CreatePullSubscription()
|
||||
}
|
||||
@@ -3,9 +3,6 @@ package nats
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
configpkg "caatsm/internal/infra/config"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
@@ -48,97 +45,4 @@ var _ = Describe("Consumer helpers", func() {
|
||||
Expect(mapReplayPolicy("")).To(Equal(nats.ReplayInstantPolicy))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("normalizeConsumerConfig", func() {
|
||||
It("applies default values when config fields are empty", func() {
|
||||
cfg := &configpkg.Config{
|
||||
NATS: configpkg.NATSConfig{
|
||||
Mode: "",
|
||||
},
|
||||
App: configpkg.AppConfig{},
|
||||
Timeouts: configpkg.TimeoutsConfig{},
|
||||
}
|
||||
|
||||
normCfg := normalizeConsumerConfig(cfg)
|
||||
|
||||
Expect(normCfg.consumerName).To(Equal("telegram-consumer"))
|
||||
Expect(normCfg.streamName).To(Equal("TELEGRAM"))
|
||||
Expect(normCfg.batchSize).To(Equal(50))
|
||||
Expect(normCfg.batchTimeout).To(Equal(2 * time.Second))
|
||||
Expect(normCfg.monitorInterval).To(Equal(30 * time.Second))
|
||||
Expect(normCfg.ackWait).To(Equal(30 * time.Second))
|
||||
})
|
||||
|
||||
It("uses provided values when config fields are set", func() {
|
||||
cfg := &configpkg.Config{
|
||||
NATS: configpkg.NATSConfig{
|
||||
Consumer: "custom-consumer",
|
||||
Stream: "CUSTOM_STREAM",
|
||||
ConsumerRules: configpkg.ConsumerRulesConfig{
|
||||
AckWait: 60 * time.Second,
|
||||
},
|
||||
},
|
||||
App: configpkg.AppConfig{
|
||||
BatchSize: 100,
|
||||
BatchTimeout: 5 * time.Second,
|
||||
MonitorInterval: 60 * time.Second,
|
||||
},
|
||||
Timeouts: configpkg.TimeoutsConfig{
|
||||
AckWait: 45 * time.Second,
|
||||
},
|
||||
}
|
||||
|
||||
normCfg := normalizeConsumerConfig(cfg)
|
||||
|
||||
Expect(normCfg.consumerName).To(Equal("custom-consumer"))
|
||||
Expect(normCfg.streamName).To(Equal("CUSTOM_STREAM"))
|
||||
Expect(normCfg.batchSize).To(Equal(100))
|
||||
Expect(normCfg.batchTimeout).To(Equal(5 * time.Second))
|
||||
Expect(normCfg.monitorInterval).To(Equal(60 * time.Second))
|
||||
Expect(normCfg.ackWait).To(Equal(60 * time.Second)) // Uses ConsumerRules.AckWait
|
||||
})
|
||||
|
||||
It("falls back to Timeouts.AckWait when ConsumerRules.AckWait is zero", func() {
|
||||
cfg := &configpkg.Config{
|
||||
NATS: configpkg.NATSConfig{
|
||||
ConsumerRules: configpkg.ConsumerRulesConfig{
|
||||
AckWait: 0,
|
||||
},
|
||||
},
|
||||
Timeouts: configpkg.TimeoutsConfig{
|
||||
AckWait: 45 * time.Second,
|
||||
},
|
||||
}
|
||||
|
||||
normCfg := normalizeConsumerConfig(cfg)
|
||||
|
||||
Expect(normCfg.ackWait).To(Equal(45 * time.Second))
|
||||
})
|
||||
|
||||
It("sets dlqSubject when DLQ is enabled", func() {
|
||||
cfg := &configpkg.Config{
|
||||
DLQ: configpkg.DLQConfig{
|
||||
Enabled: true,
|
||||
Subject: "caatsm.dlq",
|
||||
},
|
||||
}
|
||||
|
||||
normCfg := normalizeConsumerConfig(cfg)
|
||||
|
||||
Expect(normCfg.dlqSubject).To(Equal("caatsm.dlq"))
|
||||
})
|
||||
|
||||
It("clears dlqSubject when DLQ is disabled", func() {
|
||||
cfg := &configpkg.Config{
|
||||
DLQ: configpkg.DLQConfig{
|
||||
Enabled: false,
|
||||
Subject: "caatsm.dlq",
|
||||
},
|
||||
}
|
||||
|
||||
normCfg := normalizeConsumerConfig(cfg)
|
||||
|
||||
Expect(normCfg.dlqSubject).To(Equal(""))
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -1,90 +0,0 @@
|
||||
package nats
|
||||
|
||||
import (
|
||||
"caatsm/internal/infra/config"
|
||||
"context"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
// MessageFetcher defines the interface for fetching messages from NATS
|
||||
type MessageFetcher interface {
|
||||
FetchBatch(ctx context.Context, sub *nats.Subscription) ([]*nats.Msg, error)
|
||||
HandleFetchError(ctx context.Context, err error, sub **nats.Subscription, fetchErrorStreak *int) (bool, error)
|
||||
}
|
||||
|
||||
// defaultMessageFetcher implements MessageFetcher interface
|
||||
type defaultMessageFetcher struct {
|
||||
batchSize int
|
||||
batchTimeout time.Duration
|
||||
logger *zap.Logger
|
||||
conn *nats.Conn
|
||||
js nats.JetStreamContext
|
||||
consumerManager *ConsumerManager
|
||||
streamManager *StreamManager
|
||||
config *consumerConfig
|
||||
cfg *config.Config
|
||||
}
|
||||
|
||||
func (f *defaultMessageFetcher) FetchBatch(ctx context.Context, sub *nats.Subscription) ([]*nats.Msg, error) {
|
||||
return f.fetchBatch(ctx, sub)
|
||||
}
|
||||
|
||||
// fetchBatch fetches a batch of messages from the subscription with context awareness
|
||||
func (f *defaultMessageFetcher) fetchBatch(ctx context.Context, sub *nats.Subscription) ([]*nats.Msg, error) {
|
||||
// Check context before fetching
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
default:
|
||||
}
|
||||
|
||||
// Use a shorter timeout for better responsiveness to cancellation
|
||||
timeout := f.batchTimeout
|
||||
if timeout > 500*time.Millisecond {
|
||||
timeout = 500 * time.Millisecond
|
||||
}
|
||||
|
||||
return sub.Fetch(f.batchSize, nats.MaxWait(timeout))
|
||||
}
|
||||
|
||||
func (f *defaultMessageFetcher) HandleFetchError(ctx context.Context, err error, sub **nats.Subscription, fetchErrorStreak *int) (bool, error) {
|
||||
// Context cancellation - stop processing
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
f.logger.Info("Fetch error due to context cancellation", zap.Error(err))
|
||||
return false, err
|
||||
}
|
||||
|
||||
// Timeout is normal - continue
|
||||
if errors.Is(err, nats.ErrTimeout) {
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// Connection issues - apply simple backoff
|
||||
*fetchErrorStreak++
|
||||
backoff := f.calculateExponentialBackoff(*fetchErrorStreak)
|
||||
f.logger.Warn("Fetch error, applying backoff",
|
||||
zap.Error(err),
|
||||
zap.Int("error_streak", *fetchErrorStreak),
|
||||
zap.Duration("backoff", backoff),
|
||||
)
|
||||
|
||||
if !sleepWithContext(ctx, backoff) {
|
||||
return false, ctx.Err()
|
||||
}
|
||||
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// calculateExponentialBackoff calculates exponential backoff duration with a cap
|
||||
func (f *defaultMessageFetcher) calculateExponentialBackoff(streak int) time.Duration {
|
||||
if streak <= 0 {
|
||||
return 0
|
||||
}
|
||||
// Simple exponential backoff: 2^(streak-1) seconds, capped at 30 seconds
|
||||
backoff := time.Duration(1<<uint(min(streak-1, 5))) * time.Second
|
||||
return min(backoff, 30*time.Second)
|
||||
}
|
||||
@@ -1,89 +0,0 @@
|
||||
package nats
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
configpkg "caatsm/internal/infra/config"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"go.uber.org/zap"
|
||||
"go.uber.org/zap/zaptest"
|
||||
)
|
||||
|
||||
var _ = Describe("MessageFetcher", func() {
|
||||
var (
|
||||
fetcher *defaultMessageFetcher
|
||||
logger *zap.Logger
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
logger = zaptest.NewLogger(GinkgoT())
|
||||
fetcher = &defaultMessageFetcher{
|
||||
logger: logger,
|
||||
config: &consumerConfig{
|
||||
streamName: "TEST_STREAM",
|
||||
consumerName: "test-consumer",
|
||||
},
|
||||
cfg: &configpkg.Config{
|
||||
NATS: configpkg.NATSConfig{
|
||||
ConsumerRules: configpkg.ConsumerRulesConfig{
|
||||
Backoff: []time.Duration{5 * time.Second, 30 * time.Second},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
})
|
||||
|
||||
Describe("HandleFetchError", func() {
|
||||
var ctx context.Context
|
||||
|
||||
BeforeEach(func() {
|
||||
ctx = context.Background()
|
||||
})
|
||||
|
||||
It("returns true for timeout errors", func() {
|
||||
var sub *nats.Subscription
|
||||
fetchErrorStreak := 0
|
||||
shouldContinue, err := fetcher.HandleFetchError(ctx, nats.ErrTimeout, &sub, &fetchErrorStreak)
|
||||
Expect(shouldContinue).To(BeTrue())
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
It("handles ErrNoResponders with backoff", func() {
|
||||
var sub *nats.Subscription
|
||||
fetchErrorStreak := 0
|
||||
shouldContinue, err := fetcher.HandleFetchError(ctx, nats.ErrNoResponders, &sub, &fetchErrorStreak)
|
||||
Expect(shouldContinue).To(BeTrue())
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(fetchErrorStreak).To(Equal(1))
|
||||
})
|
||||
|
||||
It("handles resource not found errors", func() {
|
||||
var sub *nats.Subscription
|
||||
fetchErrorStreak := 0
|
||||
resourceErr := errors.New("stream not found")
|
||||
shouldContinue, err := fetcher.HandleFetchError(ctx, resourceErr, &sub, &fetchErrorStreak)
|
||||
// Simplified error handling just applies backoff and continues
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(shouldContinue).To(BeTrue())
|
||||
Expect(fetchErrorStreak).To(Equal(1))
|
||||
})
|
||||
|
||||
It("handles generic errors with backoff", func() {
|
||||
// We need a non-nil subscription to avoid recovery attempt
|
||||
dummySub := &nats.Subscription{}
|
||||
sub := dummySub
|
||||
|
||||
fetchErrorStreak := 0
|
||||
genericErr := errors.New("generic error")
|
||||
shouldContinue, err := fetcher.HandleFetchError(ctx, genericErr, &sub, &fetchErrorStreak)
|
||||
Expect(shouldContinue).To(BeTrue())
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(fetchErrorStreak).To(Equal(1))
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -1,47 +0,0 @@
|
||||
package nats
|
||||
|
||||
import (
|
||||
"github.com/nats-io/nats.go"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"go.uber.org/zap/zaptest"
|
||||
)
|
||||
|
||||
var _ = Describe("MessageHandler", func() {
|
||||
var (
|
||||
processor *defaultBatchProcessor
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
processor = &defaultBatchProcessor{
|
||||
logger: zaptest.NewLogger(GinkgoT()),
|
||||
streamName: "TEST_STREAM",
|
||||
consumerName: "test-consumer",
|
||||
}
|
||||
})
|
||||
|
||||
Describe("resolveMsgID", func() {
|
||||
It("extracts message ID from header", func() {
|
||||
msg := &nats.Msg{
|
||||
Header: nats.Header{},
|
||||
}
|
||||
msg.Header.Set("Nats-Msg-Id", "msg-123")
|
||||
|
||||
id, source, err := processor.resolveMsgID(msg)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(id).To(Equal("msg-123"))
|
||||
Expect(source).To(Equal("header"))
|
||||
})
|
||||
|
||||
It("returns error when header and metadata are missing", func() {
|
||||
msg := &nats.Msg{
|
||||
Header: nats.Header{},
|
||||
}
|
||||
|
||||
// Without metadata, this should return an error
|
||||
_, _, err := processor.resolveMsgID(msg)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("fetch metadata"))
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -1,326 +0,0 @@
|
||||
package nats
|
||||
|
||||
import (
|
||||
"caatsm/internal/app"
|
||||
"caatsm/internal/infra/log"
|
||||
obsmetrics "caatsm/internal/infra/metrics"
|
||||
"caatsm/internal/infra/telemetry"
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
// MessageProcessor defines the interface for processing message batches
|
||||
type MessageProcessor interface {
|
||||
ProcessBatch(ctx context.Context, msgs []*nats.Msg)
|
||||
ProcessMessage(ctx context.Context, msg *nats.Msg) error
|
||||
}
|
||||
|
||||
// ProcessingErrorResult represents the result of handling a processing error
|
||||
type ProcessingErrorResult struct {
|
||||
IsPermanent bool
|
||||
ShouldApplyBackpressure bool
|
||||
BackpressureDelay time.Duration
|
||||
}
|
||||
|
||||
// defaultBatchProcessor implements MessageProcessor interface
|
||||
type defaultBatchProcessor struct {
|
||||
processor *app.MessageProcessor
|
||||
dlqHandler DLQHandler
|
||||
logger *zap.Logger
|
||||
telemetry telemetry.Recorder
|
||||
// Configuration needed for processing
|
||||
streamName string
|
||||
consumerName string
|
||||
backoff []time.Duration
|
||||
// Pointer to consecutive errors counter (shared with Consumer)
|
||||
consecutiveProcessErrors *int
|
||||
}
|
||||
|
||||
func (p *defaultBatchProcessor) ProcessBatch(ctx context.Context, msgs []*nats.Msg) {
|
||||
for _, msg := range msgs {
|
||||
// Check context before processing each message
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
p.logger.Info("Stopping batch processing due to cancellation",
|
||||
zap.Int("remaining_messages", len(msgs)),
|
||||
)
|
||||
return
|
||||
default:
|
||||
}
|
||||
p.processSingleMessage(ctx, msg)
|
||||
}
|
||||
}
|
||||
|
||||
// processSingleMessage processes a single message with error handling and backpressure.
|
||||
func (p *defaultBatchProcessor) processSingleMessage(ctx context.Context, msg *nats.Msg) {
|
||||
start := time.Now()
|
||||
|
||||
if err := p.ProcessMessage(ctx, msg); err != nil {
|
||||
p.handleMessageError(ctx, msg, err, time.Since(start))
|
||||
return
|
||||
}
|
||||
|
||||
// Successful processing resets the error streak.
|
||||
if p.consecutiveProcessErrors != nil && *p.consecutiveProcessErrors > 0 {
|
||||
*p.consecutiveProcessErrors = 0
|
||||
}
|
||||
|
||||
elapsed := time.Since(start)
|
||||
|
||||
// ACK the message
|
||||
if ackErr := msg.Ack(); ackErr != nil {
|
||||
p.logger.Error("Failed to ACK message", zap.Error(ackErr))
|
||||
// Still record metrics even if ACK fails
|
||||
}
|
||||
p.telemetry.RecordMessageHandled(ctx, p.streamName, p.consumerName, "ok", elapsed)
|
||||
}
|
||||
|
||||
// ProcessMessage processes a single message.
|
||||
func (p *defaultBatchProcessor) ProcessMessage(ctx context.Context, msg *nats.Msg) error {
|
||||
ctx, span := otel.Tracer("caatsm/nats").Start(ctx, "Consumer.processMessage")
|
||||
defer span.End()
|
||||
|
||||
// Set semantic messaging attributes
|
||||
span.SetAttributes(
|
||||
attribute.String("messaging.system", "nats"),
|
||||
attribute.String("messaging.operation.name", "receive"),
|
||||
attribute.String("messaging.destination.name", msg.Subject),
|
||||
attribute.String("messaging.consumer.group.name", p.consumerName),
|
||||
attribute.String("caatsm.stream", p.streamName),
|
||||
)
|
||||
|
||||
msgID, source, err := p.resolveMsgID(msg)
|
||||
if err != nil {
|
||||
span.RecordError(err)
|
||||
span.SetStatus(codes.Error, err.Error())
|
||||
return fmt.Errorf("unable to resolve message id: %w", err)
|
||||
}
|
||||
if source != "header" {
|
||||
p.logger.Warn("Message missing NATS id header; using fallback",
|
||||
zap.String("subject", msg.Subject),
|
||||
zap.String("msg_id_source", source),
|
||||
zap.String("msg_id", msgID),
|
||||
)
|
||||
}
|
||||
|
||||
// Attach structured logging context including stream/consumer and NATS metadata.
|
||||
jsSeq := uint64(0)
|
||||
if meta, metaErr := msg.Metadata(); metaErr == nil {
|
||||
jsSeq = meta.Sequence.Stream
|
||||
span.SetAttributes(
|
||||
attribute.Int64("nats.js.stream_seq", int64(meta.Sequence.Stream)),
|
||||
attribute.Int64("nats.js.consumer_seq", int64(meta.Sequence.Consumer)),
|
||||
)
|
||||
}
|
||||
|
||||
msgLogger := log.WithMessageContext(p.logger, log.MessageFields{
|
||||
Service: "caatsm-consumer",
|
||||
TransportMsgID: msgID,
|
||||
Stream: p.streamName,
|
||||
Consumer: p.consumerName,
|
||||
Subject: msg.Subject,
|
||||
JSSequence: jsSeq,
|
||||
})
|
||||
|
||||
msgLogger.Debug("Processing message",
|
||||
zap.Int("data_size", len(msg.Data)),
|
||||
zap.String("msg_id_source", source),
|
||||
)
|
||||
|
||||
// Call processor
|
||||
if err := p.processor.Handle(ctx, msg.Data, msgID); err != nil {
|
||||
span.RecordError(err)
|
||||
span.SetStatus(codes.Error, err.Error())
|
||||
return fmt.Errorf("processor error: %w", err)
|
||||
}
|
||||
|
||||
span.SetAttributes(attribute.String("telegram.msg_id", msgID))
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveMsgID extracts or generates a message ID.
|
||||
func (p *defaultBatchProcessor) resolveMsgID(msg *nats.Msg) (string, string, error) {
|
||||
if id := msg.Header.Get("Nats-Msg-Id"); id != "" {
|
||||
return id, "header", nil
|
||||
}
|
||||
|
||||
meta, err := msg.Metadata()
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("fetch metadata: %w", err)
|
||||
}
|
||||
|
||||
return fmt.Sprintf("js-%d", meta.Sequence.Stream), "metadata", nil
|
||||
}
|
||||
|
||||
// handleMessageError handles errors that occur during message processing.
|
||||
func (p *defaultBatchProcessor) handleMessageError(ctx context.Context, msg *nats.Msg, err error, elapsed time.Duration) {
|
||||
// Check if context is cancelled before processing
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
p.logger.Warn("Skipping error handling due to context cancellation",
|
||||
zap.String("subject", msg.Subject),
|
||||
)
|
||||
return
|
||||
default:
|
||||
}
|
||||
|
||||
// Extract message ID for better error logging
|
||||
msgID, _, _ := p.resolveMsgID(msg)
|
||||
if msgID == "" {
|
||||
msgID = "unknown"
|
||||
}
|
||||
|
||||
isPermanent := app.IsPermanent(err)
|
||||
p.logger.Error("Failed to process message",
|
||||
zap.String("subject", msg.Subject),
|
||||
zap.String("msg_id", msgID),
|
||||
zap.Error(err),
|
||||
zap.Bool("permanent", isPermanent),
|
||||
)
|
||||
|
||||
result := obsmetrics.ResultFail
|
||||
if isPermanent {
|
||||
result = obsmetrics.ResultPermanentFail
|
||||
}
|
||||
p.telemetry.RecordMessageHandled(ctx, p.streamName, p.consumerName, result, elapsed)
|
||||
|
||||
consecutiveErrors := 0
|
||||
if p.consecutiveProcessErrors != nil {
|
||||
consecutiveErrors = *p.consecutiveProcessErrors
|
||||
}
|
||||
|
||||
processingResult := ProcessingErrorResult{IsPermanent: isPermanent}
|
||||
if !isPermanent && consecutiveErrors >= 10 {
|
||||
processingResult.ShouldApplyBackpressure = true
|
||||
processingResult.BackpressureDelay = time.Duration(consecutiveErrors) * 100 * time.Millisecond
|
||||
if processingResult.BackpressureDelay > 5*time.Second {
|
||||
processingResult.BackpressureDelay = 5 * time.Second
|
||||
}
|
||||
}
|
||||
|
||||
if processingResult.IsPermanent {
|
||||
p.handlePermanentError(ctx, msg, err)
|
||||
return
|
||||
}
|
||||
|
||||
p.handleTransientError(ctx, msg, processingResult)
|
||||
}
|
||||
|
||||
// handlePermanentError handles permanent/poison messages.
|
||||
func (p *defaultBatchProcessor) handlePermanentError(ctx context.Context, msg *nats.Msg, err error) {
|
||||
if p.consecutiveProcessErrors != nil {
|
||||
*p.consecutiveProcessErrors = 0
|
||||
}
|
||||
|
||||
// Extract message ID for better logging
|
||||
msgID, _, _ := p.resolveMsgID(msg)
|
||||
if msgID == "" {
|
||||
msgID = "unknown"
|
||||
}
|
||||
|
||||
// Poison/permanent message: route to DLQ if configured, then ACK
|
||||
dlqRouted := false
|
||||
if p.dlqHandler != nil {
|
||||
if dlqErr := p.dlqHandler.RouteToDLQ(ctx, msg, err); dlqErr != nil {
|
||||
p.logger.Error("Failed to route permanent-error message to DLQ",
|
||||
zap.String("subject", msg.Subject),
|
||||
zap.String("msg_id", msgID),
|
||||
zap.Error(dlqErr),
|
||||
zap.NamedError("original_error", err),
|
||||
)
|
||||
// Note: We still ACK the message even if DLQ routing fails to prevent
|
||||
// infinite redelivery of poison messages. The error is logged for manual investigation.
|
||||
} else {
|
||||
dlqRouted = true
|
||||
p.logger.Info("Permanent-error message routed to DLQ",
|
||||
zap.String("subject", msg.Subject),
|
||||
zap.String("msg_id", msgID),
|
||||
)
|
||||
}
|
||||
} else {
|
||||
p.logger.Warn("Permanent-error message but DLQ handler not configured - message will be ACKed without DLQ routing",
|
||||
zap.String("subject", msg.Subject),
|
||||
zap.String("msg_id", msgID),
|
||||
zap.String("hint", "Enable DLQ by setting dlq.enabled=true and dlq.subject in config to route poison messages for inspection"),
|
||||
)
|
||||
}
|
||||
|
||||
// ACK the message to prevent redelivery
|
||||
// Even if DLQ routing failed, we ACK to avoid infinite retries of poison messages
|
||||
if ackErr := msg.Ack(); ackErr != nil {
|
||||
p.logger.Error("Failed to ACK permanent-error message",
|
||||
zap.String("subject", msg.Subject),
|
||||
zap.String("msg_id", msgID),
|
||||
zap.Bool("dlq_routed", dlqRouted),
|
||||
zap.Error(ackErr),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// handleTransientError handles transient errors with backpressure and redelivery.
|
||||
func (p *defaultBatchProcessor) handleTransientError(ctx context.Context, msg *nats.Msg, processingResult ProcessingErrorResult) {
|
||||
// Increment error streak
|
||||
if p.consecutiveProcessErrors != nil {
|
||||
if *p.consecutiveProcessErrors < 0 {
|
||||
*p.consecutiveProcessErrors = 0
|
||||
}
|
||||
*p.consecutiveProcessErrors++
|
||||
}
|
||||
|
||||
if processingResult.ShouldApplyBackpressure {
|
||||
consecutiveErrors := 0
|
||||
if p.consecutiveProcessErrors != nil {
|
||||
consecutiveErrors = *p.consecutiveProcessErrors
|
||||
}
|
||||
p.logger.Warn("Applying backpressure due to consecutive processing errors",
|
||||
zap.Int("consecutive_errors", consecutiveErrors),
|
||||
zap.Duration("sleep", processingResult.BackpressureDelay),
|
||||
)
|
||||
// Use context-aware sleep instead of blocking time.Sleep
|
||||
if !sleepWithContext(ctx, processingResult.BackpressureDelay) {
|
||||
// Context canceled, stop processing
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Transient error: request redelivery with optional delay
|
||||
p.telemetry.RecordRetry(ctx, p.streamName, p.consumerName, obsmetrics.RetryReasonProcessorError)
|
||||
if nakErr := p.nakWithStrategy(msg); nakErr != nil {
|
||||
p.logger.Error("Failed to NAK message", zap.Error(nakErr))
|
||||
}
|
||||
}
|
||||
|
||||
// nakWithStrategy sends a NAK with appropriate delay based on retry attempt.
|
||||
func (p *defaultBatchProcessor) nakWithStrategy(msg *nats.Msg) error {
|
||||
if len(p.backoff) == 0 {
|
||||
return msg.Nak()
|
||||
}
|
||||
|
||||
meta, err := msg.Metadata()
|
||||
if err != nil {
|
||||
p.logger.Warn("Failed to read metadata for backoff strategy", zap.Error(err))
|
||||
return msg.Nak()
|
||||
}
|
||||
|
||||
attempt := int(meta.NumDelivered)
|
||||
index := attempt - 1
|
||||
if index < 0 {
|
||||
index = 0
|
||||
}
|
||||
if index >= len(p.backoff) {
|
||||
index = len(p.backoff) - 1
|
||||
}
|
||||
delay := p.backoff[index]
|
||||
if delay <= 0 {
|
||||
return msg.Nak()
|
||||
}
|
||||
|
||||
return msg.NakWithDelay(delay)
|
||||
}
|
||||
@@ -1,37 +0,0 @@
|
||||
package nats
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
"go.uber.org/zap/zaptest"
|
||||
)
|
||||
|
||||
var _ = Describe("Metrics", func() {
|
||||
var (
|
||||
c *Consumer
|
||||
ctx context.Context
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
ctx = context.Background()
|
||||
c = &Consumer{
|
||||
config: consumerConfig{
|
||||
streamName: "TEST_STREAM",
|
||||
consumerName: "test-consumer",
|
||||
monitorInterval: 30 * time.Second, // Set a valid interval
|
||||
},
|
||||
logger: zaptest.NewLogger(GinkgoT()),
|
||||
}
|
||||
})
|
||||
|
||||
Describe("emitConsumerStats", func() {
|
||||
It("handles context cancellation", func() {
|
||||
ctx, cancel := context.WithCancel(ctx)
|
||||
cancel()
|
||||
c.emitConsumerStats(ctx)
|
||||
// Should return without panic
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,181 @@
|
||||
package nats
|
||||
|
||||
import (
|
||||
"caatsm/internal/infra/config"
|
||||
obsmetrics "caatsm/internal/infra/metrics"
|
||||
"context"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
// ConsumerMonitor handles health monitoring and stats emission for the consumer.
|
||||
type ConsumerMonitor struct {
|
||||
logger *zap.Logger
|
||||
cfg *config.Config
|
||||
js nats.JetStreamContext
|
||||
|
||||
// Configuration
|
||||
streamName string
|
||||
consumerName string
|
||||
monitorInterval time.Duration
|
||||
|
||||
// State
|
||||
lastMessageTime time.Time
|
||||
lastMessageSequence uint64
|
||||
messageGapMutex sync.RWMutex
|
||||
}
|
||||
|
||||
// NewConsumerMonitor creates a new ConsumerMonitor.
|
||||
func NewConsumerMonitor(
|
||||
logger *zap.Logger,
|
||||
cfg *config.Config,
|
||||
js nats.JetStreamContext,
|
||||
streamName string,
|
||||
consumerName string,
|
||||
monitorInterval time.Duration,
|
||||
) *ConsumerMonitor {
|
||||
return &ConsumerMonitor{
|
||||
logger: logger,
|
||||
cfg: cfg,
|
||||
js: js,
|
||||
streamName: streamName,
|
||||
consumerName: consumerName,
|
||||
monitorInterval: monitorInterval,
|
||||
}
|
||||
}
|
||||
|
||||
// Start begins the monitoring loop.
|
||||
func (m *ConsumerMonitor) Start(ctx context.Context) {
|
||||
ticker := time.NewTicker(m.monitorInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
// Record initial metric (0) to ensure it appears in Prometheus even before first tick
|
||||
m.logger.Info("Starting NATS consumer stats emission goroutine, recording initial pending metric",
|
||||
zap.String("stream", m.streamName),
|
||||
zap.String("consumer", m.consumerName),
|
||||
zap.Duration("interval", m.monitorInterval),
|
||||
)
|
||||
obsmetrics.RecordNATSConsumerPending(m.streamName, m.consumerName, 0)
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
m.emitStats()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// emitStats gathers and records consumer statistics.
|
||||
func (m *ConsumerMonitor) emitStats() {
|
||||
if m.js != nil {
|
||||
info, err := m.js.ConsumerInfo(m.streamName, m.consumerName)
|
||||
if err != nil {
|
||||
m.logger.Warn("Failed to fetch consumer info for pending messages metric",
|
||||
zap.String("stream", m.streamName),
|
||||
zap.String("consumer", m.consumerName),
|
||||
zap.Error(err),
|
||||
)
|
||||
} else {
|
||||
// Record pending messages for monitoring
|
||||
m.logger.Debug("Recording NATS consumer pending messages metric",
|
||||
zap.String("stream", m.streamName),
|
||||
zap.String("consumer", m.consumerName),
|
||||
zap.Uint64("pending", info.NumPending),
|
||||
)
|
||||
obsmetrics.RecordNATSConsumerPending(m.streamName, m.consumerName, info.NumPending)
|
||||
}
|
||||
}
|
||||
|
||||
// Record AFTN health metrics
|
||||
gapSeconds := m.GetMessageGapSeconds()
|
||||
healthy := m.IsHealthy()
|
||||
|
||||
obsmetrics.RecordMessageGap(m.streamName, m.consumerName, gapSeconds)
|
||||
obsmetrics.RecordSerialReaderHealth(m.streamName, m.consumerName, healthy)
|
||||
|
||||
if !healthy {
|
||||
m.logger.Warn("Serial reader appears stalled - no messages received recently",
|
||||
zap.String("stream", m.streamName),
|
||||
zap.String("consumer", m.consumerName),
|
||||
zap.Float64("gap_seconds", gapSeconds),
|
||||
zap.Duration("threshold", m.cfg.AFTN.MessageGapThreshold),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// TrackMessage updates health metrics based on a received message.
|
||||
func (m *ConsumerMonitor) TrackMessage(msg *nats.Msg) {
|
||||
if msg == nil {
|
||||
return
|
||||
}
|
||||
|
||||
m.messageGapMutex.Lock()
|
||||
defer m.messageGapMutex.Unlock()
|
||||
|
||||
now := time.Now()
|
||||
m.lastMessageTime = now
|
||||
|
||||
// Extract sequence number from message metadata
|
||||
if meta, err := msg.Metadata(); err == nil {
|
||||
currentSeq := meta.Sequence.Stream
|
||||
|
||||
// Detect sequence gaps if we have a previous sequence
|
||||
if m.lastMessageSequence > 0 && m.cfg.AFTN.EnableSequenceGapDetection {
|
||||
if currentSeq > m.lastMessageSequence+1 {
|
||||
gapSize := currentSeq - m.lastMessageSequence - 1
|
||||
m.logger.Warn("Message sequence gap detected",
|
||||
zap.String("stream", m.streamName),
|
||||
zap.String("consumer", m.consumerName),
|
||||
zap.Uint64("last_sequence", m.lastMessageSequence),
|
||||
zap.Uint64("current_sequence", currentSeq),
|
||||
zap.Uint64("gap_size", gapSize),
|
||||
)
|
||||
obsmetrics.RecordSequenceGap(m.streamName, m.consumerName, gapSize)
|
||||
}
|
||||
}
|
||||
|
||||
m.lastMessageSequence = currentSeq
|
||||
}
|
||||
}
|
||||
|
||||
// GetMessageGapSeconds returns the number of seconds since the last message was received.
|
||||
func (m *ConsumerMonitor) GetMessageGapSeconds() float64 {
|
||||
m.messageGapMutex.RLock()
|
||||
defer m.messageGapMutex.RUnlock()
|
||||
|
||||
if m.lastMessageTime.IsZero() {
|
||||
return 0
|
||||
}
|
||||
|
||||
return time.Since(m.lastMessageTime).Seconds()
|
||||
}
|
||||
|
||||
// IsHealthy returns true if messages are being received within the threshold.
|
||||
func (m *ConsumerMonitor) IsHealthy() bool {
|
||||
m.messageGapMutex.RLock()
|
||||
defer m.messageGapMutex.RUnlock()
|
||||
|
||||
// If we haven't received any messages yet, consider it healthy (initial state)
|
||||
if m.lastMessageTime.IsZero() {
|
||||
return true
|
||||
}
|
||||
|
||||
gap := time.Since(m.lastMessageTime)
|
||||
return gap < m.cfg.AFTN.MessageGapThreshold
|
||||
}
|
||||
|
||||
// RecordInitialPending logs and records the initial pending messages count.
|
||||
// This is exposed to allow recording immediately upon startup.
|
||||
func (m *ConsumerMonitor) RecordInitialPending(pending uint64) {
|
||||
m.logger.Info("Recording initial NATS consumer pending messages metric",
|
||||
zap.String("stream", m.streamName),
|
||||
zap.String("consumer", m.consumerName),
|
||||
zap.Uint64("pending", pending),
|
||||
)
|
||||
obsmetrics.RecordNATSConsumerPending(m.streamName, m.consumerName, pending)
|
||||
}
|
||||
@@ -1,131 +0,0 @@
|
||||
package nats
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
// StreamManager handles JetStream stream lifecycle management
|
||||
type StreamManager struct {
|
||||
js nats.JetStreamContext
|
||||
streamName string
|
||||
subjects []string
|
||||
logger *zap.Logger
|
||||
}
|
||||
|
||||
// StreamConfig holds configuration for creating a JetStream stream
|
||||
type StreamConfig struct {
|
||||
MaxMsgs int64
|
||||
MaxBytes int64
|
||||
MaxAge time.Duration
|
||||
Discard string // "old" or "new"
|
||||
Storage string // "file" or "memory"
|
||||
Replicas int
|
||||
}
|
||||
|
||||
// NewStreamManager creates a new stream manager
|
||||
func NewStreamManager(js nats.JetStreamContext, streamName string, subjects []string, logger *zap.Logger) *StreamManager {
|
||||
return &StreamManager{
|
||||
js: js,
|
||||
streamName: streamName,
|
||||
subjects: subjects,
|
||||
logger: logger,
|
||||
}
|
||||
}
|
||||
|
||||
// EnsureStream ensures that the configured JetStream stream exists, creating it if necessary
|
||||
func (sm *StreamManager) EnsureStream(cfg *StreamConfig) error {
|
||||
// Check if stream already exists
|
||||
info, err := sm.js.StreamInfo(sm.streamName)
|
||||
if err == nil {
|
||||
// Stream exists - check if we need to add any missing subjects
|
||||
existingSubjects := make(map[string]bool)
|
||||
for _, subj := range info.Config.Subjects {
|
||||
existingSubjects[subj] = true
|
||||
}
|
||||
|
||||
// Check if any configured subjects are missing
|
||||
missingSubjects := []string{}
|
||||
for _, subj := range sm.subjects {
|
||||
if !existingSubjects[subj] {
|
||||
missingSubjects = append(missingSubjects, subj)
|
||||
}
|
||||
}
|
||||
|
||||
if len(missingSubjects) > 0 {
|
||||
// Update stream to include missing subjects
|
||||
updatedSubjects := info.Config.Subjects
|
||||
updatedSubjects = append(updatedSubjects, missingSubjects...)
|
||||
info.Config.Subjects = updatedSubjects
|
||||
|
||||
_, updateErr := sm.js.UpdateStream(&info.Config)
|
||||
if updateErr != nil {
|
||||
return fmt.Errorf("failed to update stream %s with new subjects %v: %w", sm.streamName, missingSubjects, updateErr)
|
||||
}
|
||||
|
||||
sm.logger.Info("Updated JetStream stream with new subjects",
|
||||
zap.String("stream", sm.streamName),
|
||||
zap.Strings("added_subjects", missingSubjects),
|
||||
zap.Strings("all_subjects", updatedSubjects),
|
||||
)
|
||||
} else {
|
||||
sm.logger.Info("JetStream stream verified",
|
||||
zap.String("stream", sm.streamName),
|
||||
zap.Strings("subjects", sm.subjects),
|
||||
)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// If stream doesn't exist, create it
|
||||
if errors.Is(err, nats.ErrStreamNotFound) {
|
||||
streamCfg := &nats.StreamConfig{
|
||||
Name: sm.streamName,
|
||||
Subjects: sm.subjects,
|
||||
}
|
||||
|
||||
// Apply limits if provided
|
||||
if cfg != nil {
|
||||
if cfg.MaxMsgs > 0 {
|
||||
streamCfg.MaxMsgs = cfg.MaxMsgs
|
||||
}
|
||||
if cfg.MaxBytes > 0 {
|
||||
streamCfg.MaxBytes = cfg.MaxBytes
|
||||
}
|
||||
if cfg.MaxAge > 0 {
|
||||
streamCfg.MaxAge = cfg.MaxAge
|
||||
}
|
||||
if cfg.Discard == "new" {
|
||||
streamCfg.Discard = nats.DiscardNew
|
||||
} else {
|
||||
streamCfg.Discard = nats.DiscardOld
|
||||
}
|
||||
if cfg.Storage == "memory" {
|
||||
streamCfg.Storage = nats.MemoryStorage
|
||||
} else {
|
||||
streamCfg.Storage = nats.FileStorage
|
||||
}
|
||||
if cfg.Replicas > 0 {
|
||||
streamCfg.Replicas = cfg.Replicas
|
||||
}
|
||||
}
|
||||
|
||||
_, err := sm.js.AddStream(streamCfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create stream %s: %w", sm.streamName, err)
|
||||
}
|
||||
|
||||
sm.logger.Info("Created JetStream stream",
|
||||
zap.String("stream", sm.streamName),
|
||||
zap.Strings("subjects", sm.subjects),
|
||||
)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Other error (e.g., permission denied)
|
||||
return fmt.Errorf("stream %s not found or inaccessible: %w", sm.streamName, err)
|
||||
}
|
||||
@@ -42,6 +42,9 @@ type Recorder interface {
|
||||
|
||||
// RecordJSAPICall records a JetStream API call.
|
||||
RecordJSAPICall(operation string)
|
||||
|
||||
// RecordAFTNValidationError records an AFTN protocol validation failure.
|
||||
RecordAFTNValidationError(ctx context.Context, errorType string)
|
||||
}
|
||||
|
||||
// ProvideRecorder wires a composite Recorder based on configuration flags.
|
||||
@@ -100,6 +103,9 @@ func (n *noopRecorder) RecordDLQPublishFailure(ctx context.Context, stream, cons
|
||||
func (n *noopRecorder) RecordJSAPICall(operation string) {
|
||||
}
|
||||
|
||||
func (n *noopRecorder) RecordAFTNValidationError(ctx context.Context, errorType string) {
|
||||
}
|
||||
|
||||
// compositeRecorder fans out all calls to a slice of underlying recorders.
|
||||
type compositeRecorder struct {
|
||||
recorders []Recorder
|
||||
@@ -167,6 +173,12 @@ func (c *compositeRecorder) RecordJSAPICall(operation string) {
|
||||
}
|
||||
}
|
||||
|
||||
func (c *compositeRecorder) RecordAFTNValidationError(ctx context.Context, errorType string) {
|
||||
for _, r := range c.recorders {
|
||||
r.RecordAFTNValidationError(ctx, errorType)
|
||||
}
|
||||
}
|
||||
|
||||
// promRecorder delegates to the Prometheus metrics helpers in the
|
||||
// internal/infra/metrics package.
|
||||
type promRecorder struct{}
|
||||
@@ -213,6 +225,10 @@ func (p *promRecorder) RecordJSAPICall(operation string) {
|
||||
obsmetrics.RecordJSAPICall(operation)
|
||||
}
|
||||
|
||||
func (p *promRecorder) RecordAFTNValidationError(ctx context.Context, errorType string) {
|
||||
obsmetrics.RecordAFTNValidationError(ctx, errorType)
|
||||
}
|
||||
|
||||
// otelRecorder creates and records OpenTelemetry metrics for the CAATSM
|
||||
// processor. It intentionally focuses on a small set of high-value metrics to
|
||||
// avoid duplicating the full Prometheus surface.
|
||||
@@ -305,4 +321,9 @@ func (o *otelRecorder) RecordDLQPublishFailure(ctx context.Context, stream, cons
|
||||
func (o *otelRecorder) RecordJSAPICall(operation string) {
|
||||
}
|
||||
|
||||
func (o *otelRecorder) RecordAFTNValidationError(ctx context.Context, errorType string) {
|
||||
// AFTN validation metrics are primarily tracked via Prometheus.
|
||||
// This is a no-op for OTEL recorder.
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
package port
|
||||
|
||||
import "caatsm/internal/domain/weather"
|
||||
|
||||
// WeatherParser defines the interface for parsing weather reports
|
||||
type WeatherParser interface {
|
||||
// CanParse determines if the raw string can be parsed as a weather report
|
||||
CanParse(raw string) bool
|
||||
|
||||
// Parse parses a raw weather report string and returns a WeatherMessage
|
||||
Parse(raw string) (weather.WeatherMessage, error)
|
||||
}
|
||||
|
||||
+5
-1
@@ -4,6 +4,7 @@ package di
|
||||
|
||||
import (
|
||||
"caatsm/internal/adapter/parser"
|
||||
weatherparser "caatsm/internal/adapter/parser/weather"
|
||||
"caatsm/internal/app"
|
||||
"caatsm/internal/infra/config"
|
||||
"caatsm/internal/infra/log"
|
||||
@@ -46,7 +47,10 @@ var runtimeSet = wire.NewSet(
|
||||
nats.ProvideJetStream,
|
||||
nats.ProvidePublisher,
|
||||
|
||||
// Parser
|
||||
// Weather Parser
|
||||
weatherparser.NewWeatherParser,
|
||||
|
||||
// Parser (composite, depends on weather parser)
|
||||
parser.ProvideParser,
|
||||
|
||||
// Telemetry
|
||||
|
||||
+8
-5
@@ -8,6 +8,7 @@ package di
|
||||
|
||||
import (
|
||||
"caatsm/internal/adapter/parser"
|
||||
"caatsm/internal/adapter/parser/weather"
|
||||
"caatsm/internal/app"
|
||||
"caatsm/internal/infra/config"
|
||||
"caatsm/internal/infra/log"
|
||||
@@ -21,7 +22,8 @@ import (
|
||||
// Injectors from wire.go:
|
||||
|
||||
func buildAppComponents() (*appComponents, error) {
|
||||
parserParser := parser.ProvideParser()
|
||||
weatherParser := weather.NewWeatherParser()
|
||||
parserParser := parser.ProvideParser(weatherParser)
|
||||
configConfig, err := config.ProvideConfig()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -51,7 +53,7 @@ func buildAppComponents() (*appComponents, error) {
|
||||
return nil, err
|
||||
}
|
||||
recorder := telemetry.ProvideRecorder(configConfig)
|
||||
messageProcessor := app.NewMessageProcessor(parserParser, repository, publisher, recorder, logger)
|
||||
messageProcessor := app.NewMessageProcessor(parserParser, repository, publisher, recorder, logger, configConfig)
|
||||
consumer, err := nats.ProvideConsumer(conn, jetStreamContext, messageProcessor, configConfig, recorder, logger)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -69,7 +71,8 @@ func buildAppComponents() (*appComponents, error) {
|
||||
}
|
||||
|
||||
func buildAppComponentsWithConfig(cfg *config.Config) (*appComponents, error) {
|
||||
parserParser := parser.ProvideParser()
|
||||
weatherParser := weather.NewWeatherParser()
|
||||
parserParser := parser.ProvideParser(weatherParser)
|
||||
logger, err := log.ProvideLogger(cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -95,7 +98,7 @@ func buildAppComponentsWithConfig(cfg *config.Config) (*appComponents, error) {
|
||||
return nil, err
|
||||
}
|
||||
recorder := telemetry.ProvideRecorder(cfg)
|
||||
messageProcessor := app.NewMessageProcessor(parserParser, repository, publisher, recorder, logger)
|
||||
messageProcessor := app.NewMessageProcessor(parserParser, repository, publisher, recorder, logger, cfg)
|
||||
consumer, err := nats.ProvideConsumer(conn, jetStreamContext, messageProcessor, cfg, recorder, logger)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -132,7 +135,7 @@ func InitializeAppWithConfig(cfg *config.Config) (*app.MessageProcessor, *nats.C
|
||||
return comps.Processor, comps.Consumer, comps.Monitoring, nil
|
||||
}
|
||||
|
||||
var runtimeSet = wire.NewSet(log.ProvideLogger, postgres.ProvideDB, postgres.ProvideRepository, nats.ProvideNATSConn, nats.ProvideJetStream, nats.ProvidePublisher, parser.ProvideParser, telemetry.ProvideRecorder, app.NewMessageProcessor, nats.ProvideConsumer, monitoring.ProvideServer)
|
||||
var runtimeSet = wire.NewSet(log.ProvideLogger, postgres.ProvideDB, postgres.ProvideRepository, nats.ProvideNATSConn, nats.ProvideJetStream, nats.ProvidePublisher, weather.NewWeatherParser, parser.ProvideParser, telemetry.ProvideRecorder, app.NewMessageProcessor, nats.ProvideConsumer, monitoring.ProvideServer)
|
||||
|
||||
type appComponents struct {
|
||||
Processor *app.MessageProcessor
|
||||
|
||||
@@ -82,7 +82,7 @@ func TestJetStreamToTimescaleFlow(t *testing.T) {
|
||||
}
|
||||
|
||||
telemetryRecorder := telemetryinfra.NewNoop()
|
||||
proc := app.NewMessageProcessor(parser.ProvideParser(), repo, publisher, telemetryRecorder, logger)
|
||||
proc := app.NewMessageProcessor(parser.ProvideParser(), repo, publisher, telemetryRecorder, logger, cfg)
|
||||
consumer, err := natsinfra.ProvideConsumer(conn, js, proc, cfg, telemetryRecorder, logger)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to init consumer: %v", err)
|
||||
|
||||
Reference in New Issue
Block a user