go-caatsm/docs/secret-management.md

Secret Management Guide

This document describes best practices for managing secrets and sensitive configuration in the CAATSM application.

Current Approach

The application currently supports secrets via environment variables with the CAATSM_ prefix:

export CAATSM_POSTGRES_URL="postgres://user:password@localhost:5432/aviation"
export CAATSM_NATS_AUTH_TOKEN="your-token"
export CAATSM_NATS_AUTH_PASSWORD="secure-password"

Security considerations:

• Environment variables are visible to all processes on the system
• Secrets may be logged in process lists or shell history
• No automatic rotation or expiration
• Manual management required

Recommended Secret Management Systems

For production deployments, use a dedicated secret management system:

Option 1: HashiCorp Vault (Recommended)

Vault provides secure secret storage with dynamic secrets, encryption, and access control.

Setup

1. Install Vault:

   # Download and install Vault
   wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_linux_amd64.zip
   unzip vault_1.15.0_linux_amd64.zip
   sudo mv vault /usr/local/bin/
   

2. Start Vault (dev mode for testing):

   vault server -dev
   

3. Store secrets:

   export VAULT_ADDR='http://127.0.0.1:8200'
   vault kv put secret/caatsm \
     postgres_url="postgres://user:pass@db:5432/aviation" \
     nats_token="your-token" \
     nats_password="secure-password"
   

Integration

Create a wrapper script or init container to fetch secrets from Vault:

#!/bin/bash
# fetch-secrets.sh

export VAULT_ADDR="${VAULT_ADDR:-http://vault:8200}"
export VAULT_TOKEN="${VAULT_TOKEN}"

# Fetch secrets from Vault
vault kv get -format=json secret/caatsm | jq -r '.data.data | to_entries | .[] | "export CAATSM_\(.key | ascii_upcase | gsub("-"; "_"))=\(.value)"' > /tmp/secrets.env

# Source secrets
source /tmp/secrets.env

# Start application
exec ./bin/receiver listen

Kubernetes Integration

Use Vault Agent Sidecar or Vault Secrets Operator:

apiVersion: v1
kind: Pod
metadata:
  name: caatsm-receiver
spec:
  containers:
  - name: vault-agent
    image: vault:latest
    command: ["/bin/sh", "-c"]
    args:
      - |
        vault agent -config=/vault/config/agent.hcl
  - name: caatsm-receiver
    image: caatsm/receiver:latest
    envFrom:
    - secretRef:
        name: caatsm-secrets

Option 2: AWS Secrets Manager

For AWS deployments, use AWS Secrets Manager for centralized secret management.

Setup

1. Store secrets:

   aws secretsmanager create-secret \
     --name caatsm/production \
     --secret-string '{
       "postgres_url": "postgres://user:pass@db:5432/aviation",
       "nats_token": "your-token",
       "nats_password": "secure-password"
     }'
   

2. Retrieve secrets:

   aws secretsmanager get-secret-value \
     --secret-id caatsm/production \
     --query SecretString \
     --output text | jq -r 'to_entries | .[] | "export CAATSM_\(.key | ascii_upcase | gsub("-"; "_"))=\(.value)"'
   

Integration

Use AWS SDK or CLI in init container:

#!/bin/bash
# fetch-aws-secrets.sh

SECRET_JSON=$(aws secretsmanager get-secret-value \
  --secret-id caatsm/production \
  --query SecretString \
  --output text)

echo "$SECRET_JSON" | jq -r 'to_entries | .[] | "export CAATSM_\(.key | ascii_upcase | gsub("-"; "_"))=\(.value)"' > /tmp/secrets.env

source /tmp/secrets.env
exec ./bin/receiver listen

Option 3: Kubernetes Secrets

For Kubernetes deployments, use Kubernetes Secrets.

Setup

1. Create secret:

   kubectl create secret generic caatsm-secrets \
     --from-literal=postgres-url="postgres://user:pass@db:5432/aviation" \
     --from-literal=nats-token="your-token" \
     --from-literal=nats-password="secure-password"
   

2. Use in deployment:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: caatsm-receiver
   spec:
     template:
       spec:
         containers:
         - name: receiver
           image: caatsm/receiver:latest
           env:
           - name: CAATSM_POSTGRES_URL
             valueFrom:
               secretKeyRef:
                 name: caatsm-secrets
                 key: postgres-url
           - name: CAATSM_NATS_AUTH_TOKEN
             valueFrom:
               secretKeyRef:
                 name: caatsm-secrets
                 key: nats-token
   

Best Practices

• Encrypt at rest: Enable encryption for etcd (Kubernetes backend)
• RBAC: Restrict access to secrets using Role-Based Access Control
• External Secrets Operator: Use External Secrets Operator for integration with external secret stores

Option 4: Docker Secrets

For Docker Swarm deployments, use Docker Secrets.

Setup

1. Create secret:

   echo "your-secret-value" | docker secret create caatsm_nats_token -
   

2. Use in service:

   version: '3.8'
   services:
     receiver:
       image: caatsm/receiver:latest
       secrets:
         - caatsm_nats_token
       environment:
         - CAATSM_NATS_AUTH_TOKEN_FILE=/run/secrets/caatsm_nats_token
   

Secret Rotation

Manual Rotation

1. Update secret in secret management system
2. Restart application to pick up new secret
3. Verify application is working correctly
4. Remove old secret after verification

Automated Rotation

For AWS Secrets Manager, enable automatic rotation:

aws secretsmanager rotate-secret \
  --secret-id caatsm/production \
  --rotation-lambda-arn arn:aws:lambda:region:account:function:rotate-secret

For Vault, use dynamic secrets or scheduled rotation policies.

Security Best Practices

1. Principle of Least Privilege

• Minimal access: Grant only necessary permissions
• Service accounts: Use dedicated service accounts for applications
• Secret scoping: Limit secrets to specific services/environments

2. Encryption

• Encryption at rest: Ensure secrets are encrypted in storage
• Encryption in transit: Use TLS for secret retrieval
• Key management: Use proper key management (HSM, KMS, etc.)

3. Audit and Monitoring

• Audit logs: Enable audit logging for secret access
• Monitoring: Monitor secret access patterns
• Alerts: Set up alerts for unusual access patterns

4. Secret Lifecycle

• Rotation: Rotate secrets regularly (e.g., every 90 days)
• Expiration: Set expiration dates for secrets
• Revocation: Have a process for revoking compromised secrets

5. Development vs Production

• Separate stores: Use different secret stores for dev/staging/prod
• No production secrets in code: Never commit production secrets
• Local development: Use local secret files or dev vault instance

Configuration Examples

Environment Variables (Current)

# Development
export CAATSM_POSTGRES_URL="postgres://user:pass@localhost:5432/aviation?sslmode=disable"
export CAATSM_NATS_URL="nats://localhost:4222"
export CAATSM_NATS_AUTH_TOKEN="dev-token"

# Production (via secret management)
# Secrets loaded from Vault/AWS/K8s before application start

Configuration File (Not Recommended for Secrets)

# config.prod.toml
# DO NOT store secrets in config files
# Use environment variables or secret management instead

[postgres]
# URL should come from CAATSM_POSTGRES_URL env var
url = ""  # Empty, will be overridden by env var

[nats.auth]
# Token should come from CAATSM_NATS_AUTH_TOKEN env var
token = ""  # Empty, will be overridden by env var

Secret Injection Patterns

Pattern 1: Init Container (Kubernetes)

apiVersion: v1
kind: Pod
metadata:
  name: caatsm-receiver
spec:
  initContainers:
  - name: fetch-secrets
    image: vault:latest
    command: ["/bin/sh", "-c"]
    args:
      - |
        vault kv get -format=json secret/caatsm | \
        jq -r '.data.data | to_entries | .[] | "\(.key | ascii_upcase | gsub("-"; "_"))=\(.value)"' > \
        /shared/secrets.env
    volumeMounts:
    - name: shared-secrets
      mountPath: /shared
  containers:
  - name: receiver
    image: caatsm/receiver:latest
    envFrom:
    - configMapRef:
        name: caatsm-config
    env:
    - name: CAATSM_SECRETS_FILE
      value: /shared/secrets.env
    volumeMounts:
    - name: shared-secrets
      mountPath: /shared
  volumes:
  - name: shared-secrets
    emptyDir: {}

Pattern 2: Sidecar Container

apiVersion: v1
kind: Pod
metadata:
  name: caatsm-receiver
spec:
  containers:
  - name: vault-agent
    image: vault:latest
    command: ["vault", "agent", "-config=/vault/config/agent.hcl"]
    volumeMounts:
    - name: vault-config
      mountPath: /vault/config
  - name: receiver
    image: caatsm/receiver:latest
    envFrom:
    - secretRef:
        name: caatsm-secrets
  volumes:
  - name: vault-config
    configMap:
      name: vault-agent-config

Pattern 3: Application-Level Integration

For applications that need to fetch secrets at runtime:

// Example: Fetch secrets from Vault at startup
func loadSecretsFromVault() error {
    client, err := vault.NewClient(vault.DefaultConfig())
    if err != nil {
        return err
    }
    
    secret, err := client.Logical().Read("secret/data/caatsm")
    if err != nil {
        return err
    }
    
    // Set environment variables
    for k, v := range secret.Data["data"].(map[string]interface{}) {
        os.Setenv("CAATSM_"+strings.ToUpper(k), v.(string))
    }
    
    return nil
}

Troubleshooting

Secret Not Found

Symptoms:

• Application fails to start
• Connection errors to database/NATS

Solutions:

• Verify secret exists in secret store
• Check secret name/path is correct
• Verify application has permissions to access secret
• Check secret format (JSON, plain text, etc.)

Secret Access Denied

Symptoms:

• Authentication errors when fetching secrets
• Permission denied errors

Solutions:

• Verify IAM roles/service accounts have correct permissions
• Check Vault policies or AWS IAM policies
• Verify authentication tokens/credentials are valid

Secret Rotation Issues

Symptoms:

• Application fails after secret rotation
• Connection errors after rotation

Solutions:

• Implement graceful secret reloading
• Use connection pooling with automatic reconnection
• Test rotation process in staging first

References

• HashiCorp Vault Documentation
• AWS Secrets Manager Documentation
• Kubernetes Secrets Documentation
• External Secrets Operator
• 12-Factor App: Config