vps/runbooks/pdns-health.md

PowerDNS health (hk2)

Purpose

Read-only health check of the /opt/pdns PowerDNS stack.

Scope

• Applicable: hk2.chans.xyz, /opt/pdns.
• Read-only: does not change PowerDNS, DNS records, or secrets.

Read-only checks for the /opt/pdns stack on hk2.chans.xyz (ns1.wsvc.info).

Facts: hosts/hk2.chans.xyz.md · Upstream: docs/pdns-upstream.md

Prefer the Ansible health report (ansible/playbooks/health-report.yml with --limit powerdns) for routine checks. Keep the SSH commands below for focused diagnosis, especially direct authoritative DNS queries.

Focused SSH diagnostics

Use these only after the Ansible health report needs investigation.

Quick remote

ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose ps -a'

Expect auth, db, poweradmin healthy; backup Up; pgweb Up. The legacy PDA orphan powerdns-admin was removed 2026-08-12 (W1N-59).

Version / security poll

Expect Authoritative 5.0.6 (or newer). Startup logs should not show a Mandatory Security Update banner. If they do, check https://doc.powerdns.com/authoritative/security-advisories/ and bump the powerdns/pdns-auth-50 image pin in /opt/pdns/compose.yml.

ssh -4 windy@hk2.chans.xyz 'docker logs pdns-auth 2>&1 | grep -i "Security Update" | tail -5'
ssh -4 windy@hk2.chans.xyz 'docker inspect pdns-auth --format "{{.Config.Image}}"'

DNS (from laptop / WSL)

Prefer querying auth/secondary IPs directly (local stub / public resolvers may be flaky or cached):

NS1=154.36.174.161
NS2=202.91.35.141
for z in windy.me wsvc.info chans.xyz; do
  echo "== $z =="
  dig @$NS1 +short NS "$z"
  dig @$NS1 +short SOA "$z"
  dig @$NS2 +short SOA "$z"
done
dig @$NS1 +short A ns1.wsvc.info   # expect 154.36.174.161
dig @$NS1 +short A ns2.wsvc.info   # expect 202.91.35.141

Served SOA serial can differ from the backend/API serial when SOA-EDIT=INCEPTION-INCREMENT is active — compare ns1 vs ns2 served SOA, not dig vs pdnsutil/API.

API (on host)

ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose exec -T auth \
  python3 -c "import json,os,urllib.request; print(json.load(urllib.request.urlopen(urllib.request.Request(\"http://127.0.0.1:8081/api/v1/servers/localhost\", headers={\"X-API-Key\": os.environ[\"PDNS_API_KEY\"]})))[\"version\")"'

Or use host loopback with the key from .env (do not paste the key into chat/logs).

Web UIs

# If local stub DNS is flaky, pin SNI to the auth IP:
curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pdns.wsvc.info:443:154.36.174.161 https://pdns.wsvc.info/   # 302 → login
curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pgweb.wsvc.info:443:154.36.174.161 https://pgweb.wsvc.info/  # 401 without basic auth

Backup

ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && ./scripts/manual-backup.sh && ls -lt backup/*.sql.gz | head'
ssh -4 windy@hk2.chans.xyz 'tail -30 /opt/pdns/backup/backup.log'

Scheduler should log scheduler started … (daily 03:00). If you see can't execute 'bash' or Alpine apk errors, the backup image/entrypoint was reverted — see hosts/hk2.chans.xyz.md.

Config sanity (docs-aligned)

ssh -4 windy@hk2.chans.xyz 'grep -E "^(primary|also-notify|only-notify|allow-axfr-ips|gpgsql-dnssec|default-soa-edit)" /opt/pdns/auth/pdns.conf'

Expect: primary=yes, also-notify=202.91.35.141, only-notify= empty, gpgsql-dnssec=yes, SOA-EDIT INCEPTION-INCREMENT.

Scheduled local check

The sanitized Ansible health profile is pdns (ansible/playbooks/healthchecks.yml). It runs locally through vps-healthcheck.timer, writes a sanitized JSON result to /var/lib/vps-health/latest.json, and uses the API key only inside the PowerDNS container. It does not modify PowerDNS, DNS records, or secrets.

Safety

• Read-only: never mutate PowerDNS configuration or DNS records during this check.
• Do not paste the API key into chat/logs.
• If live state conflicts with an expected value, STOP and report.

After config changes

• auth/pdns.conf, auth/templates.d/secrets.j2, or auth-related .env → use the Ansible Compose reconcile playbook with target auth
• Poweradmin env / Traefik labels → use the Ansible Compose reconcile playbook with target poweradmin; add service_reconcile_restart_traefik=true only if its UI returns 404
• backup schedule / retention env → use the Ansible Compose reconcile playbook with target backup