vps/runbooks/pdns-health.md

# PowerDNS health (hk2)

## Purpose

Read-only health check of the `/opt/pdns` PowerDNS stack.

## Scope

- Applicable: [hk2.chans.xyz](../hosts/hk2.chans.xyz.md), `/opt/pdns`.
- Read-only: does not change PowerDNS, DNS records, or secrets.

Read-only checks for the `/opt/pdns` stack on **hk2.chans.xyz** (`ns1.wsvc.info`).

Facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) · Upstream: [docs/pdns-upstream.md](../docs/pdns-upstream.md)

Prefer the Ansible health report (`ansible/playbooks/health-report.yml` with
`--limit powerdns`) for routine checks. Keep the SSH commands below for focused
diagnosis, especially direct authoritative DNS queries.

## Focused SSH diagnostics

Use these only after the Ansible health report needs investigation.

### Quick remote

```bash
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose ps -a'
```

Expect `auth`, `db`, `poweradmin` healthy; `backup` Up; `pgweb` Up. The legacy PDA orphan `powerdns-admin` was removed 2026-08-12 (W1N-59).

### Version / security poll

Expect Authoritative **5.0.6** (or newer). Startup logs should **not** show a Mandatory `Security Update` banner. If they do, check https://doc.powerdns.com/authoritative/security-advisories/ and bump the `powerdns/pdns-auth-50` image pin in `/opt/pdns/compose.yml`.

```bash
ssh -4 windy@hk2.chans.xyz 'docker logs pdns-auth 2>&1 | grep -i "Security Update" | tail -5'
ssh -4 windy@hk2.chans.xyz 'docker inspect pdns-auth --format "{{.Config.Image}}"'
```

### DNS (from laptop / WSL)

Prefer querying auth/secondary IPs directly (local stub / public resolvers may be flaky or cached):

```bash
NS1=154.36.174.161
NS2=202.91.35.141
for z in windy.me wsvc.info chans.xyz; do
  echo "== $z =="
  dig @$NS1 +short NS "$z"
  dig @$NS1 +short SOA "$z"
  dig @$NS2 +short SOA "$z"
done
dig @$NS1 +short A ns1.wsvc.info   # expect 154.36.174.161
dig @$NS1 +short A ns2.wsvc.info   # expect 202.91.35.141
```

Served SOA serial can differ from the backend/API serial when `SOA-EDIT=INCEPTION-INCREMENT` is active — compare **ns1 vs ns2 served SOA**, not dig vs `pdnsutil`/API.

### API (on host)

```bash
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose exec -T auth \
  python3 -c "import json,os,urllib.request; print(json.load(urllib.request.urlopen(urllib.request.Request(\"http://127.0.0.1:8081/api/v1/servers/localhost\", headers={\"X-API-Key\": os.environ[\"PDNS_API_KEY\"]})))[\"version\")"'
```

Or use host loopback with the key from `.env` (do not paste the key into chat/logs).

### Web UIs

```bash
# If local stub DNS is flaky, pin SNI to the auth IP:
curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pdns.wsvc.info:443:154.36.174.161 https://pdns.wsvc.info/   # 302 → login
curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pgweb.wsvc.info:443:154.36.174.161 https://pgweb.wsvc.info/  # 401 without basic auth
```

### Backup

```bash
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && ./scripts/manual-backup.sh && ls -lt backup/*.sql.gz | head'
ssh -4 windy@hk2.chans.xyz 'tail -30 /opt/pdns/backup/backup.log'
```

Scheduler should log `scheduler started … (daily 03:00)`. If you see `can't execute 'bash'` or Alpine `apk` errors, the backup image/entrypoint was reverted — see [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md).

### Config sanity (docs-aligned)

```bash
ssh -4 windy@hk2.chans.xyz 'grep -E "^(primary|also-notify|only-notify|allow-axfr-ips|gpgsql-dnssec|default-soa-edit)" /opt/pdns/auth/pdns.conf'
```

Expect: `primary=yes`, `also-notify=202.91.35.141`, `only-notify=` empty, `gpgsql-dnssec=yes`, SOA-EDIT `INCEPTION-INCREMENT`.

## Scheduled local check

The sanitized Ansible health profile is `pdns` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses the API key only inside the PowerDNS container. It does not modify PowerDNS, DNS records, or secrets.


## Safety

- Read-only: never mutate PowerDNS configuration or DNS records during this check.
- Do not paste the API key into chat/logs.
- If live state conflicts with an expected value, `STOP` and report.

## After config changes

- `auth/pdns.conf`, `auth/templates.d/secrets.j2`, or auth-related `.env` → use the Ansible Compose reconcile playbook with target `auth`
- Poweradmin env / Traefik labels → use the Ansible Compose reconcile playbook with target `poweradmin`; add `service_reconcile_restart_traefik=true` only if its UI returns 404
- backup schedule / retention env → use the Ansible Compose reconcile playbook with target `backup`