vps/runbooks/pgdb-restore.md

Runbook: pgdb database restore (pg_restore, custom format)

Purpose

Restore one pgdb database (hass, scribe, or postgres) from a pg-backup custom-format dump via pg_restore. The dump → restore path was exercised on 2026-08-29 (hass restore test, states 10014 rows); this runbook makes it repeatable and safe.

Scope

• Applicable: single-database restore on pgdb (192.168.55.15) from /opt/database/backups/{daily,weekly,monthly}/*.dump.
• Not applicable: full-stack/VM recovery, HAOS-side changes, secret rotation, or restoring to a different host (adjust -h and auth first).

Ownership

• Owner: personal ops (Windy)
• Last reviewed: 2026-08-29
• Related systems: pgdb (/opt/database), HA 192.168.55.11 (hass/scribe clients), pgdb-health

Preconditions

• Health baseline recorded (pgdb-health): stack healthy, current backup is from today and passes pg_restore -l.
• A valid dump exists for the target database (see Step 1).
• Explicit user approval obtained for the chosen restore mode (see ## Approval gates) and recorded in the Linear vps project.
• Disk headroom on /srv/pgdata for a second copy of the database (temp-DB mode).

Inputs

Input	Source	Required	Validation
Target database (hass/scribe/postgres)	hosts/pgdb.md	yes	exists in pg_database
Dump path (inside pg-backup, /backups/...)	ls /opt/database/backups/daily/	yes	pg_restore -l lists TOC
Restore mode (temp-DB swap recommended / in-place --clean)	operator + user	yes	approval record
Downtime window	user	yes (for in-place)	user-confirmed

Safety

Non-negotiable rules

• Never delete the existing configuration or the last known-good dump as the first recovery action.
• Prefer read-only diagnosis before any mutation; the restore itself is the only mutation.
• Never print passwords — use PGPASSWORD="$POSTGRES_PASSWORD" inside the pg-backup container.
• If actual state conflicts with this runbook (e.g. dump corrupt, DB missing), STOP.
• Do not invent missing parameters; do not bypass failed checks or approval.

Stop conditions

• Dump fails pg_restore -l (corrupt/truncated) — do not restore from it.
• Target DB contains data newer than the dump and the operator did not accept the data-loss window.
• Disk space insufficient for the temp copy.
• Approval missing or stale (recorded in Linear).
• Post-restore verification fails — do not declare success.

Approval gates

Action	Risk	Explicit approval	Recorded at
Create temp DB + pg_restore into it (no data loss)	Low	No (executor can run after plan review)	Linear vps issue
Drop/rename the live target DB (temp-DB swap)	High	Yes — user confirmation	Linear vps issue
In-place restore with --clean --if-exists	High	Yes — user confirmation	Linear vps issue

Procedure

Step 1 — Diagnose

Action

ssh -F /dev/null -o BatchMode=yes windy@192.168.55.15 'ls -l --time-style=long-iso /opt/database/backups/daily/'
ssh -F /dev/null -o BatchMode=yes windy@192.168.55.15 'cd /opt/database && docker compose exec -T pg-backup sh -c "PGPASSWORD=\"\$POSTGRES_PASSWORD\" pg_restore -h timescaledb -U postgres -l /backups/daily/<db>-latest.dump" | head -6'
ssh -F /dev/null -o BatchMode=yes windy@192.168.55.15 'cd /opt/database && docker compose exec -T timescaledb psql -U postgres -Atc "select datname, pg_size_pretty(pg_database_size(datname)) from pg_database where datistemplate=false order by 1;"'
ssh -F /dev/null -o BatchMode=yes windy@192.168.55.15 'df -h /srv/pgdata'

Expected

• Dump exists for today and is custom format (file -L → PostgreSQL custom database dump).
• pg_restore -l lists the archive TOC (dbname matches the target) without error.
• Target DB exists; disk has headroom for a second copy.

Decision

• Dump valid + DB present → Step 2.
• Dump invalid or target missing → STOP; record evidence, do not restore.

Step 2 — Approve

Action — Present the plan: target DB, dump (path + creation time), restore mode (temp-DB swap recommended), estimated write downtime (none for temp mode; brief for in-place), and rollback plan. Obtain explicit user approval and record it in the Linear vps project issue for this work.

Expected — User approval recorded with the chosen mode.

Decision

• Approved → Step 3.
• Not approved / mode changed → STOP and report; re-plan if needed.

Step 3 — Restore into a temp database (recommended, no data loss)

Action

ssh -F /dev/null -o BatchMode=yes windy@192.168.55.15 'cd /opt/database
docker compose exec -T timescaledb psql -U postgres -c "CREATE DATABASE <db>_restore OWNER <owner>;"
docker compose exec -T pg-backup sh -c "PGPASSWORD=\"\$POSTGRES_PASSWORD\" pg_restore -h timescaledb -U postgres -d <db>_restore /backups/daily/<db>-latest.dump"
docker compose exec -T timescaledb psql -U postgres -Atc "select count(*) from pg_database where datname='"'"'<db>_restore'"'"';"'

<owner> is the database owner from hosts/pgdb.md (hass → hass, scribe/postgres → postgres). Run inside the pg-backup container so the password stays server-side; -h timescaledb routes over the compose network.

Expected — CREATE DATABASE succeeds; pg_restore exits 0 with no error lines; the temp DB exists.

Decision

• Restore OK → Step 4.
• Restore errors → diagnose (dump integrity, permissions), fix, or STOP; the live DB is untouched in this mode.

Verification

ssh -F /dev/null -o BatchMode=yes windy@192.168.55.15 'cd /opt/database && docker compose exec -T timescaledb psql -U postgres -d <db>_restore -Atc "select count(*) from <key_table>;"'

Compare against the pre-restore baseline (e.g. states ≈ dump's row count). Row counts match → proceed to swap; mismatch → STOP and inspect.

Step 4 — Swap temp DB into place (approval-gated)

Action — After user approval: drop the live DB and rename the temp (or pg_dump the live DB as a safety copy first).

ssh -F /dev/null -o BatchMode=yes windy@192.168.55.15 'cd /opt/database
docker compose exec -T timescaledb psql -U postgres -c "DROP DATABASE <db>;"
docker compose exec -T timescaledb psql -U postgres -c "ALTER DATABASE <db>_restore RENAME TO <db>;"'

Expected — Rename succeeds; pg_database shows <db> with the restored size.

Decision

• Swap OK → Step 5.
• Swap fails → STOP; see Rollback.

Verification

• select count(*) on key tables matches the dump.
• HA sessions re-establish: pg_stat_activity shows hass|hass|192.168.55.11 on <db> (scribe writes resume).
• pgweb bookmarks show restored data (read-only browse).

Rollback

• Trigger: verification fails or swap is wrong.
• Action: restore again from the dump taken before the operation; if the live DB was dropped, recreate <db> and restore the pre-change dump (--clean not needed on an empty DB).
• Verify: row counts + HA sessions match pre-change baseline; record in Linear.

In-place variant (only with explicit approval)

ssh -F /dev/null -o BatchMode=yes windy@192.168.55.15 'cd /opt/database && docker compose exec -T pg-backup sh -c "PGPASSWORD=\"\$POSTGRES_PASSWORD\" pg_restore -h timescaledb -U postgres --clean --if-exists -d <db> /backups/daily/<db>-latest.dump"'

Destructive to current data — requires the High-risk approval gate and a fresh dump of the current state taken immediately before. Follow Steps 4–5 verification afterwards.

Final Verification

Success requires all of:

• Target DB row counts match the dump (states, states_raw, events etc. sane).
• HA (192.168.55.11) reconnects and writes resume (write-activity sample grows).
• pgdb-health passes (containers, backups still current, logs clean).
• Outcome recorded in hosts/pgdb.md and the Linear vps project.

Failure Handling

1. Stop further changes; do not guess.
2. Collect: dump path + pg_restore/psql output, DB sizes, log tail.
3. Record completed steps, actual vs expected, and whether rollback ran.
4. Escalate to the user with evidence; continue only with explicit direction.

References

• hosts/pgdb.md — roles, owners, backup layout, verification history
• pgdb-health — pre/post health baseline
• pgdb-update — image/compose upgrades
• Linear vps: W1N-226 (first check), W1N-227 (compose-化 + restore test), W1N-228 (this runbook)