2026-08-17 15:59:00 +08:00
|
|
|
# Runbook: roll back a change
|
|
|
|
|
|
|
|
|
|
## Purpose
|
|
|
|
|
|
|
|
|
|
Restore a known-good state after a change failed verification or caused a
|
|
|
|
|
regression.
|
|
|
|
|
|
|
|
|
|
## Scope
|
|
|
|
|
|
|
|
|
|
- Applicable: Compose services, DNS records, firewall rules, and Ansible-managed config that have a rollback point.
|
|
|
|
|
- Not applicable: destructive migrations without a backup (route to the owning runbook, e.g. `vaultwarden-sqlite-to-postgres.md` §Rollback).
|
|
|
|
|
|
2026-08-17 17:36:39 +08:00
|
|
|
## Ownership
|
|
|
|
|
|
|
|
|
|
- Owner: personal ops (Windy)
|
|
|
|
|
- Last reviewed: 2026-08-17
|
|
|
|
|
- Related systems: all production hosts
|
|
|
|
|
|
2026-08-17 15:59:00 +08:00
|
|
|
## Preconditions
|
|
|
|
|
|
|
|
|
|
- A rollback point exists: backup, prior config, or known-good commit/playbook state.
|
|
|
|
|
- The failing change and its symptoms are recorded.
|
|
|
|
|
|
|
|
|
|
## Safety
|
|
|
|
|
|
|
|
|
|
- Confirm the rollback target before acting; do not roll back unrelated changes.
|
|
|
|
|
- Verify after rollback; a rollback that is not verified is not complete.
|
|
|
|
|
|
|
|
|
|
## Procedure
|
|
|
|
|
|
|
|
|
|
### Step 1 — Confirm the target
|
|
|
|
|
|
|
|
|
|
**Action** — record the change to revert (playbook run, config edit, DNS change) and its intended prior state.
|
|
|
|
|
|
|
|
|
|
**Expected** — the exact prior state is known and reachable.
|
|
|
|
|
|
|
|
|
|
**Decision** — prior state known → Step 2; unknown → `STOP` and escalate.
|
|
|
|
|
|
|
|
|
|
### Step 2 — Revert
|
|
|
|
|
|
|
|
|
|
**Action** — apply the documented reversal for the change type:
|
|
|
|
|
|
|
|
|
|
- Compose config: re-apply the prior config and reconcile (or `git revert` the change then reconcile).
|
|
|
|
|
- DNS: restore the previous record value on PowerDNS.
|
|
|
|
|
- Firewall: re-apply the prior allowlist via the gated playbook.
|
|
|
|
|
- General repo change: `git revert <sha>`.
|
|
|
|
|
|
|
|
|
|
**Expected** — the prior configuration is in place.
|
|
|
|
|
|
|
|
|
|
**Verification** — re-run the relevant health report / endpoint check and confirm green.
|
|
|
|
|
|
|
|
|
|
**Rollback** — if the rollback itself fails, `STOP`; keep the original management session open and escalate.
|
|
|
|
|
|
|
|
|
|
## Final Verification
|
|
|
|
|
|
|
|
|
|
- Health report green after rollback.
|
|
|
|
|
- The incident and rollback are recorded in the Linear `vps` issue.
|
|
|
|
|
|
|
|
|
|
## References
|
|
|
|
|
|
|
|
|
|
- [`release.md`](release.md)
|
|
|
|
|
- [`ansible-operations.md`](ansible-operations.md)
|
