docs(runbooks): add runbook spec, template, index and 6 first-batch runbooks; light-enhance existing 10

- RUNBOOKS.md: repo-level spec (six-field model, naming, safety, maturity path) - runbooks/_template.md + README.md: standard template and 16-entry routing index - new: issue-to-merge, fix-ci, release, rollback, network-change, network-recovery - light-enhance 10 existing runbooks with Purpose/Scope/Safety headers - AGENTS.md: point step 3 at index/spec, add runbook execution rules - docs/agent-runbook-guide.md: archive of Manus AI guide
2026-08-17 15:59:46 +08:00
parent 047ac03346
commit b0c01b2551
21 changed files with 1140 additions and 2 deletions
@@ -0,0 +1,70 @@
+# Runbook: fix a failing health/playbook run
+
+> Adapted from the upstream guide's `fix-ci`. This repo has no software CI; the
+> equivalent "pipeline" is the Ansible **health report** and the gated playbooks.
+> This runbook covers diagnosing and fixing a failed or warning/critical run.
+
+## Purpose
+
+Diagnose and fix a failing Ansible health-report or playbook run without
+skipping checks or changing unrelated code.
+
+## Scope
+
+- Applicable: `ansible-playbook playbooks/health-report.yml` and the gated playbooks under `ansible/playbooks/`.
+- Not applicable: production changes beyond fixing the run; network/DNS changes → `network-change.md`.
+
+## Safety
+
+- Do not skip or weaken a failing check to make it pass.
+- Do not change unrelated hosts or services.
+- Prefer read-only diagnosis before mutation; destructive fixes require approval.
+
+## Procedure
+
+### Step 1 — Reproduce and read
+
+**Action** — re-run the failing playbook with `--limit <host>` and capture the task that failed.
+
+```bash
+cd ansible
+ansible-playbook playbooks/health-report.yml --limit <host> -v
+```
+
+**Expected** — a specific failed task, host, and message (warning vs critical).
+
+**Decision** — clear failure → Step 2; ambiguous → `STOP` and collect `-vvv` output + the relevant `latest.json`.
+
+### Step 2 — Diagnose
+
+**Action** — inspect the corresponding service on the host using the matching health runbook (`mailcow-health.md`, `vaultwarden-health.md`, `pdns-health.md`, etc.).
+
+**Expected** — a root cause (container down, cert expired, queue backlog, drift).
+
+**Decision** — root cause found → Step 3; live state conflicts with the runbook's assumptions → `STOP`.
+
+### Step 3 — Fix within scope
+
+**Action** — apply the minimal fix the service runbook prescribes (e.g. `compose-reconcile` for a config drift, or a documented update). Use only allowlisted/gated playbooks.
+
+**Verification** — re-run the health report and confirm it passes.
+
+**Rollback** — revert to the prior config/state and re-run; see `rollback.md` for the general procedure.
+
+## Troubleshooting
+
+### Troubleshooting A — Intermittent/flaky failure
+
+- Evidence: timing, DNS stub flakiness (use `1.1.1.1`/`8.8.8.8` for probes).
+- Allowed: re-run once with the documented resolver workaround.
+- Next: still failing → `STOP` and escalate.
+
+## Final Verification
+
+- Health report passes for the affected host.
+- No checks were skipped or weakened; the fix is committed/documented.
+
+## References
+
+- [`ansible-operations.md`](ansible-operations.md)
+- Per-service health runbooks under [`runbooks/`](.)