windyboy
b0c01b2551
- RUNBOOKS.md: repo-level spec (six-field model, naming, safety, maturity path) - runbooks/_template.md + README.md: standard template and 16-entry routing index - new: issue-to-merge, fix-ci, release, rollback, network-change, network-recovery - light-enhance 10 existing runbooks with Purpose/Scope/Safety headers - AGENTS.md: point step 3 at index/spec, add runbook execution rules - docs/agent-runbook-guide.md: archive of Manus AI guide
71 lines
2.5 KiB
Markdown
71 lines
2.5 KiB
Markdown
# Runbook: fix a failing health/playbook run
|
|
|
|
> Adapted from the upstream guide's `fix-ci`. This repo has no software CI; the
|
|
> equivalent "pipeline" is the Ansible **health report** and the gated playbooks.
|
|
> This runbook covers diagnosing and fixing a failed or warning/critical run.
|
|
|
|
## Purpose
|
|
|
|
Diagnose and fix a failing Ansible health-report or playbook run without
|
|
skipping checks or changing unrelated code.
|
|
|
|
## Scope
|
|
|
|
- Applicable: `ansible-playbook playbooks/health-report.yml` and the gated playbooks under `ansible/playbooks/`.
|
|
- Not applicable: production changes beyond fixing the run; network/DNS changes → `network-change.md`.
|
|
|
|
## Safety
|
|
|
|
- Do not skip or weaken a failing check to make it pass.
|
|
- Do not change unrelated hosts or services.
|
|
- Prefer read-only diagnosis before mutation; destructive fixes require approval.
|
|
|
|
## Procedure
|
|
|
|
### Step 1 — Reproduce and read
|
|
|
|
**Action** — re-run the failing playbook with `--limit <host>` and capture the task that failed.
|
|
|
|
```bash
|
|
cd ansible
|
|
ansible-playbook playbooks/health-report.yml --limit <host> -v
|
|
```
|
|
|
|
**Expected** — a specific failed task, host, and message (warning vs critical).
|
|
|
|
**Decision** — clear failure → Step 2; ambiguous → `STOP` and collect `-vvv` output + the relevant `latest.json`.
|
|
|
|
### Step 2 — Diagnose
|
|
|
|
**Action** — inspect the corresponding service on the host using the matching health runbook (`mailcow-health.md`, `vaultwarden-health.md`, `pdns-health.md`, etc.).
|
|
|
|
**Expected** — a root cause (container down, cert expired, queue backlog, drift).
|
|
|
|
**Decision** — root cause found → Step 3; live state conflicts with the runbook's assumptions → `STOP`.
|
|
|
|
### Step 3 — Fix within scope
|
|
|
|
**Action** — apply the minimal fix the service runbook prescribes (e.g. `compose-reconcile` for a config drift, or a documented update). Use only allowlisted/gated playbooks.
|
|
|
|
**Verification** — re-run the health report and confirm it passes.
|
|
|
|
**Rollback** — revert to the prior config/state and re-run; see `rollback.md` for the general procedure.
|
|
|
|
## Troubleshooting
|
|
|
|
### Troubleshooting A — Intermittent/flaky failure
|
|
|
|
- Evidence: timing, DNS stub flakiness (use `1.1.1.1`/`8.8.8.8` for probes).
|
|
- Allowed: re-run once with the documented resolver workaround.
|
|
- Next: still failing → `STOP` and escalate.
|
|
|
|
## Final Verification
|
|
|
|
- Health report passes for the affected host.
|
|
- No checks were skipped or weakened; the fix is committed/documented.
|
|
|
|
## References
|
|
|
|
- [`ansible-operations.md`](ansible-operations.md)
|
|
- Per-service health runbooks under [`runbooks/`](.)
|