windy/vps

Files

T

windyboy b0c01b2551 docs(runbooks): add runbook spec, template, index and 6 first-batch runbooks; light-enhance existing 10

- RUNBOOKS.md: repo-level spec (six-field model, naming, safety, maturity path)
- runbooks/_template.md + README.md: standard template and 16-entry routing index
- new: issue-to-merge, fix-ci, release, rollback, network-change, network-recovery
- light-enhance 10 existing runbooks with Purpose/Scope/Safety headers
- AGENTS.md: point step 3 at index/spec, add runbook execution rules
- docs/agent-runbook-guide.md: archive of Manus AI guide

2026-08-17 15:59:46 +08:00

2.5 KiB

Raw Blame History

Runbook: fix a failing health/playbook run

Adapted from the upstream guide's fix-ci. This repo has no software CI; the equivalent "pipeline" is the Ansible health report and the gated playbooks. This runbook covers diagnosing and fixing a failed or warning/critical run.

Purpose

Diagnose and fix a failing Ansible health-report or playbook run without skipping checks or changing unrelated code.

Scope

Applicable: ansible-playbook playbooks/health-report.yml and the gated playbooks under ansible/playbooks/.
Not applicable: production changes beyond fixing the run; network/DNS changes → network-change.md.

Safety

Do not skip or weaken a failing check to make it pass.
Do not change unrelated hosts or services.
Prefer read-only diagnosis before mutation; destructive fixes require approval.

Procedure

Step 1 — Reproduce and read

Action — re-run the failing playbook with --limit <host> and capture the task that failed.

cd ansible
ansible-playbook playbooks/health-report.yml --limit <host> -v

Expected — a specific failed task, host, and message (warning vs critical).

Decision — clear failure → Step 2; ambiguous → STOP and collect -vvv output + the relevant latest.json.

Step 2 — Diagnose

Action — inspect the corresponding service on the host using the matching health runbook (mailcow-health.md, vaultwarden-health.md, pdns-health.md, etc.).

Expected — a root cause (container down, cert expired, queue backlog, drift).

Decision — root cause found → Step 3; live state conflicts with the runbook's assumptions → STOP.

Step 3 — Fix within scope

Action — apply the minimal fix the service runbook prescribes (e.g. compose-reconcile for a config drift, or a documented update). Use only allowlisted/gated playbooks.

Verification — re-run the health report and confirm it passes.

Rollback — revert to the prior config/state and re-run; see rollback.md for the general procedure.

Troubleshooting

Troubleshooting A — Intermittent/flaky failure

Evidence: timing, DNS stub flakiness (use 1.1.1.1/8.8.8.8 for probes).
Allowed: re-run once with the documented resolver workaround.
Next: still failing → STOP and escalate.

Final Verification

Health report passes for the affected host.
No checks were skipped or weakened; the fix is committed/documented.

References

ansible-operations.md
Per-service health runbooks under runbooks/

2.5 KiB Raw Blame History