windy/vps

Files

T

windyboyandCursor 343c5db415 feat: add gated Compose deploy and make inventory the host source of truth

Keep sanitized Compose sources in-repo with a confirmation-gated Ansible
playbook, add repo-wide validation, tighten runbook ownership/STOP/review
metadata, and archive stale research docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

2026-08-17 17:36:39 +08:00

2.7 KiB

Raw Blame History

Runbook: fix a failing health/playbook run

Adapted from the upstream guide's fix-ci. This repo has no software CI; the equivalent "pipeline" is the Ansible health report and the gated playbooks. This runbook covers diagnosing and fixing a failed or warning/critical run.

Purpose

Diagnose and fix a failing Ansible health-report or playbook run without skipping checks or changing unrelated code.

Scope

Applicable: ansible-playbook playbooks/health-report.yml and the gated playbooks under ansible/playbooks/.
Not applicable: production changes beyond fixing the run; network/DNS changes → network-change.md.

Safety

Do not skip or weaken a failing check to make it pass.
Do not change unrelated hosts or services.
Prefer read-only diagnosis before mutation; destructive fixes require approval.

Ownership

Owner: personal ops (Windy)
Last reviewed: 2026-08-17
Related systems: Ansible health report / gated playbooks

Procedure

Step 1 — Reproduce and read

Action — re-run the failing playbook with --limit <host> and capture the task that failed.

cd ansible
ansible-playbook playbooks/health-report.yml --limit <host> -v

Expected — a specific failed task, host, and message (warning vs critical).

Decision — clear failure → Step 2; ambiguous → STOP and collect -vvv output + the relevant latest.json.

Step 2 — Diagnose

Action — inspect the corresponding service on the host using the matching health runbook (mailcow-health.md, vaultwarden-health.md, pdns-health.md, etc.).

Expected — a root cause (container down, cert expired, queue backlog, drift).

Decision — root cause found → Step 3; live state conflicts with the runbook's assumptions → STOP.

Step 3 — Fix within scope

Action — apply the minimal fix the service runbook prescribes (e.g. compose-reconcile for a config drift, or a documented update). Use only allowlisted/gated playbooks.

Verification — re-run the health report and confirm it passes.

Rollback — revert to the prior config/state and re-run; see rollback.md for the general procedure.

Troubleshooting

Troubleshooting A — Intermittent/flaky failure

Evidence: timing, DNS stub flakiness (use 1.1.1.1/8.8.8.8 for probes).
Allowed: re-run once with the documented resolver workaround.
Next: still failing → STOP and escalate.

Final Verification

Health report passes for the affected host.
No checks were skipped or weakened; the fix is committed/documented.

References

ansible-operations.md
Per-service health runbooks under runbooks/

2.7 KiB Raw Blame History