# Runbook: fix a failing health/playbook run

> Adapted from the upstream guide's `fix-ci`. This repo has no software CI; the
> equivalent "pipeline" is the Ansible **health report** and the gated playbooks.
> This runbook covers diagnosing and fixing a failed or warning/critical run.

## Purpose

Diagnose and fix a failing Ansible health-report or playbook run without
skipping checks or changing unrelated code.

## Scope

- Applicable: `ansible-playbook playbooks/health-report.yml` and the gated playbooks under `ansible/playbooks/`.
- Not applicable: production changes beyond fixing the run; network/DNS changes → `network-change.md`.

## Safety

- Do not skip or weaken a failing check to make it pass.
- Do not change unrelated hosts or services.
- Prefer read-only diagnosis before mutation; destructive fixes require approval.

## Procedure

### Step 1 — Reproduce and read

**Action** — re-run the failing playbook with `--limit <host>` and capture the task that failed.

```bash
cd ansible
ansible-playbook playbooks/health-report.yml --limit <host> -v
```

**Expected** — a specific failed task, host, and message (warning vs critical).

**Decision** — clear failure → Step 2; ambiguous → `STOP` and collect `-vvv` output + the relevant `latest.json`.

### Step 2 — Diagnose

**Action** — inspect the corresponding service on the host using the matching health runbook (`mailcow-health.md`, `vaultwarden-health.md`, `pdns-health.md`, etc.).

**Expected** — a root cause (container down, cert expired, queue backlog, drift).

**Decision** — root cause found → Step 3; live state conflicts with the runbook's assumptions → `STOP`.

### Step 3 — Fix within scope

**Action** — apply the minimal fix the service runbook prescribes (e.g. `compose-reconcile` for a config drift, or a documented update). Use only allowlisted/gated playbooks.

**Verification** — re-run the health report and confirm it passes.

**Rollback** — revert to the prior config/state and re-run; see `rollback.md` for the general procedure.

## Troubleshooting

### Troubleshooting A — Intermittent/flaky failure

- Evidence: timing, DNS stub flakiness (use `1.1.1.1`/`8.8.8.8` for probes).
- Allowed: re-run once with the documented resolver workaround.
- Next: still failing → `STOP` and escalate.

## Final Verification

- Health report passes for the affected host.
- No checks were skipped or weakened; the fix is committed/documented.

## References

- [`ansible-operations.md`](ansible-operations.md)
- Per-service health runbooks under [`runbooks/`](.)