# Runbook: network outage / service recovery

## Purpose

Recover from a network outage or service failure, starting from read-only
diagnosis and mutating only when the root cause is confirmed.

## Scope

- Applicable: unreachable VPS services, LAN gateway/DNS failures, DNS resolution failures.
- Not applicable: planned changes (→ `network-change.md`), SSH access recovery (→ `AGENTS.md` §SSH access safety).

## Safety

- Read-only diagnosis first; do not mutate while the root cause is unknown.
- If live state conflicts with a runbook's assumptions, `STOP` and report.
- Keep the current verified management session open as the recovery path.

## Procedure

### Step 1 — Diagnose (read-only)

**Action** — gather evidence without changing anything:

```bash
# From laptop, pin DNS to a public resolver if the stub is flaky
dig @1.1.1.1 +short <host> A
curl -4 -sS -I --max-time 10 https://<host>/
# From a reachable host, inspect the service
ssh -4 windy@<host> 'docker compose ps -a; df -h /; tail -n 50 /var/lib/vps-health/latest.json'
```

**Expected** — a clear picture: is it DNS, connectivity, host, or service?

**Decision** — root cause localized → Step 2; ambiguous or conflicting → `STOP` and escalate (provider console if host is unreachable).

### Step 2 — Confirm and route

**Action** — match the failure to the owning runbook (`mailcow-health.md`, `pdns-health.md`, `matrix-health.md`, etc.) or `network-change.md` for a config fix.

**Expected** — an applicable runbook with a recovery action.

**Decision** — applicable → follow it; none → `STOP` (diagnose only, do not mutate).

### Step 3 — Recover (gated)

**Action** — apply only the runbook's documented recovery, with approval.

**Verification** — re-run the health report / endpoint check and confirm recovery.

**Rollback** — if recovery makes it worse, revert per `rollback.md`.

## Final Verification

- Service reachable and health report green.
- Incident and recovery recorded in the Linear `vps` issue.

## References

- [`network-change.md`](network-change.md)
- Per-service health runbooks under [`runbooks/`](.)