windyboyandCursor
343c5db415
Keep sanitized Compose sources in-repo with a confirmation-gated Ansible playbook, add repo-wide validation, tighten runbook ownership/STOP/review metadata, and archive stale research docs. Co-authored-by: Cursor <cursoragent@cursor.com>
68 lines
2.2 KiB
Markdown
68 lines
2.2 KiB
Markdown
# Runbook: network outage / service recovery
|
|
|
|
## Purpose
|
|
|
|
Recover from a network outage or service failure, starting from read-only
|
|
diagnosis and mutating only when the root cause is confirmed.
|
|
|
|
## Scope
|
|
|
|
- Applicable: unreachable VPS services, LAN gateway/DNS failures, DNS resolution failures.
|
|
- Not applicable: planned changes (→ `network-change.md`), SSH access recovery (→ `AGENTS.md` §SSH access safety).
|
|
|
|
## Ownership
|
|
|
|
- Owner: personal ops (Windy)
|
|
- Last reviewed: 2026-08-17
|
|
- Related systems: VPS services / LAN gateway / DNS
|
|
|
|
## Safety
|
|
|
|
- Read-only diagnosis first; do not mutate while the root cause is unknown.
|
|
- If live state conflicts with a runbook's assumptions, `STOP` and report.
|
|
- Keep the current verified management session open as the recovery path.
|
|
|
|
## Procedure
|
|
|
|
### Step 1 — Diagnose (read-only)
|
|
|
|
**Action** — gather evidence without changing anything:
|
|
|
|
```bash
|
|
# From laptop, pin DNS to a public resolver if the stub is flaky
|
|
dig @1.1.1.1 +short <host> A
|
|
curl -4 -sS -I --max-time 10 https://<host>/
|
|
# From a reachable host, inspect the service
|
|
ssh -4 windy@<host> 'docker compose ps -a; df -h /; tail -n 50 /var/lib/vps-health/latest.json'
|
|
```
|
|
|
|
**Expected** — a clear picture: is it DNS, connectivity, host, or service?
|
|
|
|
**Decision** — root cause localized → Step 2; ambiguous or conflicting → `STOP` and escalate (provider console if host is unreachable).
|
|
|
|
### Step 2 — Confirm and route
|
|
|
|
**Action** — match the failure to the owning runbook (`mailcow-health.md`, `pdns-health.md`, `matrix-health.md`, etc.) or `network-change.md` for a config fix.
|
|
|
|
**Expected** — an applicable runbook with a recovery action.
|
|
|
|
**Decision** — applicable → follow it; none → `STOP` (diagnose only, do not mutate).
|
|
|
|
### Step 3 — Recover (gated)
|
|
|
|
**Action** — apply only the runbook's documented recovery, with approval.
|
|
|
|
**Verification** — re-run the health report / endpoint check and confirm recovery.
|
|
|
|
**Rollback** — if recovery makes it worse, revert per `rollback.md`.
|
|
|
|
## Final Verification
|
|
|
|
- Service reachable and health report green.
|
|
- Incident and recovery recorded in the Linear `vps` issue.
|
|
|
|
## References
|
|
|
|
- [`network-change.md`](network-change.md)
|
|
- Per-service health runbooks under [`runbooks/`](.)
|