windy/vps

Files

T

windyboy b0c01b2551 docs(runbooks): add runbook spec, template, index and 6 first-batch runbooks; light-enhance existing 10

- RUNBOOKS.md: repo-level spec (six-field model, naming, safety, maturity path)
- runbooks/_template.md + README.md: standard template and 16-entry routing index
- new: issue-to-merge, fix-ci, release, rollback, network-change, network-recovery
- light-enhance 10 existing runbooks with Purpose/Scope/Safety headers
- AGENTS.md: point step 3 at index/spec, add runbook execution rules
- docs/agent-runbook-guide.md: archive of Manus AI guide

2026-08-17 15:59:46 +08:00

2.1 KiB

Raw Blame History

Runbook: network outage / service recovery

Purpose

Recover from a network outage or service failure, starting from read-only diagnosis and mutating only when the root cause is confirmed.

Scope

Applicable: unreachable VPS services, LAN gateway/DNS failures, DNS resolution failures.
Not applicable: planned changes (→ network-change.md), SSH access recovery (→ AGENTS.md §SSH access safety).

Safety

Read-only diagnosis first; do not mutate while the root cause is unknown.
If live state conflicts with a runbook's assumptions, STOP and report.
Keep the current verified management session open as the recovery path.

Procedure

Step 1 — Diagnose (read-only)

Action — gather evidence without changing anything:

# From laptop, pin DNS to a public resolver if the stub is flaky
dig @1.1.1.1 +short <host> A
curl -4 -sS -I --max-time 10 https://<host>/
# From a reachable host, inspect the service
ssh -4 windy@<host> 'docker compose ps -a; df -h /; tail -n 50 /var/lib/vps-health/latest.json'

Expected — a clear picture: is it DNS, connectivity, host, or service?

Decision — root cause localized → Step 2; ambiguous or conflicting → STOP and escalate (provider console if host is unreachable).

Step 2 — Confirm and route

Action — match the failure to the owning runbook (mailcow-health.md, pdns-health.md, matrix-health.md, etc.) or network-change.md for a config fix.

Expected — an applicable runbook with a recovery action.

Decision — applicable → follow it; none → STOP (diagnose only, do not mutate).

Step 3 — Recover (gated)

Action — apply only the runbook's documented recovery, with approval.

Verification — re-run the health report / endpoint check and confirm recovery.

Rollback — if recovery makes it worse, revert per rollback.md.

Final Verification

Service reachable and health report green.
Incident and recovery recorded in the Linear vps issue.

References

network-change.md
Per-service health runbooks under runbooks/

2.1 KiB Raw Blame History