windy/vps

Files

T

windyboyandCursor 343c5db415 feat: add gated Compose deploy and make inventory the host source of truth

Keep sanitized Compose sources in-repo with a confirmation-gated Ansible
playbook, add repo-wide validation, tighten runbook ownership/STOP/review
metadata, and archive stale research docs.

Co-authored-by: Cursor <cursoragent@cursor.com>

2026-08-17 17:36:39 +08:00

2.2 KiB

Raw Blame History

Runbook: network outage / service recovery

Purpose

Recover from a network outage or service failure, starting from read-only diagnosis and mutating only when the root cause is confirmed.

Scope

Applicable: unreachable VPS services, LAN gateway/DNS failures, DNS resolution failures.
Not applicable: planned changes (→ network-change.md), SSH access recovery (→ AGENTS.md §SSH access safety).

Ownership

Owner: personal ops (Windy)
Last reviewed: 2026-08-17
Related systems: VPS services / LAN gateway / DNS

Safety

Read-only diagnosis first; do not mutate while the root cause is unknown.
If live state conflicts with a runbook's assumptions, STOP and report.
Keep the current verified management session open as the recovery path.

Procedure

Step 1 — Diagnose (read-only)

Action — gather evidence without changing anything:

# From laptop, pin DNS to a public resolver if the stub is flaky
dig @1.1.1.1 +short <host> A
curl -4 -sS -I --max-time 10 https://<host>/
# From a reachable host, inspect the service
ssh -4 windy@<host> 'docker compose ps -a; df -h /; tail -n 50 /var/lib/vps-health/latest.json'

Expected — a clear picture: is it DNS, connectivity, host, or service?

Decision — root cause localized → Step 2; ambiguous or conflicting → STOP and escalate (provider console if host is unreachable).

Step 2 — Confirm and route

Action — match the failure to the owning runbook (mailcow-health.md, pdns-health.md, matrix-health.md, etc.) or network-change.md for a config fix.

Expected — an applicable runbook with a recovery action.

Decision — applicable → follow it; none → STOP (diagnose only, do not mutate).

Step 3 — Recover (gated)

Action — apply only the runbook's documented recovery, with approval.

Verification — re-run the health report / endpoint check and confirm recovery.

Rollback — if recovery makes it worse, revert per rollback.md.

Final Verification

Service reachable and health report green.
Incident and recovery recorded in the Linear vps issue.

References

network-change.md
Per-service health runbooks under runbooks/

2.2 KiB Raw Blame History