docs(runbooks): add runbook spec, template, index and 6 first-batch runbooks; light-enhance existing 10
- RUNBOOKS.md: repo-level spec (six-field model, naming, safety, maturity path) - runbooks/_template.md + README.md: standard template and 16-entry routing index - new: issue-to-merge, fix-ci, release, rollback, network-change, network-recovery - light-enhance 10 existing runbooks with Purpose/Scope/Safety headers - AGENTS.md: point step 3 at index/spec, add runbook execution rules - docs/agent-runbook-guide.md: archive of Manus AI guide
This commit is contained in:
windyboy
@@ -0,0 +1,34 @@
|
||||
# Runbook index
|
||||
|
||||
Entry point for all runbooks. Before operational work, read the repo entry
|
||||
[`AGENTS.md`](../AGENTS.md) and the spec [`RUNBOOKS.md`](../RUNBOOKS.md). New
|
||||
runbooks start from [`_template.md`](_template.md).
|
||||
|
||||
## Route by intent
|
||||
|
||||
| Intent | Runbook | Type |
|
||||
|---|---|---|
|
||||
| mailcow health check | [mailcow-health.md](mailcow-health.md) | read-only |
|
||||
| mailcow update | [mailcow-update.md](mailcow-update.md) | change (gated) |
|
||||
| mailcow SMTP/IMAP client | [mailcow-smtp-client.md](mailcow-smtp-client.md) | reference |
|
||||
| Vaultwarden health check | [vaultwarden-health.md](vaultwarden-health.md) | read-only |
|
||||
| Vaultwarden SQLite→PG migrate | [vaultwarden-sqlite-to-postgres.md](vaultwarden-sqlite-to-postgres.md) | change (destructive) |
|
||||
| PowerDNS health check | [pdns-health.md](pdns-health.md) | read-only |
|
||||
| RustDesk health check | [rustdesk-health.md](rustdesk-health.md) | read-only |
|
||||
| Matrix health check | [matrix-health.md](matrix-health.md) | read-only |
|
||||
| AdGuard Home health check | [adguard-home-health.md](adguard-home-health.md) | read-only |
|
||||
| Routine Ansible operations | [ansible-operations.md](ansible-operations.md) | change (allowlisted) |
|
||||
| Linear issue → mergeable change | [issue-to-merge.md](issue-to-merge.md) | delivery |
|
||||
| Failing health/playbook run | [fix-ci.md](fix-ci.md) | change |
|
||||
| Release a reviewed change to production | [release.md](release.md) | change (gated) |
|
||||
| Roll back a change | [rollback.md](rollback.md) | change (gated) |
|
||||
| Controlled network configuration | [network-change.md](network-change.md) | change (gated) |
|
||||
| Network outage / service recovery | [network-recovery.md](network-recovery.md) | recovery |
|
||||
|
||||
## Notes
|
||||
|
||||
- `fix-ci.md`, `release.md`, `rollback.md`, `network-change.md`, `network-recovery.md`
|
||||
are adapted from the upstream guide to this repo's VPS-ops context (execution
|
||||
layer is Ansible + SSH + Linear, not a software CI/CD pipeline).
|
||||
- Health runbooks are read-only; they stop (`STOP`) when live state conflicts
|
||||
with the expected state instead of mutating production.
|
||||
@@ -0,0 +1,119 @@
|
||||
# Runbook: <名称>
|
||||
|
||||
## Purpose
|
||||
|
||||
<说明本 Runbook 要解决的问题及成功结果，1–2 行。>
|
||||
|
||||
## Scope
|
||||
|
||||
- 适用环境：<production / staging / LAN …>
|
||||
- 适用对象：<服务、主机、组件或告警类型>
|
||||
- 不适用情形：<需要改用其他 runbook 或转人工的场景>
|
||||
|
||||
## Ownership
|
||||
|
||||
- Owner：<团队或角色>
|
||||
- Last reviewed：<YYYY-MM-DD>
|
||||
- Related systems：<主机名 / 服务名>
|
||||
|
||||
## Preconditions
|
||||
|
||||
- <执行前必须满足的权限、备份、窗口、健康状态或已知信息>
|
||||
|
||||
## Inputs
|
||||
|
||||
| 输入 | 来源 | 是否必需 | 校验方法 |
|
||||
|---|---|---:|---|
|
||||
| <参数> | <来源> | 是/否 | <如何确认有效> |
|
||||
|
||||
## Safety
|
||||
|
||||
### Non-negotiable rules
|
||||
|
||||
- 先只读诊断，后执行变更。
|
||||
- 不得把删除现有配置作为首次恢复动作。
|
||||
- 不得猜测或编造缺失参数。
|
||||
- 不得绕过失败的测试、检查或审批。
|
||||
- 每次变更后必须完成对应验证。
|
||||
- 破坏性操作必须获得明确批准。
|
||||
|
||||
### Stop conditions
|
||||
|
||||
- 实际状态与本文档的前提或预期结果冲突。
|
||||
- 缺少必要输入、权限、审批或回滚能力。
|
||||
- 验证失败且本文档没有明确的下一步。
|
||||
- 影响范围超出 Scope。
|
||||
|
||||
### Approval gates
|
||||
|
||||
| 动作 | 风险级别 | 是否需要明确批准 | 批准记录位置 |
|
||||
|---|---|---:|---|
|
||||
| <动作> | 低/中/高 | 是/否 | <Issue / PR / 变更单> |
|
||||
|
||||
## Procedure
|
||||
|
||||
### Step 1 — Diagnose
|
||||
|
||||
**Action**
|
||||
|
||||
<执行只读诊断动作。>
|
||||
|
||||
**Expected**
|
||||
|
||||
<列出预期输出、状态或证据。>
|
||||
|
||||
**Decision**
|
||||
|
||||
- 若 <条件 A>，进入 Step 2。
|
||||
- 若 <条件 B>，进入 Troubleshooting A。
|
||||
- 若无法判断或状态冲突，`STOP` 并记录证据。
|
||||
|
||||
### Step 2 — Change
|
||||
|
||||
**Action**
|
||||
|
||||
<描述单一、可审计的变更动作。>
|
||||
|
||||
**Expected**
|
||||
|
||||
<变更后应出现的状态。>
|
||||
|
||||
**Verification**
|
||||
|
||||
<给出可重复执行的验证命令、测试、监控指标或检查清单。>
|
||||
|
||||
**Rollback**
|
||||
|
||||
- 触发条件：<什么情况需要回滚>
|
||||
- 回滚动作：<如何撤销>
|
||||
- 回滚验证：<如何确认恢复成功>
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Troubleshooting A — <异常名称>
|
||||
|
||||
- 证据收集：<日志、指标、命令输出、链接>
|
||||
- 允许动作：<仅限已验证且低风险的动作>
|
||||
- 下一步：<回到某步 / 转入另一 runbook / STOP 并升级>
|
||||
|
||||
## Final Verification
|
||||
|
||||
只有同时满足以下标准，流程才算成功：
|
||||
|
||||
- <功能或服务状态>
|
||||
- <自动化测试或健康检查>
|
||||
- <监控指标或告警状态>
|
||||
- <变更记录、PR 或 Issue 已更新>
|
||||
|
||||
## Failure Handling
|
||||
|
||||
若未能完成：
|
||||
|
||||
1. 停止进一步变更。
|
||||
2. 收集 <命令输出、时间范围、请求 ID、日志链接、截图或复现步骤>。
|
||||
3. 记录已完成步骤、实际结果、未满足的预期和是否执行过回滚。
|
||||
4. 按 <升级渠道> 交接，不继续猜测。
|
||||
|
||||
## References
|
||||
|
||||
- <关联 Issue、PR、架构文档、仪表盘、配置仓库或外部文档>
|
||||
@@ -1,5 +1,14 @@
|
||||
# AdGuard Home health — dns.windy.lan
|
||||
|
||||
## Purpose
|
||||
|
||||
Read-only health check of the AdGuard Home LAN DNS service.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: [dns.windy.lan](../hosts/dns.windy.lan.md) (`192.168.66.36`).
|
||||
- Read-only: does not expose query-log contents or secrets; does not change configuration.
|
||||
|
||||
This runbook is read-only. It does not expose query-log contents or secrets.
|
||||
|
||||
Routine checks run through Ansible on demand:
|
||||
@@ -58,3 +67,9 @@ a known-bad-signature test; an enabled DO bit alone is not validation.
|
||||
|
||||
Private PTR forwarding is intentionally absent because the EdgeRouter does
|
||||
not currently answer private PTR requests.
|
||||
|
||||
## Safety
|
||||
|
||||
- Read-only: never change the DNS policy or the `agh-ui-access.service` nftables rule during this check.
|
||||
- Do not infer a broken DNS policy from an empty `allowed_clients`.
|
||||
- If live state conflicts with an expected value, `STOP` and report.
|
||||
|
||||
@@ -1,8 +1,23 @@
|
||||
# Runbook: routine operations through Ansible
|
||||
|
||||
## Purpose
|
||||
|
||||
Routine operations (health, reconcile, maintenance) through the Ansible playbooks.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: every inventory host, run from `ansible/`.
|
||||
- Not applicable: arbitrary remote commands — the reconcile playbook is allowlisted and gated.
|
||||
|
||||
Run commands from `ansible/`. The inventory forces IPv4 and uses the `windy`
|
||||
account with sudo. Do a read-only health pass before any reconciliation.
|
||||
|
||||
## Safety
|
||||
|
||||
- Read-only health pass before any reconciliation.
|
||||
- Mutating playbooks require explicit confirmation variables; do not bypass them.
|
||||
- If a reconcile target or service name is not allowlisted, `STOP` — do not invent one.
|
||||
|
||||
## Health report (read-only)
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# Runbook: fix a failing health/playbook run
|
||||
|
||||
> Adapted from the upstream guide's `fix-ci`. This repo has no software CI; the
|
||||
> equivalent "pipeline" is the Ansible **health report** and the gated playbooks.
|
||||
> This runbook covers diagnosing and fixing a failed or warning/critical run.
|
||||
|
||||
## Purpose
|
||||
|
||||
Diagnose and fix a failing Ansible health-report or playbook run without
|
||||
skipping checks or changing unrelated code.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: `ansible-playbook playbooks/health-report.yml` and the gated playbooks under `ansible/playbooks/`.
|
||||
- Not applicable: production changes beyond fixing the run; network/DNS changes → `network-change.md`.
|
||||
|
||||
## Safety
|
||||
|
||||
- Do not skip or weaken a failing check to make it pass.
|
||||
- Do not change unrelated hosts or services.
|
||||
- Prefer read-only diagnosis before mutation; destructive fixes require approval.
|
||||
|
||||
## Procedure
|
||||
|
||||
### Step 1 — Reproduce and read
|
||||
|
||||
**Action** — re-run the failing playbook with `--limit <host>` and capture the task that failed.
|
||||
|
||||
```bash
|
||||
cd ansible
|
||||
ansible-playbook playbooks/health-report.yml --limit <host> -v
|
||||
```
|
||||
|
||||
**Expected** — a specific failed task, host, and message (warning vs critical).
|
||||
|
||||
**Decision** — clear failure → Step 2; ambiguous → `STOP` and collect `-vvv` output + the relevant `latest.json`.
|
||||
|
||||
### Step 2 — Diagnose
|
||||
|
||||
**Action** — inspect the corresponding service on the host using the matching health runbook (`mailcow-health.md`, `vaultwarden-health.md`, `pdns-health.md`, etc.).
|
||||
|
||||
**Expected** — a root cause (container down, cert expired, queue backlog, drift).
|
||||
|
||||
**Decision** — root cause found → Step 3; live state conflicts with the runbook's assumptions → `STOP`.
|
||||
|
||||
### Step 3 — Fix within scope
|
||||
|
||||
**Action** — apply the minimal fix the service runbook prescribes (e.g. `compose-reconcile` for a config drift, or a documented update). Use only allowlisted/gated playbooks.
|
||||
|
||||
**Verification** — re-run the health report and confirm it passes.
|
||||
|
||||
**Rollback** — revert to the prior config/state and re-run; see `rollback.md` for the general procedure.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Troubleshooting A — Intermittent/flaky failure
|
||||
|
||||
- Evidence: timing, DNS stub flakiness (use `1.1.1.1`/`8.8.8.8` for probes).
|
||||
- Allowed: re-run once with the documented resolver workaround.
|
||||
- Next: still failing → `STOP` and escalate.
|
||||
|
||||
## Final Verification
|
||||
|
||||
- Health report passes for the affected host.
|
||||
- No checks were skipped or weakened; the fix is committed/documented.
|
||||
|
||||
## References
|
||||
|
||||
- [`ansible-operations.md`](ansible-operations.md)
|
||||
- Per-service health runbooks under [`runbooks/`](.)
|
||||
@@ -0,0 +1,89 @@
|
||||
# Runbook: issue → mergeable change
|
||||
|
||||
## Purpose
|
||||
|
||||
Turn an approved Linear `vps` issue into a reviewed, mergeable change in this
|
||||
repo (docs, runbooks, hosts facts, or Ansible playbooks).
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: repo content under `docs/`, `runbooks/`, `hosts/`, `inventory/`, `ansible/`.
|
||||
- Not applicable: mutating production state directly — that goes through `release.md` / `ansible-operations.md`.
|
||||
|
||||
## Ownership
|
||||
|
||||
- Owner: personal ops (Windy)
|
||||
- Last reviewed: 2026-08-17
|
||||
- Related systems: Linear MCP (`vps` project), git
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Source | Required | Validation |
|
||||
|---|---|---:|---|
|
||||
| Issue identifier | Linear (`vps` project) | Yes | `linear_get_issue <id>` returns a description |
|
||||
| Current repo state | `git status` / `git log` | Yes | Clean or intended worktree |
|
||||
|
||||
## Safety
|
||||
|
||||
- Scope is locked to the issue: do not bundle unrelated changes.
|
||||
- Never commit secrets (see `AGENTS.md` §Safety).
|
||||
- Verify every change; do not merge a change whose verification was skipped.
|
||||
|
||||
## Procedure
|
||||
|
||||
### Step 1 — Read the issue
|
||||
|
||||
**Action** — `linear_get_issue <id>`, read description and acceptance criteria.
|
||||
|
||||
**Expected** — clear scope, action, and verification for the change.
|
||||
|
||||
**Decision** — if the issue is ambiguous or lacks verification criteria, `STOP`
|
||||
and ask for clarification (add a `needs-info` label if applicable). Otherwise go to Step 2.
|
||||
|
||||
### Step 2 — Inspect and change
|
||||
|
||||
**Action** — read the relevant files, then make the minimal change the issue asks for.
|
||||
|
||||
**Expected** — diff is scoped to the issue.
|
||||
|
||||
**Decision** — if the change needs production mutation, `STOP` and route to
|
||||
`release.md`. Otherwise go to Step 3.
|
||||
|
||||
### Step 3 — Verify
|
||||
|
||||
**Action** — run the applicable verification (link check for docs; `ansible-playbook --check` / `--syntax-check` for playbooks).
|
||||
|
||||
**Verification** — see `RUNBOOKS.md` §校验; the concrete checks must match the change type.
|
||||
|
||||
**Decision** — verification passed → Step 4; failed → Troubleshooting A.
|
||||
|
||||
### Step 4 — Commit and link
|
||||
|
||||
**Action** — commit with a message containing the full issue ID (e.g. `W1N-123: …`); open a PR if the change is substantial; link the issue via `linear_save_comment`.
|
||||
|
||||
**Verification** — `git log -1` shows the issue ID; the issue has the commit/PR pointer.
|
||||
|
||||
**Rollback** — `git revert <sha>` or `git checkout <branch>` to drop the change; re-verify after.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Troubleshooting A — Verification failed
|
||||
|
||||
- Evidence: command output, failing check.
|
||||
- Allowed: fix the change within scope; re-run verification.
|
||||
- Next: still failing → `STOP` and report in the issue.
|
||||
|
||||
## Final Verification
|
||||
|
||||
- Change matches the issue scope.
|
||||
- Verification passed and the issue is updated with evidence.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
If unfinished: stop, collect the failed check output, record completed steps, and
|
||||
hand back to the issue — do not guess.
|
||||
|
||||
## References
|
||||
|
||||
- [`docs/agents/issue-tracker.md`](../docs/agents/issue-tracker.md)
|
||||
- [`RUNBOOKS.md`](../RUNBOOKS.md)
|
||||
@@ -1,5 +1,14 @@
|
||||
# Runbook: mailcow health (mx2)
|
||||
|
||||
## Purpose
|
||||
|
||||
Read-only health check of the mailcow stack on mx2.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: [mx2.windy.me](../hosts/mx2.windy.me.md), `/opt/mail`.
|
||||
- Read-only: does not change mailcow configuration or service state.
|
||||
|
||||
Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
|
||||
Path: `/opt/mail`
|
||||
Prefer: the Ansible health report (`ansible/playbooks/health-report.yml --limit mailcow`),
|
||||
@@ -71,6 +80,11 @@ The sanitized Ansible health profile is `mailcow` (`ansible/playbooks/healthchec
|
||||
The server-local timer emits a sanitized result at `/var/lib/vps-health/latest.json`.
|
||||
It does not change Mailcow configuration or service state.
|
||||
|
||||
## Safety
|
||||
|
||||
- Read-only: never mutate configuration or service state during this check.
|
||||
- If live state conflicts with an expected value below, `STOP` and report; do not "fix" on the fly.
|
||||
|
||||
## Pass criteria
|
||||
|
||||
- Compose stack up; watchdog ~100%
|
||||
|
||||
@@ -1,5 +1,14 @@
|
||||
# Runbook: use mailcow SMTP / IMAP (client)
|
||||
|
||||
## Purpose
|
||||
|
||||
Reference for configuring mail clients against the mailcow SMTP/IMAP endpoints.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: [mx2.windy.me](../hosts/mx2.windy.me.md) client submission (587/465) and IMAP/POP (993/995).
|
||||
- Not applicable: server-side mailcow configuration or administration.
|
||||
|
||||
Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
|
||||
Prerequisite: a mailbox on `windy.me` (password from mailcow UI, not the admin account unless it is that mailbox).
|
||||
|
||||
@@ -52,3 +61,8 @@ Do not commit or paste real passwords into this repo.
|
||||
- Port/TLS mode mismatch (587 vs 465)
|
||||
- Account active in mailcow; not rate-limited / fail2banned after bad attempts
|
||||
- Apps that store SMTP in their own config (e.g. Vaultwarden `config.json`) may keep a **stale** password even when `.env` is correct — verify AUTH against the effective config ([vaultwarden-health](vaultwarden-health.md) §5)
|
||||
|
||||
## Safety
|
||||
|
||||
- Do not commit or paste real passwords into this repo or chat.
|
||||
- Use submission (587/465) for client sending; never use port 25 as a desktop/app outbound port.
|
||||
|
||||
@@ -1,9 +1,30 @@
|
||||
# Runbook: mailcow update (mx2)
|
||||
|
||||
## Purpose
|
||||
|
||||
Update the mailcow stack on mx2 to the latest supported release.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: [mx2.windy.me](../hosts/mx2.windy.me.md), `/opt/mail`.
|
||||
- Not applicable: config changes beyond the update, DB migration, secret rotation.
|
||||
|
||||
## Approval gates
|
||||
|
||||
| Action | Risk | Explicit approval |
|
||||
|---|---|---|
|
||||
| Run `./update.sh` (recreates containers, brief mail interruption) | Medium | Yes — user confirmation required |
|
||||
|
||||
Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
|
||||
Path: `/opt/mail`
|
||||
**Confirm with the user before running an update.**
|
||||
|
||||
## Safety
|
||||
|
||||
- Never run the update without explicit user confirmation.
|
||||
- Never pass secrets into the chat log; do not commit `mailcow.conf`.
|
||||
- If a step fails, capture `docker compose ps` and logs and stop before further changes.
|
||||
|
||||
## Before
|
||||
|
||||
1. Run [mailcow-health](mailcow-health.md) (Ansible health report). Record baseline.
|
||||
|
||||
@@ -1,5 +1,14 @@
|
||||
# Matrix Health Check
|
||||
|
||||
## Purpose
|
||||
|
||||
Read-only health check of the Matrix homeserver (ESS on K3s).
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: [synapse.chans.xyz](../hosts/synapse.chans.xyz.md), namespace `ess`.
|
||||
- Read-only: does not change pods, ingress, certificates, or configuration.
|
||||
|
||||
Monitor the Matrix homeserver running on `synapse.chans.xyz` (ESS chart `26.7.2`, K3s node).
|
||||
|
||||
Prefer `cd ansible && ansible-playbook playbooks/health-report.yml --limit matrix`
|
||||
@@ -90,3 +99,9 @@ Backup automation is currently paused. `/var/backups/matrix/` is retained for a
|
||||
| Well-known returns 404/redirect | Root `chans.xyz` ingress missing or misconfigured |
|
||||
| 502 Bad Gateway | Synapse pod restarting or DB down |
|
||||
| SMTP emails not sent | MAS SMTP config incomplete; TCP reachable but AUTH failing — see `runbooks/vaultwarden-health.md` |
|
||||
|
||||
## Safety
|
||||
|
||||
- Read-only: never mutate pods, ingress, certificates, or configuration during this check.
|
||||
- Backup automation is paused; do not treat `/var/backups/matrix/` as a recovery source.
|
||||
- If live state conflicts with an expected value, `STOP` and report.
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
# Runbook: controlled network change
|
||||
|
||||
## Purpose
|
||||
|
||||
Apply a controlled network configuration change (DNS records, firewall, LAN
|
||||
gateway, VLAN) with impact assessment, approval, and a rollback path.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: PowerDNS zone records, `us4` firewalld allowlist, LAN gateway/VLAN/DNS changes, WireGuard.
|
||||
- Not applicable: SSH access-policy changes (see `AGENTS.md` §SSH access safety — mandatory lockout-risk procedure).
|
||||
|
||||
## Preconditions
|
||||
|
||||
- A change record (Linear `vps` issue) describes the change, its reason, and rollback.
|
||||
- Read-only impact assessment done (current config captured, blast radius known).
|
||||
|
||||
## Safety
|
||||
|
||||
- Never change DNS or network config without a change record and approval.
|
||||
- Capture the current config first; never delete existing config as the first action.
|
||||
- For DNS: record the current record values and TTL before editing.
|
||||
- For firewall: retain an independent SSH rollback session before applying (see `ansible-operations.md` §us4).
|
||||
|
||||
## Procedure
|
||||
|
||||
### Step 1 — Assess and capture
|
||||
|
||||
**Action** — capture the current state (e.g. `dig` for DNS, `--check --diff` for firewall, `show` for gateway).
|
||||
|
||||
**Expected** — a baseline of current config and an identified blast radius.
|
||||
|
||||
**Decision** — change fully specified with rollback → Step 2; missing → `STOP`.
|
||||
|
||||
### Step 2 — Approve
|
||||
|
||||
**Action** — confirm approval is recorded in the issue/change record.
|
||||
|
||||
**Decision** — approved → Step 3; not approved → `STOP`.
|
||||
|
||||
### Step 3 — Change
|
||||
|
||||
**Action** — apply the single change (edit the record, run the gated playbook, or change gateway config) and only that change.
|
||||
|
||||
**Expected** — the new value/state is in effect.
|
||||
|
||||
**Verification** — re-query/verify the new state and confirm dependent services still pass health.
|
||||
|
||||
**Rollback** — restore the captured prior config and re-verify.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Troubleshooting A — Change broke dependent service
|
||||
|
||||
- Evidence: health report / endpoint failure.
|
||||
- Allowed: roll back to the captured prior config.
|
||||
- Next: verify; if still broken, escalate.
|
||||
|
||||
## Final Verification
|
||||
|
||||
- New state verified; dependent services healthy.
|
||||
- Change and outcome recorded in the issue.
|
||||
|
||||
## References
|
||||
|
||||
- [`ansible-operations.md`](ansible-operations.md)
|
||||
- [`rollback.md`](rollback.md)
|
||||
- [`network-recovery.md`](network-recovery.md)
|
||||
@@ -0,0 +1,61 @@
|
||||
# Runbook: network outage / service recovery
|
||||
|
||||
## Purpose
|
||||
|
||||
Recover from a network outage or service failure, starting from read-only
|
||||
diagnosis and mutating only when the root cause is confirmed.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: unreachable VPS services, LAN gateway/DNS failures, DNS resolution failures.
|
||||
- Not applicable: planned changes (→ `network-change.md`), SSH access recovery (→ `AGENTS.md` §SSH access safety).
|
||||
|
||||
## Safety
|
||||
|
||||
- Read-only diagnosis first; do not mutate while the root cause is unknown.
|
||||
- If live state conflicts with a runbook's assumptions, `STOP` and report.
|
||||
- Keep the current verified management session open as the recovery path.
|
||||
|
||||
## Procedure
|
||||
|
||||
### Step 1 — Diagnose (read-only)
|
||||
|
||||
**Action** — gather evidence without changing anything:
|
||||
|
||||
```bash
|
||||
# From laptop, pin DNS to a public resolver if the stub is flaky
|
||||
dig @1.1.1.1 +short <host> A
|
||||
curl -4 -sS -I --max-time 10 https://<host>/
|
||||
# From a reachable host, inspect the service
|
||||
ssh -4 windy@<host> 'docker compose ps -a; df -h /; tail -n 50 /var/lib/vps-health/latest.json'
|
||||
```
|
||||
|
||||
**Expected** — a clear picture: is it DNS, connectivity, host, or service?
|
||||
|
||||
**Decision** — root cause localized → Step 2; ambiguous or conflicting → `STOP` and escalate (provider console if host is unreachable).
|
||||
|
||||
### Step 2 — Confirm and route
|
||||
|
||||
**Action** — match the failure to the owning runbook (`mailcow-health.md`, `pdns-health.md`, `matrix-health.md`, etc.) or `network-change.md` for a config fix.
|
||||
|
||||
**Expected** — an applicable runbook with a recovery action.
|
||||
|
||||
**Decision** — applicable → follow it; none → `STOP` (diagnose only, do not mutate).
|
||||
|
||||
### Step 3 — Recover (gated)
|
||||
|
||||
**Action** — apply only the runbook's documented recovery, with approval.
|
||||
|
||||
**Verification** — re-run the health report / endpoint check and confirm recovery.
|
||||
|
||||
**Rollback** — if recovery makes it worse, revert per `rollback.md`.
|
||||
|
||||
## Final Verification
|
||||
|
||||
- Service reachable and health report green.
|
||||
- Incident and recovery recorded in the Linear `vps` issue.
|
||||
|
||||
## References
|
||||
|
||||
- [`network-change.md`](network-change.md)
|
||||
- Per-service health runbooks under [`runbooks/`](.)
|
||||
@@ -1,5 +1,14 @@
|
||||
# PowerDNS health (hk2)
|
||||
|
||||
## Purpose
|
||||
|
||||
Read-only health check of the `/opt/pdns` PowerDNS stack.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: [hk2.chans.xyz](../hosts/hk2.chans.xyz.md), `/opt/pdns`.
|
||||
- Read-only: does not change PowerDNS, DNS records, or secrets.
|
||||
|
||||
Read-only checks for the `/opt/pdns` stack on **hk2.chans.xyz** (`ns1.wsvc.info`).
|
||||
|
||||
Facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) · Upstream: [docs/pdns-upstream.md](../docs/pdns-upstream.md)
|
||||
@@ -87,6 +96,12 @@ Expect: `primary=yes`, `also-notify=202.91.35.141`, `only-notify=` empty, `gpgsq
|
||||
The sanitized Ansible health profile is `pdns` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses the API key only inside the PowerDNS container. It does not modify PowerDNS, DNS records, or secrets.
|
||||
|
||||
|
||||
## Safety
|
||||
|
||||
- Read-only: never mutate PowerDNS configuration or DNS records during this check.
|
||||
- Do not paste the API key into chat/logs.
|
||||
- If live state conflicts with an expected value, `STOP` and report.
|
||||
|
||||
## After config changes
|
||||
|
||||
- `auth/pdns.conf`, `auth/templates.d/secrets.j2`, or auth-related `.env` → use the Ansible Compose reconcile playbook with target `auth`
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
# Runbook: release a reviewed change to production
|
||||
|
||||
## Purpose
|
||||
|
||||
Apply a reviewed configuration change to production hosts in a controlled,
|
||||
verifiable, rollback-safe way.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: reviewed changes to Compose services, DNS, firewall, or Ansible-managed config on production hosts.
|
||||
- Not applicable: destructive migrations (`vaultwarden-sqlite-to-postgres.md`), SSH access-policy changes (`AGENTS.md` §SSH access safety), or network changes needing a change ticket (`network-change.md`).
|
||||
|
||||
## Preconditions
|
||||
|
||||
- The change is reviewed and its intent matches a Linear issue / change record.
|
||||
- Read-only health pass (or `maintenance-preview`) has been run and is green.
|
||||
- A rollback point exists (backup, prior config, or known-good git/playbook state).
|
||||
|
||||
## Safety
|
||||
|
||||
- Never release without a verification and a rollback point.
|
||||
- Never bypass gated confirmation variables.
|
||||
- One host at a time; re-run health after each.
|
||||
|
||||
## Procedure
|
||||
|
||||
### Step 1 — Pre-flight
|
||||
|
||||
**Action** — run the read-only health report for the target host(s).
|
||||
|
||||
```bash
|
||||
cd ansible && ansible-playbook playbooks/health-report.yml --limit <host>
|
||||
```
|
||||
|
||||
**Expected** — green baseline.
|
||||
|
||||
**Decision** — green → Step 2; warning/critical → fix first (`fix-ci.md`), `STOP` the release.
|
||||
|
||||
### Step 2 — Approve
|
||||
|
||||
**Action** — confirm the change is approved in the issue/change record; confirm the window.
|
||||
|
||||
**Decision** — approval recorded → Step 3; missing → `STOP`.
|
||||
|
||||
### Step 3 — Apply
|
||||
|
||||
**Action** — run the gated playbook for the change, with its confirmation variable.
|
||||
|
||||
```bash
|
||||
# example: reviewed Compose reconcile
|
||||
cd ansible && ansible-playbook playbooks/compose-reconcile.yml --limit <host> \
|
||||
-e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["<svc>"]}'
|
||||
```
|
||||
|
||||
**Expected** — playbook completes; service reports the new state.
|
||||
|
||||
**Verification** — re-run the health report and confirm green; confirm the specific change took effect (version, config, endpoint).
|
||||
|
||||
**Rollback** — on verification failure, revert the change and re-verify (see `rollback.md`).
|
||||
|
||||
## Final Verification
|
||||
|
||||
- Health report green post-release.
|
||||
- The change is visible and the issue/change record is updated.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
On failure: stop further changes, collect the playbook output and health report, and route to `rollback.md` if the change was applied.
|
||||
|
||||
## References
|
||||
|
||||
- [`ansible-operations.md`](ansible-operations.md)
|
||||
- [`rollback.md`](rollback.md)
|
||||
@@ -0,0 +1,56 @@
|
||||
# Runbook: roll back a change
|
||||
|
||||
## Purpose
|
||||
|
||||
Restore a known-good state after a change failed verification or caused a
|
||||
regression.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: Compose services, DNS records, firewall rules, and Ansible-managed config that have a rollback point.
|
||||
- Not applicable: destructive migrations without a backup (route to the owning runbook, e.g. `vaultwarden-sqlite-to-postgres.md` §Rollback).
|
||||
|
||||
## Preconditions
|
||||
|
||||
- A rollback point exists: backup, prior config, or known-good commit/playbook state.
|
||||
- The failing change and its symptoms are recorded.
|
||||
|
||||
## Safety
|
||||
|
||||
- Confirm the rollback target before acting; do not roll back unrelated changes.
|
||||
- Verify after rollback; a rollback that is not verified is not complete.
|
||||
|
||||
## Procedure
|
||||
|
||||
### Step 1 — Confirm the target
|
||||
|
||||
**Action** — record the change to revert (playbook run, config edit, DNS change) and its intended prior state.
|
||||
|
||||
**Expected** — the exact prior state is known and reachable.
|
||||
|
||||
**Decision** — prior state known → Step 2; unknown → `STOP` and escalate.
|
||||
|
||||
### Step 2 — Revert
|
||||
|
||||
**Action** — apply the documented reversal for the change type:
|
||||
|
||||
- Compose config: re-apply the prior config and reconcile (or `git revert` the change then reconcile).
|
||||
- DNS: restore the previous record value on PowerDNS.
|
||||
- Firewall: re-apply the prior allowlist via the gated playbook.
|
||||
- General repo change: `git revert <sha>`.
|
||||
|
||||
**Expected** — the prior configuration is in place.
|
||||
|
||||
**Verification** — re-run the relevant health report / endpoint check and confirm green.
|
||||
|
||||
**Rollback** — if the rollback itself fails, `STOP`; keep the original management session open and escalate.
|
||||
|
||||
## Final Verification
|
||||
|
||||
- Health report green after rollback.
|
||||
- The incident and rollback are recorded in the Linear `vps` issue.
|
||||
|
||||
## References
|
||||
|
||||
- [`release.md`](release.md)
|
||||
- [`ansible-operations.md`](ansible-operations.md)
|
||||
@@ -1,5 +1,14 @@
|
||||
# RustDesk server health (hk2)
|
||||
|
||||
## Purpose
|
||||
|
||||
Read-only health check of the `/opt/rustdesk` server stack.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: [hk2.chans.xyz](../hosts/hk2.chans.xyz.md), `/opt/rustdesk`.
|
||||
- Read-only: does not change RustDesk configuration or service state.
|
||||
|
||||
Read-only checks for the `/opt/rustdesk` stack on **hk2.chans.xyz**.
|
||||
|
||||
Facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md)
|
||||
@@ -66,6 +75,11 @@ The sanitized Ansible health profile is `rustdesk` (deployed via
|
||||
locally through `vps-healthcheck.timer`, writes a sanitized JSON result to
|
||||
`/var/lib/vps-health/latest.json`, and never exposes secrets.
|
||||
|
||||
## Safety
|
||||
|
||||
- Read-only: never mutate configuration or service state during this check.
|
||||
- If live state conflicts with an expected value, `STOP` and report.
|
||||
|
||||
## After config changes
|
||||
|
||||
Reviewed changes to `/opt/rustdesk/compose.yml` (relay address, image pin, port
|
||||
|
||||
@@ -1,5 +1,14 @@
|
||||
# Runbook: Vaultwarden health (us2)
|
||||
|
||||
## Purpose
|
||||
|
||||
Read-only health check of the Vaultwarden stack (Postgres backend) on us2.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: [us2.wsvc.info](../hosts/us2.wsvc.info.md), `/opt/vaultwarden`.
|
||||
- Read-only: does not change Vaultwarden, Traefik, SMTP, or secrets.
|
||||
|
||||
Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)
|
||||
Path: `/opt/vaultwarden`
|
||||
URL: https://auth.wsvc.info/
|
||||
@@ -96,6 +105,12 @@ Expect `AUTH_OK`. `535` usually means stale password in `config.json` (see step
|
||||
The sanitized Ansible health profile is `vaultwarden` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses only server-side credentials for the SMTP AUTH probe. It does not modify Vaultwarden, Traefik, SMTP, or secrets.
|
||||
|
||||
|
||||
## Safety
|
||||
|
||||
- Read-only: never mutate configuration, containers, or secrets during this check.
|
||||
- Never print passwords — compare lengths/hashes only.
|
||||
- If live state conflicts with an expected value below, `STOP` and report.
|
||||
|
||||
## Pass criteria
|
||||
|
||||
- Healthy compose; `DATABASE_URL` points at `pg`
|
||||
|
||||
@@ -1,5 +1,20 @@
|
||||
# Runbook: Vaultwarden SQLite → Postgres (us2)
|
||||
|
||||
## Purpose
|
||||
|
||||
Replay checklist for the exceptional SQLite→Postgres migration on us2.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: [us2.wsvc.info](../hosts/us2.wsvc.info.md), `/opt/vaultwarden`.
|
||||
- Not applicable: routine reconcile — this is destructive and intentionally not automated.
|
||||
|
||||
## Safety
|
||||
|
||||
- Destructive: requires explicit approval; confirm backups exist before starting.
|
||||
- Do not commit `.env` or password-bearing files to git.
|
||||
- If any step diverges from this checklist, `STOP` and report.
|
||||
|
||||
Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)
|
||||
Upstream: [Using the PostgreSQL Backend](https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend) · [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user