docs(runbooks): add runbook spec, template, index and 6 first-batch runbooks; light-enhance existing 10

- RUNBOOKS.md: repo-level spec (six-field model, naming, safety, maturity path) - runbooks/_template.md + README.md: standard template and 16-entry routing index - new: issue-to-merge, fix-ci, release, rollback, network-change, network-recovery - light-enhance 10 existing runbooks with Purpose/Scope/Safety headers - AGENTS.md: point step 3 at index/spec, add runbook execution rules - docs/agent-runbook-guide.md: archive of Manus AI guide
2026-08-17 15:59:46 +08:00
parent 047ac03346
commit b0c01b2551
21 changed files with 1140 additions and 2 deletions
@@ -8,7 +8,10 @@ Also readable as `agent.md` (symlink → this file).

 1. Read [`inventory/hosts.md`](inventory/hosts.md) for the machine list.
 2. Open the matching [`hosts/<name>.md`](hosts/) for SSH, roles, paths, and quirks.
-3. For common tasks, follow a runbook under [`runbooks/`](runbooks/).
+3. For common tasks, follow a runbook under [`runbooks/`](runbooks/). Pick the
+   most specific applicable one from [`runbooks/README.md`](runbooks/README.md);
+   the spec is [`RUNBOOKS.md`](RUNBOOKS.md) and new runbooks start from
+   [`runbooks/_template.md`](runbooks/_template.md).
 4. Prefer read-only checks first; change only after confirming current state.
 5. For routine checks and approved service reconciliation, run the matching
   Ansible playbook from `ansible/`; see [routine Ansible operations](runbooks/ansible-operations.md).
@@ -20,6 +23,15 @@ Also readable as `agent.md` (symlink → this file).
   scope, action, verification, and remaining follow-up; never put passwords,
   tokens, private keys, recovery keys, or private room IDs in Linear.

+### Runbook execution rules
+
+Before operational work: inspect `runbooks/`, select the most specific
+applicable runbook, follow its steps in order, do not skip verification steps,
+and respect its STOP and approval conditions. If no runbook applies, diagnose
+only — do not mutate production state. When live state conflicts with a
+runbook's assumptions, `STOP` and report; never invent missing parameters or
+bypass failed checks. The spec is [`RUNBOOKS.md`](RUNBOOKS.md).
+
 ## Active hosts (quick map)

 | Host | Role | SSH | Facts |
@@ -82,6 +94,14 @@ Also readable as `agent.md` (symlink → this file).
 | UniFi local-service proxy bypass | [docs/unifi-openclash-localhost.md](docs/unifi-openclash-localhost.md) |
 | UniFi SSO login setting (Ansible) | `cd ansible && ansible-playbook playbooks/unifi-sso.yml --limit unifi` |
 | Routine Ansible operations | [runbooks/ansible-operations.md](runbooks/ansible-operations.md) |
+| Issue → mergeable change | [runbooks/issue-to-merge.md](runbooks/issue-to-merge.md) |
+| Fix failing health/playbook run | [runbooks/fix-ci.md](runbooks/fix-ci.md) |
+| Release a reviewed change | [runbooks/release.md](runbooks/release.md) |
+| Roll back a change | [runbooks/rollback.md](runbooks/rollback.md) |
+| Controlled network change | [runbooks/network-change.md](runbooks/network-change.md) |
+| Network outage recovery | [runbooks/network-recovery.md](runbooks/network-recovery.md) |
+
+Full index: [runbooks/README.md](runbooks/README.md). Spec: [RUNBOOKS.md](RUNBOOKS.md).

 Routine mailcow health: `cd ansible && ansible-playbook playbooks/health-report.yml --limit mailcow`. The local stub resolver is flaky; DNS probes use `1.1.1.1` / `8.8.8.8`.

@@ -131,8 +151,9 @@ Bills, rough notes, and personal clutter stay in the Obsidian vault. This repo h

 ```
 AGENTS.md / agent.md   # this entry (agent.md → AGENTS.md)
+RUNBOOKS.md            # runbook spec (six-field model, naming, review rules)
 inventory/hosts.md     # machine index
 hosts/                 # per-host facts
-runbooks/              # step-by-step ops
+runbooks/              # step-by-step ops (README.md = index, _template.md = template)
 docs/                  # upstream doc indexes / design notes
 ```
@@ -0,0 +1,68 @@
+# RUNBOOKS — 仓库级规范
+
+本文件统一所有 Runbook 的字段、命名、评审与变更规则。上游参考：[docs/agent-runbook-guide.md](docs/agent-runbook-guide.md)。
+
+## 目录结构
+
+```text
+runbooks/
+├── README.md                 # 意图 → 文件 路由索引（本目录的入口）
+├── _template.md              # 新建 runbook 的标准模板（复制后填写）
+├── <intent>.md               # 每份 runbook 只描述一种可识别的操作意图
+└── ...
+```
+
+## 最小字段模型
+
+每份 runbook 必须显式包含以下控制信息，否则盲目执行或错误恢复的风险会升高：
+
+| 字段 | 作用 | 写作要求 |
+|---|---|---|
+| **Action** | 定义当前要执行的动作 | 可观察、可执行的动词；避免“检查一下”“适当调整” |
+| **Expected** | 描述正常状态或预期输出 | 具体信号、阈值、状态码、测试结果或页面表现 |
+| **Decision** | 定义分支与下一跳 | “条件 → 下一步”；无法判断时指向 `STOP` |
+| **Verification** | 确认变更真正生效 | 每个有副作用的步骤后执行，不可跳过 |
+| **Stop condition** | 规定何时不得继续 | 列出信息缺失、状态冲突、权限不足、验证失败等 |
+| **Rollback** | 如何恢复到变更前状态 | 触发条件、前提、撤销步骤、回滚后验证 |
+
+> 只读类 runbook 不产生副作用，可省略 Rollback；但必须保留 Stop condition（状态与预期冲突即 `STOP` 并记录证据）。
+
+## 命名与拆分规则
+
+- 文件名采用小写连字符，反映**操作意图**而非目标主机，例如 `mailcow-health.md`、`release.md`。
+- 一份文件只描述一种意图。流程出现明显分叉时拆分为独立文件，不堆叠“万能流程”。
+- 只读诊断与变更操作应分离：health 类 runbook 保持只读，变更走 `ansible-operations.md`、`release.md`、`rollback.md` 或对应 gated playbook。
+
+## 章节约定
+
+- 每份 runbook 顶部含 `## Purpose`（1–2 行）与 `## Scope`（适用/不适用情形）。
+- 变更型 runbook 额外含 `## Approval gates` 表；破坏性/不可逆操作必须获得明确批准。
+- 统一在 `## Safety` 或正文中复用以下通用安全规则（更严格要求优先）。
+
+```markdown
+## Safety Rules
+
+- Never delete an existing configuration as the first recovery action.
+- Prefer read-only diagnosis before mutation.
+- After every mutation, verify the expected state.
+- If actual state conflicts with this runbook, STOP.
+- Do not invent missing parameters.
+- Do not bypass failed tests.
+- Destructive actions require explicit approval.
+```
+
+## 评审与变更规则
+
+- 新建/修改 runbook 与代码同仓评审，随系统演进更新。
+- 每份 runbook 标注 `Last reviewed`；流程执行过程中发现的偏差记入对应的 Linear `vps` 项目 issue。
+- 破坏性流程（迁移、删除、DNS 变更、网络变更）保持人工审批，不自动下沉。
+
+## 成熟路径
+
+1. **人工处理** → 现场处置与复盘，记录证据。
+2. **Markdown runbook** → 固化步骤与证据要求，Agent 可辅助诊断。
+3. **Agent + runbook** → 严格按流程执行，受 Stop/Approval 约束。
+4. **Script / Ansible / Skill** → 把已稳定、低歧义、可验证的操作程序化（本仓库的执行层是 Ansible playbook）。
+5. **人工审批 + 自动执行** → 审批门控下的自动变更（如 gated playbook + 确认变量）。
+
+原则：先证据后变更，先小范围后扩大，先验证后结束，不确定则停止。
@@ -0,0 +1,326 @@
+# Agent Runbook 实用指南（v1）
+
+> **定位**：本指南用于把团队的重复性运维、交付与故障处理经验写成可由 Agent 安全执行的流程。它适用于以 Git 仓库为中心的工程协作模式，优先采用 **Markdown + Git 版本控制 + 明确的 Agent 路由规则**，而不是一开始引入复杂的自动化平台。
+
+> 本文件为上游参考存档。仓库内落地规范见 [`RUNBOOKS.md`](../RUNBOOKS.md)，标准模板见 [`runbooks/_template.md`](../runbooks/_template.md)，索引见 [`runbooks/README.md`](../runbooks/README.md)。
+
+## 1. 什么是 Agent Runbook
+
+Runbook 是预先设计的、可重复执行的操作流程，用于处理部署、告警、故障、配置变更、CI 修复等标准化工作。传统 Runbook 的主要读者是人；**Agent Runbook 则必须把人的隐性判断显式化**，使 Agent 能知道做什么、看到什么才算正常、下一步去哪里、何时停止以及如何撤销。
+
+Google SRE 强调在事故发生前设计响应流程、系统化排障，并逐步将重复性运维工作自动化。[1] [2] AWS Systems Manager Automation 则把可执行 Runbook 建模为顺序步骤：每个步骤调用一个动作，前一步输出可以传递给后续步骤。[3] 这两种思路共同构成了 Agent Runbook 的实用基础。
+
+| 层次 | 核心问题 | 应承担的职责 |
+|---|---|---|
+| `AGENTS.md` | **何时使用哪份流程？** | 工作路由、通用操作约束、无匹配流程时的默认行为 |
+| `runbooks/*.md` | **这件事按什么流程做？** | 前置条件、分步操作、决策分支、验证、停止条件与回滚 |
+| Skill / MCP / Tool | **有哪些可调用能力？** | 具体能力、参数、权限边界和使用说明 |
+| Shell / GitHub / Linear / SSH 等 | **实际如何执行？** | 对系统、代码库或外部服务执行操作 |
+
+## 2. 设计目标与适用边界
+
+Agent Runbook 的目标不是让 Agent 在所有异常下“想办法修好”，而是在一个**已知、受控、可验证、可回退**的边界中提高执行一致性。它应当优先覆盖高频、后果明确、流程稳定的操作，例如 CI 失败定位、Issue 到合并请求、发布前检查、标准部署、回滚及网络变更。
+
+| 适合纳入 Runbook | 暂不适合直接自动执行 |
+|---|---|
+| 明确输入、固定步骤、可观察结果的操作 | 目标或验收标准尚不清楚的探索性任务 |
+| 可在每次修改后验证状态的变更 | 缺失关键参数、权限或上下文的任务 |
+| 具有安全回滚路径的发布与配置调整 | 高破坏性、不可逆或影响面未知的操作 |
+| 可由权限与审批规则约束的运维流程 | 与既有流程事实冲突、无法判断根因的异常场景 |
+
+> **基本原则**：当实际状态与 Runbook 的假设冲突，Agent 应停止并呈报，而不是补全未知信息、绕过检查或继续试错。
+
+## 3. Agent Runbook 的最小字段
+
+与普通人工 Runbook 相比，Agent Runbook 必须显式包含以下六类控制信息。缺少其中任一项，都会增加盲目执行或错误恢复的风险。
+
+| 字段 | 作用 | 写作要求 |
+|---|---|---|
+| **Action** | 定义当前要执行的动作 | 使用可观察、可执行的动词；避免“检查一下”“适当调整”等模糊表述 |
+| **Expected** | 描述正常状态或预期输出 | 给出具体信号、阈值、状态码、测试结果或页面表现 |
+| **Decision** | 定义分支与下一跳 | 用“条件 → 下一步”的形式；无法判断时指向 `STOP` |
+| **Verification** | 确认变更真正生效 | 在每个有副作用的步骤后执行，不能被跳过 |
+| **Stop condition** | 规定何时不得继续 | 明确列出信息缺失、状态冲突、权限不足、验证失败等条件 |
+| **Rollback** | 描述如何恢复到变更前状态 | 标明触发条件、前提、撤销步骤及回滚后的验证方式 |
+
+## 4. 推荐目录与路由机制
+
+建议把流程与代码一起保存在 Git 仓库中。这样 Runbook 可以评审、版本化、随系统演进更新，也能与相关 Issue、PR 和配置建立可追溯关系。
+
+```text
+repo/
+├── AGENTS.md
+├── RUNBOOKS.md
+├── runbooks/
+│   ├── README.md
+│   ├── issue-to-merge.md
+│   ├── fix-ci.md
+│   ├── release.md
+│   ├── rollback.md
+│   ├── network-change.md
+│   └── network-recovery.md
+└── ...
+```
+
+### `AGENTS.md`：只做路由与通用约束
+
+`AGENTS.md` 不应重复流程细节。它只需要规定 Agent 在进行操作类工作前，先查找最具体且适用的 Runbook，并严格遵守其中的步骤、验证、停止和审批要求。
+
+```markdown
+# Operational Rules
+
+Before performing operational work:
+
+1. Inspect `runbooks/`.
+2. Select the most specific applicable runbook.
+3. Follow its steps in order.
+4. Do not skip verification steps.
+5. Respect STOP and approval conditions.
+6. If no runbook applies, diagnose only; do not mutate production state.
+
+## Routing
+
+- CI failure → `runbooks/fix-ci.md`
+- GitHub issue implementation → `runbooks/issue-to-merge.md`
+- Deployment → `runbooks/release.md`
+- Rollback → `runbooks/rollback.md`
+- Network configuration → `runbooks/network-change.md`
+- Network outage → `runbooks/network-recovery.md`
+```
+
+### `RUNBOOKS.md`：仓库级规范
+
+`RUNBOOKS.md` 用于统一所有 Runbook 的字段、命名、评审要求和变更规则。每份 Runbook 只描述一种可识别的操作意图；如果流程已有明显分叉，应拆分为独立文件，而不是堆叠成长篇“万能流程”。
+
+## 5. 规范模板
+
+以下模板可直接保存为 `runbooks/_template.md` 使用。
+
+```markdown
+# Runbook: <名称>
+
+## Purpose
+说明本 Runbook 要解决的问题及成功结果。
+
+## Scope
+- 适用环境：<如 development / staging / production>
+- 适用对象：<服务、仓库、组件或告警类型>
+- 不适用情形：<需要改用其他 Runbook 或转人工的场景>
+
+## Ownership
+- Owner：<团队或角色>
+- Last reviewed：<YYYY-MM-DD>
+- Related systems：<系统名称>
+
+## Preconditions
+- <执行前必须满足的权限、备份、窗口、健康状态或已知信息>
+
+## Inputs
+| 输入 | 来源 | 是否必需 | 校验方法 |
+|---|---|---:|---|
+| <参数> | <来源> | 是/否 | <如何确认有效> |
+
+## Safety
+### Non-negotiable rules
+- 先只读诊断，后执行变更。
+- 不得把删除现有配置作为首次恢复动作。
+- 不得猜测或编造缺失参数。
+- 不得绕过失败的测试、检查或审批。
+- 每次变更后必须完成对应验证。
+- 破坏性操作必须获得明确批准。
+
+### Stop conditions
+- 实际状态与本文档的前提或预期结果冲突。
+- 缺少必要输入、权限、审批或回滚能力。
+- 验证失败且本文档没有明确的下一步。
+- 影响范围超出 Scope。
+
+### Approval gates
+| 动作 | 风险级别 | 是否需要明确批准 | 批准记录位置 |
+|---|---|---:|---|
+| <动作> | 低/中/高 | 是/否 | <Issue / PR / 变更单> |
+
+## Procedure
+
+### Step 1 — Diagnose
+
+**Action**
+
+<执行只读诊断动作。>
+
+**Expected**
+
+<列出预期输出、状态或证据。>
+
+**Decision**
+
+- 若 <条件 A>，进入 Step 2。
+- 若 <条件 B>，进入 Troubleshooting A。
+- 若无法判断或状态冲突，`STOP` 并记录证据。
+
+### Step 2 — Change
+
+**Action**
+
+<描述单一、可审计的变更动作。>
+
+**Expected**
+
+<变更后应出现的状态。>
+
+**Verification**
+
+<给出可重复执行的验证命令、测试、监控指标或检查清单。>
+
+**Rollback**
+
+- 触发条件：<什么情况需要回滚>
+- 回滚动作：<如何撤销>
+- 回滚验证：<如何确认恢复成功>
+
+## Troubleshooting
+
+### Troubleshooting A — <异常名称>
+
+- 证据收集：<日志、指标、命令输出、链接>
+- 允许动作：<仅限已验证且低风险的动作>
+- 下一步：<回到某步 / 转入另一 Runbook / STOP 并升级>
+
+## Final Verification
+
+只有同时满足以下标准，流程才算成功：
+
+- <功能或服务状态>
+- <自动化测试或健康检查>
+- <监控指标或告警状态>
+- <变更记录、PR 或 Issue 已更新>
+
+## Failure Handling
+
+若未能完成：
+
+1. 停止进一步变更。
+2. 收集 <命令输出、时间范围、请求 ID、日志链接、截图或复现步骤>。
+3. 记录已完成步骤、实际结果、未满足的预期和是否执行过回滚。
+4. 按 <升级渠道> 交接，不继续猜测。
+
+## References
+
+- <关联 Issue、PR、架构文档、仪表盘、配置仓库或外部文档>
+```
+
+## 6. 编写步骤的标准写法
+
+每个步骤应只承担一个清晰目的，并使用“动作—预期—决策”的闭环表达。如下表所示，前者会导致 Agent 自主扩大操作范围，后者则为其提供安全边界。
+
+| 不推荐写法 | 推荐写法 |
+|---|---|
+| “检查部署是否正常，不正常就修复。” | “读取部署状态与最近一次发布记录。若所有副本 `Ready` 且版本等于目标版本，进入 Final Verification；若副本未就绪，收集事件与日志并进入 Troubleshooting A；若版本不匹配且原因未知，`STOP`。” |
+| “必要时修改配置。” | “仅当配置差异与变更单 `CHG-123` 完全一致且审批已记录时，应用指定键的值；应用后运行健康检查；失败则按 Rollback 回退。” |
+| “测试失败时可先跳过。” | “任何必需测试失败均不得继续部署。记录失败测试、日志和提交版本；仅按 Troubleshooting B 处理。” |
+
+## 7. 通用安全规则
+
+以下规则适合在每份 Runbook 的 `Safety` 章节中复用。若某流程存在更严格要求，应以更严格要求为准。
+
+```markdown
+## Safety Rules
+
+- Never delete an existing configuration as the first recovery action.
+- Prefer read-only diagnosis before mutation.
+- After every mutation, verify the expected state.
+- If actual state conflicts with this runbook, STOP.
+- Do not invent missing parameters.
+- Do not bypass failed tests.
+- Destructive actions require explicit approval.
+```
+
+这些约束体现了一个关键顺序：**先证据，后变更；先小范围，后扩大；先验证，后结束；不确定则停止。** 特别是停止条件必须可操作，例如“权限不足”“缺少变更单”“生产状态与前提不一致”“错误率超过 1%”等，而不应写成“情况复杂时停止”。
+
+## 8. 运行与审计流程
+
+Agent 执行 Runbook 时，应按照固定运行模型工作。每一步的输入、动作、输出和下一跳都应可追踪，这与 AWS 自动化 Runbook 的顺序步骤和输出传递思想一致。[3]
+
+```text
+输入与前置条件
+        ↓
+只读诊断
+        ↓
+确认预期状态或决策分支
+        ↓
+获取审批（如需要）
+        ↓
+执行最小变更
+        ↓
+立即验证
+        ↓
+成功收尾 / 回滚 / 停止并升级
+```
+
+| 阶段 | Agent 必须产出的证据 | 禁止行为 |
+|---|---|---|
+| 输入确认 | 参数来源、环境、目标资源、权限与审批状态 | 用猜测值补全必需参数 |
+| 诊断 | 命令输出、日志、指标或页面状态 | 在未诊断前直接修改生产状态 |
+| 变更 | 实际执行内容、变更范围、时间 | 将多个无关变更混在一起执行 |
+| 验证 | 测试、健康检查、监控状态与预期对比 | 以“命令执行成功”代替业务验证 |
+| 失败处理 | 已做步骤、异常证据、回滚状态和升级对象 | 无限制重试或绕过失败检查 |
+
+## 9. 从人工操作到自动化的成熟路径
+
+不建议在流程尚未稳定时先构建复杂 DSL 或全自动编排。应先积累真实案例，把可重复部分固化为 Markdown Runbook，再把已稳定、低歧义、可验证的操作迁移到脚本、CI、Skill 或自动化系统。Google SRE 将能够由机器替代的重复性人工工作视为应逐步消除的 toil。[4]
+
+| 阶段 | 主要形式 | 人的角色 | 自动化边界 |
+|---|---|---|---|
+| 1. 人工处理 | 现场处置与复盘 | 执行、判断、记录 | 不自动化 |
+| 2. Markdown Runbook | 固化步骤与证据要求 | 审核流程与异常判断 | Agent 可辅助诊断 |
+| 3. Agent + Runbook | 严格按流程执行 | 审批高风险动作、处理例外 | 受停止条件约束的执行 |
+| 4. Script / Skill / CI / Automation | 把稳定步骤程序化 | 处理异常和维护自动化 | 自动完成重复性操作 |
+| 5. 人工审批 + 自动执行 | 常规流程端到端运行 | 决策、审计与治理 | 审批门控下的自动变更 |
+
+## 10. 上线前检查清单
+
+在将一份新 Runbook 交给 Agent 使用前，建议由流程所有者按以下清单审核。
+
+| 检查项 | 合格标准 |
+|---|---|
+| 问题边界 | Purpose 与 Scope 清楚描述适用和不适用情形 |
+| 输入 | 所有必需输入都有来源、格式和校验方法 |
+| 步骤 | 每一步均有 Action、Expected 与明确的下一跳 |
+| 变更控制 | 所有修改动作都有 Verification；关键动作有 Rollback |
+| 安全控制 | Stop conditions、审批门槛和禁止行为已列明 |
+| 异常处理 | 失败时知道收集什么证据、交给谁，而非继续猜测 |
+| 可维护性 | 有 Owner、最近复审日期与关联文档；已在版本控制中评审 |
+| 可演练性 | 已在安全环境或历史案例上走通至少一次 |
+
+## 11. 建议的首批 Runbook
+
+首次落地时，应优先选择频率较高、输入相对明确、变更可回退的场景。以下集合通常能覆盖大部分工程协作的基础需求。
+
+| Runbook | 目的 | 关键安全控制 |
+|---|---|---|
+| `issue-to-merge.md` | 从已明确 Issue 到可评审变更 | Scope 锁定、测试门槛、PR 证据 |
+| `fix-ci.md` | 诊断并修复 CI 失败 | 不跳过测试、不修改无关代码 |
+| `release.md` | 执行标准发布 | 发布窗口、审批、健康检查、回滚点 |
+| `rollback.md` | 恢复到已知稳定版本 | 明确触发条件、版本选择、回滚后验证 |
+| `network-change.md` | 实施受控网络配置变更 | 影响评估、变更单、回退配置 |
+| `network-recovery.md` | 处理网络异常与服务恢复 | 只读诊断优先、状态冲突即停止 |
+
+## 12. 结论
+
+Agent Runbook 的价值不在于把每一项运维工作立即自动化，而在于将团队的工程判断编码为**可路由、可验证、可停止、可回滚**的操作系统。对于多数团队，从仓库中的 `AGENTS.md`、`RUNBOOKS.md` 和一组 Markdown Runbook 起步，已经足够实用。
+
+当某个流程经过多次执行、输入稳定、异常分支收敛且验证可靠后，再将其下沉为脚本、CI 或其他自动化能力。这样既能逐步降低重复性 toil，也能始终保留人类对高风险和例外情形的决策权。[4]
+
+## References
+
+[1]: https://sre.google/sre-book/managing-incidents/ "Google SRE Book — Managing Incidents"
+[2]: https://sre.google/sre-book/effective-troubleshooting/ "Google SRE Book — Effective Troubleshooting"
+[3]: https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-documents.html "AWS Systems Manager — Creating your own runbooks"
+[4]: https://sre.google/sre-book/eliminating-toil/ "Google SRE Book — Eliminating Toil"
+[5]: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-automation.html "AWS Systems Manager Automation"
+[6]: https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/automation-runbook-reference.html "AWS Systems Manager Automation Runbook Reference"
+[7]: https://learn.microsoft.com/en-us/azure/automation/manage-runbooks "Microsoft Learn — Manage runbooks in Azure Automation"
+
+---
+
+**来源**：Manus AI《Agent Runbook 实用指南（v1.0）》，本仓库存档为规范参考。
@@ -0,0 +1,34 @@
+# Runbook index
+
+Entry point for all runbooks. Before operational work, read the repo entry
+[`AGENTS.md`](../AGENTS.md) and the spec [`RUNBOOKS.md`](../RUNBOOKS.md). New
+runbooks start from [`_template.md`](_template.md).
+
+## Route by intent
+
+| Intent | Runbook | Type |
+|---|---|---|
+| mailcow health check | [mailcow-health.md](mailcow-health.md) | read-only |
+| mailcow update | [mailcow-update.md](mailcow-update.md) | change (gated) |
+| mailcow SMTP/IMAP client | [mailcow-smtp-client.md](mailcow-smtp-client.md) | reference |
+| Vaultwarden health check | [vaultwarden-health.md](vaultwarden-health.md) | read-only |
+| Vaultwarden SQLite→PG migrate | [vaultwarden-sqlite-to-postgres.md](vaultwarden-sqlite-to-postgres.md) | change (destructive) |
+| PowerDNS health check | [pdns-health.md](pdns-health.md) | read-only |
+| RustDesk health check | [rustdesk-health.md](rustdesk-health.md) | read-only |
+| Matrix health check | [matrix-health.md](matrix-health.md) | read-only |
+| AdGuard Home health check | [adguard-home-health.md](adguard-home-health.md) | read-only |
+| Routine Ansible operations | [ansible-operations.md](ansible-operations.md) | change (allowlisted) |
+| Linear issue → mergeable change | [issue-to-merge.md](issue-to-merge.md) | delivery |
+| Failing health/playbook run | [fix-ci.md](fix-ci.md) | change |
+| Release a reviewed change to production | [release.md](release.md) | change (gated) |
+| Roll back a change | [rollback.md](rollback.md) | change (gated) |
+| Controlled network configuration | [network-change.md](network-change.md) | change (gated) |
+| Network outage / service recovery | [network-recovery.md](network-recovery.md) | recovery |
+
+## Notes
+
+- `fix-ci.md`, `release.md`, `rollback.md`, `network-change.md`, `network-recovery.md`
+  are adapted from the upstream guide to this repo's VPS-ops context (execution
+  layer is Ansible + SSH + Linear, not a software CI/CD pipeline).
+- Health runbooks are read-only; they stop (`STOP`) when live state conflicts
+  with the expected state instead of mutating production.
@@ -0,0 +1,119 @@
+# Runbook: <名称>
+
+## Purpose
+
+<说明本 Runbook 要解决的问题及成功结果，1–2 行。>
+
+## Scope
+
+- 适用环境：<production / staging / LAN …>
+- 适用对象：<服务、主机、组件或告警类型>
+- 不适用情形：<需要改用其他 runbook 或转人工的场景>
+
+## Ownership
+
+- Owner：<团队或角色>
+- Last reviewed：<YYYY-MM-DD>
+- Related systems：<主机名 / 服务名>
+
+## Preconditions
+
+- <执行前必须满足的权限、备份、窗口、健康状态或已知信息>
+
+## Inputs
+
+| 输入 | 来源 | 是否必需 | 校验方法 |
+|---|---|---:|---|
+| <参数> | <来源> | 是/否 | <如何确认有效> |
+
+## Safety
+
+### Non-negotiable rules
+
+- 先只读诊断，后执行变更。
+- 不得把删除现有配置作为首次恢复动作。
+- 不得猜测或编造缺失参数。
+- 不得绕过失败的测试、检查或审批。
+- 每次变更后必须完成对应验证。
+- 破坏性操作必须获得明确批准。
+
+### Stop conditions
+
+- 实际状态与本文档的前提或预期结果冲突。
+- 缺少必要输入、权限、审批或回滚能力。
+- 验证失败且本文档没有明确的下一步。
+- 影响范围超出 Scope。
+
+### Approval gates
+
+| 动作 | 风险级别 | 是否需要明确批准 | 批准记录位置 |
+|---|---|---:|---|
+| <动作> | 低/中/高 | 是/否 | <Issue / PR / 变更单> |
+
+## Procedure
+
+### Step 1 — Diagnose
+
+**Action**
+
+<执行只读诊断动作。>
+
+**Expected**
+
+<列出预期输出、状态或证据。>
+
+**Decision**
+
+- 若 <条件 A>，进入 Step 2。
+- 若 <条件 B>，进入 Troubleshooting A。
+- 若无法判断或状态冲突，`STOP` 并记录证据。
+
+### Step 2 — Change
+
+**Action**
+
+<描述单一、可审计的变更动作。>
+
+**Expected**
+
+<变更后应出现的状态。>
+
+**Verification**
+
+<给出可重复执行的验证命令、测试、监控指标或检查清单。>
+
+**Rollback**
+
+- 触发条件：<什么情况需要回滚>
+- 回滚动作：<如何撤销>
+- 回滚验证：<如何确认恢复成功>
+
+## Troubleshooting
+
+### Troubleshooting A — <异常名称>
+
+- 证据收集：<日志、指标、命令输出、链接>
+- 允许动作：<仅限已验证且低风险的动作>
+- 下一步：<回到某步 / 转入另一 runbook / STOP 并升级>
+
+## Final Verification
+
+只有同时满足以下标准，流程才算成功：
+
+- <功能或服务状态>
+- <自动化测试或健康检查>
+- <监控指标或告警状态>
+- <变更记录、PR 或 Issue 已更新>
+
+## Failure Handling
+
+若未能完成：
+
+1. 停止进一步变更。
+2. 收集 <命令输出、时间范围、请求 ID、日志链接、截图或复现步骤>。
+3. 记录已完成步骤、实际结果、未满足的预期和是否执行过回滚。
+4. 按 <升级渠道> 交接，不继续猜测。
+
+## References
+
+- <关联 Issue、PR、架构文档、仪表盘、配置仓库或外部文档>
@@ -1,5 +1,14 @@
 # AdGuard Home health — dns.windy.lan

+## Purpose
+
+Read-only health check of the AdGuard Home LAN DNS service.
+
+## Scope
+
+- Applicable: [dns.windy.lan](../hosts/dns.windy.lan.md) (`192.168.66.36`).
+- Read-only: does not expose query-log contents or secrets; does not change configuration.
+
 This runbook is read-only.  It does not expose query-log contents or secrets.

 Routine checks run through Ansible on demand:
@@ -58,3 +67,9 @@ a known-bad-signature test; an enabled DO bit alone is not validation.

 Private PTR forwarding is intentionally absent because the EdgeRouter does
 not currently answer private PTR requests.
+
+## Safety
+
+- Read-only: never change the DNS policy or the `agh-ui-access.service` nftables rule during this check.
+- Do not infer a broken DNS policy from an empty `allowed_clients`.
+- If live state conflicts with an expected value, `STOP` and report.
@@ -1,8 +1,23 @@
 # Runbook: routine operations through Ansible

+## Purpose
+
+Routine operations (health, reconcile, maintenance) through the Ansible playbooks.
+
+## Scope
+
+- Applicable: every inventory host, run from `ansible/`.
+- Not applicable: arbitrary remote commands — the reconcile playbook is allowlisted and gated.
+
 Run commands from `ansible/`. The inventory forces IPv4 and uses the `windy`
 account with sudo. Do a read-only health pass before any reconciliation.

+## Safety
+
+- Read-only health pass before any reconciliation.
+- Mutating playbooks require explicit confirmation variables; do not bypass them.
+- If a reconcile target or service name is not allowlisted, `STOP` — do not invent one.
+
 ## Health report (read-only)

 ```bash
@@ -0,0 +1,70 @@
+# Runbook: fix a failing health/playbook run
+
+> Adapted from the upstream guide's `fix-ci`. This repo has no software CI; the
+> equivalent "pipeline" is the Ansible **health report** and the gated playbooks.
+> This runbook covers diagnosing and fixing a failed or warning/critical run.
+
+## Purpose
+
+Diagnose and fix a failing Ansible health-report or playbook run without
+skipping checks or changing unrelated code.
+
+## Scope
+
+- Applicable: `ansible-playbook playbooks/health-report.yml` and the gated playbooks under `ansible/playbooks/`.
+- Not applicable: production changes beyond fixing the run; network/DNS changes → `network-change.md`.
+
+## Safety
+
+- Do not skip or weaken a failing check to make it pass.
+- Do not change unrelated hosts or services.
+- Prefer read-only diagnosis before mutation; destructive fixes require approval.
+
+## Procedure
+
+### Step 1 — Reproduce and read
+
+**Action** — re-run the failing playbook with `--limit <host>` and capture the task that failed.
+
+```bash
+cd ansible
+ansible-playbook playbooks/health-report.yml --limit <host> -v
+```
+
+**Expected** — a specific failed task, host, and message (warning vs critical).
+
+**Decision** — clear failure → Step 2; ambiguous → `STOP` and collect `-vvv` output + the relevant `latest.json`.
+
+### Step 2 — Diagnose
+
+**Action** — inspect the corresponding service on the host using the matching health runbook (`mailcow-health.md`, `vaultwarden-health.md`, `pdns-health.md`, etc.).
+
+**Expected** — a root cause (container down, cert expired, queue backlog, drift).
+
+**Decision** — root cause found → Step 3; live state conflicts with the runbook's assumptions → `STOP`.
+
+### Step 3 — Fix within scope
+
+**Action** — apply the minimal fix the service runbook prescribes (e.g. `compose-reconcile` for a config drift, or a documented update). Use only allowlisted/gated playbooks.
+
+**Verification** — re-run the health report and confirm it passes.
+
+**Rollback** — revert to the prior config/state and re-run; see `rollback.md` for the general procedure.
+
+## Troubleshooting
+
+### Troubleshooting A — Intermittent/flaky failure
+
+- Evidence: timing, DNS stub flakiness (use `1.1.1.1`/`8.8.8.8` for probes).
+- Allowed: re-run once with the documented resolver workaround.
+- Next: still failing → `STOP` and escalate.
+
+## Final Verification
+
+- Health report passes for the affected host.
+- No checks were skipped or weakened; the fix is committed/documented.
+
+## References
+
+- [`ansible-operations.md`](ansible-operations.md)
+- Per-service health runbooks under [`runbooks/`](.)
@@ -0,0 +1,89 @@
+# Runbook: issue → mergeable change
+
+## Purpose
+
+Turn an approved Linear `vps` issue into a reviewed, mergeable change in this
+repo (docs, runbooks, hosts facts, or Ansible playbooks).
+
+## Scope
+
+- Applicable: repo content under `docs/`, `runbooks/`, `hosts/`, `inventory/`, `ansible/`.
+- Not applicable: mutating production state directly — that goes through `release.md` / `ansible-operations.md`.
+
+## Ownership
+
+- Owner: personal ops (Windy)
+- Last reviewed: 2026-08-17
+- Related systems: Linear MCP (`vps` project), git
+
+## Inputs
+
+| Input | Source | Required | Validation |
+|---|---|---:|---|
+| Issue identifier | Linear (`vps` project) | Yes | `linear_get_issue <id>` returns a description |
+| Current repo state | `git status` / `git log` | Yes | Clean or intended worktree |
+
+## Safety
+
+- Scope is locked to the issue: do not bundle unrelated changes.
+- Never commit secrets (see `AGENTS.md` §Safety).
+- Verify every change; do not merge a change whose verification was skipped.
+
+## Procedure
+
+### Step 1 — Read the issue
+
+**Action** — `linear_get_issue <id>`, read description and acceptance criteria.
+
+**Expected** — clear scope, action, and verification for the change.
+
+**Decision** — if the issue is ambiguous or lacks verification criteria, `STOP`
+and ask for clarification (add a `needs-info` label if applicable). Otherwise go to Step 2.
+
+### Step 2 — Inspect and change
+
+**Action** — read the relevant files, then make the minimal change the issue asks for.
+
+**Expected** — diff is scoped to the issue.
+
+**Decision** — if the change needs production mutation, `STOP` and route to
+`release.md`. Otherwise go to Step 3.
+
+### Step 3 — Verify
+
+**Action** — run the applicable verification (link check for docs; `ansible-playbook --check` / `--syntax-check` for playbooks).
+
+**Verification** — see `RUNBOOKS.md` §校验; the concrete checks must match the change type.
+
+**Decision** — verification passed → Step 4; failed → Troubleshooting A.
+
+### Step 4 — Commit and link
+
+**Action** — commit with a message containing the full issue ID (e.g. `W1N-123: …`); open a PR if the change is substantial; link the issue via `linear_save_comment`.
+
+**Verification** — `git log -1` shows the issue ID; the issue has the commit/PR pointer.
+
+**Rollback** — `git revert <sha>` or `git checkout <branch>` to drop the change; re-verify after.
+
+## Troubleshooting
+
+### Troubleshooting A — Verification failed
+
+- Evidence: command output, failing check.
+- Allowed: fix the change within scope; re-run verification.
+- Next: still failing → `STOP` and report in the issue.
+
+## Final Verification
+
+- Change matches the issue scope.
+- Verification passed and the issue is updated with evidence.
+
+## Failure Handling
+
+If unfinished: stop, collect the failed check output, record completed steps, and
+hand back to the issue — do not guess.
+
+## References
+
+- [`docs/agents/issue-tracker.md`](../docs/agents/issue-tracker.md)
+- [`RUNBOOKS.md`](../RUNBOOKS.md)
@@ -1,5 +1,14 @@
 # Runbook: mailcow health (mx2)

+## Purpose
+
+Read-only health check of the mailcow stack on mx2.
+
+## Scope
+
+- Applicable: [mx2.windy.me](../hosts/mx2.windy.me.md), `/opt/mail`.
+- Read-only: does not change mailcow configuration or service state.
+
 Target: [mx2.windy.me](../hosts/mx2.windy.me.md)  
 Path: `/opt/mail`  
 Prefer: the Ansible health report (`ansible/playbooks/health-report.yml --limit mailcow`),
@@ -71,6 +80,11 @@ The sanitized Ansible health profile is `mailcow` (`ansible/playbooks/healthchec
 The server-local timer emits a sanitized result at `/var/lib/vps-health/latest.json`.
 It does not change Mailcow configuration or service state.

+## Safety
+
+- Read-only: never mutate configuration or service state during this check.
+- If live state conflicts with an expected value below, `STOP` and report; do not "fix" on the fly.
+
 ## Pass criteria

 - Compose stack up; watchdog ~100%
@@ -1,5 +1,14 @@
 # Runbook: use mailcow SMTP / IMAP (client)

+## Purpose
+
+Reference for configuring mail clients against the mailcow SMTP/IMAP endpoints.
+
+## Scope
+
+- Applicable: [mx2.windy.me](../hosts/mx2.windy.me.md) client submission (587/465) and IMAP/POP (993/995).
+- Not applicable: server-side mailcow configuration or administration.
+
 Target: [mx2.windy.me](../hosts/mx2.windy.me.md)  
 Prerequisite: a mailbox on `windy.me` (password from mailcow UI, not the admin account unless it is that mailbox).

@@ -52,3 +61,8 @@ Do not commit or paste real passwords into this repo.
 - Port/TLS mode mismatch (587 vs 465)
 - Account active in mailcow; not rate-limited / fail2banned after bad attempts
 - Apps that store SMTP in their own config (e.g. Vaultwarden `config.json`) may keep a **stale** password even when `.env` is correct — verify AUTH against the effective config ([vaultwarden-health](vaultwarden-health.md) §5)
+
+## Safety
+
+- Do not commit or paste real passwords into this repo or chat.
+- Use submission (587/465) for client sending; never use port 25 as a desktop/app outbound port.
@@ -1,9 +1,30 @@
 # Runbook: mailcow update (mx2)

+## Purpose
+
+Update the mailcow stack on mx2 to the latest supported release.
+
+## Scope
+
+- Applicable: [mx2.windy.me](../hosts/mx2.windy.me.md), `/opt/mail`.
+- Not applicable: config changes beyond the update, DB migration, secret rotation.
+
+## Approval gates
+
+| Action | Risk | Explicit approval |
+|---|---|---|
+| Run `./update.sh` (recreates containers, brief mail interruption) | Medium | Yes — user confirmation required |
+
 Target: [mx2.windy.me](../hosts/mx2.windy.me.md)  
 Path: `/opt/mail`  
 **Confirm with the user before running an update.**

+## Safety
+
+- Never run the update without explicit user confirmation.
+- Never pass secrets into the chat log; do not commit `mailcow.conf`.
+- If a step fails, capture `docker compose ps` and logs and stop before further changes.
+
 ## Before

 1. Run [mailcow-health](mailcow-health.md) (Ansible health report). Record baseline.
@@ -1,5 +1,14 @@
 # Matrix Health Check

+## Purpose
+
+Read-only health check of the Matrix homeserver (ESS on K3s).
+
+## Scope
+
+- Applicable: [synapse.chans.xyz](../hosts/synapse.chans.xyz.md), namespace `ess`.
+- Read-only: does not change pods, ingress, certificates, or configuration.
+
 Monitor the Matrix homeserver running on `synapse.chans.xyz` (ESS chart `26.7.2`, K3s node).

 Prefer `cd ansible && ansible-playbook playbooks/health-report.yml --limit matrix`
@@ -90,3 +99,9 @@ Backup automation is currently paused. `/var/backups/matrix/` is retained for a
 | Well-known returns 404/redirect | Root `chans.xyz` ingress missing or misconfigured |
 | 502 Bad Gateway | Synapse pod restarting or DB down |
 | SMTP emails not sent | MAS SMTP config incomplete; TCP reachable but AUTH failing — see `runbooks/vaultwarden-health.md` |
+
+## Safety
+
+- Read-only: never mutate pods, ingress, certificates, or configuration during this check.
+- Backup automation is paused; do not treat `/var/backups/matrix/` as a recovery source.
+- If live state conflicts with an expected value, `STOP` and report.
@@ -0,0 +1,68 @@
+# Runbook: controlled network change
+
+## Purpose
+
+Apply a controlled network configuration change (DNS records, firewall, LAN
+gateway, VLAN) with impact assessment, approval, and a rollback path.
+
+## Scope
+
+- Applicable: PowerDNS zone records, `us4` firewalld allowlist, LAN gateway/VLAN/DNS changes, WireGuard.
+- Not applicable: SSH access-policy changes (see `AGENTS.md` §SSH access safety — mandatory lockout-risk procedure).
+
+## Preconditions
+
+- A change record (Linear `vps` issue) describes the change, its reason, and rollback.
+- Read-only impact assessment done (current config captured, blast radius known).
+
+## Safety
+
+- Never change DNS or network config without a change record and approval.
+- Capture the current config first; never delete existing config as the first action.
+- For DNS: record the current record values and TTL before editing.
+- For firewall: retain an independent SSH rollback session before applying (see `ansible-operations.md` §us4).
+
+## Procedure
+
+### Step 1 — Assess and capture
+
+**Action** — capture the current state (e.g. `dig` for DNS, `--check --diff` for firewall, `show` for gateway).
+
+**Expected** — a baseline of current config and an identified blast radius.
+
+**Decision** — change fully specified with rollback → Step 2; missing → `STOP`.
+
+### Step 2 — Approve
+
+**Action** — confirm approval is recorded in the issue/change record.
+
+**Decision** — approved → Step 3; not approved → `STOP`.
+
+### Step 3 — Change
+
+**Action** — apply the single change (edit the record, run the gated playbook, or change gateway config) and only that change.
+
+**Expected** — the new value/state is in effect.
+
+**Verification** — re-query/verify the new state and confirm dependent services still pass health.
+
+**Rollback** — restore the captured prior config and re-verify.
+
+## Troubleshooting
+
+### Troubleshooting A — Change broke dependent service
+
+- Evidence: health report / endpoint failure.
+- Allowed: roll back to the captured prior config.
+- Next: verify; if still broken, escalate.
+
+## Final Verification
+
+- New state verified; dependent services healthy.
+- Change and outcome recorded in the issue.
+
+## References
+
+- [`ansible-operations.md`](ansible-operations.md)
+- [`rollback.md`](rollback.md)
+- [`network-recovery.md`](network-recovery.md)
@@ -0,0 +1,61 @@
+# Runbook: network outage / service recovery
+
+## Purpose
+
+Recover from a network outage or service failure, starting from read-only
+diagnosis and mutating only when the root cause is confirmed.
+
+## Scope
+
+- Applicable: unreachable VPS services, LAN gateway/DNS failures, DNS resolution failures.
+- Not applicable: planned changes (→ `network-change.md`), SSH access recovery (→ `AGENTS.md` §SSH access safety).
+
+## Safety
+
+- Read-only diagnosis first; do not mutate while the root cause is unknown.
+- If live state conflicts with a runbook's assumptions, `STOP` and report.
+- Keep the current verified management session open as the recovery path.
+
+## Procedure
+
+### Step 1 — Diagnose (read-only)
+
+**Action** — gather evidence without changing anything:
+
+```bash
+# From laptop, pin DNS to a public resolver if the stub is flaky
+dig @1.1.1.1 +short <host> A
+curl -4 -sS -I --max-time 10 https://<host>/
+# From a reachable host, inspect the service
+ssh -4 windy@<host> 'docker compose ps -a; df -h /; tail -n 50 /var/lib/vps-health/latest.json'
+```
+
+**Expected** — a clear picture: is it DNS, connectivity, host, or service?
+
+**Decision** — root cause localized → Step 2; ambiguous or conflicting → `STOP` and escalate (provider console if host is unreachable).
+
+### Step 2 — Confirm and route
+
+**Action** — match the failure to the owning runbook (`mailcow-health.md`, `pdns-health.md`, `matrix-health.md`, etc.) or `network-change.md` for a config fix.
+
+**Expected** — an applicable runbook with a recovery action.
+
+**Decision** — applicable → follow it; none → `STOP` (diagnose only, do not mutate).
+
+### Step 3 — Recover (gated)
+
+**Action** — apply only the runbook's documented recovery, with approval.
+
+**Verification** — re-run the health report / endpoint check and confirm recovery.
+
+**Rollback** — if recovery makes it worse, revert per `rollback.md`.
+
+## Final Verification
+
+- Service reachable and health report green.
+- Incident and recovery recorded in the Linear `vps` issue.
+
+## References
+
+- [`network-change.md`](network-change.md)
+- Per-service health runbooks under [`runbooks/`](.)
@@ -1,5 +1,14 @@
 # PowerDNS health (hk2)

+## Purpose
+
+Read-only health check of the `/opt/pdns` PowerDNS stack.
+
+## Scope
+
+- Applicable: [hk2.chans.xyz](../hosts/hk2.chans.xyz.md), `/opt/pdns`.
+- Read-only: does not change PowerDNS, DNS records, or secrets.
+
 Read-only checks for the `/opt/pdns` stack on **hk2.chans.xyz** (`ns1.wsvc.info`).

 Facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) · Upstream: [docs/pdns-upstream.md](../docs/pdns-upstream.md)
@@ -87,6 +96,12 @@ Expect: `primary=yes`, `also-notify=202.91.35.141`, `only-notify=` empty, `gpgsq
 The sanitized Ansible health profile is `pdns` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses the API key only inside the PowerDNS container. It does not modify PowerDNS, DNS records, or secrets.


+## Safety
+
+- Read-only: never mutate PowerDNS configuration or DNS records during this check.
+- Do not paste the API key into chat/logs.
+- If live state conflicts with an expected value, `STOP` and report.
+
 ## After config changes

 - `auth/pdns.conf`, `auth/templates.d/secrets.j2`, or auth-related `.env` → use the Ansible Compose reconcile playbook with target `auth`
@@ -0,0 +1,73 @@
+# Runbook: release a reviewed change to production
+
+## Purpose
+
+Apply a reviewed configuration change to production hosts in a controlled,
+verifiable, rollback-safe way.
+
+## Scope
+
+- Applicable: reviewed changes to Compose services, DNS, firewall, or Ansible-managed config on production hosts.
+- Not applicable: destructive migrations (`vaultwarden-sqlite-to-postgres.md`), SSH access-policy changes (`AGENTS.md` §SSH access safety), or network changes needing a change ticket (`network-change.md`).
+
+## Preconditions
+
+- The change is reviewed and its intent matches a Linear issue / change record.
+- Read-only health pass (or `maintenance-preview`) has been run and is green.
+- A rollback point exists (backup, prior config, or known-good git/playbook state).
+
+## Safety
+
+- Never release without a verification and a rollback point.
+- Never bypass gated confirmation variables.
+- One host at a time; re-run health after each.
+
+## Procedure
+
+### Step 1 — Pre-flight
+
+**Action** — run the read-only health report for the target host(s).
+
+```bash
+cd ansible && ansible-playbook playbooks/health-report.yml --limit <host>
+```
+
+**Expected** — green baseline.
+
+**Decision** — green → Step 2; warning/critical → fix first (`fix-ci.md`), `STOP` the release.
+
+### Step 2 — Approve
+
+**Action** — confirm the change is approved in the issue/change record; confirm the window.
+
+**Decision** — approval recorded → Step 3; missing → `STOP`.
+
+### Step 3 — Apply
+
+**Action** — run the gated playbook for the change, with its confirmation variable.
+
+```bash
+# example: reviewed Compose reconcile
+cd ansible && ansible-playbook playbooks/compose-reconcile.yml --limit <host> \
+  -e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["<svc>"]}'
+```
+
+**Expected** — playbook completes; service reports the new state.
+
+**Verification** — re-run the health report and confirm green; confirm the specific change took effect (version, config, endpoint).
+
+**Rollback** — on verification failure, revert the change and re-verify (see `rollback.md`).
+
+## Final Verification
+
+- Health report green post-release.
+- The change is visible and the issue/change record is updated.
+
+## Failure Handling
+
+On failure: stop further changes, collect the playbook output and health report, and route to `rollback.md` if the change was applied.
+
+## References
+
+- [`ansible-operations.md`](ansible-operations.md)
+- [`rollback.md`](rollback.md)
@@ -0,0 +1,56 @@
+# Runbook: roll back a change
+
+## Purpose
+
+Restore a known-good state after a change failed verification or caused a
+regression.
+
+## Scope
+
+- Applicable: Compose services, DNS records, firewall rules, and Ansible-managed config that have a rollback point.
+- Not applicable: destructive migrations without a backup (route to the owning runbook, e.g. `vaultwarden-sqlite-to-postgres.md` §Rollback).
+
+## Preconditions
+
+- A rollback point exists: backup, prior config, or known-good commit/playbook state.
+- The failing change and its symptoms are recorded.
+
+## Safety
+
+- Confirm the rollback target before acting; do not roll back unrelated changes.
+- Verify after rollback; a rollback that is not verified is not complete.
+
+## Procedure
+
+### Step 1 — Confirm the target
+
+**Action** — record the change to revert (playbook run, config edit, DNS change) and its intended prior state.
+
+**Expected** — the exact prior state is known and reachable.
+
+**Decision** — prior state known → Step 2; unknown → `STOP` and escalate.
+
+### Step 2 — Revert
+
+**Action** — apply the documented reversal for the change type:
+
+- Compose config: re-apply the prior config and reconcile (or `git revert` the change then reconcile).
+- DNS: restore the previous record value on PowerDNS.
+- Firewall: re-apply the prior allowlist via the gated playbook.
+- General repo change: `git revert <sha>`.
+
+**Expected** — the prior configuration is in place.
+
+**Verification** — re-run the relevant health report / endpoint check and confirm green.
+
+**Rollback** — if the rollback itself fails, `STOP`; keep the original management session open and escalate.
+
+## Final Verification
+
+- Health report green after rollback.
+- The incident and rollback are recorded in the Linear `vps` issue.
+
+## References
+
+- [`release.md`](release.md)
+- [`ansible-operations.md`](ansible-operations.md)
@@ -1,5 +1,14 @@
 # RustDesk server health (hk2)

+## Purpose
+
+Read-only health check of the `/opt/rustdesk` server stack.
+
+## Scope
+
+- Applicable: [hk2.chans.xyz](../hosts/hk2.chans.xyz.md), `/opt/rustdesk`.
+- Read-only: does not change RustDesk configuration or service state.
+
 Read-only checks for the `/opt/rustdesk` stack on **hk2.chans.xyz**.

 Facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md)
@@ -66,6 +75,11 @@ The sanitized Ansible health profile is `rustdesk` (deployed via
 locally through `vps-healthcheck.timer`, writes a sanitized JSON result to
 `/var/lib/vps-health/latest.json`, and never exposes secrets.

+## Safety
+
+- Read-only: never mutate configuration or service state during this check.
+- If live state conflicts with an expected value, `STOP` and report.
+
 ## After config changes

 Reviewed changes to `/opt/rustdesk/compose.yml` (relay address, image pin, port
@@ -1,5 +1,14 @@
 # Runbook: Vaultwarden health (us2)

+## Purpose
+
+Read-only health check of the Vaultwarden stack (Postgres backend) on us2.
+
+## Scope
+
+- Applicable: [us2.wsvc.info](../hosts/us2.wsvc.info.md), `/opt/vaultwarden`.
+- Read-only: does not change Vaultwarden, Traefik, SMTP, or secrets.
+
 Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)  
 Path: `/opt/vaultwarden`  
 URL: https://auth.wsvc.info/  
@@ -96,6 +105,12 @@ Expect `AUTH_OK`. `535` usually means stale password in `config.json` (see step
 The sanitized Ansible health profile is `vaultwarden` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses only server-side credentials for the SMTP AUTH probe. It does not modify Vaultwarden, Traefik, SMTP, or secrets.


+## Safety
+
+- Read-only: never mutate configuration, containers, or secrets during this check.
+- Never print passwords — compare lengths/hashes only.
+- If live state conflicts with an expected value below, `STOP` and report.
+
 ## Pass criteria

 - Healthy compose; `DATABASE_URL` points at `pg`
@@ -1,5 +1,20 @@
 # Runbook: Vaultwarden SQLite → Postgres (us2)

+## Purpose
+
+Replay checklist for the exceptional SQLite→Postgres migration on us2.
+
+## Scope
+
+- Applicable: [us2.wsvc.info](../hosts/us2.wsvc.info.md), `/opt/vaultwarden`.
+- Not applicable: routine reconcile — this is destructive and intentionally not automated.
+
+## Safety
+
+- Destructive: requires explicit approval; confirm backups exist before starting.
+- Do not commit `.env` or password-bearing files to git.
+- If any step diverges from this checklist, `STOP` and report.
+
 Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)  
 Upstream: [Using the PostgreSQL Backend](https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend) · [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)