docs(ha): add Home Assistant maintenance runbook + ha CLI script (W1N-69)

- runbooks/home-assistant-maintenance.md: access pattern (sudo -n -i ha), command reference verified on host, recovery ops, families not scripted, docs-vs-CLI discrepancies - runbooks/scripts/ha-maintenance.sh: read-only health/logs + --yes-gated update/restart/rebuild/rollback/reboot/backup/restore/app modes - AGENTS.md: register runbook in table; hosts/hass: access pattern + link
2026-08-13 17:54:33 +08:00
parent f5842568b9
commit b5617fd3a9
4 changed files with 406 additions and 4 deletions
@@ -76,6 +76,7 @@ Also readable as `agent.md` (symlink → this file).
 | Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) |
 | RustDesk health (hk2) | [runbooks/rustdesk-health.md](runbooks/rustdesk-health.md) |
 | AdGuard Home health | [runbooks/adguard-home-health.md](runbooks/adguard-home-health.md) |
+| Home Assistant maintenance | [runbooks/home-assistant-maintenance.md](runbooks/home-assistant-maintenance.md) + [scripts/ha-maintenance.sh](runbooks/scripts/ha-maintenance.sh) |
 | Matrix upstream refs | [docs/matrix-upstream.md](docs/matrix-upstream.md) |
 | Hermes Agent Matrix channel | [docs/hermes-matrix.md](docs/hermes-matrix.md) |
 | UniFi local-service proxy bypass | [docs/unifi-openclash-localhost.md](docs/unifi-openclash-localhost.md) |
@@ -13,10 +13,20 @@
 | Web UI | `http://hass.windy.lan:8123` (LAN); WAN port-forward `hass` on gw → `:8123` |

 Use `hassio` for routine SSH inspection. Key-only login was verified on
-2026-08-13 from the WSL client (`BatchMode=yes`). The `ha` supervisor CLI is
-present (`/usr/bin/ha`) but returns `unauthorized` without a supervisor API
-token; use the web UI or SSH shell for read-only checks unless a token is
-configured out of band.
+2026-08-13 from the WSL client (`BatchMode=yes`).
+
+The `ha` supervisor CLI (`/usr/bin/ha`) authenticates with `SUPERVISOR_TOKEN`.
+Interactive login works because `~hassio/.zprofile` runs `exec sudo -i`, which
+loads a root environment carrying the supervisor API token. Non-interactive
+`ssh hassio 'command'` does not source `.zprofile` and fails with
+`unauthorized: missing or invalid API token`. Run `ha` non-interactively via:
+
+```bash
+ssh -o BatchMode=yes hassio@hass.windy.lan 'sudo -n -i ha core info'
+```
+
+Verified 2026-08-13 that `sudo -n -i ha core info` works from the WSL client.
+Never copy the supervisor token into this repository.

 The current SSH ED25519 host-key fingerprint is
 `SHA256:DMcMOgDzFsFTon1fndXowEP7jlyOK3/AX3PVK8BATvk` (verified 2026-08-13).
@@ -99,5 +109,6 @@ getent hosts hass.windy.lan

 ## Related docs

+- [runbooks/home-assistant-maintenance.md](../runbooks/home-assistant-maintenance.md) — `ha` CLI maintenance runbook + [script](../runbooks/scripts/ha-maintenance.sh)
 - [docs/lan-overview.md](../docs/lan-overview.md) — LAN map and gw port-forward
 - [hosts/dns.windy.lan.md](dns.windy.lan.md) — `hass.windy.lan` / `hass.local` rewrites
@@ -0,0 +1,210 @@
+# Runbook: Home Assistant maintenance (hass.windy.lan)
+
+Target: [hass.windy.lan](../hosts/hass.windy.lan.md) (HAOS, `machine: green`)
+Upstream: HAOS 18.1 / Core 2026.8.x / Supervisor 2026.07.5 (verified 2026-08-13)
+
+This runbook covers routine Home Assistant maintenance through the **`ha`
+supervisor CLI**. All commands are wrapped by a single script
+[`scripts/ha-maintenance.sh`](scripts/ha-maintenance.sh); the sections below
+document the exact commands it runs, for manual/agent use.
+
+## Access pattern
+
+`ha` authenticates to the Supervisor with `SUPERVISOR_TOKEN`. Interactive SSH
+login works because `~hassio/.zprofile` runs `exec sudo -i`; the root login
+environment carries the token. Non-interactive use must be:
+
+```bash
+ssh -o BatchMode=yes hassio@hass.windy.lan 'sudo -n -i ha <cmd>'
+```
+
+Running `ha` as `hassio` directly (or `sudo -n` without `-i`) returns
+`unauthorized: missing or invalid API token`.
+
+**MOTD:** every `sudo -n -i` login prints the SSH & Web Terminal MOTD banner.
+The script runs its whole procedure in one remote login (`sudo -n -i bash -s`)
+so the banner appears once, then strips it with `awk` up to the
+`System is ready! Use browser or app to configure.` line.
+
+## Script usage
+
+```bash
+cd runbooks/scripts
+
+./ha-maintenance.sh                                # read-only health snapshot
+./ha-maintenance.sh --check-config                 # validate core configuration
+./ha-maintenance.sh --logs core 200                # tail core logs
+./ha-maintenance.sh --logs supervisor              # tail supervisor logs (default 100)
+./ha-maintenance.sh --logs host 50                 # tail host journald logs
+./ha-maintenance.sh --logs apps:<slug>             # tail an add-on log
+
+# Mutating — refuse to run without --yes:
+./ha-maintenance.sh --update --yes                 # refresh + update core(--backup)/supervisor/os
+./ha-maintenance.sh --restart-core --yes           # restart Core
+./ha-maintenance.sh --restart-core --safe-mode --yes   # restart Core in safe mode
+./ha-maintenance.sh --rebuild-core --yes           # rebuild Core image (after options change)
+./ha-maintenance.sh --rollback-os --yes            # boot previous OS slot (A/B rollback)
+./ha-maintenance.sh --reboot --yes                 # reboot the HAOS host
+./ha-maintenance.sh --backup [NAME] --yes          # full backup (optionally named)
+./ha-maintenance.sh --restore <slug> --yes         # restore a backup (DESTRUCTIVE)
+./ha-maintenance.sh --app restart core_mosquitto --yes   # add-on lifecycle
+```
+
+- `--app` action is one of `start|stop|restart|update`; needs an add-on slug.
+- `HA_HOST` / `HA_SSH_USER` override the defaults (`hass.windy.lan` / `hassio`).
+- `--restore` overwrites the current installation — run only from a planned
+  recovery, with the backup verified.
+
+## Command reference (verified 2026-08-13)
+
+All verified against the live host. MOTD prepends each command's output; strip
+with the `awk` pattern above or read the last block.
+
+### Routine / read-only
+
+| Purpose | Command |
+|---|---|
+| General overview | `ha info` |
+| Core version/status | `ha core info` |
+| Core config validation | `ha core check` |
+| Core stats | `ha core stats` |
+| Supervisor status | `ha supervisor info` (incl. add-on list) |
+| Supervisor stats | `ha supervisor stats` |
+| OS status | `ha os info` (boot slots A/B) |
+| Host status | `ha host info` (disk free/total, kernel) |
+| Network | `ha network info` (`supervisor_internet`) |
+| Hardware | `ha hardware info` |
+| Pending updates | `ha available-updates` |
+| Reload stores/versions | `ha refresh-updates` |
+| Job manager | `ha jobs info` |
+| Resolution center | `ha resolution info` |
+| Core logs | `ha core logs -n 100` (`-f` follow, `-b` boot id) |
+| Supervisor logs | `ha supervisor logs -n 100` |
+| Host journald logs | `ha host logs -n 100` |
+| Add-on logs | `ha apps logs <slug> -n 100` |
+| Add-on list | `ha supervisor info` → `addons:` (started/stopped/error) |
+| Security integrity | `ha security integrity` |
+
+### Mutating (require --yes)
+
+| Purpose | Command |
+|---|---|
+| Update core (with partial backup) | `ha core update --backup` |
+| Update supervisor | `ha supervisor update` |
+| Update OS | `ha os update` |
+| Update add-on | `ha apps update <slug>` |
+| Restart core | `ha core restart` / `ha core restart --safe-mode` |
+| Rebuild core | `ha core rebuild` |
+| OS rollback | `ha os boot-slot other` |
+| Reboot host | `ha host reboot` |
+| Full backup | `ha backups new [--name NAME]` |
+| Restore backup | `ha backups restore <slug>` |
+| Add-on start/stop/restart | `ha apps start\|stop\|restart <slug>` |
+
+## Procedure
+
+### 1. Health snapshot (read-only)
+
+```bash
+./ha-maintenance.sh
+```
+
+Review: supervisor `healthy: true`/`supported: true`; core/OS `update_available`;
+add-on states (any `state: error`?); `resolution info` issues; disk free.
+
+### 2. Validate config after any `configuration.yaml` change
+
+```bash
+./ha-maintenance.sh --check-config
+```
+
+Expect `Command completed successfully.` before a Core restart.
+
+### 3. Inspect logs
+
+```bash
+./ha-maintenance.sh --logs core 200
+./ha-maintenance.sh --logs supervisor
+./ha-maintenance.sh --logs apps:core_mosquitto
+```
+
+### 4. Apply updates (mutating)
+
+```bash
+./ha-maintenance.sh --update --yes
+```
+
+Runs `refresh-updates` → `core update --backup` (partial backup first) →
+`supervisor update` → `os update`, then re-prints pending updates. Prefer the
+web UI (**Settings → System → Updates**) for a human-supervised pass.
+
+### 5. Recovery operations (mutating, only when needed)
+
+```bash
+./ha-maintenance.sh --restart-core --safe-mode --yes   # start Core without custom integrations
+./ha-maintenance.sh --rollback-os --yes                # OS update broke boot? go back one slot
+./ha-maintenance.sh --restore <slug> --yes             # full restore; overwrites current install
+```
+
+OS update policy: HAOS uses two boot slots (A/B); `ha os info` shows which slot
+booted. After a bad OS update, `ha os boot-slot other` boots the previous slot.
+
+### 6. Backup before major changes
+
+```bash
+./ha-maintenance.sh --backup pre-migration --yes       # named backup
+```
+
+## Command families intentionally NOT scripted
+
+These exist in `ha` but are either rare, dangerous, or better done in the web
+UI; documented here so nothing is a surprise. Use `ha <family> --help` on the
+host for exact syntax.
+
+| Family | Notes |
+|---|---|
+| `ha audio` | Audio device management; peripheral. |
+| `ha authentication` | `auth list/reset/cache`; user password ops — do in web UI. `auth list` is local-terminal only. |
+| `ha cli` | Internal CLI backend info/update; self-maintained. |
+| `ha dns` | Internal DNS server; only relevant if Supervisor DNS add-on in use. |
+| `ha docker` | Host Docker backend info/options/registries; HAOS-managed. |
+| `ha mounts` | Network storage (NFS/CIFS) mounts — configure in **Settings → System → Storage**. |
+| `ha multicast` / `ha observer` | Internal services; self-maintained. |
+| `ha network scan/update/vlan` | WiFi AP scan & interface config — prefer web UI networking. |
+| `ha host disks/options/shutdown/reload` | Disk ops / host options; `shutdown` is equivalent to `--reboot` but off. |
+| `ha os datadisk list/move/wipe` | Data-disk migration; `wipe` is **local-terminal only** and erases all data. |
+| `ha os import` | Import config from USB stick. |
+| `ha os boards` / `os config` | Board / OS settings. |
+| `ha core options` / `supervisor options` | Core/OS config options (e.g. `--duplicate-log-file`); changes need `ha core rebuild` + restart. |
+| `ha backups freeze/thaw/remove/options` | Freeze/thaw for external backup tools; removal is destructive. |
+| `ha jobs options/reset` | Job-manager tuning. |
+| `ha resolution check/healthcheck/issue/suggestion` | Resolution center management; `healthcheck` runs fixups. |
+| `ha store add/delete/repair` | Repository management — add repos in web UI app store. |
+| `ha security info/options` | Security backend options. |
+
+## Docs vs actual CLI discrepancies
+
+The [official HAOS common-tasks docs](https://www.home-assistant.io/common-tasks/os/)
+reference `ha backups list` and `ha host update`. **Neither exists in the
+installed CLI** (2026-08-13): backups are inspected with `ha backups info
+<slug>` (a slug is required) and `host` has no `update` subcommand. Trust the
+server CLI (`ha <cmd> --help`) over the docs.
+
+## Known issues on hass.windy.lan (observed 2026-08-13)
+
+From the health snapshot — follow-ups are optional, no action taken:
+
+- **2 add-ons in `state: error`**: `core_openthread_border_router`,
+  `a0d7b954_ssh` (duplicate Advanced SSH & Web Terminal install).
+- **`ha resolution info` issues**: `systemd_unit_failed`
+  (`systemd-vconsole-setup.service`), `no_current_backup`, 2×
+  `corrupt_repository` (store `d5369777`, `a0d7b954`).
+- `host info` reports `disk_life_time: 10` (disk lifetime warning threshold).
+
+## Pass criteria
+
+- Health snapshot completes; supervisor `healthy`/`supported: true`
+- Mutating modes refuse to run without `--yes` (incl. `--restore`, `--app`)
+- `--check-config` returns success
+- Update / rollback / restore / reboot confirmed only after explicit `--yes`
+- Update the **Verified** line on [hosts/hass.windy.lan.md](../hosts/hass.windy.lan.md)
@@ -0,0 +1,180 @@
+#!/usr/bin/env bash
+#
+# ha-maintenance.sh — Home Assistant maintenance via the `ha` supervisor CLI.
+#
+# Target : hass.windy.lan (HAOS / Core), SSH user `hassio`.
+# Access : non-interactive `sudo -n -i ha <cmd>`; the root login environment
+#          carries the supervisor API token (interactive login loads
+#          ~hassio/.zprofile which runs `exec sudo -i`). The whole remote
+#          procedure runs in ONE `sudo -n -i bash -s` login so the SSH & Web
+#          Terminal MOTD is printed only once, then stripped locally.
+#          See hosts/hass.windy.lan.md and runbooks/home-assistant-maintenance.md.
+#
+# Usage:
+#   ./ha-maintenance.sh                                read-only health snapshot
+#   ./ha-maintenance.sh --check-config                 validate core configuration
+#   ./ha-maintenance.sh --logs core|supervisor|host|apps:<slug> [N]
+#   ./ha-maintenance.sh --update --yes                 refresh + update core(--backup)/supervisor/os
+#   ./ha-maintenance.sh --restart-core [--safe-mode] --yes
+#   ./ha-maintenance.sh --rebuild-core --yes           rebuild core image (after options change)
+#   ./ha-maintenance.sh --rollback-os --yes            boot the previous OS slot (A/B rollback)
+#   ./ha-maintenance.sh --backup [NAME] --yes          full backup (optionally named)
+#   ./ha-maintenance.sh --restore <slug> --yes         restore a backup (DESTRUCTIVE)
+#   ./ha-maintenance.sh --app start|stop|restart|update <slug> --yes
+#
+# Environment: HA_HOST (default hass.windy.lan), HA_SSH_USER (default hassio).
+# Mutating actions always require --yes; this script never prompts.
+
+set -uo pipefail
+
+HOST="${HA_HOST:-hass.windy.lan}"
+SSH_USER="${HA_SSH_USER:-hassio}"
+
+MODE="health"
+TARGET="core"
+LINES=100
+SAFE_MODE=0
+CONFIRM=0
+ARG1=""
+ARG2=""
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --check-config) MODE="check-config" ;;
+    --logs)
+      MODE="logs"
+      if [ $# -gt 1 ]; then TARGET="$2"; shift; fi
+      if [ $# -gt 1 ] && [[ "$2" =~ ^[0-9]+$ ]]; then LINES="$2"; shift; fi
+      ;;
+    --update) MODE="update" ;;
+    --restart-core) MODE="restart-core" ;;
+    --safe-mode) SAFE_MODE=1 ;;
+    --rebuild-core) MODE="rebuild-core" ;;
+    --rollback-os) MODE="rollback-os" ;;
+    --reboot) MODE="reboot" ;;
+    --backup)
+      MODE="backup"
+      if [ $# -gt 1 ] && [[ "$2" != --* ]]; then ARG1="$2"; shift; fi
+      ;;
+    --restore)
+      MODE="restore"
+      if [ $# -gt 1 ] && [[ "$2" != --* ]]; then ARG1="$2"; shift
+      else echo "ha-maintenance: --restore needs a backup slug" >&2; exit 2; fi
+      ;;
+    --app)
+      MODE="app"
+      if [ $# -gt 1 ] && [[ "$2" != --* ]]; then ARG1="$2"; shift; fi
+      if [ $# -gt 1 ] && [[ "$2" != --* ]]; then ARG2="$2"; shift; fi
+      ;;
+    --yes) CONFIRM=1 ;;
+    *)
+      echo "ha-maintenance: unknown option '$1'" >&2
+      echo "usage: $0 [--check-config|--logs TARGET [N]|--update|--restart-core [--safe-mode]|--rebuild-core|--rollback-os|--reboot|--backup [NAME]|--restore SLUG|--app ACTION SLUG] [--yes]" >&2
+      exit 2
+      ;;
+  esac
+  shift
+done
+
+if [ "$MODE" = "app" ]; then
+  case "$ARG1" in
+    start|stop|restart|update) ;;
+    *) echo "ha-maintenance: --app action must be start|stop|restart|update (got '$ARG1')" >&2; exit 2 ;;
+  esac
+  if [ -z "$ARG2" ]; then
+    echo "ha-maintenance: --app needs an app slug, e.g. --app restart core_mosquitto" >&2
+    exit 2
+  fi
+fi
+
+case "$MODE" in
+  update|restart-core|rebuild-core|rollback-os|reboot|backup|restore|app)
+    if [ "$CONFIRM" -ne 1 ]; then
+      echo "ha-maintenance: refusing '$MODE' without --yes (mutating action)" >&2
+      exit 1
+    fi
+    ;;
+esac
+
+REMOTE_SCRIPT=$(cat <<'EOF'
+set -u
+ha() { command ha --no-progress "$@"; }
+
+case "$1" in
+  health)
+    printf '\n=== CORE ===\n'
+    ha core info
+    printf '\n=== SUPERVISOR ===\n'
+    ha supervisor info | grep -E '^(healthy|supported|version|version_latest|update_available|auto_update|timezone|logging):'
+    printf '\n=== ADDON STATES ===\n'
+    ha supervisor info | grep -E '^  (name|state):'
+    printf '\n=== OS ===\n'
+    ha os info
+    printf '\n=== HOST ===\n'
+    ha host info
+    printf '\n=== NETWORK ===\n'
+    ha network info | grep -E '^(address|ip_address|supervisor_internet|vlan|wifi):'
+    printf '\n=== AVAILABLE UPDATES ===\n'
+    ha available-updates
+    printf '\n=== JOBS ===\n'
+    ha jobs info
+    printf '\n=== RESOLUTION ===\n'
+    ha resolution info
+    printf '\n=== CORE STATS ===\n'
+    ha core stats
+    ;;
+  check-config)
+    ha core check
+    ;;
+  logs)
+    case "$2" in
+      core)       ha core logs -n "$3" ;;
+      supervisor) ha supervisor logs -n "$3" ;;
+      host)       ha host logs -n "$3" ;;
+      apps:*)     ha apps logs -n "$3" "${2#apps:}" ;;
+      *)          echo "unknown log target: $2" >&2; exit 2 ;;
+    esac
+    ;;
+  update)
+    ha refresh-updates
+    printf '\n=== UPDATE CORE (with backup) ===\n';  ha core update --backup
+    printf '\n=== UPDATE SUPERVISOR ===\n';          ha supervisor update
+    printf '\n=== UPDATE OS ===\n';                  ha os update
+    printf '\n=== PENDING AFTER ===\n';              ha available-updates
+    ;;
+  restart-core)
+    if [ "$4" = "1" ]; then ha core restart --safe-mode; else ha core restart; fi
+    ;;
+  rebuild-core)
+    ha core rebuild
+    ;;
+  rollback-os)
+    ha os boot-slot other
+    ;;
+  reboot)
+    ha host reboot
+    ;;
+  backup)
+    if [ -n "${2:-}" ]; then ha backups new --name "$2"; else ha backups new; fi
+    ;;
+  restore)
+    echo "WARNING: restoring backup $2"
+    ha backups restore "$2"
+    ;;
+  app)
+    ha apps "$2" "$3"
+    ;;
+esac
+EOF
+)
+
+if [ "$MODE" = "logs" ]; then
+  ARG1="$TARGET"
+  ARG2="$LINES"
+fi
+
+# One remote login shell (MOTD printed once), then drop the MOTD locally.
+printf '%s\n' "$REMOTE_SCRIPT" \
+  | ssh -o BatchMode=yes -o ConnectTimeout=10 "${SSH_USER}@${HOST}" \
+      'sudo -n -i bash -s' "$MODE" "$ARG1" "$ARG2" "$SAFE_MODE" 2>/dev/null \
+  | awk 'BEGIN{on=0} /^System is ready/{on=1; next} on'