vps/runbooks/home-assistant-maintenance.md

# Runbook: Home Assistant maintenance (hass.windy.lan)

Target: [hass.windy.lan](../hosts/hass.windy.lan.md) (physical x88 Pro box, HAOS `machine: green`)
Upstream: HAOS 18.1 / Core 2026.8.1 / Supervisor 2026.07.5 (verified 2026-08-14)

This runbook covers routine Home Assistant maintenance through the **`ha`
supervisor CLI**. All commands are wrapped by a single script
[`scripts/ha-maintenance.sh`](scripts/ha-maintenance.sh); the sections below
document the exact commands it runs, for manual/agent use.

## Purpose

Run routine Home Assistant maintenance on `hass.windy.lan` (health snapshot,
config validation, log inspection, updates, and recovery) through the `ha`
supervisor CLI.

## Scope

Applies to `hass.windy.lan` only (HAOS, `machine: green`). Covers both
read-only checks and gated mutating operations; the "Command families
intentionally NOT scripted" table below lists what is deliberately out of
scope.

## Ownership

- Owner: personal ops (Windy)
- Last reviewed: 2026-08-17
- Related systems: hass.windy.lan (HAOS, `machine: green`)

## Safety

- Prefer read-only checks first; the health snapshot mutates nothing.
- Every mutating mode (update / restart / rebuild / rollback / reboot /
  backup / restore / add-on lifecycle) refuses to run without `--yes`.
- `--restore` overwrites the current installation; `--rollback-os`,
  `--reboot`, and `--rebuild-core` are disruptive. Run them only from a
  planned recovery with the backup verified.
- Never commit `SUPERVISOR_TOKEN` or a long-lived `HA_TOKEN`; read entity
  state via Supervisor (`SUPERVISOR_TOKEN` after `sudo -n -i`).
- The `--restart-core` wrapper exits 1 silently on ssh failure — treat an
  empty/exit-1 result as failure and confirm with `ha core info`.
- If live state conflicts with a documented expectation, `STOP` and report;
  do not improvise command families outside this script.

## Access pattern

`ha` authenticates to the Supervisor with `SUPERVISOR_TOKEN`. Interactive SSH
login works because `~hassio/.zprofile` runs `exec sudo -i`; the root login
environment carries the token. Non-interactive use must be:

```bash
ssh -o BatchMode=yes hassio@hass.windy.lan 'sudo -n -i ha <cmd>'
```

Running `ha` as `hassio` directly (or `sudo -n` without `-i`) returns
`unauthorized: missing or invalid API token`.

**MOTD:** every `sudo -n -i` login prints the SSH & Web Terminal MOTD banner.
The script runs its whole procedure in one remote login (`sudo -n -i bash -s`)
so the banner appears once, then strips it with `awk` up to the
`System is ready! Use browser or app to configure.` line.

**Restart wrapper (verified 2026-08-14, W1N-107):**
`./ha-maintenance.sh --restart-core --yes` exited 1 with no output in <1s
and **did not restart Core**. The wrapper pipes a remote script through
`ssh … 2>/dev/null | awk …`; with `set -uo pipefail`, an ssh failure is
silent and the pipeline returns empty/exit 1 **before any remote command
runs**. That is not a MOTD-strip artifact after a successful restart.
The working restart was
`ssh -o BatchMode=yes hassio@hass.windy.lan 'sudo -n -i ha core restart'`
(~131s, `Command completed successfully.`). Treat empty/exit 1 as
failure; confirm with elapsed time and `ha core info`.

## Script usage

```bash
cd runbooks/scripts

./ha-maintenance.sh                                # read-only health snapshot
./ha-maintenance.sh --check-config                 # validate core configuration
./ha-maintenance.sh --logs core 2500               # tail core logs (use 2500 after a restart)
./ha-maintenance.sh --logs supervisor              # tail supervisor logs (default 100)
./ha-maintenance.sh --logs host 50                 # tail host journald logs
./ha-maintenance.sh --logs apps:<slug>             # tail an add-on log

# Mutating — refuse to run without --yes:
./ha-maintenance.sh --update --yes                 # refresh + update core(--backup)/supervisor/os
./ha-maintenance.sh --restart-core --yes           # restart Core
./ha-maintenance.sh --restart-core --safe-mode --yes   # restart Core in safe mode
./ha-maintenance.sh --rebuild-core --yes           # rebuild Core image (after options change)
./ha-maintenance.sh --rollback-os --yes            # boot previous OS slot (A/B rollback)
./ha-maintenance.sh --reboot --yes                 # reboot the HAOS host
./ha-maintenance.sh --backup [NAME] --yes          # full backup (optionally named)
./ha-maintenance.sh --restore <slug> --yes         # restore a backup (DESTRUCTIVE)
./ha-maintenance.sh --app restart core_mosquitto --yes   # add-on lifecycle
```

- `--app` action is one of `start|stop|restart|update`; needs an add-on slug.
- `HA_HOST` / `HA_SSH_USER` override the defaults (`hass.windy.lan` / `hassio`).
- `--restore` overwrites the current installation — run only from a planned
  recovery, with the backup verified.

## Command reference (verified 2026-08-14)

All verified against the live host. MOTD prepends each command's output; strip
with the `awk` pattern above or read the last block.

### Routine / read-only

| Purpose | Command |
|---|---|
| General overview | `ha info` |
| Core version/status | `ha core info` (this CLI build has no `state:` field; success is a normal info dump) |
| Core config validation | `ha core check` |
| Core stats | `ha core stats` |
| Supervisor status | `ha supervisor info` (incl. add-on list) |
| Supervisor stats | `ha supervisor stats` |
| OS status | `ha os info` (boot slots A/B) |
| Host status | `ha host info` (disk free/total, kernel) |
| Network | `ha network info` (`supervisor_internet`) |
| Hardware | `ha hardware info` |
| Pending updates | `ha available-updates` |
| Reload stores/versions | `ha refresh-updates` |
| Job manager | `ha jobs info` |
| Resolution center | `ha resolution info` |
| Core logs | `ha core logs -n 100` (`-f` follow, `-b` boot id). Default 100 misses setup; use `-n 2500` after a custom-component restart. `/config/home-assistant.log` may be missing — `ha core logs` is the source of truth. |
| Supervisor logs | `ha supervisor logs -n 100` |
| Host journald logs | `ha host logs -n 100` |
| Add-on logs | `ha apps logs <slug> -n 100` |
| Add-on list | `ha supervisor info` → `addons:` (started/stopped/error) |
| Security integrity | `ha security integrity` |

### Mutating (require --yes)

| Purpose | Command |
|---|---|
| Update core (with partial backup) | `ha core update --backup` |
| Update supervisor | `ha supervisor update` |
| Update OS | `ha os update` |
| Update add-on | `ha apps update <slug>` |
| Restart core | `ha core restart` / `ha core restart --safe-mode` |
| Rebuild core | `ha core rebuild` |
| OS rollback | `ha os boot-slot other` |
| Reboot host | `ha host reboot` |
| Full backup | `ha backups new [--name NAME]` |
| Restore backup | `ha backups restore <slug>` |
| Add-on start/stop/restart | `ha apps start\|stop\|restart <slug>` |

## Procedure

### 1. Health snapshot (read-only)

```bash
./ha-maintenance.sh
```

Review: supervisor `healthy: true`/`supported: true`; core/OS `update_available`;
add-on states (any `state: error`?); `resolution info` issues; disk free.

### 2. Validate config after any `configuration.yaml` change

```bash
./ha-maintenance.sh --check-config
```

Expect `Command completed successfully.` before a Core restart.

`ha core check` / a YAML reload is **not** enough after copying Python
custom-component files — restart Core.

### 3. Inspect logs

```bash
./ha-maintenance.sh --logs core 2500   # after a Core restart / custom-component copy
./ha-maintenance.sh --logs supervisor
./ha-maintenance.sh --logs apps:core_mosquitto
```

Default `--logs core` (100 lines) is too short to catch coordinator pickle /
setup errors. `/config/home-assistant.log` may be absent while
`ha core logs` still has history.

### 4. Apply updates (mutating)

```bash
./ha-maintenance.sh --update --yes
```

Runs `refresh-updates` → `core update --backup` (partial backup first) →
`supervisor update` → `os update`, then re-prints pending updates. Prefer the
web UI (**Settings → System → Updates**) for a human-supervised pass.

### 5. Recovery operations (mutating, only when needed)

```bash
./ha-maintenance.sh --restart-core --safe-mode --yes   # start Core without custom integrations
./ha-maintenance.sh --rollback-os --yes                # OS update broke boot? go back one slot
./ha-maintenance.sh --restore <slug> --yes             # full restore; overwrites current install
```

OS update policy: HAOS uses two boot slots (A/B); `ha os info` shows which slot
booted. After a bad OS update, `ha os boot-slot other` boots the previous slot.

### 6. Backup before major changes

```bash
./ha-maintenance.sh --backup pre-migration --yes       # named backup
```

### 7. Install or update a custom component (manual zip)

Home Assistant loads custom integrations from
`/config/custom_components/<domain>/` (on this HAOS host `/config` ≡
`/homeassistant`). Official lookup:
`<config>/custom_components/<domain>` then built-in
`homeassistant/components/<domain>`
([Integration file structure](https://developers.home-assistant.io/docs/creating_integration_file_structure)).
A folder named after the domain, with at least `manifest.json` and
`__init__.py`, is enough. **Restart Core** after copying — `ha core check`
and a YAML reload do not pick up new Python packages.

This host's live trees are **file copies**, not git clones. Do not
`git pull` inside `custom_components/`.

#### Official plugin paths (CSG)

[windyboy/china_southern_power_grid_stat README](https://github.com/windyboy/china_southern_power_grid_stat):
[HACS](https://hacs.xyz/) **or**
[手动下载安装](https://github.com/windyboy/china_southern_power_grid_stat/releases).

This host uses the zip path. **Do not HACS-update this integration here.**
HACS still tracks upstream `CubicPill/china_southern_power_grid_stat`
`v1.2.0` and would overwrite the fork. Releases have no uploaded zip
assets — use GitHub **Source code (zip)** / zipball of the tag.

Worked SSH example (tag, backup, `rsync`, `__pycache__`, restart):
[hosts/hass.windy.lan.md](../hosts/hass.windy.lan.md) § Manual
custom-component install.

#### Procedure

1. **Backup the live tree off `custom_components/`.** HA scans every
   directory under `custom_components/` whose `manifest.json` `domain`
   matches. A `*.bak-*` folder next to the live tree makes Core import
   the backup (`No module named '...bak-YYYYMMDD-...'`, W1N-106). CSG
   backups: `/homeassistant/.csg-backups/`.
2. **Copy only the inner `custom_components/<domain>/` tree**, not the
   repo root and not an extra nested folder.
3. **Wipe `__pycache__` as root.** `rsync --delete` as `hassio` cannot
   unlink Core-owned `.pyc` (permission denied, exit 23); stale
   `cpython-314` bytecode can keep the old coordinator in memory until
   restart. Then restart:

```bash
ssh -o BatchMode=yes hassio@hass.windy.lan \
  'sudo -n -i rm -rf /homeassistant/custom_components/<domain>/__pycache__ \
     /homeassistant/custom_components/<domain>/*/__pycache__ &&
   sudo -n -i ha core restart'
```

4. **Wait 1–2 min**, then `ha core info` (this CLI build has no `state:`
   field; success is a normal info dump). Confirm `manifest.json`
   `version` matches the tag.
5. **Read enough Core logs.** Default `ha core logs` is too short to
   catch setup. Use `-n 2500` (or `--logs core 2500`) and look for
   `Setting up <domain>` plus the first coordinator errors.
6. **First poll can time out.** If last-month sensors have numbers but
   this-month stay `unknown`/`unavailable`, reload the config entry
   (UI: integration → Reload). Supervisor:

```bash
# entry id from .storage/core.config_entries (CSG: 01KGCQDSZCF523A9X6SV3BZ1B9)
ssh -o BatchMode=yes hassio@hass.windy.lan \
  'sudo -n -i python3 -c "
import os, urllib.request
req = urllib.request.Request(
    \"http://supervisor/core/api/config/config_entries/entry/<ENTRY_ID>/reload\",
    method=\"POST\",
    headers={\"Authorization\": \"Bearer \" + os.environ[\"SUPERVISOR_TOKEN\"]},
)
print(urllib.request.urlopen(req, timeout=60).status)
"'
```

7. **Do not edit the dashboard or `templates/csg_sensors.yaml` for an
   install.** Entity IDs did not change across v1.3.0/v1.3.1/v1.3.2.
   Template `| float(0)` turns native `unavailable` into fake zeros; that
   is a follow-up, not part of the zip install.

#### Verify (CSG, after v1.3.2 / W1N-118)

| Check | Expect |
|---|---|
| `manifest.json` `version` | `1.3.2` |
| `ha core logs` after this restart | `Setting up china_southern_power_grid_stat`; **no** `cannot pickle 'mappingproxy'` |
| Config entry | `state: loaded` |
| `sensor.0800041935246530_balance` | numeric (may be `0.0`) |
| `sensor.0800041935246530_this_month_total_usage` | numeric after reload if first poll timed out |
| Native `*_total_cost` / `current_ladder` | may stay `unknown` (CSG marketing calendar SQL error); dashboard uses W1N-114 `csg_*` ladder/cost templates |

`monetary` + `total_increasing` warnings on this-month/year cost sensors
are a remaining plugin issue, not an install failure.

There is no long-lived `HA_TOKEN` in the agent environment. Read entity
states via Supervisor (`SUPERVISOR_TOKEN` after `sudo -n -i`) at
`http://supervisor/core/api/states/<entity_id>`.

## Command families intentionally NOT scripted

These exist in `ha` but are either rare, dangerous, or better done in the web
UI; documented here so nothing is a surprise. Use `ha <family> --help` on the
host for exact syntax.

| Family | Notes |
|---|---|
| `ha audio` | Audio device management; peripheral. |
| `ha authentication` | `auth list/reset/cache`; user password ops — do in web UI. `auth list` is local-terminal only. |
| `ha cli` | Internal CLI backend info/update; self-maintained. |
| `ha dns` | Internal DNS server; only relevant if Supervisor DNS add-on in use. |
| `ha docker` | Host Docker backend info/options/registries; HAOS-managed. |
| `ha mounts` | Network storage (NFS/CIFS) mounts — configure in **Settings → System → Storage**. |
| `ha multicast` / `ha observer` | Internal services; self-maintained. |
| `ha network scan/update/vlan` | WiFi AP scan & interface config — prefer web UI networking. |
| `ha host disks/options/shutdown/reload` | Disk ops / host options; `shutdown` is equivalent to `--reboot` but off. |
| `ha os datadisk list/move/wipe` | Data-disk migration; `wipe` is **local-terminal only** and erases all data. |
| `ha os import` | Import config from USB stick. |
| `ha os boards` / `os config` | Board / OS settings. |
| `ha core options` / `supervisor options` | Core/OS config options (e.g. `--duplicate-log-file`); changes need `ha core rebuild` + restart. |
| `ha backups freeze/thaw/remove/options` | Freeze/thaw for external backup tools; removal is destructive. |
| `ha jobs options/reset` | Job-manager tuning. |
| `ha resolution check/healthcheck/issue/suggestion` | Resolution center management; `healthcheck` runs fixups. |
| `ha store add/delete/repair` | Repository management — add repos in web UI app store. |
| `ha security info/options` | Security backend options. |

## Docs vs actual CLI discrepancies

The [official HAOS common-tasks docs](https://www.home-assistant.io/common-tasks/os/)
also mention `ha host update`, which **does not exist** in this CLI
(2026-08-14). Docs' `ha backups list` is not a subcommand either:
`ha backups --help` lists freeze/info/new/options/reload/remove/restore/thaw;
extra positional args (`list`, `nonsense`, ...) are ignored and the default
list still prints with exit 0. The list command is plain `ha backups`.
Per-backup: `ha backups info <slug>` (slug required). Trust the server CLI
(`ha <cmd> --help`) over the docs.

This CLI's `ha core info` also has no `state:` field (verified 2026-08-14).
Wait for a successful info dump after restart, not a `state: running` line.

## Known issues on hass.windy.lan (2026-08-14)

2026-08-13 snapshot items were resolved same day (W1N-70/71/72/73/74/75/76):
OTBR and the duplicate SSH add-on uninstalled, resolution-center empty,
full backup `pre-maintenance-20260813` (slug `411a4ba5`). Remaining:

- **Bluetooth hci0 instability (RTL8821CS)**: `bluetooth_auto_recovery`
  power-reset times out every ~2 min; kernel `hci0 hardware error`. No BLE
  entities exist, so no user impact. HAOS image ships `x88-bt-hci-recovery`
  workaround units.
- `host info` reports `disk_life_time: 10` (boot eMMC ~10% life left) —
  monitor on each snapshot; plan disk replacement / data-disk migration.
- **Home PPPoE IPv4 to CSG is blackholed** (`curl -4` to
  `218.19.148.218:443` times out). `end0` IPv6 works (`curl -6
  https://95598.csg.cn` → HTTP 200). Entry `ip_family: ipv4` still
  matches the stored option; first post-restart poll can still time out
  — reload the config entry rather than reinstalling.
- **WSL HTTP proxy**: LAN `hass.windy.lan:8123` through Mihomo returns
  empty `502`. Bypass proxy or add `.windy.lan` to `NO_PROXY` before
  debugging UI/API from the workstation
  ([hosts/hass.windy.lan.md](../hosts/hass.windy.lan.md) § HTTP proxy
  gotcha).
- **No long-lived HA token in the agent environment.** Read entity
  states via Supervisor (`SUPERVISOR_TOKEN` after `sudo -n -i`) at
  `http://supervisor/core/api/states/...`, not a committed `HA_TOKEN`.

## Pass criteria

- Health snapshot completes; supervisor `healthy`/`supported: true`
- Mutating modes refuse to run without `--yes` (incl. `--restore`, `--app`)
- `--check-config` returns success
- Update / rollback / restore / reboot confirmed only after explicit `--yes`
- Custom-component zip install: live `manifest.json` version matches the
  tag; backups not under `custom_components/`; Core restarted; logs show
  `Setting up <domain>` without import / pickle errors
- Update the **Verified** line on [hosts/hass.windy.lan.md](../hosts/hass.windy.lan.md)