vps/runbooks/home-assistant-maintenance.md

Runbook: Home Assistant maintenance (hass.windy.lan)

Target: hass.windy.lan (physical x88 Pro box, HAOS machine: green) Upstream: HAOS 18.1 / Core 2026.8.1 / Supervisor 2026.07.5 (verified 2026-08-14)

This runbook covers routine Home Assistant maintenance through the ha supervisor CLI. All commands are wrapped by a single script scripts/ha-maintenance.sh; the sections below document the exact commands it runs, for manual/agent use.

Purpose

Run routine Home Assistant maintenance on hass.windy.lan (health snapshot, config validation, log inspection, updates, and recovery) through the ha supervisor CLI.

Scope

Applies to hass.windy.lan only (HAOS, machine: green). Covers both read-only checks and gated mutating operations; the "Command families intentionally NOT scripted" table below lists what is deliberately out of scope.

Ownership

• Owner: personal ops (Windy)
• Last reviewed: 2026-08-17
• Related systems: hass.windy.lan (HAOS, machine: green)

Safety

• Prefer read-only checks first; the health snapshot mutates nothing.
• Every mutating mode (update / restart / rebuild / rollback / reboot / backup / restore / add-on lifecycle) refuses to run without --yes.
• --restore overwrites the current installation; --rollback-os, --reboot, and --rebuild-core are disruptive. Run them only from a planned recovery with the backup verified.
• Never commit SUPERVISOR_TOKEN or a long-lived HA_TOKEN; read entity state via Supervisor (SUPERVISOR_TOKEN after sudo -n -i).
• The --restart-core wrapper exits 1 silently on ssh failure — treat an empty/exit-1 result as failure and confirm with ha core info.
• If live state conflicts with a documented expectation, STOP and report; do not improvise command families outside this script.

Access pattern

ha authenticates to the Supervisor with SUPERVISOR_TOKEN. Interactive SSH login works because ~hassio/.zprofile runs exec sudo -i; the root login environment carries the token. Non-interactive use must be:

ssh -o BatchMode=yes hassio@hass.windy.lan 'sudo -n -i ha <cmd>'

Running ha as hassio directly (or sudo -n without -i) returns unauthorized: missing or invalid API token.

MOTD: every sudo -n -i login prints the SSH & Web Terminal MOTD banner. The script runs its whole procedure in one remote login (sudo -n -i bash -s) so the banner appears once, then strips it with awk up to the System is ready! Use browser or app to configure. line.

Restart wrapper (verified 2026-08-14, W1N-107): ./ha-maintenance.sh --restart-core --yes exited 1 with no output in <1s and did not restart Core. The wrapper pipes a remote script through ssh … 2>/dev/null | awk …; with set -uo pipefail, an ssh failure is silent and the pipeline returns empty/exit 1 before any remote command runs. That is not a MOTD-strip artifact after a successful restart. The working restart was ssh -o BatchMode=yes hassio@hass.windy.lan 'sudo -n -i ha core restart' (~131s, Command completed successfully.). Treat empty/exit 1 as failure; confirm with elapsed time and ha core info.

Script usage

cd runbooks/scripts

./ha-maintenance.sh                                # read-only health snapshot
./ha-maintenance.sh --check-config                 # validate core configuration
./ha-maintenance.sh --logs core 2500               # tail core logs (use 2500 after a restart)
./ha-maintenance.sh --logs supervisor              # tail supervisor logs (default 100)
./ha-maintenance.sh --logs host 50                 # tail host journald logs
./ha-maintenance.sh --logs apps:<slug>             # tail an add-on log

# Mutating — refuse to run without --yes:
./ha-maintenance.sh --update --yes                 # refresh + update core(--backup)/supervisor/os
./ha-maintenance.sh --restart-core --yes           # restart Core
./ha-maintenance.sh --restart-core --safe-mode --yes   # restart Core in safe mode
./ha-maintenance.sh --rebuild-core --yes           # rebuild Core image (after options change)
./ha-maintenance.sh --rollback-os --yes            # boot previous OS slot (A/B rollback)
./ha-maintenance.sh --reboot --yes                 # reboot the HAOS host
./ha-maintenance.sh --backup [NAME] --yes          # full backup (optionally named)
./ha-maintenance.sh --restore <slug> --yes         # restore a backup (DESTRUCTIVE)
./ha-maintenance.sh --app restart core_mosquitto --yes   # add-on lifecycle

• --app action is one of start|stop|restart|update; needs an add-on slug.
• HA_HOST / HA_SSH_USER override the defaults (hass.windy.lan / hassio).
• --restore overwrites the current installation — run only from a planned recovery, with the backup verified.

Command reference (verified 2026-08-14)

All verified against the live host. MOTD prepends each command's output; strip with the awk pattern above or read the last block.

Routine / read-only

Purpose	Command
General overview	ha info
Core version/status	ha core info (this CLI build has no state: field; success is a normal info dump)
Core config validation	ha core check
Core stats	ha core stats
Supervisor status	ha supervisor info (incl. add-on list)
Supervisor stats	ha supervisor stats
OS status	ha os info (boot slots A/B)
Host status	ha host info (disk free/total, kernel)
Network	ha network info (supervisor_internet)
Hardware	ha hardware info
Pending updates	ha available-updates
Reload stores/versions	ha refresh-updates
Job manager	ha jobs info
Resolution center	ha resolution info
Core logs	ha core logs -n 100 (-f follow, -b boot id). Default 100 misses setup; use -n 2500 after a custom-component restart. /config/home-assistant.log may be missing — ha core logs is the source of truth.
Supervisor logs	ha supervisor logs -n 100
Host journald logs	ha host logs -n 100
Add-on logs	ha apps logs <slug> -n 100
Add-on list	ha supervisor info → addons: (started/stopped/error)
Security integrity	ha security integrity

Mutating (require --yes)

Purpose	Command
Update core (with partial backup)	ha core update --backup
Update supervisor	ha supervisor update
Update OS	ha os update
Update add-on	ha apps update <slug>
Restart core	ha core restart / ha core restart --safe-mode
Rebuild core	ha core rebuild
OS rollback	ha os boot-slot other
Reboot host	ha host reboot
Full backup	ha backups new [--name NAME]
Restore backup	ha backups restore <slug>
Add-on start/stop/restart	ha apps start|stop|restart <slug>

Procedure

1. Health snapshot (read-only)

./ha-maintenance.sh

Review: supervisor healthy: true/supported: true; core/OS update_available; add-on states (any state: error?); resolution info issues; disk free.

2. Validate config after any configuration.yaml change

./ha-maintenance.sh --check-config

Expect Command completed successfully. before a Core restart.

ha core check / a YAML reload is not enough after copying Python custom-component files — restart Core.

3. Inspect logs

./ha-maintenance.sh --logs core 2500   # after a Core restart / custom-component copy
./ha-maintenance.sh --logs supervisor
./ha-maintenance.sh --logs apps:core_mosquitto

Default --logs core (100 lines) is too short to catch coordinator pickle / setup errors. /config/home-assistant.log may be absent while ha core logs still has history.

4. Apply updates (mutating)

./ha-maintenance.sh --update --yes

Runs refresh-updates → core update --backup (partial backup first) → supervisor update → os update, then re-prints pending updates. Prefer the web UI (Settings → System → Updates) for a human-supervised pass.

5. Recovery operations (mutating, only when needed)

./ha-maintenance.sh --restart-core --safe-mode --yes   # start Core without custom integrations
./ha-maintenance.sh --rollback-os --yes                # OS update broke boot? go back one slot
./ha-maintenance.sh --restore <slug> --yes             # full restore; overwrites current install

OS update policy: HAOS uses two boot slots (A/B); ha os info shows which slot booted. After a bad OS update, ha os boot-slot other boots the previous slot.

6. Backup before major changes

./ha-maintenance.sh --backup pre-migration --yes       # named backup

7. Install or update a custom component (manual zip)

Home Assistant loads custom integrations from /config/custom_components/<domain>/ (on this HAOS host /config ≡ /homeassistant). Official lookup: <config>/custom_components/<domain> then built-in homeassistant/components/<domain> (Integration file structure). A folder named after the domain, with at least manifest.json and __init__.py, is enough. Restart Core after copying — ha core check and a YAML reload do not pick up new Python packages.

This host's live trees are file copies, not git clones. Do not git pull inside custom_components/.

Official plugin paths (CSG)

windyboy/china_southern_power_grid_stat README: HACS or 手动下载安装.

This host uses the zip path. Do not HACS-update this integration here. HACS still tracks upstream CubicPill/china_southern_power_grid_stat v1.2.0 and would overwrite the fork. Releases have no uploaded zip assets — use GitHub Source code (zip) / zipball of the tag.

Worked SSH example (tag, backup, rsync, __pycache__, restart): hosts/hass.windy.lan.md § Manual custom-component install.

Procedure

1. Backup the live tree off custom_components/. HA scans every directory under custom_components/ whose manifest.json domain matches. A *.bak-* folder next to the live tree makes Core import the backup (No module named '...bak-YYYYMMDD-...', W1N-106). CSG backups: /homeassistant/.csg-backups/.
2. Copy only the inner custom_components/<domain>/ tree, not the repo root and not an extra nested folder.
3. Wipe __pycache__ as root. rsync --delete as hassio cannot unlink Core-owned .pyc (permission denied, exit 23); stale cpython-314 bytecode can keep the old coordinator in memory until restart. Then restart:

ssh -o BatchMode=yes hassio@hass.windy.lan \
  'sudo -n -i rm -rf /homeassistant/custom_components/<domain>/__pycache__ \
     /homeassistant/custom_components/<domain>/*/__pycache__ &&
   sudo -n -i ha core restart'

4. Wait 1–2 min, then ha core info (this CLI build has no state: field; success is a normal info dump). Confirm manifest.json version matches the tag.
5. Read enough Core logs. Default ha core logs is too short to catch setup. Use -n 2500 (or --logs core 2500) and look for Setting up <domain> plus the first coordinator errors.
6. First poll can time out. If last-month sensors have numbers but this-month stay unknown/unavailable, reload the config entry (UI: integration → Reload). Supervisor:

# entry id from .storage/core.config_entries (CSG: 01KGCQDSZCF523A9X6SV3BZ1B9)
ssh -o BatchMode=yes hassio@hass.windy.lan \
  'sudo -n -i python3 -c "
import os, urllib.request
req = urllib.request.Request(
    \"http://supervisor/core/api/config/config_entries/entry/<ENTRY_ID>/reload\",
    method=\"POST\",
    headers={\"Authorization\": \"Bearer \" + os.environ[\"SUPERVISOR_TOKEN\"]},
)
print(urllib.request.urlopen(req, timeout=60).status)
"'

7. Do not edit the dashboard or templates/csg_sensors.yaml for an install. Entity IDs did not change across v1.3.0/v1.3.1/v1.3.2. Template | float(0) turns native unavailable into fake zeros; that is a follow-up, not part of the zip install.

Verify (CSG, after v1.3.2 / W1N-118)

Check	Expect
manifest.json version	1.3.2
ha core logs after this restart	Setting up china_southern_power_grid_stat; no cannot pickle 'mappingproxy'
Config entry	state: loaded
sensor.0800041935246530_balance	numeric (may be 0.0)
sensor.0800041935246530_this_month_total_usage	numeric after reload if first poll timed out
Native *_total_cost / current_ladder	may stay unknown (CSG marketing calendar SQL error); dashboard uses W1N-114 csg_* ladder/cost templates

monetary + total_increasing warnings on this-month/year cost sensors are a remaining plugin issue, not an install failure.

There is no long-lived HA_TOKEN in the agent environment. Read entity states via Supervisor (SUPERVISOR_TOKEN after sudo -n -i) at http://supervisor/core/api/states/<entity_id>.

Command families intentionally NOT scripted

These exist in ha but are either rare, dangerous, or better done in the web UI; documented here so nothing is a surprise. Use ha <family> --help on the host for exact syntax.

Family	Notes
ha audio	Audio device management; peripheral.
ha authentication	auth list/reset/cache; user password ops — do in web UI. auth list is local-terminal only.
ha cli	Internal CLI backend info/update; self-maintained.
ha dns	Internal DNS server; only relevant if Supervisor DNS add-on in use.
ha docker	Host Docker backend info/options/registries; HAOS-managed.
ha mounts	Network storage (NFS/CIFS) mounts — configure in Settings → System → Storage.
ha multicast / ha observer	Internal services; self-maintained.
ha network scan/update/vlan	WiFi AP scan & interface config — prefer web UI networking.
ha host disks/options/shutdown/reload	Disk ops / host options; shutdown is equivalent to --reboot but off.
ha os datadisk list/move/wipe	Data-disk migration; wipe is local-terminal only and erases all data.
ha os import	Import config from USB stick.
ha os boards / os config	Board / OS settings.
ha core options / supervisor options	Core/OS config options (e.g. --duplicate-log-file); changes need ha core rebuild + restart.
ha backups freeze/thaw/remove/options	Freeze/thaw for external backup tools; removal is destructive.
ha jobs options/reset	Job-manager tuning.
ha resolution check/healthcheck/issue/suggestion	Resolution center management; healthcheck runs fixups.
ha store add/delete/repair	Repository management — add repos in web UI app store.
ha security info/options	Security backend options.

Docs vs actual CLI discrepancies

The official HAOS common-tasks docs also mention ha host update, which does not exist in this CLI (2026-08-14). Docs' ha backups list is not a subcommand either: ha backups --help lists freeze/info/new/options/reload/remove/restore/thaw; extra positional args (list, nonsense, ...) are ignored and the default list still prints with exit 0. The list command is plain ha backups. Per-backup: ha backups info <slug> (slug required). Trust the server CLI (ha <cmd> --help) over the docs.

This CLI's ha core info also has no state: field (verified 2026-08-14). Wait for a successful info dump after restart, not a state: running line.

Known issues on hass.windy.lan (2026-08-14)

2026-08-13 snapshot items were resolved same day (W1N-70/71/72/73/74/75/76): OTBR and the duplicate SSH add-on uninstalled, resolution-center empty, full backup pre-maintenance-20260813 (slug 411a4ba5). Remaining:

• Bluetooth hci0 instability (RTL8821CS): bluetooth_auto_recovery power-reset times out every ~2 min; kernel hci0 hardware error. No BLE entities exist, so no user impact. HAOS image ships x88-bt-hci-recovery workaround units.
• host info reports disk_life_time: 10 (boot eMMC ~10% life left) — monitor on each snapshot; plan disk replacement / data-disk migration.
• Home PPPoE IPv4 to CSG is blackholed (curl -4 to 218.19.148.218:443 times out). end1 IPv6 works (curl -6 https://95598.csg.cn → HTTP 200). Entry ip_family: ipv4 still matches the stored option; first post-restart poll can still time out — reload the config entry rather than reinstalling.
• WSL HTTP proxy: LAN hass.windy.lan:8123 through Mihomo returns empty 502. Bypass proxy or add .windy.lan to NO_PROXY before debugging UI/API from the workstation (hosts/hass.windy.lan.md § HTTP proxy gotcha).
• No long-lived HA token in the agent environment. Read entity states via Supervisor (SUPERVISOR_TOKEN after sudo -n -i) at http://supervisor/core/api/states/..., not a committed HA_TOKEN.

Pass criteria

• Health snapshot completes; supervisor healthy/supported: true
• Mutating modes refuse to run without --yes (incl. --restore, --app)
• --check-config returns success
• Update / rollback / restore / reboot confirmed only after explicit --yes
• Custom-component zip install: live manifest.json version matches the tag; backups not under custom_components/; Core restarted; logs show Setting up <domain> without import / pickle errors
• Update the Verified line on hosts/hass.windy.lan.md