vps/runbooks/host-disk-cleanup.md

# Runbook: Host disk cleanup (unbounded container logs / apt cache / docker artifacts)

## Purpose

Reclaim space on a root filesystem that is filling up (≥70% used) on a Docker
Compose host, by fixing unbounded container log growth at the source, clearing
apt/journal caches, and removing unused Docker images/volumes. Success: root
usage drops to a safe band (≤55% used, or per acceptance in the tracking issue)
and log growth stays bounded afterwards.

## Scope

- 适用环境: production single-root-fs hosts running Docker Compose stacks
  (first application: `hk2.chans.xyz`; reusable for `mx2.windy.me` / `us2.wsvc.info`
  which run the same unbounded-`json.log` pattern).
- 适用对象: root filesystem usage; container stdout/stderr log files
  (`/var/lib/docker/containers/*/*-json.log`); `/var/cache/apt`; systemd journal;
  unused Docker images / anonymous volumes / build cache.
- 不适用情形: hosts without systemd-journald or without Docker; LAN/HAOS hosts
  (use their own runbooks); cases needing disk *growth* (provider resize) rather
  than cleanup; anything touching service data volumes or `/opt/*` configs
  (STOP and use the service-specific runbook instead).

## Ownership

- Owner: windy (operator) + agent executing per approval
- Last reviewed: 2026-09-02
- Related systems: hk2.chans.xyz (PowerDNS auth / AdGuard Home / Traefik / RustDesk compose stacks)

## Preconditions

- SSH access to the target host with **passwordless sudo** (`sudo -n true` must succeed).
- A recorded `df -h` baseline and `docker system df` baseline.
- **Explicit user approval** for every service touch listed in Approval gates
  (recorded in the tracking issue, e.g. Plane `vps` VPS-81).
- No open incident on the target host.
- Container log growth root cause identified in Diagnose before mutating.

## Inputs

| Input | Source | Required | Validation |
|---|---:|---|
| Target host | inventory/hosts.md | yes | SSH login + `uname -r` |
| df/docker baseline | live read-only probe | yes | recorded before first mutation |
| Approved service touches | user confirmation in tracking issue | yes | issue comment states approval |
| Image keep-list (rollback pins) | operator decision in issue | yes | review `docker image ls` before rmi |
| Backup of any config edited | local copy with timestamp | yes | exists before edit |

## Safety

### Non-negotiable rules

- Prefer read-only diagnosis before mutation (never mutate on an unmeasured disk).
- Never use `rm` on a live container log — use `truncate -s 0` (keeps the fd valid).
- Never run `docker image prune -a` when a keep-list is intended — no keep-list
  exists; delete explicitly with `docker rmi`.
- Never run `docker volume prune -a` — plain `docker volume prune` (no `-a`)
  removes only unused anonymous volumes; named/in-use volumes stay.
- After every mutation, verify the expected state (`df -h`, container status).
- Destructive actions require explicit approval (Approval gates).

### Stop conditions

- Live state conflicts with this runbook's preconditions or expectations (e.g.
  root usage differs wildly from baseline, or a container is unhealthy).
- Missing approval, missing backup, or missing rollback ability.
- A verification step fails with no documented next step.
- Any step would touch a volume/container/mount that is not on the approved list.

### Approval gates

| Action | Risk | Explicit approval | Approval record |
|---|---:|---|---|
| `docker restart <chatty container>` | low (sec-level blip of that service only) | yes | tracking issue (VPS-81 T1) |
| `systemctl restart systemd-journald` | low (sec-level, no state loss) | yes | tracking issue (VPS-81 T2) |
| `apt-get clean` | low (re-downloadable) | no | — |
| `journalctl --vacuum-*` / journald drop-in | low | no (restart above is gated) | — |
| `docker rmi` of unused images | medium (rollback pin removed unless kept) | yes (keep-list) | tracking issue (VPS-81 T3) |
| `docker volume prune` | medium (data in anonymous volumes lost) | yes | tracking issue (VPS-81 T4) |
| `docker builder prune` | low | no | — |

## Procedure

### Step 1 — Diagnose

**Action**

Read-only: `df -h`, `df -i`, `sudo du -x -h --max-depth=1 /`, `docker system df`,
and locate oversized container logs:
`sudo ls -la /var/lib/docker/containers/*/*-json.log`. Map a big log to its
container (`docker inspect -f '{{.Name}} {{.LogPath}}' <id>`), then inspect what
it logs (`sudo tail -c 400000 <logpath>`; count `[debug]` lines) and find the
config flag driving it (e.g. AGH `log.verbose` in its YAML; note `log.file: ""`
means the app's own rotation keys are inert and output goes to the container log).

**Expected**

A full accounting of root usage and identification of: (a) any unbounded
container log and its root-cause flag; (b) reclaimable apt cache; (c) journal
size and journald limits; (d) unused images (0 dangling expected) and unused
anonymous volumes.

**Decision**

- If root is ≥70% used or any container log is unbounded → Step 2.
- If root is healthy and logs are bounded → STOP (no change needed; record evidence).
- If state conflicts with expectations (e.g. missing sudo, unexpected mount) → STOP.

### Step 2 — Fix noisy container logging at the source, then truncate

**Action**

1. Back up the app config: `sudo cp <config> <config>.bak-YYYYMMDD-<issue>`.
2. Disable the debug/verbose flag (e.g. `log.verbose: true → false` in the AGH YAML).
3. Apply config with a container restart: `docker restart <container>` (config-level
   change; **no recreate** needed and daemon.json rotation would not apply anyway).
4. Truncate the accumulated logs: `sudo truncate -s 0 <json.log>` for the chatty
   container(s) (and any other oversized ones, e.g. traefik).
5. Record `df -h` before/after.

**Expected**

`docker logs <container>` no longer shows the `[debug]` flood; the `*-json.log`
stops growing; several GB reclaimed.

**Verification**

- `sudo tail -c 200000 <json.log>` after ≥1 minute → no new debug lines.
- `df -h` improvement recorded.
- Container still `Up (healthy)`.

**Rollback**

- Trigger: log volume unchanged, service degraded, or debug output is actually needed.
- Action: restore the config backup and `docker restart <container>`.
- Verify: original verbose behaviour back; container healthy.

### Step 3 — Clear apt cache and cap journald

**Action**

1. `sudo apt-get clean` (clears only `/var/cache/apt/archives`; `/var/lib/apt/lists`
   is not cleared by it and regenerates on `apt update` — optional/low value, skip).
2. `sudo journalctl --vacuum-size=100M`.
3. Write drop-in `/etc/systemd/journald.conf.d/00-disk-<issue>.conf`:
   `[Journal]` + `SystemMaxUse=200M`.
4. `sudo systemctl restart systemd-journald` (approved service touch).
5. Record `df -h` before/after.

**Expected**

Archives cleared (~1.4G on hk2), journal ≤100M, future journal capped at 200M.

**Verification**

- `du -sh /var/cache/apt/archives` → ~0.
- `journalctl --disk-usage` → ≤100M.
- `systemctl show systemd-journald -p ...` or restart log confirms new limit;
  `journalctl -b` still readable.

**Rollback**

- Trigger: journald fails to start or logs lost unexpectedly.
- Action: remove the drop-in, `sudo systemctl restart systemd-journald`.
- Verify: journald active, prior journal entries still listed.

### Step 4 — Remove unused Docker images (explicit keep-list)

**Action**

1. Enumerate unused images: `docker image ls` cross-checked against the images of
   running containers (`docker ps --format '{{.Image}}'`). Re-enumerate at
   execution time — the list drifts.
2. Present the exact removal list to the operator; keep the agreed rollback pin(s)
   (e.g. `powerdns/pdns-auth-50:5.0.5`) and delete the rest explicitly:
   `docker rmi <repo:tag> ...` (per image).
3. Record `df -h` before/after.

**Expected**

Only in-use images + kept pins remain; ~1–2.5G reclaimed (reclaim is an upper
bound — layers shared with kept images are not freed; measure with `df`, do not
promise the estimate).

**Verification**

- `docker image ls` shows only the expected set.
- `docker system df` images reclaimable ≈ 0 for the removed set.
- All containers still `Up`.

**Rollback**

- Trigger: an image that was actually needed was removed.
- Action: re-pull it from the registry (`docker pull <repo:tag>`); if a kept pin
  must change, update the compose pin and `up -d`.
- Verify: image present; affected service healthy.

### Step 5 — Remove unused anonymous volumes and build cache

**Action**

1. Enumerate volumes: `docker volume ls`, and confirm which are referenced by
   containers (`docker inspect` Mounts). Expected targets: anonymous volumes with
   no container reference.
2. `docker volume prune` (**no `-a`**) — engine ≥ v23 removes only unused
   anonymous volumes; in-use volumes (e.g. PG data) are protected by container
   references in every version.
3. `docker builder prune -f`.
4. Record `df -h` before/after.

**Expected**

Unused anonymous volumes (~1.2G on hk2) and build cache gone; in-use volumes intact.

**Verification**

- `docker volume ls` shows only in-use volumes.
- Services that own volumes (e.g. postgres) report healthy and data present.
- `df -h` improvement recorded.

**Rollback**

- Trigger: data loss suspected in a removed volume.
- Action: restore from backup if the volume ever contained data; verify against
  the pre-prune enumeration (targets must be anonymous + unreferenced before prune).
- Note: this is why target enumeration is recorded before pruning.

## Troubleshooting

### Troubleshooting A — Log still grows after disabling verbose

- Evidence: `sudo tail -c 200000 <json.log>` still shows new lines; app config re-checked.
- Allowed actions: check for a second verbose source (container entrypoint flags,
  other apps in the same log); check `docker inspect <c> --format '{{.HostConfig.LogConfig}}'`.
- Next step: back to Step 2 or STOP if a container-level log-opts change (recreate)
  would be needed — that is a separate approval.

### Troubleshooting B — `docker rmi` fails (image in use)

- Evidence: `image is being used by stopped container ...`.
- Allowed actions: identify the stopped container (`docker ps -a`); confirm it is
  not needed; remove it only with explicit approval.
- Next step: re-run rmi for the remaining images; never force-delete blindly.

### Troubleshooting C — `docker volume prune` would remove more than expected

- Evidence: prune dry-run/listing includes a named or referenced volume.
- Allowed actions: abort; do not add `-a`; re-check references.
- Next step: STOP and report to the operator with the enumeration.

## Final Verification

The flow is successful only when all of the following hold:

- `df -h` root usage is in the agreed band (VPS-81: 76% → ≤55% used; measure, do not assume).
- `docker system df` shows reclaimable ≈ 0 for images/volumes targeted.
- All containers `Up` (health checks pass); public services verified
  (`dig @<host-ip> SOA <zone>` for DNS hosts; service URLs reachable).
- Tracking issue updated with before/after `df`, actions, and the one-week
  observation checkpoint for log growth.

## Failure Handling

If the flow cannot complete:

1. Stop further mutation.
2. Collect command output, timestamps, and the exact step that failed.
3. Record completed steps, actual results, unmet expectations, and whether a
   rollback ran.
4. Hand over per the tracking issue with evidence; do not guess further.

## References

- Plane `vps` issue VPS-81 "hk2: 释放根盘空间" (+ subtasks VPS-82…88) — plan, review findings, approvals.
- [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) — host facts.
- [RUNBOOKS.md](../RUNBOOKS.md) — runbook spec; [runbooks/README.md](README.md) — index.