vps/runbooks/host-disk-cleanup.md

Runbook: Host disk cleanup (unbounded container logs / apt cache / docker artifacts)

Purpose

Reclaim space on a root filesystem that is filling up (≥70% used) on a Docker Compose host, by fixing unbounded container log growth at the source, clearing apt/journal caches, and removing unused Docker images/volumes. Success: root usage drops to a safe band (≤55% used, or per acceptance in the tracking issue) and log growth stays bounded afterwards.

Scope

• 适用环境: production single-root-fs hosts running Docker Compose stacks (first application: hk2.chans.xyz; reusable for mx2.windy.me / us2.wsvc.info which run the same unbounded-json.log pattern).
• 适用对象: root filesystem usage; container stdout/stderr log files (/var/lib/docker/containers/*/*-json.log); /var/cache/apt; systemd journal; unused Docker images / anonymous volumes / build cache.
• 不适用情形: hosts without systemd-journald or without Docker; LAN/HAOS hosts (use their own runbooks); cases needing disk growth (provider resize) rather than cleanup; anything touching service data volumes or /opt/* configs (STOP and use the service-specific runbook instead).

Ownership

• Owner: windy (operator) + agent executing per approval
• Last reviewed: 2026-09-02
• Related systems: hk2.chans.xyz (PowerDNS auth / AdGuard Home / Traefik / RustDesk compose stacks)

Preconditions

• SSH access to the target host with passwordless sudo (sudo -n true must succeed).
• A recorded df -h baseline and docker system df baseline.
• Explicit user approval for every service touch listed in Approval gates (recorded in the tracking issue, e.g. Plane vps VPS-81).
• No open incident on the target host.
• Container log growth root cause identified in Diagnose before mutating.

Inputs

| Input | Source | Required | Validation | |---|---:|---| | Target host | inventory/hosts.md | yes | SSH login + uname -r | | df/docker baseline | live read-only probe | yes | recorded before first mutation | | Approved service touches | user confirmation in tracking issue | yes | issue comment states approval | | Image keep-list (rollback pins) | operator decision in issue | yes | review docker image ls before rmi | | Backup of any config edited | local copy with timestamp | yes | exists before edit |

Safety

Non-negotiable rules

• Prefer read-only diagnosis before mutation (never mutate on an unmeasured disk).
• Never use rm on a live container log — use truncate -s 0 (keeps the fd valid).
• Never run docker image prune -a when a keep-list is intended — no keep-list exists; delete explicitly with docker rmi.
• Never run docker volume prune -a — plain docker volume prune (no -a) removes only unused anonymous volumes; named/in-use volumes stay.
• After every mutation, verify the expected state (df -h, container status).
• Destructive actions require explicit approval (Approval gates).

Stop conditions

• Live state conflicts with this runbook's preconditions or expectations (e.g. root usage differs wildly from baseline, or a container is unhealthy).
• Missing approval, missing backup, or missing rollback ability.
• A verification step fails with no documented next step.
• Any step would touch a volume/container/mount that is not on the approved list.

Approval gates

Action	Risk	Explicit approval	Approval record
docker restart <chatty container>	low (sec-level blip of that service only)	yes	tracking issue (VPS-81 T1)
systemctl restart systemd-journald	low (sec-level, no state loss)	yes	tracking issue (VPS-81 T2)
apt-get clean	low (re-downloadable)	no	—
journalctl --vacuum-* / journald drop-in	low	no (restart above is gated)	—
docker rmi of unused images	medium (rollback pin removed unless kept)	yes (keep-list)	tracking issue (VPS-81 T3)
docker volume prune	medium (data in anonymous volumes lost)	yes	tracking issue (VPS-81 T4)
docker builder prune	low	no	—

Procedure

Step 1 — Diagnose

Action

Read-only: df -h, df -i, sudo du -x -h --max-depth=1 /, docker system df, and locate oversized container logs: sudo ls -la /var/lib/docker/containers/*/*-json.log. Map a big log to its container (docker inspect -f '{{.Name}} {{.LogPath}}' <id>), then inspect what it logs (sudo tail -c 400000 <logpath>; count [debug] lines) and find the config flag driving it (e.g. AGH log.verbose in its YAML; note log.file: "" means the app's own rotation keys are inert and output goes to the container log).

Expected

A full accounting of root usage and identification of: (a) any unbounded container log and its root-cause flag; (b) reclaimable apt cache; (c) journal size and journald limits; (d) unused images (0 dangling expected) and unused anonymous volumes.

Decision

• If root is ≥70% used or any container log is unbounded → Step 2.
• If root is healthy and logs are bounded → STOP (no change needed; record evidence).
• If state conflicts with expectations (e.g. missing sudo, unexpected mount) → STOP.

Step 2 — Fix noisy container logging at the source, then truncate

Action

1. Back up the app config: sudo cp <config> <config>.bak-YYYYMMDD-<issue>.
2. Disable the debug/verbose flag (e.g. log.verbose: true → false in the AGH YAML).
3. Apply config with a container restart: docker restart <container> (config-level change; no recreate needed and daemon.json rotation would not apply anyway).
4. Truncate the accumulated logs: sudo truncate -s 0 <json.log> for the chatty container(s) (and any other oversized ones, e.g. traefik).
5. Record df -h before/after.

Expected

docker logs <container> no longer shows the [debug] flood; the *-json.log stops growing; several GB reclaimed.

Verification

• sudo tail -c 200000 <json.log> after ≥1 minute → no new debug lines.
• df -h improvement recorded.
• Container still Up (healthy).

Rollback

• Trigger: log volume unchanged, service degraded, or debug output is actually needed.
• Action: restore the config backup and docker restart <container>.
• Verify: original verbose behaviour back; container healthy.

Step 3 — Clear apt cache and cap journald

Action

1. sudo apt-get clean (clears only /var/cache/apt/archives; /var/lib/apt/lists is not cleared by it and regenerates on apt update — optional/low value, skip).
2. sudo journalctl --vacuum-size=100M.
3. Write drop-in /etc/systemd/journald.conf.d/00-disk-<issue>.conf: [Journal] + SystemMaxUse=200M.
4. sudo systemctl restart systemd-journald (approved service touch).
5. Record df -h before/after.

Expected

Archives cleared (~1.4G on hk2), journal ≤100M, future journal capped at 200M.

Verification

• du -sh /var/cache/apt/archives → ~0.
• journalctl --disk-usage → ≤100M.
• systemctl show systemd-journald -p ... or restart log confirms new limit; journalctl -b still readable.

Rollback

• Trigger: journald fails to start or logs lost unexpectedly.
• Action: remove the drop-in, sudo systemctl restart systemd-journald.
• Verify: journald active, prior journal entries still listed.

Step 4 — Remove unused Docker images (explicit keep-list)

Action

1. Enumerate unused images: docker image ls cross-checked against the images of running containers (docker ps --format '{{.Image}}'). Re-enumerate at execution time — the list drifts.
2. Present the exact removal list to the operator; keep the agreed rollback pin(s) (e.g. powerdns/pdns-auth-50:5.0.5) and delete the rest explicitly: docker rmi <repo:tag> ... (per image).
3. Record df -h before/after.

Expected

Only in-use images + kept pins remain; ~1–2.5G reclaimed (reclaim is an upper bound — layers shared with kept images are not freed; measure with df, do not promise the estimate).

Verification

• docker image ls shows only the expected set.
• docker system df images reclaimable ≈ 0 for the removed set.
• All containers still Up.

Rollback

• Trigger: an image that was actually needed was removed.
• Action: re-pull it from the registry (docker pull <repo:tag>); if a kept pin must change, update the compose pin and up -d.
• Verify: image present; affected service healthy.

Step 5 — Remove unused anonymous volumes and build cache

Action

1. Enumerate volumes: docker volume ls, and confirm which are referenced by containers (docker inspect Mounts). Expected targets: anonymous volumes with no container reference.
2. docker volume prune (no -a) — engine ≥ v23 removes only unused anonymous volumes; in-use volumes (e.g. PG data) are protected by container references in every version.
3. docker builder prune -f.
4. Record df -h before/after.

Expected

Unused anonymous volumes (~1.2G on hk2) and build cache gone; in-use volumes intact.

Verification

• docker volume ls shows only in-use volumes.
• Services that own volumes (e.g. postgres) report healthy and data present.
• df -h improvement recorded.

Rollback

• Trigger: data loss suspected in a removed volume.
• Action: restore from backup if the volume ever contained data; verify against the pre-prune enumeration (targets must be anonymous + unreferenced before prune).
• Note: this is why target enumeration is recorded before pruning.

Troubleshooting

Troubleshooting A — Log still grows after disabling verbose

• Evidence: sudo tail -c 200000 <json.log> still shows new lines; app config re-checked.
• Allowed actions: check for a second verbose source (container entrypoint flags, other apps in the same log); check docker inspect <c> --format '{{.HostConfig.LogConfig}}'.
• Next step: back to Step 2 or STOP if a container-level log-opts change (recreate) would be needed — that is a separate approval.

Troubleshooting B — docker rmi fails (image in use)

• Evidence: image is being used by stopped container ....
• Allowed actions: identify the stopped container (docker ps -a); confirm it is not needed; remove it only with explicit approval.
• Next step: re-run rmi for the remaining images; never force-delete blindly.

Troubleshooting C — docker volume prune would remove more than expected

• Evidence: prune dry-run/listing includes a named or referenced volume.
• Allowed actions: abort; do not add -a; re-check references.
• Next step: STOP and report to the operator with the enumeration.

Final Verification

The flow is successful only when all of the following hold:

• df -h root usage is in the agreed band (VPS-81: 76% → ≤55% used; measure, do not assume).
• docker system df shows reclaimable ≈ 0 for images/volumes targeted.
• All containers Up (health checks pass); public services verified (dig @<host-ip> SOA <zone> for DNS hosts; service URLs reachable).
• Tracking issue updated with before/after df, actions, and the one-week observation checkpoint for log growth.

Failure Handling

If the flow cannot complete:

1. Stop further mutation.
2. Collect command output, timestamps, and the exact step that failed.
3. Record completed steps, actual results, unmet expectations, and whether a rollback ran.
4. Hand over per the tracking issue with evidence; do not guess further.

References

• Plane vps issue VPS-81 "hk2: 释放根盘空间" (+ subtasks VPS-82…88) — plan, review findings, approvals.
• hosts/hk2.chans.xyz.md — host facts.
• RUNBOOKS.md — runbook spec; runbooks/README.md — index.