Compare commits
27
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ff1a92110c | ||
|
|
cc3fb99c14 | ||
|
|
6879d79cc6 | ||
|
|
c445c5f512 | ||
|
|
3c83246f24 | ||
|
|
c0c975584a | ||
|
|
0034cec925 | ||
|
|
c9dcde1274 | ||
|
|
037c4ccaa5 | ||
|
|
17bb171578 | ||
|
|
cecf7e6331 | ||
|
|
e957bc2bb1 | ||
|
|
de52cb8b57 | ||
|
|
b15e19bce9 | ||
|
|
bee54a6858 | ||
|
|
d6747028b4 | ||
|
|
f174aa1219 | ||
|
|
5b5f6042e6 | ||
|
|
50136b2ffd | ||
|
|
6707cebc88 | ||
|
|
908ff5412a | ||
|
|
e58283210a | ||
|
|
8c73d1f894 | ||
|
|
bc0a86245d | ||
|
|
13032fd0bb | ||
|
|
d2063e7496 | ||
|
|
ab808088b8 |
@@ -30,3 +30,6 @@ skills-lock.json
|
||||
.vscode/
|
||||
.idea/
|
||||
*~
|
||||
# Agent working scratch (not repo content).
|
||||
.agent-work/
|
||||
.tmp-*
|
||||
|
||||
@@ -30,9 +30,13 @@ This repo is the **agent ops handbook + fact source** for maintaining personal V
|
||||
> read-only unless a change is approved).
|
||||
|
||||
7. Record each material VPS operation, incident, configuration change, or
|
||||
verification outcome in the corresponding **Linear `vps` project**. Include
|
||||
scope, action, verification, and remaining follow-up; never put passwords,
|
||||
tokens, private keys, recovery keys, or private room IDs in Linear.
|
||||
verification outcome in the corresponding **Plane `vps` project**
|
||||
(self-hosted `plane.chans.xyz`, Plane MCP `mcp__plane__*`, following the
|
||||
`plane-workflow` skill). **Linear is retired as a record source (2026-09-03)
|
||||
— do not create Linear issues;** existing W1N-* entries are read-only
|
||||
history. Include scope, action, verification, and remaining follow-up; never
|
||||
put passwords, tokens, private keys, recovery keys, or private room IDs in
|
||||
Plane or Linear.
|
||||
|
||||
### Runbook execution rules
|
||||
|
||||
@@ -59,7 +63,7 @@ not maintain a second copy of the machine table here.
|
||||
| SMTP `mx2.windy.me:587` (STARTTLS) or `:465` | mx2 | client submission; full email + mailbox password — [runbook](runbooks/mailcow-smtp-client.md) |
|
||||
| IMAP `mx2.windy.me:993` | mx2 | same mailbox credentials |
|
||||
| https://auth.wsvc.info | us2 (`/opt/vaultwarden`) | Vaultwarden (Postgres, **operational**) — client Server URL |
|
||||
| `repo.windy.me:2222` | us2 (`/opt/soft-serve`) | Soft Serve (stub details) |
|
||||
| `repo.windy.me` (git SSH `:2222` / web HTTPS) | us2 (`/opt/gitea`) | Gitea — 1.27.3-rootless pinned, backup sidecar; details in [hosts/us2.wsvc.info.md](hosts/us2.wsvc.info.md) |
|
||||
| DNS `ns1.wsvc.info:53` | hk2 (`/opt/pdns`, Auth **5.0.6**) | PowerDNS auth — zones `windy.me`, `wsvc.info`, `chans.xyz` |
|
||||
| https://pdns.wsvc.info | hk2 (`poweradmin`) | Poweradmin UI |
|
||||
| https://pgweb.wsvc.info | hk2 (`pgweb`) | PowerDNS Postgres browser |
|
||||
@@ -67,6 +71,7 @@ not maintain a second copy of the machine table here.
|
||||
| https://synapse.chans.xyz | synapse | Synapse Client-Server + Federation API |
|
||||
| https://account.chans.xyz | synapse | Matrix Authentication Service (local passwords) |
|
||||
| https://admin.chans.xyz | synapse | Element Admin console (MAS admin auth) |
|
||||
| https://plane.chans.xyz | synapse (`plane`, Helm `plane-ce` 1.8.0 / v1.4.1) | Plane project management (self-hosted, K3s) |
|
||||
|
||||
### Upstream docs
|
||||
|
||||
@@ -76,6 +81,8 @@ not maintain a second copy of the machine table here.
|
||||
|
||||
**Matrix (ESS on synapse):** Matrix homeserver running on `synapse.chans.xyz` via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone `chans.xyz` managed by hk2 PowerDNS. Before changing config, read [docs/matrix-upstream.md](docs/matrix-upstream.md) and [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: [matrix-health](runbooks/matrix-health.md).
|
||||
|
||||
**Plane (on synapse):** Self-hosted Plane project management at `plane.chans.xyz`, Helm release `plane-app` (chart `plane-ce-1.8.0`, app `v1.4.1`) in ns `plane` on the same K3s node as Matrix. Config from `/home/windy/plane-k3s/values.yaml`; workload/cert/ingress details in [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). Its Postgres/MinIO PVCs are **not** backed up.
|
||||
|
||||
**RustDesk:** Self-hosted RustDesk server on `hk2.chans.xyz` (`/opt/rustdesk`, containers `hbbs`/`hbbr`, image pinned `1.1.14`). The `hbbs -r` relay hostname must resolve to the host's public IP `154.36.174.161` — use `hk2.chans.xyz` (never `hk2.wsvc.info`, which has no DNS record). Health: [rustdesk-health](runbooks/rustdesk-health.md).
|
||||
|
||||
## Runbooks & scripts
|
||||
@@ -91,8 +98,10 @@ not maintain a second copy of the machine table here.
|
||||
| PowerDNS health (hk2) | [runbooks/pdns-health.md](runbooks/pdns-health.md) |
|
||||
| PowerDNS upstream refs | [docs/pdns-upstream.md](docs/pdns-upstream.md) |
|
||||
| Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) |
|
||||
| Plane health | [runbooks/plane-health.md](runbooks/plane-health.md) |
|
||||
| RustDesk health (hk2) | [runbooks/rustdesk-health.md](runbooks/rustdesk-health.md) |
|
||||
| AdGuard Home health | [runbooks/adguard-home-health.md](runbooks/adguard-home-health.md) |
|
||||
| Host disk cleanup | [runbooks/host-disk-cleanup.md](runbooks/host-disk-cleanup.md) |
|
||||
| Home Assistant maintenance | [runbooks/home-assistant-maintenance.md](runbooks/home-assistant-maintenance.md) + [scripts/ha-maintenance.sh](runbooks/scripts/ha-maintenance.sh) |
|
||||
| matrix_e2ee update (hass.windy.lan) | [runbooks/matrix-e2ee-update.md](runbooks/matrix-e2ee-update.md) |
|
||||
| Matrix upstream refs | [docs/matrix-upstream.md](docs/matrix-upstream.md) |
|
||||
@@ -100,6 +109,7 @@ not maintain a second copy of the machine table here.
|
||||
| UniFi local-service proxy bypass | [docs/unifi-openclash-localhost.md](docs/unifi-openclash-localhost.md) |
|
||||
| UniFi SSO login setting (Ansible) | `cd ansible && ansible-playbook playbooks/unifi-sso.yml --limit unifi` |
|
||||
| Routine Ansible operations | [runbooks/ansible-operations.md](runbooks/ansible-operations.md) |
|
||||
| Routine make commands | `make help` (wraps `ansible-operations.md` read-only + gated flows) |
|
||||
| Issue → mergeable change | [runbooks/issue-to-merge.md](runbooks/issue-to-merge.md) |
|
||||
| Fix failing health/playbook run | [runbooks/fix-ci.md](runbooks/fix-ci.md) |
|
||||
| Release a reviewed change | [runbooks/release.md](runbooks/release.md) |
|
||||
@@ -115,7 +125,12 @@ Routine mailcow health: `cd ansible && ansible-playbook playbooks/health-report.
|
||||
|
||||
### Issue tracker
|
||||
|
||||
Issues are tracked in Linear and created/updated via the Linear MCP (`vps` project). See `docs/agents/issue-tracker.md`.
|
||||
Issues are tracked in **Plane** — self-hosted at `plane.chans.xyz`, project
|
||||
`vps` — and created/updated via the Plane MCP (`mcp__plane__*`), following the
|
||||
`plane-workflow` skill. **Linear is retired as a record source (2026-09-03); do
|
||||
not create Linear issues.** Existing W1N-* entries are read-only history.
|
||||
`docs/agents/issue-tracker.md` documents the retired Linear workflow and is
|
||||
stale; treat this section as authoritative.
|
||||
|
||||
### Triage labels
|
||||
|
||||
@@ -166,4 +181,5 @@ hosts/ # per-host facts
|
||||
runbooks/ # step-by-step ops (README.md = index, _template.md = template)
|
||||
docs/ # upstream refs / design notes / research records (active + archive/)
|
||||
scripts/validate-repo.sh # repo-wide validation (run before merging)
|
||||
Makefile # routine validate / health / gated ansible wrappers
|
||||
```
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
# VPS ops hub — routine validate / health / gated Ansible wrappers.
|
||||
# See runbooks/ansible-operations.md for playbook semantics.
|
||||
|
||||
SHELL := /usr/bin/env bash
|
||||
.SHELLFLAGS := -eu -o pipefail -c
|
||||
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
REPO_ROOT := $(CURDIR)
|
||||
ANSIBLE_DIR := $(REPO_ROOT)/ansible
|
||||
export ANSIBLE_LOCAL_TEMP := $(REPO_ROOT)/.ansible/tmp
|
||||
export ANSIBLE_HOME := $(REPO_ROOT)/.ansible
|
||||
|
||||
LIMIT ?=
|
||||
EXTRA ?=
|
||||
VERBOSE ?= 0
|
||||
CONFIRM ?= 0
|
||||
TARGETS ?=
|
||||
TRAEFIK ?= 0
|
||||
|
||||
LIMIT_FLAG := $(if $(LIMIT),--limit $(LIMIT),)
|
||||
VERBOSE_FLAG := $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vvv,))
|
||||
|
||||
.PHONY: help validate check deps galaxy syntax ansible-prep \
|
||||
ping inventory audit health health-mailcow health-matrix \
|
||||
maint-preview baseline compose-check \
|
||||
install-healthchecks install-matrix-healthchecks compose-deploy reconcile
|
||||
|
||||
help:
|
||||
@printf '%s\n' \
|
||||
'VPS ops hub — make targets (run from repo root)' \
|
||||
'' \
|
||||
'Variables: LIMIT=<group|host> CONFIRM=1 TARGETS=<svc[,svc]> TRAEFIK=1 VERBOSE=0|1|2 EXTRA=...' \
|
||||
'' \
|
||||
'Local / repo:' \
|
||||
' validate, check scripts/validate-repo.sh (pre-merge gate)' \
|
||||
' deps, galaxy ansible-galaxy collection install' \
|
||||
' syntax ansible-playbook --syntax-check all playbooks' \
|
||||
'' \
|
||||
'Read-only remote (ansible):' \
|
||||
' ping ansible managed -m ping' \
|
||||
' inventory ansible-inventory --graph' \
|
||||
' audit playbooks/audit.yml' \
|
||||
' health [LIMIT=…] playbooks/health-report.yml' \
|
||||
' health-mailcow health --limit mailcow' \
|
||||
' health-matrix health --limit matrix' \
|
||||
' maint-preview playbooks/maintenance-preview.yml' \
|
||||
' baseline playbooks/baseline.yml' \
|
||||
' compose-check compose-deploy --check --diff (requires LIMIT=)' \
|
||||
'' \
|
||||
'Mutating (require CONFIRM=1; host-scoped targets require LIMIT=):' \
|
||||
' install-healthchecks playbooks/healthchecks.yml' \
|
||||
' install-matrix-healthchecks playbooks/matrix-healthchecks.yml' \
|
||||
' compose-deploy playbooks/compose-deploy.yml' \
|
||||
' reconcile playbooks/compose-reconcile.yml (requires TARGETS=)' \
|
||||
'' \
|
||||
'Examples:' \
|
||||
' make validate' \
|
||||
' make health LIMIT=mailcow' \
|
||||
' make compose-check LIMIT=vaultwarden' \
|
||||
' make compose-deploy LIMIT=vaultwarden CONFIRM=1' \
|
||||
' make reconcile LIMIT=powerdns TARGETS=auth CONFIRM=1' \
|
||||
' make reconcile LIMIT=vaultwarden TARGETS=vaultwarden TRAEFIK=1 CONFIRM=1' \
|
||||
'' \
|
||||
'Advanced (not wrapped — use ansible-playbook directly):' \
|
||||
' us4-firewalld, unifi-sso, k3s-server, matrix-stack, wireguard-harden,' \
|
||||
' restic, rustdesk, email-alerts, mailcow update runbook'
|
||||
|
||||
validate check:
|
||||
@bash "$(REPO_ROOT)/scripts/validate-repo.sh"
|
||||
|
||||
deps galaxy: ansible-prep
|
||||
@command -v ansible-galaxy >/dev/null 2>&1 || { echo "ansible-galaxy not found; install Ansible first." >&2; exit 1; }
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-galaxy collection install -r requirements.yml
|
||||
|
||||
syntax: ansible-prep
|
||||
@if ! command -v ansible-playbook >/dev/null 2>&1; then \
|
||||
echo "ansible-playbook not found; syntax check skipped." >&2; \
|
||||
exit 0; \
|
||||
fi
|
||||
@fail=0; \
|
||||
for p in "$(ANSIBLE_DIR)"/playbooks/*.yml; do \
|
||||
if ! (cd "$(ANSIBLE_DIR)" && ansible-playbook --syntax-check "playbooks/$$(basename "$$p")" >/dev/null 2>&1); then \
|
||||
echo "syntax-check failed: $$p" >&2; \
|
||||
fail=1; \
|
||||
fi; \
|
||||
done; \
|
||||
exit $$fail
|
||||
|
||||
ansible-prep:
|
||||
@mkdir -p "$(ANSIBLE_HOME)/tmp" "$(ANSIBLE_HOME)/ssh-control"
|
||||
|
||||
define require_ansible
|
||||
@command -v ansible-playbook >/dev/null 2>&1 || { echo "ansible-playbook not found; install Ansible first." >&2; exit 1; }
|
||||
endef
|
||||
|
||||
define require_limit
|
||||
@if [ -z "$(LIMIT)" ]; then \
|
||||
echo "LIMIT is required (e.g. LIMIT=mailcow, LIMIT=vaultwarden, LIMIT=powerdns)." >&2; \
|
||||
exit 1; \
|
||||
fi
|
||||
endef
|
||||
|
||||
define require_confirm
|
||||
@if [ "$(CONFIRM)" != "1" ]; then \
|
||||
echo "Mutating operation blocked. Re-run with CONFIRM=1" >&2; \
|
||||
exit 1; \
|
||||
fi
|
||||
endef
|
||||
|
||||
define require_targets
|
||||
@if [ -z "$(TARGETS)" ]; then \
|
||||
echo "TARGETS is required (comma-separated service names, e.g. TARGETS=auth or TARGETS=vaultwarden)." >&2; \
|
||||
exit 1; \
|
||||
fi
|
||||
endef
|
||||
|
||||
ping: ansible-prep
|
||||
$(require_ansible)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible managed -m ping $(LIMIT_FLAG) $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
inventory: ansible-prep
|
||||
$(require_ansible)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-inventory --graph $(EXTRA)
|
||||
|
||||
audit: ansible-prep
|
||||
$(require_ansible)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/audit.yml $(LIMIT_FLAG) $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
health: ansible-prep
|
||||
$(require_ansible)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/health-report.yml $(LIMIT_FLAG) $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
health-mailcow: ansible-prep
|
||||
$(require_ansible)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/health-report.yml --limit mailcow $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
health-matrix: ansible-prep
|
||||
$(require_ansible)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/health-report.yml --limit matrix $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
maint-preview: ansible-prep
|
||||
$(require_ansible)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/maintenance-preview.yml $(LIMIT_FLAG) $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
baseline: ansible-prep
|
||||
$(require_ansible)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/baseline.yml $(LIMIT_FLAG) $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
compose-check: ansible-prep
|
||||
$(require_ansible)
|
||||
$(require_limit)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/compose-deploy.yml --check --diff --limit $(LIMIT) $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
install-healthchecks: ansible-prep
|
||||
$(require_ansible)
|
||||
$(require_confirm)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/healthchecks.yml $(LIMIT_FLAG) $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
install-matrix-healthchecks: ansible-prep
|
||||
$(require_ansible)
|
||||
$(require_confirm)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/matrix-healthchecks.yml $(LIMIT_FLAG) $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
compose-deploy: ansible-prep
|
||||
$(require_ansible)
|
||||
$(require_limit)
|
||||
$(require_confirm)
|
||||
@cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/compose-deploy.yml --limit $(LIMIT) \
|
||||
-e '{"compose_deploy_confirm": true}' $(VERBOSE_FLAG) $(EXTRA)
|
||||
|
||||
reconcile: ansible-prep
|
||||
$(require_ansible)
|
||||
$(require_limit)
|
||||
$(require_targets)
|
||||
$(require_confirm)
|
||||
@json=$$(python3 -c 'import json,sys; t=[x.strip() for x in sys.argv[1].split(",") if x.strip()]; \
|
||||
(not t) and sys.exit("TARGETS must contain at least one non-empty service name"); \
|
||||
d={"service_reconcile_confirm": True, "service_reconcile_targets": t}; \
|
||||
(sys.argv[2]=="1") and d.update({"service_reconcile_restart_traefik": True}); \
|
||||
print(json.dumps(d))' "$(TARGETS)" "$(TRAEFIK)"); \
|
||||
cd "$(ANSIBLE_DIR)" && ansible-playbook playbooks/compose-reconcile.yml --limit $(LIMIT) \
|
||||
-e "$$json" $(VERBOSE_FLAG) $(EXTRA)
|
||||
@@ -31,9 +31,27 @@ compose_ps() {
|
||||
}
|
||||
|
||||
check_compose() {
|
||||
local output
|
||||
output="$(compose_ps)" || { record critical 'compose_ps_failed'; return; }
|
||||
if grep -qiE 'Exited|Restarting|[[:space:]]Dead[[:space:]]' <<<"$output"; then
|
||||
local output services bad
|
||||
# Only flag containers of *active* services (config --services excludes
|
||||
# debug/profile-gated services such as vaultwarden's pgweb, which is
|
||||
# intentionally stopped unless started with --profile debug).
|
||||
services="$(docker compose --project-directory '{{ compose_project_dir }}' config --services 2>/dev/null)" || { record critical 'compose_ps_failed'; return; }
|
||||
output="$(docker compose --project-directory '{{ compose_project_dir }}' ps --all --format json 2>&1)" || { record critical 'compose_ps_failed'; return; }
|
||||
bad="$(printf '%s\n' "$output" | python3 -c '
|
||||
import json, sys
|
||||
services = set(sys.argv[1].split())
|
||||
for line in sys.stdin:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
c = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if c.get("Service") in services and c.get("State") in ("exited", "restarting", "dead"):
|
||||
print(c.get("Service"))
|
||||
' "$services")"
|
||||
if [[ -n "$bad" ]]; then
|
||||
record critical 'compose_unhealthy_container'
|
||||
else
|
||||
record ok 'compose_ok'
|
||||
|
||||
@@ -14,5 +14,11 @@ rm -f '{{ healthcheck_state_dir }}/latest-{{ healthcheck_profile_scripts[profile
|
||||
this_rc="${PIPESTATUS[0]}"
|
||||
[ "$this_rc" -gt "$rc" ] && rc="$this_rc"
|
||||
{% endfor %}
|
||||
aggregate_result{% for profile in healthcheck_profiles %} {{ healthcheck_profile_scripts[profile] | replace('.sh', '') }}{% endfor %}
|
||||
# Collect profile check names line-by-line (robust against Jinja trim_blocks
|
||||
# whitespace control, which would otherwise merge this into one line).
|
||||
aggregate_args=""
|
||||
{% for profile in healthcheck_profiles %}
|
||||
aggregate_args="$aggregate_args {{ healthcheck_profile_scripts[profile] | replace('.sh', '') }}"
|
||||
{% endfor %}
|
||||
aggregate_result $aggregate_args
|
||||
exit "$rc"
|
||||
|
||||
@@ -15,11 +15,12 @@ grep -Fq 'vw-db' <<<"$health" || record critical 'postgres_missing'
|
||||
check_https 'https://auth.wsvc.info/' '^200$'
|
||||
check_tls_days auth.wsvc.info 443
|
||||
|
||||
# Read effective config only inside the service and report booleans/fingerprints,
|
||||
# never its SMTP password or other secret fields.
|
||||
smtp_result="$(docker compose --project-directory '{{ compose_project_dir }}' exec -T vaultwarden python3 - <<'PY' 2>&1
|
||||
# Read effective config from the mounted vw-data dir on the host and run the
|
||||
# SMTP AUTH probe from the host (the vaultwarden image has no python3; the
|
||||
# host does). Never print the SMTP password.
|
||||
smtp_result="$(python3 - <<'PY' 2>&1
|
||||
import json, pathlib, smtplib, ssl
|
||||
cfg=json.loads(pathlib.Path('/data/config.json').read_text())
|
||||
cfg=json.loads(pathlib.Path('{{ compose_project_dir }}/vw-data/config.json').read_text())
|
||||
host=cfg.get('smtp_host'); port=int(cfg.get('smtp_port') or 0)
|
||||
user=cfg.get('smtp_username')
|
||||
smtp_secret=cfg.get('smtp_password')
|
||||
|
||||
@@ -11,6 +11,8 @@ files: every secret is a `${VAR}` reference resolved from the **server-local
|
||||
| `vaultwarden` | us2 (`/opt/vaultwarden`) | `compose/vaultwarden/compose.yml` | static file + `compose-deploy.yml` |
|
||||
| `pdns` | hk2 (`/opt/pdns`) | `compose/pdns/compose.yml` | static file + `compose-deploy.yml` |
|
||||
| `pgdb` | pgdb (`/opt/database`, 无 ansible) | `compose/pgdb/compose.yml` | static file(手动部署:scp → `docker compose config -q` → `up -d`;服务器文件名 `docker-compose.yml`) |
|
||||
| `soft-serve` | us2 (`/opt/soft-serve`, 已退役停用) | `compose/soft-serve/compose.yml` (+ `Dockerfile.backup`, `scripts/`) | static file(参考镜像; 2026-09-18 被 gitea 替换 VPS-94, 数据保留作回滚) |
|
||||
| `gitea` | us2 (`/opt/gitea`) | `compose/gitea/compose.yml` (+ `Dockerfile.backup`, `scripts/`) | static file(参考镜像, 未接入 compose-deploy; 服务器文件为准; 2026-09-18 替换 soft-serve, VPS-94) |
|
||||
| `adguardhome` | dns.windy.lan (`/opt/adguardhome`) | — (待从 LAN 提取) | static file (pending) |
|
||||
| `unifi` | ubnt (`/home/windy/unifi-9`) | — (待从 LAN 提取) | static file (pending) |
|
||||
| `wireguard` | us4 (`/opt/wireguard`) | `ansible/templates/wireguard-compose.yml.j2` | role-rendered (inventory vars) |
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
# compose/gitea — 秘密一律走服务器本地 .env, 不入库
|
||||
# 迁移期一次性: Gitea 管理员生成的 token (mirror-migrate.sh 读取, 用后撤销)
|
||||
GITEA_MIGRATE_USER=
|
||||
GITEA_MIGRATE_TOKEN=
|
||||
@@ -0,0 +1,3 @@
|
||||
FROM alpine:3.20
|
||||
RUN apk add --no-cache sqlite rsync tzdata
|
||||
WORKDIR /scripts
|
||||
@@ -0,0 +1,59 @@
|
||||
# Gitea on us2 — reference compose (Plane VPS-94, 迁移完成 2026-09-18)
|
||||
# 参考镜像, 服务器 /opt/gitea 文件为准 (同 soft-serve 约定, 未接入 compose-deploy)
|
||||
# rootless 镜像: uid 1000 原生非 root; 数据 /var/lib/gitea (宿主 ./data), 配置 /etc/gitea (宿主 ./config)
|
||||
# SSH: 容器内监听 2322 (非特权, SSH_LISTEN_PORT), 对外 repo.windy.me:2222 经 Traefik TCP entrypoint `ssh`
|
||||
services:
|
||||
gitea:
|
||||
image: gitea/gitea@sha256:1c17ecaead42eb3b5391553d8708103a4beb0e86edf5b9ebc1eb269c318845f2 # 1.27.3-rootless
|
||||
container_name: gitea
|
||||
restart: unless-stopped
|
||||
user: "1000:1000"
|
||||
environment:
|
||||
TZ: Asia/Shanghai
|
||||
volumes:
|
||||
- ./data:/var/lib/gitea
|
||||
- ./config:/etc/gitea
|
||||
- ./secrets:/secrets:ro # 复用的 soft-serve host key (SSH_SERVER_HOST_KEYS)
|
||||
networks:
|
||||
- traefik
|
||||
labels:
|
||||
- traefik.enable=true
|
||||
# Web UI: repo.windy.me (2026-09-18 操作者决定复用现有域名, 免 DNS 变更)
|
||||
- traefik.http.routers.gitea-web.rule=Host(`repo.windy.me`)
|
||||
- traefik.http.routers.gitea-web.entrypoints=websecure
|
||||
- traefik.http.routers.gitea-web.tls.certresolver=letsencrypt
|
||||
- traefik.http.services.gitea-web.loadbalancer.server.port=3000
|
||||
# SSH: 接管 :2222 (entrypoint 已存在, router 动态生效, 无需重启 Traefik)
|
||||
- traefik.tcp.routers.gitea-ssh.entrypoints=ssh
|
||||
- traefik.tcp.routers.gitea-ssh.rule=HostSNI(`*`)
|
||||
- traefik.tcp.routers.gitea-ssh.tls=false
|
||||
- traefik.tcp.services.gitea-ssh.loadbalancer.server.port=2322
|
||||
|
||||
gitea-backup:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.backup
|
||||
container_name: gitea-backup
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ./data:/data:ro
|
||||
- ./config:/config:ro
|
||||
- ./backups:/backup
|
||||
- ./scripts:/scripts
|
||||
environment:
|
||||
TZ: Asia/Shanghai
|
||||
BACKUP_UID: 1000
|
||||
BACKUP_GID: 1000
|
||||
entrypoint: >
|
||||
/bin/sh -ec "
|
||||
umask 077 &&
|
||||
touch /backup/backup.log &&
|
||||
crontab /scripts/crontab.txt &&
|
||||
echo '[INFO] gitea backup cron installed' &&
|
||||
crond -f -l 8
|
||||
"
|
||||
|
||||
networks:
|
||||
traefik:
|
||||
external: true
|
||||
name: vw-net
|
||||
Executable
+20
@@ -0,0 +1,20 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
umask 077
|
||||
D() { date "+%Y-%m-%d %H:%M:%S"; }
|
||||
TS=$(date +%Y%m%d_%H%M%S)
|
||||
OUT="/backup/gitea_${TS}"
|
||||
mkdir -p "$OUT"
|
||||
echo "[$(D)] Starting gitea backup -> $OUT"
|
||||
# rootless 布局: app.ini=/etc/gitea(宿主 ./config), db+repos=/var/lib/gitea/data(宿主 ./data/data)
|
||||
# app.ini 含 SECRET_KEY/INTERNAL_TOKEN — 恢复 2FA/session/mirror 凭据必需
|
||||
tar czf "$OUT/app.ini.tar.gz" -C /config app.ini
|
||||
sqlite3 /data/data/gitea.db ".backup '$OUT/gitea.db'"
|
||||
rsync -a /data/data/git/repositories/ "$OUT/repos/"
|
||||
tar czf "$OUT/repos.tar.gz" -C "$OUT" repos
|
||||
rm -rf "$OUT/repos"
|
||||
chmod 600 "$OUT"/*
|
||||
if [ -n "${BACKUP_UID:-}" ] && [ -n "${BACKUP_GID:-}" ]; then
|
||||
chown -R "$BACKUP_UID:$BACKUP_GID" "$OUT" /backup/backup.log
|
||||
fi
|
||||
echo "[$(D)] Backup OK: $(du -sh "$OUT" | cut -f1)"
|
||||
@@ -0,0 +1,4 @@
|
||||
# Run gitea backup daily at 02:00
|
||||
0 2 * * * /bin/sh /scripts/backup.sh >> /backup/backup.log 2>&1
|
||||
# Prune backups older than 14 days daily at 03:00
|
||||
0 3 * * * /bin/sh /scripts/prune.sh >> /backup/backup.log 2>&1
|
||||
Executable
+41
@@ -0,0 +1,41 @@
|
||||
#!/bin/sh
|
||||
# 一次性迁移辅助 (Plane VPS-94 Phase 2): 在 gitea 容器内执行。
|
||||
# 已于 2026-09-18 执行完成 (16 仓), 留档备查; 复用时按 VPS-94 流程重生成一次性 token。
|
||||
# 用法:
|
||||
# GITEA_MIGRATE_USER=<user> GITEA_MIGRATE_TOKEN=<token> \
|
||||
# docker exec -e GITEA_MIGRATE_USER -e GITEA_MIGRATE_TOKEN gitea \
|
||||
# /scripts/mirror-migrate.sh [public_repo ...]
|
||||
# 每仓: API 建仓 (默认 private, 参数中列出的为 public) -> push --mirror。
|
||||
# default_branch 按源仓 symbolic-ref HEAD 设置, 避免非 main 源仓在 Gitea 显示为空。
|
||||
# 结束后按 VPS-94 Phase 3 逐仓核对 git ls-remote ref 全集。
|
||||
set -eu
|
||||
MUSER="${GITEA_MIGRATE_USER:?need GITEA_MIGRATE_USER}"
|
||||
TOKEN="${GITEA_MIGRATE_TOKEN:?need GITEA_MIGRATE_TOKEN}"
|
||||
SRC="/migration-src"
|
||||
API="http://localhost:3000/api/v1"
|
||||
PUBLIC_REPOS=" $* "
|
||||
|
||||
migrate_one() {
|
||||
dir="$1"
|
||||
git -C "$dir" rev-parse --git-dir >/dev/null 2>&1 || { echo "[SKIP] $dir (not a git repo)"; return 0; }
|
||||
name=$(basename "$dir"); name=${name%.git}
|
||||
def_branch=$(git -C "$dir" symbolic-ref --short HEAD)
|
||||
case "$PUBLIC_REPOS" in *" $name "*) private=false ;; *) private=true ;; esac
|
||||
echo "[MIGRATE] $name (default=$def_branch private=$private)"
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' -X POST "$API/user/repos" \
|
||||
-H "Authorization: token $TOKEN" -H "Content-Type: application/json" \
|
||||
-d "{\"name\":\"$name\",\"private\":$private,\"default_branch\":\"$def_branch\",\"auto_init\":false}")
|
||||
case "$code" in
|
||||
201) : ;;
|
||||
409) echo " [WARN] $name 已存在, 直接补推" ;;
|
||||
*) echo " [FAIL] create HTTP $code"; return 1 ;;
|
||||
esac
|
||||
git -C "$dir" push --mirror "http://$MUSER:$TOKEN@localhost:3000/$MUSER/$name.git"
|
||||
echo " [OK] $name pushed"
|
||||
}
|
||||
|
||||
for dir in "$SRC"/*.git "$SRC"/cdia; do
|
||||
[ -d "$dir" ] || continue
|
||||
migrate_one "$dir"
|
||||
done
|
||||
echo "[DONE] 全部处理完毕; 迁移后记得撤销一次性 token"
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
D() { date "+%Y-%m-%d %H:%M:%S"; }
|
||||
ls -dt /backup/gitea_* 2>/dev/null | tail -n +15 | xargs -r rm -rf
|
||||
echo "[$(D)] Pruned. Kept $(ls -d /backup/gitea_* 2>/dev/null | wc -l) backups (max 14)"
|
||||
@@ -0,0 +1,24 @@
|
||||
[Unit]
|
||||
Description=Reconcile pgdb compose stack (timescaledb + pgweb + pg-backup) at boot
|
||||
Documentation=file:///opt/database/docker-compose.yml
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
WorkingDirectory=/opt/database
|
||||
# Idempotent boot-time reconcile. docker's own restore can fail to bind the
|
||||
# published ports (192.168.55.15:5432/8081) when the VM IP is not yet usable
|
||||
# right after boot (EADDRNOTAVAIL, observed 2026-08-30): timescaledb/pgweb
|
||||
# then stay stopped until a manual `docker compose up`. This unit retries
|
||||
# `docker compose up -d` (a no-op when the stack is healthy) until the port
|
||||
# listens, and force-recreates as a last resort to recover a network-detached
|
||||
# container. Data lives on bind mounts (/srv/pgdata, /opt/database/backups),
|
||||
# so recreation is safe.
|
||||
ExecStart=/bin/bash -c 'for i in $(seq 1 12); do docker compose up -d --remove-orphans; sleep 2; if ss -tln | grep -q "192.168.55.15:5432"; then exit 0; fi; sleep 3; done; echo "pgdb-compose: retries exhausted, force-recreating"; docker compose up -d --force-recreate; sleep 10; ss -tln | grep -q "192.168.55.15:5432"'
|
||||
TimeoutStartSec=180
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,2 @@
|
||||
# Soft Serve initial admin public key (used only on first boot)
|
||||
SOFT_SERVE_INITIAL_ADMIN_KEYS=ssh-ed25519 AAAA... # replace with admin public key
|
||||
@@ -0,0 +1,3 @@
|
||||
FROM alpine:3.20
|
||||
RUN apk add --no-cache sqlite tzdata
|
||||
WORKDIR /scripts
|
||||
@@ -0,0 +1,59 @@
|
||||
services:
|
||||
soft-serve:
|
||||
image: charmcli/soft-serve:v0.12.2
|
||||
container_name: soft-serve
|
||||
restart: unless-stopped
|
||||
# non-root (uid 1000 = windy; 与 backup sidecar BACKUP_UID 一致)
|
||||
user: "1000:1000"
|
||||
|
||||
environment:
|
||||
SOFT_SERVE_DATA_PATH: /var/lib/soft-serve
|
||||
SOFT_SERVE_INITIAL_ADMIN: windy
|
||||
SOFT_SERVE_INITIAL_ADMIN_KEYS: ${SOFT_SERVE_INITIAL_ADMIN_KEYS}
|
||||
|
||||
volumes:
|
||||
- ./data:/var/lib/soft-serve
|
||||
- soft-serve-app:/soft-serve
|
||||
|
||||
networks:
|
||||
- traefik
|
||||
|
||||
labels:
|
||||
- traefik.enable=true
|
||||
|
||||
# SSH over TCP via Traefik (entryPoint ssh -> container port 23231)
|
||||
- traefik.tcp.routers.softserve-ssh.entrypoints=ssh
|
||||
- traefik.tcp.routers.softserve-ssh.rule=HostSNI(`*`)
|
||||
- traefik.tcp.routers.softserve-ssh.tls=false
|
||||
- traefik.tcp.services.softserve-ssh.loadbalancer.server.port=23231
|
||||
|
||||
soft-serve-backup:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.backup
|
||||
container_name: soft-serve-backup
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ./data:/data:ro
|
||||
- ./backups:/backup
|
||||
- ./scripts:/scripts
|
||||
environment:
|
||||
TZ: Asia/Shanghai
|
||||
BACKUP_UID: 1000
|
||||
BACKUP_GID: 1000
|
||||
entrypoint: >
|
||||
/bin/sh -ec "
|
||||
umask 077 &&
|
||||
touch /backup/backup.log &&
|
||||
crontab /scripts/crontab.txt &&
|
||||
echo '[INFO] soft-serve backup cron installed' &&
|
||||
crond -f -l 8
|
||||
"
|
||||
|
||||
volumes:
|
||||
soft-serve-app:
|
||||
|
||||
networks:
|
||||
traefik:
|
||||
external: true
|
||||
name: vw-net
|
||||
@@ -0,0 +1,15 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
umask 077
|
||||
D() { date "+%Y-%m-%d %H:%M:%S"; }
|
||||
TS=$(date +%Y%m%d_%H%M%S)
|
||||
OUT="/backup/soft-serve_${TS}"
|
||||
mkdir -p "$OUT"
|
||||
echo "[$(D)] Starting soft-serve backup -> $OUT"
|
||||
tar czf "$OUT/repos-config.tar.gz" -C /data repos hooks config.yaml ssh
|
||||
sqlite3 /data/soft-serve.db ".backup '$OUT/soft-serve.db'"
|
||||
chmod 600 "$OUT/repos-config.tar.gz" "$OUT/soft-serve.db"
|
||||
if [ -n "${BACKUP_UID:-}" ] && [ -n "${BACKUP_GID:-}" ]; then
|
||||
chown -R "$BACKUP_UID:$BACKUP_GID" "$OUT" /backup/backup.log
|
||||
fi
|
||||
echo "[$(D)] Backup OK: $(du -sh "$OUT" | cut -f1)"
|
||||
@@ -0,0 +1,4 @@
|
||||
# Run soft-serve backup daily at 02:00
|
||||
0 2 * * * /bin/sh /scripts/backup.sh >> /backup/backup.log 2>&1
|
||||
# Prune backups older than 14 days daily at 03:00
|
||||
0 3 * * * /bin/sh /scripts/prune.sh >> /backup/backup.log 2>&1
|
||||
@@ -0,0 +1,5 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
D() { date "+%Y-%m-%d %H:%M:%S"; }
|
||||
ls -dt /backup/soft-serve_* 2>/dev/null | tail -n +15 | xargs -r rm -rf
|
||||
echo "[$(D)] Pruned. Kept $(ls -d /backup/soft-serve_* 2>/dev/null | wc -l) backups (max 14)"
|
||||
@@ -1,6 +1,6 @@
|
||||
services:
|
||||
vaultwarden:
|
||||
image: vaultwarden/server:1.37.1
|
||||
image: vaultwarden/server:1.37.2
|
||||
container_name: vaultwarden
|
||||
restart: unless-stopped
|
||||
env_file: ".env"
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Plane CE 加固草稿(docs/plane-hardening/)
|
||||
|
||||
> **状态:草稿,未应用、未提交。** 对应追踪:Plane vps 项目条目(2026-09-03,**记录源**;Linear W1N-277 已取消,Linear 自 2026-09-03 起不再作为记录源)。
|
||||
> 线上实例:`plane.chans.xyz`(synapse K3s,ns `plane`,release `plane-app` = chart `plane-ce-1.8.0` / app `v1.4.1`)。
|
||||
> 依据:2026-09-03 只读核查(13 条审查意见中 11 条属实、#3 基本属实、#9 指标归属错误)+ 上游 chart 模板逐条核对。
|
||||
|
||||
## 文件
|
||||
|
||||
| 文件 | 内容 |
|
||||
|------|------|
|
||||
| `values.hardened.yaml` | 可选硬化 values(external secrets 引用、requireExplicitSecrets、minio pin、上传限额对齐);含 HTTP→HTTPS `extraObjects` 示例 |
|
||||
| `secrets.yaml.example` | 6 组外部 Secret 结构占位(只含 key 名,真实值仅存宿主机) |
|
||||
| `backup/plane-backup.yaml` | **PostgreSQL 备份 CronJob**(pg_dump `-Fc`,hostPath `/var/backups/plane`;MinIO 已按实际用量剔除) |
|
||||
| `backup/README.md` | 备份方案说明(排程/容量/保留/还原/阻塞) |
|
||||
|
||||
## 应用顺序(每步先 diff 后执行,全部需用户逐项确认)
|
||||
|
||||
### 现在就值得做:DB 备份(P0,见 backup/)
|
||||
`plane-backup.yaml` 部署 + 手动触发验证一次即可;88 MB 库每日快照几乎零成本。
|
||||
|
||||
### 可选(顺手做一次,不是必须)
|
||||
- **Phase A 密钥外部化**(零行为变化、无停机,约 15 分钟):按 `secrets.yaml.example`
|
||||
在宿主机建 6 个 Secret(值先复制当前集群),用 `values.hardened.yaml` 跑
|
||||
`helm diff upgrade` → `helm upgrade`;验证后删除 chart 生成的旧 Secret。
|
||||
价值:默认密钥不再落在 chart 公开常量上,作为保险。
|
||||
- **MCP API Key 轮换**:若审查对话出过你的环境,Plane 后台重生成 + 更新
|
||||
`/home/windy/plane-k3s/mcp/mcp.env`(0600)+ 重启 Cursor MCP。
|
||||
- **/god-mode IP 白名单**:若在意管理后台被公网爆破。chart 1.8.0 的 IngressRoute
|
||||
不支持给单条路由追加 middleware → 需 post-renderer 或 upgrade 后 `kubectl patch`
|
||||
(升级会覆盖,需固化);源 IP 清单待提供。
|
||||
|
||||
### 明确暂缓/跳过(个人单节点,等出现症状再处理)
|
||||
- SECRET_KEY 等轮换(Phase B):等真要配 SMTP/OAuth 前再做(避免旧密文不可解)。
|
||||
- NetworkPolicy、有状态组件 resources limits(chart 无 values 开关,需 post-render/patch)、
|
||||
HTTP→HTTPS(草稿已给 `extraObjects` 示例)、metrics-server/Sentry。
|
||||
|
||||
## 关键限制(chart 1.8.0 模板已核对)
|
||||
- `external_secrets.*_existingSecret` 设置后,对应 Secret **必须**包含模板所需全部 key
|
||||
(缺失不自动补),见 `secrets.yaml.example` 注释。
|
||||
- `app_keys_existingSecret` 的 envFrom 在所有 workload 上**最后注入**(后置生效),
|
||||
保证 app/live 共享密钥一致——不要在其后再放同名 key 的 Secret。
|
||||
- `DATABASE_URL`/`AMQP_URL`/`REDIS_URL` 是 chart 生成的派生 URL,内嵌明文密码;
|
||||
外部化后轮换 DB/队列密码时必须同步更新 `plane-app-env`。
|
||||
- minio 的 `MINIO_ROOT_*` 与 `AWS_*` 同源于一个 Secret;升级时 bucket Job 会重跑
|
||||
(需 admin 权限凭据)——换 svcacct 前先确认权限覆盖该 Job。
|
||||
@@ -0,0 +1,48 @@
|
||||
# Plane CE 备份方案(DB-only)— DRAFT (2026-09-03), 未应用
|
||||
|
||||
> 关联:`plane-backup.yaml`(CronJob);追踪:Plane vps 项目条目(记录源,2026-09-03 起不用 Linear)。
|
||||
> 现状(实测):pg 全库 **88 MB**(310 issues / 1 user);MinIO uploads **264 KB**(几乎空)。
|
||||
|
||||
## 范围决策(2026-09-03,实际角度)
|
||||
|
||||
- **做:PostgreSQL 逻辑备份** —— 覆盖现实故障(误删、升级失败、磁盘坏、重装),成本≈0。
|
||||
- **不做:MinIO/附件备份** —— 桶仅 264 KB,个人实例附件可接受丢失;不为它付日常维护。
|
||||
日后附件明显变多再按原完整版思路加 `mc mirror`(历史版本见本目录 git 历史/Plane 条目评论)。
|
||||
- 异机同步暂不启用(见下"局限/阻塞")。
|
||||
|
||||
## 方案
|
||||
|
||||
集群内 CronJob(ns `plane`,每天 **01:30 UTC = 03:30 本地**,控制器按 UTC 跑):
|
||||
|
||||
1. 单容器 `postgres:15.7-alpine`:`pg_dump -Fc`(自定义压缩格式)打 `plane` 库
|
||||
→ `/var/backups/plane/pg/plane-<UTC时间戳>.dump`(hostPath `DirectoryOrCreate`)
|
||||
2. 保留 7 天(`find -mtime +7 -delete`),成功/失败历史各留 3/2
|
||||
3. 凭据:现 chart Secret `plane-app-pgdb-secrets`(Phase A 外部化后改 `plane-pgdb-credentials`)
|
||||
|
||||
## 容量
|
||||
|
||||
- 库 88 MB → `-Fc` 快照约 10–40 MB/天 × 7 天 ≈ **<300 MB**,对 83 G 可用盘可忽略。
|
||||
|
||||
## 还原(未演练;应用前先做一次隔离测试)
|
||||
|
||||
```bash
|
||||
# 目标 PG15 实例(临时起一个 postgres:15.7-alpine 容器或另一台机):
|
||||
# 先建空库: createdb plane (user=plane)
|
||||
pg_restore -h <target> -U plane -d plane --clean --if-exists /var/backups/plane/pg/plane-<TS>.dump
|
||||
# 还原后确认 310 issues 量级一致;附件为空属预期(未备份 MinIO)
|
||||
```
|
||||
|
||||
## 验收(应用前逐项过)
|
||||
|
||||
- [ ] CronJob 建立后手动触发一次:`kubectl -n plane create job --from=cronjob/plane-backup plane-backup-manual-1`,Job `Completed`
|
||||
- [ ] `/var/backups/plane/pg/plane-*.dump` 可被 `pg_restore -l` 列出
|
||||
- [ ] 备份 Job 只依赖 pgdb 服务,不依赖 Plane 应用 Pod(应用故障期间也能出备份)
|
||||
- [ ] 保留清理 dry-run(`find ... -print`)正确;`df -h /` 前后对比记录
|
||||
|
||||
## 局限 / 阻塞
|
||||
|
||||
- **本地方案不是离机备份**:单节点磁盘/整机故障即丢。如日后要离机,纳入
|
||||
[Restic 异机 repository 决策与存取隔离](https://plane.chans.xyz/space/projects/56874283-7e1d-43a8-afa4-631cf1c4ad5b/issues/7825d564-ae15-446b-bced-be26b648346b/)
|
||||
(与 Matrix 备份同一决策);恢复演练纪律见
|
||||
[服务级 restore runbook 与隔离复元演练](https://plane.chans.xyz/space/projects/56874283-7e1d-43a8-afa4-631cf1c4ad5b/issues/a9bea3ba-c958-4a74-b2f1-6bbb653f21d3/)。
|
||||
- 提醒:同一节点 **Matrix 数据价值远高于 Plane 且同样无备份** —— 若投入备份精力,顺序上 Matrix 优先。
|
||||
@@ -0,0 +1,63 @@
|
||||
# Plane CE PostgreSQL backup CronJob — DRAFT (2026-09-03), NOT applied.
|
||||
# ns: plane (synapse K3s single node). Output: hostPath /var/backups/plane (root disk, auto-created).
|
||||
#
|
||||
# Scope decision (2026-09-03, practical): DB-only. MinIO dropped — uploads bucket
|
||||
# measured at 264 KB / 444 KB total; attachments are acceptable loss for this
|
||||
# personal 1-user instance (310 issues / 88 MB DB). Revisit only if usage grows.
|
||||
#
|
||||
# Credentials: read from the CURRENT chart-generated Secret (works today). After the
|
||||
# optional external-secrets migration (docs/plane-hardening/README.md Phase A) switch
|
||||
# the secretKeyRef name to plane-pgdb-credentials.
|
||||
#
|
||||
# Apply:
|
||||
# ssh windy@synapse.chans.xyz 'sudo k3s kubectl apply -n plane -f -' < plane-backup.yaml
|
||||
# Manual run + verify:
|
||||
# sudo k3s kubectl -n plane create job --from=cronjob/plane-backup plane-backup-manual-1
|
||||
# sudo k3s kubectl -n plane get cronjob,job,pods | grep plane-backup
|
||||
# sudo ls -lh /var/backups/plane/pg
|
||||
# Restore steps + tuning: see backup/README.md
|
||||
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: plane-backup
|
||||
namespace: plane
|
||||
spec:
|
||||
# 01:30 UTC daily = 03:30 local (CEST). CronJob controller runs in UTC.
|
||||
schedule: "30 1 * * *"
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 2
|
||||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
volumes:
|
||||
- name: backup
|
||||
hostPath:
|
||||
path: /var/backups/plane
|
||||
type: DirectoryOrCreate
|
||||
containers:
|
||||
- name: pg-dump
|
||||
image: postgres:15.7-alpine
|
||||
env:
|
||||
- name: PGPASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plane-app-pgdb-secrets # -> plane-pgdb-credentials after Phase A
|
||||
key: POSTGRES_PASSWORD
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -euo pipefail
|
||||
TS=$(date -u +%Y%m%dT%H%M%SZ)
|
||||
mkdir -p /backup/pg
|
||||
pg_dump -h plane-app-pgdb.plane.svc.cluster.local -U plane -d plane \
|
||||
-Fc -f "/backup/pg/plane-${TS}.dump"
|
||||
find /backup/pg -type f -name 'plane-*.dump' -mtime +7 -delete
|
||||
echo "pg_dump done: /backup/pg/plane-${TS}.dump ($(du -h /backup/pg/plane-${TS}.dump | cut -f1))"
|
||||
volumeMounts:
|
||||
- name: backup
|
||||
mountPath: /backup
|
||||
@@ -0,0 +1,91 @@
|
||||
# External Secret structure for Plane CE hardening — EXAMPLE ONLY.
|
||||
# No real values here; this file is safe to commit. Real values live only on the
|
||||
# host (/home/windy/plane-k3s, 0600/0700) and in the cluster.
|
||||
#
|
||||
# Phase A — create each Secret with the CURRENT cluster values first (zero change):
|
||||
# # current source Secrets (chart-generated):
|
||||
# kubectl -n plane get secret plane-app-app-secrets -o jsonpath='{.data.SECRET_KEY}' | base64 -d
|
||||
# kubectl -n plane get secret plane-app-live-secrets -o jsonpath='{.data.REDIS_URL}' | base64 -d
|
||||
# kubectl -n plane get secret plane-app-pgdb-secrets -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d
|
||||
# kubectl -n plane get secret plane-app-rabbitmq-secrets -o jsonpath='{.data.RABBITMQ_DEFAULT_PASS}' | base64 -d
|
||||
# kubectl -n plane get secret plane-app-doc-store-secrets -o jsonpath='{.data}' | base64 -d
|
||||
#
|
||||
# e.g. kubectl -n plane create secret generic plane-app-keys \
|
||||
# --from-literal=SECRET_KEY="$(<copy from above>)" \
|
||||
# --from-literal=LIVE_SERVER_SECRET_KEY="$(<copy from above>)"
|
||||
#
|
||||
# All keys below are REQUIRED by chart templates/plane-ce-1.8.0 (verified 2026-09-03):
|
||||
# missing keys are NOT auto-filled once an existingSecret is referenced.
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plane-app-keys # external_secrets.app_keys_existingSecret
|
||||
namespace: plane
|
||||
type: Opaque
|
||||
stringData:
|
||||
SECRET_KEY: "" # current: copy from plane-app-app-secrets; rotate only in Phase B
|
||||
LIVE_SERVER_SECRET_KEY: "" # current: same value as above / plane-app-live-secrets
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plane-app-env # external_secrets.app_env_existingSecret
|
||||
namespace: plane
|
||||
type: Opaque
|
||||
stringData:
|
||||
REDIS_URL: "" # redis://plane-app-redis.plane.svc.cluster.local:6379/
|
||||
DATABASE_URL: "" # postgresql://plane:plane@plane-app-pgdb.plane.svc.cluster.local/plane
|
||||
AMQP_URL: "" # amqp://plane:plane@plane-app-rabbitmq.plane.svc.cluster.local/
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plane-live-env # external_secrets.live_env_existingSecret
|
||||
namespace: plane
|
||||
type: Opaque
|
||||
stringData:
|
||||
REDIS_URL: "" # redis://plane-app-redis.plane.svc.cluster.local:6379/
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plane-pgdb-credentials # external_secrets.pgdb_existingSecret
|
||||
namespace: plane
|
||||
type: Opaque
|
||||
stringData:
|
||||
POSTGRES_PASSWORD: "" # Phase A: keep current ('plane'); Phase B: ALTER USER first, then sync
|
||||
POSTGRES_DB: "plane"
|
||||
POSTGRES_USER: "plane"
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plane-rabbitmq-credentials # external_secrets.rabbitmq_existingSecret
|
||||
namespace: plane
|
||||
type: Opaque
|
||||
stringData:
|
||||
RABBITMQ_DEFAULT_USER: "plane"
|
||||
RABBITMQ_DEFAULT_PASS: "" # Phase A: keep current; Phase B: rabbitmqctl change_password first
|
||||
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plane-minio-credentials # external_secrets.doc_store_existingSecret
|
||||
namespace: plane
|
||||
type: Opaque
|
||||
stringData:
|
||||
FILE_SIZE_LIMIT: "20971520" # must match env.doc_upload_size_limit
|
||||
AWS_S3_BUCKET_NAME: "uploads"
|
||||
USE_MINIO: "1"
|
||||
MINIO_ROOT_USER: "admin"
|
||||
MINIO_ROOT_PASSWORD: "" # root creds take effect on first init only
|
||||
AWS_ACCESS_KEY_ID: "admin"
|
||||
AWS_SECRET_ACCESS_KEY: "" # == MINIO_ROOT_PASSWORD while minio.local_setup
|
||||
AWS_S3_ENDPOINT_URL: "http://plane-app-minio:9000"
|
||||
@@ -0,0 +1,109 @@
|
||||
# Plane CE hardened values — DRAFT (2026-09-03), NOT applied.
|
||||
# Target file on host: /home/windy/plane-k3s/values.yaml (synapse.chans.xyz)
|
||||
# Reference release: plane-app, chart plane-ce-1.8.0 (values.yaml L1-362 + templates verified 2026-09-03).
|
||||
# No secrets in this file. Secret *values* live only in k8s Secrets (see secrets.yaml.example).
|
||||
#
|
||||
# Two phases:
|
||||
# Phase A: externalize secrets (reference names below) with CURRENT values copied -> zero change.
|
||||
# Phase B: rotate credentials one by one (see README.md). SECRET_KEY rotation is cheap only while
|
||||
# SMTP/OAuth are unconfigured (no encrypted config rows yet).
|
||||
|
||||
planeVersion: v1.4.1
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
appHost: plane.chans.xyz
|
||||
ingressClass: traefik
|
||||
traefik:
|
||||
# 20 MiB (chart default). Keep aligned with env.doc_upload_size_limit below.
|
||||
maxRequestBodyBytes: 20971520
|
||||
|
||||
ssl:
|
||||
createIssuer: true
|
||||
issuer: http # HTTP-01; ssl_token_existingSecret not needed
|
||||
email: admin@chans.xyz
|
||||
generateCerts: true
|
||||
|
||||
postgres:
|
||||
storageClass: local-path
|
||||
volumeSize: 5Gi
|
||||
# NOTE: chart 1.8.0 exposes NO resources knob for the bundled datastores
|
||||
# (stateful templates render no resources block). Add limits via
|
||||
# --post-renderer/kustomize or `kubectl -n plane patch sts ...` re-applied on
|
||||
# every upgrade (P2 task; see README.md).
|
||||
|
||||
redis:
|
||||
storageClass: local-path
|
||||
# image: valkey/valkey:7.2.11-alpine # already pinned by chart default; uncomment to make explicit
|
||||
|
||||
minio:
|
||||
# P2: pin. Digest of the currently running :latest (2026-09-03, pod plane-app-minio-wl-0).
|
||||
image: minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e
|
||||
# image_mc: minio/mc@sha256:... # optional: pin one-shot bucket-init client the same way
|
||||
storageClass: local-path
|
||||
volumeSize: 5Gi
|
||||
|
||||
rabbitmq:
|
||||
storageClass: local-path
|
||||
|
||||
env:
|
||||
# Fail the render instead of ever falling back to the chart's PUBLIC constants
|
||||
# (values.yaml L340-341 in chart 1.8.0). Requires external_secrets below.
|
||||
requireExplicitSecrets: true
|
||||
|
||||
# SECRET_KEY / LIVE_SERVER_SECRET_KEY are deliberately OMITTED here.
|
||||
# They live in k8s Secret `plane-app-keys` (referenced below). With
|
||||
# requireExplicitSecrets=true and app_keys_existingSecret set, the chart renders
|
||||
# neither key itself and app+live workloads both envFrom `plane-app-keys` LAST
|
||||
# (later envFrom wins), which keeps the shared signing key consistent.
|
||||
|
||||
pgdb_name: plane
|
||||
docstore_bucket: uploads
|
||||
# Align app-side upload cap with the Traefik body limit (was 5242880/5MiB).
|
||||
# Keep both at 20MiB, or lower both together.
|
||||
doc_upload_size_limit: "20971520"
|
||||
|
||||
external_secrets:
|
||||
# Shared signing keys (used by app + live). REQUIRED keys: SECRET_KEY, LIVE_SERVER_SECRET_KEY.
|
||||
app_keys_existingSecret: plane-app-keys
|
||||
# REQUIRED keys: REDIS_URL, DATABASE_URL, AMQP_URL (chart-derived URLs; update on DB/queue rotation).
|
||||
app_env_existingSecret: plane-app-env
|
||||
# REQUIRED keys: REDIS_URL.
|
||||
live_env_existingSecret: plane-live-env
|
||||
# REQUIRED keys: POSTGRES_PASSWORD, POSTGRES_DB, POSTGRES_USER.
|
||||
pgdb_existingSecret: plane-pgdb-credentials
|
||||
# REQUIRED keys: RABBITMQ_DEFAULT_USER, RABBITMQ_DEFAULT_PASS.
|
||||
rabbitmq_existingSecret: plane-rabbitmq-credentials
|
||||
# REQUIRED keys: FILE_SIZE_LIMIT, AWS_S3_BUCKET_NAME, USE_MINIO, MINIO_ROOT_USER,
|
||||
# MINIO_ROOT_PASSWORD, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_S3_ENDPOINT_URL.
|
||||
doc_store_existingSecret: plane-minio-credentials
|
||||
# ssl_token_existingSecret: '' # DNS-01 only (cloudflare/digitalocean); unused with HTTP-01
|
||||
|
||||
# Optional, P2: HTTP -> HTTPS 301. The chart's own IngressRoute binds only
|
||||
# 'websecure' (http:// currently 404s). extraObjects is rendered verbatim (toYaml).
|
||||
# Uncomment and `helm upgrade` once reviewed:
|
||||
# extraObjects:
|
||||
# - apiVersion: traefik.io/v1alpha1
|
||||
# kind: Middleware
|
||||
# metadata:
|
||||
# name: plane-https-redirect
|
||||
# namespace: plane
|
||||
# spec:
|
||||
# redirectScheme:
|
||||
# scheme: https
|
||||
# permanent: true
|
||||
# - apiVersion: traefik.io/v1alpha1
|
||||
# kind: IngressRoute
|
||||
# metadata:
|
||||
# name: plane-http-to-https
|
||||
# namespace: plane
|
||||
# spec:
|
||||
# entryPoints: [web]
|
||||
# routes:
|
||||
# - match: Host(`plane.chans.xyz`)
|
||||
# kind: Rule
|
||||
# middlewares:
|
||||
# - name: plane-https-redirect
|
||||
# services:
|
||||
# - name: plane-app-web
|
||||
# port: 3000
|
||||
@@ -60,6 +60,14 @@ OpenClash runs `/etc/openclash/clash` (clash_meta core) with configuration
|
||||
> AAAA now `2404:6800…` (new upstream answering, was `2607:f8b0…` via hk2),
|
||||
> taobao/intercept/clash-fake-ip all unchanged. Backup:
|
||||
> `config.yaml.bak-multi-doh-20260813-105421`.
|
||||
> 2026-09-01: removed `dns.quad9.net` from `foreign_upstream` — recurring
|
||||
> `WARN foreign_upstream … unexpected EOF` bursts (481 log entries) against
|
||||
> Quad9 DoH; endpoint answers on probe but gets intermittently
|
||||
> connection-reset from this network (same failure class as the excluded
|
||||
> `dns.quad101.net`). Remaining upstreams `adg.chans.xyz` (hk2) +
|
||||
> `dns.cloudflare.com` both verified live; google.com A + youtube.com AAAA
|
||||
> resolve through mosdns :6052 after restart. Backup:
|
||||
> `config.yaml.bak-quad9-remove-20260901-201801`.
|
||||
- nft: OpenClash injects TPROXY/redirect + DNS-hijack rules into
|
||||
`table inet fw4`; a residual `table inet passwall` exists with 0 packets (unused)
|
||||
|
||||
|
||||
+346
-9
@@ -228,6 +228,102 @@ cost/ladder sensors can stay `unknown` because CSG
|
||||
dashboard uses template ladder/cost entities instead. Do not change
|
||||
`templates/csg_sensors.yaml` or the 电力监控 dashboard for an install.
|
||||
|
||||
**`templates/csg_sensors.yaml` hardened 2026-08-29 (W1N-239):** added
|
||||
`availability` templates to all 12 `csg_*` sensors (numeric sensors can't
|
||||
render `unknown`/`unavailable` in `state`; availability suppresses
|
||||
rendering instead — native CSG down ⇒ derived sensors show `unavailable`,
|
||||
no more fake zeros / "一档" / `0%`). `csg_yesterday_kwh` now falls back to
|
||||
`last_month_by_day`'s last entry when `this_month_by_day` is empty (month
|
||||
start); ladder constants (`t1/t2/p1/p2/p3`) deduped into per-block
|
||||
`variables:` (Block B + Block D); `csg_mom_change` parses `date`
|
||||
defensively. Backup:
|
||||
`/homeassistant/.csg-backups/csg_sensors.yaml.bak-20260829-w1n239`.
|
||||
**Verified:** `ha core check` OK; Core restart required (trigger-based
|
||||
template blocks don't settle on `template.reload` — W1N-114 precedent);
|
||||
post-restart all 12 entities numeric & consistent (302.47 kWh→180.28 元,
|
||||
324.03 kWh→194.06 元, mom_change -3.6%, yesterday 7.66 kWh/2026-08-28),
|
||||
no template errors in Core logs.
|
||||
|
||||
**`csg_sensors.yaml` off-by-one fixed 2026-08-29 (W1N-241):** CSG data
|
||||
lags 1 day (`sum(this_month_by_day)` == `this_month_total_usage`, data
|
||||
stops at yesterday), but templates used `now().day` as "days elapsed" →
|
||||
`csg_predicted_usage` underestimated ~1 daily avg (~3%) and
|
||||
`csg_mom_change` compared this-month 28 days vs last-month 29 days
|
||||
(-3.6% vs true -0.3%). Both now derive the day number from
|
||||
`this_month_by_day[-1].date` (fallback `now().day` when empty). Added
|
||||
`sensor.csg_this_month_daily_avg` (month-to-date avg, 302.47/28=10.8) and
|
||||
`sensor.csg_prediction_progress` (usage/predicted %, 90.3) in Block C
|
||||
(trigger adds `csg_predicted_usage`). Backup:
|
||||
`/homeassistant/.csg-backups/csg_sensors.yaml.bak-20260829-w1n241`.
|
||||
**Verified (8/29):** predicted 324.03→334.81, mom_change -3.6→-0.3,
|
||||
daily_avg 10.8, progress 90.3, predicted_cost 194.06→200.94 (334.81 kWh
|
||||
ladder), ladder cost 180.28 unchanged, `ha core check` OK after restart,
|
||||
no template errors; 14 csg_* entities total.
|
||||
|
||||
**电力监控面板(`lovelace.dashboard_unknown` / view `power-monitor`)
|
||||
updated 2026-08-29 (W1N-240 + W1N-242):** 「本月累计」gauge 对齐夏季阶梯:
|
||||
`max:650`、segments `0/260/600`(绿/橙/红 = 一/二/三档;冬季 11-01 需切
|
||||
`max:450`、`0/200/400` — **seasonal switch point**,见下文)。「📊 统计
|
||||
数据」卡新增本年/去年 4 行(原生传感器,口径标注「电费(账单)」、本年
|
||||
「(至今)」)+ 本月日均/预测进度 2 行(`csg_this_month_daily_avg` /
|
||||
`csg_prediction_progress`,W1N-242);面板共引用 **20** 个实体。改前备份:
|
||||
`/homeassistant/.lovelace-backups/dashboard-unknown-power-monitor-20260829-204845.json`
|
||||
(W1N-240)、`-20260829-210708.json`(W1N-242)
|
||||
(改法:WS `lovelace/config/save`,参数 `url_path: dashboard-unknown` +
|
||||
`config`;勿直改 `.storage/`)。验证:WS 读回 18→20 实体 diff ✓、gauge
|
||||
配置一致 ✓、URL `http://hass.windy.lan:8123/dashboard-unknown/power-monitor`。
|
||||
|
||||
**`csg_sensors.yaml` W1N-242:** `csg_predicted_usage` /
|
||||
`csg_mom_change` / `csg_this_month_daily_avg` 三处取 `days[-1]` 前补
|
||||
`sort(attribute='date')`(与 `csg_yesterday_kwh` 一致,防上游乱序取错
|
||||
数据日)。备份 `csg_sensors.yaml.bak-20260829-w1n242`。验证:Core
|
||||
restart 后回归值不变(334.81 / -0.3 / 10.8 / 90.3 / 200.94 / 180.28)。
|
||||
|
||||
**CSG 面板重构 2026-09-04(VPS-90,先核对计价后展示层改动):** 核对
|
||||
`power-monitor` 计价与 8 月账单一致(198.65 vs 账单 198.64,差 ≤0.01 元,
|
||||
因模板用公众圆整价 0.589/0.639/0.889、账单用 6 位精确价),不改阶梯常量。
|
||||
改动:① `csg_sensors.yaml` Block B 新增
|
||||
`sensor.csg_this_month_avg_price`(本月阶梯电费÷本月用电,`元/kWh`,
|
||||
availability 照 W1N-239 惯例;**csg_* 实体 14→15**);② 面板改名「环比上月」
|
||||
→「环比上月同期」;glance「本月/上月」grid 去重为单卡「上月」(本月用电/电费
|
||||
行归 💰核心数据卡);⚡阶梯电价卡加「本月实际均价」行(当前档位/当前电价/
|
||||
本月实际均价/档位剩余;面板唯一实体引用 20→21);③ `automations.yaml` 加
|
||||
2 条提醒:`automation.csg_mian_ban_qie_dong_ji_dang_ti_xing`(10-25 09:00)
|
||||
与 `automation.csg_mian_ban_qie_xia_ji_dang_ti_xing`(4-25 09:00)经
|
||||
`matrix_e2ee.send_message` 提醒切 gauge。④ 金额单位混排(原生 CNY vs 模板
|
||||
元)**维持**:`config/entity_registry/update` 拒绝自定义文本单位
|
||||
(`extra keys not allowed … Got '元'`),用户确认接受。备份:
|
||||
`.lovelace-backups/dashboard-unknown-power-monitor-20260904-204757-pre-refactor.json`、
|
||||
`.csg-backups/csg_sensors.yaml.bak-20260904-204757-pre-refactor`(及
|
||||
`-205301-pre-avgprice`)、`.automations-backups/automations.yaml.bak-*`。
|
||||
**WS 改法(2026.8,本机实测)**:core/主机 python 无 ws 库、core 容器内经
|
||||
supervisor 代理 WS 被拒(loop prevention),用
|
||||
`docker run --rm --network host -e SUPERVISOR_TOKEN`(supervisor 镜像
|
||||
`aarch64-hassio-supervisor:2026.08.0`)连 `ws://172.30.32.2/core/websocket`;
|
||||
命令名 `lovelace/config`(读)+ `lovelace/config/save`(写),
|
||||
`lovelace/config/get` 已不存在(unknown_command)。验证:新实体
|
||||
0.589 元/kWh、15 个 csg_* 数值齐全、回归值不变(14.09/198.65/331.22/
|
||||
304.99/181.89)、automations on、`ha core check` OK、日志无 template 错误。
|
||||
|
||||
**CSG 长期归档(W1N-243, 2026-08-29):** scribe 库新增 `csg_history`
|
||||
表(逐日 usage/cost/ladder/balance + 逐月累计;2026-07-01 起回填,永久),
|
||||
由 TimescaleDB 每日任务 **1008** `csg_daily_snapshot()`(22:30
|
||||
Asia/Shanghai;**TS job 非 pg_cron**)upsert 维护。日费用在原生
|
||||
`latest_day_cost` 缺失时回退 = 昨日用电 × 当前档费率(模板
|
||||
`csg_current_ladder_tariff` 0.639);月费用回退模板
|
||||
`csg_this_month_ladder_cost`。**语义**:day 行 usage/cost 为该日值,
|
||||
ladder/balance 为 22:30 快照值。详见 [hosts/pgdb.md](../hosts/pgdb.md)。
|
||||
|
||||
> **Seasonal gauge switch (W1N-240 已知事项):** 每年 **11-01** 把
|
||||
> `power-monitor` 视图「本月累计」gauge 切到冬季 `max:450` /
|
||||
> `0/200/400`,**5-01** 切回夏季 `max:650` / `0/260/600`(与模板
|
||||
> `now().month` 季节逻辑对齐;模板常量在 Block B/D `variables`)。
|
||||
> **提醒 automation(2026-09-04 起,VPS-90):**
|
||||
> `automation.csg_mian_ban_qie_dong_ji_dang_ti_xing`(10-25)与
|
||||
> `automation.csg_mian_ban_qie_xia_ji_dang_ti_xing`(4-25)09:00 经
|
||||
> `matrix_e2ee.send_message` 发操作步骤提醒;gauge 的 max/segments 无法
|
||||
> 模板化,仍需人工改卡配置。
|
||||
|
||||
Home PPPoE IPv4 to CSG is still blackholed (`curl -4` to `218.19.148.218:443`
|
||||
times out). `end0` IPv6 is enabled (`ipv6.method: auto`); from HA,
|
||||
`curl -6 https://95598.csg.cn` returns HTTP 200 via `240e:f9:8060::1:16`.
|
||||
@@ -319,12 +415,17 @@ deployed 2026-08-18 from `216cc99` (backup
|
||||
reworked in v0.3.1/v0.3.2 to wait for a peer-initiated inbound SAS from
|
||||
Element with emoji comparison), see
|
||||
[docs/home-assistant-matrix.md § Device verification](../docs/home-assistant-matrix.md).
|
||||
### Scribe long-term history (verified 2026-08-29)
|
||||
### Scribe long-term history (3.8.0 setup 2026-08-29; 4.4.0 verified 2026-09-13)
|
||||
|
||||
- **Scribe 3.8.0** (`/homeassistant/custom_components/scribe/`), configured from
|
||||
`configuration.yaml` (`scribe:` block; config entry
|
||||
- **Scribe 4.4.0** (`/homeassistant/custom_components/scribe/`, HACS repo
|
||||
`jonathan-gtd/scribe`, = latest stable 2026-09-12; upgraded 2026-09-13 together
|
||||
with Core 2026.9.1 / HAOS 18.2), configured from
|
||||
`/homeassistant/scribe.yaml` — W1N-238 moved the block out of
|
||||
`configuration.yaml` on 2026-08-29 (main config now carries
|
||||
`scribe: !include scribe.yaml`; content moved verbatim; backup
|
||||
`configuration.yaml.bak-20260829-201724-w1n238`). Config entry
|
||||
`01KC2VFJWEQ3XDHY6TQKHPDVRB`, `source: import` — UI "Configure → Advanced"
|
||||
edits are overridden by the YAML on restart; treat YAML as authoritative).
|
||||
edits are overridden by the YAML on restart; treat YAML as authoritative.
|
||||
- TimescaleDB at `192.168.55.15:5432/scribe` (DB user `hass`; host in inventory,
|
||||
see [hosts/pgdb.md](../hosts/pgdb.md)). Database re-initialized 2026-08-29 14:06 CST
|
||||
(user-handled; earlier `relation "entities" does not exist` errors resolved).
|
||||
@@ -342,11 +443,166 @@ deployed 2026-08-18 from `216cc99` (backup
|
||||
entities (`scribe_states_written`, `scribe_events_written`, rates, sizes).
|
||||
- Verified post-restart 14:23 CST: writer started, `scribe_events_written=1`
|
||||
(homeassistant_start), states ~110/min, buffer 3, no scribe log errors.
|
||||
- **Scribe 3.8.0 has no retention option** (upstream feature, newer versions
|
||||
only). Retention requires upgrading Scribe first; the DB is a fresh 3.x-schema
|
||||
so an upgrade is safe. Do not expect retention YAML keys to validate on 3.8.0.
|
||||
- Recorder stays external-Postgres with `purge_keep_days: 30` for daily UI
|
||||
history; Scribe is the permanent archive.
|
||||
- **4.x upgrade核对 2026-09-13(只读 + 一处配置变更)**:live `manifest.json` =
|
||||
4.4.0。两个 4.0 breaking change 在本机都不需要动作——数据库是 3.x 结构
|
||||
(`states_raw` PK `(metadata_id, time)` 在,4.2.0 的启动态去重因此可用),
|
||||
TimescaleDB 2.29.2 已装。4.1.0 修了 `db_url` 优先级,YAML 里的
|
||||
`!secret scribe_url` 现在是权威。`scribe.yaml` 现有键在 4.4.0 全部仍然合法
|
||||
(未知键被忽略,`extra=vol.ALLOW_EXTRA`)。**配置优先级 YAML > entry
|
||||
`options` > entry `data` > 默认值**,而 `_resolve_settings` 读的是
|
||||
`hass.data[DOMAIN]["yaml_config"]`(只有 `async_setup` 会写),所以
|
||||
**YAML 改动必须重启 Core,reload config entry 不重读 YAML。**
|
||||
- **`stats_io_interval: 300`(2026-09-13 添加**,备份
|
||||
`/homeassistant/scribe.yaml.bak-20260913-191558`)。4.4.0 不再让 HA 每 30s
|
||||
轮询 I/O 统计传感器,改由集成自己每 60s 发布,间隔成为配置项。scribe 自己的
|
||||
传感器此前是本机自写历史的主要来源(变更前 24h:11 019 / 87 461 行状态 =
|
||||
12.6%),60s → 300s 把这部分降约 5 倍(每个 I/O 传感器约 1440 → 288 行/天)。
|
||||
验证:`ha core check` OK;重启 88s;`ScribeWriter started successfully`;
|
||||
无 scribe error/warning;scribe Repairs 问题 0 条;传感器发布间隔实测正好
|
||||
300s(11:19:26 → 11:24:26 UTC)。
|
||||
- **Retention 现在可用但刻意不设**:`retention_states` / `retention_events`
|
||||
(4.0.0)按间隔丢 chunk,留空 = 永久保留,符合本机定位(Scribe 是永久归档,
|
||||
recorder 保留 365 天)。注意 retention 是**绕过** entry `data` 副本读取的
|
||||
(`from_entry_data=False`),所以删掉 YAML 行即撤销策略。`db_schema`、
|
||||
`enable_rollups`、`scribe.purge` 同样未用:图表走 `sensor_minute` +
|
||||
`timescale_database_reader`(见 [hosts/pgdb.md](pgdb.md)),不吃 scribe 自己的
|
||||
视图,配置里也没有任何 `scribe.query` 调用。`flush_interval` 仍是 entry
|
||||
`data` 钉住的 5s——上游下一个版本把默认改成 30s,但 entry 值优先,要采用只能
|
||||
在 YAML 显式写 `flush_interval: 30`。
|
||||
- Recorder stays external-Postgres with `purge_keep_days: 365` (W1N-243,
|
||||
2026-08-29, raised from 30 — ~300 MB/yr, 1% of the 30G pgdb disk) for
|
||||
native UI per-change history; Scribe is the permanent archive. Long-term
|
||||
statistics stay permanent (not purged by `purge_keep_days`). Note:
|
||||
extending retention does **not** recover pre-2026-08-29 raw history
|
||||
(already purged); only `csg_history` day/month values cover that period.
|
||||
|
||||
### Config layout: scribe.yaml + templates/ merge (W1N-238, verified 2026-08-29)
|
||||
|
||||
- `configuration.yaml` line 29: `scribe: !include scribe.yaml`; line 9:
|
||||
`template: !include_dir_merge_list templates`. No `packages/`.
|
||||
- `scribe.yaml` (config root): the Scribe block, content identical to the
|
||||
former inline one; import semantics unchanged.
|
||||
- `templates/`: `csg_sensors.yaml` (12 template sensors, top-level **list**)
|
||||
+ `quick_sensors.yaml` (scaffold for Quick-derived `quick_*` sensors, empty
|
||||
list with convention header). **`!include_dir_merge_list` merges per-file
|
||||
lists; non-list files are silently skipped** — every file in `templates/`
|
||||
must be a top-level list (`- sensor:` blocks). Directory include only picks
|
||||
up `*.yaml`, so the `.bak` / `.pre-*` backups in the dir are ignored. After
|
||||
adding sensors, verify template-platform entity count = 12 + N (entity
|
||||
registry `platform: template`).
|
||||
- Convention (per review + W1N-233): pure sums/averages stay min_max helpers
|
||||
(e.g. `sensor.dang_qian_zong_gong_lu`); only template-logic derivations
|
||||
(ladder pricing, cross-entity conditions) go into `quick_sensors.yaml`.
|
||||
- Post-change verification 20:19 CST: `ha core check` ok, 92 s restart
|
||||
(2026.8.3), `binary_sensor.scribe_database_connection` on,
|
||||
`scribe_states_written` 18581→19426 growing, template entities still 12,
|
||||
csg sensors numeric, no scribe/template log errors.
|
||||
|
||||
### Timescale Plotly card + database reader (verified 2026-08-29)
|
||||
|
||||
Chart stack over the Scribe TimescaleDB archive. Upstream pair (no HACS;
|
||||
manual copies): reader `remmob/timescale_database_reader` **v1.1.0** (main
|
||||
`bb8776a`) + card `remmob/timescale-plotly-card` **2.2.0** (main `217961d`).
|
||||
|
||||
- **Reader integration**: `/homeassistant/custom_components/timescale_database_reader/`.
|
||||
Config entry `01M165P77QT1FQEAVPNZHDT82W` ("Scribe", `source: user`): connects
|
||||
`hass@192.168.55.15:5432/scribe` (credentials = `secrets.yaml` `scribe_url`),
|
||||
`table: sensor_minute`. Exposes no entities/services — it serves WS command
|
||||
`timescale/query` (window ≤ 365 d, ≤ 50 000 rows, `downsample` bucket seconds).
|
||||
Benign startup warning `Error executing test query: column "time" does not
|
||||
exist`: the self-test SQL assumes the LTSS column name; the scribe table uses
|
||||
`minute` — real queries work (verified: 70 rows for a live power sensor).
|
||||
- **Card**: `/homeassistant/www/community/timescale-plotly-card/timescale-plotly-card.js`
|
||||
(root-owned, same convention as HACS dirs). Lovelace resource (storage)
|
||||
id `2e360d17b5aa4ce59c2fd13c43b51215` →
|
||||
`/hacsfiles/timescale-plotly-card/timescale-plotly-card.js`, type `module`.
|
||||
Card config matches the entry by `database: scribe` (name from the reader
|
||||
entry). Updates: replace the file, resource URL unchanged — browsers need a
|
||||
hard refresh or a bumped `?v=` query on the resource URL.
|
||||
- **pgdb side** (`sensor_minute_aggregate` cagg + `sensor_minute` hypertable +
|
||||
every-minute refresh job): see [hosts/pgdb.md](pgdb.md) § Databases.
|
||||
- **Agent-side HA WebSocket without a long-lived token** (verified 2026-08-29):
|
||||
connect `ws://supervisor/core/websocket` with header
|
||||
`Authorization: Bearer $SUPERVISOR_TOKEN`, then send
|
||||
`{"type":"auth","access_token":"$SUPERVISOR_TOKEN"}` — the Supervisor proxy
|
||||
swaps it for a core token (works as the internal Supervisor admin user). Note
|
||||
`lovelace/resources/create` in HA 2026.8 takes `res_type` (NOT
|
||||
`resource_type`).
|
||||
- Scribe stores numeric sensor values in `states_raw.value` with `state` NULL,
|
||||
so `sensor_minute.state` shows `'0'` for numeric sensors; the card plots
|
||||
`avg_state` (from `value`) — expected, not a bug.
|
||||
- **Quick 仪表盘(`dashboard-quick`)图表套件**(2026-08-29 创建,经 WS
|
||||
`lovelace/config/save` 写入;W1N-230 修复 + W1N-231 round-2 改进):
|
||||
5 张 timescale 卡——大功率电器/常驻负载功率(按量级拆图,避免尖峰压扁
|
||||
<70 W 基线)、按插座用电量(`energy_mode` + cumulative/diff,数据质量前提
|
||||
见 pgdb 的 refresh 过程补丁)、室内外温湿度(温度左轴/湿度右轴,4 位置同色
|
||||
配对)、人体感应活动状态(3 个 `motion_state`,banded `state_map`
|
||||
none/small/medium/large → 0-11,per-entity `line_color` 红/蓝/绿)。
|
||||
空调实体引用为 `kong_diao_*`(`kong_tiao` 是笔误,W1N-230 修复;`grep -c
|
||||
kong_tiao` 应为 0)。灯区:2×2 嵌套 grid(`grid_options: {columns: "full"}`,
|
||||
内层 `columns: 2`)+ 4 卡统一 `mushroom-light-card`(显式 name、
|
||||
`use_light_color: false`、内联亮度/色温控制),heading icon
|
||||
`mdi:lightbulb-group`。heading badges:环境 4 温度(迷你/mini数显/数显/广州)、
|
||||
大功率电器 空调/电脑当前功率、常驻负载 总功率
|
||||
(`sensor.dang_qian_zong_gong_lu`,min_max **sum** helper,`round_digits: 0`,
|
||||
任一源掉线 fail-closed → unknown)。常驻负载图卡级 `fill: 'tozeroy'` +
|
||||
冰箱/主网络 per-entity `fill_color`(线色 20% 透明)+ 其余 5 条 `fill: false`
|
||||
(per-entity fill 逐系列退出,卡 JS `seriesConfig.fill !== false`)。
|
||||
布局:视图 `type: sections` + `max_columns: 4`;灯/用电/环境/人体感应
|
||||
`column_span: 4`,功率两图拆两个 `column_span: 2` 分区**并排**(等高 280px,
|
||||
桌面并排、手机回落堆叠;去卡内 title 省半宽图垂直空间)。
|
||||
**分区/卡片是两套尺寸键,不可混用**:分区宽 = `column_span`
|
||||
(`hui-sections-view.ts` 缺省按 1 列渲染,绝不省略);卡片宽 =
|
||||
`grid_options: {columns: <n|"full">}`(`hui-card.ts` 只读 `config.grid_options`,
|
||||
写在卡片上的 `column_span` 被静默忽略;缺省 12 列,分区内格 = 12 × 分区
|
||||
span,故 span-4 分区里缺省卡片只有 1/4 宽)。
|
||||
修改前备份:`/homeassistant/.lovelace-backups/dashboard-quick-*.json`
|
||||
(W1N-230 修复: `20260829-190256`;round-2 改进: `20260829-194040`)。
|
||||
|
||||
- **Quick 时间范围扩容 (2026-09-13, VPS-92)**: 用户反馈「48 小时不够」。
|
||||
各 timescale 卡可选档上调——大功率电器/常驻负载 `…,24h` → `+3d,7d`;
|
||||
环境 `6h,12h,24h,48h` → `+7d,14d,30d`;人体感应 `…,24h` → `+3d,7d`;
|
||||
用电量(按插座) `energy_time_ranges` `today,week,month,custom` → `+3mo`。
|
||||
**默认档未改**(6h / 6h / today / 24h / 12h)。卡片 JS 只接受
|
||||
`<n>m|<n>h|<n>d`(`parseDurationToMs` 正则 `/^(\d+)(m|h|d)$/`,
|
||||
仅 m/h/d,无 w)与命名档 `today|week|month|3mo|6mo|year|years|custom`;
|
||||
`energy_mode` 卡必须用后者。**数据下界注意**:scribe `sensor_minute`
|
||||
目前最早只到 **2026-08-29**,所以 >15d 的档(14d 边缘、30d 明显)前半段
|
||||
会是空白,等归档继续累积才好看。备份
|
||||
`.lovelace-backups/dashboard-quick-20260913-190912-pre-timerange.json`。
|
||||
|
||||
### 地图仪表盘:CARTO keyed tiles via `custom:map-card` (verified 2026-08-30, W1N-261)
|
||||
|
||||
- **背景:** CARTO 自 2026-08-26 起对无 key 栅格瓦片打 "API KEY REQUIRED"
|
||||
水印,内置地图卡/zone 编辑器全部受影响。Core 2026.8.3 的 `MapCardConfig`
|
||||
**没有任何瓦片配置项**(frontend 20260729.7 源码核对:
|
||||
`setup-leaflet-map.ts` 硬编码 CARTO voyager URL)。上游修复是 2026.9.0b1
|
||||
起改用 OSMF 矢量瓦片(frontend PR #53816),stable 预计 2026-09-02 前后。
|
||||
- **变更:** 「地图」仪表盘(url_path `map`,storage)唯一 map 卡替换为
|
||||
`custom:map-card`([nathan-gs/ha-map-card](https://github.com/nathan-gs/ha-map-card)
|
||||
**v1.16.0**,手动安装非 HACS):`tile_layer_url` =
|
||||
`https://{s}.basemaps.cartocdn.com/rastertiles/voyager/{z}/{x}/{y}.png?key=<CARTO_KEY>`
|
||||
(配 `tile_layer_options: {subdomains: abcd, maxZoom: 20}` + OSM/CARTO
|
||||
attribution)。实体不变:2 person + 4 zone(zone 用 `display: icon` +
|
||||
`circle: auto`,circle 读实体 `radius` 属性画半径圈)。
|
||||
- **CARTO key 是 secret**: 只存在于服务端 lovelace 存储(dashboard `map`
|
||||
的卡片配置)和用户本人处;勿写入本仓库或 Linear。
|
||||
- **文件/资源:** `/homeassistant/www/community/ha-map-card/map-card.js`
|
||||
(root:root 644,678554 B,sha256
|
||||
`f30dfb606e858d2216d5198d8cf758ce956d127006ebd7d66d4329153a247ec2`);
|
||||
Lovelace resource(storage)id `9d2b50b52c60420d89ebd041f722cf60` →
|
||||
`/hacsfiles/ha-map-card/map-card.js`,type module(WS
|
||||
`lovelace/resources/create`,2026.8 参数名 `res_type`)。升级 = 手动替换
|
||||
该文件(不在 HACS 管理下,浏览器需强刷)。
|
||||
- **备份:** `/homeassistant/.lovelace-backups/dashboard-map-map-20260830-133714.json`
|
||||
(还原 = 把备份里的 `views[0].cards[0]` 写回后再 WS `lovelace/config/save`
|
||||
url_path `map`)。
|
||||
- **验证 8/30:** 同瓦片无 key=水印 / 带 key=干净(256×256 PNG 视觉对比);
|
||||
resource HTTP 200 text/javascript;WS 读回卡片配置(type/entities/key/
|
||||
attribution/options)全部符合;HA 主机 `curl -4` 带 key 瓦片 200。
|
||||
- **Follow-up:** Core 升 2026.9.0 stable 后内置地图/zone 编辑器自动切
|
||||
OSMF 矢量瓦片;届时可保留 custom 卡(继续 keyed CARTO)或用备份还原
|
||||
内置卡。zone 编辑器等其余内置地图的水印在 2026.9 前无解。
|
||||
|
||||
## Known issues
|
||||
|
||||
@@ -407,6 +663,87 @@ advertised correctly over v4+v6. Both ESP32-C2 bulbs now announce `_matter._tcp`
|
||||
:5540 sessions** (device-side failure mode C; no errors logged — see
|
||||
[docs/matter-pairing-troubleshoot.md §8](../docs/matter-pairing-troubleshoot.md)).
|
||||
|
||||
## 马桶换气电源(Matter 插座,半计量)+ 电量估算 (2026-09-13)
|
||||
|
||||
**设备**:Matter `Smart Plug`(SIXWGH,`model_id 3596`,hw 1.0 / sw 1.3.0),node 18
|
||||
(0x12),`device_id 5ef1850953466d6e7a9c6b901fbebe1c`,config entry
|
||||
`01JF51VQ48PGJGXX3RNAG6MVAA`,区域**卫生间** (`wei_sheng_jian`),label `power`;
|
||||
2026-09-13 17:58 CST 配对。实体:
|
||||
`switch.wei_sheng_jian_ma_tong_huan_qi_dian_yuan`(插座)、
|
||||
`sensor.…_dian_yuan`(电源 W)、`sensor.…_dian_ya`(电压 V)、
|
||||
`sensor.…_you_gong_dian_liu`(有功电流 A)、`sensor.…_dian_li`(电力 kWh,
|
||||
**永久 unknown**)。
|
||||
|
||||
**根因(实测 Matter 属性,node 18)**:电量簇 0x0091 `FeatureMap = 13`
|
||||
(IMPE|CUME|PERE,即**声明**支持导入/累计/周期电量),但
|
||||
`CumulativeEnergyImported (0x0001)` 恒为 `null`,`PeriodicEnergyImported
|
||||
(0x0003)` 带载也恒为 `{Energy: 0}`;`CumulativeEnergyExported (0x0002)`
|
||||
不存在(EXPE 未声明,自洽)。HA 只用 `CumulativeEnergyImported` 建能量实体
|
||||
(`components/matter/sensor.py:1083`,`allow_none_value=True`)→ 该实体
|
||||
**永远不会出数**。**功率计量本身正常**:0x0090 `FeatureMap = 2` (ALTC),
|
||||
Voltage / ActiveCurrent / ActivePower 都随负载变化(实测 220.3 V / 118 mA /
|
||||
24.7 W,HA `电源` 0.0→24.9 W 有历史)。厂商 `update` 实体报无新固件。
|
||||
|
||||
**处理(方案 A:功率积分补电量)**:
|
||||
|
||||
- 新建 **Integration (Riemann sum) 辅助元素**:config entry
|
||||
`01M2D53T188FW8WEC547ENHSVH`(domain `integration`,state `loaded`),
|
||||
source `sensor.wei_sheng_jian_ma_tong_huan_qi_dian_yuan_dian_yuan`,
|
||||
`method: trapezoidal`、`unit_prefix: k`、`unit_time: h`、`round: 3`、
|
||||
`max_sub_interval: 60s`。
|
||||
- 实体 `sensor.wei_sheng_jian_ma_tong_huan_qi_dian_yuan_energy`(创建时 HA
|
||||
自动生成 `…_dian_yuan_ma_tong_huan_qi_dian_yuan_dian_liang`,随后立即
|
||||
`config/entity_registry/update` 改名为 `<插座>_energy` 以对齐约定;
|
||||
该实体新建、无引用,改名安全),friendly name「马桶换气电源 电力」,
|
||||
unit kWh、`device_class: energy`、**`state_class: total`**——能源仪表盘
|
||||
允许 `TOTAL` 与 `TOTAL_INCREASING`(`components/energy/validate.py:279`)。
|
||||
- **能源仪表盘** (`/energy`):grid 源 `[8]` 由 `…_dian_li` 改为 `…_energy`,
|
||||
其余 8 条插座源未动。注意这 9 条「插座」全部以 `type: grid` 注册,被当作
|
||||
全屋用电代理;`switch` 卡片所在的 Grid 卡片此前第 9 行是空的,即本次修复点。
|
||||
- **Quick 仪表盘**:「用电量(按插座)」图第 11 项由 `…_dian_li` 改为
|
||||
`…_energy`;新增 `column_span: 2` 的「开关」区块(heading + tile
|
||||
`switch.…` + `toggle` feature + 功率徽标)→ 视图 6→7 分区。
|
||||
|
||||
**口径警告**:`…_energy` 是**估算值**(Riemann 积分,只在 HA 运行期间累计、
|
||||
非账单级),与另外 8 个原生计量插座的累计电量口径不同;功率传感器更新
|
||||
间隔约 5–10 s(实测 24.9/24.8/25.0 W 抖动),加 `max_sub_interval: 60s`
|
||||
保证静默时也继续累计。
|
||||
|
||||
**Agent 侧建辅助元素的方法(2026-09-13 实测)**:HA 的 config flow 走
|
||||
**REST**(WS 只有 `config_entries/flow/progress|subscribe`,没有 start)。
|
||||
经 supervisor 代理即可,无需 HA 长连接/长寿命 token:
|
||||
|
||||
```bash
|
||||
# SUPERVISOR_TOKEN 由 sudo -n -i 提供
|
||||
curl -s -X POST -H "Authorization: Bearer $SUPERVISOR_TOKEN" \
|
||||
-H "Content-Type: application/json" -d '{"handler":"integration"}' \
|
||||
http://supervisor/core/api/config/config_entries/flow # → {flow_id, step_id:"user", data_schema}
|
||||
curl -s -X POST -H "Authorization: Bearer $SUPERVISOR_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"name":"…","source":"sensor.x","method":"trapezoidal","round":3,
|
||||
"unit_prefix":"k","unit_time":"h","max_sub_interval":{"minutes":1}}' \
|
||||
http://supervisor/core/api/config/config_entries/flow/<flow_id> # → create_entry
|
||||
```
|
||||
|
||||
`auth/long_lived_access_token` 在 supervisor 代理身份下**失败**
|
||||
(`unknown_error`),故无法用长寿命 token 开浏览器会话;`DurationSelector`
|
||||
的值是 `{"minutes":1}` 形式(`cv.time_period`)。
|
||||
|
||||
**备份/回滚**:`.lovelace-backups/dashboard-quick-20260913-181251-pre-ma-tong-plug.json`
|
||||
(改动前原件)、`…-20260913-183210-pre-repoint.json`(改名/换源前);
|
||||
`.ha-backups/energy-20260913-183135-pre-ma-tong-repoint.json`(能源 prefs)。
|
||||
回滚 = 把能源 prefs 的源 [8] 指回 `…_dian_li` + 还原 Quick 面板 JSON;
|
||||
如需彻底放弃估算电量 = 删除 config entry `01M2D53T188FW8WEC547ENHSVH`。
|
||||
|
||||
**验证 (2026-09-13 18:3x)**:`…_energy` 0.002→0.003 kWh 且随 24.6 W 负载
|
||||
增长(换气扇关掉后回落 0.0 W,累计值保留);`recorder/list_statistic_ids`
|
||||
已含该实体;Quick 面板 WS 读回 7 分区、用电量图 11 项指向新实体、旧
|
||||
`_dian_li` 引用 0 处;能源 prefs 读回 9 源、第 9 条为新实体。
|
||||
**`energy/validate` 已全绿**(9 源 0 issue):创建后 ~5 min 内曾报
|
||||
`statistics_not_defined`(recorder 的统计任务周期是 5 min,`statistics_meta`
|
||||
行由该任务建立),18:39 复核时已自动消失——建辅助元素后**不要**把这条
|
||||
瞬时告警当作失败。
|
||||
|
||||
## Related docs
|
||||
|
||||
- [runbooks/home-assistant-maintenance.md](../runbooks/home-assistant-maintenance.md) — `ha` CLI maintenance runbook + [script](../runbooks/scripts/ha-maintenance.sh); custom-component zip install is §7
|
||||
|
||||
@@ -185,6 +185,23 @@ dig @202.91.35.141 SOA wsvc.info +short
|
||||
|
||||
On-server docs: `/opt/pdns/README.md`, `CHANGELOG.md`.
|
||||
|
||||
## Disk / logging (VPS-81, 2026-09-02)
|
||||
|
||||
Root disk cleanup performed (runbook: [host-disk-cleanup](../runbooks/host-disk-cleanup.md)):
|
||||
|
||||
- Root `/` (20G vda1): 76% used → **38% used** (15G → 7.1G; free 4.7G → 12G).
|
||||
- **AGH log flood root cause fixed**: `/opt/adguard/conf/AdGuardHome.yaml`
|
||||
`log.verbose: true → false` (backup `AdGuardHome.yaml.bak-20260902-vps81`).
|
||||
Verbose debug was streaming to stderr → container `json.log` (~120MB/day);
|
||||
`log.file: ""` makes AGH's own rotation keys inert. Restart only (no recreate).
|
||||
- Journald capped: `/etc/systemd/journald.conf.d/00-vps81.conf`
|
||||
`SystemMaxUse=200M`; journal vacuumed to ~96M.
|
||||
- Docker: engine **29.7.2**; 14 unused images removed (kept `pdns-auth-50:5.0.5`
|
||||
rollback pin); 12 orphan anonymous volumes + build cache pruned. In-use
|
||||
volumes intact (`pdns_dbdata`, `b594d738…` PG data, `e855d078…` backup).
|
||||
- Follow-up: re-check AGH `json.log` growth **2026-09-09** (one-week checkpoint);
|
||||
global docker log rotation only if still needed.
|
||||
|
||||
## Verified
|
||||
|
||||
Last checked: **2026-08-01 21:40 CST** — operational; docs audit recorded.
|
||||
@@ -194,3 +211,4 @@ Last checked: **2026-08-01 21:40 CST** — operational; docs audit recorded.
|
||||
- `only-notify=` + `also-notify=202.91.35.141`; MASTER `domains.master` cleared
|
||||
- https://pdns.wsvc.info → **302**; https://pgweb.wsvc.info → **401**
|
||||
- Hardening backlog: API/DB credential rotation + TSIG rotate (see upstream doc)
|
||||
- 2026-09-02 (VPS-81): post-cleanup verified — 10 containers Up (adguardhome healthy), DNS SOA/NS + web endpoints OK; see Disk/logging section above.
|
||||
|
||||
+12
-4
@@ -8,8 +8,8 @@
|
||||
| IPv4 | `192.168.55.15` (LAN55) |
|
||||
| DNS | (none) |
|
||||
| SSH | `ssh -4 windy@192.168.55.15`(key auth 已验证可用 2026-08-29;agent 沙箱用 `ssh -F /dev/null -o BatchMode=yes`;password auth 亦可) |
|
||||
| Host | PVE 管理的 QEMU VM(i440FX),Debian 13 (trixie),内核 6.12.105 |
|
||||
| Resources | 2 GB RAM / 30 GB disk(26 G 空闲) |
|
||||
| Host | PVE 管理的 QEMU VM(i440FX,**VMID 100**),Debian 13 (trixie),内核 6.12.105;宿主机 **pve2 `192.168.55.25`**(Proxmox 9.2.2,SSH `root@192.168.55.25`,`onboot: 1`,QEMU guest agent 已装;2026-08-31 补记) |
|
||||
| Resources | 3 GB RAM(08-30 13:58 由 2G 上调、删除 balloon/ksm/shares 后重启生效)/ 30 GB disk(26 G 空闲) |
|
||||
| Docker | 29.7.2;容器 `timescaledb` = `timescale/timescaledb:latest-pg18`(PG **18.6** + TimescaleDB **2.29.2**,Apache-2.0 版) |
|
||||
| Ports | `192.168.55.15:5432`(PG,IPv4 only);`192.168.55.15:8081`(pgweb GUI,basic auth) |
|
||||
|
||||
@@ -17,8 +17,8 @@
|
||||
|
||||
| DB | Owner | Size | 用途 |
|
||||
|---|---|---|---|
|
||||
| `hass` | hass | ~14 MB | HA recorder(states/events/statistics),客户端 HAOS `192.168.55.11` |
|
||||
| `scribe` | postgres | ~11 MB | HA scribe 集成(entities/areas/devices 注册表同步 + `states_raw` hypertable) |
|
||||
| `hass` | hass | ~406 MB(2026-09-13) | HA recorder(states/events/statistics),客户端 HAOS `192.168.55.11` |
|
||||
| `scribe` | postgres | ~2.6 GB(2026-09-13) | HA scribe 集成(entities/areas/devices 注册表同步 + `states_raw`/`events` hypertable + `csg_history` 长期归档表);体积由 `sensor_minute` 图表管道主导(2.26 GB),见 Known issues |
|
||||
| `postgres` | postgres | ~9 MB | 默认库 |
|
||||
|
||||
## Ops notes
|
||||
@@ -35,17 +35,25 @@
|
||||
- **备份**:由 `pg-backup` 容器接管(2026-08-29),宿主机 cron 与 `/opt/database/pg-backup.sh` 已退役。恢复用 `pg_restore`(custom format)——2026-08-29 已实测还原 hass 库 dump(states 10014 行)成功。
|
||||
- **认证**:外部连接 scram-sha-256(密码必填,改密码有效);容器内 loopback 为 trust(官方镜像默认)。
|
||||
- **回滚**:旧启动命令保留在 `/opt/database/run`(容器无状态,数据在 /srv/pgdata);旧匿名卷 `9375195843b950f4e04c34872409ca095e1136520dd019a8e86e2794be06c236`(根盘 ~82M)保留作兜底,确认稳定后可 `docker volume rm`。
|
||||
- **开机自愈**(2026-08-30):新增 systemd oneshot `pgdb-compose.service`(enabled,源码在仓库 `compose/pgdb/pgdb-compose.service`):`After=network-online.target docker.service`,开机后幂等执行 `docker compose up -d`,重试直到 `192.168.55.15:5432` 监听,重试耗尽 `--force-recreate` 兜底(数据在 bind mount,无损)。原因:2026-08-30 开机竞态——docker 恢复容器时 VM IP 尚未可绑(EADDRNOTAVAIL),timescaledb/pgweb 启动失败且 docker 不重试。手动重跑:`sudo systemctl restart pgdb-compose.service`。
|
||||
- 本机无防火墙(ufw/nft/iptables 均未装)——待办:如要彻底隔离可加 ufw 白名单 192.168.55.11。
|
||||
- `/opt/database/backups/` 根下残留 `*-2026-08-29_1359.dump`(compose 化之前旧备份机制产物)与 `backup.log`——健康检查只看 `daily/`,残留可清理。
|
||||
- **Runbooks**:[pgdb-health](../runbooks/pgdb-health.md)(只读健康检查)、[pgdb-restore](../runbooks/pgdb-restore.md)(pg_restore 还原)、[pgdb-update](../runbooks/pgdb-update.md)(镜像/compose 升级)。
|
||||
- **CSG 长期归档(2026-08-29, W1N-243)**:`csg_history` 表(`period date / kind('day'|'month') / usage_kwh / cost / ladder / balance / updated_at`,PK(period,kind),`GRANT SELECT TO hass`)保存南方电网有价值数据:day = 逐日(昨日用电/费用/阶梯/余额,2026-07-01 起),month = 当月累计(用电/费用,2025-01 起)。由 TimescaleDB 每日任务 **1008** `csg_daily_snapshot()`(22:30 Asia/Shanghai;**TS job 非 pg_cron**,本库未装 pg_cron)upsert 维护:取「最新有值行」防瞬态 unknown 竞态;日费用缺原生 `latest_day_cost` 时回退 = 昨日用电 × 当前档费率(模板 `csg_current_ladder_tariff` 0.639);月费用回退模板 `csg_this_month_ladder_cost`。验证:day 08-28 = 7.66 / 4.89474 / 二档 / 0,month 08 = 302.47 / 180.28。回填来源:集成 attributes `history_data`(59 天)+ `by_month`(19 月)——08-29 前唯一残存历史。回滚:`DROP TABLE csg_history` + `SELECT delete_job(1008)`。
|
||||
|
||||
## Known issues
|
||||
|
||||
- 2026-09-13:**`sensor_minute` 体积构成与压缩窗口(只读诊断,暂不处理)**。`scribe` 库 2.6 GB = `sensor_minute` **2.26 GB**(850 万行 / 16 天,约 56–59 万行/天 = 331 实体 × 1440 分钟 LOCF)+ `states_raw` 290 MB + `events` 1.5 MB;`hass` 库另 406 MB。2.26 GB 中 1.50 GB 是 chunk `[09-03,09-10]`、0.76 GB 是 `[09-10,09-17]`,**都还没到压缩窗口**——TimescaleDB 的 `compress_after` 按 **chunk 结束时间**判断(09-10 结束 + 7 天 = **09-17** 才合格),所以「7 天 chunk + 7 天 compress_after」的设计下限就是盘上常驻近 14 天原始数据;已压缩的 `[08-27,09-03]` 从 1.04 GB → **1.5 MB**(LOCF 重复度极高,~700:1)。任务 1005 健康(30 成功 / 0 失败,最近 09-13 04:18 跑过但无合格 chunk);1002/1003/1006/1007 亦全 Success。稳态估算 ≈ 2 个未压缩 chunk(3–4.5 GB)+ 已压缩归档(约 1.5 MB/周 ≈ 80 MB/年)≈ **4–5 GB 平台期**;pgdata 卷 32 G 当前用 3.2 G,可用 27 G,**无需处理**。复查点 **2026-09-17 之后**:`_hyper_4_6_chunk` 应转为 `compressed=true` 且库体积回落;若仍为 false 才需动手(手动 `compress_chunk()` 或调小 `compress_after`)。可选调优:chunk 间隔 7 天 → 1 天 + `compress_after` → 2 天,把常驻未压缩量压到 <1 GB(`set_chunk_time_interval` 只对新 chunk 生效,旧 chunk 不重切)。诊断命令:`select chunk_name, is_compressed from timescaledb_information.chunks where hypertable_name='sensor_minute';` + `pg_database_size('scribe')`。**注意:这是 pgdb 侧对象,HA/scribe 的 `retention_states` 管不到它;HA 侧唯一杠杆是少记/少画(等于砍图)。**
|
||||
- 2026-08-29:HA 侧 HACS 集成 `custom_components.scribe`(YAML `scribe: db_url:`,连 `scribe` 库)建表被拒(`permission denied for schema public`,hass 无 CREATE 权限),之后持续报 `relation "entities" does not exist`。**已解决**:① `GRANT CREATE ON SCHEMA public TO hass;`(scribe 库)② 重启 HA Core 触发重跑建表。重启后自动创建 `entities`(1591 行)/`users`/`areas`/`devices`/`integrations`/`states_raw` 表并启用 TimescaleDB 时间序列能力。报错已停止(最后一条 06:06 UTC),`states_raw` 持续写入。2026-08-29 复查:scribe 现有**两个** hypertable——`states_raw`(segmentby `metadata_id`、orderby `time`)与 `events`(segmentby `event_type`、orderby `time`),均 1 维 `time`;压缩已配置(`timescaledb_information.compression_settings` 可见对应行;2.29.x 该视图无 `compression_enabled` 列)。
|
||||
- 2026-08-29:**timescale reader 图表对象**(配套 hass 的 `timescale_database_reader` 集成 + `timescale-plotly-card`,上游 SQL `remmob/timescale_database_reader` `SQL/scribe/01+02` @ `bb8776a`,以 postgres 执行):`sensor_minute_aggregate` 连续聚合(1 分钟桶,last(state)/last(value),实时聚合开启)+ `sensor_minute_aggregate_entity` 视图(join `entities`)+ `sensor_minute` hypertable(`minute`/`entity_id`/`state`/`value`,LOCF 前向填充)。任务:1005 `sensor_minute` 压缩(7 天)、1006 `sensor_minute` 保留(10 年)、1007 `every_minute_refresh` 每分钟增量刷新(含 5 分钟回溯窗口修正)。授权:`GRANT SELECT ON sensor_minute_aggregate, sensor_minute_aggregate_entity, sensor_minute, entities TO hass`。种子 19529 行(331 实体,自首个数据点起)。**刻意跳过**了上游脚本对 `states_raw` 的 3 个月保留 + 压缩策略语句——与"`states_raw` 永久归档"定位冲突,如需磁盘回收属用户决策(scribe 自己的压缩任务 1000/1001 未动)。
|
||||
- 2026-08-29:**`sensor_minute_refresh` 本地补丁(类比 tianqi 补丁,重跑上游 02 SQL 后需重打)**:值 CASE 的 `ELSE 0` → `ELSE NULL`。原因:scribe 对 unavailable 分钟 value 为 NULL,上游刷新过程兜底写 0;对差分模式的用电图,0→计数器回升会把插座的**生命周期累计值**(最高 1588 kWh)算进掉线那一小时。同日一次性清理既有脏 0:头部占位行 DELETE 505 行(各实体首次非零分钟之前的 value=0);`sensor.%_energy` 与温湿度实体的 value=0 → NULL(10+16 行,物理上不可能的真 0,图表渲染为断点)。功率实体的中途 0 是真实待机读数,保留。
|
||||
- `hass` 库的 recorder 表仍为普通表(无 hypertable);`scribe` 集成负责时间序列历史(`states_raw` + `events` hypertable)。
|
||||
|
||||
## Verification history
|
||||
|
||||
- 2026-08-31:**13:58 重启根因确认,非停电**(W1N-263):pve2(`192.168.55.25`)任务日志显示 08-30 **13:58:00 `root@pam` 在 PVE Web UI 修改 VM 100 配置**(`-delete allow-ksm,balloon,shares -memory 3072`),**13:58:06 点 Reboot**(`qmreboot` → 客机 13:58:08 干净 ACPI 关机 → 13:58:13 自动重启)。宿主机全程在线(08-30 09:00 开机至今连续运行 1d12h+),`.66.26` PVE 及各 VM 均无重启——排除停电。HA recorder 在窗口(13:58:46–47)报 2 次 `Connection refused`,DB 恢复后自动重连,**无数据丢失**(`hass.states`/`scribe.states_raw` 13:55–14:02 逐分钟无缺口,recorder 内存队列吸收回写)。13:58:47 三容器已起,13:58:56 自愈单元 `pgdb-compose.service` 执行成功——本次自愈按设计工作。同日下午 12:54–12:55 另有一次**客机内自重启**(无 PVE 任务,工作站 SSH 会话相邻)。08-29 22:19→08-30 09:00 宿主机停机 10h41m 为**干净关机**(systemd 有序关闭,非停电)。
|
||||
- 2026-08-30:**开机竞态故障 + 修复**(W1N-260):09:01 开机后 docker 恢复容器时绑定 `192.168.55.15:5432/8081` 失败(EADDRNOTAVAIL)→ timescaledb/pgweb 停摆至 12:16,pg-backup 开机备份失败(解析不到 timescaledb)→ unhealthy。12:22 `docker compose up -d --force-recreate` 修复(三容器回 `database_default`、端口发布、今日备份、pgweb 恢复);用户重启 HA Core 后写入管道恢复。12:43 新增开机自愈 unit `pgdb-compose.service`(enabled,已实测幂等 reconcile)。pgdb-health 8 项全绿。
|
||||
- 2026-08-29:首次检查(只读)+ 修复 scribe 权限 + 安装夜间备份。见 Linear vps 项目登记。
|
||||
- 2026-08-29:**compose 改造完成**(W1N-227,用户已验收):裸 `docker run` → `/opt/database/docker-compose.yml` 三服务(timescaledb + pgweb + pg-backup);superuser 换强密码;端口收紧 IPv4;备份容器化(TZ=Asia/Shanghai,cron 02:00 本地);`pg_restore` 还原实测通过;pgweb UI 用户确认可查 hass/scribe 数据。源码在仓库 `compose/pgdb/`。
|
||||
- 2026-08-29:**运维 runbook 落地**(W1N-228,已验收):新增 `runbooks/pgdb-health.md`(只读,8 项诊断全绿)、`pgdb-restore.md`(流程式,temp-DB 安全还原 + 审批门)、`pgdb-update.md`(门控命令式,回滚=/opt/database/run + 旧卷);README 索引与 validate-repo.sh 分类同步更新;runbook 命令已对活主机逐条实测(含 `pg_restore -l` 校验当日 dump)。同日修正:SSH key auth 可用(facts 原记"密钥未安装"已过时);scribe 新增 `events` hypertable。
|
||||
- 2026-08-29:**CSG 长期归档 + recorder 365d**(W1N-243):建 `csg_history` 表 + attributes 回填(逐日 59 + 逐月 19)+ 每日任务 1008(函数 v2:最新有值行读取、日费用阶梯回退);hass `purge_keep_days` 30→365(备份 `configuration.yaml.bak-20260829-purge365`)。见 Linear vps W1N-243。
|
||||
|
||||
@@ -24,6 +24,7 @@ ssh -4 windy@synapse.chans.xyz
|
||||
| DB | ESS embedded PostgreSQL 17 (PVC 20Gi, local-path) |
|
||||
| Cache | ESS embedded Redis (PVC 2Gi) |
|
||||
| Chart | `oci://ghcr.io/element-hq/ess-helm/matrix-stack`, version `26.7.2` |
|
||||
| Plane | Helm `plane-ce-1.8.0` (app `v1.4.1`), namespace `plane` — self-hosted Plane project management |
|
||||
|
||||
### Matrix service endpoints
|
||||
|
||||
@@ -51,8 +52,50 @@ All other ports internal only (no K3s API, no database, no Redis exposed).
|
||||
|
||||
- `ess` — all ESS workloads (Synapse, MAS, Element, Postgres, Redis, HAProxy)
|
||||
- `matrix-system` — cluster base resources (ResourceQuota, LimitRange, mrtc-placeholder)
|
||||
- `plane` — Plane project management (Helm release `plane-app`)
|
||||
- `cert-manager` — cert-manager
|
||||
|
||||
## Plane (project management)
|
||||
|
||||
Self-hosted [Plane](https://github.com/makeplane/plane) on the same K3s node, deployed via the official `plane-ce` Helm chart.
|
||||
|
||||
| Item | Detail |
|
||||
|------|--------|
|
||||
| Release | `plane-app` (ns `plane`), chart `plane-ce-1.8.0`, app `v1.4.1`, revision 1 |
|
||||
| URL | https://plane.chans.xyz |
|
||||
| Install date | 2026-09-01 |
|
||||
| Values source | `/home/windy/plane-k3s/values.yaml` (plain file, not a git repo) |
|
||||
| Images | `artifacts.plane.so/makeplane/*` (`plane-frontend`, `plane-backend`, `plane-admin`, `plane-live`), pullPolicy `Always` |
|
||||
| Ingress | Traefik `IngressRoute` `plane-app-ingress` — `/`→web, `/api` `/auth`→api, `/spaces`→space, `/god-mode`→admin, `/live`→live, `/uploads`→minio; `maxRequestBodyBytes` 20Mi |
|
||||
| TLS | Own namespace `Issuer` `plane-app-cert-issuer` (HTTP-01, LE prod, `admin@chans.xyz`); cert `plane-app-ssl-cert` (CN `plane.chans.xyz`) |
|
||||
| DB | Bundled Postgres `15.7-alpine` (PVC 5Gi, local-path) |
|
||||
| Cache/queue | Bundled Redis (PVC 100Mi), RabbitMQ `3.13.6-management-alpine` (PVC 100Mi) |
|
||||
| Storage | Bundled MinIO (`minio/minio:latest`, root user `admin`, PVC 5Gi) — S3 for uploads/docs |
|
||||
| Resources | Every workload: cpu 50m/500m, mem 50Mi/1000Mi, replicas 1 |
|
||||
| SMTP | Not configured (no `smtp` values) — Plane invites/password resets won't email yet |
|
||||
|
||||
Workloads (all 1/1 Running): 7 Deployments (`plane-app-{admin,api,beat-worker,live,space,web,worker}-wl`) + 4 StatefulSets (`plane-app-{minio,pgdb,rabbitmq,redis}-wl`); init Jobs `api-migrate-1` / `minio-bucket-1` Completed. All PVCs Bound on `local-path` (root disk).
|
||||
|
||||
### Plane configuration notes
|
||||
|
||||
- **`planeVersion: v1.4.1`** pinned in values.yaml; chart tracks Plane's own tags.
|
||||
- **Secrets**: Helm-generated Opaque secrets (`plane-app-app-secrets`, `-doc-store-secrets`, `-pgdb-secrets`, `-rabbitmq-secrets`, `-live-secrets`); `requireExplicitSecrets: false`. Values live in `$SECRET_KEY`, `DATABASE_URL`, `AMQP_URL`, `REDIS_URL` etc.
|
||||
- **Sentry / CORS**: `sentry_dsn` and `cors_allowed_origins` empty (defaults fine for single-host).
|
||||
- **MinIO is `latest` tag** — pin a version for reproducibility.
|
||||
- **Backup**: NOT covered by `/var/backups/matrix` (which is paused anyway) — Plane Postgres/MinIO PVCs have no backup tier yet.
|
||||
|
||||
### Plane verification
|
||||
|
||||
```bash
|
||||
# Release + workloads
|
||||
sudo helm list -A
|
||||
sudo k3s kubectl -n plane get deploy,sts,pods -o wide
|
||||
# Cert + ingress
|
||||
sudo k3s kubectl -n plane get certificate,ingressroute
|
||||
# Endpoint
|
||||
curl -4 -s -o /dev/null -w '%{http_code}\n' https://plane.chans.xyz/
|
||||
```
|
||||
|
||||
## Local backup
|
||||
|
||||
| Item | Detail |
|
||||
@@ -62,7 +105,7 @@ All other ports internal only (no K3s API, no database, no Redis exposed).
|
||||
| Retention | 7 days |
|
||||
| Disk warning | 80% (healthcheck), 90% (backup stops) |
|
||||
| Content | Planned: PostgreSQL `synapse` + `mas` logical dumps, media store archive, `/etc/matrix-bootstrap` |
|
||||
| Status | **Not operational** — no current Matrix backup or recovery tier |
|
||||
| Status | **Not operational** — no current Matrix backup or recovery tier. **Plane data (its own Postgres + MinIO PVCs in ns `plane`) is also not covered by any backup.** |
|
||||
|
||||
## Health checks
|
||||
|
||||
@@ -101,5 +144,6 @@ diagnosis and imperative recovery work.
|
||||
- MatrixRTC / Element Call / LiveKit / Coturn not deployed (`mrtc.chans.xyz` reserved only)
|
||||
- SMTP email not yet configured (requires manual secret bootstrap followed by a
|
||||
reviewed Ansible stack deployment)
|
||||
- Plane `minio` image uses `latest` tag (pin a version)
|
||||
- No off-site Restic backup
|
||||
- Single-node K3s (no HA for control plane)
|
||||
|
||||
+26
-12
@@ -5,12 +5,12 @@
|
||||
| Role | Multi-service VPS (Vaultwarden, Traefik, Soft Serve, …) |
|
||||
| SSH | `ssh -4 windy@us2.wsvc.info` (prefer IPv4 from WSL) |
|
||||
| IPv4 | `193.9.44.165` |
|
||||
| Also DNS | `auth.wsvc.info` → this host; `repo.windy.me` → this host (Soft Serve) |
|
||||
| Also DNS | `auth.wsvc.info` → this host; `repo.windy.me` → this host (Gitea) |
|
||||
| Public HTTPS | Traefik on `:80` / `:443` (`/opt/traefik`) |
|
||||
|
||||
## Vaultwarden (Bitwarden-compatible)
|
||||
|
||||
**Status: operational** (Postgres live, HTTPS 200, healthy containers, SMTP AUTH OK — last probe 2026-08-01 18:55 CST).
|
||||
**Status: operational** (Postgres live, HTTPS 200, healthy containers, SMTP AUTH OK — last probe 2026-08-29).
|
||||
|
||||
Upstream docs: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
|
||||
|
||||
@@ -21,9 +21,9 @@ Upstream docs: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
|
||||
| Env file | `/opt/vaultwarden/.env` |
|
||||
| Admin overrides | `/opt/vaultwarden/vw-data/config.json` (**wins over env**) |
|
||||
| Public URL / `DOMAIN` | `https://auth.wsvc.info` |
|
||||
| Image | `vaultwarden/server:1.37.1` (pinned) |
|
||||
| Image | `vaultwarden/server:1.37.2` (pinned) |
|
||||
| Live DB | **Postgres 16** (`vw-db` / service `pg`) via compose `DATABASE_URL` |
|
||||
| Data (probe) | users=1, ciphers=1327 |
|
||||
| Data (probe) | users=1, ciphers=1360 |
|
||||
| Cold SQLite | `backups/sqlite-cold/db.sqlite3.pre-pg-20260801` (not used live) |
|
||||
| Pre-migrate backup | `backups/pre-pg-migrate-20260801_161204/` |
|
||||
| Data dir | `./vw-data` → `/data` (attachments, rsa keys, `config.json`) |
|
||||
@@ -50,7 +50,7 @@ Upstream docs: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
|
||||
|
||||
| Container | Status |
|
||||
|-----------|--------|
|
||||
| `vaultwarden` | Up (healthy), `vaultwarden/server:1.37.1` |
|
||||
| `vaultwarden` | Up (healthy), `vaultwarden/server:1.37.2` |
|
||||
| `vw-db` | Up (healthy) — **live** Postgres |
|
||||
| `vaultwarden-backup` | Up (`pg_dump`) |
|
||||
| `vaultwarden-pgweb` | Exited (profile `debug`) |
|
||||
@@ -80,19 +80,33 @@ ansible-playbook playbooks/compose-reconcile.yml --limit vaultwarden \
|
||||
|
||||
| Container | Status | Image / notes |
|
||||
|-----------|--------|---------------|
|
||||
| `soft-serve` | Up | `ghcr.io/charmbracelet/soft-serve:latest` (`repo.windy.me:2222`) |
|
||||
| `gitea` | Up | `gitea/gitea@sha256:1c17ecaead42e…` (1.27.3-rootless) — SSH `repo.windy.me:2222`, web `https://repo.windy.me` |
|
||||
| `gitea-backup` | Up | alpine + sqlite3/rsync sidecar (daily backup 02:00 / prune 03:00, crond) |
|
||||
|
||||
### Gitea (replaced Soft Serve 2026-09-18; [Plane VPS-94](https://plane.chans.xyz))
|
||||
|
||||
- `/opt/gitea/compose.yml` (+ `Dockerfile.backup`, `scripts/`, `config/app.ini`, `data/`, `secrets/`, `backups/`); 镜像: `compose/gitea/`(参考, 服务器文件为准)
|
||||
- **rootless 镜像** uid 1000:1000; SQLite `/opt/gitea/data/data/gitea.db`; repos `/opt/gitea/data/data/git/repositories/`; app.ini `/opt/gitea/config/app.ini`(600, 含 SECRET_KEY)
|
||||
- SSH: 内置 server 容器内 `:2322`(`SSH_LISTEN_PORT` 非特权), Traefik TCP entrypoint `ssh`(`:2222` → `gitea:2322`, `HostSNI(*)`, `tls=false`) on `vw-net`; clone URL `ssh://git@repo.windy.me:2222/windy/<repo>.git`(owner 段 `windy`)
|
||||
- **host key 复用 soft-serve**(`SSH_SERVER_HOST_KEYS=/secrets/soft_serve_host_ed25519`, ed25519, 指纹 `SHA256:PdxZRe74…`): 客户端 known_hosts 零变更; 仅公钥认证(密码认证未启用)
|
||||
- Web: `https://repo.windy.me`(Traefik websecure + letsencrypt); `DISABLE_REGISTRATION=true`, Actions 关闭; 管理员 `windy`(凭据仅存服务器 `/opt/gitea/.admin-credentials`, 勿入库/入 Plane)
|
||||
- 仓库: 16 个(顶层 11 + `cdia/` 4 + `windyboy/go-caatsm`), 2026-09-18 自 soft-serve `push --mirror` 迁移, 逐仓 `ls-remote` ref 全集 + HEAD symref 两端一致; 可见性仅 `dotfiles-personal` private, 其余 public(与 soft-serve 现状一致)
|
||||
- 备份: sidecar 每日 02:00 → `backups/gitea_<TS>/{app.ini.tar.gz, gitea.db, repos.tar.gz}`(app.ini 含恢复必需 SECRET_KEY), 03:00 prune 保留 14 份; 已验证手动备份产物 109.9M
|
||||
- 回滚: `/opt/soft-serve` 未删(compose stop + sidecar 停, 数据与旧备份冻结保留), 回滚 = Traefik `:2222` 指回 `soft-serve:23231` + 客户端 remote 回改旧无 owner 段路径; 观察 2–4 周后清理(历史: W1N-244~248)
|
||||
| `traefik` | Up | `traefik:v3.6.2` (`/opt/traefik`, public `:80`/`:443`) |
|
||||
| `nghttpx-proxy` + `squid-backend` | Up | HTTP forward-proxy stack (`/opt/nghttpx`), network `nghttpx_internal-net`; details TBD |
|
||||
|
||||
Directories for `authelia`, `conduit`, `dendrite`, `mastodon`, `rustdesk`, `zitadel`, etc. exist under `/opt` but have no running containers; treat them as dormant, not documented services.
|
||||
**Disk cleanup 2026-09-18** ([Plane vps VPS-93](https://plane.chans.xyz)): root 71% → **23%** (~33G freed) keeping soft-serve / vaultwarden / traefik (nghttpx kept running per operator choice). Removed: unused Docker images + orphan volumes (incl. `zitadel_data` 801M), dormant `/opt` dirs (dendrite + its disabled `dendrite.service` unit, mastodon, dailysync, keycloak, media-repo, authelia, conduit, npm, manager, fusion, zitadel, rustdesk), rootless podman storage (6.4G stale goauthentik), home dev caches, apt cache, journal 3.8G→162M (+`SystemMaxUse=200M` drop-in, active next boot), truncated container logs (nghttpx 550M / traefik / squid). Follow-up: nghttpx-proxy logs grow ~25M/day (INFO per-connection); root-cause log-level/rotation fix still open (needs container restart approval).
|
||||
|
||||
Remaining running services on this host: `gitea`, `vaultwarden` stack, `traefik`, `nghttpx-proxy` + `squid-backend` (undocumented forward proxy, `/opt/nghttpx`). `/opt/soft-serve` kept stopped as rollback (2–4 weeks, data intact). `/home/windy/authelia` (76M) left in place — outside approved cleanup scope.
|
||||
|
||||
## Verified
|
||||
|
||||
Last checked: **2026-08-01 18:55 CST** — operational.
|
||||
Last checked: **2026-09-18** — operational; disk cleanup done (see note above, Plane vps VPS-93). Prior full probe: 2026-08-29.
|
||||
|
||||
- `vaultwarden` + `vw-db` healthy; `DATABASE_URL` → `pg:5432/vaultwarden`
|
||||
- `https://auth.wsvc.info/` **200**, `/admin` **200**, `/api/config` OK (`disableUserRegistration: true`)
|
||||
- Identity wrong-password → **400** business error (DB readable, not 500)
|
||||
- SMTP: container → `mx2:587` OK; STARTTLS cert CN=`mx2.windy.me`; **AUTH OK** with effective `config.json` password (synced with `.env` / `.smtp-credentials`)
|
||||
- LE cert CN=`auth.wsvc.info`
|
||||
- PG counts: users=1, ciphers=1327
|
||||
- SMTP: container → `mx2:587` OK; **AUTH OK** with effective `config.json` password (synced with `.env` / `.smtp-credentials`, fingerprint match)
|
||||
- PG counts: users=1, ciphers=1360
|
||||
- Image `vaultwarden/server:1.37.2` (**upgraded 2026-08-29** from 1.37.1; required for Bitwarden clients 2026.8.0+); post-upgrade 404 fixed by Traefik restart, then 200
|
||||
- vps-health local check **installed 2026-08-29** (`vps-healthcheck.timer` daily 06:15 + `/usr/local/lib/vps-health/run`); `health-report.yml --limit vaultwarden` now passes (**ok**, was failing due to missing check infra + script bugs fixed: trim_blocks render, pgweb debug-profile false positive, SMTP probe moved host-side since image lacks python3)
|
||||
|
||||
+2
-1
@@ -17,7 +17,7 @@ the software deployed there, see [the LAN overview](../docs/lan-overview.md).
|
||||
| mx2.windy.me | mailcow (primary MX prio 20) | `ssh -4 windy@mx2.windy.me` | 194.163.160.244 | ✓ (mx2) | active | [hosts/mx2.windy.me.md](../hosts/mx2.windy.me.md) |
|
||||
| us2.wsvc.info | Vaultwarden/Postgres (+ Traefik, Soft Serve, …) | `ssh -4 windy@us2.wsvc.info` | 193.9.44.165 | ✓ (us2) | active | [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md) |
|
||||
| mx.windy.me | mail (secondary MX prio 30) | TBD | see AAAA/A | — (stub) | stub | — |
|
||||
| repo.windy.me | Soft Serve git (on us2) | `ssh -p 2222 windy@repo.windy.me` | 193.9.44.165 | — (service on us2) | stub | see us2 |
|
||||
| repo.windy.me | Gitea git (on us2) | `ssh -p 2222 windy@repo.windy.me` | 193.9.44.165 | — (service on us2) | stub | see us2 |
|
||||
| auth.wsvc.info | Vaultwarden public hostname | — (HTTPS) | → us2 | — (alias) | active | see us2 |
|
||||
| us1.wsvc.info | PowerDNS secondary (ns2 host) | TBD | 202.91.35.141 | — (stub) | stub | Auth 5.0.5; see hk2 |
|
||||
| us4.wsvc.info | WireGuard VPN | `ssh -4 windy@us4.wsvc.info` | 185.201.226.122 | ✓ (us4) | active | [hosts/us4.wsvc.info.md](../hosts/us4.wsvc.info.md) |
|
||||
@@ -48,4 +48,5 @@ the software deployed there, see [the LAN overview](../docs/lan-overview.md).
|
||||
| https://synapse.chans.xyz | Synapse API | Client-Server + Federation API |
|
||||
| https://account.chans.xyz | MAS | Matrix Authentication Service (local passwords) |
|
||||
| https://admin.chans.xyz | Element Admin | Admin console (MAS admin auth) |
|
||||
| https://plane.chans.xyz | Plane | Project management (Helm `plane-ce` v1.4.1, ns `plane`) |
|
||||
| `mrtc.chans.xyz` | MatrixRTC | **Reserved** – not deployed |
|
||||
|
||||
@@ -16,10 +16,12 @@ runbooks start from [`_template.md`](_template.md).
|
||||
| PowerDNS health check | [pdns-health.md](pdns-health.md) | read-only |
|
||||
| RustDesk health check | [rustdesk-health.md](rustdesk-health.md) | read-only |
|
||||
| Matrix health check | [matrix-health.md](matrix-health.md) | read-only |
|
||||
| Plane health check | [plane-health.md](plane-health.md) | read-only |
|
||||
| pgdb health check | [pgdb-health.md](pgdb-health.md) | read-only |
|
||||
| pgdb DB restore (pg_restore) | [pgdb-restore.md](pgdb-restore.md) | change (procedure) |
|
||||
| pgdb image/compose update | [pgdb-update.md](pgdb-update.md) | change (gated) |
|
||||
| AdGuard Home health check | [adguard-home-health.md](adguard-home-health.md) | read-only |
|
||||
| Host disk cleanup (logs/apt/docker) | [host-disk-cleanup.md](host-disk-cleanup.md) | change (gated) |
|
||||
| Matter packet capture | [matter-packet-capture.md](matter-packet-capture.md) | read-only |
|
||||
| Home Assistant maintenance | [home-assistant-maintenance.md](home-assistant-maintenance.md) | change (gated) |
|
||||
| matrix_e2ee integration update | [matrix-e2ee-update.md](matrix-e2ee-update.md) | change (gated) |
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Runbook: Home Assistant maintenance (hass.windy.lan)
|
||||
|
||||
Target: [hass.windy.lan](../hosts/hass.windy.lan.md) (physical x88 Pro box, HAOS `machine: green`)
|
||||
Upstream: HAOS 18.1 / Core 2026.8.1 / Supervisor 2026.07.5 (verified 2026-08-14)
|
||||
Upstream: HAOS 18.2 / Supervisor 2026.09.0 / Core 2026.9.1 (verified 2026-09-13)
|
||||
|
||||
This runbook covers routine Home Assistant maintenance through the **`ha`
|
||||
supervisor CLI**. All commands are wrapped by a single script
|
||||
@@ -284,8 +284,10 @@ print(urllib.request.urlopen(req, timeout=60).status)
|
||||
|
||||
7. **Do not edit the dashboard or `templates/csg_sensors.yaml` for an
|
||||
install.** Entity IDs did not change across v1.3.0/v1.3.1/v1.3.2.
|
||||
Template `| float(0)` turns native `unavailable` into fake zeros; that
|
||||
is a follow-up, not part of the zip install.
|
||||
(The old `| float(0)` fake-zero follow-up was resolved 2026-08-29 by
|
||||
W1N-239: template sensors now carry `availability` templates and show
|
||||
`unavailable` instead of fake zeros when native CSG sensors are down.
|
||||
Template edits go through that issue, not the install path.)
|
||||
|
||||
#### Verify (CSG, after v1.3.2 / W1N-118)
|
||||
|
||||
@@ -305,6 +307,40 @@ There is no long-lived `HA_TOKEN` in the agent environment. Read entity
|
||||
states via Supervisor (`SUPERVISOR_TOKEN` after `sudo -n -i`) at
|
||||
`http://supervisor/core/api/states/<entity_id>`.
|
||||
|
||||
#### CSG display refactor 2026-09-04 (VPS-90)
|
||||
|
||||
Template/dashboard changes made **after** pricing cross-check (8月账单
|
||||
198.64 元 vs 模板 198.65 元,≤0.01 元;阶梯常量 0.589/0.639/0.889、
|
||||
260/600 夏档未动):
|
||||
|
||||
- `templates/csg_sensors.yaml` Block B 新增
|
||||
`sensor.csg_this_month_avg_price`(本月阶梯电费÷本月用电,`元/kWh`);
|
||||
**csg_* template sensors = 15**。
|
||||
- Panel `power-monitor`(`lovelace.dashboard_unknown`):环比行改名
|
||||
「环比上月同期」;glance「本月/上月」去重为单卡「上月」(本月行归
|
||||
💰核心数据卡);⚡阶梯电价卡加「本月实际均价」行。实体引用 20→21。
|
||||
- `automations.yaml` +2 提醒:`automation.csg_mian_ban_qie_dong_ji_dang_ti_xing`
|
||||
(10-25)/ `automation.csg_mian_ban_qie_xia_ji_dang_ti_xing`(4-25)
|
||||
09:00 Matrix 提醒人工切「本月累计」gauge 季节档(max/segments 不可模板化)。
|
||||
- 金额单位混排(原生 CNY vs 模板 元)**保留**:`config/entity_registry/update`
|
||||
拒绝自定义文本单位(`extra keys not allowed … Got '元'`),已定案接受。
|
||||
|
||||
**WS 改面板(2026.8,本机实测,后续沿用)**: core/主机 python 无 ws 库、
|
||||
core 容器内经 supervisor 代理 WS 被拒(loop prevention)。用
|
||||
|
||||
```bash
|
||||
ssh -o BatchMode=yes hassio@hass.windy.lan \
|
||||
'sudo -n -i sh -c "docker run --rm -i --network host -e SUPERVISOR_TOKEN \
|
||||
--entrypoint python3 r.hassbus.com/home-assistant/aarch64-hassio-supervisor:2026.08.0 \
|
||||
- < /tmp/x.py"'
|
||||
```
|
||||
|
||||
连 `ws://172.30.32.2/core/websocket`(aiohttp,header `Authorization: Bearer
|
||||
$SUPERVISOR_TOKEN`,随后 auth 帧同 token)。命令名 **`lovelace/config`**(读)
|
||||
+ **`lovelace/config/save`**(写,url_path + 全量 config);`lovelace/config/get`
|
||||
已不存在(unknown_command)。备份与细节见
|
||||
[hosts/hass.windy.lan.md](../hosts/hass.windy.lan.md) § CSG 面板重构 2026-09-04。
|
||||
|
||||
## Command families intentionally NOT scripted
|
||||
|
||||
These exist in `ha` but are either rare, dangerous, or better done in the web
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
# Runbook: Host disk cleanup (unbounded container logs / apt cache / docker artifacts)
|
||||
|
||||
## Purpose
|
||||
|
||||
Reclaim space on a root filesystem that is filling up (≥70% used) on a Docker
|
||||
Compose host, by fixing unbounded container log growth at the source, clearing
|
||||
apt/journal caches, and removing unused Docker images/volumes. Success: root
|
||||
usage drops to a safe band (≤55% used, or per acceptance in the tracking issue)
|
||||
and log growth stays bounded afterwards.
|
||||
|
||||
## Scope
|
||||
|
||||
- 适用环境: production single-root-fs hosts running Docker Compose stacks
|
||||
(first application: `hk2.chans.xyz`; reusable for `mx2.windy.me` / `us2.wsvc.info`
|
||||
which run the same unbounded-`json.log` pattern).
|
||||
- 适用对象: root filesystem usage; container stdout/stderr log files
|
||||
(`/var/lib/docker/containers/*/*-json.log`); `/var/cache/apt`; systemd journal;
|
||||
unused Docker images / anonymous volumes / build cache.
|
||||
- 不适用情形: hosts without systemd-journald or without Docker; LAN/HAOS hosts
|
||||
(use their own runbooks); cases needing disk *growth* (provider resize) rather
|
||||
than cleanup; anything touching service data volumes or `/opt/*` configs
|
||||
(STOP and use the service-specific runbook instead).
|
||||
|
||||
## Ownership
|
||||
|
||||
- Owner: windy (operator) + agent executing per approval
|
||||
- Last reviewed: 2026-09-02
|
||||
- Related systems: hk2.chans.xyz (PowerDNS auth / AdGuard Home / Traefik / RustDesk compose stacks)
|
||||
|
||||
## Preconditions
|
||||
|
||||
- SSH access to the target host with **passwordless sudo** (`sudo -n true` must succeed).
|
||||
- A recorded `df -h` baseline and `docker system df` baseline.
|
||||
- **Explicit user approval** for every service touch listed in Approval gates
|
||||
(recorded in the tracking issue, e.g. Plane `vps` VPS-81).
|
||||
- No open incident on the target host.
|
||||
- Container log growth root cause identified in Diagnose before mutating.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Source | Required | Validation |
|
||||
|---|---:|---|
|
||||
| Target host | inventory/hosts.md | yes | SSH login + `uname -r` |
|
||||
| df/docker baseline | live read-only probe | yes | recorded before first mutation |
|
||||
| Approved service touches | user confirmation in tracking issue | yes | issue comment states approval |
|
||||
| Image keep-list (rollback pins) | operator decision in issue | yes | review `docker image ls` before rmi |
|
||||
| Backup of any config edited | local copy with timestamp | yes | exists before edit |
|
||||
|
||||
## Safety
|
||||
|
||||
### Non-negotiable rules
|
||||
|
||||
- Prefer read-only diagnosis before mutation (never mutate on an unmeasured disk).
|
||||
- Never use `rm` on a live container log — use `truncate -s 0` (keeps the fd valid).
|
||||
- Never run `docker image prune -a` when a keep-list is intended — no keep-list
|
||||
exists; delete explicitly with `docker rmi`.
|
||||
- Never run `docker volume prune -a` — plain `docker volume prune` (no `-a`)
|
||||
removes only unused anonymous volumes; named/in-use volumes stay.
|
||||
- After every mutation, verify the expected state (`df -h`, container status).
|
||||
- Destructive actions require explicit approval (Approval gates).
|
||||
|
||||
### Stop conditions
|
||||
|
||||
- Live state conflicts with this runbook's preconditions or expectations (e.g.
|
||||
root usage differs wildly from baseline, or a container is unhealthy).
|
||||
- Missing approval, missing backup, or missing rollback ability.
|
||||
- A verification step fails with no documented next step.
|
||||
- Any step would touch a volume/container/mount that is not on the approved list.
|
||||
|
||||
### Approval gates
|
||||
|
||||
| Action | Risk | Explicit approval | Approval record |
|
||||
|---|---:|---|---|
|
||||
| `docker restart <chatty container>` | low (sec-level blip of that service only) | yes | tracking issue (VPS-81 T1) |
|
||||
| `systemctl restart systemd-journald` | low (sec-level, no state loss) | yes | tracking issue (VPS-81 T2) |
|
||||
| `apt-get clean` | low (re-downloadable) | no | — |
|
||||
| `journalctl --vacuum-*` / journald drop-in | low | no (restart above is gated) | — |
|
||||
| `docker rmi` of unused images | medium (rollback pin removed unless kept) | yes (keep-list) | tracking issue (VPS-81 T3) |
|
||||
| `docker volume prune` | medium (data in anonymous volumes lost) | yes | tracking issue (VPS-81 T4) |
|
||||
| `docker builder prune` | low | no | — |
|
||||
|
||||
## Procedure
|
||||
|
||||
### Step 1 — Diagnose
|
||||
|
||||
**Action**
|
||||
|
||||
Read-only: `df -h`, `df -i`, `sudo du -x -h --max-depth=1 /`, `docker system df`,
|
||||
and locate oversized container logs:
|
||||
`sudo ls -la /var/lib/docker/containers/*/*-json.log`. Map a big log to its
|
||||
container (`docker inspect -f '{{.Name}} {{.LogPath}}' <id>`), then inspect what
|
||||
it logs (`sudo tail -c 400000 <logpath>`; count `[debug]` lines) and find the
|
||||
config flag driving it (e.g. AGH `log.verbose` in its YAML; note `log.file: ""`
|
||||
means the app's own rotation keys are inert and output goes to the container log).
|
||||
|
||||
**Expected**
|
||||
|
||||
A full accounting of root usage and identification of: (a) any unbounded
|
||||
container log and its root-cause flag; (b) reclaimable apt cache; (c) journal
|
||||
size and journald limits; (d) unused images (0 dangling expected) and unused
|
||||
anonymous volumes.
|
||||
|
||||
**Decision**
|
||||
|
||||
- If root is ≥70% used or any container log is unbounded → Step 2.
|
||||
- If root is healthy and logs are bounded → STOP (no change needed; record evidence).
|
||||
- If state conflicts with expectations (e.g. missing sudo, unexpected mount) → STOP.
|
||||
|
||||
### Step 2 — Fix noisy container logging at the source, then truncate
|
||||
|
||||
**Action**
|
||||
|
||||
1. Back up the app config: `sudo cp <config> <config>.bak-YYYYMMDD-<issue>`.
|
||||
2. Disable the debug/verbose flag (e.g. `log.verbose: true → false` in the AGH YAML).
|
||||
3. Apply config with a container restart: `docker restart <container>` (config-level
|
||||
change; **no recreate** needed and daemon.json rotation would not apply anyway).
|
||||
4. Truncate the accumulated logs: `sudo truncate -s 0 <json.log>` for the chatty
|
||||
container(s) (and any other oversized ones, e.g. traefik).
|
||||
5. Record `df -h` before/after.
|
||||
|
||||
**Expected**
|
||||
|
||||
`docker logs <container>` no longer shows the `[debug]` flood; the `*-json.log`
|
||||
stops growing; several GB reclaimed.
|
||||
|
||||
**Verification**
|
||||
|
||||
- `sudo tail -c 200000 <json.log>` after ≥1 minute → no new debug lines.
|
||||
- `df -h` improvement recorded.
|
||||
- Container still `Up (healthy)`.
|
||||
|
||||
**Rollback**
|
||||
|
||||
- Trigger: log volume unchanged, service degraded, or debug output is actually needed.
|
||||
- Action: restore the config backup and `docker restart <container>`.
|
||||
- Verify: original verbose behaviour back; container healthy.
|
||||
|
||||
### Step 3 — Clear apt cache and cap journald
|
||||
|
||||
**Action**
|
||||
|
||||
1. `sudo apt-get clean` (clears only `/var/cache/apt/archives`; `/var/lib/apt/lists`
|
||||
is not cleared by it and regenerates on `apt update` — optional/low value, skip).
|
||||
2. `sudo journalctl --vacuum-size=100M`.
|
||||
3. Write drop-in `/etc/systemd/journald.conf.d/00-disk-<issue>.conf`:
|
||||
`[Journal]` + `SystemMaxUse=200M`.
|
||||
4. `sudo systemctl restart systemd-journald` (approved service touch).
|
||||
5. Record `df -h` before/after.
|
||||
|
||||
**Expected**
|
||||
|
||||
Archives cleared (~1.4G on hk2), journal ≤100M, future journal capped at 200M.
|
||||
|
||||
**Verification**
|
||||
|
||||
- `du -sh /var/cache/apt/archives` → ~0.
|
||||
- `journalctl --disk-usage` → ≤100M.
|
||||
- `systemctl show systemd-journald -p ...` or restart log confirms new limit;
|
||||
`journalctl -b` still readable.
|
||||
|
||||
**Rollback**
|
||||
|
||||
- Trigger: journald fails to start or logs lost unexpectedly.
|
||||
- Action: remove the drop-in, `sudo systemctl restart systemd-journald`.
|
||||
- Verify: journald active, prior journal entries still listed.
|
||||
|
||||
### Step 4 — Remove unused Docker images (explicit keep-list)
|
||||
|
||||
**Action**
|
||||
|
||||
1. Enumerate unused images: `docker image ls` cross-checked against the images of
|
||||
running containers (`docker ps --format '{{.Image}}'`). Re-enumerate at
|
||||
execution time — the list drifts.
|
||||
2. Present the exact removal list to the operator; keep the agreed rollback pin(s)
|
||||
(e.g. `powerdns/pdns-auth-50:5.0.5`) and delete the rest explicitly:
|
||||
`docker rmi <repo:tag> ...` (per image).
|
||||
3. Record `df -h` before/after.
|
||||
|
||||
**Expected**
|
||||
|
||||
Only in-use images + kept pins remain; ~1–2.5G reclaimed (reclaim is an upper
|
||||
bound — layers shared with kept images are not freed; measure with `df`, do not
|
||||
promise the estimate).
|
||||
|
||||
**Verification**
|
||||
|
||||
- `docker image ls` shows only the expected set.
|
||||
- `docker system df` images reclaimable ≈ 0 for the removed set.
|
||||
- All containers still `Up`.
|
||||
|
||||
**Rollback**
|
||||
|
||||
- Trigger: an image that was actually needed was removed.
|
||||
- Action: re-pull it from the registry (`docker pull <repo:tag>`); if a kept pin
|
||||
must change, update the compose pin and `up -d`.
|
||||
- Verify: image present; affected service healthy.
|
||||
|
||||
### Step 5 — Remove unused anonymous volumes and build cache
|
||||
|
||||
**Action**
|
||||
|
||||
1. Enumerate volumes: `docker volume ls`, and confirm which are referenced by
|
||||
containers (`docker inspect` Mounts). Expected targets: anonymous volumes with
|
||||
no container reference.
|
||||
2. `docker volume prune` (**no `-a`**) — engine ≥ v23 removes only unused
|
||||
anonymous volumes; in-use volumes (e.g. PG data) are protected by container
|
||||
references in every version.
|
||||
3. `docker builder prune -f`.
|
||||
4. Record `df -h` before/after.
|
||||
|
||||
**Expected**
|
||||
|
||||
Unused anonymous volumes (~1.2G on hk2) and build cache gone; in-use volumes intact.
|
||||
|
||||
**Verification**
|
||||
|
||||
- `docker volume ls` shows only in-use volumes.
|
||||
- Services that own volumes (e.g. postgres) report healthy and data present.
|
||||
- `df -h` improvement recorded.
|
||||
|
||||
**Rollback**
|
||||
|
||||
- Trigger: data loss suspected in a removed volume.
|
||||
- Action: restore from backup if the volume ever contained data; verify against
|
||||
the pre-prune enumeration (targets must be anonymous + unreferenced before prune).
|
||||
- Note: this is why target enumeration is recorded before pruning.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Troubleshooting A — Log still grows after disabling verbose
|
||||
|
||||
- Evidence: `sudo tail -c 200000 <json.log>` still shows new lines; app config re-checked.
|
||||
- Allowed actions: check for a second verbose source (container entrypoint flags,
|
||||
other apps in the same log); check `docker inspect <c> --format '{{.HostConfig.LogConfig}}'`.
|
||||
- Next step: back to Step 2 or STOP if a container-level log-opts change (recreate)
|
||||
would be needed — that is a separate approval.
|
||||
|
||||
### Troubleshooting B — `docker rmi` fails (image in use)
|
||||
|
||||
- Evidence: `image is being used by stopped container ...`.
|
||||
- Allowed actions: identify the stopped container (`docker ps -a`); confirm it is
|
||||
not needed; remove it only with explicit approval.
|
||||
- Next step: re-run rmi for the remaining images; never force-delete blindly.
|
||||
|
||||
### Troubleshooting C — `docker volume prune` would remove more than expected
|
||||
|
||||
- Evidence: prune dry-run/listing includes a named or referenced volume.
|
||||
- Allowed actions: abort; do not add `-a`; re-check references.
|
||||
- Next step: STOP and report to the operator with the enumeration.
|
||||
|
||||
## Final Verification
|
||||
|
||||
The flow is successful only when all of the following hold:
|
||||
|
||||
- `df -h` root usage is in the agreed band (VPS-81: 76% → ≤55% used; measure, do not assume).
|
||||
- `docker system df` shows reclaimable ≈ 0 for images/volumes targeted.
|
||||
- All containers `Up` (health checks pass); public services verified
|
||||
(`dig @<host-ip> SOA <zone>` for DNS hosts; service URLs reachable).
|
||||
- Tracking issue updated with before/after `df`, actions, and the one-week
|
||||
observation checkpoint for log growth.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
If the flow cannot complete:
|
||||
|
||||
1. Stop further mutation.
|
||||
2. Collect command output, timestamps, and the exact step that failed.
|
||||
3. Record completed steps, actual results, unmet expectations, and whether a
|
||||
rollback ran.
|
||||
4. Hand over per the tracking issue with evidence; do not guess further.
|
||||
|
||||
## References
|
||||
|
||||
- Plane `vps` issue VPS-81 "hk2: 释放根盘空间" (+ subtasks VPS-82…88) — plan, review findings, approvals.
|
||||
- [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) — host facts.
|
||||
- [RUNBOOKS.md](../RUNBOOKS.md) — runbook spec; [runbooks/README.md](README.md) — index.
|
||||
@@ -0,0 +1,93 @@
|
||||
# Runbook: Plane Health Check
|
||||
|
||||
## Purpose
|
||||
|
||||
Read-only health check of the self-hosted Plane project-management instance
|
||||
(`plane.chans.xyz`) running on the synapse K3s cluster.
|
||||
|
||||
## Scope
|
||||
|
||||
- Applicable: [synapse.chans.xyz](../hosts/synapse.chans.xyz.md), namespace `plane`.
|
||||
- Read-only: does not change pods, ingress, certificates, secrets, or configuration.
|
||||
- Not applicable: Plane upgrade, values changes, or data recovery — those need a
|
||||
reviewed change (see `ansible-operations.md` / `release.md`).
|
||||
|
||||
## Ownership
|
||||
|
||||
- Owner: personal ops (Windy)
|
||||
- Last reviewed: 2026-09-02
|
||||
- Related systems: synapse.chans.xyz (Helm `plane-app`, chart `plane-ce-1.8.0`, app `v1.4.1`)
|
||||
|
||||
## Pass criteria
|
||||
|
||||
All of the following must hold; any conflict means `STOP` and record evidence.
|
||||
|
||||
- `sudo helm list -A` shows `plane-app` in ns `plane`, STATUS `deployed`.
|
||||
- All 7 Deployments + 4 StatefulSets in ns `plane` are `1/1 Running` with 0 recent restarts.
|
||||
- Init Jobs `api-migrate-*` / `minio-bucket-*` are `Completed`.
|
||||
- Certificate `plane-app-ssl-cert` is `READY=True` (CN `plane.chans.xyz`).
|
||||
- `https://plane.chans.xyz/` returns HTTP 200 with a valid Let's Encrypt cert.
|
||||
- Root disk usage below the 80% warning threshold.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Release and workloads
|
||||
|
||||
```bash
|
||||
ssh -4 windy@synapse.chans.xyz 'sudo helm list -A'
|
||||
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get deploy,sts,pods -o wide'
|
||||
```
|
||||
|
||||
Expected: `plane-app` deployed; all workloads `1/1 Running`, `RESTARTS` low;
|
||||
no `CrashLoopBackOff` / `Evicted`. Otherwise `STOP` and record evidence.
|
||||
|
||||
### 2. TLS certificate and ingress
|
||||
|
||||
```bash
|
||||
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get certificate,issuer,ingressroute'
|
||||
```
|
||||
|
||||
Expected: `plane-app-ssl-cert` `READY=True`; `plane-app-ingress` present with
|
||||
routes for `/`, `/api`, `/spaces`, `/god-mode`, `/live`, `/uploads`.
|
||||
|
||||
If `READY=False` with a pending HTTP-01 challenge, the usual cause is the node
|
||||
DNS chain (coreDNS → systemd-resolved → uplink) failing to resolve
|
||||
`plane.chans.xyz` — check `resolvectl query plane.chans.xyz` vs
|
||||
`dig +short plane.chans.xyz @8.8.8.8`. If the record exists publicly but the
|
||||
node fails, `sudo resolvectl flush-caches` and wait for the cert-manager retry;
|
||||
do not mutate the issuer.
|
||||
|
||||
### 3. Endpoint verification
|
||||
|
||||
```bash
|
||||
curl -4 -s -o /dev/null -w '%{http_code}\n' https://plane.chans.xyz/
|
||||
echo | openssl s_client -connect plane.chans.xyz:443 -servername plane.chans.xyz 2>/dev/null | openssl x509 -noout -subject -issuer -dates
|
||||
```
|
||||
|
||||
Expected: HTTP 200, cert `CN=plane.chans.xyz` issued by Let's Encrypt with a
|
||||
future `notAfter`. `https://plane.chans.xyz/api/` may 404 — the API serves
|
||||
under `/api/...` paths only, so a bare 404 there is not a failure.
|
||||
|
||||
### 4. Storage and disk
|
||||
|
||||
```bash
|
||||
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get pvc'
|
||||
ssh -4 windy@synapse.chans.xyz 'df -hP /'
|
||||
```
|
||||
|
||||
Expected: all 4 PVCs `Bound` (minio 5Gi, pgdb 5Gi, rabbitmq 100Mi, redis 100Mi
|
||||
on `local-path`); root disk < 80%.
|
||||
|
||||
## Safety
|
||||
|
||||
- Read-only: never mutate pods, ingress, certificates, or configuration during
|
||||
this check.
|
||||
- Plane data (ns `plane` Postgres + MinIO PVCs) has **no backup tier**; treat
|
||||
the instance as at-risk until a backup design exists.
|
||||
- If live state conflicts with the expected values above, `STOP` and record
|
||||
evidence; do not "fix in passing".
|
||||
|
||||
## References
|
||||
|
||||
- [hosts/synapse.chans.xyz.md](../hosts/synapse.chans.xyz.md) — plane stack facts
|
||||
- [matrix-health.md](matrix-health.md) — sibling service on the same cluster
|
||||
@@ -138,7 +138,7 @@ fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
say "== 4. Runbook spec compliance (RUNBOOKS.md) =="
|
||||
PROCEDURE_RUNBOOKS="fix-ci.md issue-to-merge.md network-change.md network-recovery.md pgdb-restore.md release.md rollback.md"
|
||||
PROCEDURE_RUNBOOKS="fix-ci.md host-disk-cleanup.md issue-to-merge.md network-change.md network-recovery.md pgdb-restore.md release.md rollback.md"
|
||||
GATED_REF_RUNBOOKS="mailcow-update.md ansible-operations.md home-assistant-maintenance.md matrix-e2ee-update.md pgdb-update.md vaultwarden-sqlite-to-postgres.md"
|
||||
RUNBOOK_INDEX="runbooks/README.md"
|
||||
for f in runbooks/*.md; do
|
||||
|
||||
Reference in New Issue
Block a user