From e08e5d50deddbc44ac732b1c92f05b1928fcdb4b Mon Sep 17 00:00:00 2001 From: windyboy Date: Sat, 19 Sep 2026 14:22:05 +0800 Subject: [PATCH] =?UTF-8?q?docs(us2):=20gitea-health=20runbook=20+=20?= =?UTF-8?q?=E6=94=B6=E5=8F=A3=20Gitea=20=E6=96=87=E6=A1=A3=E6=BC=82?= =?UTF-8?q?=E7=A7=BB=20(Plane=20VPS-94)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - runbooks/gitea-health.md 新建: 只读健康检查(容器/版本pin/web/SSH/备份/内存闸门)+ AGENTS 与索引登记 - AGENTS.md: Gitea upstream 段(digest 升级流程/app.ini 600 含密钥)、SSH 示例改 git@、runbook 表加行 - inventory/hosts.md + hosts/us2: repo.windy.me SSH 用户改 git@; us2 角色行 Gitea 化 - hosts/us2: 仓库数修正 16→17(windy/13 含 vps、go-caatsm + cdia/4); Verified 补 2026-09-19 独立审核结果 - 本仓库 remote 已改写为 ssh://git@repo.windy.me:2222/windy/vps.git(VPS-94 Phase 3 漏项) --- AGENTS.md | 5 +- hosts/us2.wsvc.info.md | 6 +- inventory/hosts.md | 4 +- runbooks/README.md | 1 + runbooks/gitea-health.md | 123 +++++++++++++++++++++++++++++++++++++++ 5 files changed, 133 insertions(+), 6 deletions(-) create mode 100644 runbooks/gitea-health.md diff --git a/AGENTS.md b/AGENTS.md index afc5dc7..aff9e8c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,7 +23,7 @@ This repo is the **agent ops handbook + fact source** for maintaining personal V > so plain `ssh` aborts with `Bad owner or permissions on ...`. Always use > `ssh -F /dev/null` from the agent shell and pass options explicitly > (`~/.ssh/config` is skipped; e.g. `ssh -F /dev/null -p 2222 - > -i ~/.ssh/id_ed25519 windy@repo.windy.me`). `sudo` never works in the + > -i ~/.ssh/id_ed25519 git@repo.windy.me`). `sudo` never works in the > sandbox (`NoNewPrivs`, no capabilities, `/` read-only). The host itself is > healthy — to inspect or act on the real host from the sandbox use > `/mnt/c/WINDOWS/system32/wsl.exe -u root -- ` (real root: keep @@ -85,6 +85,8 @@ not maintain a second copy of the machine table here. **RustDesk:** Self-hosted RustDesk server on `hk2.chans.xyz` (`/opt/rustdesk`, containers `hbbs`/`hbbr`, image pinned `1.1.14`). The `hbbs -r` relay hostname must resolve to the host's public IP `154.36.174.161` — use `hk2.chans.xyz` (never `hk2.wsvc.info`, which has no DNS record). Health: [rustdesk-health](runbooks/rustdesk-health.md). +**Gitea (on us2):** Self-hosted Gitea at `repo.windy.me` (web HTTPS + git SSH `:2222`; clone `ssh://git@repo.windy.me:2222/windy/.git` — user must be `git@`, path needs the owner segment). Replaced Soft Serve 2026-09-18 (Plane VPS-94; `/opt/soft-serve` kept stopped as rollback). Before changing `/opt/gitea`, read [hosts/us2.wsvc.info.md](hosts/us2.wsvc.info.md). Image digest-pinned (`1.27.3-rootless`); upgrade = new digest in `compose.yml` + `docker compose up -d` + re-run health. `config/app.ini` is mode 600 and holds SECRET_KEY/INTERNAL_TOKEN — never commit; the backup sidecar includes it because restores need it. Health: [gitea-health](runbooks/gitea-health.md). + ## Runbooks & scripts | Task | Path | @@ -100,6 +102,7 @@ not maintain a second copy of the machine table here. | Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) | | Plane health | [runbooks/plane-health.md](runbooks/plane-health.md) | | RustDesk health (hk2) | [runbooks/rustdesk-health.md](runbooks/rustdesk-health.md) | +| Gitea health (us2) | [runbooks/gitea-health.md](runbooks/gitea-health.md) | | AdGuard Home health | [runbooks/adguard-home-health.md](runbooks/adguard-home-health.md) | | Host disk cleanup | [runbooks/host-disk-cleanup.md](runbooks/host-disk-cleanup.md) | | Home Assistant maintenance | [runbooks/home-assistant-maintenance.md](runbooks/home-assistant-maintenance.md) + [scripts/ha-maintenance.sh](runbooks/scripts/ha-maintenance.sh) | diff --git a/hosts/us2.wsvc.info.md b/hosts/us2.wsvc.info.md index 2647a57..304b2b7 100644 --- a/hosts/us2.wsvc.info.md +++ b/hosts/us2.wsvc.info.md @@ -2,7 +2,7 @@ | Item | Value | |------|--------| -| Role | Multi-service VPS (Vaultwarden, Traefik, Soft Serve, …) | +| Role | Multi-service VPS (Vaultwarden, Traefik, Gitea, …) | | SSH | `ssh -4 windy@us2.wsvc.info` (prefer IPv4 from WSL) | | IPv4 | `193.9.44.165` | | Also DNS | `auth.wsvc.info` → this host; `repo.windy.me` → this host (Gitea) | @@ -90,7 +90,7 @@ ansible-playbook playbooks/compose-reconcile.yml --limit vaultwarden \ - SSH: 内置 server 容器内 `:2322`(`SSH_LISTEN_PORT` 非特权), Traefik TCP entrypoint `ssh`(`:2222` → `gitea:2322`, `HostSNI(*)`, `tls=false`) on `vw-net`; clone URL `ssh://git@repo.windy.me:2222/windy/.git`(owner 段 `windy`) - **host key 复用 soft-serve**(`SSH_SERVER_HOST_KEYS=/secrets/soft_serve_host_ed25519`, ed25519, 指纹 `SHA256:PdxZRe74…`): 客户端 known_hosts 零变更; 仅公钥认证(密码认证未启用) - Web: `https://repo.windy.me`(Traefik websecure + letsencrypt); `DISABLE_REGISTRATION=true`, Actions 关闭; 管理员 `windy`(凭据仅存服务器 `/opt/gitea/.admin-credentials`, 勿入库/入 Plane) -- 仓库: 16 个(顶层 11 + `cdia/` 4 + `windyboy/go-caatsm`), 2026-09-18 自 soft-serve `push --mirror` 迁移, 逐仓 `ls-remote` ref 全集 + HEAD symref 两端一致; 可见性仅 `dotfiles-personal` private, 其余 public(与 soft-serve 现状一致) +- 仓库: 17 个(`windy/` 13 — 含 vps、自 `windyboy` 并入的 go-caatsm — + `cdia/` 4; 2026-09-19 复核), 2026-09-18 自 soft-serve `push --mirror` 迁移, 逐仓 `ls-remote` ref 全集 + HEAD symref 两端一致; 可见性仅 `dotfiles-personal` private, 其余 public(与 soft-serve 现状一致) - 备份: sidecar 每日 02:00 → `backups/gitea_/{app.ini.tar.gz, gitea.db, repos.tar.gz}`(app.ini 含恢复必需 SECRET_KEY), 03:00 prune 保留 14 份; 已验证手动备份产物 109.9M - 回滚: `/opt/soft-serve` 未删(compose stop + sidecar 停, 数据与旧备份冻结保留), 回滚 = Traefik `:2222` 指回 `soft-serve:23231` + 客户端 remote 回改旧无 owner 段路径; 观察 2–4 周后清理(历史: W1N-244~248) | `traefik` | Up | `traefik:v3.6.2` (`/opt/traefik`, public `:80`/`:443`) | @@ -102,7 +102,7 @@ Remaining running services on this host: `gitea`, `vaultwarden` stack, `traefik` ## Verified -Last checked: **2026-09-18** — operational; disk cleanup done (see note above, Plane vps VPS-93). Prior full probe: 2026-08-29. +Last checked: **2026-09-19** — operational. Gitea independently audited (read-only): `1.27.3` = latest 1.27.x patch; HTTPS 200 (LE cert → 2026-12-17); SSH `git@…:2222` key auth OK; backup 02:00 OK (110M); memory available 584 MB; logs clean (record: Plane VPS-94 comment 2026-09-19). Prior: 2026-09-18 migration + disk cleanup (VPS-93); 2026-08-29 full Vaultwarden probe. - `vaultwarden` + `vw-db` healthy; `DATABASE_URL` → `pg:5432/vaultwarden` - `https://auth.wsvc.info/` **200**, `/admin` **200**, `/api/config` OK (`disableUserRegistration: true`) diff --git a/inventory/hosts.md b/inventory/hosts.md index 7268f22..40adac4 100644 --- a/inventory/hosts.md +++ b/inventory/hosts.md @@ -15,9 +15,9 @@ the software deployed there, see [the LAN overview](../docs/lan-overview.md). | Host | Role | SSH | IPv4 | Ansible | Status | Facts | |------|------|-----|------|---------|--------|-------| | mx2.windy.me | mailcow (primary MX prio 20) | `ssh -4 windy@mx2.windy.me` | 194.163.160.244 | ✓ (mx2) | active | [hosts/mx2.windy.me.md](../hosts/mx2.windy.me.md) | -| us2.wsvc.info | Vaultwarden/Postgres (+ Traefik, Soft Serve, …) | `ssh -4 windy@us2.wsvc.info` | 193.9.44.165 | ✓ (us2) | active | [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md) | +| us2.wsvc.info | Vaultwarden/Postgres (+ Traefik, Gitea, …) | `ssh -4 windy@us2.wsvc.info` | 193.9.44.165 | ✓ (us2) | active | [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md) | | mx.windy.me | mail (secondary MX prio 30) | TBD | see AAAA/A | — (stub) | stub | — | -| repo.windy.me | Gitea git (on us2) | `ssh -p 2222 windy@repo.windy.me` | 193.9.44.165 | — (service on us2) | stub | see us2 | +| repo.windy.me | Gitea git (on us2) | `ssh -p 2222 git@repo.windy.me` | 193.9.44.165 | — (service on us2) | stub | see us2 | | auth.wsvc.info | Vaultwarden public hostname | — (HTTPS) | → us2 | — (alias) | active | see us2 | | us1.wsvc.info | PowerDNS secondary (ns2 host) | TBD | 202.91.35.141 | — (stub) | stub | Auth 5.0.5; see hk2 | | us4.wsvc.info | WireGuard VPN | `ssh -4 windy@us4.wsvc.info` | 185.201.226.122 | ✓ (us4) | active | [hosts/us4.wsvc.info.md](../hosts/us4.wsvc.info.md) | diff --git a/runbooks/README.md b/runbooks/README.md index e44b067..e8f2961 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -15,6 +15,7 @@ runbooks start from [`_template.md`](_template.md). | Vaultwarden SQLite→PG migrate | [vaultwarden-sqlite-to-postgres.md](vaultwarden-sqlite-to-postgres.md) | change (destructive) | | PowerDNS health check | [pdns-health.md](pdns-health.md) | read-only | | RustDesk health check | [rustdesk-health.md](rustdesk-health.md) | read-only | +| Gitea health check | [gitea-health.md](gitea-health.md) | read-only | | Matrix health check | [matrix-health.md](matrix-health.md) | read-only | | Plane health check | [plane-health.md](plane-health.md) | read-only | | pgdb health check | [pgdb-health.md](pgdb-health.md) | read-only | diff --git a/runbooks/gitea-health.md b/runbooks/gitea-health.md new file mode 100644 index 0000000..c967836 --- /dev/null +++ b/runbooks/gitea-health.md @@ -0,0 +1,123 @@ +# Gitea health (us2) + +## Purpose + +Read-only health check of the `/opt/gitea` stack: web UI, git SSH, backup +sidecar, and resource headroom. + +## Scope + +- Applicable: [us2.wsvc.info](../hosts/us2.wsvc.info.md), `/opt/gitea`. +- Read-only: does not change Gitea configuration or service state. +- Not applicable: Traefik-wide issues (see hosts/us2 notes); Soft Serve + rollback (see the rollback section of hosts/us2.wsvc.info.md). + +## Ownership + +- Owner: personal ops (Windy) +- Last reviewed: 2026-09-19 +- Related systems: us2.wsvc.info, Traefik (`vw-net`), repo.windy.me DNS + +Facts: [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md). Reference +compose mirror: [compose/gitea/](../compose/gitea/) (server files win). + +## Focused SSH diagnostics + +From the agent shell use `ssh -F /dev/null -4 windy@us2.wsvc.info`. + +### Containers + +```bash +ssh -F /dev/null -4 windy@us2.wsvc.info 'cd /opt/gitea && docker compose ps -a' +``` + +Expect `gitea` and `gitea-backup` both `Up`. `soft-serve*` containers are +intentionally `Exited` (rollback retention, keep as-is). + +### Version / image pin + +```bash +ssh -F /dev/null -4 windy@us2.wsvc.info \ + 'docker exec gitea /usr/local/bin/gitea --version; grep "^ *image:" /opt/gitea/compose.yml' +``` + +Expect `gitea version 1.27.3` and the digest pin +`gitea/gitea@sha256:1c17ecaead42…` (full digest in compose.yml). If upstream +has a newer 1.27.x patch with security fixes, plan an upgrade — do not +auto-pull `latest`. + +### Web UI (from laptop / WSL) + +```bash +curl -4sI https://repo.windy.me/ | head -3 # expect HTTP/2 200 +echo | openssl s_client -connect repo.windy.me:443 -servername repo.windy.me 2>/dev/null \ + | openssl x509 -noout -dates # expect LE cert not expired +``` + +### Git SSH (the critical check) + +```bash +ssh -F /dev/null -o BatchMode=yes -p 2222 git@repo.windy.me 2>&1 | head -2 +``` + +Expect `Hi there, windy! You've successfully authenticated …` and no shell. +Login user **must be `git@`** (`windy@` fails: publickey denied). Clone URLs +need the owner segment: `ssh://git@repo.windy.me:2222/windy/.git`. +The host key is the reused Soft Serve ed25519 key (fingerprint prefix +`SHA256:PdxZRe74…`) — a changed fingerprint means someone rotated it. + +### Registration / hardening (server-side, definitive) + +```bash +ssh -F /dev/null -4 windy@us2.wsvc.info \ + 'grep -E "DISABLE_REGISTRATION|INSTALL_LOCK" /opt/gitea/config/app.ini; stat -c "%a %U" /opt/gitea/config/app.ini' +``` + +Expect `DISABLE_REGISTRATION = true`, `INSTALL_LOCK = true`, file mode `600`. + +### Backup freshness + +```bash +ssh -F /dev/null -4 windy@us2.wsvc.info 'tail -4 /opt/gitea/backups/backup.log; ls -lt /opt/gitea/backups/ | head -4' +``` + +Expect a `Backup OK` line from today's 02:00 run (products include +`app.ini.tar.gz`, `gitea.db`, `repos.tar.gz`) and ≤ 14 retained sets. A +backup set older than ~25h with no log error means cron silently died — +investigate before proceeding. + +### Logs and memory gate + +```bash +ssh -F /dev/null -4 windy@us2.wsvc.info \ + 'docker logs gitea --since 24h 2>&1 | grep -iE "error|panic" | tail -5; free -m | head -2' +``` + +Expect no recurring errors (single SSH auth-failure warnings from the +Traefik proxy IP are background noise) and `available` ≥ 150 MB +(VPS-94 memory gate; gitea idles ~400 MB RSS). + +## Safety + +- Read-only: never mutate configuration or service state during this check. +- If live state conflicts with an expected value (wrong digest, HTTPS + non-200, stale backup, registration enabled, `available` < 150 MB), + `STOP` and report per repo rules. + +## After config changes + +`/opt/gitea` has no Ansible coverage yet. Reviewed changes +(compose edit, digest bump) are applied manually and verified: + +```bash +ssh -F /dev/null -4 windy@us2.wsvc.info 'cd /opt/gitea && docker compose up -d && docker compose ps' +``` + +Then re-run the diagnostics above. Keep [compose/gitea/](../compose/gitea/) +in sync (reference copy, server files win). + +## References + +- [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md) — deployment facts, rollback path +- Plane `vps` project VPS-94 — migration plan, acceptance checks, audit record +- Upstream: https://docs.gitea.com/ (self-hosted admin/config docs)