vps/runbooks/gitea-health.md

Gitea health (us2)

Purpose

Read-only health check of the /opt/gitea stack: web UI, git SSH, backup sidecar, and resource headroom.

Scope

• Applicable: us2.wsvc.info, /opt/gitea.
• Read-only: does not change Gitea configuration or service state.
• Not applicable: Traefik-wide issues (see hosts/us2 notes); Soft Serve rollback (see the rollback section of hosts/us2.wsvc.info.md).

Ownership

• Owner: personal ops (Windy)
• Last reviewed: 2026-09-19
• Related systems: us2.wsvc.info, Traefik (vw-net), repo.windy.me DNS

Facts: hosts/us2.wsvc.info.md. Reference compose mirror: compose/gitea/ (server files win).

Focused SSH diagnostics

From the agent shell use ssh -F /dev/null -4 windy@us2.wsvc.info.

Containers

ssh -F /dev/null -4 windy@us2.wsvc.info 'cd /opt/gitea && docker compose ps -a'

Expect gitea and gitea-backup both Up. soft-serve* containers are intentionally Exited (rollback retention, keep as-is).

Version / image pin

ssh -F /dev/null -4 windy@us2.wsvc.info \
  'docker exec gitea /usr/local/bin/gitea --version; grep "^ *image:" /opt/gitea/compose.yml'

Expect gitea version 1.27.3 and the digest pin gitea/gitea@sha256:1c17ecaead42… (full digest in compose.yml). If upstream has a newer 1.27.x patch with security fixes, plan an upgrade — do not auto-pull latest.

Web UI (from laptop / WSL)

curl -4sI https://repo.windy.me/ | head -3        # expect HTTP/2 200
echo | openssl s_client -connect repo.windy.me:443 -servername repo.windy.me 2>/dev/null \
  | openssl x509 -noout -dates                     # expect LE cert not expired

Git SSH (the critical check)

ssh -F /dev/null -o BatchMode=yes -p 2222 git@repo.windy.me 2>&1 | head -2

Expect Hi there, windy! You've successfully authenticated … and no shell. Login user must be git@ (windy@ fails: publickey denied). Clone URLs need the owner segment: ssh://git@repo.windy.me:2222/windy/<repo>.git. The host key is the reused Soft Serve ed25519 key (fingerprint prefix SHA256:PdxZRe74…) — a changed fingerprint means someone rotated it.

Registration / hardening (server-side, definitive)

ssh -F /dev/null -4 windy@us2.wsvc.info \
  'grep -E "DISABLE_REGISTRATION|INSTALL_LOCK" /opt/gitea/config/app.ini; stat -c "%a %U" /opt/gitea/config/app.ini'

Expect DISABLE_REGISTRATION = true, INSTALL_LOCK = true, file mode 600.

Backup freshness

ssh -F /dev/null -4 windy@us2.wsvc.info 'tail -4 /opt/gitea/backups/backup.log; ls -lt /opt/gitea/backups/ | head -4'

Expect a Backup OK line from today's 02:00 run (products include app.ini.tar.gz, gitea.db, repos.tar.gz) and ≤ 14 retained sets. A backup set older than ~25h with no log error means cron silently died — investigate before proceeding.

Logs and memory gate

ssh -F /dev/null -4 windy@us2.wsvc.info \
  'docker logs gitea --since 24h 2>&1 | grep -iE "error|panic" | tail -5; free -m | head -2'

Expect no recurring errors (single SSH auth-failure warnings from the Traefik proxy IP are background noise) and available ≥ 150 MB (VPS-94 memory gate; gitea idles ~400 MB RSS).

Safety

• Read-only: never mutate configuration or service state during this check.
• If live state conflicts with an expected value (wrong digest, HTTPS non-200, stale backup, registration enabled, available < 150 MB), STOP and report per repo rules.

After config changes

/opt/gitea has no Ansible coverage yet. Reviewed changes (compose edit, digest bump) are applied manually and verified:

ssh -F /dev/null -4 windy@us2.wsvc.info 'cd /opt/gitea && docker compose up -d && docker compose ps'

Then re-run the diagnostics above. Keep compose/gitea/ in sync (reference copy, server files win).

References

• hosts/us2.wsvc.info.md — deployment facts, rollback path
• Plane vps project VPS-94 — migration plan, acceptance checks, audit record
• Upstream: https://docs.gitea.com/ (self-hosted admin/config docs)