Initial VPS operations handbook

2026-08-03 12:26:42 +08:00
commit b73125e5bc
97 changed files with 3641 additions and 0 deletions
@@ -0,0 +1,96 @@
+# Runbook: Vaultwarden health (us2)
+
+Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)  
+Path: `/opt/vaultwarden`  
+URL: https://auth.wsvc.info/  
+Upstream: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
+
+## 1. Containers
+
+```bash
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose ps -a'
+```
+
+Expect `vaultwarden` + `vw-db` **Up (healthy)**. `pgweb` should be stopped unless started with `--profile debug`.
+
+## 2. Live DB is Postgres
+
+```bash
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden sh -c "printenv DATABASE_URL" | sed -E "s#://[^@]+@#://REDACTED@#"'
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T pg psql -U vaultwarden -d vaultwarden -c "SELECT COUNT(*) AS users FROM users; SELECT COUNT(*) AS ciphers FROM ciphers;"'
+```
+
+## 3. Effective config (config.json wins)
+
+```bash
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden cat /data/config.json' \
+  | python3 -c "import sys,json;d=json.load(sys.stdin);print({k:d.get(k) for k in ['domain','smtp_host','smtp_port','smtp_security','smtp_username','smtp_from','ip_header','signups_allowed']});print('smtp_password_set',bool(d.get('smtp_password')))"
+```
+
+Expect SMTP **mx2.windy.me** / **587** / **starttls**, `ip_header` **X-Forwarded-For**.
+
+Also confirm `.env` and `config.json` SMTP passwords match (lengths/hashes only — never print secrets):
+
+```bash
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
+import json, re, pathlib, hashlib, subprocess
+env = pathlib.Path(".env").read_text()
+def grab(k):
+    m = re.search(rf"^{re.escape(k)}=(.*)$", env, re.M)
+    return m.group(1).strip().strip("\"'\''") if m else None
+cfg = json.loads(subprocess.check_output(
+    ["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
+ep, cp = grab("SMTP_PASSWORD"), cfg.get("smtp_password") or ""
+def fp(s): return {"len": len(s), "sha256_8": hashlib.sha256(s.encode()).hexdigest()[:8]}
+print("env_pass", fp(ep or ""), "cfg_pass", fp(cp), "match", ep == cp)
+EOF'
+```
+
+If they diverge, **auth uses `config.json`** — sync password there (or via `/admin`) and update `.smtp-credentials`.
+
+## 4. External HTTPS
+
+```bash
+curl -4 -sS -I --max-time 15 https://auth.wsvc.info/ | head -15
+```
+
+If **404** right after recreating VW: `ssh -4 windy@us2.wsvc.info 'docker restart traefik'` then retry.
+
+## 5. SMTP reachability + AUTH
+
+TCP (expect `587:0`; `465` often times out from us2):
+
+```bash
+ssh -4 windy@us2.wsvc.info 'docker compose -f /opt/vaultwarden/docker-compose.yml exec -T vaultwarden sh -c "timeout 5 bash -c \"cat </dev/null >/dev/tcp/mx2.windy.me/587\"; echo 587:\$?"'
+```
+
+AUTH with the **effective** `config.json` password (run on us2; do not print the password):
+
+```bash
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
+import json, smtplib, ssl, subprocess
+cfg = json.loads(subprocess.check_output(
+    ["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
+host, port = cfg["smtp_host"], int(cfg["smtp_port"])
+user, pw = cfg["smtp_username"], cfg["smtp_password"]
+ctx = ssl.create_default_context()
+with smtplib.SMTP(host, port, timeout=15) as s:
+    s.ehlo(); s.starttls(context=ctx); s.ehlo(); s.login(user, pw)
+print("AUTH_OK", host, port, user)
+EOF'
+```
+
+Expect `AUTH_OK`. `535` usually means stale password in `config.json` (see step 3).
+
+## Scheduled local check
+
+The sanitized Ansible health profile is `vaultwarden` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses only server-side credentials for the SMTP AUTH probe. It does not modify Vaultwarden, Traefik, SMTP, or secrets.
+
+
+## Pass criteria
+
+- Healthy compose; `DATABASE_URL` points at `pg`
+- HTTPS 200 + `/admin` 200; PG counts sane
+- Effective SMTP 587/starttls; `.env` ↔ `config.json` password match; **AUTH_OK**
+- Wrong-password login returns 400 business error (not 500)
+- Update **Verified** on [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md)