Runbook: Vaultwarden health (us2)

Target: us2.wsvc.info
Path: /opt/vaultwarden
URL: https://auth.wsvc.info/
Upstream: docs/vaultwarden-upstream.md

Prefer the Ansible health report (ansible/playbooks/health-report.yml with --limit vaultwarden) for the routine check. The SSH commands below are focused diagnostics for an already identified problem.

Focused SSH diagnostics

Use these only after the Ansible health report needs investigation.

1. Containers

ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose ps -a'

Expect vaultwarden + vw-db Up (healthy). pgweb should be stopped unless started with --profile debug.

2. Live DB is Postgres

ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden sh -c "printenv DATABASE_URL" | sed -E "s#://[^@]+@#://REDACTED@#"'
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T pg psql -U vaultwarden -d vaultwarden -c "SELECT COUNT(*) AS users FROM users; SELECT COUNT(*) AS ciphers FROM ciphers;"'

3. Effective config (config.json wins)

ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden cat /data/config.json' \
  | python3 -c "import sys,json;d=json.load(sys.stdin);print({k:d.get(k) for k in ['domain','smtp_host','smtp_port','smtp_security','smtp_username','smtp_from','ip_header','signups_allowed']});print('smtp_password_set',bool(d.get('smtp_password')))"

Expect SMTP mx2.windy.me / 587 / starttls, ip_header X-Forwarded-For.

Also confirm .env and config.json SMTP passwords match (lengths/hashes only — never print secrets):

ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
import json, re, pathlib, hashlib, subprocess
env = pathlib.Path(".env").read_text()
def grab(k):
    m = re.search(rf"^{re.escape(k)}=(.*)$", env, re.M)
    return m.group(1).strip().strip("\"'\''") if m else None
cfg = json.loads(subprocess.check_output(
    ["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
ep, cp = grab("SMTP_PASSWORD"), cfg.get("smtp_password") or ""
def fp(s): return {"len": len(s), "sha256_8": hashlib.sha256(s.encode()).hexdigest()[:8]}
print("env_pass", fp(ep or ""), "cfg_pass", fp(cp), "match", ep == cp)
EOF'

If they diverge, auth uses config.json — sync password there (or via /admin) and update .smtp-credentials.

4. External HTTPS

curl -4 -sS -I --max-time 15 https://auth.wsvc.info/ | head -15

If 404 right after recreating VW: use the Ansible Compose reconcile playbook with service_reconcile_restart_traefik=true, then retry.

5. SMTP reachability + AUTH

TCP (expect 587:0; 465 often times out from us2):

ssh -4 windy@us2.wsvc.info 'docker compose -f /opt/vaultwarden/docker-compose.yml exec -T vaultwarden sh -c "timeout 5 bash -c \"cat </dev/null >/dev/tcp/mx2.windy.me/587\"; echo 587:\$?"'

AUTH with the effective config.json password (run on us2; do not print the password):

ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
import json, smtplib, ssl, subprocess
cfg = json.loads(subprocess.check_output(
    ["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
host, port = cfg["smtp_host"], int(cfg["smtp_port"])
user, pw = cfg["smtp_username"], cfg["smtp_password"]
ctx = ssl.create_default_context()
with smtplib.SMTP(host, port, timeout=15) as s:
    s.ehlo(); s.starttls(context=ctx); s.ehlo(); s.login(user, pw)
print("AUTH_OK", host, port, user)
EOF'

Expect AUTH_OK. 535 usually means stale password in config.json (see step 3).

Scheduled local check

The sanitized Ansible health profile is vaultwarden (ansible/playbooks/healthchecks.yml). It runs locally through vps-healthcheck.timer, writes a sanitized JSON result to /var/lib/vps-health/latest.json, and uses only server-side credentials for the SMTP AUTH probe. It does not modify Vaultwarden, Traefik, SMTP, or secrets.

Pass criteria

Healthy compose; DATABASE_URL points at pg
HTTPS 200 + /admin 200; PG counts sane
Effective SMTP 587/starttls; .env ↔ config.json password match; AUTH_OK
Wrong-password login returns 400 business error (not 500)
Update Verified on hosts/us2.wsvc.info.md

4.4 KiB Raw Blame History