Runbook: Vaultwarden health (us2)

Target: us2.wsvc.info
Path: /opt/vaultwarden
URL: https://auth.wsvc.info/
Upstream: docs/vaultwarden-upstream.md

1. Containers

ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose ps -a'

Expect vaultwarden + vw-db Up (healthy). pgweb should be stopped unless started with --profile debug.

2. Live DB is Postgres

ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden sh -c "printenv DATABASE_URL" | sed -E "s#://[^@]+@#://REDACTED@#"'
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T pg psql -U vaultwarden -d vaultwarden -c "SELECT COUNT(*) AS users FROM users; SELECT COUNT(*) AS ciphers FROM ciphers;"'

3. Effective config (config.json wins)

ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden cat /data/config.json' \
  | python3 -c "import sys,json;d=json.load(sys.stdin);print({k:d.get(k) for k in ['domain','smtp_host','smtp_port','smtp_security','smtp_username','smtp_from','ip_header','signups_allowed']});print('smtp_password_set',bool(d.get('smtp_password')))"

Expect SMTP mx2.windy.me / 587 / starttls, ip_header X-Forwarded-For.

Also confirm .env and config.json SMTP passwords match (lengths/hashes only — never print secrets):

ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
import json, re, pathlib, hashlib, subprocess
env = pathlib.Path(".env").read_text()
def grab(k):
    m = re.search(rf"^{re.escape(k)}=(.*)$", env, re.M)
    return m.group(1).strip().strip("\"'\''") if m else None
cfg = json.loads(subprocess.check_output(
    ["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
ep, cp = grab("SMTP_PASSWORD"), cfg.get("smtp_password") or ""
def fp(s): return {"len": len(s), "sha256_8": hashlib.sha256(s.encode()).hexdigest()[:8]}
print("env_pass", fp(ep or ""), "cfg_pass", fp(cp), "match", ep == cp)
EOF'

If they diverge, auth uses config.json — sync password there (or via /admin) and update .smtp-credentials.

4. External HTTPS

curl -4 -sS -I --max-time 15 https://auth.wsvc.info/ | head -15

If 404 right after recreating VW: ssh -4 windy@us2.wsvc.info 'docker restart traefik' then retry.

5. SMTP reachability + AUTH

TCP (expect 587:0; 465 often times out from us2):

ssh -4 windy@us2.wsvc.info 'docker compose -f /opt/vaultwarden/docker-compose.yml exec -T vaultwarden sh -c "timeout 5 bash -c \"cat </dev/null >/dev/tcp/mx2.windy.me/587\"; echo 587:\$?"'

AUTH with the effective config.json password (run on us2; do not print the password):

ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
import json, smtplib, ssl, subprocess
cfg = json.loads(subprocess.check_output(
    ["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
host, port = cfg["smtp_host"], int(cfg["smtp_port"])
user, pw = cfg["smtp_username"], cfg["smtp_password"]
ctx = ssl.create_default_context()
with smtplib.SMTP(host, port, timeout=15) as s:
    s.ehlo(); s.starttls(context=ctx); s.ehlo(); s.login(user, pw)
print("AUTH_OK", host, port, user)
EOF'

Expect AUTH_OK. 535 usually means stale password in config.json (see step 3).

Scheduled local check

The sanitized Ansible health profile is vaultwarden (ansible/playbooks/healthchecks.yml). It runs locally through vps-healthcheck.timer, writes a sanitized JSON result to /var/lib/vps-health/latest.json, and uses only server-side credentials for the SMTP AUTH probe. It does not modify Vaultwarden, Traefik, SMTP, or secrets.

Pass criteria

Healthy compose; DATABASE_URL points at pg
HTTPS 200 + /admin 200; PG counts sane
Effective SMTP 587/starttls; .env ↔ config.json password match; AUTH_OK
Wrong-password login returns 400 business error (not 500)
Update Verified on hosts/us2.wsvc.info.md

4.0 KiB Raw Blame History