vps/runbooks/vaultwarden-health.md

# Runbook: Vaultwarden health (us2)

## Purpose

Read-only health check of the Vaultwarden stack (Postgres backend) on us2.

## Scope

- Applicable: [us2.wsvc.info](../hosts/us2.wsvc.info.md), `/opt/vaultwarden`.
- Read-only: does not change Vaultwarden, Traefik, SMTP, or secrets.

## Ownership

- Owner: personal ops (Windy)
- Last reviewed: 2026-08-17
- Related systems: us2.wsvc.info (`/opt/vaultwarden`)

Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)  
Path: `/opt/vaultwarden`  
URL: https://auth.wsvc.info/  
Upstream: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)

Prefer the Ansible health report (`ansible/playbooks/health-report.yml` with
`--limit vaultwarden`) for the routine check. The SSH commands below are
focused diagnostics for an already identified problem.

## Focused SSH diagnostics

Use these only after the Ansible health report needs investigation.

### 1. Containers

```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose ps -a'
```

Expect `vaultwarden` + `vw-db` **Up (healthy)**. `pgweb` should be stopped unless started with `--profile debug`.

### 2. Live DB is Postgres

```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden sh -c "printenv DATABASE_URL" | sed -E "s#://[^@]+@#://REDACTED@#"'
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T pg psql -U vaultwarden -d vaultwarden -c "SELECT COUNT(*) AS users FROM users; SELECT COUNT(*) AS ciphers FROM ciphers;"'
```

### 3. Effective config (config.json wins)

```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden cat /data/config.json' \
  | python3 -c "import sys,json;d=json.load(sys.stdin);print({k:d.get(k) for k in ['domain','smtp_host','smtp_port','smtp_security','smtp_username','smtp_from','ip_header','signups_allowed']});print('smtp_password_set',bool(d.get('smtp_password')))"
```

Expect SMTP **mx2.windy.me** / **587** / **starttls**, `ip_header` **X-Forwarded-For**.

Also confirm `.env` and `config.json` SMTP passwords match (lengths/hashes only — never print secrets):

```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
import json, re, pathlib, hashlib, subprocess
env = pathlib.Path(".env").read_text()
def grab(k):
    m = re.search(rf"^{re.escape(k)}=(.*)$", env, re.M)
    return m.group(1).strip().strip("\"'\''") if m else None
cfg = json.loads(subprocess.check_output(
    ["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
ep, cp = grab("SMTP_PASSWORD"), cfg.get("smtp_password") or ""
def fp(s): return {"len": len(s), "sha256_8": hashlib.sha256(s.encode()).hexdigest()[:8]}
print("env_pass", fp(ep or ""), "cfg_pass", fp(cp), "match", ep == cp)
EOF'
```

If they diverge, **auth uses `config.json`** — sync password there (or via `/admin`) and update `.smtp-credentials`.

### 4. External HTTPS

```bash
curl -4 -sS -I --max-time 15 https://auth.wsvc.info/ | head -15
```

If **404** right after recreating VW: use the Ansible Compose reconcile playbook
with `service_reconcile_restart_traefik=true`, then retry.

### 5. SMTP reachability + AUTH

TCP (expect `587:0`; `465` often times out from us2):

```bash
ssh -4 windy@us2.wsvc.info 'docker compose -f /opt/vaultwarden/docker-compose.yml exec -T vaultwarden sh -c "timeout 5 bash -c \"cat </dev/null >/dev/tcp/mx2.windy.me/587\"; echo 587:\$?"'
```

AUTH with the **effective** `config.json` password (run on us2; do not print the password):

```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
import json, smtplib, ssl, subprocess
cfg = json.loads(subprocess.check_output(
    ["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
host, port = cfg["smtp_host"], int(cfg["smtp_port"])
user, pw = cfg["smtp_username"], cfg["smtp_password"]
ctx = ssl.create_default_context()
with smtplib.SMTP(host, port, timeout=15) as s:
    s.ehlo(); s.starttls(context=ctx); s.ehlo(); s.login(user, pw)
print("AUTH_OK", host, port, user)
EOF'
```

Expect `AUTH_OK`. `535` usually means stale password in `config.json` (see step 3).

## Scheduled local check

The sanitized Ansible health profile is `vaultwarden` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses only server-side credentials for the SMTP AUTH probe. It does not modify Vaultwarden, Traefik, SMTP, or secrets.


## Safety

- Read-only: never mutate configuration, containers, or secrets during this check.
- Never print passwords — compare lengths/hashes only.
- If live state conflicts with an expected value below, `STOP` and report.

## Pass criteria

- Healthy compose; `DATABASE_URL` points at `pg`
- HTTPS 200 + `/admin` 200; PG counts sane
- Effective SMTP 587/starttls; `.env` ↔ `config.json` password match; **AUTH_OK**
- Wrong-password login returns 400 business error (not 500)
- Update **Verified** on [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md)