windy/vps
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Initial VPS operations handbook

Browse Source
This commit is contained in:
windyboy
2026-08-03 12:26:42 +08:00
commit b73125e5bc
97 changed files with 3641 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
runbooks/mailcow-health.md
+72
View File
@@ -0,0 +1,72 @@
# Runbook: mailcow health (mx2)
Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
Path: `/opt/mail`
Prefer: `./scripts/check-mx2.sh` from this repo, or the steps below over SSH.
## 1. Containers
```bash
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose ps -a'
```
Expect mailcow (`cow-*`) services **Up**. Note any Exit/Restarting.
## 2. Watchdog
```bash
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose logs --tail=40 watchdog-mailcow'
```
Expect recent lines at **100%** for Nginx, Postfix, Dovecot, MySQL, Redis, SOGo, Rspamd, ACME, etc.
## 3. Queue
```bash
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose exec -T postfix-mailcow postqueue -p'
```
Healthy: empty / `Mail queue is empty`.
## 4. Local listeners
```bash
ssh windy@mx2.windy.me 'ss -tlnp | grep -E ":(25|465|587|143|993|995|80|443|4190)\\s"'
```
## 5. External checks (from laptop / agent host)
```bash
curl -sI --max-time 10 http://mx2.windy.me/ | head -5
curl -skI --max-time 10 https://mx2.windy.me/ | head -15
echo | openssl s_client -connect mx2.windy.me:443 -servername mx2.windy.me 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates
timeout 8 bash -c 'exec 3<>/dev/tcp/mx2.windy.me/25; echo -e "EHLO test.local\r\nQUIT\r\n" >&3; cat <&3' | head -20
```
Expect: HTTP 301 → HTTPS; HTTPS 200; LE cert for `mx2.windy.me`; SMTP banner `Postcow`.
## 6. DNS summary
Use a public resolver if the local stub is flaky (WSL `127.0.0.53`):
```bash
dig @1.1.1.1 +short mx2.windy.me A
dig @1.1.1.1 +short mx2.windy.me AAAA
dig @1.1.1.1 +short -x 194.163.160.244
dig @1.1.1.1 +short windy.me MX
dig @1.1.1.1 +short windy.me TXT | grep -i spf
```
## Scheduled local check
The sanitized Ansible health profile is `mailcow` (`ansible/playbooks/healthchecks.yml`). It preserves the existing standalone `scripts/check-mx2.sh` as the richer operator check, while the server-local timer emits a sanitized result at `/var/lib/vps-health/latest.json`. Neither check changes Mailcow configuration or service state.
## Pass criteria
- Compose stack up; watchdog ~100%
- Queue empty (or explain backlog)
- TLS valid; mail ports reachable externally
- A/AAAA/PTR/MX consistent with [host facts](../hosts/mx2.windy.me.md)
Update the **Verified** line on the host page after a successful check.
runbooks/mailcow-smtp-client.md
+54
View File
@@ -0,0 +1,54 @@
# Runbook: use mailcow SMTP / IMAP (client)
Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
Prerequisite: a mailbox on `windy.me` (password from mailcow UI, not the admin account unless it is that mailbox).
Clients should use **submission** to send. Do **not** use port 25 as a desktop/app outbound port.
## Send (SMTP)
| Item | Value |
|------|--------|
| Server | `mx2.windy.me` |
| Port | **587** (STARTTLS) — preferred |
| Alt port | **465** (SSL/TLS) |
| Username | Full address, e.g. `zhiqiang@windy.me` |
| Password | Mailbox password |
| Auth | Required (LOGIN / PLAIN) |
Match encryption to the port: 587 → STARTTLS; 465 → SSL/TLS (not STARTTLS).
From address should be that mailbox or an allowed alias on the same domain.
## Receive (IMAP / POP)
| Protocol | Server | Port |
|----------|--------|------|
| IMAPS | `mx2.windy.me` | **993** |
| POP3S | `mx2.windy.me` | **995** |
Username = full email address; same mailbox password.
## Webmail
- UI / SOGo: https://mx2.windy.me
- Log in with the same mailbox credentials.
## CLI smoke test (optional)
```bash
swaks --to someone@example.com \
--from YOU@windy.me \
--server mx2.windy.me --port 587 -tls \
--auth LOGIN --auth-user YOU@windy.me --auth-password 'MAILBOX_PASSWORD'
```
Do not commit or paste real passwords into this repo.
## Auth failures — quick checks
- Username is the **full** email, not a local part alone
- Password is the **mailbox** password (not unrelated host/root secrets)
- Port/TLS mode mismatch (587 vs 465)
- Account active in mailcow; not rate-limited / fail2banned after bad attempts
- Apps that store SMTP in their own config (e.g. Vaultwarden `config.json`) may keep a **stale** password even when `.env` is correct — verify AUTH against the effective config ([vaultwarden-health](vaultwarden-health.md) §5)
runbooks/mailcow-update.md
+43
View File
@@ -0,0 +1,43 @@
# Runbook: mailcow update (mx2)
Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
Path: `/opt/mail`
**Confirm with the user before running an update.**
## Before
1. Run [mailcow-health](mailcow-health.md) (or `./scripts/check-mx2.sh`). Record baseline.
2. Note config quirks in `mailcow.conf`, especially:
- `SKIP_CLAMD=y` — after update, `clamd` may still be running until services are recreated per mailcow docs.
- `SKIP_LETS_ENCRYPT`, ports, hostname — should stay `mx2.windy.me`.
3. Disk/memory: `df -h /` and `free -h` on the host.
## Update
```bash
ssh -t windy@mx2.windy.me 'cd /opt/mail && ./update.sh'
```
- Prefer interactive SSH (`-t`) so prompts work.
- Do not pass secrets into the chat log; do not commit `mailcow.conf`.
If update asks to stop/recreate containers, allow it unless the user said otherwise.
## After
1. Re-run health checks ([mailcow-health](mailcow-health.md)).
2. Confirm UI: `https://mx2.windy.me`
3. Confirm SMTP banner and queue empty.
4. If `SKIP_CLAMD=y` but `cow-clamd-mailcow-1` is still Up and that is unwanted:
```bash
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose up -d --force-recreate'
```
Only after user OK; recreate briefly interrupts mail.
5. Update **Verified** on [hosts/mx2.windy.me.md](../hosts/mx2.windy.me.md) with date and outcome.
## Rollback
Mailcow updates are image/git based; there is no one-click rollback in this runbook. If something fails, capture `docker compose ps` and `docker compose logs --tail=100` for the failing service and stop before further changes.
runbooks/matrix-health.md
+85
View File
@@ -0,0 +1,85 @@
# Matrix Health Check
Monitor the Matrix homeserver running on `synapse.chans.xyz` (ESS chart `26.7.2`, K3s node).
## Basic checks
```bash
# Overall status
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods,svc,ingress,certificate -o wide'
# Pod restarts
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods -o wide | grep -E "CrashLoop|Error|Evicted" || echo "No unhealthy pods"'
# Disk
ssh -4 windy@synapse.chans.xyz 'df -hP /'
```
## Certificate verification
```bash
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get certificate'
```
All certificates should show `READY=True`. Domains:
- `chans.xyz`
- `synapse.chans.xyz`
- `chat.chans.xyz`
- `account.chans.xyz`
- `admin.chans.xyz`
- `mrtc.chans.xyz` (placeholder only)
## External endpoint verification
```bash
# Client API
curl -4 -s https://synapse.chans.xyz/_matrix/client/versions | python3 -m json.tool | head -5
# Well-known server discovery
curl -4 -s https://chans.xyz/.well-known/matrix/server
# Well-known client discovery
curl -4 -s https://chans.xyz/.well-known/matrix/client
# Federation tester (manual)
open https://federationtester.matrix.org/#chans.xyz
```
## Local health-check automation
The node has a systemd health-check timer that runs daily:
```bash
ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-k3s-healthcheck.timer'
ssh -4 windy@synapse.chans.xyz 'cat /var/lib/vps-health/latest.json 2>/dev/null || echo "No data yet"'
```
Checks cover:
- K3s node Ready
- Pods in `ess` namespace (no CrashLoopBackOff / Evicted)
- Root disk usage (warning ≥80%, critical ≥90%)
- Backup freshness
## Backup status
```bash
ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-backup.timer'
ssh -4 windy@synapse.chans.xyz 'ls -la /var/backups/matrix/'
```
Backup automation is currently paused. `/var/backups/matrix/` is retained for a future, reviewed design and must not be treated as a recovery source.
- PostgreSQL dumps (synapse + mas)
- Media archive
- Bootstrap secrets archive
- Backup manifest with SHA-256
- Retention: 7 days
## Common pitfalls
| Symptom | Likely cause |
|---|---|
| `/_matrix/client/versions` fails | Traefik ingress not routing to Synapse |
| HTTPS returns non-LE cert | cert-manager `letsencrypt-prod` not ready |
| Well-known returns 404/redirect | Root `chans.xyz` ingress missing or misconfigured |
| 502 Bad Gateway | Synapse pod restarting or DB down |
| SMTP emails not sent | MAS SMTP config incomplete; TCP reachable but AUTH failing — see `runbooks/vaultwarden-health.md` |
runbooks/pdns-health.md
+86
View File
@@ -0,0 +1,86 @@
# PowerDNS health (hk2)
Read-only checks for the `/opt/pdns` stack on **hk2.chans.xyz** (`ns1.wsvc.info`).
Facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) · Upstream: [docs/pdns-upstream.md](../docs/pdns-upstream.md)
## Quick remote
```bash
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose ps -a'
```
Expect `auth`, `db`, `poweradmin` healthy; `backup` Up; `pgweb` Up. Ignore stopped orphan `powerdns-admin` unless cleaning orphans.
## Version / security poll
Expect Authoritative **5.0.6** (or newer). Startup logs should **not** show a Mandatory `Security Update` banner. If they do, check https://doc.powerdns.com/authoritative/security-advisories/ and bump the `powerdns/pdns-auth-50` image pin in `/opt/pdns/compose.yml`.
```bash
ssh -4 windy@hk2.chans.xyz 'docker logs pdns-auth 2>&1 | grep -i "Security Update" | tail -5'
ssh -4 windy@hk2.chans.xyz 'docker inspect pdns-auth --format "{{.Config.Image}}"'
```
## DNS (from laptop / WSL)
Prefer querying auth/secondary IPs directly (local stub / public resolvers may be flaky or cached):
```bash
NS1=154.36.174.161
NS2=202.91.35.141
for z in windy.me wsvc.info chans.xyz; do
echo "== $z =="
dig @$NS1 +short NS "$z"
dig @$NS1 +short SOA "$z"
dig @$NS2 +short SOA "$z"
done
dig @$NS1 +short A ns1.wsvc.info # expect 154.36.174.161
dig @$NS1 +short A ns2.wsvc.info # expect 202.91.35.141
```
Served SOA serial can differ from the backend/API serial when `SOA-EDIT=INCEPTION-INCREMENT` is active — compare **ns1 vs ns2 served SOA**, not dig vs `pdnsutil`/API.
## API (on host)
```bash
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose exec -T auth \
python3 -c "import json,os,urllib.request; print(json.load(urllib.request.urlopen(urllib.request.Request(\"http://127.0.0.1:8081/api/v1/servers/localhost\", headers={\"X-API-Key\": os.environ[\"PDNS_API_KEY\"]})))[\"version\")"'
```
Or use host loopback with the key from `.env` (do not paste the key into chat/logs).
## Web UIs
```bash
# If local stub DNS is flaky, pin SNI to the auth IP:
curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pdns.wsvc.info:443:154.36.174.161 https://pdns.wsvc.info/ # 302 → login
curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pgweb.wsvc.info:443:154.36.174.161 https://pgweb.wsvc.info/ # 401 without basic auth
```
## Backup
```bash
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && ./scripts/manual-backup.sh && ls -lt backup/*.sql.gz | head'
ssh -4 windy@hk2.chans.xyz 'tail -30 /opt/pdns/backup/backup.log'
```
Scheduler should log `scheduler started … (daily 03:00)`. If you see `can't execute 'bash'` or Alpine `apk` errors, the backup image/entrypoint was reverted — see [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md).
## Config sanity (docs-aligned)
```bash
ssh -4 windy@hk2.chans.xyz 'grep -E "^(primary|also-notify|only-notify|allow-axfr-ips|gpgsql-dnssec|default-soa-edit)" /opt/pdns/auth/pdns.conf'
```
Expect: `primary=yes`, `also-notify=202.91.35.141`, `only-notify=` empty, `gpgsql-dnssec=yes`, SOA-EDIT `INCEPTION-INCREMENT`.
## Scheduled local check
The sanitized Ansible health profile is `pdns` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses the API key only inside the PowerDNS container. It does not modify PowerDNS, DNS records, or secrets.
## After config changes
- `auth/pdns.conf`, `auth/templates.d/secrets.j2`, or auth-related `.env` → `docker compose up -d --force-recreate auth`
- Poweradmin env / Traefik labels → `docker compose up -d --no-deps --force-recreate poweradmin` (restart Traefik if UI 404s)
- backup schedule / retention env → `docker compose up -d --force-recreate --no-deps backup`
runbooks/vaultwarden-health.md
+96
View File
@@ -0,0 +1,96 @@
# Runbook: Vaultwarden health (us2)
Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)
Path: `/opt/vaultwarden`
URL: https://auth.wsvc.info/
Upstream: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
## 1. Containers
```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose ps -a'
```
Expect `vaultwarden` + `vw-db` **Up (healthy)**. `pgweb` should be stopped unless started with `--profile debug`.
## 2. Live DB is Postgres
```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden sh -c "printenv DATABASE_URL" | sed -E "s#://[^@]+@#://REDACTED@#"'
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T pg psql -U vaultwarden -d vaultwarden -c "SELECT COUNT(*) AS users FROM users; SELECT COUNT(*) AS ciphers FROM ciphers;"'
```
## 3. Effective config (config.json wins)
```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden cat /data/config.json' \
| python3 -c "import sys,json;d=json.load(sys.stdin);print({k:d.get(k) for k in ['domain','smtp_host','smtp_port','smtp_security','smtp_username','smtp_from','ip_header','signups_allowed']});print('smtp_password_set',bool(d.get('smtp_password')))"
```
Expect SMTP **mx2.windy.me** / **587** / **starttls**, `ip_header` **X-Forwarded-For**.
Also confirm `.env` and `config.json` SMTP passwords match (lengths/hashes only — never print secrets):
```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
import json, re, pathlib, hashlib, subprocess
env = pathlib.Path(".env").read_text()
def grab(k):
m = re.search(rf"^{re.escape(k)}=(.*)$", env, re.M)
return m.group(1).strip().strip("\"'\''") if m else None
cfg = json.loads(subprocess.check_output(
["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
ep, cp = grab("SMTP_PASSWORD"), cfg.get("smtp_password") or ""
def fp(s): return {"len": len(s), "sha256_8": hashlib.sha256(s.encode()).hexdigest()[:8]}
print("env_pass", fp(ep or ""), "cfg_pass", fp(cp), "match", ep == cp)
EOF'
```
If they diverge, **auth uses `config.json`** — sync password there (or via `/admin`) and update `.smtp-credentials`.
## 4. External HTTPS
```bash
curl -4 -sS -I --max-time 15 https://auth.wsvc.info/ | head -15
```
If **404** right after recreating VW: `ssh -4 windy@us2.wsvc.info 'docker restart traefik'` then retry.
## 5. SMTP reachability + AUTH
TCP (expect `587:0`; `465` often times out from us2):
```bash
ssh -4 windy@us2.wsvc.info 'docker compose -f /opt/vaultwarden/docker-compose.yml exec -T vaultwarden sh -c "timeout 5 bash -c \"cat </dev/null >/dev/tcp/mx2.windy.me/587\"; echo 587:\$?"'
```
AUTH with the **effective** `config.json` password (run on us2; do not print the password):
```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
import json, smtplib, ssl, subprocess
cfg = json.loads(subprocess.check_output(
["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
host, port = cfg["smtp_host"], int(cfg["smtp_port"])
user, pw = cfg["smtp_username"], cfg["smtp_password"]
ctx = ssl.create_default_context()
with smtplib.SMTP(host, port, timeout=15) as s:
s.ehlo(); s.starttls(context=ctx); s.ehlo(); s.login(user, pw)
print("AUTH_OK", host, port, user)
EOF'
```
Expect `AUTH_OK`. `535` usually means stale password in `config.json` (see step 3).
## Scheduled local check
The sanitized Ansible health profile is `vaultwarden` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses only server-side credentials for the SMTP AUTH probe. It does not modify Vaultwarden, Traefik, SMTP, or secrets.
## Pass criteria
- Healthy compose; `DATABASE_URL` points at `pg`
- HTTPS 200 + `/admin` 200; PG counts sane
- Effective SMTP 587/starttls; `.env` ↔ `config.json` password match; **AUTH_OK**
- Wrong-password login returns 400 business error (not 500)
- Update **Verified** on [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md)
runbooks/vaultwarden-sqlite-to-postgres.md
+37
View File
@@ -0,0 +1,37 @@
# Runbook: Vaultwarden SQLite → Postgres (us2)
Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)
Upstream: [Using the PostgreSQL Backend](https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend) · [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
**Done on 2026-08-01** for this host. Keep this as the replay checklist.
## Summary of what we did
1. Backup `vw-data` + `.env` → `backups/pre-pg-migrate-<timestamp>/`
2. Baseline counts from SQLite (then: users=1, ciphers=1327)
3. Drop/recreate empty Postgres DB; pin image `vaultwarden/server:1.37.1`
4. Set compose `DATABASE_URL=postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:${DB_PORT}/${DB_NAME}`
5. Start VW once → diesel schema → stop
6. `pgloader` **data only** (exclude `__diesel_schema_migrations`) with CAST bigint→integer
7. Remaining intentional bigints: `attachments.file_size`, `twofactor.last_used`, `twofactor_duo_ctx.exp`
8. Align `config.json` SMTP **587/starttls**, `ip_header=X-Forwarded-For`, `DOMAIN` without trailing slash
9. Rotate `DB_PASS` / `PGWEB_PASS`; `pgweb` behind compose `profiles: [debug]`
10. Move live sqlite file to `backups/sqlite-cold/` (cold only)
11. After VW recreate, **restart Traefik** so Docker provider picks up the router
## Critical pitfalls
- `config.json` overrides `.env` (admin wins). Fix SMTP there or via `/admin`.
- SMTP password drift: if `.env` AUTH works but VW mail fails with `535`, sync `vw-data/config.json` (+ `.smtp-credentials`) to the mailbox password — do not assume env alone is enough.
- From this network, **mx2:465 times out**; use **587 + starttls**.
- After recreating `vaultwarden`, Traefik may 404 until `docker restart traefik`.
- pgloader needs a **writable** directory for the sqlite file (CANTOPEN if mount is awkward).
- After pgloader, wrong integer types → silent “wrong password”; see [Discussion #7134](https://github.com/dani-garcia/vaultwarden/discussions/7134).
- Do not commit `.env` or load files with passwords into git.
## Rollback
1. `docker compose stop vaultwarden`
2. Restore `vw-data` from `backups/pre-pg-migrate-*` (including sqlite)
3. Remove `DATABASE_URL` from compose `environment`
4. `docker compose up -d vaultwarden` && `docker restart traefik`