Initial VPS operations handbook

2026-08-03 12:26:42 +08:00
commit b73125e5bc
97 changed files with 3641 additions and 0 deletions
@@ -0,0 +1,85 @@
+# Matrix Health Check
+
+Monitor the Matrix homeserver running on `synapse.chans.xyz` (ESS chart `26.7.2`, K3s node).
+
+## Basic checks
+
+```bash
+# Overall status
+ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods,svc,ingress,certificate -o wide'
+
+# Pod restarts
+ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods -o wide | grep -E "CrashLoop|Error|Evicted" || echo "No unhealthy pods"'
+
+# Disk
+ssh -4 windy@synapse.chans.xyz 'df -hP /'
+```
+
+## Certificate verification
+
+```bash
+ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get certificate'
+```
+
+All certificates should show `READY=True`. Domains:
+- `chans.xyz`
+- `synapse.chans.xyz`
+- `chat.chans.xyz`
+- `account.chans.xyz`
+- `admin.chans.xyz`
+- `mrtc.chans.xyz` (placeholder only)
+
+## External endpoint verification
+
+```bash
+# Client API
+curl -4 -s https://synapse.chans.xyz/_matrix/client/versions | python3 -m json.tool | head -5
+
+# Well-known server discovery
+curl -4 -s https://chans.xyz/.well-known/matrix/server
+
+# Well-known client discovery
+curl -4 -s https://chans.xyz/.well-known/matrix/client
+
+# Federation tester (manual)
+open https://federationtester.matrix.org/#chans.xyz
+```
+
+## Local health-check automation
+
+The node has a systemd health-check timer that runs daily:
+
+```bash
+ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-k3s-healthcheck.timer'
+ssh -4 windy@synapse.chans.xyz 'cat /var/lib/vps-health/latest.json 2>/dev/null || echo "No data yet"'
+```
+
+Checks cover:
+- K3s node Ready
+- Pods in `ess` namespace (no CrashLoopBackOff / Evicted)
+- Root disk usage (warning ≥80%, critical ≥90%)
+- Backup freshness
+
+## Backup status
+
+```bash
+ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-backup.timer'
+ssh -4 windy@synapse.chans.xyz 'ls -la /var/backups/matrix/'
+```
+
+Backup automation is currently paused. `/var/backups/matrix/` is retained for a future, reviewed design and must not be treated as a recovery source.
+- PostgreSQL dumps (synapse + mas)
+- Media archive
+- Bootstrap secrets archive
+- Backup manifest with SHA-256
+- Retention: 7 days
+
+## Common pitfalls
+
+| Symptom | Likely cause |
+|---|---|
+| `/_matrix/client/versions` fails | Traefik ingress not routing to Synapse |
+| HTTPS returns non-LE cert | cert-manager `letsencrypt-prod` not ready |
+| Well-known returns 404/redirect | Root `chans.xyz` ingress missing or misconfigured |
+| 502 Bad Gateway | Synapse pod restarting or DB down |
+| SMTP emails not sent | MAS SMTP config incomplete; TCP reachable but AUTH failing — see `runbooks/vaultwarden-health.md` |