windyboy
b0c01b2551
- RUNBOOKS.md: repo-level spec (six-field model, naming, safety, maturity path) - runbooks/_template.md + README.md: standard template and 16-entry routing index - new: issue-to-merge, fix-ci, release, rollback, network-change, network-recovery - light-enhance 10 existing runbooks with Purpose/Scope/Safety headers - AGENTS.md: point step 3 at index/spec, add runbook execution rules - docs/agent-runbook-guide.md: archive of Manus AI guide
3.2 KiB
3.2 KiB
Matrix Health Check
Purpose
Read-only health check of the Matrix homeserver (ESS on K3s).
Scope
- Applicable: synapse.chans.xyz, namespace
ess. - Read-only: does not change pods, ingress, certificates, or configuration.
Monitor the Matrix homeserver running on synapse.chans.xyz (ESS chart 26.7.2, K3s node).
Prefer cd ansible && ansible-playbook playbooks/health-report.yml --limit matrix
for the routine report. The SSH commands below remain diagnostic detail.
Focused SSH diagnostics
Use these only after the Ansible health report needs investigation.
Basic checks
# Overall status
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods,svc,ingress,certificate -o wide'
# Pod restarts
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods -o wide | grep -E "CrashLoop|Error|Evicted" || echo "No unhealthy pods"'
# Disk
ssh -4 windy@synapse.chans.xyz 'df -hP /'
Certificate verification
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get certificate'
All certificates should show READY=True. Domains:
chans.xyzsynapse.chans.xyzchat.chans.xyzaccount.chans.xyzadmin.chans.xyzmrtc.chans.xyz(placeholder only)
External endpoint verification
# Client API
curl -4 -s https://synapse.chans.xyz/_matrix/client/versions | python3 -m json.tool | head -5
# Well-known server discovery
curl -4 -s https://chans.xyz/.well-known/matrix/server
# Well-known client discovery
curl -4 -s https://chans.xyz/.well-known/matrix/client
# Federation tester (manual)
open https://federationtester.matrix.org/#chans.xyz
Local health-check automation
The node has a systemd health-check timer that runs daily:
ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-k3s-healthcheck.timer'
ssh -4 windy@synapse.chans.xyz 'cat /var/lib/vps-health/latest.json 2>/dev/null || echo "No data yet"'
Checks cover:
- K3s node Ready
- Pods in
essnamespace (no CrashLoopBackOff / Evicted) - Root disk usage (warning ≥80%, critical ≥90%)
- Backup freshness
Backup status
ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-backup.timer'
ssh -4 windy@synapse.chans.xyz 'ls -la /var/backups/matrix/'
Backup automation is currently paused. /var/backups/matrix/ is retained for a future, reviewed design and must not be treated as a recovery source.
- PostgreSQL dumps (synapse + mas)
- Media archive
- Bootstrap secrets archive
- Backup manifest with SHA-256
- Retention: 7 days
Common pitfalls
| Symptom | Likely cause |
|---|---|
/_matrix/client/versions fails |
Traefik ingress not routing to Synapse |
| HTTPS returns non-LE cert | cert-manager letsencrypt-prod not ready |
| Well-known returns 404/redirect | Root chans.xyz ingress missing or misconfigured |
| 502 Bad Gateway | Synapse pod restarting or DB down |
| SMTP emails not sent | MAS SMTP config incomplete; TCP reachable but AUTH failing — see runbooks/vaultwarden-health.md |
Safety
- Read-only: never mutate pods, ingress, certificates, or configuration during this check.
- Backup automation is paused; do not treat
/var/backups/matrix/as a recovery source. - If live state conflicts with an expected value,
STOPand report.