Matrix Health Check

Monitor the Matrix homeserver running on synapse.chans.xyz (ESS chart 26.7.2, K3s node).

Prefer cd ansible && ansible-playbook playbooks/health-report.yml --limit matrix for the routine report. The SSH commands below remain diagnostic detail.

Focused SSH diagnostics

Use these only after the Ansible health report needs investigation.

Basic checks

# Overall status
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods,svc,ingress,certificate -o wide'

# Pod restarts
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods -o wide | grep -E "CrashLoop|Error|Evicted" || echo "No unhealthy pods"'

# Disk
ssh -4 windy@synapse.chans.xyz 'df -hP /'

Certificate verification

ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get certificate'

All certificates should show READY=True. Domains:

chans.xyz
synapse.chans.xyz
chat.chans.xyz
account.chans.xyz
admin.chans.xyz
mrtc.chans.xyz (placeholder only)

External endpoint verification

# Client API
curl -4 -s https://synapse.chans.xyz/_matrix/client/versions | python3 -m json.tool | head -5

# Well-known server discovery
curl -4 -s https://chans.xyz/.well-known/matrix/server

# Well-known client discovery
curl -4 -s https://chans.xyz/.well-known/matrix/client

# Federation tester (manual)
open https://federationtester.matrix.org/#chans.xyz

Local health-check automation

The node has a systemd health-check timer that runs daily:

ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-k3s-healthcheck.timer'
ssh -4 windy@synapse.chans.xyz 'cat /var/lib/vps-health/latest.json 2>/dev/null || echo "No data yet"'

Checks cover:

K3s node Ready
Pods in ess namespace (no CrashLoopBackOff / Evicted)
Root disk usage (warning ≥80%, critical ≥90%)
Backup freshness

Backup status

ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-backup.timer'
ssh -4 windy@synapse.chans.xyz 'ls -la /var/backups/matrix/'

Backup automation is currently paused. /var/backups/matrix/ is retained for a future, reviewed design and must not be treated as a recovery source.

PostgreSQL dumps (synapse + mas)
Media archive
Bootstrap secrets archive
Backup manifest with SHA-256
Retention: 7 days

Common pitfalls

Symptom	Likely cause
`/_matrix/client/versions` fails	Traefik ingress not routing to Synapse
HTTPS returns non-LE cert	cert-manager `letsencrypt-prod` not ready
Well-known returns 404/redirect	Root `chans.xyz` ingress missing or misconfigured
502 Bad Gateway	Synapse pod restarting or DB down
SMTP emails not sent	MAS SMTP config incomplete; TCP reachable but AUTH failing — see `runbooks/vaultwarden-health.md`

2.7 KiB Raw Permalink Blame History