vps/runbooks/matrix-health.md

Matrix Health Check

Purpose

Read-only health check of the Matrix homeserver (ESS on K3s).

Scope

• Applicable: synapse.chans.xyz, namespace ess.
• Read-only: does not change pods, ingress, certificates, or configuration.

Ownership

• Owner: personal ops (Windy)
• Last reviewed: 2026-08-17
• Related systems: synapse.chans.xyz (ESS chart 26.7.2, K3s)

Monitor the Matrix homeserver running on synapse.chans.xyz (ESS chart 26.7.2, K3s node).

Prefer cd ansible && ansible-playbook playbooks/health-report.yml --limit matrix for the routine report. The SSH commands below remain diagnostic detail.

Focused SSH diagnostics

Use these only after the Ansible health report needs investigation.

Basic checks

# Overall status
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods,svc,ingress,certificate -o wide'

# Pod restarts
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods -o wide | grep -E "CrashLoop|Error|Evicted" || echo "No unhealthy pods"'

# Disk
ssh -4 windy@synapse.chans.xyz 'df -hP /'

Certificate verification

ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get certificate'

All certificates should show READY=True. Domains:

• chans.xyz
• synapse.chans.xyz
• chat.chans.xyz
• account.chans.xyz
• admin.chans.xyz
• mrtc.chans.xyz (placeholder only)

External endpoint verification

# Client API
curl -4 -s https://synapse.chans.xyz/_matrix/client/versions | python3 -m json.tool | head -5

# Well-known server discovery
curl -4 -s https://chans.xyz/.well-known/matrix/server

# Well-known client discovery
curl -4 -s https://chans.xyz/.well-known/matrix/client

# Federation tester (manual)
open https://federationtester.matrix.org/#chans.xyz

Local health-check automation

The node has a systemd health-check timer that runs daily:

ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-k3s-healthcheck.timer'
ssh -4 windy@synapse.chans.xyz 'cat /var/lib/vps-health/latest.json 2>/dev/null || echo "No data yet"'

Checks cover:

• K3s node Ready
• Pods in ess namespace (no CrashLoopBackOff / Evicted)
• Root disk usage (warning ≥80%, critical ≥90%)
• Backup freshness

Backup status

ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-backup.timer'
ssh -4 windy@synapse.chans.xyz 'ls -la /var/backups/matrix/'

Backup automation is currently paused. /var/backups/matrix/ is retained for a future, reviewed design and must not be treated as a recovery source.

• PostgreSQL dumps (synapse + mas)
• Media archive
• Bootstrap secrets archive
• Backup manifest with SHA-256
• Retention: 7 days

Common pitfalls

Symptom	Likely cause
/_matrix/client/versions fails	Traefik ingress not routing to Synapse
HTTPS returns non-LE cert	cert-manager letsencrypt-prod not ready
Well-known returns 404/redirect	Root chans.xyz ingress missing or misconfigured
502 Bad Gateway	Synapse pod restarting or DB down
SMTP emails not sent	MAS SMTP config incomplete; TCP reachable but AUTH failing — see runbooks/vaultwarden-health.md

Safety

• Read-only: never mutate pods, ingress, certificates, or configuration during this check.
• Backup automation is paused; do not treat /var/backups/matrix/ as a recovery source.
• If live state conflicts with an expected value, STOP and report.