vps/runbooks/matrix-health.md

# Matrix Health Check

## Purpose

Read-only health check of the Matrix homeserver (ESS on K3s).

## Scope

- Applicable: [synapse.chans.xyz](../hosts/synapse.chans.xyz.md), namespace `ess`.
- Read-only: does not change pods, ingress, certificates, or configuration.

## Ownership

- Owner: personal ops (Windy)
- Last reviewed: 2026-08-17
- Related systems: synapse.chans.xyz (ESS chart `26.7.2`, K3s)

Monitor the Matrix homeserver running on `synapse.chans.xyz` (ESS chart `26.7.2`, K3s node).

Prefer `cd ansible && ansible-playbook playbooks/health-report.yml --limit matrix`
for the routine report. The SSH commands below remain diagnostic detail.

## Focused SSH diagnostics

Use these only after the Ansible health report needs investigation.

### Basic checks

```bash
# Overall status
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods,svc,ingress,certificate -o wide'

# Pod restarts
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods -o wide | grep -E "CrashLoop|Error|Evicted" || echo "No unhealthy pods"'

# Disk
ssh -4 windy@synapse.chans.xyz 'df -hP /'
```

### Certificate verification

```bash
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get certificate'
```

All certificates should show `READY=True`. Domains:
- `chans.xyz`
- `synapse.chans.xyz`
- `chat.chans.xyz`
- `account.chans.xyz`
- `admin.chans.xyz`
- `mrtc.chans.xyz` (placeholder only)

### External endpoint verification

```bash
# Client API
curl -4 -s https://synapse.chans.xyz/_matrix/client/versions | python3 -m json.tool | head -5

# Well-known server discovery
curl -4 -s https://chans.xyz/.well-known/matrix/server

# Well-known client discovery
curl -4 -s https://chans.xyz/.well-known/matrix/client

# Federation tester (manual)
open https://federationtester.matrix.org/#chans.xyz
```

## Local health-check automation

The node has a systemd health-check timer that runs daily:

```bash
ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-k3s-healthcheck.timer'
ssh -4 windy@synapse.chans.xyz 'cat /var/lib/vps-health/latest.json 2>/dev/null || echo "No data yet"'
```

Checks cover:
- K3s node Ready
- Pods in `ess` namespace (no CrashLoopBackOff / Evicted)
- Root disk usage (warning ≥80%, critical ≥90%)
- Backup freshness

## Backup status

```bash
ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-backup.timer'
ssh -4 windy@synapse.chans.xyz 'ls -la /var/backups/matrix/'
```

Backup automation is currently paused. `/var/backups/matrix/` is retained for a future, reviewed design and must not be treated as a recovery source.
- PostgreSQL dumps (synapse + mas)
- Media archive
- Bootstrap secrets archive
- Backup manifest with SHA-256
- Retention: 7 days

## Common pitfalls

| Symptom | Likely cause |
|---|---|
| `/_matrix/client/versions` fails | Traefik ingress not routing to Synapse |
| HTTPS returns non-LE cert | cert-manager `letsencrypt-prod` not ready |
| Well-known returns 404/redirect | Root `chans.xyz` ingress missing or misconfigured |
| 502 Bad Gateway | Synapse pod restarting or DB down |
| SMTP emails not sent | MAS SMTP config incomplete; TCP reachable but AUTH failing — see `runbooks/vaultwarden-health.md` |

## Safety

- Read-only: never mutate pods, ingress, certificates, or configuration during this check.
- Backup automation is paused; do not treat `/var/backups/matrix/` as a recovery source.
- If live state conflicts with an expected value, `STOP` and report.