windyboy
b0c01b2551
- RUNBOOKS.md: repo-level spec (six-field model, naming, safety, maturity path) - runbooks/_template.md + README.md: standard template and 16-entry routing index - new: issue-to-merge, fix-ci, release, rollback, network-change, network-recovery - light-enhance 10 existing runbooks with Purpose/Scope/Safety headers - AGENTS.md: point step 3 at index/spec, add runbook execution rules - docs/agent-runbook-guide.md: archive of Manus AI guide
108 lines
3.2 KiB
Markdown
108 lines
3.2 KiB
Markdown
# Matrix Health Check
|
|
|
|
## Purpose
|
|
|
|
Read-only health check of the Matrix homeserver (ESS on K3s).
|
|
|
|
## Scope
|
|
|
|
- Applicable: [synapse.chans.xyz](../hosts/synapse.chans.xyz.md), namespace `ess`.
|
|
- Read-only: does not change pods, ingress, certificates, or configuration.
|
|
|
|
Monitor the Matrix homeserver running on `synapse.chans.xyz` (ESS chart `26.7.2`, K3s node).
|
|
|
|
Prefer `cd ansible && ansible-playbook playbooks/health-report.yml --limit matrix`
|
|
for the routine report. The SSH commands below remain diagnostic detail.
|
|
|
|
## Focused SSH diagnostics
|
|
|
|
Use these only after the Ansible health report needs investigation.
|
|
|
|
### Basic checks
|
|
|
|
```bash
|
|
# Overall status
|
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods,svc,ingress,certificate -o wide'
|
|
|
|
# Pod restarts
|
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods -o wide | grep -E "CrashLoop|Error|Evicted" || echo "No unhealthy pods"'
|
|
|
|
# Disk
|
|
ssh -4 windy@synapse.chans.xyz 'df -hP /'
|
|
```
|
|
|
|
### Certificate verification
|
|
|
|
```bash
|
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get certificate'
|
|
```
|
|
|
|
All certificates should show `READY=True`. Domains:
|
|
- `chans.xyz`
|
|
- `synapse.chans.xyz`
|
|
- `chat.chans.xyz`
|
|
- `account.chans.xyz`
|
|
- `admin.chans.xyz`
|
|
- `mrtc.chans.xyz` (placeholder only)
|
|
|
|
### External endpoint verification
|
|
|
|
```bash
|
|
# Client API
|
|
curl -4 -s https://synapse.chans.xyz/_matrix/client/versions | python3 -m json.tool | head -5
|
|
|
|
# Well-known server discovery
|
|
curl -4 -s https://chans.xyz/.well-known/matrix/server
|
|
|
|
# Well-known client discovery
|
|
curl -4 -s https://chans.xyz/.well-known/matrix/client
|
|
|
|
# Federation tester (manual)
|
|
open https://federationtester.matrix.org/#chans.xyz
|
|
```
|
|
|
|
## Local health-check automation
|
|
|
|
The node has a systemd health-check timer that runs daily:
|
|
|
|
```bash
|
|
ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-k3s-healthcheck.timer'
|
|
ssh -4 windy@synapse.chans.xyz 'cat /var/lib/vps-health/latest.json 2>/dev/null || echo "No data yet"'
|
|
```
|
|
|
|
Checks cover:
|
|
- K3s node Ready
|
|
- Pods in `ess` namespace (no CrashLoopBackOff / Evicted)
|
|
- Root disk usage (warning ≥80%, critical ≥90%)
|
|
- Backup freshness
|
|
|
|
## Backup status
|
|
|
|
```bash
|
|
ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-backup.timer'
|
|
ssh -4 windy@synapse.chans.xyz 'ls -la /var/backups/matrix/'
|
|
```
|
|
|
|
Backup automation is currently paused. `/var/backups/matrix/` is retained for a future, reviewed design and must not be treated as a recovery source.
|
|
- PostgreSQL dumps (synapse + mas)
|
|
- Media archive
|
|
- Bootstrap secrets archive
|
|
- Backup manifest with SHA-256
|
|
- Retention: 7 days
|
|
|
|
## Common pitfalls
|
|
|
|
| Symptom | Likely cause |
|
|
|---|---|
|
|
| `/_matrix/client/versions` fails | Traefik ingress not routing to Synapse |
|
|
| HTTPS returns non-LE cert | cert-manager `letsencrypt-prod` not ready |
|
|
| Well-known returns 404/redirect | Root `chans.xyz` ingress missing or misconfigured |
|
|
| 502 Bad Gateway | Synapse pod restarting or DB down |
|
|
| SMTP emails not sent | MAS SMTP config incomplete; TCP reachable but AUTH failing — see `runbooks/vaultwarden-health.md` |
|
|
|
|
## Safety
|
|
|
|
- Read-only: never mutate pods, ingress, certificates, or configuration during this check.
|
|
- Backup automation is paused; do not treat `/var/backups/matrix/` as a recovery source.
|
|
- If live state conflicts with an expected value, `STOP` and report.
|