94 lines
3.4 KiB
Markdown
94 lines
3.4 KiB
Markdown
|
# Runbook: Plane Health Check
|
||
|
|
|
||
|
|
## Purpose
|
||
|
|
|
||
|
|
Read-only health check of the self-hosted Plane project-management instance
|
||
|
|
(`plane.chans.xyz`) running on the synapse K3s cluster.
|
||
|
|
|
||
|
|
## Scope
|
||
|
|
|
||
|
|
- Applicable: [synapse.chans.xyz](../hosts/synapse.chans.xyz.md), namespace `plane`.
|
||
|
|
- Read-only: does not change pods, ingress, certificates, secrets, or configuration.
|
||
|
|
- Not applicable: Plane upgrade, values changes, or data recovery — those need a
|
||
|
|
reviewed change (see `ansible-operations.md` / `release.md`).
|
||
|
|
|
||
|
|
## Ownership
|
||
|
|
|
||
|
|
- Owner: personal ops (Windy)
|
||
|
|
- Last reviewed: 2026-09-02
|
||
|
|
- Related systems: synapse.chans.xyz (Helm `plane-app`, chart `plane-ce-1.8.0`, app `v1.4.1`)
|
||
|
|
|
||
|
|
## Pass criteria
|
||
|
|
|
||
|
|
All of the following must hold; any conflict means `STOP` and record evidence.
|
||
|
|
|
||
|
|
- `sudo helm list -A` shows `plane-app` in ns `plane`, STATUS `deployed`.
|
||
|
|
- All 7 Deployments + 4 StatefulSets in ns `plane` are `1/1 Running` with 0 recent restarts.
|
||
|
|
- Init Jobs `api-migrate-*` / `minio-bucket-*` are `Completed`.
|
||
|
|
- Certificate `plane-app-ssl-cert` is `READY=True` (CN `plane.chans.xyz`).
|
||
|
|
- `https://plane.chans.xyz/` returns HTTP 200 with a valid Let's Encrypt cert.
|
||
|
|
- Root disk usage below the 80% warning threshold.
|
||
|
|
|
||
|
|
## Procedure
|
||
|
|
|
||
|
|
### 1. Release and workloads
|
||
|
|
|
||
|
|
```bash
|
||
|
|
ssh -4 windy@synapse.chans.xyz 'sudo helm list -A'
|
||
|
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get deploy,sts,pods -o wide'
|
||
|
|
```
|
||
|
|
|
||
|
|
Expected: `plane-app` deployed; all workloads `1/1 Running`, `RESTARTS` low;
|
||
|
|
no `CrashLoopBackOff` / `Evicted`. Otherwise `STOP` and record evidence.
|
||
|
|
|
||
|
|
### 2. TLS certificate and ingress
|
||
|
|
|
||
|
|
```bash
|
||
|
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get certificate,issuer,ingressroute'
|
||
|
|
```
|
||
|
|
|
||
|
|
Expected: `plane-app-ssl-cert` `READY=True`; `plane-app-ingress` present with
|
||
|
|
routes for `/`, `/api`, `/spaces`, `/god-mode`, `/live`, `/uploads`.
|
||
|
|
|
||
|
|
If `READY=False` with a pending HTTP-01 challenge, the usual cause is the node
|
||
|
|
DNS chain (coreDNS → systemd-resolved → uplink) failing to resolve
|
||
|
|
`plane.chans.xyz` — check `resolvectl query plane.chans.xyz` vs
|
||
|
|
`dig +short plane.chans.xyz @8.8.8.8`. If the record exists publicly but the
|
||
|
|
node fails, `sudo resolvectl flush-caches` and wait for the cert-manager retry;
|
||
|
|
do not mutate the issuer.
|
||
|
|
|
||
|
|
### 3. Endpoint verification
|
||
|
|
|
||
|
|
```bash
|
||
|
|
curl -4 -s -o /dev/null -w '%{http_code}\n' https://plane.chans.xyz/
|
||
|
|
echo | openssl s_client -connect plane.chans.xyz:443 -servername plane.chans.xyz 2>/dev/null | openssl x509 -noout -subject -issuer -dates
|
||
|
|
```
|
||
|
|
|
||
|
|
Expected: HTTP 200, cert `CN=plane.chans.xyz` issued by Let's Encrypt with a
|
||
|
|
future `notAfter`. `https://plane.chans.xyz/api/` may 404 — the API serves
|
||
|
|
under `/api/...` paths only, so a bare 404 there is not a failure.
|
||
|
|
|
||
|
|
### 4. Storage and disk
|
||
|
|
|
||
|
|
```bash
|
||
|
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get pvc'
|
||
|
|
ssh -4 windy@synapse.chans.xyz 'df -hP /'
|
||
|
|
```
|
||
|
|
|
||
|
|
Expected: all 4 PVCs `Bound` (minio 5Gi, pgdb 5Gi, rabbitmq 100Mi, redis 100Mi
|
||
|
|
on `local-path`); root disk < 80%.
|
||
|
|
|
||
|
|
## Safety
|
||
|
|
|
||
|
|
- Read-only: never mutate pods, ingress, certificates, or configuration during
|
||
|
|
this check.
|
||
|
|
- Plane data (ns `plane` Postgres + MinIO PVCs) has **no backup tier**; treat
|
||
|
|
the instance as at-risk until a backup design exists.
|
||
|
|
- If live state conflicts with the expected values above, `STOP` and record
|
||
|
|
evidence; do not "fix in passing".
|
||
|
|
|
||
|
|
## References
|
||
|
|
|
||
|
|
- [hosts/synapse.chans.xyz.md](../hosts/synapse.chans.xyz.md) — plane stack facts
|
||
|
|
- [matrix-health.md](matrix-health.md) — sibling service on the same cluster
|