From c0c975584ad1406d0c8b022c74fa168a4182a079 Mon Sep 17 00:00:00 2001 From: windyboy Date: Sun, 13 Sep 2026 19:12:32 +0800 Subject: [PATCH] =?UTF-8?q?docs(plane):=20=E8=87=AA=E6=89=98=E7=AE=A1=20Pl?= =?UTF-8?q?ane=20=E8=90=BD=E5=9C=B0=E4=BA=8B=E5=AE=9E=E5=85=A5=E4=BB=93?= =?UTF-8?q?=E5=BA=93=20+=20plane-health=20runbook=20+=20hardening=20?= =?UTF-8?q?=E8=8D=89=E7=A8=BF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录源 Linear→Plane (2026-09-03 起, Plane MCP) + plane.chans.xyz 服务行/upstream 段; inventory + hosts/synapse.chans.xyz.md 补 Plane 部署事实 (Helm plane-ce-1.8.0 / app v1.4.1, ns plane, IngressRoute/自有证书 issuer/PVC 5+5Gi local-path/无备份层); 新增 runbooks/plane-health.md (只读健康检查) 与 docs/plane-hardening/ 草稿 (values.hardened.yaml、secrets.yaml.example 占位、backup/ CronJob), 均为未应用设计稿; .gitignore 增加 .tmp-* agent 临时文件。 --- .gitignore | 1 + AGENTS.md | 19 ++- docs/plane-hardening/README.md | 45 ++++++++ docs/plane-hardening/backup/README.md | 48 ++++++++ docs/plane-hardening/backup/plane-backup.yaml | 63 ++++++++++ docs/plane-hardening/secrets.yaml.example | 91 +++++++++++++++ docs/plane-hardening/values.hardened.yaml | 109 ++++++++++++++++++ hosts/synapse.chans.xyz.md | 46 +++++++- inventory/hosts.md | 1 + runbooks/README.md | 1 + runbooks/plane-health.md | 93 +++++++++++++++ 11 files changed, 512 insertions(+), 5 deletions(-) create mode 100644 docs/plane-hardening/README.md create mode 100644 docs/plane-hardening/backup/README.md create mode 100644 docs/plane-hardening/backup/plane-backup.yaml create mode 100644 docs/plane-hardening/secrets.yaml.example create mode 100644 docs/plane-hardening/values.hardened.yaml create mode 100644 runbooks/plane-health.md diff --git a/.gitignore b/.gitignore index 8139c0b..2eb7452 100644 --- a/.gitignore +++ b/.gitignore @@ -32,3 +32,4 @@ skills-lock.json *~ # Agent working scratch (not repo content). .agent-work/ +.tmp-* diff --git a/AGENTS.md b/AGENTS.md index a84f947..20d2c09 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,9 +30,12 @@ This repo is the **agent ops handbook + fact source** for maintaining personal V > read-only unless a change is approved). 7. Record each material VPS operation, incident, configuration change, or - verification outcome in the corresponding **Linear `vps` project**. Include - scope, action, verification, and remaining follow-up; never put passwords, - tokens, private keys, recovery keys, or private room IDs in Linear. + verification outcome in the corresponding **Plane `vps` project** + (self-hosted `plane.chans.xyz`, Plane MCP `mcp__plane__*`). **Linear is + retired as a record source (2026-09-03) — do not create Linear issues;** + existing W1N-* entries are read-only history. Include scope, action, + verification, and remaining follow-up; never put passwords, tokens, private + keys, recovery keys, or private room IDs in Plane or Linear. ### Runbook execution rules @@ -67,6 +70,7 @@ not maintain a second copy of the machine table here. | https://synapse.chans.xyz | synapse | Synapse Client-Server + Federation API | | https://account.chans.xyz | synapse | Matrix Authentication Service (local passwords) | | https://admin.chans.xyz | synapse | Element Admin console (MAS admin auth) | +| https://plane.chans.xyz | synapse (`plane`, Helm `plane-ce` 1.8.0 / v1.4.1) | Plane project management (self-hosted, K3s) | ### Upstream docs @@ -76,6 +80,8 @@ not maintain a second copy of the machine table here. **Matrix (ESS on synapse):** Matrix homeserver running on `synapse.chans.xyz` via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone `chans.xyz` managed by hk2 PowerDNS. Before changing config, read [docs/matrix-upstream.md](docs/matrix-upstream.md) and [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: [matrix-health](runbooks/matrix-health.md). +**Plane (on synapse):** Self-hosted Plane project management at `plane.chans.xyz`, Helm release `plane-app` (chart `plane-ce-1.8.0`, app `v1.4.1`) in ns `plane` on the same K3s node as Matrix. Config from `/home/windy/plane-k3s/values.yaml`; workload/cert/ingress details in [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). Its Postgres/MinIO PVCs are **not** backed up. + **RustDesk:** Self-hosted RustDesk server on `hk2.chans.xyz` (`/opt/rustdesk`, containers `hbbs`/`hbbr`, image pinned `1.1.14`). The `hbbs -r` relay hostname must resolve to the host's public IP `154.36.174.161` — use `hk2.chans.xyz` (never `hk2.wsvc.info`, which has no DNS record). Health: [rustdesk-health](runbooks/rustdesk-health.md). ## Runbooks & scripts @@ -91,6 +97,7 @@ not maintain a second copy of the machine table here. | PowerDNS health (hk2) | [runbooks/pdns-health.md](runbooks/pdns-health.md) | | PowerDNS upstream refs | [docs/pdns-upstream.md](docs/pdns-upstream.md) | | Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) | +| Plane health | [runbooks/plane-health.md](runbooks/plane-health.md) | | RustDesk health (hk2) | [runbooks/rustdesk-health.md](runbooks/rustdesk-health.md) | | AdGuard Home health | [runbooks/adguard-home-health.md](runbooks/adguard-home-health.md) | | Home Assistant maintenance | [runbooks/home-assistant-maintenance.md](runbooks/home-assistant-maintenance.md) + [scripts/ha-maintenance.sh](runbooks/scripts/ha-maintenance.sh) | @@ -116,7 +123,11 @@ Routine mailcow health: `cd ansible && ansible-playbook playbooks/health-report. ### Issue tracker -Issues are tracked in Linear and created/updated via the Linear MCP (`vps` project). See `docs/agents/issue-tracker.md`. +Issues are tracked in **Plane** — self-hosted at `plane.chans.xyz`, project +`vps` — and created/updated via the Plane MCP (`mcp__plane__*`). **Linear is +retired as a record source (2026-09-03); do not create Linear issues.** Existing +W1N-* entries are read-only history. `docs/agents/issue-tracker.md` documents +the retired Linear workflow and is stale; treat this section as authoritative. ### Triage labels diff --git a/docs/plane-hardening/README.md b/docs/plane-hardening/README.md new file mode 100644 index 0000000..792dc73 --- /dev/null +++ b/docs/plane-hardening/README.md @@ -0,0 +1,45 @@ +# Plane CE 加固草稿（docs/plane-hardening/） + +> **状态：草稿，未应用、未提交。** 对应追踪：Plane vps 项目条目（2026-09-03，**记录源**；Linear W1N-277 已取消，Linear 自 2026-09-03 起不再作为记录源）。 +> 线上实例：`plane.chans.xyz`（synapse K3s，ns `plane`，release `plane-app` = chart `plane-ce-1.8.0` / app `v1.4.1`）。 +> 依据：2026-09-03 只读核查（13 条审查意见中 11 条属实、#3 基本属实、#9 指标归属错误）+ 上游 chart 模板逐条核对。 + +## 文件 + +| 文件 | 内容 | +|------|------| +| `values.hardened.yaml` | 可选硬化 values（external secrets 引用、requireExplicitSecrets、minio pin、上传限额对齐）；含 HTTP→HTTPS `extraObjects` 示例 | +| `secrets.yaml.example` | 6 组外部 Secret 结构占位（只含 key 名，真实值仅存宿主机） | +| `backup/plane-backup.yaml` | **PostgreSQL 备份 CronJob**（pg_dump `-Fc`，hostPath `/var/backups/plane`；MinIO 已按实际用量剔除） | +| `backup/README.md` | 备份方案说明（排程/容量/保留/还原/阻塞） | + +## 应用顺序（每步先 diff 后执行，全部需用户逐项确认） + +### 现在就值得做：DB 备份（P0，见 backup/） +`plane-backup.yaml` 部署 + 手动触发验证一次即可；88 MB 库每日快照几乎零成本。 + +### 可选（顺手做一次，不是必须） +- **Phase A 密钥外部化**（零行为变化、无停机，约 15 分钟）：按 `secrets.yaml.example` + 在宿主机建 6 个 Secret（值先复制当前集群），用 `values.hardened.yaml` 跑 + `helm diff upgrade` → `helm upgrade`；验证后删除 chart 生成的旧 Secret。 + 价值：默认密钥不再落在 chart 公开常量上，作为保险。 +- **MCP API Key 轮换**：若审查对话出过你的环境，Plane 后台重生成 + 更新 + `/home/windy/plane-k3s/mcp/mcp.env`（0600）+ 重启 Cursor MCP。 +- **/god-mode IP 白名单**：若在意管理后台被公网爆破。chart 1.8.0 的 IngressRoute + 不支持给单条路由追加 middleware → 需 post-renderer 或 upgrade 后 `kubectl patch` + （升级会覆盖，需固化）；源 IP 清单待提供。 + +### 明确暂缓/跳过（个人单节点，等出现症状再处理） +- SECRET_KEY 等轮换（Phase B）：等真要配 SMTP/OAuth 前再做（避免旧密文不可解）。 +- NetworkPolicy、有状态组件 resources limits（chart 无 values 开关，需 post-render/patch）、 + HTTP→HTTPS（草稿已给 `extraObjects` 示例）、metrics-server/Sentry。 + +## 关键限制（chart 1.8.0 模板已核对） +- `external_secrets.*_existingSecret` 设置后，对应 Secret **必须**包含模板所需全部 key + （缺失不自动补），见 `secrets.yaml.example` 注释。 +- `app_keys_existingSecret` 的 envFrom 在所有 workload 上**最后注入**（后置生效）， + 保证 app/live 共享密钥一致——不要在其后再放同名 key 的 Secret。 +- `DATABASE_URL`/`AMQP_URL`/`REDIS_URL` 是 chart 生成的派生 URL，内嵌明文密码； + 外部化后轮换 DB/队列密码时必须同步更新 `plane-app-env`。 +- minio 的 `MINIO_ROOT_*` 与 `AWS_*` 同源于一个 Secret；升级时 bucket Job 会重跑 + （需 admin 权限凭据）——换 svcacct 前先确认权限覆盖该 Job。 diff --git a/docs/plane-hardening/backup/README.md b/docs/plane-hardening/backup/README.md new file mode 100644 index 0000000..f4faf09 --- /dev/null +++ b/docs/plane-hardening/backup/README.md @@ -0,0 +1,48 @@ +# Plane CE 备份方案（DB-only）— DRAFT (2026-09-03), 未应用 + +> 关联：`plane-backup.yaml`（CronJob）；追踪：Plane vps 项目条目（记录源，2026-09-03 起不用 Linear）。 +> 现状（实测）：pg 全库 **88 MB**（310 issues / 1 user）；MinIO uploads **264 KB**（几乎空）。 + +## 范围决策（2026-09-03，实际角度） + +- **做：PostgreSQL 逻辑备份** —— 覆盖现实故障（误删、升级失败、磁盘坏、重装），成本≈0。 +- **不做：MinIO/附件备份** —— 桶仅 264 KB，个人实例附件可接受丢失；不为它付日常维护。 + 日后附件明显变多再按原完整版思路加 `mc mirror`（历史版本见本目录 git 历史/Plane 条目评论）。 +- 异机同步暂不启用（见下"局限/阻塞"）。 + +## 方案 + +集群内 CronJob（ns `plane`，每天 **01:30 UTC = 03:30 本地**，控制器按 UTC 跑）： + +1. 单容器 `postgres:15.7-alpine`：`pg_dump -Fc`（自定义压缩格式）打 `plane` 库 + → `/var/backups/plane/pg/plane-.dump`（hostPath `DirectoryOrCreate`） +2. 保留 7 天（`find -mtime +7 -delete`），成功/失败历史各留 3/2 +3. 凭据：现 chart Secret `plane-app-pgdb-secrets`（Phase A 外部化后改 `plane-pgdb-credentials`） + +## 容量 + +- 库 88 MB → `-Fc` 快照约 10–40 MB/天 × 7 天 ≈ **<300 MB**，对 83 G 可用盘可忽略。 + +## 还原（未演练；应用前先做一次隔离测试） + +```bash +# 目标 PG15 实例（临时起一个 postgres:15.7-alpine 容器或另一台机）: +# 先建空库: createdb plane (user=plane) +pg_restore -h -U plane -d plane --clean --if-exists /var/backups/plane/pg/plane-.dump +# 还原后确认 310 issues 量级一致；附件为空属预期（未备份 MinIO） +``` + +## 验收（应用前逐项过） + +- [ ] CronJob 建立后手动触发一次：`kubectl -n plane create job --from=cronjob/plane-backup plane-backup-manual-1`，Job `Completed` +- [ ] `/var/backups/plane/pg/plane-*.dump` 可被 `pg_restore -l` 列出 +- [ ] 备份 Job 只依赖 pgdb 服务，不依赖 Plane 应用 Pod（应用故障期间也能出备份） +- [ ] 保留清理 dry-run（`find ... -print`）正确；`df -h /` 前后对比记录 + +## 局限 / 阻塞 + +- **本地方案不是离机备份**：单节点磁盘/整机故障即丢。如日后要离机，纳入 + [Restic 异机 repository 决策与存取隔离](https://plane.chans.xyz/space/projects/56874283-7e1d-43a8-afa4-631cf1c4ad5b/issues/7825d564-ae15-446b-bced-be26b648346b/) + （与 Matrix 备份同一决策）；恢复演练纪律见 + [服务级 restore runbook 与隔离复元演练](https://plane.chans.xyz/space/projects/56874283-7e1d-43a8-afa4-631cf1c4ad5b/issues/a9bea3ba-c958-4a74-b2f1-6bbb653f21d3/)。 +- 提醒：同一节点 **Matrix 数据价值远高于 Plane 且同样无备份** —— 若投入备份精力，顺序上 Matrix 优先。 diff --git a/docs/plane-hardening/backup/plane-backup.yaml b/docs/plane-hardening/backup/plane-backup.yaml new file mode 100644 index 0000000..4c1b819 --- /dev/null +++ b/docs/plane-hardening/backup/plane-backup.yaml @@ -0,0 +1,63 @@ +# Plane CE PostgreSQL backup CronJob — DRAFT (2026-09-03), NOT applied. +# ns: plane (synapse K3s single node). Output: hostPath /var/backups/plane (root disk, auto-created). +# +# Scope decision (2026-09-03, practical): DB-only. MinIO dropped — uploads bucket +# measured at 264 KB / 444 KB total; attachments are acceptable loss for this +# personal 1-user instance (310 issues / 88 MB DB). Revisit only if usage grows. +# +# Credentials: read from the CURRENT chart-generated Secret (works today). After the +# optional external-secrets migration (docs/plane-hardening/README.md Phase A) switch +# the secretKeyRef name to plane-pgdb-credentials. +# +# Apply: +# ssh windy@synapse.chans.xyz 'sudo k3s kubectl apply -n plane -f -' < plane-backup.yaml +# Manual run + verify: +# sudo k3s kubectl -n plane create job --from=cronjob/plane-backup plane-backup-manual-1 +# sudo k3s kubectl -n plane get cronjob,job,pods | grep plane-backup +# sudo ls -lh /var/backups/plane/pg +# Restore steps + tuning: see backup/README.md + +apiVersion: batch/v1 +kind: CronJob +metadata: + name: plane-backup + namespace: plane +spec: + # 01:30 UTC daily = 03:30 local (CEST). CronJob controller runs in UTC. + schedule: "30 1 * * *" + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 2 + jobTemplate: + spec: + backoffLimit: 2 + template: + spec: + restartPolicy: OnFailure + volumes: + - name: backup + hostPath: + path: /var/backups/plane + type: DirectoryOrCreate + containers: + - name: pg-dump + image: postgres:15.7-alpine + env: + - name: PGPASSWORD + valueFrom: + secretKeyRef: + name: plane-app-pgdb-secrets # -> plane-pgdb-credentials after Phase A + key: POSTGRES_PASSWORD + command: ["/bin/sh", "-c"] + args: + - | + set -euo pipefail + TS=$(date -u +%Y%m%dT%H%M%SZ) + mkdir -p /backup/pg + pg_dump -h plane-app-pgdb.plane.svc.cluster.local -U plane -d plane \ + -Fc -f "/backup/pg/plane-${TS}.dump" + find /backup/pg -type f -name 'plane-*.dump' -mtime +7 -delete + echo "pg_dump done: /backup/pg/plane-${TS}.dump ($(du -h /backup/pg/plane-${TS}.dump | cut -f1))" + volumeMounts: + - name: backup + mountPath: /backup diff --git a/docs/plane-hardening/secrets.yaml.example b/docs/plane-hardening/secrets.yaml.example new file mode 100644 index 0000000..b8bc08a --- /dev/null +++ b/docs/plane-hardening/secrets.yaml.example @@ -0,0 +1,91 @@ +# External Secret structure for Plane CE hardening — EXAMPLE ONLY. +# No real values here; this file is safe to commit. Real values live only on the +# host (/home/windy/plane-k3s, 0600/0700) and in the cluster. +# +# Phase A — create each Secret with the CURRENT cluster values first (zero change): +# # current source Secrets (chart-generated): +# kubectl -n plane get secret plane-app-app-secrets -o jsonpath='{.data.SECRET_KEY}' | base64 -d +# kubectl -n plane get secret plane-app-live-secrets -o jsonpath='{.data.REDIS_URL}' | base64 -d +# kubectl -n plane get secret plane-app-pgdb-secrets -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d +# kubectl -n plane get secret plane-app-rabbitmq-secrets -o jsonpath='{.data.RABBITMQ_DEFAULT_PASS}' | base64 -d +# kubectl -n plane get secret plane-app-doc-store-secrets -o jsonpath='{.data}' | base64 -d +# +# e.g. kubectl -n plane create secret generic plane-app-keys \ +# --from-literal=SECRET_KEY="$()" \ +# --from-literal=LIVE_SERVER_SECRET_KEY="$()" +# +# All keys below are REQUIRED by chart templates/plane-ce-1.8.0 (verified 2026-09-03): +# missing keys are NOT auto-filled once an existingSecret is referenced. + +--- +apiVersion: v1 +kind: Secret +metadata: + name: plane-app-keys # external_secrets.app_keys_existingSecret + namespace: plane +type: Opaque +stringData: + SECRET_KEY: "" # current: copy from plane-app-app-secrets; rotate only in Phase B + LIVE_SERVER_SECRET_KEY: "" # current: same value as above / plane-app-live-secrets + +--- +apiVersion: v1 +kind: Secret +metadata: + name: plane-app-env # external_secrets.app_env_existingSecret + namespace: plane +type: Opaque +stringData: + REDIS_URL: "" # redis://plane-app-redis.plane.svc.cluster.local:6379/ + DATABASE_URL: "" # postgresql://plane:plane@plane-app-pgdb.plane.svc.cluster.local/plane + AMQP_URL: "" # amqp://plane:plane@plane-app-rabbitmq.plane.svc.cluster.local/ + +--- +apiVersion: v1 +kind: Secret +metadata: + name: plane-live-env # external_secrets.live_env_existingSecret + namespace: plane +type: Opaque +stringData: + REDIS_URL: "" # redis://plane-app-redis.plane.svc.cluster.local:6379/ + +--- +apiVersion: v1 +kind: Secret +metadata: + name: plane-pgdb-credentials # external_secrets.pgdb_existingSecret + namespace: plane +type: Opaque +stringData: + POSTGRES_PASSWORD: "" # Phase A: keep current ('plane'); Phase B: ALTER USER first, then sync + POSTGRES_DB: "plane" + POSTGRES_USER: "plane" + +--- +apiVersion: v1 +kind: Secret +metadata: + name: plane-rabbitmq-credentials # external_secrets.rabbitmq_existingSecret + namespace: plane +type: Opaque +stringData: + RABBITMQ_DEFAULT_USER: "plane" + RABBITMQ_DEFAULT_PASS: "" # Phase A: keep current; Phase B: rabbitmqctl change_password first + +--- +apiVersion: v1 +kind: Secret +metadata: + name: plane-minio-credentials # external_secrets.doc_store_existingSecret + namespace: plane +type: Opaque +stringData: + FILE_SIZE_LIMIT: "20971520" # must match env.doc_upload_size_limit + AWS_S3_BUCKET_NAME: "uploads" + USE_MINIO: "1" + MINIO_ROOT_USER: "admin" + MINIO_ROOT_PASSWORD: "" # root creds take effect on first init only + AWS_ACCESS_KEY_ID: "admin" + AWS_SECRET_ACCESS_KEY: "" # == MINIO_ROOT_PASSWORD while minio.local_setup + AWS_S3_ENDPOINT_URL: "http://plane-app-minio:9000" diff --git a/docs/plane-hardening/values.hardened.yaml b/docs/plane-hardening/values.hardened.yaml new file mode 100644 index 0000000..ccf02d2 --- /dev/null +++ b/docs/plane-hardening/values.hardened.yaml @@ -0,0 +1,109 @@ +# Plane CE hardened values — DRAFT (2026-09-03), NOT applied. +# Target file on host: /home/windy/plane-k3s/values.yaml (synapse.chans.xyz) +# Reference release: plane-app, chart plane-ce-1.8.0 (values.yaml L1-362 + templates verified 2026-09-03). +# No secrets in this file. Secret *values* live only in k8s Secrets (see secrets.yaml.example). +# +# Two phases: +# Phase A: externalize secrets (reference names below) with CURRENT values copied -> zero change. +# Phase B: rotate credentials one by one (see README.md). SECRET_KEY rotation is cheap only while +# SMTP/OAuth are unconfigured (no encrypted config rows yet). + +planeVersion: v1.4.1 + +ingress: + enabled: true + appHost: plane.chans.xyz + ingressClass: traefik + traefik: + # 20 MiB (chart default). Keep aligned with env.doc_upload_size_limit below. + maxRequestBodyBytes: 20971520 + +ssl: + createIssuer: true + issuer: http # HTTP-01; ssl_token_existingSecret not needed + email: admin@chans.xyz + generateCerts: true + +postgres: + storageClass: local-path + volumeSize: 5Gi + # NOTE: chart 1.8.0 exposes NO resources knob for the bundled datastores + # (stateful templates render no resources block). Add limits via + # --post-renderer/kustomize or `kubectl -n plane patch sts ...` re-applied on + # every upgrade (P2 task; see README.md). + +redis: + storageClass: local-path + # image: valkey/valkey:7.2.11-alpine # already pinned by chart default; uncomment to make explicit + +minio: + # P2: pin. Digest of the currently running :latest (2026-09-03, pod plane-app-minio-wl-0). + image: minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e + # image_mc: minio/mc@sha256:... # optional: pin one-shot bucket-init client the same way + storageClass: local-path + volumeSize: 5Gi + +rabbitmq: + storageClass: local-path + +env: + # Fail the render instead of ever falling back to the chart's PUBLIC constants + # (values.yaml L340-341 in chart 1.8.0). Requires external_secrets below. + requireExplicitSecrets: true + + # SECRET_KEY / LIVE_SERVER_SECRET_KEY are deliberately OMITTED here. + # They live in k8s Secret `plane-app-keys` (referenced below). With + # requireExplicitSecrets=true and app_keys_existingSecret set, the chart renders + # neither key itself and app+live workloads both envFrom `plane-app-keys` LAST + # (later envFrom wins), which keeps the shared signing key consistent. + + pgdb_name: plane + docstore_bucket: uploads + # Align app-side upload cap with the Traefik body limit (was 5242880/5MiB). + # Keep both at 20MiB, or lower both together. + doc_upload_size_limit: "20971520" + +external_secrets: + # Shared signing keys (used by app + live). REQUIRED keys: SECRET_KEY, LIVE_SERVER_SECRET_KEY. + app_keys_existingSecret: plane-app-keys + # REQUIRED keys: REDIS_URL, DATABASE_URL, AMQP_URL (chart-derived URLs; update on DB/queue rotation). + app_env_existingSecret: plane-app-env + # REQUIRED keys: REDIS_URL. + live_env_existingSecret: plane-live-env + # REQUIRED keys: POSTGRES_PASSWORD, POSTGRES_DB, POSTGRES_USER. + pgdb_existingSecret: plane-pgdb-credentials + # REQUIRED keys: RABBITMQ_DEFAULT_USER, RABBITMQ_DEFAULT_PASS. + rabbitmq_existingSecret: plane-rabbitmq-credentials + # REQUIRED keys: FILE_SIZE_LIMIT, AWS_S3_BUCKET_NAME, USE_MINIO, MINIO_ROOT_USER, + # MINIO_ROOT_PASSWORD, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_S3_ENDPOINT_URL. + doc_store_existingSecret: plane-minio-credentials + # ssl_token_existingSecret: '' # DNS-01 only (cloudflare/digitalocean); unused with HTTP-01 + +# Optional, P2: HTTP -> HTTPS 301. The chart's own IngressRoute binds only +# 'websecure' (http:// currently 404s). extraObjects is rendered verbatim (toYaml). +# Uncomment and `helm upgrade` once reviewed: +# extraObjects: +# - apiVersion: traefik.io/v1alpha1 +# kind: Middleware +# metadata: +# name: plane-https-redirect +# namespace: plane +# spec: +# redirectScheme: +# scheme: https +# permanent: true +# - apiVersion: traefik.io/v1alpha1 +# kind: IngressRoute +# metadata: +# name: plane-http-to-https +# namespace: plane +# spec: +# entryPoints: [web] +# routes: +# - match: Host(`plane.chans.xyz`) +# kind: Rule +# middlewares: +# - name: plane-https-redirect +# services: +# - name: plane-app-web +# port: 3000 diff --git a/hosts/synapse.chans.xyz.md b/hosts/synapse.chans.xyz.md index 4f4afca..e7ecf09 100644 --- a/hosts/synapse.chans.xyz.md +++ b/hosts/synapse.chans.xyz.md @@ -24,6 +24,7 @@ ssh -4 windy@synapse.chans.xyz | DB | ESS embedded PostgreSQL 17 (PVC 20Gi, local-path) | | Cache | ESS embedded Redis (PVC 2Gi) | | Chart | `oci://ghcr.io/element-hq/ess-helm/matrix-stack`, version `26.7.2` | +| Plane | Helm `plane-ce-1.8.0` (app `v1.4.1`), namespace `plane` — self-hosted Plane project management | ### Matrix service endpoints @@ -51,8 +52,50 @@ All other ports internal only (no K3s API, no database, no Redis exposed). - `ess` — all ESS workloads (Synapse, MAS, Element, Postgres, Redis, HAProxy) - `matrix-system` — cluster base resources (ResourceQuota, LimitRange, mrtc-placeholder) +- `plane` — Plane project management (Helm release `plane-app`) - `cert-manager` — cert-manager +## Plane (project management) + +Self-hosted [Plane](https://github.com/makeplane/plane) on the same K3s node, deployed via the official `plane-ce` Helm chart. + +| Item | Detail | +|------|--------| +| Release | `plane-app` (ns `plane`), chart `plane-ce-1.8.0`, app `v1.4.1`, revision 1 | +| URL | https://plane.chans.xyz | +| Install date | 2026-09-01 | +| Values source | `/home/windy/plane-k3s/values.yaml` (plain file, not a git repo) | +| Images | `artifacts.plane.so/makeplane/*` (`plane-frontend`, `plane-backend`, `plane-admin`, `plane-live`), pullPolicy `Always` | +| Ingress | Traefik `IngressRoute` `plane-app-ingress` — `/`→web, `/api` `/auth`→api, `/spaces`→space, `/god-mode`→admin, `/live`→live, `/uploads`→minio; `maxRequestBodyBytes` 20Mi | +| TLS | Own namespace `Issuer` `plane-app-cert-issuer` (HTTP-01, LE prod, `admin@chans.xyz`); cert `plane-app-ssl-cert` (CN `plane.chans.xyz`) | +| DB | Bundled Postgres `15.7-alpine` (PVC 5Gi, local-path) | +| Cache/queue | Bundled Redis (PVC 100Mi), RabbitMQ `3.13.6-management-alpine` (PVC 100Mi) | +| Storage | Bundled MinIO (`minio/minio:latest`, root user `admin`, PVC 5Gi) — S3 for uploads/docs | +| Resources | Every workload: cpu 50m/500m, mem 50Mi/1000Mi, replicas 1 | +| SMTP | Not configured (no `smtp` values) — Plane invites/password resets won't email yet | + +Workloads (all 1/1 Running): 7 Deployments (`plane-app-{admin,api,beat-worker,live,space,web,worker}-wl`) + 4 StatefulSets (`plane-app-{minio,pgdb,rabbitmq,redis}-wl`); init Jobs `api-migrate-1` / `minio-bucket-1` Completed. All PVCs Bound on `local-path` (root disk). + +### Plane configuration notes + +- **`planeVersion: v1.4.1`** pinned in values.yaml; chart tracks Plane's own tags. +- **Secrets**: Helm-generated Opaque secrets (`plane-app-app-secrets`, `-doc-store-secrets`, `-pgdb-secrets`, `-rabbitmq-secrets`, `-live-secrets`); `requireExplicitSecrets: false`. Values live in `$SECRET_KEY`, `DATABASE_URL`, `AMQP_URL`, `REDIS_URL` etc. +- **Sentry / CORS**: `sentry_dsn` and `cors_allowed_origins` empty (defaults fine for single-host). +- **MinIO is `latest` tag** — pin a version for reproducibility. +- **Backup**: NOT covered by `/var/backups/matrix` (which is paused anyway) — Plane Postgres/MinIO PVCs have no backup tier yet. + +### Plane verification + +```bash +# Release + workloads +sudo helm list -A +sudo k3s kubectl -n plane get deploy,sts,pods -o wide +# Cert + ingress +sudo k3s kubectl -n plane get certificate,ingressroute +# Endpoint +curl -4 -s -o /dev/null -w '%{http_code}\n' https://plane.chans.xyz/ +``` + ## Local backup | Item | Detail | @@ -62,7 +105,7 @@ All other ports internal only (no K3s API, no database, no Redis exposed). | Retention | 7 days | | Disk warning | 80% (healthcheck), 90% (backup stops) | | Content | Planned: PostgreSQL `synapse` + `mas` logical dumps, media store archive, `/etc/matrix-bootstrap` | -| Status | **Not operational** — no current Matrix backup or recovery tier | +| Status | **Not operational** — no current Matrix backup or recovery tier. **Plane data (its own Postgres + MinIO PVCs in ns `plane`) is also not covered by any backup.** | ## Health checks @@ -101,5 +144,6 @@ diagnosis and imperative recovery work. - MatrixRTC / Element Call / LiveKit / Coturn not deployed (`mrtc.chans.xyz` reserved only) - SMTP email not yet configured (requires manual secret bootstrap followed by a reviewed Ansible stack deployment) +- Plane `minio` image uses `latest` tag (pin a version) - No off-site Restic backup - Single-node K3s (no HA for control plane) diff --git a/inventory/hosts.md b/inventory/hosts.md index 962cdb7..c5628cd 100644 --- a/inventory/hosts.md +++ b/inventory/hosts.md @@ -48,4 +48,5 @@ the software deployed there, see [the LAN overview](../docs/lan-overview.md). | https://synapse.chans.xyz | Synapse API | Client-Server + Federation API | | https://account.chans.xyz | MAS | Matrix Authentication Service (local passwords) | | https://admin.chans.xyz | Element Admin | Admin console (MAS admin auth) | +| https://plane.chans.xyz | Plane | Project management (Helm `plane-ce` v1.4.1, ns `plane`) | | `mrtc.chans.xyz` | MatrixRTC | **Reserved** – not deployed | diff --git a/runbooks/README.md b/runbooks/README.md index 614b9ec..3c25f6f 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -16,6 +16,7 @@ runbooks start from [`_template.md`](_template.md). | PowerDNS health check | [pdns-health.md](pdns-health.md) | read-only | | RustDesk health check | [rustdesk-health.md](rustdesk-health.md) | read-only | | Matrix health check | [matrix-health.md](matrix-health.md) | read-only | +| Plane health check | [plane-health.md](plane-health.md) | read-only | | pgdb health check | [pgdb-health.md](pgdb-health.md) | read-only | | pgdb DB restore (pg_restore) | [pgdb-restore.md](pgdb-restore.md) | change (procedure) | | pgdb image/compose update | [pgdb-update.md](pgdb-update.md) | change (gated) | diff --git a/runbooks/plane-health.md b/runbooks/plane-health.md new file mode 100644 index 0000000..6e3aaf1 --- /dev/null +++ b/runbooks/plane-health.md @@ -0,0 +1,93 @@ +# Runbook: Plane Health Check + +## Purpose + +Read-only health check of the self-hosted Plane project-management instance +(`plane.chans.xyz`) running on the synapse K3s cluster. + +## Scope + +- Applicable: [synapse.chans.xyz](../hosts/synapse.chans.xyz.md), namespace `plane`. +- Read-only: does not change pods, ingress, certificates, secrets, or configuration. +- Not applicable: Plane upgrade, values changes, or data recovery — those need a + reviewed change (see `ansible-operations.md` / `release.md`). + +## Ownership + +- Owner: personal ops (Windy) +- Last reviewed: 2026-09-02 +- Related systems: synapse.chans.xyz (Helm `plane-app`, chart `plane-ce-1.8.0`, app `v1.4.1`) + +## Pass criteria + +All of the following must hold; any conflict means `STOP` and record evidence. + +- `sudo helm list -A` shows `plane-app` in ns `plane`, STATUS `deployed`. +- All 7 Deployments + 4 StatefulSets in ns `plane` are `1/1 Running` with 0 recent restarts. +- Init Jobs `api-migrate-*` / `minio-bucket-*` are `Completed`. +- Certificate `plane-app-ssl-cert` is `READY=True` (CN `plane.chans.xyz`). +- `https://plane.chans.xyz/` returns HTTP 200 with a valid Let's Encrypt cert. +- Root disk usage below the 80% warning threshold. + +## Procedure + +### 1. Release and workloads + +```bash +ssh -4 windy@synapse.chans.xyz 'sudo helm list -A' +ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get deploy,sts,pods -o wide' +``` + +Expected: `plane-app` deployed; all workloads `1/1 Running`, `RESTARTS` low; +no `CrashLoopBackOff` / `Evicted`. Otherwise `STOP` and record evidence. + +### 2. TLS certificate and ingress + +```bash +ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get certificate,issuer,ingressroute' +``` + +Expected: `plane-app-ssl-cert` `READY=True`; `plane-app-ingress` present with +routes for `/`, `/api`, `/spaces`, `/god-mode`, `/live`, `/uploads`. + +If `READY=False` with a pending HTTP-01 challenge, the usual cause is the node +DNS chain (coreDNS → systemd-resolved → uplink) failing to resolve +`plane.chans.xyz` — check `resolvectl query plane.chans.xyz` vs +`dig +short plane.chans.xyz @8.8.8.8`. If the record exists publicly but the +node fails, `sudo resolvectl flush-caches` and wait for the cert-manager retry; +do not mutate the issuer. + +### 3. Endpoint verification + +```bash +curl -4 -s -o /dev/null -w '%{http_code}\n' https://plane.chans.xyz/ +echo | openssl s_client -connect plane.chans.xyz:443 -servername plane.chans.xyz 2>/dev/null | openssl x509 -noout -subject -issuer -dates +``` + +Expected: HTTP 200, cert `CN=plane.chans.xyz` issued by Let's Encrypt with a +future `notAfter`. `https://plane.chans.xyz/api/` may 404 — the API serves +under `/api/...` paths only, so a bare 404 there is not a failure. + +### 4. Storage and disk + +```bash +ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get pvc' +ssh -4 windy@synapse.chans.xyz 'df -hP /' +``` + +Expected: all 4 PVCs `Bound` (minio 5Gi, pgdb 5Gi, rabbitmq 100Mi, redis 100Mi +on `local-path`); root disk < 80%. + +## Safety + +- Read-only: never mutate pods, ingress, certificates, or configuration during + this check. +- Plane data (ns `plane` Postgres + MinIO PVCs) has **no backup tier**; treat + the instance as at-risk until a backup design exists. +- If live state conflicts with the expected values above, `STOP` and record + evidence; do not "fix in passing". + +## References + +- [hosts/synapse.chans.xyz.md](../hosts/synapse.chans.xyz.md) — plane stack facts +- [matrix-health.md](matrix-health.md) — sibling service on the same cluster