docs(plane): 自托管 Plane 落地事实入仓库 + plane-health runbook + hardening 草稿
记录源 Linear→Plane (2026-09-03 起, Plane MCP) + plane.chans.xyz 服务行/upstream 段; inventory + hosts/synapse.chans.xyz.md 补 Plane 部署事实 (Helm plane-ce-1.8.0 / app v1.4.1, ns plane, IngressRoute/自有证书 issuer/PVC 5+5Gi local-path/无备份层); 新增 runbooks/plane-health.md (只读健康检查) 与 docs/plane-hardening/ 草稿 (values.hardened.yaml、secrets.yaml.example 占位、backup/ CronJob), 均为未应用设计稿; .gitignore 增加 .tmp-* agent 临时文件。
This commit is contained in:
@@ -32,3 +32,4 @@ skills-lock.json
|
|||||||
*~
|
*~
|
||||||
# Agent working scratch (not repo content).
|
# Agent working scratch (not repo content).
|
||||||
.agent-work/
|
.agent-work/
|
||||||
|
.tmp-*
|
||||||
|
|||||||
@@ -30,9 +30,12 @@ This repo is the **agent ops handbook + fact source** for maintaining personal V
|
|||||||
> read-only unless a change is approved).
|
> read-only unless a change is approved).
|
||||||
|
|
||||||
7. Record each material VPS operation, incident, configuration change, or
|
7. Record each material VPS operation, incident, configuration change, or
|
||||||
verification outcome in the corresponding **Linear `vps` project**. Include
|
verification outcome in the corresponding **Plane `vps` project**
|
||||||
scope, action, verification, and remaining follow-up; never put passwords,
|
(self-hosted `plane.chans.xyz`, Plane MCP `mcp__plane__*`). **Linear is
|
||||||
tokens, private keys, recovery keys, or private room IDs in Linear.
|
retired as a record source (2026-09-03) — do not create Linear issues;**
|
||||||
|
existing W1N-* entries are read-only history. Include scope, action,
|
||||||
|
verification, and remaining follow-up; never put passwords, tokens, private
|
||||||
|
keys, recovery keys, or private room IDs in Plane or Linear.
|
||||||
|
|
||||||
### Runbook execution rules
|
### Runbook execution rules
|
||||||
|
|
||||||
@@ -67,6 +70,7 @@ not maintain a second copy of the machine table here.
|
|||||||
| https://synapse.chans.xyz | synapse | Synapse Client-Server + Federation API |
|
| https://synapse.chans.xyz | synapse | Synapse Client-Server + Federation API |
|
||||||
| https://account.chans.xyz | synapse | Matrix Authentication Service (local passwords) |
|
| https://account.chans.xyz | synapse | Matrix Authentication Service (local passwords) |
|
||||||
| https://admin.chans.xyz | synapse | Element Admin console (MAS admin auth) |
|
| https://admin.chans.xyz | synapse | Element Admin console (MAS admin auth) |
|
||||||
|
| https://plane.chans.xyz | synapse (`plane`, Helm `plane-ce` 1.8.0 / v1.4.1) | Plane project management (self-hosted, K3s) |
|
||||||
|
|
||||||
### Upstream docs
|
### Upstream docs
|
||||||
|
|
||||||
@@ -76,6 +80,8 @@ not maintain a second copy of the machine table here.
|
|||||||
|
|
||||||
**Matrix (ESS on synapse):** Matrix homeserver running on `synapse.chans.xyz` via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone `chans.xyz` managed by hk2 PowerDNS. Before changing config, read [docs/matrix-upstream.md](docs/matrix-upstream.md) and [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: [matrix-health](runbooks/matrix-health.md).
|
**Matrix (ESS on synapse):** Matrix homeserver running on `synapse.chans.xyz` via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone `chans.xyz` managed by hk2 PowerDNS. Before changing config, read [docs/matrix-upstream.md](docs/matrix-upstream.md) and [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: [matrix-health](runbooks/matrix-health.md).
|
||||||
|
|
||||||
|
**Plane (on synapse):** Self-hosted Plane project management at `plane.chans.xyz`, Helm release `plane-app` (chart `plane-ce-1.8.0`, app `v1.4.1`) in ns `plane` on the same K3s node as Matrix. Config from `/home/windy/plane-k3s/values.yaml`; workload/cert/ingress details in [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). Its Postgres/MinIO PVCs are **not** backed up.
|
||||||
|
|
||||||
**RustDesk:** Self-hosted RustDesk server on `hk2.chans.xyz` (`/opt/rustdesk`, containers `hbbs`/`hbbr`, image pinned `1.1.14`). The `hbbs -r` relay hostname must resolve to the host's public IP `154.36.174.161` — use `hk2.chans.xyz` (never `hk2.wsvc.info`, which has no DNS record). Health: [rustdesk-health](runbooks/rustdesk-health.md).
|
**RustDesk:** Self-hosted RustDesk server on `hk2.chans.xyz` (`/opt/rustdesk`, containers `hbbs`/`hbbr`, image pinned `1.1.14`). The `hbbs -r` relay hostname must resolve to the host's public IP `154.36.174.161` — use `hk2.chans.xyz` (never `hk2.wsvc.info`, which has no DNS record). Health: [rustdesk-health](runbooks/rustdesk-health.md).
|
||||||
|
|
||||||
## Runbooks & scripts
|
## Runbooks & scripts
|
||||||
@@ -91,6 +97,7 @@ not maintain a second copy of the machine table here.
|
|||||||
| PowerDNS health (hk2) | [runbooks/pdns-health.md](runbooks/pdns-health.md) |
|
| PowerDNS health (hk2) | [runbooks/pdns-health.md](runbooks/pdns-health.md) |
|
||||||
| PowerDNS upstream refs | [docs/pdns-upstream.md](docs/pdns-upstream.md) |
|
| PowerDNS upstream refs | [docs/pdns-upstream.md](docs/pdns-upstream.md) |
|
||||||
| Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) |
|
| Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) |
|
||||||
|
| Plane health | [runbooks/plane-health.md](runbooks/plane-health.md) |
|
||||||
| RustDesk health (hk2) | [runbooks/rustdesk-health.md](runbooks/rustdesk-health.md) |
|
| RustDesk health (hk2) | [runbooks/rustdesk-health.md](runbooks/rustdesk-health.md) |
|
||||||
| AdGuard Home health | [runbooks/adguard-home-health.md](runbooks/adguard-home-health.md) |
|
| AdGuard Home health | [runbooks/adguard-home-health.md](runbooks/adguard-home-health.md) |
|
||||||
| Home Assistant maintenance | [runbooks/home-assistant-maintenance.md](runbooks/home-assistant-maintenance.md) + [scripts/ha-maintenance.sh](runbooks/scripts/ha-maintenance.sh) |
|
| Home Assistant maintenance | [runbooks/home-assistant-maintenance.md](runbooks/home-assistant-maintenance.md) + [scripts/ha-maintenance.sh](runbooks/scripts/ha-maintenance.sh) |
|
||||||
@@ -116,7 +123,11 @@ Routine mailcow health: `cd ansible && ansible-playbook playbooks/health-report.
|
|||||||
|
|
||||||
### Issue tracker
|
### Issue tracker
|
||||||
|
|
||||||
Issues are tracked in Linear and created/updated via the Linear MCP (`vps` project). See `docs/agents/issue-tracker.md`.
|
Issues are tracked in **Plane** — self-hosted at `plane.chans.xyz`, project
|
||||||
|
`vps` — and created/updated via the Plane MCP (`mcp__plane__*`). **Linear is
|
||||||
|
retired as a record source (2026-09-03); do not create Linear issues.** Existing
|
||||||
|
W1N-* entries are read-only history. `docs/agents/issue-tracker.md` documents
|
||||||
|
the retired Linear workflow and is stale; treat this section as authoritative.
|
||||||
|
|
||||||
### Triage labels
|
### Triage labels
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# Plane CE 加固草稿(docs/plane-hardening/)
|
||||||
|
|
||||||
|
> **状态:草稿,未应用、未提交。** 对应追踪:Plane vps 项目条目(2026-09-03,**记录源**;Linear W1N-277 已取消,Linear 自 2026-09-03 起不再作为记录源)。
|
||||||
|
> 线上实例:`plane.chans.xyz`(synapse K3s,ns `plane`,release `plane-app` = chart `plane-ce-1.8.0` / app `v1.4.1`)。
|
||||||
|
> 依据:2026-09-03 只读核查(13 条审查意见中 11 条属实、#3 基本属实、#9 指标归属错误)+ 上游 chart 模板逐条核对。
|
||||||
|
|
||||||
|
## 文件
|
||||||
|
|
||||||
|
| 文件 | 内容 |
|
||||||
|
|------|------|
|
||||||
|
| `values.hardened.yaml` | 可选硬化 values(external secrets 引用、requireExplicitSecrets、minio pin、上传限额对齐);含 HTTP→HTTPS `extraObjects` 示例 |
|
||||||
|
| `secrets.yaml.example` | 6 组外部 Secret 结构占位(只含 key 名,真实值仅存宿主机) |
|
||||||
|
| `backup/plane-backup.yaml` | **PostgreSQL 备份 CronJob**(pg_dump `-Fc`,hostPath `/var/backups/plane`;MinIO 已按实际用量剔除) |
|
||||||
|
| `backup/README.md` | 备份方案说明(排程/容量/保留/还原/阻塞) |
|
||||||
|
|
||||||
|
## 应用顺序(每步先 diff 后执行,全部需用户逐项确认)
|
||||||
|
|
||||||
|
### 现在就值得做:DB 备份(P0,见 backup/)
|
||||||
|
`plane-backup.yaml` 部署 + 手动触发验证一次即可;88 MB 库每日快照几乎零成本。
|
||||||
|
|
||||||
|
### 可选(顺手做一次,不是必须)
|
||||||
|
- **Phase A 密钥外部化**(零行为变化、无停机,约 15 分钟):按 `secrets.yaml.example`
|
||||||
|
在宿主机建 6 个 Secret(值先复制当前集群),用 `values.hardened.yaml` 跑
|
||||||
|
`helm diff upgrade` → `helm upgrade`;验证后删除 chart 生成的旧 Secret。
|
||||||
|
价值:默认密钥不再落在 chart 公开常量上,作为保险。
|
||||||
|
- **MCP API Key 轮换**:若审查对话出过你的环境,Plane 后台重生成 + 更新
|
||||||
|
`/home/windy/plane-k3s/mcp/mcp.env`(0600)+ 重启 Cursor MCP。
|
||||||
|
- **/god-mode IP 白名单**:若在意管理后台被公网爆破。chart 1.8.0 的 IngressRoute
|
||||||
|
不支持给单条路由追加 middleware → 需 post-renderer 或 upgrade 后 `kubectl patch`
|
||||||
|
(升级会覆盖,需固化);源 IP 清单待提供。
|
||||||
|
|
||||||
|
### 明确暂缓/跳过(个人单节点,等出现症状再处理)
|
||||||
|
- SECRET_KEY 等轮换(Phase B):等真要配 SMTP/OAuth 前再做(避免旧密文不可解)。
|
||||||
|
- NetworkPolicy、有状态组件 resources limits(chart 无 values 开关,需 post-render/patch)、
|
||||||
|
HTTP→HTTPS(草稿已给 `extraObjects` 示例)、metrics-server/Sentry。
|
||||||
|
|
||||||
|
## 关键限制(chart 1.8.0 模板已核对)
|
||||||
|
- `external_secrets.*_existingSecret` 设置后,对应 Secret **必须**包含模板所需全部 key
|
||||||
|
(缺失不自动补),见 `secrets.yaml.example` 注释。
|
||||||
|
- `app_keys_existingSecret` 的 envFrom 在所有 workload 上**最后注入**(后置生效),
|
||||||
|
保证 app/live 共享密钥一致——不要在其后再放同名 key 的 Secret。
|
||||||
|
- `DATABASE_URL`/`AMQP_URL`/`REDIS_URL` 是 chart 生成的派生 URL,内嵌明文密码;
|
||||||
|
外部化后轮换 DB/队列密码时必须同步更新 `plane-app-env`。
|
||||||
|
- minio 的 `MINIO_ROOT_*` 与 `AWS_*` 同源于一个 Secret;升级时 bucket Job 会重跑
|
||||||
|
(需 admin 权限凭据)——换 svcacct 前先确认权限覆盖该 Job。
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# Plane CE 备份方案(DB-only)— DRAFT (2026-09-03), 未应用
|
||||||
|
|
||||||
|
> 关联:`plane-backup.yaml`(CronJob);追踪:Plane vps 项目条目(记录源,2026-09-03 起不用 Linear)。
|
||||||
|
> 现状(实测):pg 全库 **88 MB**(310 issues / 1 user);MinIO uploads **264 KB**(几乎空)。
|
||||||
|
|
||||||
|
## 范围决策(2026-09-03,实际角度)
|
||||||
|
|
||||||
|
- **做:PostgreSQL 逻辑备份** —— 覆盖现实故障(误删、升级失败、磁盘坏、重装),成本≈0。
|
||||||
|
- **不做:MinIO/附件备份** —— 桶仅 264 KB,个人实例附件可接受丢失;不为它付日常维护。
|
||||||
|
日后附件明显变多再按原完整版思路加 `mc mirror`(历史版本见本目录 git 历史/Plane 条目评论)。
|
||||||
|
- 异机同步暂不启用(见下"局限/阻塞")。
|
||||||
|
|
||||||
|
## 方案
|
||||||
|
|
||||||
|
集群内 CronJob(ns `plane`,每天 **01:30 UTC = 03:30 本地**,控制器按 UTC 跑):
|
||||||
|
|
||||||
|
1. 单容器 `postgres:15.7-alpine`:`pg_dump -Fc`(自定义压缩格式)打 `plane` 库
|
||||||
|
→ `/var/backups/plane/pg/plane-<UTC时间戳>.dump`(hostPath `DirectoryOrCreate`)
|
||||||
|
2. 保留 7 天(`find -mtime +7 -delete`),成功/失败历史各留 3/2
|
||||||
|
3. 凭据:现 chart Secret `plane-app-pgdb-secrets`(Phase A 外部化后改 `plane-pgdb-credentials`)
|
||||||
|
|
||||||
|
## 容量
|
||||||
|
|
||||||
|
- 库 88 MB → `-Fc` 快照约 10–40 MB/天 × 7 天 ≈ **<300 MB**,对 83 G 可用盘可忽略。
|
||||||
|
|
||||||
|
## 还原(未演练;应用前先做一次隔离测试)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 目标 PG15 实例(临时起一个 postgres:15.7-alpine 容器或另一台机):
|
||||||
|
# 先建空库: createdb plane (user=plane)
|
||||||
|
pg_restore -h <target> -U plane -d plane --clean --if-exists /var/backups/plane/pg/plane-<TS>.dump
|
||||||
|
# 还原后确认 310 issues 量级一致;附件为空属预期(未备份 MinIO)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 验收(应用前逐项过)
|
||||||
|
|
||||||
|
- [ ] CronJob 建立后手动触发一次:`kubectl -n plane create job --from=cronjob/plane-backup plane-backup-manual-1`,Job `Completed`
|
||||||
|
- [ ] `/var/backups/plane/pg/plane-*.dump` 可被 `pg_restore -l` 列出
|
||||||
|
- [ ] 备份 Job 只依赖 pgdb 服务,不依赖 Plane 应用 Pod(应用故障期间也能出备份)
|
||||||
|
- [ ] 保留清理 dry-run(`find ... -print`)正确;`df -h /` 前后对比记录
|
||||||
|
|
||||||
|
## 局限 / 阻塞
|
||||||
|
|
||||||
|
- **本地方案不是离机备份**:单节点磁盘/整机故障即丢。如日后要离机,纳入
|
||||||
|
[Restic 异机 repository 决策与存取隔离](https://plane.chans.xyz/space/projects/56874283-7e1d-43a8-afa4-631cf1c4ad5b/issues/7825d564-ae15-446b-bced-be26b648346b/)
|
||||||
|
(与 Matrix 备份同一决策);恢复演练纪律见
|
||||||
|
[服务级 restore runbook 与隔离复元演练](https://plane.chans.xyz/space/projects/56874283-7e1d-43a8-afa4-631cf1c4ad5b/issues/a9bea3ba-c958-4a74-b2f1-6bbb653f21d3/)。
|
||||||
|
- 提醒:同一节点 **Matrix 数据价值远高于 Plane 且同样无备份** —— 若投入备份精力,顺序上 Matrix 优先。
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# Plane CE PostgreSQL backup CronJob — DRAFT (2026-09-03), NOT applied.
|
||||||
|
# ns: plane (synapse K3s single node). Output: hostPath /var/backups/plane (root disk, auto-created).
|
||||||
|
#
|
||||||
|
# Scope decision (2026-09-03, practical): DB-only. MinIO dropped — uploads bucket
|
||||||
|
# measured at 264 KB / 444 KB total; attachments are acceptable loss for this
|
||||||
|
# personal 1-user instance (310 issues / 88 MB DB). Revisit only if usage grows.
|
||||||
|
#
|
||||||
|
# Credentials: read from the CURRENT chart-generated Secret (works today). After the
|
||||||
|
# optional external-secrets migration (docs/plane-hardening/README.md Phase A) switch
|
||||||
|
# the secretKeyRef name to plane-pgdb-credentials.
|
||||||
|
#
|
||||||
|
# Apply:
|
||||||
|
# ssh windy@synapse.chans.xyz 'sudo k3s kubectl apply -n plane -f -' < plane-backup.yaml
|
||||||
|
# Manual run + verify:
|
||||||
|
# sudo k3s kubectl -n plane create job --from=cronjob/plane-backup plane-backup-manual-1
|
||||||
|
# sudo k3s kubectl -n plane get cronjob,job,pods | grep plane-backup
|
||||||
|
# sudo ls -lh /var/backups/plane/pg
|
||||||
|
# Restore steps + tuning: see backup/README.md
|
||||||
|
|
||||||
|
apiVersion: batch/v1
|
||||||
|
kind: CronJob
|
||||||
|
metadata:
|
||||||
|
name: plane-backup
|
||||||
|
namespace: plane
|
||||||
|
spec:
|
||||||
|
# 01:30 UTC daily = 03:30 local (CEST). CronJob controller runs in UTC.
|
||||||
|
schedule: "30 1 * * *"
|
||||||
|
concurrencyPolicy: Forbid
|
||||||
|
successfulJobsHistoryLimit: 3
|
||||||
|
failedJobsHistoryLimit: 2
|
||||||
|
jobTemplate:
|
||||||
|
spec:
|
||||||
|
backoffLimit: 2
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
restartPolicy: OnFailure
|
||||||
|
volumes:
|
||||||
|
- name: backup
|
||||||
|
hostPath:
|
||||||
|
path: /var/backups/plane
|
||||||
|
type: DirectoryOrCreate
|
||||||
|
containers:
|
||||||
|
- name: pg-dump
|
||||||
|
image: postgres:15.7-alpine
|
||||||
|
env:
|
||||||
|
- name: PGPASSWORD
|
||||||
|
valueFrom:
|
||||||
|
secretKeyRef:
|
||||||
|
name: plane-app-pgdb-secrets # -> plane-pgdb-credentials after Phase A
|
||||||
|
key: POSTGRES_PASSWORD
|
||||||
|
command: ["/bin/sh", "-c"]
|
||||||
|
args:
|
||||||
|
- |
|
||||||
|
set -euo pipefail
|
||||||
|
TS=$(date -u +%Y%m%dT%H%M%SZ)
|
||||||
|
mkdir -p /backup/pg
|
||||||
|
pg_dump -h plane-app-pgdb.plane.svc.cluster.local -U plane -d plane \
|
||||||
|
-Fc -f "/backup/pg/plane-${TS}.dump"
|
||||||
|
find /backup/pg -type f -name 'plane-*.dump' -mtime +7 -delete
|
||||||
|
echo "pg_dump done: /backup/pg/plane-${TS}.dump ($(du -h /backup/pg/plane-${TS}.dump | cut -f1))"
|
||||||
|
volumeMounts:
|
||||||
|
- name: backup
|
||||||
|
mountPath: /backup
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
# External Secret structure for Plane CE hardening — EXAMPLE ONLY.
|
||||||
|
# No real values here; this file is safe to commit. Real values live only on the
|
||||||
|
# host (/home/windy/plane-k3s, 0600/0700) and in the cluster.
|
||||||
|
#
|
||||||
|
# Phase A — create each Secret with the CURRENT cluster values first (zero change):
|
||||||
|
# # current source Secrets (chart-generated):
|
||||||
|
# kubectl -n plane get secret plane-app-app-secrets -o jsonpath='{.data.SECRET_KEY}' | base64 -d
|
||||||
|
# kubectl -n plane get secret plane-app-live-secrets -o jsonpath='{.data.REDIS_URL}' | base64 -d
|
||||||
|
# kubectl -n plane get secret plane-app-pgdb-secrets -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d
|
||||||
|
# kubectl -n plane get secret plane-app-rabbitmq-secrets -o jsonpath='{.data.RABBITMQ_DEFAULT_PASS}' | base64 -d
|
||||||
|
# kubectl -n plane get secret plane-app-doc-store-secrets -o jsonpath='{.data}' | base64 -d
|
||||||
|
#
|
||||||
|
# e.g. kubectl -n plane create secret generic plane-app-keys \
|
||||||
|
# --from-literal=SECRET_KEY="$(<copy from above>)" \
|
||||||
|
# --from-literal=LIVE_SERVER_SECRET_KEY="$(<copy from above>)"
|
||||||
|
#
|
||||||
|
# All keys below are REQUIRED by chart templates/plane-ce-1.8.0 (verified 2026-09-03):
|
||||||
|
# missing keys are NOT auto-filled once an existingSecret is referenced.
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: plane-app-keys # external_secrets.app_keys_existingSecret
|
||||||
|
namespace: plane
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
SECRET_KEY: "" # current: copy from plane-app-app-secrets; rotate only in Phase B
|
||||||
|
LIVE_SERVER_SECRET_KEY: "" # current: same value as above / plane-app-live-secrets
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: plane-app-env # external_secrets.app_env_existingSecret
|
||||||
|
namespace: plane
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
REDIS_URL: "" # redis://plane-app-redis.plane.svc.cluster.local:6379/
|
||||||
|
DATABASE_URL: "" # postgresql://plane:plane@plane-app-pgdb.plane.svc.cluster.local/plane
|
||||||
|
AMQP_URL: "" # amqp://plane:plane@plane-app-rabbitmq.plane.svc.cluster.local/
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: plane-live-env # external_secrets.live_env_existingSecret
|
||||||
|
namespace: plane
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
REDIS_URL: "" # redis://plane-app-redis.plane.svc.cluster.local:6379/
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: plane-pgdb-credentials # external_secrets.pgdb_existingSecret
|
||||||
|
namespace: plane
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
POSTGRES_PASSWORD: "" # Phase A: keep current ('plane'); Phase B: ALTER USER first, then sync
|
||||||
|
POSTGRES_DB: "plane"
|
||||||
|
POSTGRES_USER: "plane"
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: plane-rabbitmq-credentials # external_secrets.rabbitmq_existingSecret
|
||||||
|
namespace: plane
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
RABBITMQ_DEFAULT_USER: "plane"
|
||||||
|
RABBITMQ_DEFAULT_PASS: "" # Phase A: keep current; Phase B: rabbitmqctl change_password first
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: plane-minio-credentials # external_secrets.doc_store_existingSecret
|
||||||
|
namespace: plane
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
FILE_SIZE_LIMIT: "20971520" # must match env.doc_upload_size_limit
|
||||||
|
AWS_S3_BUCKET_NAME: "uploads"
|
||||||
|
USE_MINIO: "1"
|
||||||
|
MINIO_ROOT_USER: "admin"
|
||||||
|
MINIO_ROOT_PASSWORD: "" # root creds take effect on first init only
|
||||||
|
AWS_ACCESS_KEY_ID: "admin"
|
||||||
|
AWS_SECRET_ACCESS_KEY: "" # == MINIO_ROOT_PASSWORD while minio.local_setup
|
||||||
|
AWS_S3_ENDPOINT_URL: "http://plane-app-minio:9000"
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
# Plane CE hardened values — DRAFT (2026-09-03), NOT applied.
|
||||||
|
# Target file on host: /home/windy/plane-k3s/values.yaml (synapse.chans.xyz)
|
||||||
|
# Reference release: plane-app, chart plane-ce-1.8.0 (values.yaml L1-362 + templates verified 2026-09-03).
|
||||||
|
# No secrets in this file. Secret *values* live only in k8s Secrets (see secrets.yaml.example).
|
||||||
|
#
|
||||||
|
# Two phases:
|
||||||
|
# Phase A: externalize secrets (reference names below) with CURRENT values copied -> zero change.
|
||||||
|
# Phase B: rotate credentials one by one (see README.md). SECRET_KEY rotation is cheap only while
|
||||||
|
# SMTP/OAuth are unconfigured (no encrypted config rows yet).
|
||||||
|
|
||||||
|
planeVersion: v1.4.1
|
||||||
|
|
||||||
|
ingress:
|
||||||
|
enabled: true
|
||||||
|
appHost: plane.chans.xyz
|
||||||
|
ingressClass: traefik
|
||||||
|
traefik:
|
||||||
|
# 20 MiB (chart default). Keep aligned with env.doc_upload_size_limit below.
|
||||||
|
maxRequestBodyBytes: 20971520
|
||||||
|
|
||||||
|
ssl:
|
||||||
|
createIssuer: true
|
||||||
|
issuer: http # HTTP-01; ssl_token_existingSecret not needed
|
||||||
|
email: admin@chans.xyz
|
||||||
|
generateCerts: true
|
||||||
|
|
||||||
|
postgres:
|
||||||
|
storageClass: local-path
|
||||||
|
volumeSize: 5Gi
|
||||||
|
# NOTE: chart 1.8.0 exposes NO resources knob for the bundled datastores
|
||||||
|
# (stateful templates render no resources block). Add limits via
|
||||||
|
# --post-renderer/kustomize or `kubectl -n plane patch sts ...` re-applied on
|
||||||
|
# every upgrade (P2 task; see README.md).
|
||||||
|
|
||||||
|
redis:
|
||||||
|
storageClass: local-path
|
||||||
|
# image: valkey/valkey:7.2.11-alpine # already pinned by chart default; uncomment to make explicit
|
||||||
|
|
||||||
|
minio:
|
||||||
|
# P2: pin. Digest of the currently running :latest (2026-09-03, pod plane-app-minio-wl-0).
|
||||||
|
image: minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e
|
||||||
|
# image_mc: minio/mc@sha256:... # optional: pin one-shot bucket-init client the same way
|
||||||
|
storageClass: local-path
|
||||||
|
volumeSize: 5Gi
|
||||||
|
|
||||||
|
rabbitmq:
|
||||||
|
storageClass: local-path
|
||||||
|
|
||||||
|
env:
|
||||||
|
# Fail the render instead of ever falling back to the chart's PUBLIC constants
|
||||||
|
# (values.yaml L340-341 in chart 1.8.0). Requires external_secrets below.
|
||||||
|
requireExplicitSecrets: true
|
||||||
|
|
||||||
|
# SECRET_KEY / LIVE_SERVER_SECRET_KEY are deliberately OMITTED here.
|
||||||
|
# They live in k8s Secret `plane-app-keys` (referenced below). With
|
||||||
|
# requireExplicitSecrets=true and app_keys_existingSecret set, the chart renders
|
||||||
|
# neither key itself and app+live workloads both envFrom `plane-app-keys` LAST
|
||||||
|
# (later envFrom wins), which keeps the shared signing key consistent.
|
||||||
|
|
||||||
|
pgdb_name: plane
|
||||||
|
docstore_bucket: uploads
|
||||||
|
# Align app-side upload cap with the Traefik body limit (was 5242880/5MiB).
|
||||||
|
# Keep both at 20MiB, or lower both together.
|
||||||
|
doc_upload_size_limit: "20971520"
|
||||||
|
|
||||||
|
external_secrets:
|
||||||
|
# Shared signing keys (used by app + live). REQUIRED keys: SECRET_KEY, LIVE_SERVER_SECRET_KEY.
|
||||||
|
app_keys_existingSecret: plane-app-keys
|
||||||
|
# REQUIRED keys: REDIS_URL, DATABASE_URL, AMQP_URL (chart-derived URLs; update on DB/queue rotation).
|
||||||
|
app_env_existingSecret: plane-app-env
|
||||||
|
# REQUIRED keys: REDIS_URL.
|
||||||
|
live_env_existingSecret: plane-live-env
|
||||||
|
# REQUIRED keys: POSTGRES_PASSWORD, POSTGRES_DB, POSTGRES_USER.
|
||||||
|
pgdb_existingSecret: plane-pgdb-credentials
|
||||||
|
# REQUIRED keys: RABBITMQ_DEFAULT_USER, RABBITMQ_DEFAULT_PASS.
|
||||||
|
rabbitmq_existingSecret: plane-rabbitmq-credentials
|
||||||
|
# REQUIRED keys: FILE_SIZE_LIMIT, AWS_S3_BUCKET_NAME, USE_MINIO, MINIO_ROOT_USER,
|
||||||
|
# MINIO_ROOT_PASSWORD, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_S3_ENDPOINT_URL.
|
||||||
|
doc_store_existingSecret: plane-minio-credentials
|
||||||
|
# ssl_token_existingSecret: '' # DNS-01 only (cloudflare/digitalocean); unused with HTTP-01
|
||||||
|
|
||||||
|
# Optional, P2: HTTP -> HTTPS 301. The chart's own IngressRoute binds only
|
||||||
|
# 'websecure' (http:// currently 404s). extraObjects is rendered verbatim (toYaml).
|
||||||
|
# Uncomment and `helm upgrade` once reviewed:
|
||||||
|
# extraObjects:
|
||||||
|
# - apiVersion: traefik.io/v1alpha1
|
||||||
|
# kind: Middleware
|
||||||
|
# metadata:
|
||||||
|
# name: plane-https-redirect
|
||||||
|
# namespace: plane
|
||||||
|
# spec:
|
||||||
|
# redirectScheme:
|
||||||
|
# scheme: https
|
||||||
|
# permanent: true
|
||||||
|
# - apiVersion: traefik.io/v1alpha1
|
||||||
|
# kind: IngressRoute
|
||||||
|
# metadata:
|
||||||
|
# name: plane-http-to-https
|
||||||
|
# namespace: plane
|
||||||
|
# spec:
|
||||||
|
# entryPoints: [web]
|
||||||
|
# routes:
|
||||||
|
# - match: Host(`plane.chans.xyz`)
|
||||||
|
# kind: Rule
|
||||||
|
# middlewares:
|
||||||
|
# - name: plane-https-redirect
|
||||||
|
# services:
|
||||||
|
# - name: plane-app-web
|
||||||
|
# port: 3000
|
||||||
@@ -24,6 +24,7 @@ ssh -4 windy@synapse.chans.xyz
|
|||||||
| DB | ESS embedded PostgreSQL 17 (PVC 20Gi, local-path) |
|
| DB | ESS embedded PostgreSQL 17 (PVC 20Gi, local-path) |
|
||||||
| Cache | ESS embedded Redis (PVC 2Gi) |
|
| Cache | ESS embedded Redis (PVC 2Gi) |
|
||||||
| Chart | `oci://ghcr.io/element-hq/ess-helm/matrix-stack`, version `26.7.2` |
|
| Chart | `oci://ghcr.io/element-hq/ess-helm/matrix-stack`, version `26.7.2` |
|
||||||
|
| Plane | Helm `plane-ce-1.8.0` (app `v1.4.1`), namespace `plane` — self-hosted Plane project management |
|
||||||
|
|
||||||
### Matrix service endpoints
|
### Matrix service endpoints
|
||||||
|
|
||||||
@@ -51,8 +52,50 @@ All other ports internal only (no K3s API, no database, no Redis exposed).
|
|||||||
|
|
||||||
- `ess` — all ESS workloads (Synapse, MAS, Element, Postgres, Redis, HAProxy)
|
- `ess` — all ESS workloads (Synapse, MAS, Element, Postgres, Redis, HAProxy)
|
||||||
- `matrix-system` — cluster base resources (ResourceQuota, LimitRange, mrtc-placeholder)
|
- `matrix-system` — cluster base resources (ResourceQuota, LimitRange, mrtc-placeholder)
|
||||||
|
- `plane` — Plane project management (Helm release `plane-app`)
|
||||||
- `cert-manager` — cert-manager
|
- `cert-manager` — cert-manager
|
||||||
|
|
||||||
|
## Plane (project management)
|
||||||
|
|
||||||
|
Self-hosted [Plane](https://github.com/makeplane/plane) on the same K3s node, deployed via the official `plane-ce` Helm chart.
|
||||||
|
|
||||||
|
| Item | Detail |
|
||||||
|
|------|--------|
|
||||||
|
| Release | `plane-app` (ns `plane`), chart `plane-ce-1.8.0`, app `v1.4.1`, revision 1 |
|
||||||
|
| URL | https://plane.chans.xyz |
|
||||||
|
| Install date | 2026-09-01 |
|
||||||
|
| Values source | `/home/windy/plane-k3s/values.yaml` (plain file, not a git repo) |
|
||||||
|
| Images | `artifacts.plane.so/makeplane/*` (`plane-frontend`, `plane-backend`, `plane-admin`, `plane-live`), pullPolicy `Always` |
|
||||||
|
| Ingress | Traefik `IngressRoute` `plane-app-ingress` — `/`→web, `/api` `/auth`→api, `/spaces`→space, `/god-mode`→admin, `/live`→live, `/uploads`→minio; `maxRequestBodyBytes` 20Mi |
|
||||||
|
| TLS | Own namespace `Issuer` `plane-app-cert-issuer` (HTTP-01, LE prod, `admin@chans.xyz`); cert `plane-app-ssl-cert` (CN `plane.chans.xyz`) |
|
||||||
|
| DB | Bundled Postgres `15.7-alpine` (PVC 5Gi, local-path) |
|
||||||
|
| Cache/queue | Bundled Redis (PVC 100Mi), RabbitMQ `3.13.6-management-alpine` (PVC 100Mi) |
|
||||||
|
| Storage | Bundled MinIO (`minio/minio:latest`, root user `admin`, PVC 5Gi) — S3 for uploads/docs |
|
||||||
|
| Resources | Every workload: cpu 50m/500m, mem 50Mi/1000Mi, replicas 1 |
|
||||||
|
| SMTP | Not configured (no `smtp` values) — Plane invites/password resets won't email yet |
|
||||||
|
|
||||||
|
Workloads (all 1/1 Running): 7 Deployments (`plane-app-{admin,api,beat-worker,live,space,web,worker}-wl`) + 4 StatefulSets (`plane-app-{minio,pgdb,rabbitmq,redis}-wl`); init Jobs `api-migrate-1` / `minio-bucket-1` Completed. All PVCs Bound on `local-path` (root disk).
|
||||||
|
|
||||||
|
### Plane configuration notes
|
||||||
|
|
||||||
|
- **`planeVersion: v1.4.1`** pinned in values.yaml; chart tracks Plane's own tags.
|
||||||
|
- **Secrets**: Helm-generated Opaque secrets (`plane-app-app-secrets`, `-doc-store-secrets`, `-pgdb-secrets`, `-rabbitmq-secrets`, `-live-secrets`); `requireExplicitSecrets: false`. Values live in `$SECRET_KEY`, `DATABASE_URL`, `AMQP_URL`, `REDIS_URL` etc.
|
||||||
|
- **Sentry / CORS**: `sentry_dsn` and `cors_allowed_origins` empty (defaults fine for single-host).
|
||||||
|
- **MinIO is `latest` tag** — pin a version for reproducibility.
|
||||||
|
- **Backup**: NOT covered by `/var/backups/matrix` (which is paused anyway) — Plane Postgres/MinIO PVCs have no backup tier yet.
|
||||||
|
|
||||||
|
### Plane verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Release + workloads
|
||||||
|
sudo helm list -A
|
||||||
|
sudo k3s kubectl -n plane get deploy,sts,pods -o wide
|
||||||
|
# Cert + ingress
|
||||||
|
sudo k3s kubectl -n plane get certificate,ingressroute
|
||||||
|
# Endpoint
|
||||||
|
curl -4 -s -o /dev/null -w '%{http_code}\n' https://plane.chans.xyz/
|
||||||
|
```
|
||||||
|
|
||||||
## Local backup
|
## Local backup
|
||||||
|
|
||||||
| Item | Detail |
|
| Item | Detail |
|
||||||
@@ -62,7 +105,7 @@ All other ports internal only (no K3s API, no database, no Redis exposed).
|
|||||||
| Retention | 7 days |
|
| Retention | 7 days |
|
||||||
| Disk warning | 80% (healthcheck), 90% (backup stops) |
|
| Disk warning | 80% (healthcheck), 90% (backup stops) |
|
||||||
| Content | Planned: PostgreSQL `synapse` + `mas` logical dumps, media store archive, `/etc/matrix-bootstrap` |
|
| Content | Planned: PostgreSQL `synapse` + `mas` logical dumps, media store archive, `/etc/matrix-bootstrap` |
|
||||||
| Status | **Not operational** — no current Matrix backup or recovery tier |
|
| Status | **Not operational** — no current Matrix backup or recovery tier. **Plane data (its own Postgres + MinIO PVCs in ns `plane`) is also not covered by any backup.** |
|
||||||
|
|
||||||
## Health checks
|
## Health checks
|
||||||
|
|
||||||
@@ -101,5 +144,6 @@ diagnosis and imperative recovery work.
|
|||||||
- MatrixRTC / Element Call / LiveKit / Coturn not deployed (`mrtc.chans.xyz` reserved only)
|
- MatrixRTC / Element Call / LiveKit / Coturn not deployed (`mrtc.chans.xyz` reserved only)
|
||||||
- SMTP email not yet configured (requires manual secret bootstrap followed by a
|
- SMTP email not yet configured (requires manual secret bootstrap followed by a
|
||||||
reviewed Ansible stack deployment)
|
reviewed Ansible stack deployment)
|
||||||
|
- Plane `minio` image uses `latest` tag (pin a version)
|
||||||
- No off-site Restic backup
|
- No off-site Restic backup
|
||||||
- Single-node K3s (no HA for control plane)
|
- Single-node K3s (no HA for control plane)
|
||||||
|
|||||||
@@ -48,4 +48,5 @@ the software deployed there, see [the LAN overview](../docs/lan-overview.md).
|
|||||||
| https://synapse.chans.xyz | Synapse API | Client-Server + Federation API |
|
| https://synapse.chans.xyz | Synapse API | Client-Server + Federation API |
|
||||||
| https://account.chans.xyz | MAS | Matrix Authentication Service (local passwords) |
|
| https://account.chans.xyz | MAS | Matrix Authentication Service (local passwords) |
|
||||||
| https://admin.chans.xyz | Element Admin | Admin console (MAS admin auth) |
|
| https://admin.chans.xyz | Element Admin | Admin console (MAS admin auth) |
|
||||||
|
| https://plane.chans.xyz | Plane | Project management (Helm `plane-ce` v1.4.1, ns `plane`) |
|
||||||
| `mrtc.chans.xyz` | MatrixRTC | **Reserved** – not deployed |
|
| `mrtc.chans.xyz` | MatrixRTC | **Reserved** – not deployed |
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ runbooks start from [`_template.md`](_template.md).
|
|||||||
| PowerDNS health check | [pdns-health.md](pdns-health.md) | read-only |
|
| PowerDNS health check | [pdns-health.md](pdns-health.md) | read-only |
|
||||||
| RustDesk health check | [rustdesk-health.md](rustdesk-health.md) | read-only |
|
| RustDesk health check | [rustdesk-health.md](rustdesk-health.md) | read-only |
|
||||||
| Matrix health check | [matrix-health.md](matrix-health.md) | read-only |
|
| Matrix health check | [matrix-health.md](matrix-health.md) | read-only |
|
||||||
|
| Plane health check | [plane-health.md](plane-health.md) | read-only |
|
||||||
| pgdb health check | [pgdb-health.md](pgdb-health.md) | read-only |
|
| pgdb health check | [pgdb-health.md](pgdb-health.md) | read-only |
|
||||||
| pgdb DB restore (pg_restore) | [pgdb-restore.md](pgdb-restore.md) | change (procedure) |
|
| pgdb DB restore (pg_restore) | [pgdb-restore.md](pgdb-restore.md) | change (procedure) |
|
||||||
| pgdb image/compose update | [pgdb-update.md](pgdb-update.md) | change (gated) |
|
| pgdb image/compose update | [pgdb-update.md](pgdb-update.md) | change (gated) |
|
||||||
|
|||||||
@@ -0,0 +1,93 @@
|
|||||||
|
# Runbook: Plane Health Check
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Read-only health check of the self-hosted Plane project-management instance
|
||||||
|
(`plane.chans.xyz`) running on the synapse K3s cluster.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- Applicable: [synapse.chans.xyz](../hosts/synapse.chans.xyz.md), namespace `plane`.
|
||||||
|
- Read-only: does not change pods, ingress, certificates, secrets, or configuration.
|
||||||
|
- Not applicable: Plane upgrade, values changes, or data recovery — those need a
|
||||||
|
reviewed change (see `ansible-operations.md` / `release.md`).
|
||||||
|
|
||||||
|
## Ownership
|
||||||
|
|
||||||
|
- Owner: personal ops (Windy)
|
||||||
|
- Last reviewed: 2026-09-02
|
||||||
|
- Related systems: synapse.chans.xyz (Helm `plane-app`, chart `plane-ce-1.8.0`, app `v1.4.1`)
|
||||||
|
|
||||||
|
## Pass criteria
|
||||||
|
|
||||||
|
All of the following must hold; any conflict means `STOP` and record evidence.
|
||||||
|
|
||||||
|
- `sudo helm list -A` shows `plane-app` in ns `plane`, STATUS `deployed`.
|
||||||
|
- All 7 Deployments + 4 StatefulSets in ns `plane` are `1/1 Running` with 0 recent restarts.
|
||||||
|
- Init Jobs `api-migrate-*` / `minio-bucket-*` are `Completed`.
|
||||||
|
- Certificate `plane-app-ssl-cert` is `READY=True` (CN `plane.chans.xyz`).
|
||||||
|
- `https://plane.chans.xyz/` returns HTTP 200 with a valid Let's Encrypt cert.
|
||||||
|
- Root disk usage below the 80% warning threshold.
|
||||||
|
|
||||||
|
## Procedure
|
||||||
|
|
||||||
|
### 1. Release and workloads
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'sudo helm list -A'
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get deploy,sts,pods -o wide'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: `plane-app` deployed; all workloads `1/1 Running`, `RESTARTS` low;
|
||||||
|
no `CrashLoopBackOff` / `Evicted`. Otherwise `STOP` and record evidence.
|
||||||
|
|
||||||
|
### 2. TLS certificate and ingress
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get certificate,issuer,ingressroute'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: `plane-app-ssl-cert` `READY=True`; `plane-app-ingress` present with
|
||||||
|
routes for `/`, `/api`, `/spaces`, `/god-mode`, `/live`, `/uploads`.
|
||||||
|
|
||||||
|
If `READY=False` with a pending HTTP-01 challenge, the usual cause is the node
|
||||||
|
DNS chain (coreDNS → systemd-resolved → uplink) failing to resolve
|
||||||
|
`plane.chans.xyz` — check `resolvectl query plane.chans.xyz` vs
|
||||||
|
`dig +short plane.chans.xyz @8.8.8.8`. If the record exists publicly but the
|
||||||
|
node fails, `sudo resolvectl flush-caches` and wait for the cert-manager retry;
|
||||||
|
do not mutate the issuer.
|
||||||
|
|
||||||
|
### 3. Endpoint verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -4 -s -o /dev/null -w '%{http_code}\n' https://plane.chans.xyz/
|
||||||
|
echo | openssl s_client -connect plane.chans.xyz:443 -servername plane.chans.xyz 2>/dev/null | openssl x509 -noout -subject -issuer -dates
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: HTTP 200, cert `CN=plane.chans.xyz` issued by Let's Encrypt with a
|
||||||
|
future `notAfter`. `https://plane.chans.xyz/api/` may 404 — the API serves
|
||||||
|
under `/api/...` paths only, so a bare 404 there is not a failure.
|
||||||
|
|
||||||
|
### 4. Storage and disk
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n plane get pvc'
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'df -hP /'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: all 4 PVCs `Bound` (minio 5Gi, pgdb 5Gi, rabbitmq 100Mi, redis 100Mi
|
||||||
|
on `local-path`); root disk < 80%.
|
||||||
|
|
||||||
|
## Safety
|
||||||
|
|
||||||
|
- Read-only: never mutate pods, ingress, certificates, or configuration during
|
||||||
|
this check.
|
||||||
|
- Plane data (ns `plane` Postgres + MinIO PVCs) has **no backup tier**; treat
|
||||||
|
the instance as at-risk until a backup design exists.
|
||||||
|
- If live state conflicts with the expected values above, `STOP` and record
|
||||||
|
evidence; do not "fix in passing".
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [hosts/synapse.chans.xyz.md](../hosts/synapse.chans.xyz.md) — plane stack facts
|
||||||
|
- [matrix-health.md](matrix-health.md) — sibling service on the same cluster
|
||||||
Reference in New Issue
Block a user