windy/vps
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Initial VPS operations handbook

Browse Source
This commit is contained in:
windyboy
2026-08-03 12:26:42 +08:00
commit b73125e5bc
97 changed files with 3641 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/matrix-upstream.md
+109
View File
@@ -0,0 +1,109 @@
# Matrix (Element Server Suite) Upstream References
Matrix homeserver deployed via **Element Server Suite (ESS) Community** OCI chart.
## Chart
| Property | Value |
|---|---|
| Chart name | `matrix-stack` |
| OCI registry | `oci://ghcr.io/element-hq/ess-helm/matrix-stack` |
| Version deployed | `26.7.2` |
| Chart source | <https://github.com/element-hq/ess-helm> |
| Documentation | <https://element-hq.github.io/ess-helm/> |
## Components
| Component | Docker image | Role |
|---|---|---|
| Synapse | `ghcr.io/element-hq/synapse:*` | Matrix homeserver |
| MAS | `ghcr.io/element-hq/matrix-authentication-service:*` | OIDC/OAuth authentication |
| Element Web | `ghcr.io/element-hq/element-web:*` | Web client |
| Element Admin | `ghcr.io/element-hq/element-admin:*` | Admin console |
| HAProxy | `haproxy:*` | Internal routing/health |
| PostgreSQL | `postgres:17` | Synapse + MAS database |
All images should be locked via digest in production; the chart's `appVersion` may not match default image tags.
## Domain configuration
| Hostname | Component |
|---|---|
| `chans.xyz` | Matrix identity (`serverName`) + well-known discovery |
| `synapse.chans.xyz` | Client-Server API, Federation API |
| `chat.chans.xyz` | Element Web |
| `account.chans.xyz` | Matrix Authentication Service (local passwords) |
| `admin.chans.xyz` | Element Admin (MAS admin auth) |
| `mrtc.chans.xyz` | Reserved for MatrixRTC — **not deployed** |
## TLS & Ingress
- Ingress controller: K3s built-in Traefik (IngressClass `traefik`)
- Certificate issuer: cert-manager + Let's Encrypt HTTP-01 (`letsencrypt-prod` ClusterIssuer)
- All TLS enabled: `global.ingress.tls.enabled=true`
- Issuer annotation: `cert-manager.io/cluster-issuer: letsencrypt-prod`
## Architecture
```
Internet :80/:443
→ K3s Traefik (hostPort)
→ ESS Ingresses (Traefik class)
→ ClusterIP Services
→ Pods (Synapse, MAS, Element, Admin, HAProxy)
→ PostgreSQL (StatefulSet, 20Gi PVC)
```
## Deployment
The ESS chart is deployed via Ansible (`ansible/playbooks/matrix-stack.yml`) with non-secret values at `/etc/ess/*.yaml` on the target host.
**Secrets** (signing key, database passwords, SMTP password, MAS keys) are **not** in Git. They are stored:
1. On the VPS at `/etc/matrix-bootstrap/` (root:root 0600 files)
2. In pre-created Kubernetes Secrets in the `ess` namespace
## Upgrades
```bash
# Pull latest chart version
helm pull oci://ghcr.io/element-hq/ess-helm/matrix-stack --version <new-version>
# Review changes
helm diff upgrade ess oci://ghcr.io/element-hq/ess-helm/matrix-stack \
-n ess -f /etc/ess/hostnames.yaml -f /etc/ess/tls.yaml
# Apply
helm upgrade --install ess oci://ghcr.io/element-hq/ess-helm/matrix-stack \
-n ess -f /etc/ess/hostnames.yaml -f /etc/ess/tls.yaml --wait
```
Before upgrading:
- Matrix backup automation is currently paused; establish and test a reviewed backup design before treating the deployment as recoverable.
- Review chart changelog and database migration notes
- Test in isolation if possible
## Recovery
When the planned backup automation is enabled, recovery will require the following from `/var/backups/matrix/`:
1. PostgreSQL dumps (synapse + mas)
2. Media store archive
3. Bootstrap secrets archive (`/etc/matrix-bootstrap/`)
4. Non-secret values files (`/etc/ess/*.yaml`)
5. Non-secret playbook templates (from this repo)
Steps:
1. Deploy K3s + Traefik + cert-manager per Ansible
2. Restore bootstrap secrets + create Kubernetes Secrets
3. Restore PostgreSQL dumps
4. Restore media store to PVC
5. Deploy ESS chart with same values
## References
- ESS Helm docs: <https://element-hq.github.io/ess-helm/>
- ESS Community setup: <https://github.com/element-hq/element-server-suite>
- Synapse: <https://github.com/element-hq/synapse>
- Matrix Authentication Service: <https://github.com/element-hq/matrix-authentication-service>
- Matrix spec: <https://spec.matrix.org/>
- Federation tester: <https://federationtester.matrix.org/>
docs/pdns-upstream.md
+81
View File
@@ -0,0 +1,81 @@
# PowerDNS Authoritative — upstream documentation
Use these when changing `/opt/pdns` on [hk2](../hosts/hk2.chans.xyz.md). Prefer official docs over blog posts.
## Primary references
| Topic | URL |
|-------|-----|
| Settings reference | https://doc.powerdns.com/authoritative/settings.html |
| Primary / secondary modes | https://doc.powerdns.com/authoritative/modes-of-operation.html |
| Generic PostgreSQL backend | https://doc.powerdns.com/authoritative/backends/generic-postgresql.html |
| DNSSEC ops + SOA-EDIT | https://doc.powerdns.com/authoritative/dnssec/operational.html |
| DNSSEC advice | https://doc.powerdns.com/authoritative/dnssec/advice.html |
| TSIG | https://doc.powerdns.com/authoritative/tsig.html |
| HTTP API | https://doc.powerdns.com/authoritative/http-api/ |
| Security advisories | https://doc.powerdns.com/authoritative/security-advisories/ |
| Docker image (`pdns-auth-50`) | https://hub.docker.com/r/powerdns/pdns-auth-50 |
## Settings we care about on hk2 (ns1)
| Concern | Upstream guidance | Our notes (post-2026-08-01) |
|---------|-------------------|-----------------------------|
| Version | Track advisories; upgrade when Mandatory | Pinned **`powerdns/pdns-auth-50:5.0.6`** (clears 2026-06 + 2026-07) |
| Primary | `primary=yes`; zones type MASTER/PRIMARY | Set; zones `windy.me` / `wsvc.info` / `chans.xyz` are Master |
| NOTIFY | `also-notify` always notified; empty `only-notify` disables NS-based NOTIFY | `also-notify=202.91.35.141`, `only-notify=` |
| AXFR IP ACL | `allow-axfr-ips` limits **unsigned** AXFR | Global + per-zone `ALLOW-AXFR-FROM` → ns2 only |
| TSIG AXFR | **Any host with the TSIG key may AXFR**, bypassing IP ACL | Zones use `TSIG-ALLOW-AXFR=mykey.` — rotate/rename key; protect secret |
| DNSSEC | `gpgsql-dnssec=yes` + keys via `pdnsutil` | CSK ECDSAP256SHA256 on all three zones |
| SOA-EDIT | Needed so non-PDNS secondaries see signature rolls; PDNS secondaries can also compare RRSIGs | `default-soa-edit(-signed)=INCEPTION-INCREMENT` OK (ns2 is PowerDNS). Dig SOA ≠ backend serial is expected |
| API key | Prefer `pdnsutil hash-password` (plaintext still accepted) | **Deferred hardening:** live key is weak, plaintext, reused with DB password — rotate |
| Webserver | Prefer trusted clients only | Host publishes `127.0.0.1:8081`; `webserver-allow-from` private ranges |
| `gpgsql-host` | Prefer IP to avoid resolve chicken/egg | `db` (Compose DNS) is fine inside the stack |
| `domains.master` | Used for SECONDARY upstreams | Cleared on MASTER zones (was self-IP noise) |
## Intended `auth/pdns.conf` shape (no secrets)
Live file: `/opt/pdns/auth/pdns.conf`. Secrets come from `auth/templates.d/secrets.j2` → `/etc/powerdns/pdns.d/secrets.conf` at container start.
```
local-address=0.0.0.0
local-port=53
launch=gpgsql
gpgsql-host=db
gpgsql-dnssec=yes
include-dir=/etc/powerdns/pdns.d
allow-axfr-ips=202.91.35.141
also-notify=202.91.35.141
only-notify=
primary=yes
secondary=no
api=yes
webserver=yes
webserver-address=0.0.0.0
webserver-port=8081
webserver-allow-from=127.0.0.1,172.16.0.0/12,10.0.0.0/8,192.168.0.0/16
version-string=anonymous
disable-syslog=yes
loglevel=4
default-soa-edit=INCEPTION-INCREMENT
default-soa-edit-signed=INCEPTION-INCREMENT
disable-axfr=no
```
Drop-in (from `.env`, never commit): `gpgsql-dbname`, `gpgsql-user`, `gpgsql-password`, `api-key`.
## Deferred hardening (not done yet)
1. Rotate `PDNS_API_KEY` to a long random value; store hashed via `pdnsutil hash-password` if desired; update Poweradmin `.env` / recreate `auth` + `poweradmin`.
2. Rotate `DB_PASS` / Postgres role separately from the API key.
3. Rotate TSIG `mykey` (or rename); confirm ns2 still AXFRs; remember TSIG bypasses `allow-axfr-ips`.
## Related ops docs
- Host facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md)
- Health runbook: [runbooks/pdns-health.md](../runbooks/pdns-health.md)
docs/vaultwarden-upstream.md
+53
View File
@@ -0,0 +1,53 @@
# Vaultwarden — upstream documentation (source of truth)
Use these when changing `/opt/vaultwarden` on [us2](../hosts/us2.wsvc.info.md). Prefer upstream over blog posts or Obsidian notes.
## Primary references
| Topic | URL |
|-------|-----|
| Configuration overview (methods + precedence) | https://github.com/dani-garcia/vaultwarden/wiki/Configuration-overview |
| Env / options template (canonical variable list) | https://github.com/dani-garcia/vaultwarden/blob/main/.env.template |
| Defaults / implementation source | https://github.com/dani-garcia/vaultwarden/blob/main/src/config.rs |
| PostgreSQL backend + SQLite migration | https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend |
| SMTP | https://github.com/dani-garcia/vaultwarden/wiki/SMTP-configuration |
| Reverse proxy examples (incl. Traefik) | https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples |
| Enabling admin page / `ADMIN_TOKEN` | https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page |
| Disable registration / invitations | https://github.com/dani-garcia/vaultwarden/wiki/Disable-registration-of-new-users |
| pgloader BIGINT login failure note | https://github.com/dani-garcia/vaultwarden/discussions/7134 |
| Project wiki home | https://github.com/dani-garcia/vaultwarden/wiki |
| Container image | https://github.com/dani-garcia/vaultwarden/pkgs/container/server |
## Configuration precedence (official)
From [Configuration overview](https://github.com/dani-garcia/vaultwarden/wiki/Configuration-overview):
1. Compile-time defaults (`src/config.rs`)
2. Vaultwarden `ENV_FILE` / `.env` read by the process (if used)
3. Process environment variables (e.g. Docker Compose `env_file` + `environment`) — override (2)
4. **`data/config.json` from `/admin` — highest precedence** (overrides env)
Official stance: prefer environment variables; `config.json` is **not** the recommended long-term method. Once admin saves `config.json`, those keys stop being changeable via `.env` until removed/edited in admin or the file.
## Settings we care about on us2
| Concern | Upstream guidance | Our notes (post-2026-08-01 migrate) |
|---------|-------------------|-------------------------------------|
| Public URL | Set `DOMAIN` e.g. `https://auth.wsvc.info` (no trailing slash) | Set in compose + `config.json` |
| SMTP | `587`+`starttls` or `465`+`force_tls` | Effective: **587/starttls** to mx2 (`465` unreachable from us2); `extra_hosts` pins A `194.163.160.244` |
| SMTP password | Same mailbox password as mailcow | Must match across `.env`, `config.json`, `.smtp-credentials`. **Admin/`config.json` wins** — stale admin password → SMTP `535` while `.env` still works |
| Database | `DATABASE_URL` for Postgres | Live: Postgres via compose `DATABASE_URL`; sqlite cold under `backups/sqlite-cold/` |
| Client IP behind proxy | Traefik → prefer `X-Forwarded-For` | `ip_header=X-Forwarded-For` in `config.json` |
| Admin | Argon2 `ADMIN_TOKEN` | On server only |
| Signups | Invite-only on public URL | `signups_allowed=false` |
## Agent checklist before edits
1. Read this page + [us2 host facts](../hosts/us2.wsvc.info.md).
2. On the server, compare `.env`, compose `environment`, and `/data/config.json` (including SMTP password fingerprint, not plaintext in chat/logs).
3. Change the layer that actually wins (often `config.json`); if rotating the mailbox password, update **all three** secret stores.
4. After recreating the `vaultwarden` container, restart Traefik if `auth.wsvc.info` 404s.
5. Do not commit secrets; do not copy passwords from Obsidian into this repo.
6. Verify SMTP with AUTH (not TCP-only) — [vaultwarden-health](../runbooks/vaultwarden-health.md) §5.
Replay migration steps: [runbooks/vaultwarden-sqlite-to-postgres.md](../runbooks/vaultwarden-sqlite-to-postgres.md).