commit b73125e5bcd5b6c0b6344bc7a3bb0be7ef69fccc
Author: windyboy <windyboy@gmail.com>
Date:   Mon Aug 3 12:26:42 2026 +0800

    Initial VPS operations handbook

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..46610e4
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,22 @@
+# Local secrets and service configuration (never version these).
+.env
+.env.*
+!.env.example
+mailcow.conf
+.smtp-credentials
+.admin-token
+*.pem
+*.key
+id_*
+!*.pub
+
+# Ansible runtime output.
+*.retry
+.ansible/
+facts/
+
+# Editor and operating-system files.
+.DS_Store
+.vscode/
+.idea/
+*~
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000..8e9f448
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,104 @@
+# VPS Agent Ops Hub
+
+This repo is the **agent ops handbook + fact source** for maintaining personal VPS hosts. Prefer verifying live state over assuming docs are complete.
+
+Also readable as `agent.md` (symlink → this file).
+
+## How to work
+
+1. Read [`inventory/hosts.md`](inventory/hosts.md) for the machine list.
+2. Open the matching [`hosts/<name>.md`](hosts/) for SSH, roles, paths, and quirks.
+3. For common tasks, follow a runbook under [`runbooks/`](runbooks/).
+4. Prefer read-only checks first; change only after confirming current state.
+5. Default access: `ssh -4 windy@<host>` (pubkey). Prefer **IPv4** from this WSL client (AAAA often exists but IPv6 route does not).
+
+## Active hosts (quick map)
+
+| Host | Role | SSH | Facts |
+|------|------|-----|--------|
+| **mx2.windy.me** | mailcow (`/opt/mail`, project `cow`) | `ssh -4 windy@mx2.windy.me` | [hosts/mx2.windy.me.md](hosts/mx2.windy.me.md) |
+| **us2.wsvc.info** | Vaultwarden + Traefik (+ Soft Serve, …) | `ssh -4 windy@us2.wsvc.info` | [hosts/us2.wsvc.info.md](hosts/us2.wsvc.info.md) |
+| **hk2.chans.xyz** | PowerDNS auth ns1 (`/opt/pdns`) | `ssh -4 windy@hk2.chans.xyz` | [hosts/hk2.chans.xyz.md](hosts/hk2.chans.xyz.md) |
+
+### Public services
+
+| URL / endpoint | Backend host | Notes |
+|----------------|--------------|--------|
+| https://mx2.windy.me | mx2 | mailcow UI / SOGo |
+| SMTP `mx2.windy.me:587` (STARTTLS) or `:465` | mx2 | client submission; full email + mailbox password — [runbook](runbooks/mailcow-smtp-client.md) |
+| IMAP `mx2.windy.me:993` | mx2 | same mailbox credentials |
+| https://auth.wsvc.info | us2 (`/opt/vaultwarden`) | Vaultwarden (Postgres, **operational**) — client Server URL |
+| `repo.windy.me:2222` | us2 (`/opt/soft-serve`) | Soft Serve (stub details) |
+| DNS `ns1.wsvc.info:53` | hk2 (`/opt/pdns`, Auth **5.0.6**) | PowerDNS auth — zones `windy.me`, `wsvc.info`, `chans.xyz` |
+| https://pdns.wsvc.info | hk2 (`poweradmin`) | Poweradmin UI |
+| https://pgweb.wsvc.info | hk2 (`pgweb`) | PowerDNS Postgres browser |
+| https://chat.chans.xyz | synapse (`ess`, ESS chart `26.7.2`) | Element Web (Matrix client) |
+| https://synapse.chans.xyz | synapse | Synapse Client-Server + Federation API |
+| https://account.chans.xyz | synapse | Matrix Authentication Service (local passwords) |
+| https://admin.chans.xyz | synapse | Element Admin console (MAS admin auth) |
+
+### Upstream docs
+
+**Vaultwarden:** Before changing config, read [docs/vaultwarden-upstream.md](docs/vaultwarden-upstream.md). Live DB is **Postgres** (`DATABASE_URL`); `vw-data/config.json` from `/admin` overrides `.env` (including SMTP password — keep it synced with `.env` / `.smtp-credentials`). After recreating the VW container, restart Traefik if `https://auth.wsvc.info` returns 404. SMTP health needs **AUTH**, not TCP-only — see [vaultwarden-health](runbooks/vaultwarden-health.md).
+
+**PowerDNS:** Before changing `/opt/pdns` on hk2, read [docs/pdns-upstream.md](docs/pdns-upstream.md). Auth pin **5.0.6**; primary + `also-notify`/`only-notify=` + DNSSEC OK. Deferred: rotate weak API/DB credentials and TSIG (`TSIG-ALLOW-AXFR` bypasses IP ACL). Health: [pdns-health](runbooks/pdns-health.md).
+
+**Matrix (ESS on synapse):** Matrix homeserver running on `synapse.chans.xyz` via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone `chans.xyz` managed by hk2 PowerDNS. Before changing config, read [docs/matrix-upstream.md](docs/matrix-upstream.md) and [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: [matrix-health](runbooks/matrix-health.md).
+
+## Runbooks & scripts
+
+| Task | Path |
+|------|------|
+| mailcow health | [runbooks/mailcow-health.md](runbooks/mailcow-health.md) · `./scripts/check-mx2.sh` |
+| mailcow update | [runbooks/mailcow-update.md](runbooks/mailcow-update.md) |
+| mailcow SMTP/IMAP client | [runbooks/mailcow-smtp-client.md](runbooks/mailcow-smtp-client.md) |
+| Vaultwarden health | [runbooks/vaultwarden-health.md](runbooks/vaultwarden-health.md) |
+| Vaultwarden SQLite→PG migrate | [runbooks/vaultwarden-sqlite-to-postgres.md](runbooks/vaultwarden-sqlite-to-postgres.md) |
+| Vaultwarden upstream refs | [docs/vaultwarden-upstream.md](docs/vaultwarden-upstream.md) |
+| PowerDNS health (hk2) | [runbooks/pdns-health.md](runbooks/pdns-health.md) |
+| PowerDNS upstream refs | [docs/pdns-upstream.md](docs/pdns-upstream.md) |
+| Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) |
+| Matrix upstream refs | [docs/matrix-upstream.md](docs/matrix-upstream.md) |
+
+```bash
+./scripts/check-mx2.sh
+```
+
+`check-mx2.sh` defaults to IPv4; set `MX2_ALLOW_IPV6=1` to allow IPv6. DNS checks use `1.1.1.1` / `8.8.8.8` (local stub is flaky).
+
+## Safety
+
+- Never commit secrets: passwords, API keys, private keys, `.env`, `mailcow.conf` DB passwords, Vaultwarden `ADMIN_TOKEN` / `.smtp-credentials`.
+- Secrets live **only on the servers** (e.g. `/opt/vaultwarden/.env`). Do not copy them from Obsidian into this repo.
+- Ask before destructive actions: `docker compose down -v`, DB drops, DNS changes, force-push, disk wipes.
+- Do not modify local `~/.ssh/config` or git config unless explicitly asked.
+
+## SSH access safety — mandatory change procedure
+
+**Treat any SSH authentication or authorization change as a lockout-risk change.** This includes `sshd_config` / drop-in edits, `PasswordAuthentication`, `PermitRootLogin`, `AuthorizedKeysFile`, `Match` blocks, SSH keys, accounts, groups, chroots, firewalls, and SSH ports.
+
+Before disabling an existing access method or reloading `sshd`, all of the following are mandatory:
+
+1. Keep the current verified SSH management session open as the rollback path; do not use it to run the validation connection.
+2. Install the intended management public key from a **verified local source**. Never assume a public-key file exists on the target host or copy a remote path without checking it first.
+3. Validate the candidate configuration with `sshd -t` before reloading the service.
+4. From a separate local terminal, prove the new key-only login works using the exact target account, hostname, and key (`BatchMode=yes`, password authentication disabled), and verify needed sudo access.
+5. Only after step 4 succeeds may password login, root login, the old key, or another existing access route be disabled.
+6. Keep the original management session open until the final effective SSH configuration and a fresh key-only login have both been verified.
+
+If the new key-based login cannot be proven first, stop immediately: do **not** reload an access-restricting configuration. Provider web/serial console access must be confirmed before any SSH access-policy change, but it is a recovery path—not a substitute for the validation steps above.
+
+## Obsidian
+
+Bills, rough notes, and personal clutter stay in the Obsidian vault. This repo holds **actionable facts and runbooks** only.
+
+## Layout
+
+```
+AGENTS.md / agent.md   # this entry (agent.md → AGENTS.md)
+inventory/hosts.md     # machine index
+hosts/                 # per-host facts
+runbooks/              # step-by-step ops
+docs/                  # upstream doc indexes / design notes
+scripts/               # read-only helper scripts
+```
diff --git a/ansible/ansible.cfg b/ansible/ansible.cfg
new file mode 100644
index 0000000..7487d63
--- /dev/null
+++ b/ansible/ansible.cfg
@@ -0,0 +1,13 @@
+[defaults]
+inventory = inventory/hosts.yml
+roles_path = roles
+interpreter_python = auto_silent
+stdout_callback = default
+bin_ansible_callbacks = True
+retry_files_enabled = False
+host_key_checking = True
+
+[privilege_escalation]
+become = True
+become_method = sudo
+become_ask_pass = False
diff --git a/ansible/inventory/hosts.yml b/ansible/inventory/hosts.yml
new file mode 100644
index 0000000..82f06ef
--- /dev/null
+++ b/ansible/inventory/hosts.yml
@@ -0,0 +1,74 @@
+---
+# Sanitized control-plane inventory. Canonical human-readable facts remain
+# in ../../inventory/hosts.md and ../../hosts/*.md. No passwords, keys, tokens,
+# repository URLs, or private material belong here.
+all:
+  vars:
+    ansible_user: windy
+    ansible_ssh_common_args: >-
+      -o BatchMode=yes -o ConnectTimeout=10 -o AddressFamily=inet
+    ansible_become: true
+    ansible_become_method: sudo
+  children:
+    managed:
+      hosts:
+        mx2:
+          ansible_host: mx2.windy.me
+          ansible_host_ipv4: 194.163.160.244
+          service_role: mailcow
+          compose_project_dir: /opt/mail
+          healthcheck_profile: mailcow
+        us2:
+          ansible_host: us2.wsvc.info
+          ansible_host_ipv4: 193.9.44.165
+          service_role: vaultwarden
+          compose_project_dir: /opt/vaultwarden
+          healthcheck_profile: vaultwarden
+        hk2:
+          ansible_host: hk2.chans.xyz
+          ansible_host_ipv4: 154.36.174.161
+          service_role: powerdns
+          compose_project_dir: /opt/pdns
+          healthcheck_profile: pdns
+    mailcow:
+      hosts:
+        mx2:
+    vaultwarden:
+      hosts:
+        us2:
+    powerdns:
+      hosts:
+        hk2:
+    docker_hosts:
+      children:
+        mailcow:
+        vaultwarden:
+        powerdns:
+    # Matrix is a dedicated K3s node and intentionally remains outside the
+    # Docker-oriented managed group.
+    k3s_servers:
+      hosts:
+        matrix_vps:
+          ansible_host: 169.58.86.13
+          ansible_host_ipv4: 169.58.86.13
+          service_role: matrix_k3s
+          matrix_server_name: chans.xyz
+          matrix_synapse_host: synapse.chans.xyz
+          matrix_element_host: chat.chans.xyz
+          matrix_mas_host: account.chans.xyz
+          matrix_admin_host: admin.chans.xyz
+          matrix_rtc_host: mrtc.chans.xyz
+          matrix_backup_path: /var/backups/matrix
+          matrix_bootstrap_dir: /etc/matrix-bootstrap
+          matrix_stack_enabled: false
+          # ESS OCI chart configuration
+          matrix_stack_chart_ref: oci://ghcr.io/element-hq/ess-helm/matrix-stack
+          matrix_stack_chart_version: 26.7.2
+          matrix_stack_release_name: ess
+          matrix_namespace: ess
+    matrix:
+      children:
+        k3s_servers:
+    matrix_production:
+      children:
+        matrix:
diff --git a/ansible/playbooks/audit.yml b/ansible/playbooks/audit.yml
new file mode 100644
index 0000000..00d29d9
--- /dev/null
+++ b/ansible/playbooks/audit.yml
@@ -0,0 +1,73 @@
+---
+# Read-only control-plane audit. This play intentionally contains no package,
+# file, service, container, or configuration mutation tasks.
+- name: Audit managed VPS hosts without changes
+  hosts: managed
+  gather_facts: true
+  become: false
+  any_errors_fatal: false
+  tasks:
+    - name: Verify Docker Compose command is available
+      ansible.builtin.command:
+        argv:
+          - docker
+          - compose
+          - version
+      changed_when: false
+
+    - name: Inspect configured Compose project
+      ansible.builtin.command:
+        argv:
+          - docker
+          - compose
+          - --project-directory
+          - "{{ compose_project_dir }}"
+          - ps
+          - --all
+      register: audit_compose_ps
+      changed_when: false
+      failed_when: false
+
+    - name: Inspect failed systemd units
+      ansible.builtin.command:
+        argv:
+          - systemctl
+          - --failed
+          - --no-legend
+          - --no-pager
+      register: audit_failed_units
+      changed_when: false
+      failed_when: false
+
+    - name: Inspect filesystem capacity
+      ansible.builtin.command:
+        argv:
+          - df
+          - -P
+          - -x
+          - tmpfs
+          - -x
+          - devtmpfs
+      register: audit_filesystems
+      changed_when: false
+
+    - name: Inspect active listeners
+      ansible.builtin.command:
+        argv:
+          - ss
+          - -lntup
+      register: audit_listeners
+      changed_when: false
+      failed_when: false
+
+    - name: Report sanitized audit summary
+      ansible.builtin.debug:
+        msg:
+          host: "{{ inventory_hostname }}"
+          profile: "{{ healthcheck_profile }}"
+          os: "{{ ansible_distribution }} {{ ansible_distribution_version }}"
+          kernel: "{{ ansible_kernel }}"
+          compose_rc: "{{ audit_compose_ps.rc }}"
+          failed_units: "{{ audit_failed_units.stdout_lines | default([]) }}"
+          filesystem_lines: "{{ audit_filesystems.stdout_lines | default([]) }}"
+          listener_lines: "{{ audit_listeners.stdout_lines | default([]) }}"
diff --git a/ansible/playbooks/baseline.yml b/ansible/playbooks/baseline.yml
new file mode 100644
index 0000000..95b3689
--- /dev/null
+++ b/ansible/playbooks/baseline.yml
@@ -0,0 +1,9 @@
+---
+# Baseline starts audit-only. Opt-in variables are deliberately false by default.
+- name: Apply controlled common baseline
+  hosts: managed
+  become: true
+  gather_facts: false
+  roles:
+    - role: baseline
+      tags: [baseline, audit]
diff --git a/ansible/playbooks/email-alerts.yml b/ansible/playbooks/email-alerts.yml
new file mode 100644
index 0000000..c64c616
--- /dev/null
+++ b/ansible/playbooks/email-alerts.yml
@@ -0,0 +1,9 @@
+---
+# Requires a target-local root-owned SMTP config; no credentials are passed here.
+- name: Deploy health email alert integration
+  hosts: managed
+  become: true
+  gather_facts: false
+  roles:
+    - role: email_alert
+      tags: [healthcheck, email]
diff --git a/ansible/playbooks/healthchecks.yml b/ansible/playbooks/healthchecks.yml
new file mode 100644
index 0000000..ac7f81c
--- /dev/null
+++ b/ansible/playbooks/healthchecks.yml
@@ -0,0 +1,8 @@
+---
+- name: Deploy daily local health checks
+  hosts: managed
+  become: true
+  gather_facts: false
+  roles:
+    - role: healthcheck
+      tags: [healthcheck, timers]
diff --git a/ansible/playbooks/k3s-server.yml b/ansible/playbooks/k3s-server.yml
new file mode 100644
index 0000000..c416726
--- /dev/null
+++ b/ansible/playbooks/k3s-server.yml
@@ -0,0 +1,9 @@
+---
+- name: Install or reconcile the single-node K3s server
+  hosts: k3s_servers
+  become: true
+  gather_facts: true
+  serial: 1
+  roles:
+    - role: k3s_server
+      tags: [k3s, matrix, mutating]
diff --git a/ansible/playbooks/maintenance-preview.yml b/ansible/playbooks/maintenance-preview.yml
new file mode 100644
index 0000000..b2729ef
--- /dev/null
+++ b/ansible/playbooks/maintenance-preview.yml
@@ -0,0 +1,30 @@
+---
+# Preview only. This playbook does not install updates, restart services, or
+# change DNS/secrets. A separate, manually reviewed change is required to act.
+- name: Preview pending maintenance without changes
+  hosts: managed
+  become: true
+  gather_facts: false
+  tasks:
+    - name: Check reboot requirement marker
+      ansible.builtin.stat:
+        path: /var/run/reboot-required
+      register: maintenance_reboot_marker
+
+    - name: Preview available package updates on Debian-family hosts
+      ansible.builtin.command:
+        argv:
+          - apt-get
+          - --just-print
+          - upgrade
+      register: maintenance_apt_preview
+      changed_when: false
+      failed_when: false
+      when: ansible_facts.os_family | default('Debian') == 'Debian'
+
+    - name: Report maintenance preview
+      ansible.builtin.debug:
+        msg:
+          host: "{{ inventory_hostname }}"
+          reboot_required: "{{ maintenance_reboot_marker.stat.exists }}"
+          package_preview: "{{ maintenance_apt_preview.stdout_lines | default([]) }}"
diff --git a/ansible/playbooks/matrix-backup.yml b/ansible/playbooks/matrix-backup.yml
new file mode 100644
index 0000000..079892e
--- /dev/null
+++ b/ansible/playbooks/matrix-backup.yml
@@ -0,0 +1,8 @@
+---
+- name: Install local Matrix consistency backup jobs
+  hosts: matrix
+  become: true
+  gather_facts: false
+  roles:
+    - role: matrix_backup
+      tags: [matrix, backup, mutating]
diff --git a/ansible/playbooks/matrix-cert-manager.yml b/ansible/playbooks/matrix-cert-manager.yml
new file mode 100644
index 0000000..860d845
--- /dev/null
+++ b/ansible/playbooks/matrix-cert-manager.yml
@@ -0,0 +1,11 @@
+---
+- name: Reconcile Helm, cert-manager, and the Let's Encrypt issuer
+  hosts: k3s_servers
+  become: true
+  gather_facts: false
+  serial: 1
+  roles:
+    - role: helm_client
+      tags: [helm, matrix, mutating]
+    - role: cert_manager
+      tags: [cert_manager, matrix, mutating]
diff --git a/ansible/playbooks/matrix-certificate-smoke.yml b/ansible/playbooks/matrix-certificate-smoke.yml
new file mode 100644
index 0000000..055e51e
--- /dev/null
+++ b/ansible/playbooks/matrix-certificate-smoke.yml
@@ -0,0 +1,9 @@
+---
+- name: Run the temporary Matrix HTTP-01 certificate smoke test
+  hosts: matrix
+  become: true
+  gather_facts: false
+  serial: 1
+  roles:
+    - role: matrix_certificate_smoke
+      tags: [matrix, certificates, smoke_test, mutating]
diff --git a/ansible/playbooks/matrix-cluster-base.yml b/ansible/playbooks/matrix-cluster-base.yml
new file mode 100644
index 0000000..0f6d13e
--- /dev/null
+++ b/ansible/playbooks/matrix-cluster-base.yml
@@ -0,0 +1,8 @@
+---
+- name: Create non-secret Matrix Kubernetes foundation resources
+  hosts: matrix
+  become: true
+  gather_facts: false
+  roles:
+    - role: matrix_cluster_base
+      tags: [matrix, cluster_base, mutating]
diff --git a/ansible/playbooks/matrix-healthchecks.yml b/ansible/playbooks/matrix-healthchecks.yml
new file mode 100644
index 0000000..1efc99f
--- /dev/null
+++ b/ansible/playbooks/matrix-healthchecks.yml
@@ -0,0 +1,8 @@
+---
+- name: Install Matrix K3s local health checks
+  hosts: matrix
+  become: true
+  gather_facts: false
+  roles:
+    - role: matrix_healthcheck
+      tags: [matrix, healthcheck, timers]
diff --git a/ansible/playbooks/matrix-preflight.yml b/ansible/playbooks/matrix-preflight.yml
new file mode 100644
index 0000000..3b6aa73
--- /dev/null
+++ b/ansible/playbooks/matrix-preflight.yml
@@ -0,0 +1,9 @@
+---
+# Read-only gate before a K3s or Matrix change.
+- name: Validate Matrix K3s host readiness without changes
+  hosts: matrix
+  become: true
+  gather_facts: true
+  roles:
+    - role: k3s_preflight
+      tags: [matrix, preflight, read_only]
diff --git a/ansible/playbooks/matrix-secret-contract.yml b/ansible/playbooks/matrix-secret-contract.yml
new file mode 100644
index 0000000..1fbe256
--- /dev/null
+++ b/ansible/playbooks/matrix-secret-contract.yml
@@ -0,0 +1,8 @@
+---
+- name: Validate the pre-provisioned Matrix secret contract
+  hosts: matrix
+  become: true
+  gather_facts: false
+  roles:
+    - role: matrix_secret_contract
+      tags: [matrix, secrets, validation]
diff --git a/ansible/playbooks/matrix-stack.yml b/ansible/playbooks/matrix-stack.yml
new file mode 100644
index 0000000..00b496c
--- /dev/null
+++ b/ansible/playbooks/matrix-stack.yml
@@ -0,0 +1,99 @@
+---
+# Deploy the official ESS OCI chart on a K3s node.
+# Pre-tasks create the non-secret values files on the target host.
+# The matrix_stack role then validates and deploys the chart.
+- name: Deploy the Matrix stack (ESS OCI chart)
+  hosts: matrix
+  become: true
+  gather_facts: false
+  serial: 1
+  pre_tasks:
+    - name: Ensure the ESS values directory exists
+      ansible.builtin.file:
+        path: /etc/ess
+        state: directory
+        owner: root
+        group: root
+        mode: "0700"
+
+    - name: Write hostnames values file
+      ansible.builtin.copy:
+        dest: /etc/ess/hostnames.yaml
+        owner: root
+        group: root
+        mode: "0600"
+        content: |
+          serverName: {{ matrix_server_name }}
+
+          elementWeb:
+            ingress:
+              host: {{ matrix_element_host }}
+
+          synapse:
+            ingress:
+              host: {{ matrix_synapse_host }}
+
+          matrixAuthenticationService:
+            ingress:
+              host: {{ matrix_mas_host }}
+
+          elementAdmin:
+            ingress:
+              host: {{ matrix_admin_host }}
+
+          matrixRTC:
+            ingress:
+              host: {{ matrix_rtc_host }}
+
+    - name: Write TLS values file
+      ansible.builtin.copy:
+        dest: /etc/ess/tls.yaml
+        owner: root
+        group: root
+        mode: "0600"
+        content: |
+          certManager:
+            clusterIssuer: letsencrypt-prod
+          ingress:
+            className: traefik
+            tlsEnabled: true
+
+    - name: Write single-node tuning values file
+      ansible.builtin.copy:
+        dest: /etc/ess/single-node.yaml
+        owner: root
+        group: root
+        mode: "0600"
+        content: |
+          # ESS single-node resource tuning for K3s
+          # Chart defaults are already single-node-friendly
+          postgres:
+            storage:
+              size: 20Gi
+          redis:
+            maxMemory: 128mb
+          matrixRTC:
+            enabled: false
+
+    - name: Verify values files are in place
+      ansible.builtin.stat:
+        path: "/etc/ess/{{ item }}"
+      loop:
+        - hostnames.yaml
+        - tls.yaml
+        - single-node.yaml
+      register: _values_check
+
+    - name: Assert all values files exist
+      ansible.builtin.assert:
+        that:
+          - item.stat.exists
+          - item.stat.isreg
+          - item.stat.pw_name == 'root'
+      loop: "{{ _values_check.results }}"
+      loop_control:
+        label: "{{ item.stat.path | default(item.item) }}"
+
+  roles:
+    - role: matrix_stack
+      tags: [matrix, stack, mutating]
\ No newline at end of file
diff --git a/ansible/playbooks/restic.yml b/ansible/playbooks/restic.yml
new file mode 100644
index 0000000..a8d6bbc
--- /dev/null
+++ b/ansible/playbooks/restic.yml
@@ -0,0 +1,13 @@
+---
+# Intentionally targets only services with approved local dump/data sources.
+# Set restic_enabled=true only after selecting a backend and provisioning the
+# root-only repository config directly on each target.
+- name: Deploy controlled Restic timers
+  hosts:
+    - vaultwarden
+    - powerdns
+  become: true
+  gather_facts: false
+  roles:
+    - role: restic
+      tags: [restic, backup]
diff --git a/ansible/roles/baseline/defaults/main.yml b/ansible/roles/baseline/defaults/main.yml
new file mode 100644
index 0000000..2371f34
--- /dev/null
+++ b/ansible/roles/baseline/defaults/main.yml
@@ -0,0 +1,3 @@
+---
+baseline_manage_logrotate: false
+baseline_require_chrony: false
diff --git a/ansible/roles/baseline/tasks/main.yml b/ansible/roles/baseline/tasks/main.yml
new file mode 100644
index 0000000..2fccbb2
--- /dev/null
+++ b/ansible/roles/baseline/tasks/main.yml
@@ -0,0 +1,44 @@
+---
+- name: Audit SSH daemon effective configuration
+  ansible.builtin.command:
+    argv:
+      - sshd
+      - -T
+  register: baseline_sshd_effective
+  changed_when: false
+  failed_when: false
+
+- name: Report SSH hardening observations without changing SSH
+  ansible.builtin.debug:
+    msg:
+      permit_root_login: >-
+        {{ baseline_sshd_effective.stdout_lines | select('match', '^permitrootlogin ') | list }}
+      password_authentication: >-
+        {{ baseline_sshd_effective.stdout_lines | select('match', '^passwordauthentication ') | list }}
+
+- name: Audit time synchronization service state
+  ansible.builtin.command:
+    argv:
+      - systemctl
+      - is-active
+      - systemd-timesyncd
+  register: baseline_timesync_state
+  changed_when: false
+  failed_when: false
+
+- name: Install baseline logrotate policy only with explicit opt-in
+  ansible.builtin.copy:
+    dest: /etc/logrotate.d/vps-baseline
+    content: |
+      /var/log/vps-health/*.log /var/log/vps-restic/*.log {
+          weekly
+          rotate 8
+          missingok
+          notifempty
+          compress
+          create 0640 root root
+      }
+    owner: root
+    group: root
+    mode: "0644"
+  when: baseline_manage_logrotate | bool
diff --git a/ansible/roles/cert_manager/defaults/main.yml b/ansible/roles/cert_manager/defaults/main.yml
new file mode 100644
index 0000000..a979a11
--- /dev/null
+++ b/ansible/roles/cert_manager/defaults/main.yml
@@ -0,0 +1,13 @@
+---
+cert_manager_enabled: false
+cert_manager_namespace: cert-manager
+cert_manager_release_name: cert-manager
+cert_manager_chart_repository_name: jetstack
+cert_manager_chart_repository_url: https://charts.jetstack.io
+cert_manager_chart_ref: jetstack/cert-manager
+cert_manager_chart_version: v1.19.3
+cert_manager_kubeconfig: /etc/rancher/k3s/k3s.yaml
+cert_manager_cluster_issuer_name: letsencrypt-prod
+cert_manager_acme_server: https://acme-v02.api.letsencrypt.org/directory
+cert_manager_acme_private_key_secret: letsencrypt-prod-private-key
+cert_manager_ingress_class: traefik
diff --git a/ansible/roles/cert_manager/tasks/main.yml b/ansible/roles/cert_manager/tasks/main.yml
new file mode 100644
index 0000000..3999943
--- /dev/null
+++ b/ansible/roles/cert_manager/tasks/main.yml
@@ -0,0 +1,94 @@
+---
+- name: Require explicit approval before cert-manager reconciliation
+  ansible.builtin.assert:
+    that:
+      - cert_manager_enabled | bool
+    fail_msg: >-
+      Refusing cert-manager changes. Re-run with cert_manager_enabled=true after
+      confirming public TCP 80 and 443 reach Traefik.
+
+- name: Require Helm and the K3s kubeconfig
+  ansible.builtin.stat:
+    path: "{{ item }}"
+  loop:
+    - /usr/local/bin/helm
+    - "{{ cert_manager_kubeconfig }}"
+  register: cert_manager_prerequisites
+
+- name: Assert Helm and kubeconfig are available
+  ansible.builtin.assert:
+    that:
+      - item.stat.exists
+    fail_msg: "Missing cert-manager prerequisite: {{ item.item }}"
+  loop: "{{ cert_manager_prerequisites.results }}"
+
+- name: Add or update the Jetstack Helm repository
+  ansible.builtin.command:
+    argv:
+      - helm
+      - repo
+      - add
+      - "{{ cert_manager_chart_repository_name }}"
+      - "{{ cert_manager_chart_repository_url }}"
+      - --force-update
+  environment:
+    KUBECONFIG: "{{ cert_manager_kubeconfig }}"
+  changed_when: false
+
+- name: Update Helm repositories
+  ansible.builtin.command:
+    argv: [helm, repo, update]
+  environment:
+    KUBECONFIG: "{{ cert_manager_kubeconfig }}"
+  changed_when: false
+
+- name: Install or reconcile cert-manager
+  ansible.builtin.command:
+    argv:
+      - helm
+      - upgrade
+      - --install
+      - "{{ cert_manager_release_name }}"
+      - "{{ cert_manager_chart_ref }}"
+      - --namespace
+      - "{{ cert_manager_namespace }}"
+      - --create-namespace
+      - --version
+      - "{{ cert_manager_chart_version }}"
+      - --set
+      - crds.enabled=true
+      - --wait
+      - --timeout
+      - 10m
+  environment:
+    KUBECONFIG: "{{ cert_manager_kubeconfig }}"
+  changed_when: true
+
+- name: Install ClusterIssuer manifest without secret material
+  ansible.builtin.template:
+    src: clusterissuer.yaml.j2
+    dest: "/etc/rancher/k3s/{{ cert_manager_cluster_issuer_name }}-clusterissuer.yaml"
+    owner: root
+    group: root
+    mode: "0600"
+
+- name: Apply ClusterIssuer manifest
+  ansible.builtin.command:
+    argv:
+      - k3s
+      - kubectl
+      - apply
+      - -f
+      - "/etc/rancher/k3s/{{ cert_manager_cluster_issuer_name }}-clusterissuer.yaml"
+  changed_when: false
+
+- name: Wait for ClusterIssuer readiness
+  ansible.builtin.command:
+    argv:
+      - k3s
+      - kubectl
+      - wait
+      - --for=condition=Ready
+      - "clusterissuer/{{ cert_manager_cluster_issuer_name }}"
+      - --timeout=180s
+  changed_when: false
diff --git a/ansible/roles/cert_manager/templates/clusterissuer.yaml.j2 b/ansible/roles/cert_manager/templates/clusterissuer.yaml.j2
new file mode 100644
index 0000000..af0fa89
--- /dev/null
+++ b/ansible/roles/cert_manager/templates/clusterissuer.yaml.j2
@@ -0,0 +1,13 @@
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: {{ cert_manager_cluster_issuer_name }}
+spec:
+  acme:
+    server: {{ cert_manager_acme_server }}
+    privateKeySecretRef:
+      name: {{ cert_manager_acme_private_key_secret }}
+    solvers:
+      - http01:
+          ingress:
+            class: {{ cert_manager_ingress_class }}
diff --git a/ansible/roles/email_alert/defaults/main.yml b/ansible/roles/email_alert/defaults/main.yml
new file mode 100644
index 0000000..642742d
--- /dev/null
+++ b/ansible/roles/email_alert/defaults/main.yml
@@ -0,0 +1,6 @@
+---
+email_alert_config_path: /etc/vps-health/alert-smtp.conf
+email_alert_state_path: /var/lib/vps-health/alert-state
+email_alert_recipient: ''
+email_alert_enabled: false
+email_alert_repeat_hours: 24
diff --git a/ansible/roles/email_alert/handlers/main.yml b/ansible/roles/email_alert/handlers/main.yml
new file mode 100644
index 0000000..5498019
--- /dev/null
+++ b/ansible/roles/email_alert/handlers/main.yml
@@ -0,0 +1,4 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd_service:
+    daemon_reload: true
diff --git a/ansible/roles/email_alert/tasks/main.yml b/ansible/roles/email_alert/tasks/main.yml
new file mode 100644
index 0000000..8eee232
--- /dev/null
+++ b/ansible/roles/email_alert/tasks/main.yml
@@ -0,0 +1,50 @@
+---
+# The health-check service owns the sole dispatcher hook. This role only
+# installs/removes that dispatcher according to the explicit opt-in below.
+- name: Require explicit non-secret alert recipient when email is enabled
+  ansible.builtin.assert:
+    that:
+      - email_alert_recipient | length > 0
+    fail_msg: Set email_alert_recipient outside version control before enabling alerts.
+  when: email_alert_enabled | bool
+
+- name: Install alert integration when explicitly enabled
+  when: email_alert_enabled | bool
+  block:
+    - name: Install alert state directory
+      ansible.builtin.file:
+        path: "{{ email_alert_state_path | dirname }}"
+        state: directory
+        owner: root
+        group: root
+        mode: "0750"
+
+    - name: Install secret-free alert dispatcher
+      ansible.builtin.template:
+        src: alert-dispatch.sh.j2
+        dest: /usr/local/lib/vps-health/alert-dispatch
+        owner: root
+        group: root
+        mode: "0750"
+
+- name: Remove alert integration when disabled
+  when: not (email_alert_enabled | bool)
+  block:
+    - name: Remove alert dispatcher
+      ansible.builtin.file:
+        path: /usr/local/lib/vps-health/alert-dispatch
+        state: absent
+
+    - name: Remove legacy alert service drop-in
+      ansible.builtin.file:
+        path: /etc/systemd/system/vps-healthcheck.service.d/alerting.conf
+        state: absent
+      notify: Reload systemd
+
+- name: Report required server-side alert configuration
+  ansible.builtin.debug:
+    msg: >-
+      Email alerts are {{ 'enabled' if email_alert_enabled | bool else 'disabled' }}.
+      When enabled, the root-owned {{ email_alert_config_path }} must be provisioned
+      directly on the host and must contain SMTP settings and recipient; it is never
+      created or populated by this repository.
diff --git a/ansible/roles/email_alert/templates/alert-dispatch.sh.j2 b/ansible/roles/email_alert/templates/alert-dispatch.sh.j2
new file mode 100644
index 0000000..7ca7985
--- /dev/null
+++ b/ansible/roles/email_alert/templates/alert-dispatch.sh.j2
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# Sends sanitized health results through the host's locally provisioned SMTP
+# credentials. The config is intentionally excluded from Ansible/Git.
+set -uo pipefail
+config='{{ email_alert_config_path }}'
+result='/var/lib/vps-health/latest.json'
+state='{{ email_alert_state_path }}'
+[[ -r "$config" && -r "$result" ]] || exit 0
+# shellcheck source=/dev/null
+source "$config"
+: "${SMTP_URL:?missing SMTP_URL in server-side alert config}"
+: "${ALERT_TO:?missing ALERT_TO in server-side alert config}"
+status="$(python3 -c 'import json; print(json.load(open("'"$result"'"))["status"])')"
+fingerprint="$(sha256sum "$result" | cut -d' ' -f1)"
+previous="$(cat "$state" 2>/dev/null || true)"
+now="$(date +%s)"
+last_time="${previous%%:*}"; last_fp="${previous#*:}"
+if [[ "$status" =~ ^(critical|unknown)$ ]] && [[ "$fingerprint" != "$last_fp" || $((now-${last_time:-0})) -ge {{ email_alert_repeat_hours }}*3600 ]]; then
+  subject="[${status}] VPS health $(hostname -f 2>/dev/null || hostname)"
+  curl --fail --silent --show-error --url "$SMTP_URL" --mail-rcpt "$ALERT_TO" --upload-file <(printf 'To: %s\nSubject: %s\nContent-Type: application/json\n\n%s\n' "$ALERT_TO" "$subject" "$(cat "$result")")
+  printf '%s:%s\n' "$now" "$fingerprint" > "$state"
+elif [[ "$status" =~ ^(ok|warning)$ ]]; then
+  subject="[${status}] daily VPS health $(hostname -f 2>/dev/null || hostname)"
+  curl --fail --silent --show-error --url "$SMTP_URL" --mail-rcpt "$ALERT_TO" --upload-file <(printf 'To: %s\nSubject: %s\nContent-Type: application/json\n\n%s\n' "$ALERT_TO" "$subject" "$(cat "$result")")
+fi
diff --git a/ansible/roles/email_alert/templates/alerting.conf.j2 b/ansible/roles/email_alert/templates/alerting.conf.j2
new file mode 100644
index 0000000..9ca441f
--- /dev/null
+++ b/ansible/roles/email_alert/templates/alerting.conf.j2
@@ -0,0 +1,2 @@
+[Service]
+ExecStartPost=/usr/local/lib/vps-health/alert-dispatch
diff --git a/ansible/roles/healthcheck/defaults/main.yml b/ansible/roles/healthcheck/defaults/main.yml
new file mode 100644
index 0000000..5a937f0
--- /dev/null
+++ b/ansible/roles/healthcheck/defaults/main.yml
@@ -0,0 +1,13 @@
+---
+healthcheck_install_root: /usr/local/lib/vps-health
+healthcheck_state_dir: /var/lib/vps-health
+healthcheck_log_dir: /var/log/vps-health
+healthcheck_service_name: vps-healthcheck
+healthcheck_timer_on_calendar: '*-*-* 06:15:00'
+healthcheck_timer_randomized_delay_sec: 15m
+healthcheck_backup_max_age_hours: 30
+healthcheck_tls_warn_days: 21
+healthcheck_profiles:
+  mailcow: mailcow.sh
+  vaultwarden: vaultwarden.sh
+  pdns: pdns.sh
diff --git a/ansible/roles/healthcheck/handlers/main.yml b/ansible/roles/healthcheck/handlers/main.yml
new file mode 100644
index 0000000..5498019
--- /dev/null
+++ b/ansible/roles/healthcheck/handlers/main.yml
@@ -0,0 +1,4 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd_service:
+    daemon_reload: true
diff --git a/ansible/roles/healthcheck/tasks/main.yml b/ansible/roles/healthcheck/tasks/main.yml
new file mode 100644
index 0000000..ab2b142
--- /dev/null
+++ b/ansible/roles/healthcheck/tasks/main.yml
@@ -0,0 +1,81 @@
+---
+- name: Validate known health-check profile
+  ansible.builtin.assert:
+    that:
+      - healthcheck_profile in healthcheck_profiles
+    fail_msg: "Unsupported healthcheck_profile: {{ healthcheck_profile }}"
+
+- name: Install health-check directories
+  ansible.builtin.file:
+    path: "{{ item.path }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "{{ item.mode }}"
+  loop:
+    - path: "{{ healthcheck_install_root }}"
+      mode: "0755"
+    - path: "{{ healthcheck_state_dir }}"
+      mode: "0750"
+    - path: "{{ healthcheck_log_dir }}"
+      mode: "0750"
+
+- name: Install common health-check library
+  ansible.builtin.template:
+    src: health-common.sh.j2
+    dest: "{{ healthcheck_install_root }}/health-common.sh"
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install service health-check script
+  ansible.builtin.template:
+    src: "{{ healthcheck_profiles[healthcheck_profile] }}.j2"
+    dest: "{{ healthcheck_install_root }}/{{ healthcheck_profiles[healthcheck_profile] }}"
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install health-check dispatcher
+  ansible.builtin.template:
+    src: healthcheck-runner.sh.j2
+    dest: "{{ healthcheck_install_root }}/run"
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install health-check systemd unit
+  ansible.builtin.template:
+    src: vps-healthcheck.service.j2
+    dest: "/etc/systemd/system/{{ healthcheck_service_name }}.service"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Install persistent health-check timer
+  ansible.builtin.template:
+    src: vps-healthcheck.timer.j2
+    dest: "/etc/systemd/system/{{ healthcheck_service_name }}.timer"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Install health-check log rotation
+  ansible.builtin.template:
+    src: logrotate-vps-healthcheck.j2
+    dest: /etc/logrotate.d/vps-healthcheck
+    owner: root
+    group: root
+    mode: "0644"
+
+- name: Enable health-check timer
+  ansible.builtin.systemd_service:
+    name: "{{ healthcheck_service_name }}.timer"
+    enabled: true
+    state: started
+    daemon_reload: true
+
+- name: Flush health-check systemd changes
+  ansible.builtin.meta: flush_handlers
diff --git a/ansible/roles/healthcheck/templates/health-common.sh.j2 b/ansible/roles/healthcheck/templates/health-common.sh.j2
new file mode 100644
index 0000000..1398ddb
--- /dev/null
+++ b/ansible/roles/healthcheck/templates/health-common.sh.j2
@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+# Shared contract: one JSON object per run in /var/lib/vps-health/latest.json.
+# Status values: ok, warning, critical, unknown. Exit codes: 0 healthy/warning,
+# 2 critical, 3 unknown. Check detail must never include secret values.
+set -uo pipefail
+
+RESULT_DIR='{{ healthcheck_state_dir }}'
+LOG_DIR='{{ healthcheck_log_dir }}'
+HOST_NAME="$(hostname -f 2>/dev/null || hostname)"
+CHECK_NAME='{{ healthcheck_profile }}'
+STATUS=ok
+EXIT_CODE=0
+DETAILS=()
+
+record() {
+  local severity="$1" message="$2"
+  DETAILS+=("${severity}:${message}")
+  case "$severity" in
+    critical) STATUS=critical; EXIT_CODE=2 ;;
+    unknown) [[ "$STATUS" != critical ]] && STATUS=unknown; [[ "$EXIT_CODE" -eq 0 ]] && EXIT_CODE=3 ;;
+    warning) [[ "$STATUS" == ok ]] && STATUS=warning ;;
+  esac
+}
+
+require_command() {
+  command -v "$1" >/dev/null 2>&1 || record unknown "missing_command:$1"
+}
+
+compose_ps() {
+  docker compose --project-directory '{{ compose_project_dir }}' ps --all 2>&1
+}
+
+check_compose() {
+  local output
+  output="$(compose_ps)" || { record critical 'compose_ps_failed'; return; }
+  if grep -qiE 'Exited|Restarting|[[:space:]]Dead[[:space:]]' <<<"$output"; then
+    record critical 'compose_unhealthy_container'
+  else
+    record ok 'compose_ok'
+  fi
+}
+
+check_backup_freshness() {
+  local pattern="$1" newest now age
+  newest="$(find {{ compose_project_dir | quote }} -type f -path "$pattern" -printf '%T@\n' 2>/dev/null | sort -nr | head -n1)"
+  if [[ -z "$newest" ]]; then
+    record warning 'backup_not_found'
+    return
+  fi
+  now="$(date +%s)"
+  age="$(( now - ${newest%.*} ))"
+  if (( age > {{ healthcheck_backup_max_age_hours }} * 3600 )); then
+    record critical 'backup_stale'
+  else
+    record ok 'backup_fresh'
+  fi
+}
+
+check_https() {
+  local url="$1" expected="$2" code
+  code="$(curl --silent --show-error --output /dev/null --write-out '%{http_code}' --max-time 20 "$url" 2>/dev/null)" || {
+    record critical 'https_request_failed'; return;
+  }
+  [[ "$code" =~ $expected ]] && record ok "https_${code}" || record critical "https_${code}"
+}
+
+check_tls_days() {
+  local host="$1" port="$2" expires epoch remaining
+  expires="$(timeout 20 openssl s_client -connect "${host}:${port}" -servername "$host" </dev/null 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2-)" || {
+    record unknown 'tls_read_failed'; return;
+  }
+  epoch="$(date -d "$expires" +%s 2>/dev/null)" || { record unknown 'tls_date_parse_failed'; return; }
+  remaining="$(( (epoch - $(date +%s)) / 86400 ))"
+  if (( remaining < 0 )); then record critical 'tls_expired'
+  elif (( remaining < {{ healthcheck_tls_warn_days }} )); then record warning 'tls_near_expiry'
+  else record ok 'tls_valid'; fi
+}
+
+emit_result() {
+  local tmp detail_json
+  tmp="$(mktemp "${RESULT_DIR}/latest.json.XXXXXX")"
+  detail_json="$(printf '%s\n' "${DETAILS[@]:-unknown:no_details}" | python3 -c 'import json,sys; print(json.dumps([line.rstrip() for line in sys.stdin if line.strip()]))')"
+  python3 - "$tmp" "$HOST_NAME" "$CHECK_NAME" "$STATUS" "$EXIT_CODE" "$detail_json" <<'PY'
+import json, sys
+path, host, check, status, code, details = sys.argv[1:]
+with open(path, 'w', encoding='utf-8') as f:
+    json.dump({'schema': 1, 'host': host, 'check': check, 'status': status,
+               'exit_code': int(code), 'details': json.loads(details)}, f,
+              sort_keys=True, separators=(',', ':'))
+    f.write('\n')
+PY
+  chmod 0640 "$tmp"
+  mv "$tmp" "${RESULT_DIR}/latest.json"
+  cat "${RESULT_DIR}/latest.json"
+}
diff --git a/ansible/roles/healthcheck/templates/healthcheck-runner.sh.j2 b/ansible/roles/healthcheck/templates/healthcheck-runner.sh.j2
new file mode 100644
index 0000000..84e5987
--- /dev/null
+++ b/ansible/roles/healthcheck/templates/healthcheck-runner.sh.j2
@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+set -uo pipefail
+exec '{{ healthcheck_install_root }}/{{ healthcheck_profiles[healthcheck_profile] }}' >> '{{ healthcheck_log_dir }}/healthcheck.log' 2>&1
diff --git a/ansible/roles/healthcheck/templates/logrotate-vps-healthcheck.j2 b/ansible/roles/healthcheck/templates/logrotate-vps-healthcheck.j2
new file mode 100644
index 0000000..73070a9
--- /dev/null
+++ b/ansible/roles/healthcheck/templates/logrotate-vps-healthcheck.j2
@@ -0,0 +1,9 @@
+{{ healthcheck_log_dir }}/healthcheck.log {
+    daily
+    rotate 14
+    missingok
+    notifempty
+    compress
+    delaycompress
+    create 0640 root root
+}
diff --git a/ansible/roles/healthcheck/templates/mailcow.sh.j2 b/ansible/roles/healthcheck/templates/mailcow.sh.j2
new file mode 100644
index 0000000..88fa62c
--- /dev/null
+++ b/ansible/roles/healthcheck/templates/mailcow.sh.j2
@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+set -uo pipefail
+source '{{ healthcheck_install_root }}/health-common.sh'
+
+require_command docker
+require_command curl
+require_command openssl
+require_command dig
+require_command timeout
+check_compose
+
+watchdog="$(docker compose --project-directory '{{ compose_project_dir }}' logs --tail=40 watchdog-mailcow 2>&1)" || record critical 'watchdog_log_failed'
+grep -qiE '100%|healthy' <<<"$watchdog" || record warning 'watchdog_health_not_confirmed'
+
+queue="$(docker compose --project-directory '{{ compose_project_dir }}' exec -T postfix-mailcow postqueue -p 2>&1)" || record critical 'mail_queue_check_failed'
+grep -Fqi 'Mail queue is empty' <<<"$queue" || record warning 'mail_queue_nonempty'
+
+listeners="$(ss -lnt 2>/dev/null)"
+for port in 25 465 587 993 443; do
+  grep -qE ":${port}[[:space:]]" <<<"$listeners" || record critical "listener_missing_${port}"
+done
+
+check_https 'https://mx2.windy.me/' '^200$'
+check_tls_days mx2.windy.me 443
+smtp="$(timeout 10 bash -c "exec 3<>/dev/tcp/mx2.windy.me/25; printf 'EHLO health.local\\r\\nQUIT\\r\\n' >&3; cat <&3" 2>/dev/null)" || record critical 'smtp_connect_failed'
+grep -qiE 'Postcow|ESMTP' <<<"$smtp" || record critical 'smtp_banner_unexpected'
+
+for resolver in 1.1.1.1 8.8.8.8; do
+  mx="$(dig +short +time=3 +tries=1 "@${resolver}" windy.me MX 2>/dev/null)"
+  grep -Fqi 'mx2.windy.me' <<<"$mx" && { record ok 'mx_record_ok'; break; } || record warning 'mx_record_not_confirmed'
+done
+spf="$(dig +short +time=3 +tries=1 @1.1.1.1 windy.me TXT 2>/dev/null)"
+grep -Fqi 'v=spf1' <<<"$spf" || record warning 'spf_not_confirmed'
+check_backup_freshness '*/backup/*'
+emit_result
+exit "$EXIT_CODE"
diff --git a/ansible/roles/healthcheck/templates/pdns.sh.j2 b/ansible/roles/healthcheck/templates/pdns.sh.j2
new file mode 100644
index 0000000..ea76661
--- /dev/null
+++ b/ansible/roles/healthcheck/templates/pdns.sh.j2
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+set -uo pipefail
+source '{{ healthcheck_install_root }}/health-common.sh'
+
+require_command docker
+require_command curl
+require_command dig
+check_compose
+
+image="$(docker inspect pdns-auth --format '{{ '{{' }}.Config.Image{{ '}}' }}' 2>/dev/null)" || record critical 'pdns_container_missing'
+grep -Eq ':5\.0\.[6-9]|:5\.[1-9]\.' <<<"$image" || record warning 'pdns_version_not_confirmed'
+security="$(docker logs pdns-auth 2>&1 | grep -i 'Mandatory.*Security Update' || true)"
+[[ -z "$security" ]] || record critical 'pdns_security_update_banner'
+
+# The API key remains in the container environment; the request emits only the
+# reported version and no authentication material.
+api="$(docker compose --project-directory '{{ compose_project_dir }}' exec -T auth python3 - <<'PY' 2>&1
+import json, os, urllib.request
+request=urllib.request.Request('http://127.0.0.1:8081/api/v1/servers/localhost', headers={'X-API-Key': os.environ['PDNS_API_KEY']})
+print(json.load(urllib.request.urlopen(request, timeout=10))['version'])
+PY
+)" || record critical 'pdns_api_failed'
+grep -Eq '^5\.' <<<"$api" || record critical 'pdns_api_version_invalid'
+
+for zone in windy.me wsvc.info chans.xyz; do
+  primary="$(dig +short @154.36.174.161 SOA "$zone" 2>/dev/null)"
+  secondary="$(dig +short @202.91.35.141 SOA "$zone" 2>/dev/null)"
+  [[ -n "$primary" ]] || record critical "soa_primary_missing_${zone}"
+  [[ -n "$secondary" ]] || record critical "soa_secondary_missing_${zone}"
+  [[ "$primary" == "$secondary" ]] || record warning "soa_secondary_lag_${zone}"
+done
+
+check_https 'https://pdns.wsvc.info/' '^30[12]$'
+check_https 'https://pgweb.wsvc.info/' '^(200|401)$'
+check_backup_freshness '*/backup/*.sql.gz'
+emit_result
+exit "$EXIT_CODE"
diff --git a/ansible/roles/healthcheck/templates/vaultwarden.sh.j2 b/ansible/roles/healthcheck/templates/vaultwarden.sh.j2
new file mode 100644
index 0000000..631fe36
--- /dev/null
+++ b/ansible/roles/healthcheck/templates/vaultwarden.sh.j2
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+set -uo pipefail
+source '{{ healthcheck_install_root }}/health-common.sh'
+
+require_command docker
+require_command curl
+require_command openssl
+require_command python3
+check_compose
+
+health="$(docker compose --project-directory '{{ compose_project_dir }}' ps --format json 2>&1)" || record critical 'compose_status_unavailable'
+grep -Fq 'vaultwarden' <<<"$health" || record critical 'vaultwarden_missing'
+grep -Fq 'vw-db' <<<"$health" || record critical 'postgres_missing'
+
+check_https 'https://auth.wsvc.info/' '^200$'
+check_tls_days auth.wsvc.info 443
+
+# Read effective config only inside the service and report booleans/fingerprints,
+# never its SMTP password or other secret fields.
+smtp_result="$(docker compose --project-directory '{{ compose_project_dir }}' exec -T vaultwarden python3 - <<'PY' 2>&1
+import json, pathlib, smtplib, ssl
+cfg=json.loads(pathlib.Path('/data/config.json').read_text())
+host=cfg.get('smtp_host'); port=int(cfg.get('smtp_port') or 0)
+user=cfg.get('smtp_username')
+smtp_secret=cfg.get('smtp_password')
+if not all((host, port, user, smtp_secret)):
+    raise SystemExit('smtp_config_incomplete')
+with smtplib.SMTP(host, port, timeout=15) as client:
+    client.ehlo(); client.starttls(context=ssl.create_default_context()); client.ehlo(); client.login(user, smtp_secret)
+print('smtp_auth_ok')
+PY
+)"
+grep -Fqx 'smtp_auth_ok' <<<"$smtp_result" || record critical 'smtp_auth_failed'
+
+# Detect drift without exposing the values: matching SHA-256 digests are used only
+# internally and the result is a boolean.
+drift_result="$(python3 - <<'PY'
+import hashlib, json, pathlib, re
+root=pathlib.Path('{{ compose_project_dir }}')
+env=root.joinpath('.env').read_text()
+match=re.search(r'^SMTP_PASSWORD=(.*)$', env, re.M)
+config=json.loads(root.joinpath('vw-data/config.json').read_text())
+value=(match.group(1).strip().strip('"\'') if match else '')
+print('smtp_password_match' if value and value == (config.get('smtp_password') or '') else 'smtp_password_drift')
+PY
+ 2>&1)" || record unknown 'smtp_drift_check_failed'
+grep -Fqx 'smtp_password_match' <<<"$drift_result" || record critical 'smtp_password_drift'
+
+check_backup_freshness '*/backups/*.sql.gz'
+emit_result
+exit "$EXIT_CODE"
diff --git a/ansible/roles/healthcheck/templates/vps-healthcheck.service.j2 b/ansible/roles/healthcheck/templates/vps-healthcheck.service.j2
new file mode 100644
index 0000000..3dc2adf
--- /dev/null
+++ b/ansible/roles/healthcheck/templates/vps-healthcheck.service.j2
@@ -0,0 +1,19 @@
+[Unit]
+Description=Read-only VPS health check (%i)
+Wants=network-online.target
+After=network-online.target docker.service
+
+[Service]
+Type=oneshot
+User=root
+Group=root
+UMask=0027
+ExecStart={{ healthcheck_install_root }}/run
+# The optional dispatcher is installed only by email_alert when explicitly enabled.
+# This service remains healthy if alerting is intentionally unavailable.
+ExecStartPost=-/usr/local/lib/vps-health/alert-dispatch
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectHome=true
+ProtectSystem=full
+ReadWritePaths={{ healthcheck_state_dir }} {{ healthcheck_log_dir }}
diff --git a/ansible/roles/healthcheck/templates/vps-healthcheck.timer.j2 b/ansible/roles/healthcheck/templates/vps-healthcheck.timer.j2
new file mode 100644
index 0000000..36bc43d
--- /dev/null
+++ b/ansible/roles/healthcheck/templates/vps-healthcheck.timer.j2
@@ -0,0 +1,11 @@
+[Unit]
+Description=Daily read-only VPS health check
+
+[Timer]
+OnCalendar={{ healthcheck_timer_on_calendar }}
+Persistent=true
+RandomizedDelaySec={{ healthcheck_timer_randomized_delay_sec }}
+Unit={{ healthcheck_service_name }}.service
+
+[Install]
+WantedBy=timers.target
diff --git a/ansible/roles/helm_client/defaults/main.yml b/ansible/roles/helm_client/defaults/main.yml
new file mode 100644
index 0000000..d1800e5
--- /dev/null
+++ b/ansible/roles/helm_client/defaults/main.yml
@@ -0,0 +1,6 @@
+---
+# Helm is required by the patched matrix-stack release. Existing installs are
+# reported and retained; this role only installs Helm when it is absent.
+helm_client_install_enabled: false
+helm_client_install_script_url: https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
+helm_client_binary: /usr/local/bin/helm
diff --git a/ansible/roles/helm_client/tasks/main.yml b/ansible/roles/helm_client/tasks/main.yml
new file mode 100644
index 0000000..518c5d4
--- /dev/null
+++ b/ansible/roles/helm_client/tasks/main.yml
@@ -0,0 +1,28 @@
+---
+- name: Check whether Helm is installed
+  ansible.builtin.stat:
+    path: "{{ helm_client_binary }}"
+  register: helm_client_binary_state
+
+- name: Require explicit approval to install Helm when absent
+  ansible.builtin.assert:
+    that:
+      - helm_client_install_enabled | bool
+    fail_msg: >-
+      Helm is not installed. Re-run with helm_client_install_enabled=true after
+      reviewing the upstream installer source and checksum policy.
+  when: not helm_client_binary_state.stat.exists
+
+- name: Install Helm only when explicitly approved and absent
+  ansible.builtin.shell:
+    cmd: "curl -fsSL {{ helm_client_install_script_url }} | bash"
+    creates: "{{ helm_client_binary }}"
+  when:
+    - helm_client_install_enabled | bool
+    - not helm_client_binary_state.stat.exists
+  no_log: true
+
+- name: Read Helm version
+  ansible.builtin.command:
+    argv: ["{{ helm_client_binary }}", version, --short]
+  changed_when: false
diff --git a/ansible/roles/k3s_preflight/defaults/main.yml b/ansible/roles/k3s_preflight/defaults/main.yml
new file mode 100644
index 0000000..4e8dbfc
--- /dev/null
+++ b/ansible/roles/k3s_preflight/defaults/main.yml
@@ -0,0 +1,14 @@
+---
+# Read-only checks for a dedicated, single-node Matrix K3s host.
+k3s_preflight_required_memory_mib: 6144
+k3s_preflight_required_root_free_gib: 50
+k3s_preflight_required_hosts:
+  - "{{ matrix_server_name }}"
+  - "{{ matrix_synapse_host }}"
+  - "{{ matrix_element_host }}"
+  - "{{ matrix_mas_host }}"
+  - "{{ matrix_admin_host }}"
+  - "{{ matrix_rtc_host }}"
+k3s_preflight_required_ports:
+  - 80
+  - 443
diff --git a/ansible/roles/k3s_preflight/tasks/main.yml b/ansible/roles/k3s_preflight/tasks/main.yml
new file mode 100644
index 0000000..64f8050
--- /dev/null
+++ b/ansible/roles/k3s_preflight/tasks/main.yml
@@ -0,0 +1,69 @@
+---
+- name: Read host memory in MiB
+  ansible.builtin.set_fact:
+    k3s_preflight_memory_mib: "{{ (ansible_memtotal_mb | int) }}"
+
+- name: Read root filesystem capacity facts
+  ansible.builtin.set_fact:
+    k3s_preflight_root_mount: >-
+      {{ (ansible_mounts | selectattr('mount', 'equalto', '/') | list | first) | default({}) }}
+
+- name: Assert host has the baseline resources for Matrix
+  ansible.builtin.assert:
+    that:
+      - k3s_preflight_memory_mib | int >= k3s_preflight_required_memory_mib | int
+      - (k3s_preflight_root_mount.size_available | default(0) | int) >= (k3s_preflight_required_root_free_gib | int * 1024 * 1024 * 1024)
+    fail_msg: >-
+      Matrix requires at least {{ k3s_preflight_required_memory_mib }} MiB RAM and
+      {{ k3s_preflight_required_root_free_gib }} GiB available on /. No change was made.
+
+- name: Check whether K3s is already installed
+  ansible.builtin.stat:
+    path: /usr/local/bin/k3s
+  register: k3s_preflight_binary
+
+- name: Read listeners on required public ports
+  ansible.builtin.command:
+    argv: [ss, -lntH]
+  changed_when: false
+  register: k3s_preflight_listeners
+
+- name: Assert public ports are unused before initial K3s installation
+  ansible.builtin.assert:
+    that:
+      - >-
+        (k3s_preflight_binary.stat.exists | bool) or
+        ((k3s_preflight_listeners.stdout_lines | select('search', '(:|\\.)' ~ (item | string) ~ '$') | list | length) == 0)
+    fail_msg: "Port {{ item }} is already listening; resolve the ingress ownership conflict first."
+  loop: "{{ k3s_preflight_required_ports }}"
+
+- name: Resolve Matrix hostnames over IPv4
+  ansible.builtin.command:
+    argv: [getent, ahostsv4, "{{ item }}"]
+  changed_when: false
+  register: k3s_preflight_dns
+  loop: "{{ k3s_preflight_required_hosts }}"
+
+- name: Assert every Matrix hostname resolves to the selected VPS
+  ansible.builtin.assert:
+    that:
+      - item.stdout is search(ansible_host_ipv4 | regex_escape)
+    fail_msg: >-
+      {{ item.item }} does not resolve to {{ ansible_host_ipv4 }} over IPv4. DNS must
+      be correct before HTTP-01 certificates can be issued.
+  loop: "{{ k3s_preflight_dns.results }}"
+
+- name: Check the local backup path parent filesystem
+  ansible.builtin.command:
+    argv: [df, -P, "{{ matrix_backup_path | dirname }}"]
+  changed_when: false
+  register: k3s_preflight_backup_filesystem
+
+- name: Report read-only preflight state
+  ansible.builtin.debug:
+    msg:
+      k3s_installed: "{{ k3s_preflight_binary.stat.exists }}"
+      memory_mib: "{{ k3s_preflight_memory_mib }}"
+      root_available_bytes: "{{ k3s_preflight_root_mount.size_available | default(0) }}"
+      backup_path: "{{ matrix_backup_path }}"
+      backup_filesystem: "{{ k3s_preflight_backup_filesystem.stdout_lines[-1] }}"
diff --git a/ansible/roles/k3s_server/defaults/main.yml b/ansible/roles/k3s_server/defaults/main.yml
new file mode 100644
index 0000000..39757e1
--- /dev/null
+++ b/ansible/roles/k3s_server/defaults/main.yml
@@ -0,0 +1,11 @@
+---
+# The installer downloads the current stable channel only for an uninstalled node.
+# Normal reruns never auto-upgrade an existing K3s installation.
+k3s_install_enabled: false
+k3s_install_url: https://get.k3s.io
+k3s_binary_path: /usr/local/bin/k3s
+k3s_service_name: k3s
+k3s_server_args:
+  - --disable=servicelb
+  - --disable=metrics-server
+k3s_kubeconfig: /etc/rancher/k3s/k3s.yaml
diff --git a/ansible/roles/k3s_server/tasks/main.yml b/ansible/roles/k3s_server/tasks/main.yml
new file mode 100644
index 0000000..c22ae6b
--- /dev/null
+++ b/ansible/roles/k3s_server/tasks/main.yml
@@ -0,0 +1,48 @@
+---
+- name: Check whether K3s is installed
+  ansible.builtin.stat:
+    path: "{{ k3s_binary_path }}"
+  register: k3s_server_binary
+
+- name: Require explicit approval before first K3s installation
+  ansible.builtin.assert:
+    that:
+      - k3s_install_enabled | bool
+    fail_msg: >-
+      Refusing first-time K3s installation. Re-run only after preflight succeeds
+      with k3s_install_enabled=true.
+  when: not (k3s_server_binary.stat.exists | bool)
+
+- name: Install current stable K3s only on a new node
+  ansible.builtin.shell:
+    cmd: >-
+      curl -fsSL {{ k3s_install_url }} |
+      INSTALL_K3S_EXEC='server {{ k3s_server_args | join(' ') }}' sh -
+    creates: "{{ k3s_binary_path }}"
+  when: not (k3s_server_binary.stat.exists | bool)
+  no_log: true
+
+- name: Wait for K3s API readiness
+  ansible.builtin.command:
+    argv:
+      - "{{ k3s_binary_path }}"
+      - kubectl
+      - get
+      - node
+      - --output=jsonpath={.items[0].status.conditions[?(@.type=="Ready")].status}
+  changed_when: false
+  register: k3s_server_ready
+  retries: 30
+  delay: 5
+  until: k3s_server_ready.stdout == 'True'
+
+- name: Assert Traefik is installed and K3s optional components remain disabled
+  ansible.builtin.command:
+    argv:
+      - "{{ k3s_binary_path }}"
+      - kubectl
+      - get
+      - deployment
+      - traefik
+      - --namespace=kube-system
+  changed_when: false
diff --git a/ansible/roles/matrix_backup/defaults/main.yml b/ansible/roles/matrix_backup/defaults/main.yml
new file mode 100644
index 0000000..4e61c75
--- /dev/null
+++ b/ansible/roles/matrix_backup/defaults/main.yml
@@ -0,0 +1,13 @@
+---
+matrix_backup_enabled: false
+matrix_backup_path: /var/backups/matrix
+matrix_backup_retention_days: 7
+matrix_backup_warn_percent: 80
+matrix_backup_stop_percent: 90
+matrix_backup_script_path: /usr/local/sbin/matrix-backup
+matrix_backup_service_name: matrix-backup.service
+matrix_backup_timer_name: matrix-backup.timer
+matrix_namespace: matrix-system
+matrix_backup_postgres_pod_selector: app=matrix-postgres
+matrix_backup_media_pod_selector: app=synapse,component=synapse-media-repository
+matrix_backup_bootstrap_dir: /etc/matrix-bootstrap
diff --git a/ansible/roles/matrix_backup/handlers/main.yml b/ansible/roles/matrix_backup/handlers/main.yml
new file mode 100644
index 0000000..c28484f
--- /dev/null
+++ b/ansible/roles/matrix_backup/handlers/main.yml
@@ -0,0 +1,4 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
diff --git a/ansible/roles/matrix_backup/tasks/main.yml b/ansible/roles/matrix_backup/tasks/main.yml
new file mode 100644
index 0000000..50ef309
--- /dev/null
+++ b/ansible/roles/matrix_backup/tasks/main.yml
@@ -0,0 +1,49 @@
+---
+- name: Require explicit approval before installing backup automation
+  ansible.builtin.assert:
+    that:
+      - matrix_backup_enabled | bool
+    fail_msg: >-
+      Refusing to install Matrix backup automation until matrix_backup_enabled=true
+      is supplied deliberately.
+
+- name: Create root-only Matrix backup directory
+  ansible.builtin.file:
+    path: "{{ matrix_backup_path }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Install Matrix backup script
+  ansible.builtin.template:
+    src: matrix-backup.sh.j2
+    dest: "{{ matrix_backup_script_path }}"
+    owner: root
+    group: root
+    mode: "0700"
+
+- name: Install Matrix backup systemd service
+  ansible.builtin.template:
+    src: matrix-backup.service.j2
+    dest: "/etc/systemd/system/{{ matrix_backup_service_name }}"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Install Matrix backup systemd timer
+  ansible.builtin.template:
+    src: matrix-backup.timer.j2
+    dest: "/etc/systemd/system/{{ matrix_backup_timer_name }}"
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Reload systemd
+
+- name: Enable Matrix backup timer
+  ansible.builtin.systemd:
+    name: "{{ matrix_backup_timer_name }}"
+    enabled: true
+    state: started
+    daemon_reload: true
diff --git a/ansible/roles/matrix_backup/templates/matrix-backup.service.j2 b/ansible/roles/matrix_backup/templates/matrix-backup.service.j2
new file mode 100644
index 0000000..7897467
--- /dev/null
+++ b/ansible/roles/matrix_backup/templates/matrix-backup.service.j2
@@ -0,0 +1,16 @@
+[Unit]
+Description=Create local consistent Matrix backup
+Wants=network-online.target
+After=network-online.target k3s.service
+
+[Service]
+Type=oneshot
+User=root
+Group=root
+UMask=0077
+ExecStart={{ matrix_backup_script_path }}
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectHome=true
+ProtectSystem=full
+ReadWritePaths={{ matrix_backup_path }} /etc/matrix-bootstrap
diff --git a/ansible/roles/matrix_backup/templates/matrix-backup.sh.j2 b/ansible/roles/matrix_backup/templates/matrix-backup.sh.j2
new file mode 100644
index 0000000..1ae78a1
--- /dev/null
+++ b/ansible/roles/matrix_backup/templates/matrix-backup.sh.j2
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+set -euo pipefail
+umask 077
+
+backup_root='{{ matrix_backup_path }}'
+namespace='{{ matrix_namespace }}'
+warn_percent='{{ matrix_backup_warn_percent }}'
+stop_percent='{{ matrix_backup_stop_percent }}'
+retention_days='{{ matrix_backup_retention_days }}'
+postgres_selector='{{ matrix_backup_postgres_pod_selector }}'
+media_selector='{{ matrix_backup_media_pod_selector }}'
+bootstrap_dir='{{ matrix_backup_bootstrap_dir }}'
+
+usage=$(df -P / | awk 'NR == 2 {gsub(/%/, "", $5); print $5}')
+if (( usage >= stop_percent )); then
+  printf 'Refusing Matrix backup: root filesystem usage is %s%% (stop threshold %s%%).\n' "$usage" "$stop_percent" >&2
+  exit 2
+fi
+
+stamp=$(date -u +%Y%m%dT%H%M%SZ)
+stage="$backup_root/.staging-$stamp"
+final="$backup_root/$stamp"
+mkdir -p "$stage" "$backup_root"
+trap 'rm -rf "$stage"' EXIT
+
+if ! command -v k3s >/dev/null 2>&1; then
+  printf 'K3s is unavailable; refusing Matrix backup.\n' >&2
+  exit 3
+fi
+
+postgres_pod=$(k3s kubectl -n "$namespace" get pod -l "$postgres_selector" -o jsonpath='{.items[0].metadata.name}')
+media_pod=$(k3s kubectl -n "$namespace" get pod -l "$media_selector" -o jsonpath='{.items[0].metadata.name}')
+if [[ -z "$postgres_pod" || -z "$media_pod" ]]; then
+  printf 'Required Matrix PostgreSQL or media pod is unavailable.\n' >&2
+  exit 3
+fi
+
+for database in synapse mas; do
+  k3s kubectl -n "$namespace" exec "$postgres_pod" -- \
+    pg_dump --username=postgres --format=custom --file="/tmp/$database-$stamp.dump" "$database"
+  k3s kubectl -n "$namespace" cp \
+    "$namespace/$postgres_pod:/tmp/$database-$stamp.dump" "$stage/$database.dump"
+  k3s kubectl -n "$namespace" exec "$postgres_pod" -- rm -f "/tmp/$database-$stamp.dump"
+done
+
+k3s kubectl -n "$namespace" exec "$media_pod" -- \
+  tar --create --gzip --file="/tmp/media-$stamp.tar.gz" --directory=/data media_store
+k3s kubectl -n "$namespace" cp \
+  "$namespace/$media_pod:/tmp/media-$stamp.tar.gz" "$stage/media.tar.gz"
+k3s kubectl -n "$namespace" exec "$media_pod" -- rm -f "/tmp/media-$stamp.tar.gz"
+
+tar --create --gzip --file="$stage/bootstrap.tar.gz" --directory="$(dirname "$bootstrap_dir")" "$(basename "$bootstrap_dir")"
+sha256sum "$stage"/* > "$stage/SHA256SUMS"
+printf '{"created_at":"%s","root_usage_percent":%s,"warning_threshold_percent":%s}\n' \
+  "$stamp" "$usage" "$warn_percent" > "$stage/manifest.json"
+mv "$stage" "$final"
+trap - EXIT
+find "$backup_root" -mindepth 1 -maxdepth 1 -type d -name '20*Z' -mtime +"$retention_days" -exec rm -rf {} +
+printf 'Matrix backup created: %s\n' "$final"
diff --git a/ansible/roles/matrix_backup/templates/matrix-backup.timer.j2 b/ansible/roles/matrix_backup/templates/matrix-backup.timer.j2
new file mode 100644
index 0000000..cbeaf7f
--- /dev/null
+++ b/ansible/roles/matrix_backup/templates/matrix-backup.timer.j2
@@ -0,0 +1,10 @@
+[Unit]
+Description=Run Matrix local backup daily
+
+[Timer]
+OnCalendar=*-*-* 03:15:00 UTC
+Persistent=true
+RandomizedDelaySec=15m
+
+[Install]
+WantedBy=timers.target
diff --git a/ansible/roles/matrix_certificate_smoke/defaults/main.yml b/ansible/roles/matrix_certificate_smoke/defaults/main.yml
new file mode 100644
index 0000000..ac5d16b
--- /dev/null
+++ b/ansible/roles/matrix_certificate_smoke/defaults/main.yml
@@ -0,0 +1,11 @@
+---
+# A minimal HTTPS placeholder for the explicitly reserved RTC hostname. It also
+# validates the Traefik HTTP-01 certificate path without deploying RTC services.
+matrix_certificate_smoke_enabled: false
+matrix_certificate_smoke_namespace: matrix-system
+matrix_certificate_smoke_name: mrtc-placeholder
+matrix_certificate_smoke_host: mrtc.chans.xyz
+matrix_certificate_smoke_ingress_class: traefik
+matrix_certificate_smoke_cluster_issuer: letsencrypt-prod
+matrix_certificate_smoke_kubeconfig: /etc/rancher/k3s/k3s.yaml
+matrix_certificate_smoke_cleanup: false
diff --git a/ansible/roles/matrix_certificate_smoke/tasks/main.yml b/ansible/roles/matrix_certificate_smoke/tasks/main.yml
new file mode 100644
index 0000000..49aacdc
--- /dev/null
+++ b/ansible/roles/matrix_certificate_smoke/tasks/main.yml
@@ -0,0 +1,63 @@
+---
+- name: Require explicit approval before certificate smoke test
+  ansible.builtin.assert:
+    that:
+      - matrix_certificate_smoke_enabled | bool
+    fail_msg: >-
+      Refusing certificate smoke test changes. Re-run with
+      matrix_certificate_smoke_enabled=true after creating the test DNS record.
+
+- name: Render non-sensitive certificate smoke-test resources
+  ansible.builtin.template:
+    src: smoke.yaml.j2
+    dest: "/etc/rancher/k3s/{{ matrix_certificate_smoke_name }}.yaml"
+    owner: root
+    group: root
+    mode: "0600"
+
+- name: Apply certificate smoke-test resources
+  ansible.builtin.command:
+    argv:
+      - k3s
+      - kubectl
+      - apply
+      - -f
+      - "/etc/rancher/k3s/{{ matrix_certificate_smoke_name }}.yaml"
+  changed_when: false
+
+- name: Wait for smoke-test deployment
+  ansible.builtin.command:
+    argv:
+      - k3s
+      - kubectl
+      - rollout
+      - status
+      - "deployment/{{ matrix_certificate_smoke_name }}"
+      - "--namespace={{ matrix_certificate_smoke_namespace }}"
+      - --timeout=180s
+  changed_when: false
+
+- name: Wait for smoke-test certificate
+  ansible.builtin.command:
+    argv:
+      - k3s
+      - kubectl
+      - wait
+      - --for=condition=Ready
+      - "certificate/{{ matrix_certificate_smoke_name }}-tls"
+      - "--namespace={{ matrix_certificate_smoke_namespace }}"
+      - --timeout=10m
+  changed_when: false
+  when: not (matrix_certificate_smoke_cleanup | bool)
+
+- name: Remove smoke-test resources after verification
+  ansible.builtin.command:
+    argv:
+      - k3s
+      - kubectl
+      - delete
+      - -f
+      - "/etc/rancher/k3s/{{ matrix_certificate_smoke_name }}.yaml"
+      - --ignore-not-found=true
+  changed_when: false
+  when: matrix_certificate_smoke_cleanup | bool
diff --git a/ansible/roles/matrix_certificate_smoke/templates/smoke.yaml.j2 b/ansible/roles/matrix_certificate_smoke/templates/smoke.yaml.j2
new file mode 100644
index 0000000..c8d14ac
--- /dev/null
+++ b/ansible/roles/matrix_certificate_smoke/templates/smoke.yaml.j2
@@ -0,0 +1,82 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ matrix_certificate_smoke_name }}-content
+  namespace: {{ matrix_certificate_smoke_namespace }}
+data:
+  index.html: MatrixRTC is not available yet.
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ matrix_certificate_smoke_name }}
+  namespace: {{ matrix_certificate_smoke_namespace }}
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: {{ matrix_certificate_smoke_name }}
+  template:
+    metadata:
+      labels:
+        app: {{ matrix_certificate_smoke_name }}
+    spec:
+      containers:
+        - name: static
+          image: nginx:1.27.5-alpine
+          ports:
+            - containerPort: 80
+          volumeMounts:
+            - name: content
+              mountPath: /usr/share/nginx/html/index.html
+              subPath: index.html
+              readOnly: true
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              cpu: 50m
+              memory: 64Mi
+      volumes:
+        - name: content
+          configMap:
+            name: {{ matrix_certificate_smoke_name }}-content
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ matrix_certificate_smoke_name }}
+  namespace: {{ matrix_certificate_smoke_namespace }}
+spec:
+  selector:
+    app: {{ matrix_certificate_smoke_name }}
+  ports:
+    - name: http
+      port: 80
+      targetPort: 80
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: {{ matrix_certificate_smoke_name }}
+  namespace: {{ matrix_certificate_smoke_namespace }}
+  annotations:
+    cert-manager.io/cluster-issuer: {{ matrix_certificate_smoke_cluster_issuer }}
+spec:
+  ingressClassName: {{ matrix_certificate_smoke_ingress_class }}
+  tls:
+    - hosts:
+        - {{ matrix_certificate_smoke_host }}
+      secretName: {{ matrix_certificate_smoke_name }}-tls
+  rules:
+    - host: {{ matrix_certificate_smoke_host }}
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: {{ matrix_certificate_smoke_name }}
+                port:
+                  name: http
diff --git a/ansible/roles/matrix_cluster_base/defaults/main.yml b/ansible/roles/matrix_cluster_base/defaults/main.yml
new file mode 100644
index 0000000..fc69f8d
--- /dev/null
+++ b/ansible/roles/matrix_cluster_base/defaults/main.yml
@@ -0,0 +1,15 @@
+---
+# These resources are intentionally chart-independent and secret-free.
+matrix_namespace: matrix-system
+matrix_resource_quota:
+  requests.cpu: "4"
+  requests.memory: 6Gi
+  limits.cpu: "6"
+  limits.memory: 7Gi
+  persistentvolumeclaims: "4"
+matrix_limit_range:
+  defaultRequest:
+    cpu: 50m
+    memory: 128Mi
+  default:
+    memory: 1Gi
diff --git a/ansible/roles/matrix_cluster_base/tasks/main.yml b/ansible/roles/matrix_cluster_base/tasks/main.yml
new file mode 100644
index 0000000..99adbf1
--- /dev/null
+++ b/ansible/roles/matrix_cluster_base/tasks/main.yml
@@ -0,0 +1,24 @@
+---
+- name: Require a working K3s kubeconfig
+  ansible.builtin.stat:
+    path: /etc/rancher/k3s/k3s.yaml
+  register: matrix_cluster_kubeconfig
+
+- name: Assert K3s is ready before creating cluster resources
+  ansible.builtin.assert:
+    that:
+      - matrix_cluster_kubeconfig.stat.exists
+    fail_msg: Run k3s-server.yml successfully before matrix-cluster-base.yml.
+
+- name: Install Matrix cluster base manifest
+  ansible.builtin.template:
+    src: base-resources.yaml.j2
+    dest: /etc/rancher/k3s/matrix-cluster-base.yaml
+    owner: root
+    group: root
+    mode: "0600"
+
+- name: Apply Matrix cluster base manifest
+  ansible.builtin.command:
+    argv: [k3s, kubectl, apply, -f, /etc/rancher/k3s/matrix-cluster-base.yaml]
+  changed_when: false
diff --git a/ansible/roles/matrix_cluster_base/templates/base-resources.yaml.j2 b/ansible/roles/matrix_cluster_base/templates/base-resources.yaml.j2
new file mode 100644
index 0000000..1dcec46
--- /dev/null
+++ b/ansible/roles/matrix_cluster_base/templates/base-resources.yaml.j2
@@ -0,0 +1,26 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: {{ matrix_namespace }}
+---
+apiVersion: v1
+kind: ResourceQuota
+metadata:
+  name: matrix-system-quota
+  namespace: {{ matrix_namespace }}
+spec:
+  hard:
+{{ matrix_resource_quota | to_nice_yaml(indent=4) | indent(4, true) }}
+---
+apiVersion: v1
+kind: LimitRange
+metadata:
+  name: matrix-system-defaults
+  namespace: {{ matrix_namespace }}
+spec:
+  limits:
+    - type: Container
+      defaultRequest:
+{{ matrix_limit_range.defaultRequest | to_nice_yaml(indent=8) | indent(8, true) }}
+      default:
+{{ matrix_limit_range.default | to_nice_yaml(indent=8) | indent(8, true) }}
diff --git a/ansible/roles/matrix_cluster_base/templates/letsencrypt-prod-clusterissuer.yaml.j2 b/ansible/roles/matrix_cluster_base/templates/letsencrypt-prod-clusterissuer.yaml.j2
new file mode 100644
index 0000000..a2b25ec
--- /dev/null
+++ b/ansible/roles/matrix_cluster_base/templates/letsencrypt-prod-clusterissuer.yaml.j2
@@ -0,0 +1,13 @@
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: letsencrypt-prod
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: letsencrypt-prod-private-key
+    solvers:
+      - http01:
+          ingress:
+            class: traefik
diff --git a/ansible/roles/matrix_healthcheck/defaults/main.yml b/ansible/roles/matrix_healthcheck/defaults/main.yml
new file mode 100644
index 0000000..1c1bdeb
--- /dev/null
+++ b/ansible/roles/matrix_healthcheck/defaults/main.yml
@@ -0,0 +1,13 @@
+---
+matrix_healthcheck_enabled: false
+matrix_healthcheck_script_path: /usr/local/lib/vps-health/matrix-k3s
+matrix_healthcheck_state_path: /var/lib/vps-health/matrix-k3s.json
+matrix_healthcheck_service_name: matrix-k3s-healthcheck.service
+matrix_healthcheck_timer_name: matrix-k3s-healthcheck.timer
+matrix_healthcheck_timer_on_calendar: '*-*-* 06:00:00 UTC'
+matrix_healthcheck_timer_randomized_delay_sec: 15m
+matrix_healthcheck_namespace: matrix-system
+matrix_healthcheck_backup_path: /var/backups/matrix
+matrix_healthcheck_backup_max_age_hours: 30
+matrix_healthcheck_warn_percent: 80
+matrix_healthcheck_critical_percent: 90
diff --git a/ansible/roles/matrix_healthcheck/tasks/main.yml b/ansible/roles/matrix_healthcheck/tasks/main.yml
new file mode 100644
index 0000000..82c881a
--- /dev/null
+++ b/ansible/roles/matrix_healthcheck/tasks/main.yml
@@ -0,0 +1,53 @@
+---
+- name: Require explicit approval before installing Matrix health checks
+  ansible.builtin.assert:
+    that:
+      - matrix_healthcheck_enabled | bool
+    fail_msg: Set matrix_healthcheck_enabled=true only after the Matrix namespace and backup job exist.
+
+- name: Create Matrix healthcheck script directory
+  ansible.builtin.file:
+    path: "{{ matrix_healthcheck_script_path | dirname }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Create Matrix healthcheck state directory
+  ansible.builtin.file:
+    path: "{{ matrix_healthcheck_state_path | dirname }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0750"
+
+- name: Install Matrix K3s healthcheck script
+  ansible.builtin.template:
+    src: matrix-k3s.sh.j2
+    dest: "{{ matrix_healthcheck_script_path }}"
+    owner: root
+    group: root
+    mode: "0750"
+
+- name: Install Matrix K3s healthcheck systemd service
+  ansible.builtin.template:
+    src: matrix-k3s.service.j2
+    dest: "/etc/systemd/system/{{ matrix_healthcheck_service_name }}"
+    owner: root
+    group: root
+    mode: "0644"
+
+- name: Install Matrix K3s healthcheck systemd timer
+  ansible.builtin.template:
+    src: matrix-k3s.timer.j2
+    dest: "/etc/systemd/system/{{ matrix_healthcheck_timer_name }}"
+    owner: root
+    group: root
+    mode: "0644"
+
+- name: Enable Matrix K3s healthcheck timer
+  ansible.builtin.systemd:
+    name: "{{ matrix_healthcheck_timer_name }}"
+    enabled: true
+    state: started
+    daemon_reload: true
diff --git a/ansible/roles/matrix_healthcheck/templates/matrix-k3s.service.j2 b/ansible/roles/matrix_healthcheck/templates/matrix-k3s.service.j2
new file mode 100644
index 0000000..db62d45
--- /dev/null
+++ b/ansible/roles/matrix_healthcheck/templates/matrix-k3s.service.j2
@@ -0,0 +1,16 @@
+[Unit]
+Description=Read-only Matrix K3s health check
+Wants=network-online.target
+After=network-online.target k3s.service
+
+[Service]
+Type=oneshot
+User=root
+Group=root
+UMask=0027
+ExecStart={{ matrix_healthcheck_script_path }}
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectHome=true
+ProtectSystem=full
+ReadWritePaths={{ matrix_healthcheck_state_path | dirname }}
diff --git a/ansible/roles/matrix_healthcheck/templates/matrix-k3s.sh.j2 b/ansible/roles/matrix_healthcheck/templates/matrix-k3s.sh.j2
new file mode 100644
index 0000000..87f6aef
--- /dev/null
+++ b/ansible/roles/matrix_healthcheck/templates/matrix-k3s.sh.j2
@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+namespace='{{ matrix_healthcheck_namespace }}'
+backup_root='{{ matrix_healthcheck_backup_path }}'
+max_backup_age_seconds=$(({{ matrix_healthcheck_backup_max_age_hours }} * 3600))
+warn_percent='{{ matrix_healthcheck_warn_percent }}'
+critical_percent='{{ matrix_healthcheck_critical_percent }}'
+status=ok
+messages=()
+
+set_status() {
+  local next="$1"
+  case "$next" in
+    critical) status=critical ;;
+    warning) [[ "$status" != critical ]] && status=warning ;;
+  esac
+}
+
+if ! k3s kubectl get namespace "$namespace" >/dev/null 2>&1; then
+  set_status critical
+  messages+=("namespace $namespace is unavailable")
+else
+  unready=$(k3s kubectl -n "$namespace" get pods --no-headers 2>/dev/null | awk '$2 !~ /^[0-9]+\/[0-9]+$/ || $3 != "Running" {print $1}')
+  if [[ -n "$unready" ]]; then
+    set_status critical
+    messages+=("unready pods: ${unready//$'\n'/, }")
+  fi
+fi
+
+usage=$(df -P / | awk 'NR == 2 {gsub(/%/, "", $5); print $5}')
+if (( usage >= critical_percent )); then
+  set_status critical
+  messages+=("root filesystem usage ${usage}%")
+elif (( usage >= warn_percent )); then
+  set_status warning
+  messages+=("root filesystem usage ${usage}%")
+fi
+
+latest=$(find "$backup_root" -mindepth 1 -maxdepth 1 -type d -name '20*Z' -printf '%T@ %p\n' 2>/dev/null | sort -nr | awk 'NR == 1 {print $1, $2}')
+if [[ -z "$latest" ]]; then
+  set_status critical
+  messages+=("no Matrix backup exists")
+else
+  latest_epoch=${latest%% *}
+  latest_path=${latest#* }
+  age=$(( $(date +%s) - ${latest_epoch%.*} ))
+  if (( age > max_backup_age_seconds )); then
+    set_status critical
+    messages+=("latest Matrix backup is ${age}s old")
+  elif [[ ! -f "$latest_path/SHA256SUMS" || ! -f "$latest_path/manifest.json" ]]; then
+    set_status critical
+    messages+=("latest Matrix backup is incomplete")
+  fi
+fi
+
+printf '{"service":"matrix_k3s","status":"%s","messages":[' "$status"
+for i in "${!messages[@]}"; do
+  (( i > 0 )) && printf ','
+  printf '"%s"' "${messages[$i]//\"/\\\"}"
+done
+printf '],"root_usage_percent":%s}\n' "$usage"
diff --git a/ansible/roles/matrix_healthcheck/templates/matrix-k3s.timer.j2 b/ansible/roles/matrix_healthcheck/templates/matrix-k3s.timer.j2
new file mode 100644
index 0000000..df98f2b
--- /dev/null
+++ b/ansible/roles/matrix_healthcheck/templates/matrix-k3s.timer.j2
@@ -0,0 +1,11 @@
+[Unit]
+Description=Run Matrix K3s health check daily
+
+[Timer]
+OnCalendar={{ matrix_healthcheck_timer_on_calendar }}
+Persistent=true
+RandomizedDelaySec={{ matrix_healthcheck_timer_randomized_delay_sec }}
+Unit={{ matrix_healthcheck_service_name }}
+
+[Install]
+WantedBy=timers.target
diff --git a/ansible/roles/matrix_secret_contract/defaults/main.yml b/ansible/roles/matrix_secret_contract/defaults/main.yml
new file mode 100644
index 0000000..8a47b02
--- /dev/null
+++ b/ansible/roles/matrix_secret_contract/defaults/main.yml
@@ -0,0 +1,21 @@
+---
+matrix_namespace: matrix-system
+matrix_bootstrap_dir: /etc/matrix-bootstrap
+# This role validates only names, keys, types, ownership and modes. It never reads
+# bootstrap values or emits Kubernetes Secret contents.
+matrix_required_bootstrap_files:
+  - smtp-password
+  - synapse-db-password
+  - mas-db-password
+  - synapse-signing-key
+  - synapse-macaroon-secret
+  - synapse-form-secret
+  - mas-encryption-secret
+  - mas-signing-key
+matrix_required_kubernetes_secrets:
+  - name: matrix-synapse-master-config
+    keys: [homeserver.yaml, log.config, signing.key]
+  - name: matrix-authentication-config
+    keys: [mas-config.yaml]
+  - name: matrix-pgbouncer-userlist
+    keys: [userlist.txt]
diff --git a/ansible/roles/matrix_secret_contract/tasks/main.yml b/ansible/roles/matrix_secret_contract/tasks/main.yml
new file mode 100644
index 0000000..9a8f7cf
--- /dev/null
+++ b/ansible/roles/matrix_secret_contract/tasks/main.yml
@@ -0,0 +1,61 @@
+---
+- name: Inspect Matrix bootstrap directory without reading secrets
+  ansible.builtin.stat:
+    path: "{{ matrix_bootstrap_dir }}"
+  register: matrix_secret_bootstrap_dir
+
+- name: Assert Matrix bootstrap directory is root-only
+  ansible.builtin.assert:
+    that:
+      - matrix_secret_bootstrap_dir.stat.exists
+      - matrix_secret_bootstrap_dir.stat.isdir
+      - matrix_secret_bootstrap_dir.stat.pw_name == 'root'
+      - matrix_secret_bootstrap_dir.stat.gr_name == 'root'
+      - matrix_secret_bootstrap_dir.stat.mode == '0700'
+    fail_msg: "{{ matrix_bootstrap_dir }} must be a root:root 0700 directory."
+
+- name: Inspect required individual bootstrap secret files without reading them
+  ansible.builtin.stat:
+    path: "{{ matrix_bootstrap_dir }}/{{ item }}"
+  loop: "{{ matrix_required_bootstrap_files }}"
+  register: matrix_secret_bootstrap_files
+  no_log: true
+
+- name: Assert required bootstrap secret file permissions
+  ansible.builtin.assert:
+    that:
+      - item.stat.exists
+      - item.stat.isreg
+      - item.stat.pw_name == 'root'
+      - item.stat.gr_name == 'root'
+      - item.stat.mode == '0600'
+    fail_msg: A required root-only Matrix bootstrap file is missing or has unsafe permissions.
+  loop: "{{ matrix_secret_bootstrap_files.results }}"
+  no_log: true
+
+- name: Inspect Kubernetes Secret metadata without retrieving values
+  ansible.builtin.command:
+    argv:
+      - k3s
+      - kubectl
+      - describe
+      - secret
+      - "{{ item.name }}"
+      - --namespace={{ matrix_namespace }}
+  changed_when: false
+  loop: "{{ matrix_required_kubernetes_secrets }}"
+  register: matrix_secret_kubernetes_metadata
+  no_log: true
+
+- name: Assert required Kubernetes Secret keys exist
+  ansible.builtin.assert:
+    that:
+      - item.stdout is search('(?m)^' ~ key ~ ':')
+    fail_msg: Required Matrix Kubernetes Secret key is missing.
+  loop: "{{ matrix_secret_kubernetes_metadata.results | subelements('item.keys') }}"
+  loop_control:
+    loop_var: matrix_secret_key_check
+  vars:
+    item: "{{ matrix_secret_key_check.0 }}"
+    key: "{{ matrix_secret_key_check.1 }}"
+  no_log: true
diff --git a/ansible/roles/matrix_stack/defaults/main.yml b/ansible/roles/matrix_stack/defaults/main.yml
new file mode 100644
index 0000000..01d7161
--- /dev/null
+++ b/ansible/roles/matrix_stack/defaults/main.yml
@@ -0,0 +1,17 @@
+---
+# Matrix Stack deployment using the official ESS OCI chart
+# All sensitive values are handled by the chart's built-in initSecrets
+# or provided via pre-created Kubernetes Secrets.
+
+matrix_stack_enabled: false
+matrix_namespace: ess
+matrix_stack_chart_ref: oci://ghcr.io/element-hq/ess-helm/matrix-stack
+matrix_stack_chart_version: 26.7.2
+matrix_stack_release_name: ess
+# Root-only, non-secret values directory on the target host
+matrix_stack_values_dir: /etc/ess
+# Individual values files (must not contain secrets)
+matrix_stack_values_files:
+  - hostnames.yaml
+  - tls.yaml
+  - single-node.yaml
\ No newline at end of file
diff --git a/ansible/roles/matrix_stack/tasks/main.yml b/ansible/roles/matrix_stack/tasks/main.yml
new file mode 100644
index 0000000..d56349e
--- /dev/null
+++ b/ansible/roles/matrix_stack/tasks/main.yml
@@ -0,0 +1,97 @@
+---
+- name: Require explicit approval before deploying the Matrix stack
+  ansible.builtin.assert:
+    that:
+      - matrix_stack_enabled | bool
+    fail_msg: >-
+      Refusing Matrix deployment until matrix_stack_enabled=true is set deliberately.
+
+- name: Require Helm on the Matrix host
+  ansible.builtin.command:
+    argv: [helm, version, --short]
+  changed_when: false
+
+- name: Require non-secret values directory exists on the host
+  ansible.builtin.stat:
+    path: "{{ matrix_stack_values_dir }}"
+  register: _values_dir
+
+- name: Assert values directory exists and is root-owned
+  ansible.builtin.assert:
+    that:
+      - _values_dir.stat.exists
+      - _values_dir.stat.isdir
+      - _values_dir.stat.pw_name == 'root'
+      - _values_dir.stat.gr_name == 'root'
+      - _values_dir.stat.mode == '0700'
+    fail_msg: >-
+      Create the root-owned directory {{ matrix_stack_values_dir }}
+      with mode 0700 and place the non-secret values files in it.
+
+- name: Require each non-secret values file exists on the host
+  ansible.builtin.stat:
+    path: "{{ matrix_stack_values_dir }}/{{ item }}"
+  loop: "{{ matrix_stack_values_files }}"
+  register: _values_files
+
+- name: Assert all values files exist and are root-owned
+  ansible.builtin.assert:
+    that:
+      - item.stat.exists
+      - item.stat.isreg
+      - item.stat.pw_name == 'root'
+      - item.stat.gr_name == 'root'
+      - item.stat.mode in ['0600', '0640']
+    fail_msg: >-
+      Values file {{ item.stat.path }} must be root-owned with restricted
+      permissions (0600 or 0640) and must not contain secrets.
+  loop: "{{ _values_files.results }}"
+  loop_control:
+    label: "{{ item.stat.path | default(item.item) }}"
+
+- name: Build helm value arguments
+  ansible.builtin.set_fact:
+    _helm_values_args: >-
+      {%- for f in matrix_stack_values_files -%}
+      --values {{ matrix_stack_values_dir }}/{{ f }} {% endfor -%}
+
+- name: Render the ESS chart without applying it (dry-run validation)
+  ansible.builtin.command:
+    cmd: >-
+      helm template {{ matrix_stack_release_name }}
+      {{ matrix_stack_chart_ref }}
+      --version {{ matrix_stack_chart_version }}
+      --namespace {{ matrix_namespace }}
+      --create-namespace
+      {{ _helm_values_args }}
+  environment:
+    KUBECONFIG: "{{ k3s_kubeconfig_path | default('/etc/rancher/k3s/k3s.yaml') }}"
+  changed_when: false
+  register: _chart_render
+  # No secrets in values, but render output may contain initSecrets-generated placeholders
+
+- name: Verify the chart renders without errors
+  ansible.builtin.assert:
+    that:
+      - _chart_render.rc == 0
+      - _chart_render.stdout | length > 0
+    fail_msg: >-
+      Helm template rendering failed. Check values files for syntax errors.
+      Output: {{ _chart_render.stderr | default('(none)') }}
+
+- name: Deploy the ESS chart via Helm upgrade --install
+  ansible.builtin.command:
+    cmd: >-
+      helm upgrade --install {{ matrix_stack_release_name }}
+      {{ matrix_stack_chart_ref }}
+      --version {{ matrix_stack_chart_version }}
+      --namespace {{ matrix_namespace }}
+      --create-namespace
+      {{ _helm_values_args }}
+      --wait
+      --timeout 15m
+  environment:
+    KUBECONFIG: "{{ k3s_kubeconfig_path | default('/etc/rancher/k3s/k3s.yaml') }}"
+  changed_when: true
+  register: _helm_deploy
+  # initSecrets may generate passwords at deploy time; those stay in-cluster only
\ No newline at end of file
diff --git a/ansible/roles/restic/defaults/main.yml b/ansible/roles/restic/defaults/main.yml
new file mode 100644
index 0000000..fc15455
--- /dev/null
+++ b/ansible/roles/restic/defaults/main.yml
@@ -0,0 +1,21 @@
+---
+restic_enabled: false
+restic_binary: /usr/bin/restic
+restic_config_path: /etc/vps-restic/repository.env
+restic_state_dir: /var/lib/vps-restic
+restic_log_dir: /var/log/vps-restic
+restic_backup_on_calendar: '*-*-* 04:30:00'
+restic_check_on_calendar: 'Sun *-*-* 05:30:00'
+restic_forget_on_calendar: 'Sat *-*-* 05:30:00'
+restic_keep_daily: 7
+restic_keep_weekly: 4
+restic_keep_monthly: 6
+restic_sources:
+  vaultwarden:
+    - /opt/vaultwarden/backups
+    - /opt/vaultwarden/vw-data
+  pdns:
+    - /opt/pdns/backup
+    - /opt/pdns/auth/pdns.conf
+# Mailcow is deliberately excluded pending its official consistency and restore
+# design review. No repository/backend value is supplied by this project.
diff --git a/ansible/roles/restic/handlers/main.yml b/ansible/roles/restic/handlers/main.yml
new file mode 100644
index 0000000..5498019
--- /dev/null
+++ b/ansible/roles/restic/handlers/main.yml
@@ -0,0 +1,4 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd_service:
+    daemon_reload: true
diff --git a/ansible/roles/restic/tasks/main.yml b/ansible/roles/restic/tasks/main.yml
new file mode 100644
index 0000000..7cadfc5
--- /dev/null
+++ b/ansible/roles/restic/tasks/main.yml
@@ -0,0 +1,97 @@
+---
+- name: Require explicit Restic opt-in
+  ansible.builtin.assert:
+    that:
+      - restic_enabled | bool
+    fail_msg: >-
+      Restic is disabled by default. Set restic_enabled=true and provision the
+      repository configuration only on the target host after backend approval.
+
+- name: Validate supported Restic source profile
+  ansible.builtin.assert:
+    that:
+      - healthcheck_profile in restic_sources
+    fail_msg: "No approved Restic source profile for {{ healthcheck_profile }}."
+
+- name: Verify Restic binary exists on target
+  ansible.builtin.stat:
+    path: "{{ restic_binary }}"
+  register: restic_binary_stat
+
+- name: Require target-side Restic binary
+  ansible.builtin.assert:
+    that: restic_binary_stat.stat.exists
+    fail_msg: "Install Restic through an approved host maintenance change first."
+
+- name: Verify target-side repository configuration exists
+  ansible.builtin.stat:
+    path: "{{ restic_config_path }}"
+  register: restic_config_stat
+
+- name: Require root-only repository configuration
+  ansible.builtin.assert:
+    that:
+      - restic_config_stat.stat.exists
+      - restic_config_stat.stat.mode == '0600'
+    fail_msg: >-
+      Provision {{ restic_config_path }} directly on the host with mode 0600.
+      It must contain RESTIC_REPOSITORY, RESTIC_PASSWORD_FILE, and any backend
+      credentials; do not commit or pass them via Ansible.
+
+- name: Install Restic state and log directories
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0750"
+  loop:
+    - "{{ restic_state_dir }}"
+    - "{{ restic_log_dir }}"
+
+- name: Install Restic script directory
+  ansible.builtin.file:
+    path: /usr/local/lib/vps-restic
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install Restic scripts
+  ansible.builtin.template:
+    src: "{{ item }}.j2"
+    dest: "/usr/local/lib/vps-restic/{{ item }}"
+    owner: root
+    group: root
+    mode: "0750"
+  loop:
+    - backup
+    - check
+    - forget-prune
+
+- name: Install Restic systemd units and timers
+  ansible.builtin.template:
+    src: "{{ item }}.j2"
+    dest: "/etc/systemd/system/{{ item }}"
+    owner: root
+    group: root
+    mode: "0644"
+  loop:
+    - vps-restic-backup.service
+    - vps-restic-backup.timer
+    - vps-restic-check.service
+    - vps-restic-check.timer
+    - vps-restic-forget-prune.service
+    - vps-restic-forget-prune.timer
+  notify: Reload systemd
+
+- name: Enable Restic timers
+  ansible.builtin.systemd_service:
+    name: "{{ item }}"
+    enabled: true
+    state: started
+    daemon_reload: true
+  loop:
+    - vps-restic-backup.timer
+    - vps-restic-check.timer
+    - vps-restic-forget-prune.timer
diff --git a/ansible/roles/restic/templates/backup.j2 b/ansible/roles/restic/templates/backup.j2
new file mode 100644
index 0000000..2193494
--- /dev/null
+++ b/ansible/roles/restic/templates/backup.j2
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# Repository and password credentials are host-local in {{ restic_config_path }}.
+# shellcheck source=/dev/null
+source '{{ restic_config_path }}'
+exec '{{ restic_binary }}' backup --tag '{{ healthcheck_profile }}' --tag "$(hostname -s)" {% for source in restic_sources[healthcheck_profile] %}{{ source | quote }} {% endfor %}
diff --git a/ansible/roles/restic/templates/check.j2 b/ansible/roles/restic/templates/check.j2
new file mode 100644
index 0000000..01361d1
--- /dev/null
+++ b/ansible/roles/restic/templates/check.j2
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# shellcheck source=/dev/null
+source '{{ restic_config_path }}'
+exec '{{ restic_binary }}' check --read-data-subset=5%
diff --git a/ansible/roles/restic/templates/forget-prune.j2 b/ansible/roles/restic/templates/forget-prune.j2
new file mode 100644
index 0000000..7b4ce30
--- /dev/null
+++ b/ansible/roles/restic/templates/forget-prune.j2
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# shellcheck source=/dev/null
+source '{{ restic_config_path }}'
+exec '{{ restic_binary }}' forget --prune --keep-daily {{ restic_keep_daily }} --keep-weekly {{ restic_keep_weekly }} --keep-monthly {{ restic_keep_monthly }} --tag '{{ healthcheck_profile }}'
diff --git a/ansible/roles/restic/templates/vps-restic-backup.service.j2 b/ansible/roles/restic/templates/vps-restic-backup.service.j2
new file mode 100644
index 0000000..b9e2154
--- /dev/null
+++ b/ansible/roles/restic/templates/vps-restic-backup.service.j2
@@ -0,0 +1,11 @@
+[Unit]
+Description=Restic backup for approved {{ healthcheck_profile }} sources
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=root
+Group=root
+UMask=0077
+ExecStart=/usr/local/lib/vps-restic/backup
diff --git a/ansible/roles/restic/templates/vps-restic-backup.timer.j2 b/ansible/roles/restic/templates/vps-restic-backup.timer.j2
new file mode 100644
index 0000000..3dba83f
--- /dev/null
+++ b/ansible/roles/restic/templates/vps-restic-backup.timer.j2
@@ -0,0 +1,11 @@
+[Unit]
+Description=Daily Restic backup timer
+
+[Timer]
+OnCalendar={{ restic_backup_on_calendar }}
+Persistent=true
+RandomizedDelaySec=20m
+Unit=vps-restic-backup.service
+
+[Install]
+WantedBy=timers.target
diff --git a/ansible/roles/restic/templates/vps-restic-check.service.j2 b/ansible/roles/restic/templates/vps-restic-check.service.j2
new file mode 100644
index 0000000..113730d
--- /dev/null
+++ b/ansible/roles/restic/templates/vps-restic-check.service.j2
@@ -0,0 +1,9 @@
+[Unit]
+Description=Restic repository integrity check
+
+[Service]
+Type=oneshot
+User=root
+Group=root
+UMask=0077
+ExecStart=/usr/local/lib/vps-restic/check
diff --git a/ansible/roles/restic/templates/vps-restic-check.timer.j2 b/ansible/roles/restic/templates/vps-restic-check.timer.j2
new file mode 100644
index 0000000..a6e387c
--- /dev/null
+++ b/ansible/roles/restic/templates/vps-restic-check.timer.j2
@@ -0,0 +1,11 @@
+[Unit]
+Description=Weekly Restic integrity check timer
+
+[Timer]
+OnCalendar={{ restic_check_on_calendar }}
+Persistent=true
+RandomizedDelaySec=30m
+Unit=vps-restic-check.service
+
+[Install]
+WantedBy=timers.target
diff --git a/ansible/roles/restic/templates/vps-restic-forget-prune.service.j2 b/ansible/roles/restic/templates/vps-restic-forget-prune.service.j2
new file mode 100644
index 0000000..9d5b270
--- /dev/null
+++ b/ansible/roles/restic/templates/vps-restic-forget-prune.service.j2
@@ -0,0 +1,9 @@
+[Unit]
+Description=Restic retention and prune
+
+[Service]
+Type=oneshot
+User=root
+Group=root
+UMask=0077
+ExecStart=/usr/local/lib/vps-restic/forget-prune
diff --git a/ansible/roles/restic/templates/vps-restic-forget-prune.timer.j2 b/ansible/roles/restic/templates/vps-restic-forget-prune.timer.j2
new file mode 100644
index 0000000..1d246a1
--- /dev/null
+++ b/ansible/roles/restic/templates/vps-restic-forget-prune.timer.j2
@@ -0,0 +1,11 @@
+[Unit]
+Description=Weekly Restic retention timer
+
+[Timer]
+OnCalendar={{ restic_forget_on_calendar }}
+Persistent=true
+RandomizedDelaySec=30m
+Unit=vps-restic-forget-prune.service
+
+[Install]
+WantedBy=timers.target
diff --git a/docs/matrix-upstream.md b/docs/matrix-upstream.md
new file mode 100644
index 0000000..c779429
--- /dev/null
+++ b/docs/matrix-upstream.md
@@ -0,0 +1,109 @@
+# Matrix (Element Server Suite) Upstream References
+
+Matrix homeserver deployed via **Element Server Suite (ESS) Community** OCI chart.
+
+## Chart
+
+| Property | Value |
+|---|---|
+| Chart name | `matrix-stack` |
+| OCI registry | `oci://ghcr.io/element-hq/ess-helm/matrix-stack` |
+| Version deployed | `26.7.2` |
+| Chart source | <https://github.com/element-hq/ess-helm> |
+| Documentation | <https://element-hq.github.io/ess-helm/> |
+
+## Components
+
+| Component | Docker image | Role |
+|---|---|---|
+| Synapse | `ghcr.io/element-hq/synapse:*` | Matrix homeserver |
+| MAS | `ghcr.io/element-hq/matrix-authentication-service:*` | OIDC/OAuth authentication |
+| Element Web | `ghcr.io/element-hq/element-web:*` | Web client |
+| Element Admin | `ghcr.io/element-hq/element-admin:*` | Admin console |
+| HAProxy | `haproxy:*` | Internal routing/health |
+| PostgreSQL | `postgres:17` | Synapse + MAS database |
+
+All images should be locked via digest in production; the chart's `appVersion` may not match default image tags.
+
+## Domain configuration
+
+| Hostname | Component |
+|---|---|
+| `chans.xyz` | Matrix identity (`serverName`) + well-known discovery |
+| `synapse.chans.xyz` | Client-Server API, Federation API |
+| `chat.chans.xyz` | Element Web |
+| `account.chans.xyz` | Matrix Authentication Service (local passwords) |
+| `admin.chans.xyz` | Element Admin (MAS admin auth) |
+| `mrtc.chans.xyz` | Reserved for MatrixRTC — **not deployed** |
+
+## TLS & Ingress
+
+- Ingress controller: K3s built-in Traefik (IngressClass `traefik`)
+- Certificate issuer: cert-manager + Let's Encrypt HTTP-01 (`letsencrypt-prod` ClusterIssuer)
+- All TLS enabled: `global.ingress.tls.enabled=true`
+- Issuer annotation: `cert-manager.io/cluster-issuer: letsencrypt-prod`
+
+## Architecture
+
+```
+Internet :80/:443
+  → K3s Traefik (hostPort)
+  → ESS Ingresses (Traefik class)
+  → ClusterIP Services
+  → Pods (Synapse, MAS, Element, Admin, HAProxy)
+  → PostgreSQL (StatefulSet, 20Gi PVC)
+```
+
+## Deployment
+
+The ESS chart is deployed via Ansible (`ansible/playbooks/matrix-stack.yml`) with non-secret values at `/etc/ess/*.yaml` on the target host.
+
+**Secrets** (signing key, database passwords, SMTP password, MAS keys) are **not** in Git. They are stored:
+1. On the VPS at `/etc/matrix-bootstrap/` (root:root 0600 files)
+2. In pre-created Kubernetes Secrets in the `ess` namespace
+
+## Upgrades
+
+```bash
+# Pull latest chart version
+helm pull oci://ghcr.io/element-hq/ess-helm/matrix-stack --version <new-version>
+
+# Review changes
+helm diff upgrade ess oci://ghcr.io/element-hq/ess-helm/matrix-stack \
+  -n ess -f /etc/ess/hostnames.yaml -f /etc/ess/tls.yaml
+
+# Apply
+helm upgrade --install ess oci://ghcr.io/element-hq/ess-helm/matrix-stack \
+  -n ess -f /etc/ess/hostnames.yaml -f /etc/ess/tls.yaml --wait
+```
+
+Before upgrading:
+- Matrix backup automation is currently paused; establish and test a reviewed backup design before treating the deployment as recoverable.
+- Review chart changelog and database migration notes
+- Test in isolation if possible
+
+## Recovery
+
+When the planned backup automation is enabled, recovery will require the following from `/var/backups/matrix/`:
+
+1. PostgreSQL dumps (synapse + mas)
+2. Media store archive
+3. Bootstrap secrets archive (`/etc/matrix-bootstrap/`)
+4. Non-secret values files (`/etc/ess/*.yaml`)
+5. Non-secret playbook templates (from this repo)
+
+Steps:
+1. Deploy K3s + Traefik + cert-manager per Ansible
+2. Restore bootstrap secrets + create Kubernetes Secrets
+3. Restore PostgreSQL dumps
+4. Restore media store to PVC
+5. Deploy ESS chart with same values
+
+## References
+
+- ESS Helm docs: <https://element-hq.github.io/ess-helm/>
+- ESS Community setup: <https://github.com/element-hq/element-server-suite>
+- Synapse: <https://github.com/element-hq/synapse>
+- Matrix Authentication Service: <https://github.com/element-hq/matrix-authentication-service>
+- Matrix spec: <https://spec.matrix.org/>
+- Federation tester: <https://federationtester.matrix.org/>
diff --git a/docs/pdns-upstream.md b/docs/pdns-upstream.md
new file mode 100644
index 0000000..a2e67b1
--- /dev/null
+++ b/docs/pdns-upstream.md
@@ -0,0 +1,81 @@
+# PowerDNS Authoritative — upstream documentation
+
+Use these when changing `/opt/pdns` on [hk2](../hosts/hk2.chans.xyz.md). Prefer official docs over blog posts.
+
+## Primary references
+
+| Topic | URL |
+|-------|-----|
+| Settings reference | https://doc.powerdns.com/authoritative/settings.html |
+| Primary / secondary modes | https://doc.powerdns.com/authoritative/modes-of-operation.html |
+| Generic PostgreSQL backend | https://doc.powerdns.com/authoritative/backends/generic-postgresql.html |
+| DNSSEC ops + SOA-EDIT | https://doc.powerdns.com/authoritative/dnssec/operational.html |
+| DNSSEC advice | https://doc.powerdns.com/authoritative/dnssec/advice.html |
+| TSIG | https://doc.powerdns.com/authoritative/tsig.html |
+| HTTP API | https://doc.powerdns.com/authoritative/http-api/ |
+| Security advisories | https://doc.powerdns.com/authoritative/security-advisories/ |
+| Docker image (`pdns-auth-50`) | https://hub.docker.com/r/powerdns/pdns-auth-50 |
+
+## Settings we care about on hk2 (ns1)
+
+| Concern | Upstream guidance | Our notes (post-2026-08-01) |
+|---------|-------------------|-----------------------------|
+| Version | Track advisories; upgrade when Mandatory | Pinned **`powerdns/pdns-auth-50:5.0.6`** (clears 2026-06 + 2026-07) |
+| Primary | `primary=yes`; zones type MASTER/PRIMARY | Set; zones `windy.me` / `wsvc.info` / `chans.xyz` are Master |
+| NOTIFY | `also-notify` always notified; empty `only-notify` disables NS-based NOTIFY | `also-notify=202.91.35.141`, `only-notify=` |
+| AXFR IP ACL | `allow-axfr-ips` limits **unsigned** AXFR | Global + per-zone `ALLOW-AXFR-FROM` → ns2 only |
+| TSIG AXFR | **Any host with the TSIG key may AXFR**, bypassing IP ACL | Zones use `TSIG-ALLOW-AXFR=mykey.` — rotate/rename key; protect secret |
+| DNSSEC | `gpgsql-dnssec=yes` + keys via `pdnsutil` | CSK ECDSAP256SHA256 on all three zones |
+| SOA-EDIT | Needed so non-PDNS secondaries see signature rolls; PDNS secondaries can also compare RRSIGs | `default-soa-edit(-signed)=INCEPTION-INCREMENT` OK (ns2 is PowerDNS). Dig SOA ≠ backend serial is expected |
+| API key | Prefer `pdnsutil hash-password` (plaintext still accepted) | **Deferred hardening:** live key is weak, plaintext, reused with DB password — rotate |
+| Webserver | Prefer trusted clients only | Host publishes `127.0.0.1:8081`; `webserver-allow-from` private ranges |
+| `gpgsql-host` | Prefer IP to avoid resolve chicken/egg | `db` (Compose DNS) is fine inside the stack |
+| `domains.master` | Used for SECONDARY upstreams | Cleared on MASTER zones (was self-IP noise) |
+
+## Intended `auth/pdns.conf` shape (no secrets)
+
+Live file: `/opt/pdns/auth/pdns.conf`. Secrets come from `auth/templates.d/secrets.j2` → `/etc/powerdns/pdns.d/secrets.conf` at container start.
+
+```
+local-address=0.0.0.0
+local-port=53
+
+launch=gpgsql
+gpgsql-host=db
+gpgsql-dnssec=yes
+include-dir=/etc/powerdns/pdns.d
+
+allow-axfr-ips=202.91.35.141
+also-notify=202.91.35.141
+only-notify=
+primary=yes
+secondary=no
+
+api=yes
+webserver=yes
+webserver-address=0.0.0.0
+webserver-port=8081
+webserver-allow-from=127.0.0.1,172.16.0.0/12,10.0.0.0/8,192.168.0.0/16
+
+version-string=anonymous
+disable-syslog=yes
+loglevel=4
+
+default-soa-edit=INCEPTION-INCREMENT
+default-soa-edit-signed=INCEPTION-INCREMENT
+
+disable-axfr=no
+```
+
+Drop-in (from `.env`, never commit): `gpgsql-dbname`, `gpgsql-user`, `gpgsql-password`, `api-key`.
+
+## Deferred hardening (not done yet)
+
+1. Rotate `PDNS_API_KEY` to a long random value; store hashed via `pdnsutil hash-password` if desired; update Poweradmin `.env` / recreate `auth` + `poweradmin`.
+2. Rotate `DB_PASS` / Postgres role separately from the API key.
+3. Rotate TSIG `mykey` (or rename); confirm ns2 still AXFRs; remember TSIG bypasses `allow-axfr-ips`.
+
+## Related ops docs
+
+- Host facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md)
+- Health runbook: [runbooks/pdns-health.md](../runbooks/pdns-health.md)
diff --git a/docs/vaultwarden-upstream.md b/docs/vaultwarden-upstream.md
new file mode 100644
index 0000000..59b9fc1
--- /dev/null
+++ b/docs/vaultwarden-upstream.md
@@ -0,0 +1,53 @@
+# Vaultwarden — upstream documentation (source of truth)
+
+Use these when changing `/opt/vaultwarden` on [us2](../hosts/us2.wsvc.info.md). Prefer upstream over blog posts or Obsidian notes.
+
+## Primary references
+
+| Topic | URL |
+|-------|-----|
+| Configuration overview (methods + precedence) | https://github.com/dani-garcia/vaultwarden/wiki/Configuration-overview |
+| Env / options template (canonical variable list) | https://github.com/dani-garcia/vaultwarden/blob/main/.env.template |
+| Defaults / implementation source | https://github.com/dani-garcia/vaultwarden/blob/main/src/config.rs |
+| PostgreSQL backend + SQLite migration | https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend |
+| SMTP | https://github.com/dani-garcia/vaultwarden/wiki/SMTP-configuration |
+| Reverse proxy examples (incl. Traefik) | https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples |
+| Enabling admin page / `ADMIN_TOKEN` | https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page |
+| Disable registration / invitations | https://github.com/dani-garcia/vaultwarden/wiki/Disable-registration-of-new-users |
+| pgloader BIGINT login failure note | https://github.com/dani-garcia/vaultwarden/discussions/7134 |
+| Project wiki home | https://github.com/dani-garcia/vaultwarden/wiki |
+| Container image | https://github.com/dani-garcia/vaultwarden/pkgs/container/server |
+
+## Configuration precedence (official)
+
+From [Configuration overview](https://github.com/dani-garcia/vaultwarden/wiki/Configuration-overview):
+
+1. Compile-time defaults (`src/config.rs`)
+2. Vaultwarden `ENV_FILE` / `.env` read by the process (if used)
+3. Process environment variables (e.g. Docker Compose `env_file` + `environment`) — override (2)
+4. **`data/config.json` from `/admin` — highest precedence** (overrides env)
+
+Official stance: prefer environment variables; `config.json` is **not** the recommended long-term method. Once admin saves `config.json`, those keys stop being changeable via `.env` until removed/edited in admin or the file.
+
+## Settings we care about on us2
+
+| Concern | Upstream guidance | Our notes (post-2026-08-01 migrate) |
+|---------|-------------------|-------------------------------------|
+| Public URL | Set `DOMAIN` e.g. `https://auth.wsvc.info` (no trailing slash) | Set in compose + `config.json` |
+| SMTP | `587`+`starttls` or `465`+`force_tls` | Effective: **587/starttls** to mx2 (`465` unreachable from us2); `extra_hosts` pins A `194.163.160.244` |
+| SMTP password | Same mailbox password as mailcow | Must match across `.env`, `config.json`, `.smtp-credentials`. **Admin/`config.json` wins** — stale admin password → SMTP `535` while `.env` still works |
+| Database | `DATABASE_URL` for Postgres | Live: Postgres via compose `DATABASE_URL`; sqlite cold under `backups/sqlite-cold/` |
+| Client IP behind proxy | Traefik → prefer `X-Forwarded-For` | `ip_header=X-Forwarded-For` in `config.json` |
+| Admin | Argon2 `ADMIN_TOKEN` | On server only |
+| Signups | Invite-only on public URL | `signups_allowed=false` |
+
+## Agent checklist before edits
+
+1. Read this page + [us2 host facts](../hosts/us2.wsvc.info.md).
+2. On the server, compare `.env`, compose `environment`, and `/data/config.json` (including SMTP password fingerprint, not plaintext in chat/logs).
+3. Change the layer that actually wins (often `config.json`); if rotating the mailbox password, update **all three** secret stores.
+4. After recreating the `vaultwarden` container, restart Traefik if `auth.wsvc.info` 404s.
+5. Do not commit secrets; do not copy passwords from Obsidian into this repo.
+6. Verify SMTP with AUTH (not TCP-only) — [vaultwarden-health](../runbooks/vaultwarden-health.md) §5.
+
+Replay migration steps: [runbooks/vaultwarden-sqlite-to-postgres.md](../runbooks/vaultwarden-sqlite-to-postgres.md).
diff --git a/hosts/hk2.chans.xyz.md b/hosts/hk2.chans.xyz.md
new file mode 100644
index 0000000..c1753eb
--- /dev/null
+++ b/hosts/hk2.chans.xyz.md
@@ -0,0 +1,138 @@
+# hk2.chans.xyz
+
+| Item | Value |
+|------|--------|
+| Role | PowerDNS Authoritative (ns1) + Poweradmin / Traefik |
+| SSH | `ssh -4 windy@hk2.chans.xyz` (prefer IPv4 from WSL) |
+| IPv4 | `154.36.174.161` |
+| Hostname (OS) | `C20210714091792` (Debian 13 trixie) |
+| Also DNS | `ns1.wsvc.info` → this host |
+| Secondary NS | `ns2.wsvc.info` → `202.91.35.141` (`us1.wsvc.info`, PowerDNS Auth **5.0.5**) |
+| Install path | `/opt/pdns` |
+| Compose | `/opt/pdns/compose.yml` |
+| Env file | `/opt/pdns/.env` (secrets — **never commit**) |
+| TZ | `Asia/Shanghai` |
+
+Upstream docs / config checklist: [docs/pdns-upstream.md](../docs/pdns-upstream.md)
+
+## PowerDNS Authoritative
+
+**Status: operational** (auth + db healthy, API **5.0.6**, DNS on :53 — last config audit 2026-08-01 21:40 CST).
+
+| Item | Value |
+|------|--------|
+| Image | `powerdns/pdns-auth-50:5.0.6` (pinned) |
+| Container | `pdns-auth` |
+| Config | `/opt/pdns/auth/pdns.conf` + `auth/templates.d/secrets.j2` → `pdns.d/secrets.conf` |
+| Backend | PostgreSQL 16 (`launch=gpgsql`, DB name `pdns`) |
+| DNS ports | host `53/tcp` + `53/udp` |
+| API | host `127.0.0.1:8081` only; `webserver-allow-from` private ranges |
+| Mode | `primary=yes`; `also-notify` / AXFR ACL → `202.91.35.141`; `only-notify=` empty |
+| DNSSEC | `gpgsql-dnssec=yes`; CSK ECDSAP256SHA256 |
+| SOA-EDIT | `default-soa-edit(-signed)=INCEPTION-INCREMENT` |
+| Poweradmin | `PDNS_VERSION=50`; Traefik Host(`pdns.wsvc.info`) |
+| Advisories | 2026-06 + 2026-07 cleared by 5.0.6 |
+
+### Zones (authoritative)
+
+| Zone | Kind | DNSSEC | Zone metadata |
+|------|------|--------|---------------|
+| `windy.me` | Master | yes | `ALLOW-AXFR-FROM` + `TSIG-ALLOW-AXFR=mykey.` |
+| `wsvc.info` | Master | yes | same |
+| `chans.xyz` | Master | yes | same |
+
+NS set: `ns1.wsvc.info` / `ns2.wsvc.info`. Dig SOA may differ from backend serial (SOA-EDIT) — expected.
+
+### Intended auth config (sanitized)
+
+See full shape in [docs/pdns-upstream.md](../docs/pdns-upstream.md). Live secrets only in `.env` / generated drop-in on the server.
+
+### Stack
+
+| Container | Service | Status | Image |
+|-----------|---------|--------|-------|
+| `pdns-auth` | auth | Up (healthy) | `powerdns/pdns-auth-50:5.0.6` |
+| `pdns-db` | db | Up (healthy) | `postgres:16` |
+| `poweradmin` | poweradmin | Up (healthy) | `poweradmin/poweradmin:stable` |
+| `pdns_pgweb` | pgweb | Up | `sosedoff/pgweb:0.16.2` |
+| `pdns-backup` | backup | Up | `postgres:16` (scheduler) |
+| `powerdns-admin` | *(orphan)* | Exited | legacy PDA UI — not in active compose |
+
+### Network model
+
+| Network | Purpose |
+|---------|---------|
+| `backend` | Internal (`internal: true`) — DB + service API |
+| `edge` | Host-published DNS `:53` and loopback API `:8081` |
+| `frontend` (`traefik`) | External Traefik network for web UIs |
+
+### Public / management URLs
+
+| URL | Backend | Notes |
+|-----|---------|-------|
+| https://pdns.wsvc.info | `poweradmin` via Traefik | DNS admin UI |
+| https://pgweb.wsvc.info | `pgweb` via Traefik | DB browser (HTTP basic) |
+| DNS `ns1.wsvc.info:53` | `pdns-auth` | Public authoritative |
+
+## Config audit (vs official docs)
+
+**Functional primary/secondary config: OK** after 5.0.6 upgrade + `only-notify=` / `domains.master` cleanup.
+
+| Item | Verdict |
+|------|---------|
+| `primary` + MASTER zones + gpgsql DNSSEC | OK |
+| `also-notify` + empty `only-notify` | OK (docs-recommended pattern) |
+| API bind / host port publish | OK |
+| SOA-EDIT with PowerDNS secondary | OK |
+| API key / DB password hygiene | **Deferred** — weak, plaintext, reused |
+| TSIG `mykey` | **Deferred** — any holder can AXFR (bypasses IP ACL) |
+
+## Backup
+
+| Item | Value |
+|------|--------|
+| Service | `pdns-backup` |
+| Schedule | `CRON_SCHEDULE=0 3 * * *` (03:00 Asia/Shanghai) |
+| Retention | `RETENTION_DAYS=7`, `MAX_BACKUPS=3` |
+| Output | `/opt/pdns/backup/pdns_*.sql.gz` (+ optional `roles_*.sql`) |
+| Scheduler | `scripts/backup-scheduler.sh` (daily `M H * * *` only) |
+| Manual | `cd /opt/pdns && ./scripts/manual-backup.sh` |
+
+**Quirk:** `backend` is internal — backup must not use Alpine + runtime `apk`/`crond`. Uses `postgres:16` + `backup-scheduler.sh` (fixed 2026-08-01).
+
+## Other software on this host (stubs)
+
+`/opt/traefik`, `/opt/adguard`, `/opt/remark42`, `/opt/rustdesk`, `/opt/nginx-manager`, …
+
+## Ops / runbooks
+
+- [pdns-health](../runbooks/pdns-health.md)
+- [pdns-upstream docs](../docs/pdns-upstream.md)
+
+```bash
+ssh -4 windy@hk2.chans.xyz
+cd /opt/pdns && docker compose ps -a
+docker compose logs -f auth poweradmin backup
+./scripts/manual-backup.sh
+# after pdns.conf / secrets.j2 / auth env changes:
+docker compose up -d --force-recreate auth
+```
+
+```bash
+dig @154.36.174.161 NS windy.me +short
+dig @154.36.174.161 SOA wsvc.info +short
+# compare secondary:
+dig @202.91.35.141 SOA wsvc.info +short
+```
+
+On-server docs: `/opt/pdns/README.md`, `CHANGELOG.md`.
+
+## Verified
+
+Last checked: **2026-08-01 21:40 CST** — operational; docs audit recorded.
+
+- Auth **5.0.6** healthy; no Mandatory security banner
+- Zones Master + DNSSEC; SOA in sync with ns2 (served serial)
+- `only-notify=` + `also-notify=202.91.35.141`; MASTER `domains.master` cleared
+- https://pdns.wsvc.info → **302**; https://pgweb.wsvc.info → **401**
+- Hardening backlog: API/DB credential rotation + TSIG rotate (see upstream doc)
diff --git a/hosts/mx2.windy.me.md b/hosts/mx2.windy.me.md
new file mode 100644
index 0000000..102dca1
--- /dev/null
+++ b/hosts/mx2.windy.me.md
@@ -0,0 +1,54 @@
+# mx2.windy.me
+
+| Item | Value |
+|------|--------|
+| Role | Mail server (mailcow) |
+| SSH | `ssh -4 windy@mx2.windy.me` (pubkey; root denied; prefer IPv4 from WSL) |
+| Install path | `/opt/mail` |
+| Compose project | `cow` (`COMPOSE_PROJECT_NAME=cow`) |
+| Compose file | `/opt/mail/docker-compose.yml` |
+| Config | `/opt/mail/mailcow.conf` (`.env` → symlink) |
+| Hostname | `MAILCOW_HOSTNAME=mx2.windy.me` |
+| TZ | `Asia/Shanghai` |
+| IPv4 | `194.163.160.244` |
+| IPv6 | `2a02:c207:2284:8258::1` |
+| Docker network | `IPV4_NETWORK=172.22.1` |
+| UI | https://mx2.windy.me |
+
+## Ports (host-published)
+
+`25`, `465`, `587` (SMTP), `110`/`995` (POP), `143`/`993` (IMAP), `4190` (Sieve), `80`/`443` (UI)
+
+## Domain & mailboxes
+
+- Domain: `windy.me` (primary on this host)
+- MX for `windy.me`: `mx2.windy.me` (prio 20), `mx.windy.me` (prio 30)
+- Mailboxes (examples): `zhiqiang@windy.me`, `claw@windy.me`, `hermes@windy.me`
+- ACME / Let’s Encrypt enabled (`SKIP_LETS_ENCRYPT=n`); cert SAN includes `mx2`, `autoconfig`, `autodiscover`
+- PTR (v4/v6) → `mx2.windy.me`
+
+## Ops notes
+
+- Manage with: `cd /opt/mail && docker compose …`
+- `SKIP_CLAMD=y` may still leave `clamd` running after updates; recreate if you need it truly skipped
+- Host has ~8G RAM, no swap; watch memory under load
+- Unrelated stopped containers may exist (`zc-*` monitoring stack) — not part of mailcow
+
+## Client access (SMTP / IMAP)
+
+See [mailcow-smtp-client](../runbooks/mailcow-smtp-client.md).
+
+- Send: `mx2.windy.me:587` STARTTLS (or `:465` SSL), auth = full email + mailbox password
+- IMAP: `mx2.windy.me:993` · POP: `:995`
+- Webmail: https://mx2.windy.me
+
+## Runbooks
+
+- [mailcow-health](../runbooks/mailcow-health.md)
+- [mailcow-update](../runbooks/mailcow-update.md)
+- [mailcow-smtp-client](../runbooks/mailcow-smtp-client.md)
+- Script: [`../scripts/check-mx2.sh`](../scripts/check-mx2.sh)
+
+## Verified
+
+Last checked: 2026-08-01 — `./scripts/check-mx2.sh` exit 0 (compose up, watchdog 100%, empty queue, TLS LE OK, SMTP Postcow, A/AAAA/PTR/MX/SPF OK).
diff --git a/hosts/synapse.chans.xyz.md b/hosts/synapse.chans.xyz.md
new file mode 100644
index 0000000..1322b8f
--- /dev/null
+++ b/hosts/synapse.chans.xyz.md
@@ -0,0 +1,100 @@
+# synapse.chans.xyz — Matrix homeserver (ESS)
+
+## Role
+
+Matrix homeserver running **Element Server Suite (ESS) Community**: Synapse + Matrix Authentication Service (MAS) + Element Web + Element Admin.
+
+## SSH
+
+```
+ssh -4 windy@synapse.chans.xyz
+```
+
+- User: `windy` (sudo via Ansible)
+- IPv4: `169.58.86.13`
+- OS: Ubuntu 26.04
+
+## Stack
+
+| Layer | Detail |
+|-------|--------|
+| Runtime | K3s v1.36.2+k3s1 (single-node) |
+| Ingress | K3s built-in Traefik (hostPort `:80` / `:443`) |
+| TLS | cert-manager v1.19.3 + `letsencrypt-prod` ClusterIssuer (HTTP-01) |
+| DB | ESS embedded PostgreSQL 17 (PVC 20Gi, local-path) |
+| Cache | ESS embedded Redis (PVC 2Gi) |
+| Chart | `oci://ghcr.io/element-hq/ess-helm/matrix-stack`, version `26.7.2` |
+
+### Matrix service endpoints
+
+| URL | Component |
+|-----|-----------|
+| https://chat.chans.xyz | Element Web |
+| https://synapse.chans.xyz | Synapse Client-Server / Federation API |
+| https://account.chans.xyz | Matrix Authentication Service |
+| https://admin.chans.xyz | Element Admin console |
+| https://chans.xyz/.well-known/matrix/server | Federation discovery → `synapse.chans.xyz:443` |
+| https://chans.xyz/.well-known/matrix/client | Client discovery → `https://synapse.chans.xyz` |
+| `mrtc.chans.xyz` | Reserved for MatrixRTC (not deployed) |
+
+## Public ports
+
+| Port | Protocol | Service |
+|------|----------|---------|
+| 22/tcp | SSH | OpenSSH |
+| 80/tcp | HTTP | Let's Encrypt HTTP-01 / Traefik redirect |
+| 443/tcp | HTTPS | Matrix Client-Server, Federation, Element, MAS, Admin |
+
+All other ports internal only (no K3s API, no database, no Redis exposed).
+
+## Namespace layout
+
+- `ess` — all ESS workloads (Synapse, MAS, Element, Postgres, Redis, HAProxy)
+- `matrix-system` — cluster base resources (ResourceQuota, LimitRange, mrtc-placeholder)
+- `cert-manager` — cert-manager
+
+## Local backup
+
+| Item | Detail |
+|------|--------|
+| Path | `/var/backups/matrix` (root:root, 0700) |
+| Schedule | **Paused** — `matrix-backup.timer` is disabled pending a reviewed backup design |
+| Retention | 7 days |
+| Disk warning | 80% (healthcheck), 90% (backup stops) |
+| Content | Planned: PostgreSQL `synapse` + `mas` logical dumps, media store archive, `/etc/matrix-bootstrap` |
+| Status | **Not operational** — no current Matrix backup or recovery tier |
+
+## Health checks
+
+Daily via `matrix-k3s-healthcheck.timer`, outputs to `/var/lib/vps-health/latest.json`. Checks:
+
+- K3s API / namespace health
+- ESS pod restart counts
+- Backup freshness
+- Root disk usage
+- Certificate expiry
+- External HTTPS endpoints
+
+## Bootstrap secrets
+
+Path: `/etc/matrix-bootstrap/` (root:root, 0700) — individual files per secret key (0600). SMTP password, DB passwords, signing key etc. Included in local backup.
+
+## Identity
+
+- **Server name**: `chans.xyz` (user IDs: `@user:chans.xyz`)
+- **Deployment type**: New instance (no recovery from previous homeserver)
+- **Public registration**: Disabled
+- **Authentication**: MAS local passwords
+
+## Ansible
+
+Inventory group: `matrix_production` (but NOT in `managed` which expects Docker Compose).
+
+Playbooks in `ansible/playbooks/matrix-*.yml`.
+
+## Known limitations
+
+- MatrixRTC / Element Call / LiveKit / Coturn not deployed (`mrtc.chans.xyz` reserved only)
+- SMTP email not yet configured (requires manual bootstrap + Helm upgrade)
+- No off-site Restic backup
+- Single-node K3s (no HA for control plane)
diff --git a/hosts/us2.wsvc.info.md b/hosts/us2.wsvc.info.md
new file mode 100644
index 0000000..519cfbb
--- /dev/null
+++ b/hosts/us2.wsvc.info.md
@@ -0,0 +1,91 @@
+# us2.wsvc.info
+
+| Item | Value |
+|------|--------|
+| Role | Multi-service VPS (Vaultwarden, Traefik, Soft Serve, …) |
+| SSH | `ssh -4 windy@us2.wsvc.info` (prefer IPv4 from WSL) |
+| IPv4 | `193.9.44.165` |
+| Also DNS | `auth.wsvc.info` → this host; `repo.windy.me` → this host (Soft Serve) |
+| Public HTTPS | Traefik on `:80` / `:443` (`/opt/traefik`) |
+
+## Vaultwarden (Bitwarden-compatible)
+
+**Status: operational** (Postgres live, HTTPS 200, healthy containers, SMTP AUTH OK — last probe 2026-08-01 18:55 CST).
+
+Upstream docs: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
+
+| Item | Value |
+|------|--------|
+| Install path | `/opt/vaultwarden` |
+| Compose | `/opt/vaultwarden/docker-compose.yml` |
+| Env file | `/opt/vaultwarden/.env` |
+| Admin overrides | `/opt/vaultwarden/vw-data/config.json` (**wins over env**) |
+| Public URL / `DOMAIN` | `https://auth.wsvc.info` |
+| Image | `vaultwarden/server:1.37.1` (pinned) |
+| Live DB | **Postgres 16** (`vw-db` / service `pg`) via compose `DATABASE_URL` |
+| Data (probe) | users=1, ciphers=1327 |
+| Cold SQLite | `backups/sqlite-cold/db.sqlite3.pre-pg-20260801` (not used live) |
+| Pre-migrate backup | `backups/pre-pg-migrate-20260801_161204/` |
+| Data dir | `./vw-data` → `/data` (attachments, rsa keys, `config.json`) |
+| DB backups | `vaultwarden-backup` → `pg_dump` daily 02:00 (`scripts/backup.sh`); latest e.g. `backups/vaultwarden_2026-08-01_16-23-03.sql.gz` |
+| Reverse proxy | Traefik (`vw-net`) → Host(`auth.wsvc.info`), LE (`auth.wsvc.info`, valid → 2026-10-08) |
+| SMTP | `mx2.windy.me:587` STARTTLS (`extra_hosts` → 194.163.160.244); from `zhiqiang@windy.me` |
+| SMTP secrets | Keep **`.env`**, **`vw-data/config.json`**, and **`.smtp-credentials`** in sync — `config.json` wins; drift breaks mail |
+| IP header | `X-Forwarded-For` |
+| Signups | disabled (`signups_allowed=false`); invitations allowed |
+| pgweb | compose profile `debug` (stopped by default) |
+| Secrets | `.env`, `.admin-token`, `.smtp-credentials` — **never commit** |
+
+### Official documentation
+
+| Topic | Link |
+|-------|------|
+| Config overview | https://github.com/dani-garcia/vaultwarden/wiki/Configuration-overview |
+| Postgres / migration | https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend |
+| SMTP | https://github.com/dani-garcia/vaultwarden/wiki/SMTP-configuration |
+| Proxy / Traefik | https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples |
+| Full index | [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md) |
+
+### Stack
+
+| Container | Status |
+|-----------|--------|
+| `vaultwarden` | Up (healthy), `vaultwarden/server:1.37.1` |
+| `vw-db` | Up (healthy) — **live** Postgres |
+| `vaultwarden-backup` | Up (`pg_dump`) |
+| `vaultwarden-pgweb` | Exited (profile `debug`) |
+
+Effective `config.json`: domain `https://auth.wsvc.info`, SMTP 587/starttls to `mx2.windy.me` as `zhiqiang@windy.me`, `ip_header=X-Forwarded-For`. Password matched to `.env` (fixed 2026-08-01).
+
+### Client use
+
+- Server URL: `https://auth.wsvc.info`
+- Admin: `https://auth.wsvc.info/admin`
+
+### Ops / runbooks
+
+- [vaultwarden-health](../runbooks/vaultwarden-health.md)
+- [vaultwarden-sqlite-to-postgres](../runbooks/vaultwarden-sqlite-to-postgres.md)
+- [vaultwarden-upstream docs](../docs/vaultwarden-upstream.md)
+
+```bash
+ssh -4 windy@us2.wsvc.info
+cd /opt/vaultwarden && docker compose ps -a
+# after recreating vaultwarden, if auth 404s:
+docker restart traefik
+```
+
+## Other software on this host (stubs)
+
+`/opt/traefik`, `soft-serve`, `authelia`, `dendrite`/`conduit`, `mastodon`, `rustdesk`, `zitadel`, etc.
+
+## Verified
+
+Last checked: **2026-08-01 18:55 CST** — operational.
+
+- `vaultwarden` + `vw-db` healthy; `DATABASE_URL` → `pg:5432/vaultwarden`
+- `https://auth.wsvc.info/` **200**, `/admin` **200**, `/api/config` OK (`disableUserRegistration: true`)
+- Identity wrong-password → **400** business error (DB readable, not 500)
+- SMTP: container → `mx2:587` OK; STARTTLS cert CN=`mx2.windy.me`; **AUTH OK** with effective `config.json` password (synced with `.env` / `.smtp-credentials`)
+- LE cert CN=`auth.wsvc.info`
+- PG counts: users=1, ciphers=1327
diff --git a/inventory/hosts.md b/inventory/hosts.md
new file mode 100644
index 0000000..75cd4ec
--- /dev/null
+++ b/inventory/hosts.md
@@ -0,0 +1,29 @@
+# Host inventory
+
+| Host | Role | SSH | IPv4 | Status | Facts |
+|------|------|-----|------|--------|-------|
+| mx2.windy.me | mailcow (primary MX prio 20) | `ssh -4 windy@mx2.windy.me` | 194.163.160.244 | active | [hosts/mx2.windy.me.md](../hosts/mx2.windy.me.md) |
+| us2.wsvc.info | Vaultwarden/Postgres (+ Traefik, Soft Serve, …) | `ssh -4 windy@us2.wsvc.info` | 193.9.44.165 | active | [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md) |
+| mx.windy.me | mail (secondary MX prio 30) | TBD | see AAAA/A | stub | — |
+| repo.windy.me | Soft Serve git (on us2) | `ssh -p 2222 windy@repo.windy.me` | 193.9.44.165 | stub | see us2 |
+| auth.wsvc.info | Vaultwarden public hostname | — (HTTPS) | → us2 | active | see us2 |
+| us1.wsvc.info | PowerDNS secondary (ns2 host) | TBD | 202.91.35.141 | stub | Auth 5.0.5; see hk2 |
+| us4.wsvc.info | Traefik / services | TBD | TBD | stub | — |
+| hk2.chans.xyz | PowerDNS auth (ns1) | `ssh -4 windy@hk2.chans.xyz` | 154.36.174.161 | active | [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) |
+| ns1.wsvc.info | PowerDNS public NS name | — (DNS) | → hk2 `154.36.174.161` | active | see hk2 |
+| ns2.wsvc.info | Secondary NS (AXFR/NOTIFY peer) | — (DNS) | → us1 `202.91.35.141` | active | see hk2 |
+| pdns.wsvc.info | Poweradmin UI | — (HTTPS) | → hk2 | active | see hk2 |
+| pgweb.wsvc.info | PowerDNS Postgres UI | — (HTTPS) | → hk2 | active | see hk2 |
+| **synapse.chans.xyz** | Matrix homeserver (ESS: Synapse + MAS + Element) | `ssh -4 windy@synapse.chans.xyz` | `169.58.86.13` | **active** | [hosts/synapse.chans.xyz.md](../hosts/synapse.chans.xyz.md) |
+
+`status: stub` = known to exist; fill `hosts/<name>.md` when next touched.
+
+### Matrix services (synapse.chans.xyz)
+
+| URL | Service | Notes |
+|-----|---------|-------|
+| https://chat.chans.xyz | Element Web | Matrix web client |
+| https://synapse.chans.xyz | Synapse API | Client-Server + Federation API |
+| https://account.chans.xyz | MAS | Matrix Authentication Service (local passwords) |
+| https://admin.chans.xyz | Element Admin | Admin console (MAS admin auth) |
+| `mrtc.chans.xyz` | MatrixRTC | **Reserved** – not deployed |
\ No newline at end of file
diff --git a/plans/2026-08-02-vps-maintenance-ansible-restic-refactor-v1.md b/plans/2026-08-02-vps-maintenance-ansible-restic-refactor-v1.md
new file mode 100644
index 0000000..3388f66
--- /dev/null
+++ b/plans/2026-08-02-vps-maintenance-ansible-restic-refactor-v1.md
@@ -0,0 +1,69 @@
+# VPS 維護專案重構：Ansible + systemd timers + Restic
+
+## Objective
+
+將現有 VPS 維運知識庫漸進重構為一個以 Ansible 管理一致性、以 systemd timers 執行每日唯讀檢查、以 Restic 建立加密異機備份的個人 SRE 維運專案。第一期僅納管 mx2、us2、hk2；自動化只能觀測與告警，任何更新、重啟、修復、DNS 或機密變更都必須人工確認。
+
+## Confirmed Decisions
+
+- [x] 定位為個人 SRE／維運手冊庫，而非全自動修復平台。
+- [x] 第一階段完整納管 mx2、us2、hk2；mx、us1、us4 待完成盤點與緊急存取驗證後才納管。
+- [x] 每日自動做唯讀檢查與 Email 通知；每週人工審查、每月人工維護、每季復原演練與權限盤點。
+- [x] 以 Email 作為唯一告警與摘要通道。
+- [x] 採用本機快速復原加密異機副本的 3-2-1 最小可行策略。
+- [x] 真實機密不進 Git；repo 只保存去敏設定結構與操作／驗證紀錄。
+- [x] 日常存取使用非 root SSH key；禁止 root/password SSH，維護 provider/recovery console 緊急存取。
+- [x] 採用 Ansible + systemd timers + Restic 作為技術基線。
+
+## Implementation Plan
+
+- [x] **Status: Done — 建立去敏 Ansible inventory。** 為 mx2、us2、hk2 建立主機與服務群組，將非機密主機資料與現有人類可讀的 `inventory/hosts.md` 對應；理由是提供可審查的機器可讀編排層，且不取代既有事實來源。
+- [x] **Status: Done — 實作 audit-only Ansible playbook。** 只收集連線、OS、磁碟、systemd、Docker Compose 和依賴命令狀態，不做設定或服務改動；理由是先確認控制面與真實現況。
+- [x] **Status: Done — 定義跨服務健康檢查結果合約。** 統一去敏結構化結果、日誌位置、嚴重度及 exit code，讓人工執行、systemd 和 Ansible audit 可共同消費；理由是避免各服務告警語意漂移。
+- [x] **Status: Done — 保留並模組化 Mailcow 健康檢查。** 持續檢查 Compose、watchdog、queue、listeners、HTTP/HTTPS、TLS、SMTP、DNS/PTR/MX/SPF；理由是現有腳本已覆蓋關鍵郵件服務面向。
+- [x] **Status: Done — 實作 Vaultwarden 唯讀健康檢查。** 驗證 Compose/Postgres、HTTPS、有效設定、SMTP AUTH、備份新鮮度及機密指紋一致性，禁止輸出機密；理由是 `config.json` 優先於 `.env`，且 SMTP 漂移已有已知風險。
+- [x] **Status: Done — 實作 PowerDNS 唯讀健康檢查。** 驗證容器、版本與安全公告、ns1/ns2 served SOA、API、Web UI、備份新鮮度與必要設定；理由是同時覆蓋公開 DNS 與 secondary 同步。
+- [x] **Status: Done — 使用 Ansible 部署 systemd healthcheck service/timer。** 每台主機本機執行每日檢查，具 persistent 排程、權限、logrotate 和一致錯誤處理；理由是控制端離線不應阻止巡檢。
+- [x] **Status: Done — 部署 Email 告警與每日摘要。** Critical/unknown 立即通知、健康狀態每日摘要、重複失敗抑制；SMTP 真實認證僅在各主機受限路徑保存；理由是已選定 Email 為唯一通知通道。
+- [!] **Status: Blocked — 決定 Restic 異機 repository 與存取隔離。** 選擇具加密傳輸、權限隔離與可承受保留需求的 S3/B2/SFTP 或等價目的地；理由是無異機 repository 即無法滿足已確認的 3-2-1 政策。阻塞：本地實作依要求未虛構 backend、repository 或 credentials。
+- [x] **Status: Done (templates gated) — 部署 Vaultwarden 與 PowerDNS 的 Restic 備份。** 將既有一致性資料庫 dump、必要資料目錄及復原元資料加密同步，並加入 snapshot 年齡和 `restic check` 驗證；理由是兩者已有本機備份但缺少異機保護。啟用被 repository 決策和主機端受限設定檔阻擋。
+- [!] **Status: Blocked — 完成 Mailcow 備份設計審查。** 官方流程已確認：以 `/opt/mail/helper-scripts/backup_and_restore.sh backup all`（或經明確核准的元件集）先產生一致性備份，再由 Restic 同步該輸出；不得直接複製 Docker volumes。阻塞：仍需決定本機備份位置、保留期、排程及異機 Restic repository。
+- [x] **Status: Done (templates gated) — 部署 Restic backup、retention、forget/prune 與 check timers。** 與現有資料庫 dump 時段錯開，所有 repository 認證與密碼留在伺服器端受限檔案；理由是保持備份可用與成本可控。啟用被 repository 決策和主機端受限設定檔阻擋。
+- [!] **Status: Blocked — 補齊服務級復原 runbook。** 阻塞：需在選定 Restic repository、建立實際 snapshot 並確認各服務的實際備份輸出後，才能編寫可驗證的 restore 前置條件、順序與 rollback；禁止臆造 backend/credentials 或未驗證還原命令。
+- [!] **Status: Blocked — 執行隔離式復原演練。** 阻塞：尚未選定／配置 Restic 異機 repository，亦尚無可供還原的異機 snapshot；演練不得對現有生產資料執行。
+- [x] **Status: Done — 實作受控 common baseline 與 maintenance playbook。** 僅在 audit 穩定後納入 SSH 稽核、時間同步、logrotate、更新預覽與人工確認的維護操作；理由是避免工具導入期間同時改變服務狀態。
+- [x] **Status: Done (separate change defined) — 將 PowerDNS API key、DB password 與 TSIG 輪替列為分離變更。** 已明確保持 API key、DB password 與 TSIG 為三項獨立、需人工核准的變更，並要求逐步驗證 Auth、Poweradmin、AXFR/NOTIFY、DNSSEC 與 ns2 同步；未輪替任何機密。
+- [!] **Status: Blocked — 盤點 mx、us1、us4。** 阻塞：inventory 僅記錄 `TBD` SSH／角色或缺少完整事實，尚未提供可驗證的存取方式與 provider/recovery console 資訊；不得猜測或嘗試未授權存取。
+
+## Verification Criteria
+
+- [ ] 三台 active 主機能被 Ansible 正確解析及執行 audit，且 audit 不產生主機變更。
+- [ ] 三台主機每日產生去敏健康結果；服務、容量、TLS、公開端點和備份新鮮度異常均能被偵測。
+- [ ] 正常狀態寄送 Email 摘要；受控測試異常能觸發一次可讀且不含機密的 Email 告警。
+- [ ] 每個核心服務至少有一份加密、異機、可列出且通過完整性檢查的 Restic snapshot。
+- [ ] 至少完成一次隔離式實際還原，並確認其結果符合或明確量化偏離服務 RPO/RTO。
+- [ ] 自動化排程不包含更新、重啟、修復、秘密輪替或 DNS 變更。
+- [ ] Repo、Ansible vars、產出日誌及 Email 均不包含任何真實機密。
+
+## Potential Risks and Mitigations
+
+1. **Mailcow 備份未保持資料一致性。**  
+   Mitigation: 先依官方方式完成備份範圍與還原設計，再上線排程；以隔離還原驗證作為完成門檻。
+
+2. **Ansible 設定錯誤改動生產系統。**  
+   Mitigation: 先導入 audit-only；變更 playbook 必須使用 dry-run/diff、明確 tag 與人工確認，且先限制單一 host。
+
+3. **機密出現在 Git、Ansible output 或 Email。**  
+   Mitigation: 真實值只存主機受限檔案；去敏輸出、no-log 機制和測試用機密掃描為強制要求。
+
+4. **Timer 成功但公開服務實際不可用。**  
+   Mitigation: 同時執行本機 Compose/DB 檢查與外部 HTTP、SMTP、TLS、DNS 檢查。
+
+5. **PowerDNS 密鑰輪替中斷 DNS 管理或 zone transfer。**  
+   Mitigation: API key、DB password、TSIG 分批處理；每步都驗證 primary/secondary 同步、DNSSEC 和管理 UI。
+
+## Alternative Approaches
+
+1. **純 shell + systemd，不使用 Ansible**：初期較快，但設定與排程容易跨主機漂移，且難以審查或擴充；不建議作為長期方案。
+2. **Ansible + 集中式監控平台**：可增加儀表板與趨勢，但平台本身需額外維護；等 Email 摘要無法滿足需求時再評估。
+3. **立即導入 SOPS + age**：可使加密設定進 Git，但需先完成密鑰生命週期與緊急存取設計；建議列為後續階段。
diff --git a/runbooks/mailcow-health.md b/runbooks/mailcow-health.md
new file mode 100644
index 0000000..eca5956
--- /dev/null
+++ b/runbooks/mailcow-health.md
@@ -0,0 +1,72 @@
+# Runbook: mailcow health (mx2)
+
+Target: [mx2.windy.me](../hosts/mx2.windy.me.md)  
+Path: `/opt/mail`  
+Prefer: `./scripts/check-mx2.sh` from this repo, or the steps below over SSH.
+
+## 1. Containers
+
+```bash
+ssh windy@mx2.windy.me 'cd /opt/mail && docker compose ps -a'
+```
+
+Expect mailcow (`cow-*`) services **Up**. Note any Exit/Restarting.
+
+## 2. Watchdog
+
+```bash
+ssh windy@mx2.windy.me 'cd /opt/mail && docker compose logs --tail=40 watchdog-mailcow'
+```
+
+Expect recent lines at **100%** for Nginx, Postfix, Dovecot, MySQL, Redis, SOGo, Rspamd, ACME, etc.
+
+## 3. Queue
+
+```bash
+ssh windy@mx2.windy.me 'cd /opt/mail && docker compose exec -T postfix-mailcow postqueue -p'
+```
+
+Healthy: empty / `Mail queue is empty`.
+
+## 4. Local listeners
+
+```bash
+ssh windy@mx2.windy.me 'ss -tlnp | grep -E ":(25|465|587|143|993|995|80|443|4190)\\s"'
+```
+
+## 5. External checks (from laptop / agent host)
+
+```bash
+curl -sI --max-time 10 http://mx2.windy.me/ | head -5
+curl -skI --max-time 10 https://mx2.windy.me/ | head -15
+echo | openssl s_client -connect mx2.windy.me:443 -servername mx2.windy.me 2>/dev/null \
+  | openssl x509 -noout -subject -issuer -dates
+timeout 8 bash -c 'exec 3<>/dev/tcp/mx2.windy.me/25; echo -e "EHLO test.local\r\nQUIT\r\n" >&3; cat <&3' | head -20
+```
+
+Expect: HTTP 301 → HTTPS; HTTPS 200; LE cert for `mx2.windy.me`; SMTP banner `Postcow`.
+
+## 6. DNS summary
+
+Use a public resolver if the local stub is flaky (WSL `127.0.0.53`):
+
+```bash
+dig @1.1.1.1 +short mx2.windy.me A
+dig @1.1.1.1 +short mx2.windy.me AAAA
+dig @1.1.1.1 +short -x 194.163.160.244
+dig @1.1.1.1 +short windy.me MX
+dig @1.1.1.1 +short windy.me TXT | grep -i spf
+```
+
+## Scheduled local check
+
+The sanitized Ansible health profile is `mailcow` (`ansible/playbooks/healthchecks.yml`). It preserves the existing standalone `scripts/check-mx2.sh` as the richer operator check, while the server-local timer emits a sanitized result at `/var/lib/vps-health/latest.json`. Neither check changes Mailcow configuration or service state.
+
+## Pass criteria
+
+- Compose stack up; watchdog ~100%
+- Queue empty (or explain backlog)
+- TLS valid; mail ports reachable externally
+- A/AAAA/PTR/MX consistent with [host facts](../hosts/mx2.windy.me.md)
+
+Update the **Verified** line on the host page after a successful check.
diff --git a/runbooks/mailcow-smtp-client.md b/runbooks/mailcow-smtp-client.md
new file mode 100644
index 0000000..cd41098
--- /dev/null
+++ b/runbooks/mailcow-smtp-client.md
@@ -0,0 +1,54 @@
+# Runbook: use mailcow SMTP / IMAP (client)
+
+Target: [mx2.windy.me](../hosts/mx2.windy.me.md)  
+Prerequisite: a mailbox on `windy.me` (password from mailcow UI, not the admin account unless it is that mailbox).
+
+Clients should use **submission** to send. Do **not** use port 25 as a desktop/app outbound port.
+
+## Send (SMTP)
+
+| Item | Value |
+|------|--------|
+| Server | `mx2.windy.me` |
+| Port | **587** (STARTTLS) — preferred |
+| Alt port | **465** (SSL/TLS) |
+| Username | Full address, e.g. `zhiqiang@windy.me` |
+| Password | Mailbox password |
+| Auth | Required (LOGIN / PLAIN) |
+
+Match encryption to the port: 587 → STARTTLS; 465 → SSL/TLS (not STARTTLS).
+
+From address should be that mailbox or an allowed alias on the same domain.
+
+## Receive (IMAP / POP)
+
+| Protocol | Server | Port |
+|----------|--------|------|
+| IMAPS | `mx2.windy.me` | **993** |
+| POP3S | `mx2.windy.me` | **995** |
+
+Username = full email address; same mailbox password.
+
+## Webmail
+
+- UI / SOGo: https://mx2.windy.me  
+- Log in with the same mailbox credentials.
+
+## CLI smoke test (optional)
+
+```bash
+swaks --to someone@example.com \
+  --from YOU@windy.me \
+  --server mx2.windy.me --port 587 -tls \
+  --auth LOGIN --auth-user YOU@windy.me --auth-password 'MAILBOX_PASSWORD'
+```
+
+Do not commit or paste real passwords into this repo.
+
+## Auth failures — quick checks
+
+- Username is the **full** email, not a local part alone
+- Password is the **mailbox** password (not unrelated host/root secrets)
+- Port/TLS mode mismatch (587 vs 465)
+- Account active in mailcow; not rate-limited / fail2banned after bad attempts
+- Apps that store SMTP in their own config (e.g. Vaultwarden `config.json`) may keep a **stale** password even when `.env` is correct — verify AUTH against the effective config ([vaultwarden-health](vaultwarden-health.md) §5)
diff --git a/runbooks/mailcow-update.md b/runbooks/mailcow-update.md
new file mode 100644
index 0000000..af4382a
--- /dev/null
+++ b/runbooks/mailcow-update.md
@@ -0,0 +1,43 @@
+# Runbook: mailcow update (mx2)
+
+Target: [mx2.windy.me](../hosts/mx2.windy.me.md)  
+Path: `/opt/mail`  
+**Confirm with the user before running an update.**
+
+## Before
+
+1. Run [mailcow-health](mailcow-health.md) (or `./scripts/check-mx2.sh`). Record baseline.
+2. Note config quirks in `mailcow.conf`, especially:
+   - `SKIP_CLAMD=y` — after update, `clamd` may still be running until services are recreated per mailcow docs.
+   - `SKIP_LETS_ENCRYPT`, ports, hostname — should stay `mx2.windy.me`.
+3. Disk/memory: `df -h /` and `free -h` on the host.
+
+## Update
+
+```bash
+ssh -t windy@mx2.windy.me 'cd /opt/mail && ./update.sh'
+```
+
+- Prefer interactive SSH (`-t`) so prompts work.
+- Do not pass secrets into the chat log; do not commit `mailcow.conf`.
+
+If update asks to stop/recreate containers, allow it unless the user said otherwise.
+
+## After
+
+1. Re-run health checks ([mailcow-health](mailcow-health.md)).
+2. Confirm UI: `https://mx2.windy.me`
+3. Confirm SMTP banner and queue empty.
+4. If `SKIP_CLAMD=y` but `cow-clamd-mailcow-1` is still Up and that is unwanted:
+
+   ```bash
+   ssh windy@mx2.windy.me 'cd /opt/mail && docker compose up -d --force-recreate'
+   ```
+
+   Only after user OK; recreate briefly interrupts mail.
+
+5. Update **Verified** on [hosts/mx2.windy.me.md](../hosts/mx2.windy.me.md) with date and outcome.
+
+## Rollback
+
+Mailcow updates are image/git based; there is no one-click rollback in this runbook. If something fails, capture `docker compose ps` and `docker compose logs --tail=100` for the failing service and stop before further changes.
diff --git a/runbooks/matrix-health.md b/runbooks/matrix-health.md
new file mode 100644
index 0000000..a128ffa
--- /dev/null
+++ b/runbooks/matrix-health.md
@@ -0,0 +1,85 @@
+# Matrix Health Check
+
+Monitor the Matrix homeserver running on `synapse.chans.xyz` (ESS chart `26.7.2`, K3s node).
+
+## Basic checks
+
+```bash
+# Overall status
+ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods,svc,ingress,certificate -o wide'
+
+# Pod restarts
+ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods -o wide | grep -E "CrashLoop|Error|Evicted" || echo "No unhealthy pods"'
+
+# Disk
+ssh -4 windy@synapse.chans.xyz 'df -hP /'
+```
+
+## Certificate verification
+
+```bash
+ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get certificate'
+```
+
+All certificates should show `READY=True`. Domains:
+- `chans.xyz`
+- `synapse.chans.xyz`
+- `chat.chans.xyz`
+- `account.chans.xyz`
+- `admin.chans.xyz`
+- `mrtc.chans.xyz` (placeholder only)
+
+## External endpoint verification
+
+```bash
+# Client API
+curl -4 -s https://synapse.chans.xyz/_matrix/client/versions | python3 -m json.tool | head -5
+
+# Well-known server discovery
+curl -4 -s https://chans.xyz/.well-known/matrix/server
+
+# Well-known client discovery
+curl -4 -s https://chans.xyz/.well-known/matrix/client
+
+# Federation tester (manual)
+open https://federationtester.matrix.org/#chans.xyz
+```
+
+## Local health-check automation
+
+The node has a systemd health-check timer that runs daily:
+
+```bash
+ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-k3s-healthcheck.timer'
+ssh -4 windy@synapse.chans.xyz 'cat /var/lib/vps-health/latest.json 2>/dev/null || echo "No data yet"'
+```
+
+Checks cover:
+- K3s node Ready
+- Pods in `ess` namespace (no CrashLoopBackOff / Evicted)
+- Root disk usage (warning ≥80%, critical ≥90%)
+- Backup freshness
+
+## Backup status
+
+```bash
+ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-backup.timer'
+ssh -4 windy@synapse.chans.xyz 'ls -la /var/backups/matrix/'
+```
+
+Backup automation is currently paused. `/var/backups/matrix/` is retained for a future, reviewed design and must not be treated as a recovery source.
+- PostgreSQL dumps (synapse + mas)
+- Media archive
+- Bootstrap secrets archive
+- Backup manifest with SHA-256
+- Retention: 7 days
+
+## Common pitfalls
+
+| Symptom | Likely cause |
+|---|---|
+| `/_matrix/client/versions` fails | Traefik ingress not routing to Synapse |
+| HTTPS returns non-LE cert | cert-manager `letsencrypt-prod` not ready |
+| Well-known returns 404/redirect | Root `chans.xyz` ingress missing or misconfigured |
+| 502 Bad Gateway | Synapse pod restarting or DB down |
+| SMTP emails not sent | MAS SMTP config incomplete; TCP reachable but AUTH failing — see `runbooks/vaultwarden-health.md` |
diff --git a/runbooks/pdns-health.md b/runbooks/pdns-health.md
new file mode 100644
index 0000000..c74e1fa
--- /dev/null
+++ b/runbooks/pdns-health.md
@@ -0,0 +1,86 @@
+# PowerDNS health (hk2)
+
+Read-only checks for the `/opt/pdns` stack on **hk2.chans.xyz** (`ns1.wsvc.info`).
+
+Facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) · Upstream: [docs/pdns-upstream.md](../docs/pdns-upstream.md)
+
+## Quick remote
+
+```bash
+ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose ps -a'
+```
+
+Expect `auth`, `db`, `poweradmin` healthy; `backup` Up; `pgweb` Up. Ignore stopped orphan `powerdns-admin` unless cleaning orphans.
+
+## Version / security poll
+
+Expect Authoritative **5.0.6** (or newer). Startup logs should **not** show a Mandatory `Security Update` banner. If they do, check https://doc.powerdns.com/authoritative/security-advisories/ and bump the `powerdns/pdns-auth-50` image pin in `/opt/pdns/compose.yml`.
+
+```bash
+ssh -4 windy@hk2.chans.xyz 'docker logs pdns-auth 2>&1 | grep -i "Security Update" | tail -5'
+ssh -4 windy@hk2.chans.xyz 'docker inspect pdns-auth --format "{{.Config.Image}}"'
+```
+
+## DNS (from laptop / WSL)
+
+Prefer querying auth/secondary IPs directly (local stub / public resolvers may be flaky or cached):
+
+```bash
+NS1=154.36.174.161
+NS2=202.91.35.141
+for z in windy.me wsvc.info chans.xyz; do
+  echo "== $z =="
+  dig @$NS1 +short NS "$z"
+  dig @$NS1 +short SOA "$z"
+  dig @$NS2 +short SOA "$z"
+done
+dig @$NS1 +short A ns1.wsvc.info   # expect 154.36.174.161
+dig @$NS1 +short A ns2.wsvc.info   # expect 202.91.35.141
+```
+
+Served SOA serial can differ from the backend/API serial when `SOA-EDIT=INCEPTION-INCREMENT` is active — compare **ns1 vs ns2 served SOA**, not dig vs `pdnsutil`/API.
+
+## API (on host)
+
+```bash
+ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose exec -T auth \
+  python3 -c "import json,os,urllib.request; print(json.load(urllib.request.urlopen(urllib.request.Request(\"http://127.0.0.1:8081/api/v1/servers/localhost\", headers={\"X-API-Key\": os.environ[\"PDNS_API_KEY\"]})))[\"version\")"'
+```
+
+Or use host loopback with the key from `.env` (do not paste the key into chat/logs).
+
+## Web UIs
+
+```bash
+# If local stub DNS is flaky, pin SNI to the auth IP:
+curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pdns.wsvc.info:443:154.36.174.161 https://pdns.wsvc.info/   # 302 → login
+curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pgweb.wsvc.info:443:154.36.174.161 https://pgweb.wsvc.info/  # 401 without basic auth
+```
+
+## Backup
+
+```bash
+ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && ./scripts/manual-backup.sh && ls -lt backup/*.sql.gz | head'
+ssh -4 windy@hk2.chans.xyz 'tail -30 /opt/pdns/backup/backup.log'
+```
+
+Scheduler should log `scheduler started … (daily 03:00)`. If you see `can't execute 'bash'` or Alpine `apk` errors, the backup image/entrypoint was reverted — see [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md).
+
+## Config sanity (docs-aligned)
+
+```bash
+ssh -4 windy@hk2.chans.xyz 'grep -E "^(primary|also-notify|only-notify|allow-axfr-ips|gpgsql-dnssec|default-soa-edit)" /opt/pdns/auth/pdns.conf'
+```
+
+Expect: `primary=yes`, `also-notify=202.91.35.141`, `only-notify=` empty, `gpgsql-dnssec=yes`, SOA-EDIT `INCEPTION-INCREMENT`.
+
+## Scheduled local check
+
+The sanitized Ansible health profile is `pdns` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses the API key only inside the PowerDNS container. It does not modify PowerDNS, DNS records, or secrets.
+
+
+## After config changes
+
+- `auth/pdns.conf`, `auth/templates.d/secrets.j2`, or auth-related `.env` → `docker compose up -d --force-recreate auth`
+- Poweradmin env / Traefik labels → `docker compose up -d --no-deps --force-recreate poweradmin` (restart Traefik if UI 404s)
+- backup schedule / retention env → `docker compose up -d --force-recreate --no-deps backup`
diff --git a/runbooks/vaultwarden-health.md b/runbooks/vaultwarden-health.md
new file mode 100644
index 0000000..1139d5d
--- /dev/null
+++ b/runbooks/vaultwarden-health.md
@@ -0,0 +1,96 @@
+# Runbook: Vaultwarden health (us2)
+
+Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)  
+Path: `/opt/vaultwarden`  
+URL: https://auth.wsvc.info/  
+Upstream: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
+
+## 1. Containers
+
+```bash
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose ps -a'
+```
+
+Expect `vaultwarden` + `vw-db` **Up (healthy)**. `pgweb` should be stopped unless started with `--profile debug`.
+
+## 2. Live DB is Postgres
+
+```bash
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden sh -c "printenv DATABASE_URL" | sed -E "s#://[^@]+@#://REDACTED@#"'
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T pg psql -U vaultwarden -d vaultwarden -c "SELECT COUNT(*) AS users FROM users; SELECT COUNT(*) AS ciphers FROM ciphers;"'
+```
+
+## 3. Effective config (config.json wins)
+
+```bash
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden cat /data/config.json' \
+  | python3 -c "import sys,json;d=json.load(sys.stdin);print({k:d.get(k) for k in ['domain','smtp_host','smtp_port','smtp_security','smtp_username','smtp_from','ip_header','signups_allowed']});print('smtp_password_set',bool(d.get('smtp_password')))"
+```
+
+Expect SMTP **mx2.windy.me** / **587** / **starttls**, `ip_header` **X-Forwarded-For**.
+
+Also confirm `.env` and `config.json` SMTP passwords match (lengths/hashes only — never print secrets):
+
+```bash
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
+import json, re, pathlib, hashlib, subprocess
+env = pathlib.Path(".env").read_text()
+def grab(k):
+    m = re.search(rf"^{re.escape(k)}=(.*)$", env, re.M)
+    return m.group(1).strip().strip("\"'\''") if m else None
+cfg = json.loads(subprocess.check_output(
+    ["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
+ep, cp = grab("SMTP_PASSWORD"), cfg.get("smtp_password") or ""
+def fp(s): return {"len": len(s), "sha256_8": hashlib.sha256(s.encode()).hexdigest()[:8]}
+print("env_pass", fp(ep or ""), "cfg_pass", fp(cp), "match", ep == cp)
+EOF'
+```
+
+If they diverge, **auth uses `config.json`** — sync password there (or via `/admin`) and update `.smtp-credentials`.
+
+## 4. External HTTPS
+
+```bash
+curl -4 -sS -I --max-time 15 https://auth.wsvc.info/ | head -15
+```
+
+If **404** right after recreating VW: `ssh -4 windy@us2.wsvc.info 'docker restart traefik'` then retry.
+
+## 5. SMTP reachability + AUTH
+
+TCP (expect `587:0`; `465` often times out from us2):
+
+```bash
+ssh -4 windy@us2.wsvc.info 'docker compose -f /opt/vaultwarden/docker-compose.yml exec -T vaultwarden sh -c "timeout 5 bash -c \"cat </dev/null >/dev/tcp/mx2.windy.me/587\"; echo 587:\$?"'
+```
+
+AUTH with the **effective** `config.json` password (run on us2; do not print the password):
+
+```bash
+ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
+import json, smtplib, ssl, subprocess
+cfg = json.loads(subprocess.check_output(
+    ["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
+host, port = cfg["smtp_host"], int(cfg["smtp_port"])
+user, pw = cfg["smtp_username"], cfg["smtp_password"]
+ctx = ssl.create_default_context()
+with smtplib.SMTP(host, port, timeout=15) as s:
+    s.ehlo(); s.starttls(context=ctx); s.ehlo(); s.login(user, pw)
+print("AUTH_OK", host, port, user)
+EOF'
+```
+
+Expect `AUTH_OK`. `535` usually means stale password in `config.json` (see step 3).
+
+## Scheduled local check
+
+The sanitized Ansible health profile is `vaultwarden` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses only server-side credentials for the SMTP AUTH probe. It does not modify Vaultwarden, Traefik, SMTP, or secrets.
+
+
+## Pass criteria
+
+- Healthy compose; `DATABASE_URL` points at `pg`
+- HTTPS 200 + `/admin` 200; PG counts sane
+- Effective SMTP 587/starttls; `.env` ↔ `config.json` password match; **AUTH_OK**
+- Wrong-password login returns 400 business error (not 500)
+- Update **Verified** on [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md)
diff --git a/runbooks/vaultwarden-sqlite-to-postgres.md b/runbooks/vaultwarden-sqlite-to-postgres.md
new file mode 100644
index 0000000..3d7074a
--- /dev/null
+++ b/runbooks/vaultwarden-sqlite-to-postgres.md
@@ -0,0 +1,37 @@
+# Runbook: Vaultwarden SQLite → Postgres (us2)
+
+Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)  
+Upstream: [Using the PostgreSQL Backend](https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend) · [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
+
+**Done on 2026-08-01** for this host. Keep this as the replay checklist.
+
+## Summary of what we did
+
+1. Backup `vw-data` + `.env` → `backups/pre-pg-migrate-<timestamp>/`
+2. Baseline counts from SQLite (then: users=1, ciphers=1327)
+3. Drop/recreate empty Postgres DB; pin image `vaultwarden/server:1.37.1`
+4. Set compose `DATABASE_URL=postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:${DB_PORT}/${DB_NAME}`
+5. Start VW once → diesel schema → stop
+6. `pgloader` **data only** (exclude `__diesel_schema_migrations`) with CAST bigint→integer
+7. Remaining intentional bigints: `attachments.file_size`, `twofactor.last_used`, `twofactor_duo_ctx.exp`
+8. Align `config.json` SMTP **587/starttls**, `ip_header=X-Forwarded-For`, `DOMAIN` without trailing slash
+9. Rotate `DB_PASS` / `PGWEB_PASS`; `pgweb` behind compose `profiles: [debug]`
+10. Move live sqlite file to `backups/sqlite-cold/` (cold only)
+11. After VW recreate, **restart Traefik** so Docker provider picks up the router
+
+## Critical pitfalls
+
+- `config.json` overrides `.env` (admin wins). Fix SMTP there or via `/admin`.
+- SMTP password drift: if `.env` AUTH works but VW mail fails with `535`, sync `vw-data/config.json` (+ `.smtp-credentials`) to the mailbox password — do not assume env alone is enough.
+- From this network, **mx2:465 times out**; use **587 + starttls**.
+- After recreating `vaultwarden`, Traefik may 404 until `docker restart traefik`.
+- pgloader needs a **writable** directory for the sqlite file (CANTOPEN if mount is awkward).
+- After pgloader, wrong integer types → silent “wrong password”; see [Discussion #7134](https://github.com/dani-garcia/vaultwarden/discussions/7134).
+- Do not commit `.env` or load files with passwords into git.
+
+## Rollback
+
+1. `docker compose stop vaultwarden`
+2. Restore `vw-data` from `backups/pre-pg-migrate-*` (including sqlite)
+3. Remove `DATABASE_URL` from compose `environment`
+4. `docker compose up -d vaultwarden` && `docker restart traefik`
diff --git a/scripts/check-mx2.sh b/scripts/check-mx2.sh
new file mode 100755
index 0000000..6252f4c
--- /dev/null
+++ b/scripts/check-mx2.sh
@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+# Read-only health check for mx2.windy.me mailcow.
+# Usage: ./scripts/check-mx2.sh
+# Env:
+#   MX2_HOST       default mx2.windy.me
+#   MX2_SSH_USER   default windy
+#   MX2_ALLOW_IPV6 set to 1 to allow IPv6 (default forces IPv4; WSL often lacks v6 route)
+set -euo pipefail
+
+HOST="${MX2_HOST:-mx2.windy.me}"
+SSH_USER="${MX2_SSH_USER:-windy}"
+
+# Default -4: host has AAAA but many clients (WSL) have no IPv6 route → "Network is unreachable"
+SSH_AF_OPTS=(-4)
+CURL_AF_OPTS=(-4)
+OPENSSL_AF_OPTS=(-4)
+if [[ "${MX2_ALLOW_IPV6:-0}" == "1" ]]; then
+  SSH_AF_OPTS=()
+  CURL_AF_OPTS=()
+  OPENSSL_AF_OPTS=()
+fi
+
+SSH=(ssh "${SSH_AF_OPTS[@]}" -o BatchMode=yes -o ConnectTimeout=10 "${SSH_USER}@${HOST}")
+# Prefer public resolvers: local stub (127.0.0.53) is flaky from some WSL setups
+DIG_OPTS=(+time=3 +tries=2 +short)
+DIG_SERVERS=(1.1.1.1 8.8.8.8)
+FAILURES=0
+
+section() { printf '\n===== %s =====\n' "$1"; }
+
+fail() {
+  printf 'FAIL: %s\n' "$1" >&2
+  FAILURES=$((FAILURES + 1))
+}
+
+# Strip dig commentary (timeouts are printed on stdout as ";; ...")
+dig_answers_only() {
+  grep -v '^;;' | grep -v '^$' || true
+}
+
+# dig_short <name> [type] — tries each public resolver; skips timeout noise
+dig_short() {
+  local name="$1"
+  local typ="${2:-A}"
+  local server out
+  for server in "${DIG_SERVERS[@]}"; do
+    out="$(dig @"${server}" "${DIG_OPTS[@]}" "$name" "$typ" 2>/dev/null | dig_answers_only)" || out=""
+    if [[ -n "$out" ]]; then
+      printf '%s\n' "$out"
+      return 0
+    fi
+  done
+  return 1
+}
+
+# dig_ptr <ip>
+dig_ptr() {
+  local ip="$1"
+  local server out
+  for server in "${DIG_SERVERS[@]}"; do
+    out="$(dig @"${server}" "${DIG_OPTS[@]}" -x "$ip" 2>/dev/null | dig_answers_only)" || out=""
+    if [[ -n "$out" ]]; then
+      printf '%s\n' "$out"
+      return 0
+    fi
+  done
+  return 1
+}
+
+check_ptrs() {
+  local ip ptr
+  while read -r ip; do
+    [[ -z "$ip" ]] && continue
+    ptr="$(dig_ptr "$ip" || true)"
+    printf 'PTR %s -> %s\n' "$ip" "${ptr:-<none>}"
+    grep -Fiq "${HOST}" <<<"$ptr" || fail "PTR for ${ip} does not mention ${HOST}"
+  done
+}
+
+need_cmd() {
+  command -v "$1" >/dev/null 2>&1 || {
+    echo "missing required command: $1" >&2
+    exit 127
+  }
+}
+
+need_cmd ssh
+need_cmd curl
+need_cmd openssl
+need_cmd dig
+need_cmd timeout
+
+section "SSH + compose ps"
+PS_OUT="$("${SSH[@]}" 'cd /opt/mail && docker compose ps -a')" || {
+  fail "ssh or docker compose ps"
+  echo "Cannot continue without SSH/compose." >&2
+  exit 1
+}
+printf '%s\n' "$PS_OUT"
+if grep -qiE 'Exited|Restarting|[[:space:]]Dead[[:space:]]' <<<"$PS_OUT"; then
+  fail "compose has Exited/Restarting/Dead containers"
+fi
+for svc in nginx-mailcow postfix-mailcow dovecot-mailcow mysql-mailcow; do
+  grep -Fq "$svc" <<<"$PS_OUT" || fail "missing service ${svc}"
+done
+
+section "Watchdog (tail)"
+if ! "${SSH[@]}" 'cd /opt/mail && docker compose logs --tail=40 watchdog-mailcow'; then
+  fail "watchdog logs"
+fi
+
+section "Mail queue"
+QUEUE_OUT="$("${SSH[@]}" 'cd /opt/mail && docker compose exec -T postfix-mailcow postqueue -p' 2>&1)" || {
+  fail "postqueue"
+  QUEUE_OUT=""
+}
+printf '%s\n' "$QUEUE_OUT"
+if [[ -n "$QUEUE_OUT" ]] && ! grep -Fiq 'Mail queue is empty' <<<"$QUEUE_OUT"; then
+  fail "mail queue not empty"
+fi
+
+section "Listeners"
+LISTEN_OUT="$("${SSH[@]}" 'ss -tlnp 2>/dev/null | grep -E ":(25|465|587|110|143|993|995|80|443|4190)[[:space:]]" || true')"
+printf '%s\n' "$LISTEN_OUT"
+for p in 25 465 587 110 143 993 995 80 443 4190; do
+  if ! grep -qE ":${p}[[:space:]]" <<<"$LISTEN_OUT"; then
+    fail "port ${p} not listening on host"
+  fi
+done
+
+section "HTTP / HTTPS"
+HTTP_OUT="$(curl "${CURL_AF_OPTS[@]}" -sS -I --max-time 10 "http://${HOST}/" 2>&1 | head -8)" || true
+printf '%s\n' "$HTTP_OUT"
+echo "---"
+# No -k: surface TLS trust problems
+HTTPS_OUT="$(curl "${CURL_AF_OPTS[@]}" -sS -I --max-time 10 "https://${HOST}/" 2>&1 | head -15)" || true
+printf '%s\n' "$HTTPS_OUT"
+grep -qE '^HTTP/[0-9.]+ 301' <<<"$HTTP_OUT" || fail "HTTP did not redirect (expect 301)"
+grep -qE '^HTTP/[0-9.]+ 200' <<<"$HTTPS_OUT" || fail "HTTPS did not return 200"
+
+section "TLS cert (443)"
+CERT_OUT="$(
+  timeout 15 openssl s_client "${OPENSSL_AF_OPTS[@]}" -connect "${HOST}:443" -servername "${HOST}" </dev/null 2>/dev/null \
+    | openssl x509 -noout -subject -issuer -dates 2>/dev/null
+)" || true
+printf '%s\n' "$CERT_OUT"
+if ! grep -Fq "CN=${HOST}" <<<"$CERT_OUT" && ! grep -Fq "CN = ${HOST}" <<<"$CERT_OUT"; then
+  fail "TLS subject missing ${HOST}"
+fi
+grep -Fiq "Let's Encrypt" <<<"$CERT_OUT" || fail "TLS issuer not Let's Encrypt"
+
+section "SMTP banner"
+# Connect via A record so /dev/tcp does not pick unreachable AAAA first
+SMTP_IP="$(dig_short "${HOST}" A 2>/dev/null | head -n1 || true)"
+if [[ -z "$SMTP_IP" ]]; then
+  fail "cannot resolve A for SMTP check"
+  SMTP_OUT=""
+else
+  SMTP_OUT="$(
+    timeout 8 bash -c "exec 3<>/dev/tcp/${SMTP_IP}/25; printf 'EHLO test.local\r\nQUIT\r\n' >&3; cat <&3" 2>/dev/null \
+      | head -20
+  )" || true
+fi
+printf 'connect %s:25\n' "${SMTP_IP:-?}"
+printf '%s\n' "$SMTP_OUT"
+grep -qiE 'Postcow|ESMTP' <<<"$SMTP_OUT" || fail "SMTP banner unexpected"
+
+section "DNS"
+A_REC="$(dig_short "${HOST}" A || true)"
+AAAA_REC="$(dig_short "${HOST}" AAAA || true)"
+printf 'A:    %s\n' "${A_REC:-<none>}"
+printf 'AAAA: %s\n' "${AAAA_REC:-<none>}"
+[[ -n "$A_REC" ]] || fail "missing A record"
+[[ -n "$AAAA_REC" ]] || fail "missing AAAA record"
+
+check_ptrs <<<"$A_REC"
+check_ptrs <<<"$AAAA_REC"
+
+MX_REC="$(dig_short windy.me MX || true)"
+printf 'MX:\n%s\n' "${MX_REC:-<none>}"
+grep -Fiq "${HOST}" <<<"$MX_REC" || fail "windy.me MX missing ${HOST}"
+
+SPF="$(dig_short windy.me TXT || true)"
+printf 'SPF:\n'
+if ! grep -Fi 'v=spf1' <<<"$SPF"; then
+  fail "no SPF TXT on windy.me"
+fi
+
+section "Done"
+if [[ "$FAILURES" -gt 0 ]]; then
+  echo "Health check finished with ${FAILURES} failure(s)."
+  exit 1
+fi
+echo "Health check OK. Update hosts/mx2.windy.me.md Verified line if desired."
+exit 0