Initial VPS operations handbook
This commit is contained in:
windyboy
+22
@@ -0,0 +1,22 @@
|
|||||||
|
# Local secrets and service configuration (never version these).
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
!.env.example
|
||||||
|
mailcow.conf
|
||||||
|
.smtp-credentials
|
||||||
|
.admin-token
|
||||||
|
*.pem
|
||||||
|
*.key
|
||||||
|
id_*
|
||||||
|
!*.pub
|
||||||
|
|
||||||
|
# Ansible runtime output.
|
||||||
|
*.retry
|
||||||
|
.ansible/
|
||||||
|
facts/
|
||||||
|
|
||||||
|
# Editor and operating-system files.
|
||||||
|
.DS_Store
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*~
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
# VPS Agent Ops Hub
|
||||||
|
|
||||||
|
This repo is the **agent ops handbook + fact source** for maintaining personal VPS hosts. Prefer verifying live state over assuming docs are complete.
|
||||||
|
|
||||||
|
Also readable as `agent.md` (symlink → this file).
|
||||||
|
|
||||||
|
## How to work
|
||||||
|
|
||||||
|
1. Read [`inventory/hosts.md`](inventory/hosts.md) for the machine list.
|
||||||
|
2. Open the matching [`hosts/<name>.md`](hosts/) for SSH, roles, paths, and quirks.
|
||||||
|
3. For common tasks, follow a runbook under [`runbooks/`](runbooks/).
|
||||||
|
4. Prefer read-only checks first; change only after confirming current state.
|
||||||
|
5. Default access: `ssh -4 windy@<host>` (pubkey). Prefer **IPv4** from this WSL client (AAAA often exists but IPv6 route does not).
|
||||||
|
|
||||||
|
## Active hosts (quick map)
|
||||||
|
|
||||||
|
| Host | Role | SSH | Facts |
|
||||||
|
|------|------|-----|--------|
|
||||||
|
| **mx2.windy.me** | mailcow (`/opt/mail`, project `cow`) | `ssh -4 windy@mx2.windy.me` | [hosts/mx2.windy.me.md](hosts/mx2.windy.me.md) |
|
||||||
|
| **us2.wsvc.info** | Vaultwarden + Traefik (+ Soft Serve, …) | `ssh -4 windy@us2.wsvc.info` | [hosts/us2.wsvc.info.md](hosts/us2.wsvc.info.md) |
|
||||||
|
| **hk2.chans.xyz** | PowerDNS auth ns1 (`/opt/pdns`) | `ssh -4 windy@hk2.chans.xyz` | [hosts/hk2.chans.xyz.md](hosts/hk2.chans.xyz.md) |
|
||||||
|
|
||||||
|
### Public services
|
||||||
|
|
||||||
|
| URL / endpoint | Backend host | Notes |
|
||||||
|
|----------------|--------------|--------|
|
||||||
|
| https://mx2.windy.me | mx2 | mailcow UI / SOGo |
|
||||||
|
| SMTP `mx2.windy.me:587` (STARTTLS) or `:465` | mx2 | client submission; full email + mailbox password — [runbook](runbooks/mailcow-smtp-client.md) |
|
||||||
|
| IMAP `mx2.windy.me:993` | mx2 | same mailbox credentials |
|
||||||
|
| https://auth.wsvc.info | us2 (`/opt/vaultwarden`) | Vaultwarden (Postgres, **operational**) — client Server URL |
|
||||||
|
| `repo.windy.me:2222` | us2 (`/opt/soft-serve`) | Soft Serve (stub details) |
|
||||||
|
| DNS `ns1.wsvc.info:53` | hk2 (`/opt/pdns`, Auth **5.0.6**) | PowerDNS auth — zones `windy.me`, `wsvc.info`, `chans.xyz` |
|
||||||
|
| https://pdns.wsvc.info | hk2 (`poweradmin`) | Poweradmin UI |
|
||||||
|
| https://pgweb.wsvc.info | hk2 (`pgweb`) | PowerDNS Postgres browser |
|
||||||
|
| https://chat.chans.xyz | synapse (`ess`, ESS chart `26.7.2`) | Element Web (Matrix client) |
|
||||||
|
| https://synapse.chans.xyz | synapse | Synapse Client-Server + Federation API |
|
||||||
|
| https://account.chans.xyz | synapse | Matrix Authentication Service (local passwords) |
|
||||||
|
| https://admin.chans.xyz | synapse | Element Admin console (MAS admin auth) |
|
||||||
|
|
||||||
|
### Upstream docs
|
||||||
|
|
||||||
|
**Vaultwarden:** Before changing config, read [docs/vaultwarden-upstream.md](docs/vaultwarden-upstream.md). Live DB is **Postgres** (`DATABASE_URL`); `vw-data/config.json` from `/admin` overrides `.env` (including SMTP password — keep it synced with `.env` / `.smtp-credentials`). After recreating the VW container, restart Traefik if `https://auth.wsvc.info` returns 404. SMTP health needs **AUTH**, not TCP-only — see [vaultwarden-health](runbooks/vaultwarden-health.md).
|
||||||
|
|
||||||
|
**PowerDNS:** Before changing `/opt/pdns` on hk2, read [docs/pdns-upstream.md](docs/pdns-upstream.md). Auth pin **5.0.6**; primary + `also-notify`/`only-notify=` + DNSSEC OK. Deferred: rotate weak API/DB credentials and TSIG (`TSIG-ALLOW-AXFR` bypasses IP ACL). Health: [pdns-health](runbooks/pdns-health.md).
|
||||||
|
|
||||||
|
**Matrix (ESS on synapse):** Matrix homeserver running on `synapse.chans.xyz` via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone `chans.xyz` managed by hk2 PowerDNS. Before changing config, read [docs/matrix-upstream.md](docs/matrix-upstream.md) and [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: [matrix-health](runbooks/matrix-health.md).
|
||||||
|
|
||||||
|
## Runbooks & scripts
|
||||||
|
|
||||||
|
| Task | Path |
|
||||||
|
|------|------|
|
||||||
|
| mailcow health | [runbooks/mailcow-health.md](runbooks/mailcow-health.md) · `./scripts/check-mx2.sh` |
|
||||||
|
| mailcow update | [runbooks/mailcow-update.md](runbooks/mailcow-update.md) |
|
||||||
|
| mailcow SMTP/IMAP client | [runbooks/mailcow-smtp-client.md](runbooks/mailcow-smtp-client.md) |
|
||||||
|
| Vaultwarden health | [runbooks/vaultwarden-health.md](runbooks/vaultwarden-health.md) |
|
||||||
|
| Vaultwarden SQLite→PG migrate | [runbooks/vaultwarden-sqlite-to-postgres.md](runbooks/vaultwarden-sqlite-to-postgres.md) |
|
||||||
|
| Vaultwarden upstream refs | [docs/vaultwarden-upstream.md](docs/vaultwarden-upstream.md) |
|
||||||
|
| PowerDNS health (hk2) | [runbooks/pdns-health.md](runbooks/pdns-health.md) |
|
||||||
|
| PowerDNS upstream refs | [docs/pdns-upstream.md](docs/pdns-upstream.md) |
|
||||||
|
| Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) |
|
||||||
|
| Matrix upstream refs | [docs/matrix-upstream.md](docs/matrix-upstream.md) |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/check-mx2.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
`check-mx2.sh` defaults to IPv4; set `MX2_ALLOW_IPV6=1` to allow IPv6. DNS checks use `1.1.1.1` / `8.8.8.8` (local stub is flaky).
|
||||||
|
|
||||||
|
## Safety
|
||||||
|
|
||||||
|
- Never commit secrets: passwords, API keys, private keys, `.env`, `mailcow.conf` DB passwords, Vaultwarden `ADMIN_TOKEN` / `.smtp-credentials`.
|
||||||
|
- Secrets live **only on the servers** (e.g. `/opt/vaultwarden/.env`). Do not copy them from Obsidian into this repo.
|
||||||
|
- Ask before destructive actions: `docker compose down -v`, DB drops, DNS changes, force-push, disk wipes.
|
||||||
|
- Do not modify local `~/.ssh/config` or git config unless explicitly asked.
|
||||||
|
|
||||||
|
## SSH access safety — mandatory change procedure
|
||||||
|
|
||||||
|
**Treat any SSH authentication or authorization change as a lockout-risk change.** This includes `sshd_config` / drop-in edits, `PasswordAuthentication`, `PermitRootLogin`, `AuthorizedKeysFile`, `Match` blocks, SSH keys, accounts, groups, chroots, firewalls, and SSH ports.
|
||||||
|
|
||||||
|
Before disabling an existing access method or reloading `sshd`, all of the following are mandatory:
|
||||||
|
|
||||||
|
1. Keep the current verified SSH management session open as the rollback path; do not use it to run the validation connection.
|
||||||
|
2. Install the intended management public key from a **verified local source**. Never assume a public-key file exists on the target host or copy a remote path without checking it first.
|
||||||
|
3. Validate the candidate configuration with `sshd -t` before reloading the service.
|
||||||
|
4. From a separate local terminal, prove the new key-only login works using the exact target account, hostname, and key (`BatchMode=yes`, password authentication disabled), and verify needed sudo access.
|
||||||
|
5. Only after step 4 succeeds may password login, root login, the old key, or another existing access route be disabled.
|
||||||
|
6. Keep the original management session open until the final effective SSH configuration and a fresh key-only login have both been verified.
|
||||||
|
|
||||||
|
If the new key-based login cannot be proven first, stop immediately: do **not** reload an access-restricting configuration. Provider web/serial console access must be confirmed before any SSH access-policy change, but it is a recovery path—not a substitute for the validation steps above.
|
||||||
|
|
||||||
|
## Obsidian
|
||||||
|
|
||||||
|
Bills, rough notes, and personal clutter stay in the Obsidian vault. This repo holds **actionable facts and runbooks** only.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
AGENTS.md / agent.md # this entry (agent.md → AGENTS.md)
|
||||||
|
inventory/hosts.md # machine index
|
||||||
|
hosts/ # per-host facts
|
||||||
|
runbooks/ # step-by-step ops
|
||||||
|
docs/ # upstream doc indexes / design notes
|
||||||
|
scripts/ # read-only helper scripts
|
||||||
|
```
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
[defaults]
|
||||||
|
inventory = inventory/hosts.yml
|
||||||
|
roles_path = roles
|
||||||
|
interpreter_python = auto_silent
|
||||||
|
stdout_callback = default
|
||||||
|
bin_ansible_callbacks = True
|
||||||
|
retry_files_enabled = False
|
||||||
|
host_key_checking = True
|
||||||
|
|
||||||
|
[privilege_escalation]
|
||||||
|
become = True
|
||||||
|
become_method = sudo
|
||||||
|
become_ask_pass = False
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
---
|
||||||
|
# Sanitized control-plane inventory. Canonical human-readable facts remain
|
||||||
|
# in ../../inventory/hosts.md and ../../hosts/*.md. No passwords, keys, tokens,
|
||||||
|
# repository URLs, or private material belong here.
|
||||||
|
all:
|
||||||
|
vars:
|
||||||
|
ansible_user: windy
|
||||||
|
ansible_ssh_common_args: >-
|
||||||
|
-o BatchMode=yes -o ConnectTimeout=10 -o AddressFamily=inet
|
||||||
|
ansible_become: true
|
||||||
|
ansible_become_method: sudo
|
||||||
|
children:
|
||||||
|
managed:
|
||||||
|
hosts:
|
||||||
|
mx2:
|
||||||
|
ansible_host: mx2.windy.me
|
||||||
|
ansible_host_ipv4: 194.163.160.244
|
||||||
|
service_role: mailcow
|
||||||
|
compose_project_dir: /opt/mail
|
||||||
|
healthcheck_profile: mailcow
|
||||||
|
us2:
|
||||||
|
ansible_host: us2.wsvc.info
|
||||||
|
ansible_host_ipv4: 193.9.44.165
|
||||||
|
service_role: vaultwarden
|
||||||
|
compose_project_dir: /opt/vaultwarden
|
||||||
|
healthcheck_profile: vaultwarden
|
||||||
|
hk2:
|
||||||
|
ansible_host: hk2.chans.xyz
|
||||||
|
ansible_host_ipv4: 154.36.174.161
|
||||||
|
service_role: powerdns
|
||||||
|
compose_project_dir: /opt/pdns
|
||||||
|
healthcheck_profile: pdns
|
||||||
|
mailcow:
|
||||||
|
hosts:
|
||||||
|
mx2:
|
||||||
|
vaultwarden:
|
||||||
|
hosts:
|
||||||
|
us2:
|
||||||
|
powerdns:
|
||||||
|
hosts:
|
||||||
|
hk2:
|
||||||
|
docker_hosts:
|
||||||
|
children:
|
||||||
|
mailcow:
|
||||||
|
vaultwarden:
|
||||||
|
powerdns:
|
||||||
|
# Matrix is a dedicated K3s node and intentionally remains outside the
|
||||||
|
# Docker-oriented managed group.
|
||||||
|
k3s_servers:
|
||||||
|
hosts:
|
||||||
|
matrix_vps:
|
||||||
|
ansible_host: 169.58.86.13
|
||||||
|
ansible_host_ipv4: 169.58.86.13
|
||||||
|
service_role: matrix_k3s
|
||||||
|
matrix_server_name: chans.xyz
|
||||||
|
matrix_synapse_host: synapse.chans.xyz
|
||||||
|
matrix_element_host: chat.chans.xyz
|
||||||
|
matrix_mas_host: account.chans.xyz
|
||||||
|
matrix_admin_host: admin.chans.xyz
|
||||||
|
matrix_rtc_host: mrtc.chans.xyz
|
||||||
|
matrix_backup_path: /var/backups/matrix
|
||||||
|
matrix_bootstrap_dir: /etc/matrix-bootstrap
|
||||||
|
matrix_stack_enabled: false
|
||||||
|
# ESS OCI chart configuration
|
||||||
|
matrix_stack_chart_ref: oci://ghcr.io/element-hq/ess-helm/matrix-stack
|
||||||
|
matrix_stack_chart_version: 26.7.2
|
||||||
|
matrix_stack_release_name: ess
|
||||||
|
matrix_namespace: ess
|
||||||
|
matrix:
|
||||||
|
children:
|
||||||
|
k3s_servers:
|
||||||
|
matrix_production:
|
||||||
|
children:
|
||||||
|
matrix:
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
---
|
||||||
|
# Read-only control-plane audit. This play intentionally contains no package,
|
||||||
|
# file, service, container, or configuration mutation tasks.
|
||||||
|
- name: Audit managed VPS hosts without changes
|
||||||
|
hosts: managed
|
||||||
|
gather_facts: true
|
||||||
|
become: false
|
||||||
|
any_errors_fatal: false
|
||||||
|
tasks:
|
||||||
|
- name: Verify Docker Compose command is available
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- docker
|
||||||
|
- compose
|
||||||
|
- version
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Inspect configured Compose project
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- docker
|
||||||
|
- compose
|
||||||
|
- --project-directory
|
||||||
|
- "{{ compose_project_dir }}"
|
||||||
|
- ps
|
||||||
|
- --all
|
||||||
|
register: audit_compose_ps
|
||||||
|
changed_when: false
|
||||||
|
failed_when: false
|
||||||
|
|
||||||
|
- name: Inspect failed systemd units
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- systemctl
|
||||||
|
- --failed
|
||||||
|
- --no-legend
|
||||||
|
- --no-pager
|
||||||
|
register: audit_failed_units
|
||||||
|
changed_when: false
|
||||||
|
failed_when: false
|
||||||
|
|
||||||
|
- name: Inspect filesystem capacity
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- df
|
||||||
|
- -P
|
||||||
|
- -x
|
||||||
|
- tmpfs
|
||||||
|
- -x
|
||||||
|
- devtmpfs
|
||||||
|
register: audit_filesystems
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Inspect active listeners
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- ss
|
||||||
|
- -lntup
|
||||||
|
register: audit_listeners
|
||||||
|
changed_when: false
|
||||||
|
failed_when: false
|
||||||
|
|
||||||
|
- name: Report sanitized audit summary
|
||||||
|
ansible.builtin.debug:
|
||||||
|
msg:
|
||||||
|
host: "{{ inventory_hostname }}"
|
||||||
|
profile: "{{ healthcheck_profile }}"
|
||||||
|
os: "{{ ansible_distribution }} {{ ansible_distribution_version }}"
|
||||||
|
kernel: "{{ ansible_kernel }}"
|
||||||
|
compose_rc: "{{ audit_compose_ps.rc }}"
|
||||||
|
failed_units: "{{ audit_failed_units.stdout_lines | default([]) }}"
|
||||||
|
filesystem_lines: "{{ audit_filesystems.stdout_lines | default([]) }}"
|
||||||
|
listener_lines: "{{ audit_listeners.stdout_lines | default([]) }}"
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
# Baseline starts audit-only. Opt-in variables are deliberately false by default.
|
||||||
|
- name: Apply controlled common baseline
|
||||||
|
hosts: managed
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
roles:
|
||||||
|
- role: baseline
|
||||||
|
tags: [baseline, audit]
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
# Requires a target-local root-owned SMTP config; no credentials are passed here.
|
||||||
|
- name: Deploy health email alert integration
|
||||||
|
hosts: managed
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
roles:
|
||||||
|
- role: email_alert
|
||||||
|
tags: [healthcheck, email]
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
---
|
||||||
|
- name: Deploy daily local health checks
|
||||||
|
hosts: managed
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
roles:
|
||||||
|
- role: healthcheck
|
||||||
|
tags: [healthcheck, timers]
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
- name: Install or reconcile the single-node K3s server
|
||||||
|
hosts: k3s_servers
|
||||||
|
become: true
|
||||||
|
gather_facts: true
|
||||||
|
serial: 1
|
||||||
|
roles:
|
||||||
|
- role: k3s_server
|
||||||
|
tags: [k3s, matrix, mutating]
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
---
|
||||||
|
# Preview only. This playbook does not install updates, restart services, or
|
||||||
|
# change DNS/secrets. A separate, manually reviewed change is required to act.
|
||||||
|
- name: Preview pending maintenance without changes
|
||||||
|
hosts: managed
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
tasks:
|
||||||
|
- name: Check reboot requirement marker
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: /var/run/reboot-required
|
||||||
|
register: maintenance_reboot_marker
|
||||||
|
|
||||||
|
- name: Preview available package updates on Debian-family hosts
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- apt-get
|
||||||
|
- --just-print
|
||||||
|
- upgrade
|
||||||
|
register: maintenance_apt_preview
|
||||||
|
changed_when: false
|
||||||
|
failed_when: false
|
||||||
|
when: ansible_facts.os_family | default('Debian') == 'Debian'
|
||||||
|
|
||||||
|
- name: Report maintenance preview
|
||||||
|
ansible.builtin.debug:
|
||||||
|
msg:
|
||||||
|
host: "{{ inventory_hostname }}"
|
||||||
|
reboot_required: "{{ maintenance_reboot_marker.stat.exists }}"
|
||||||
|
package_preview: "{{ maintenance_apt_preview.stdout_lines | default([]) }}"
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
---
|
||||||
|
- name: Install local Matrix consistency backup jobs
|
||||||
|
hosts: matrix
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
roles:
|
||||||
|
- role: matrix_backup
|
||||||
|
tags: [matrix, backup, mutating]
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
---
|
||||||
|
- name: Reconcile Helm, cert-manager, and the Let's Encrypt issuer
|
||||||
|
hosts: k3s_servers
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
serial: 1
|
||||||
|
roles:
|
||||||
|
- role: helm_client
|
||||||
|
tags: [helm, matrix, mutating]
|
||||||
|
- role: cert_manager
|
||||||
|
tags: [cert_manager, matrix, mutating]
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
- name: Run the temporary Matrix HTTP-01 certificate smoke test
|
||||||
|
hosts: matrix
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
serial: 1
|
||||||
|
roles:
|
||||||
|
- role: matrix_certificate_smoke
|
||||||
|
tags: [matrix, certificates, smoke_test, mutating]
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
---
|
||||||
|
- name: Create non-secret Matrix Kubernetes foundation resources
|
||||||
|
hosts: matrix
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
roles:
|
||||||
|
- role: matrix_cluster_base
|
||||||
|
tags: [matrix, cluster_base, mutating]
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
---
|
||||||
|
- name: Install Matrix K3s local health checks
|
||||||
|
hosts: matrix
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
roles:
|
||||||
|
- role: matrix_healthcheck
|
||||||
|
tags: [matrix, healthcheck, timers]
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
---
|
||||||
|
# Read-only gate before a K3s or Matrix change.
|
||||||
|
- name: Validate Matrix K3s host readiness without changes
|
||||||
|
hosts: matrix
|
||||||
|
become: true
|
||||||
|
gather_facts: true
|
||||||
|
roles:
|
||||||
|
- role: k3s_preflight
|
||||||
|
tags: [matrix, preflight, read_only]
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
---
|
||||||
|
- name: Validate the pre-provisioned Matrix secret contract
|
||||||
|
hosts: matrix
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
roles:
|
||||||
|
- role: matrix_secret_contract
|
||||||
|
tags: [matrix, secrets, validation]
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
---
|
||||||
|
# Deploy the official ESS OCI chart on a K3s node.
|
||||||
|
# Pre-tasks create the non-secret values files on the target host.
|
||||||
|
# The matrix_stack role then validates and deploys the chart.
|
||||||
|
- name: Deploy the Matrix stack (ESS OCI chart)
|
||||||
|
hosts: matrix
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
serial: 1
|
||||||
|
pre_tasks:
|
||||||
|
- name: Ensure the ESS values directory exists
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: /etc/ess
|
||||||
|
state: directory
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0700"
|
||||||
|
|
||||||
|
- name: Write hostnames values file
|
||||||
|
ansible.builtin.copy:
|
||||||
|
dest: /etc/ess/hostnames.yaml
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0600"
|
||||||
|
content: |
|
||||||
|
serverName: {{ matrix_server_name }}
|
||||||
|
|
||||||
|
elementWeb:
|
||||||
|
ingress:
|
||||||
|
host: {{ matrix_element_host }}
|
||||||
|
|
||||||
|
synapse:
|
||||||
|
ingress:
|
||||||
|
host: {{ matrix_synapse_host }}
|
||||||
|
|
||||||
|
matrixAuthenticationService:
|
||||||
|
ingress:
|
||||||
|
host: {{ matrix_mas_host }}
|
||||||
|
|
||||||
|
elementAdmin:
|
||||||
|
ingress:
|
||||||
|
host: {{ matrix_admin_host }}
|
||||||
|
|
||||||
|
matrixRTC:
|
||||||
|
ingress:
|
||||||
|
host: {{ matrix_rtc_host }}
|
||||||
|
|
||||||
|
- name: Write TLS values file
|
||||||
|
ansible.builtin.copy:
|
||||||
|
dest: /etc/ess/tls.yaml
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0600"
|
||||||
|
content: |
|
||||||
|
certManager:
|
||||||
|
clusterIssuer: letsencrypt-prod
|
||||||
|
ingress:
|
||||||
|
className: traefik
|
||||||
|
tlsEnabled: true
|
||||||
|
|
||||||
|
- name: Write single-node tuning values file
|
||||||
|
ansible.builtin.copy:
|
||||||
|
dest: /etc/ess/single-node.yaml
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0600"
|
||||||
|
content: |
|
||||||
|
# ESS single-node resource tuning for K3s
|
||||||
|
# Chart defaults are already single-node-friendly
|
||||||
|
postgres:
|
||||||
|
storage:
|
||||||
|
size: 20Gi
|
||||||
|
redis:
|
||||||
|
maxMemory: 128mb
|
||||||
|
matrixRTC:
|
||||||
|
enabled: false
|
||||||
|
|
||||||
|
- name: Verify values files are in place
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "/etc/ess/{{ item }}"
|
||||||
|
loop:
|
||||||
|
- hostnames.yaml
|
||||||
|
- tls.yaml
|
||||||
|
- single-node.yaml
|
||||||
|
register: _values_check
|
||||||
|
|
||||||
|
- name: Assert all values files exist
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- item.stat.exists
|
||||||
|
- item.stat.isreg
|
||||||
|
- item.stat.pw_name == 'root'
|
||||||
|
loop: "{{ _values_check.results }}"
|
||||||
|
loop_control:
|
||||||
|
label: "{{ item.stat.path | default(item.item) }}"
|
||||||
|
|
||||||
|
roles:
|
||||||
|
- role: matrix_stack
|
||||||
|
tags: [matrix, stack, mutating]
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
---
|
||||||
|
# Intentionally targets only services with approved local dump/data sources.
|
||||||
|
# Set restic_enabled=true only after selecting a backend and provisioning the
|
||||||
|
# root-only repository config directly on each target.
|
||||||
|
- name: Deploy controlled Restic timers
|
||||||
|
hosts:
|
||||||
|
- vaultwarden
|
||||||
|
- powerdns
|
||||||
|
become: true
|
||||||
|
gather_facts: false
|
||||||
|
roles:
|
||||||
|
- role: restic
|
||||||
|
tags: [restic, backup]
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
---
|
||||||
|
baseline_manage_logrotate: false
|
||||||
|
baseline_require_chrony: false
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
---
|
||||||
|
- name: Audit SSH daemon effective configuration
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- sshd
|
||||||
|
- -T
|
||||||
|
register: baseline_sshd_effective
|
||||||
|
changed_when: false
|
||||||
|
failed_when: false
|
||||||
|
|
||||||
|
- name: Report SSH hardening observations without changing SSH
|
||||||
|
ansible.builtin.debug:
|
||||||
|
msg:
|
||||||
|
permit_root_login: >-
|
||||||
|
{{ baseline_sshd_effective.stdout_lines | select('match', '^permitrootlogin ') | list }}
|
||||||
|
password_authentication: >-
|
||||||
|
{{ baseline_sshd_effective.stdout_lines | select('match', '^passwordauthentication ') | list }}
|
||||||
|
|
||||||
|
- name: Audit time synchronization service state
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- systemctl
|
||||||
|
- is-active
|
||||||
|
- systemd-timesyncd
|
||||||
|
register: baseline_timesync_state
|
||||||
|
changed_when: false
|
||||||
|
failed_when: false
|
||||||
|
|
||||||
|
- name: Install baseline logrotate policy only with explicit opt-in
|
||||||
|
ansible.builtin.copy:
|
||||||
|
dest: /etc/logrotate.d/vps-baseline
|
||||||
|
content: |
|
||||||
|
/var/log/vps-health/*.log /var/log/vps-restic/*.log {
|
||||||
|
weekly
|
||||||
|
rotate 8
|
||||||
|
missingok
|
||||||
|
notifempty
|
||||||
|
compress
|
||||||
|
create 0640 root root
|
||||||
|
}
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0644"
|
||||||
|
when: baseline_manage_logrotate | bool
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
---
|
||||||
|
cert_manager_enabled: false
|
||||||
|
cert_manager_namespace: cert-manager
|
||||||
|
cert_manager_release_name: cert-manager
|
||||||
|
cert_manager_chart_repository_name: jetstack
|
||||||
|
cert_manager_chart_repository_url: https://charts.jetstack.io
|
||||||
|
cert_manager_chart_ref: jetstack/cert-manager
|
||||||
|
cert_manager_chart_version: v1.19.3
|
||||||
|
cert_manager_kubeconfig: /etc/rancher/k3s/k3s.yaml
|
||||||
|
cert_manager_cluster_issuer_name: letsencrypt-prod
|
||||||
|
cert_manager_acme_server: https://acme-v02.api.letsencrypt.org/directory
|
||||||
|
cert_manager_acme_private_key_secret: letsencrypt-prod-private-key
|
||||||
|
cert_manager_ingress_class: traefik
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
---
|
||||||
|
- name: Require explicit approval before cert-manager reconciliation
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- cert_manager_enabled | bool
|
||||||
|
fail_msg: >-
|
||||||
|
Refusing cert-manager changes. Re-run with cert_manager_enabled=true after
|
||||||
|
confirming public TCP 80 and 443 reach Traefik.
|
||||||
|
|
||||||
|
- name: Require Helm and the K3s kubeconfig
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "{{ item }}"
|
||||||
|
loop:
|
||||||
|
- /usr/local/bin/helm
|
||||||
|
- "{{ cert_manager_kubeconfig }}"
|
||||||
|
register: cert_manager_prerequisites
|
||||||
|
|
||||||
|
- name: Assert Helm and kubeconfig are available
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- item.stat.exists
|
||||||
|
fail_msg: "Missing cert-manager prerequisite: {{ item.item }}"
|
||||||
|
loop: "{{ cert_manager_prerequisites.results }}"
|
||||||
|
|
||||||
|
- name: Add or update the Jetstack Helm repository
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- helm
|
||||||
|
- repo
|
||||||
|
- add
|
||||||
|
- "{{ cert_manager_chart_repository_name }}"
|
||||||
|
- "{{ cert_manager_chart_repository_url }}"
|
||||||
|
- --force-update
|
||||||
|
environment:
|
||||||
|
KUBECONFIG: "{{ cert_manager_kubeconfig }}"
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Update Helm repositories
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv: [helm, repo, update]
|
||||||
|
environment:
|
||||||
|
KUBECONFIG: "{{ cert_manager_kubeconfig }}"
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Install or reconcile cert-manager
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- helm
|
||||||
|
- upgrade
|
||||||
|
- --install
|
||||||
|
- "{{ cert_manager_release_name }}"
|
||||||
|
- "{{ cert_manager_chart_ref }}"
|
||||||
|
- --namespace
|
||||||
|
- "{{ cert_manager_namespace }}"
|
||||||
|
- --create-namespace
|
||||||
|
- --version
|
||||||
|
- "{{ cert_manager_chart_version }}"
|
||||||
|
- --set
|
||||||
|
- crds.enabled=true
|
||||||
|
- --wait
|
||||||
|
- --timeout
|
||||||
|
- 10m
|
||||||
|
environment:
|
||||||
|
KUBECONFIG: "{{ cert_manager_kubeconfig }}"
|
||||||
|
changed_when: true
|
||||||
|
|
||||||
|
- name: Install ClusterIssuer manifest without secret material
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: clusterissuer.yaml.j2
|
||||||
|
dest: "/etc/rancher/k3s/{{ cert_manager_cluster_issuer_name }}-clusterissuer.yaml"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0600"
|
||||||
|
|
||||||
|
- name: Apply ClusterIssuer manifest
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- k3s
|
||||||
|
- kubectl
|
||||||
|
- apply
|
||||||
|
- -f
|
||||||
|
- "/etc/rancher/k3s/{{ cert_manager_cluster_issuer_name }}-clusterissuer.yaml"
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Wait for ClusterIssuer readiness
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- k3s
|
||||||
|
- kubectl
|
||||||
|
- wait
|
||||||
|
- --for=condition=Ready
|
||||||
|
- "clusterissuer/{{ cert_manager_cluster_issuer_name }}"
|
||||||
|
- --timeout=180s
|
||||||
|
changed_when: false
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: ClusterIssuer
|
||||||
|
metadata:
|
||||||
|
name: {{ cert_manager_cluster_issuer_name }}
|
||||||
|
spec:
|
||||||
|
acme:
|
||||||
|
server: {{ cert_manager_acme_server }}
|
||||||
|
privateKeySecretRef:
|
||||||
|
name: {{ cert_manager_acme_private_key_secret }}
|
||||||
|
solvers:
|
||||||
|
- http01:
|
||||||
|
ingress:
|
||||||
|
class: {{ cert_manager_ingress_class }}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
---
|
||||||
|
email_alert_config_path: /etc/vps-health/alert-smtp.conf
|
||||||
|
email_alert_state_path: /var/lib/vps-health/alert-state
|
||||||
|
email_alert_recipient: ''
|
||||||
|
email_alert_enabled: false
|
||||||
|
email_alert_repeat_hours: 24
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
---
|
||||||
|
- name: Reload systemd
|
||||||
|
ansible.builtin.systemd_service:
|
||||||
|
daemon_reload: true
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
---
|
||||||
|
# The health-check service owns the sole dispatcher hook. This role only
|
||||||
|
# installs/removes that dispatcher according to the explicit opt-in below.
|
||||||
|
- name: Require explicit non-secret alert recipient when email is enabled
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- email_alert_recipient | length > 0
|
||||||
|
fail_msg: Set email_alert_recipient outside version control before enabling alerts.
|
||||||
|
when: email_alert_enabled | bool
|
||||||
|
|
||||||
|
- name: Install alert integration when explicitly enabled
|
||||||
|
when: email_alert_enabled | bool
|
||||||
|
block:
|
||||||
|
- name: Install alert state directory
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ email_alert_state_path | dirname }}"
|
||||||
|
state: directory
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0750"
|
||||||
|
|
||||||
|
- name: Install secret-free alert dispatcher
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: alert-dispatch.sh.j2
|
||||||
|
dest: /usr/local/lib/vps-health/alert-dispatch
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0750"
|
||||||
|
|
||||||
|
- name: Remove alert integration when disabled
|
||||||
|
when: not (email_alert_enabled | bool)
|
||||||
|
block:
|
||||||
|
- name: Remove alert dispatcher
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: /usr/local/lib/vps-health/alert-dispatch
|
||||||
|
state: absent
|
||||||
|
|
||||||
|
- name: Remove legacy alert service drop-in
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: /etc/systemd/system/vps-healthcheck.service.d/alerting.conf
|
||||||
|
state: absent
|
||||||
|
notify: Reload systemd
|
||||||
|
|
||||||
|
- name: Report required server-side alert configuration
|
||||||
|
ansible.builtin.debug:
|
||||||
|
msg: >-
|
||||||
|
Email alerts are {{ 'enabled' if email_alert_enabled | bool else 'disabled' }}.
|
||||||
|
When enabled, the root-owned {{ email_alert_config_path }} must be provisioned
|
||||||
|
directly on the host and must contain SMTP settings and recipient; it is never
|
||||||
|
created or populated by this repository.
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Sends sanitized health results through the host's locally provisioned SMTP
|
||||||
|
# credentials. The config is intentionally excluded from Ansible/Git.
|
||||||
|
set -uo pipefail
|
||||||
|
config='{{ email_alert_config_path }}'
|
||||||
|
result='/var/lib/vps-health/latest.json'
|
||||||
|
state='{{ email_alert_state_path }}'
|
||||||
|
[[ -r "$config" && -r "$result" ]] || exit 0
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "$config"
|
||||||
|
: "${SMTP_URL:?missing SMTP_URL in server-side alert config}"
|
||||||
|
: "${ALERT_TO:?missing ALERT_TO in server-side alert config}"
|
||||||
|
status="$(python3 -c 'import json; print(json.load(open("'"$result"'"))["status"])')"
|
||||||
|
fingerprint="$(sha256sum "$result" | cut -d' ' -f1)"
|
||||||
|
previous="$(cat "$state" 2>/dev/null || true)"
|
||||||
|
now="$(date +%s)"
|
||||||
|
last_time="${previous%%:*}"; last_fp="${previous#*:}"
|
||||||
|
if [[ "$status" =~ ^(critical|unknown)$ ]] && [[ "$fingerprint" != "$last_fp" || $((now-${last_time:-0})) -ge {{ email_alert_repeat_hours }}*3600 ]]; then
|
||||||
|
subject="[${status}] VPS health $(hostname -f 2>/dev/null || hostname)"
|
||||||
|
curl --fail --silent --show-error --url "$SMTP_URL" --mail-rcpt "$ALERT_TO" --upload-file <(printf 'To: %s\nSubject: %s\nContent-Type: application/json\n\n%s\n' "$ALERT_TO" "$subject" "$(cat "$result")")
|
||||||
|
printf '%s:%s\n' "$now" "$fingerprint" > "$state"
|
||||||
|
elif [[ "$status" =~ ^(ok|warning)$ ]]; then
|
||||||
|
subject="[${status}] daily VPS health $(hostname -f 2>/dev/null || hostname)"
|
||||||
|
curl --fail --silent --show-error --url "$SMTP_URL" --mail-rcpt "$ALERT_TO" --upload-file <(printf 'To: %s\nSubject: %s\nContent-Type: application/json\n\n%s\n' "$ALERT_TO" "$subject" "$(cat "$result")")
|
||||||
|
fi
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
[Service]
|
||||||
|
ExecStartPost=/usr/local/lib/vps-health/alert-dispatch
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
---
|
||||||
|
healthcheck_install_root: /usr/local/lib/vps-health
|
||||||
|
healthcheck_state_dir: /var/lib/vps-health
|
||||||
|
healthcheck_log_dir: /var/log/vps-health
|
||||||
|
healthcheck_service_name: vps-healthcheck
|
||||||
|
healthcheck_timer_on_calendar: '*-*-* 06:15:00'
|
||||||
|
healthcheck_timer_randomized_delay_sec: 15m
|
||||||
|
healthcheck_backup_max_age_hours: 30
|
||||||
|
healthcheck_tls_warn_days: 21
|
||||||
|
healthcheck_profiles:
|
||||||
|
mailcow: mailcow.sh
|
||||||
|
vaultwarden: vaultwarden.sh
|
||||||
|
pdns: pdns.sh
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
---
|
||||||
|
- name: Reload systemd
|
||||||
|
ansible.builtin.systemd_service:
|
||||||
|
daemon_reload: true
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
---
|
||||||
|
- name: Validate known health-check profile
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- healthcheck_profile in healthcheck_profiles
|
||||||
|
fail_msg: "Unsupported healthcheck_profile: {{ healthcheck_profile }}"
|
||||||
|
|
||||||
|
- name: Install health-check directories
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ item.path }}"
|
||||||
|
state: directory
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "{{ item.mode }}"
|
||||||
|
loop:
|
||||||
|
- path: "{{ healthcheck_install_root }}"
|
||||||
|
mode: "0755"
|
||||||
|
- path: "{{ healthcheck_state_dir }}"
|
||||||
|
mode: "0750"
|
||||||
|
- path: "{{ healthcheck_log_dir }}"
|
||||||
|
mode: "0750"
|
||||||
|
|
||||||
|
- name: Install common health-check library
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: health-common.sh.j2
|
||||||
|
dest: "{{ healthcheck_install_root }}/health-common.sh"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0755"
|
||||||
|
|
||||||
|
- name: Install service health-check script
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: "{{ healthcheck_profiles[healthcheck_profile] }}.j2"
|
||||||
|
dest: "{{ healthcheck_install_root }}/{{ healthcheck_profiles[healthcheck_profile] }}"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0755"
|
||||||
|
|
||||||
|
- name: Install health-check dispatcher
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: healthcheck-runner.sh.j2
|
||||||
|
dest: "{{ healthcheck_install_root }}/run"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0755"
|
||||||
|
|
||||||
|
- name: Install health-check systemd unit
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: vps-healthcheck.service.j2
|
||||||
|
dest: "/etc/systemd/system/{{ healthcheck_service_name }}.service"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0644"
|
||||||
|
notify: Reload systemd
|
||||||
|
|
||||||
|
- name: Install persistent health-check timer
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: vps-healthcheck.timer.j2
|
||||||
|
dest: "/etc/systemd/system/{{ healthcheck_service_name }}.timer"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0644"
|
||||||
|
notify: Reload systemd
|
||||||
|
|
||||||
|
- name: Install health-check log rotation
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: logrotate-vps-healthcheck.j2
|
||||||
|
dest: /etc/logrotate.d/vps-healthcheck
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0644"
|
||||||
|
|
||||||
|
- name: Enable health-check timer
|
||||||
|
ansible.builtin.systemd_service:
|
||||||
|
name: "{{ healthcheck_service_name }}.timer"
|
||||||
|
enabled: true
|
||||||
|
state: started
|
||||||
|
daemon_reload: true
|
||||||
|
|
||||||
|
- name: Flush health-check systemd changes
|
||||||
|
ansible.builtin.meta: flush_handlers
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Shared contract: one JSON object per run in /var/lib/vps-health/latest.json.
|
||||||
|
# Status values: ok, warning, critical, unknown. Exit codes: 0 healthy/warning,
|
||||||
|
# 2 critical, 3 unknown. Check detail must never include secret values.
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
RESULT_DIR='{{ healthcheck_state_dir }}'
|
||||||
|
LOG_DIR='{{ healthcheck_log_dir }}'
|
||||||
|
HOST_NAME="$(hostname -f 2>/dev/null || hostname)"
|
||||||
|
CHECK_NAME='{{ healthcheck_profile }}'
|
||||||
|
STATUS=ok
|
||||||
|
EXIT_CODE=0
|
||||||
|
DETAILS=()
|
||||||
|
|
||||||
|
record() {
|
||||||
|
local severity="$1" message="$2"
|
||||||
|
DETAILS+=("${severity}:${message}")
|
||||||
|
case "$severity" in
|
||||||
|
critical) STATUS=critical; EXIT_CODE=2 ;;
|
||||||
|
unknown) [[ "$STATUS" != critical ]] && STATUS=unknown; [[ "$EXIT_CODE" -eq 0 ]] && EXIT_CODE=3 ;;
|
||||||
|
warning) [[ "$STATUS" == ok ]] && STATUS=warning ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
require_command() {
|
||||||
|
command -v "$1" >/dev/null 2>&1 || record unknown "missing_command:$1"
|
||||||
|
}
|
||||||
|
|
||||||
|
compose_ps() {
|
||||||
|
docker compose --project-directory '{{ compose_project_dir }}' ps --all 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
check_compose() {
|
||||||
|
local output
|
||||||
|
output="$(compose_ps)" || { record critical 'compose_ps_failed'; return; }
|
||||||
|
if grep -qiE 'Exited|Restarting|[[:space:]]Dead[[:space:]]' <<<"$output"; then
|
||||||
|
record critical 'compose_unhealthy_container'
|
||||||
|
else
|
||||||
|
record ok 'compose_ok'
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
check_backup_freshness() {
|
||||||
|
local pattern="$1" newest now age
|
||||||
|
newest="$(find {{ compose_project_dir | quote }} -type f -path "$pattern" -printf '%T@\n' 2>/dev/null | sort -nr | head -n1)"
|
||||||
|
if [[ -z "$newest" ]]; then
|
||||||
|
record warning 'backup_not_found'
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
now="$(date +%s)"
|
||||||
|
age="$(( now - ${newest%.*} ))"
|
||||||
|
if (( age > {{ healthcheck_backup_max_age_hours }} * 3600 )); then
|
||||||
|
record critical 'backup_stale'
|
||||||
|
else
|
||||||
|
record ok 'backup_fresh'
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
check_https() {
|
||||||
|
local url="$1" expected="$2" code
|
||||||
|
code="$(curl --silent --show-error --output /dev/null --write-out '%{http_code}' --max-time 20 "$url" 2>/dev/null)" || {
|
||||||
|
record critical 'https_request_failed'; return;
|
||||||
|
}
|
||||||
|
[[ "$code" =~ $expected ]] && record ok "https_${code}" || record critical "https_${code}"
|
||||||
|
}
|
||||||
|
|
||||||
|
check_tls_days() {
|
||||||
|
local host="$1" port="$2" expires epoch remaining
|
||||||
|
expires="$(timeout 20 openssl s_client -connect "${host}:${port}" -servername "$host" </dev/null 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2-)" || {
|
||||||
|
record unknown 'tls_read_failed'; return;
|
||||||
|
}
|
||||||
|
epoch="$(date -d "$expires" +%s 2>/dev/null)" || { record unknown 'tls_date_parse_failed'; return; }
|
||||||
|
remaining="$(( (epoch - $(date +%s)) / 86400 ))"
|
||||||
|
if (( remaining < 0 )); then record critical 'tls_expired'
|
||||||
|
elif (( remaining < {{ healthcheck_tls_warn_days }} )); then record warning 'tls_near_expiry'
|
||||||
|
else record ok 'tls_valid'; fi
|
||||||
|
}
|
||||||
|
|
||||||
|
emit_result() {
|
||||||
|
local tmp detail_json
|
||||||
|
tmp="$(mktemp "${RESULT_DIR}/latest.json.XXXXXX")"
|
||||||
|
detail_json="$(printf '%s\n' "${DETAILS[@]:-unknown:no_details}" | python3 -c 'import json,sys; print(json.dumps([line.rstrip() for line in sys.stdin if line.strip()]))')"
|
||||||
|
python3 - "$tmp" "$HOST_NAME" "$CHECK_NAME" "$STATUS" "$EXIT_CODE" "$detail_json" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
path, host, check, status, code, details = sys.argv[1:]
|
||||||
|
with open(path, 'w', encoding='utf-8') as f:
|
||||||
|
json.dump({'schema': 1, 'host': host, 'check': check, 'status': status,
|
||||||
|
'exit_code': int(code), 'details': json.loads(details)}, f,
|
||||||
|
sort_keys=True, separators=(',', ':'))
|
||||||
|
f.write('\n')
|
||||||
|
PY
|
||||||
|
chmod 0640 "$tmp"
|
||||||
|
mv "$tmp" "${RESULT_DIR}/latest.json"
|
||||||
|
cat "${RESULT_DIR}/latest.json"
|
||||||
|
}
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -uo pipefail
|
||||||
|
exec '{{ healthcheck_install_root }}/{{ healthcheck_profiles[healthcheck_profile] }}' >> '{{ healthcheck_log_dir }}/healthcheck.log' 2>&1
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
{{ healthcheck_log_dir }}/healthcheck.log {
|
||||||
|
daily
|
||||||
|
rotate 14
|
||||||
|
missingok
|
||||||
|
notifempty
|
||||||
|
compress
|
||||||
|
delaycompress
|
||||||
|
create 0640 root root
|
||||||
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -uo pipefail
|
||||||
|
source '{{ healthcheck_install_root }}/health-common.sh'
|
||||||
|
|
||||||
|
require_command docker
|
||||||
|
require_command curl
|
||||||
|
require_command openssl
|
||||||
|
require_command dig
|
||||||
|
require_command timeout
|
||||||
|
check_compose
|
||||||
|
|
||||||
|
watchdog="$(docker compose --project-directory '{{ compose_project_dir }}' logs --tail=40 watchdog-mailcow 2>&1)" || record critical 'watchdog_log_failed'
|
||||||
|
grep -qiE '100%|healthy' <<<"$watchdog" || record warning 'watchdog_health_not_confirmed'
|
||||||
|
|
||||||
|
queue="$(docker compose --project-directory '{{ compose_project_dir }}' exec -T postfix-mailcow postqueue -p 2>&1)" || record critical 'mail_queue_check_failed'
|
||||||
|
grep -Fqi 'Mail queue is empty' <<<"$queue" || record warning 'mail_queue_nonempty'
|
||||||
|
|
||||||
|
listeners="$(ss -lnt 2>/dev/null)"
|
||||||
|
for port in 25 465 587 993 443; do
|
||||||
|
grep -qE ":${port}[[:space:]]" <<<"$listeners" || record critical "listener_missing_${port}"
|
||||||
|
done
|
||||||
|
|
||||||
|
check_https 'https://mx2.windy.me/' '^200$'
|
||||||
|
check_tls_days mx2.windy.me 443
|
||||||
|
smtp="$(timeout 10 bash -c "exec 3<>/dev/tcp/mx2.windy.me/25; printf 'EHLO health.local\\r\\nQUIT\\r\\n' >&3; cat <&3" 2>/dev/null)" || record critical 'smtp_connect_failed'
|
||||||
|
grep -qiE 'Postcow|ESMTP' <<<"$smtp" || record critical 'smtp_banner_unexpected'
|
||||||
|
|
||||||
|
for resolver in 1.1.1.1 8.8.8.8; do
|
||||||
|
mx="$(dig +short +time=3 +tries=1 "@${resolver}" windy.me MX 2>/dev/null)"
|
||||||
|
grep -Fqi 'mx2.windy.me' <<<"$mx" && { record ok 'mx_record_ok'; break; } || record warning 'mx_record_not_confirmed'
|
||||||
|
done
|
||||||
|
spf="$(dig +short +time=3 +tries=1 @1.1.1.1 windy.me TXT 2>/dev/null)"
|
||||||
|
grep -Fqi 'v=spf1' <<<"$spf" || record warning 'spf_not_confirmed'
|
||||||
|
check_backup_freshness '*/backup/*'
|
||||||
|
emit_result
|
||||||
|
exit "$EXIT_CODE"
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -uo pipefail
|
||||||
|
source '{{ healthcheck_install_root }}/health-common.sh'
|
||||||
|
|
||||||
|
require_command docker
|
||||||
|
require_command curl
|
||||||
|
require_command dig
|
||||||
|
check_compose
|
||||||
|
|
||||||
|
image="$(docker inspect pdns-auth --format '{{ '{{' }}.Config.Image{{ '}}' }}' 2>/dev/null)" || record critical 'pdns_container_missing'
|
||||||
|
grep -Eq ':5\.0\.[6-9]|:5\.[1-9]\.' <<<"$image" || record warning 'pdns_version_not_confirmed'
|
||||||
|
security="$(docker logs pdns-auth 2>&1 | grep -i 'Mandatory.*Security Update' || true)"
|
||||||
|
[[ -z "$security" ]] || record critical 'pdns_security_update_banner'
|
||||||
|
|
||||||
|
# The API key remains in the container environment; the request emits only the
|
||||||
|
# reported version and no authentication material.
|
||||||
|
api="$(docker compose --project-directory '{{ compose_project_dir }}' exec -T auth python3 - <<'PY' 2>&1
|
||||||
|
import json, os, urllib.request
|
||||||
|
request=urllib.request.Request('http://127.0.0.1:8081/api/v1/servers/localhost', headers={'X-API-Key': os.environ['PDNS_API_KEY']})
|
||||||
|
print(json.load(urllib.request.urlopen(request, timeout=10))['version'])
|
||||||
|
PY
|
||||||
|
)" || record critical 'pdns_api_failed'
|
||||||
|
grep -Eq '^5\.' <<<"$api" || record critical 'pdns_api_version_invalid'
|
||||||
|
|
||||||
|
for zone in windy.me wsvc.info chans.xyz; do
|
||||||
|
primary="$(dig +short @154.36.174.161 SOA "$zone" 2>/dev/null)"
|
||||||
|
secondary="$(dig +short @202.91.35.141 SOA "$zone" 2>/dev/null)"
|
||||||
|
[[ -n "$primary" ]] || record critical "soa_primary_missing_${zone}"
|
||||||
|
[[ -n "$secondary" ]] || record critical "soa_secondary_missing_${zone}"
|
||||||
|
[[ "$primary" == "$secondary" ]] || record warning "soa_secondary_lag_${zone}"
|
||||||
|
done
|
||||||
|
|
||||||
|
check_https 'https://pdns.wsvc.info/' '^30[12]$'
|
||||||
|
check_https 'https://pgweb.wsvc.info/' '^(200|401)$'
|
||||||
|
check_backup_freshness '*/backup/*.sql.gz'
|
||||||
|
emit_result
|
||||||
|
exit "$EXIT_CODE"
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -uo pipefail
|
||||||
|
source '{{ healthcheck_install_root }}/health-common.sh'
|
||||||
|
|
||||||
|
require_command docker
|
||||||
|
require_command curl
|
||||||
|
require_command openssl
|
||||||
|
require_command python3
|
||||||
|
check_compose
|
||||||
|
|
||||||
|
health="$(docker compose --project-directory '{{ compose_project_dir }}' ps --format json 2>&1)" || record critical 'compose_status_unavailable'
|
||||||
|
grep -Fq 'vaultwarden' <<<"$health" || record critical 'vaultwarden_missing'
|
||||||
|
grep -Fq 'vw-db' <<<"$health" || record critical 'postgres_missing'
|
||||||
|
|
||||||
|
check_https 'https://auth.wsvc.info/' '^200$'
|
||||||
|
check_tls_days auth.wsvc.info 443
|
||||||
|
|
||||||
|
# Read effective config only inside the service and report booleans/fingerprints,
|
||||||
|
# never its SMTP password or other secret fields.
|
||||||
|
smtp_result="$(docker compose --project-directory '{{ compose_project_dir }}' exec -T vaultwarden python3 - <<'PY' 2>&1
|
||||||
|
import json, pathlib, smtplib, ssl
|
||||||
|
cfg=json.loads(pathlib.Path('/data/config.json').read_text())
|
||||||
|
host=cfg.get('smtp_host'); port=int(cfg.get('smtp_port') or 0)
|
||||||
|
user=cfg.get('smtp_username')
|
||||||
|
smtp_secret=cfg.get('smtp_password')
|
||||||
|
if not all((host, port, user, smtp_secret)):
|
||||||
|
raise SystemExit('smtp_config_incomplete')
|
||||||
|
with smtplib.SMTP(host, port, timeout=15) as client:
|
||||||
|
client.ehlo(); client.starttls(context=ssl.create_default_context()); client.ehlo(); client.login(user, smtp_secret)
|
||||||
|
print('smtp_auth_ok')
|
||||||
|
PY
|
||||||
|
)"
|
||||||
|
grep -Fqx 'smtp_auth_ok' <<<"$smtp_result" || record critical 'smtp_auth_failed'
|
||||||
|
|
||||||
|
# Detect drift without exposing the values: matching SHA-256 digests are used only
|
||||||
|
# internally and the result is a boolean.
|
||||||
|
drift_result="$(python3 - <<'PY'
|
||||||
|
import hashlib, json, pathlib, re
|
||||||
|
root=pathlib.Path('{{ compose_project_dir }}')
|
||||||
|
env=root.joinpath('.env').read_text()
|
||||||
|
match=re.search(r'^SMTP_PASSWORD=(.*)$', env, re.M)
|
||||||
|
config=json.loads(root.joinpath('vw-data/config.json').read_text())
|
||||||
|
value=(match.group(1).strip().strip('"\'') if match else '')
|
||||||
|
print('smtp_password_match' if value and value == (config.get('smtp_password') or '') else 'smtp_password_drift')
|
||||||
|
PY
|
||||||
|
2>&1)" || record unknown 'smtp_drift_check_failed'
|
||||||
|
grep -Fqx 'smtp_password_match' <<<"$drift_result" || record critical 'smtp_password_drift'
|
||||||
|
|
||||||
|
check_backup_freshness '*/backups/*.sql.gz'
|
||||||
|
emit_result
|
||||||
|
exit "$EXIT_CODE"
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Read-only VPS health check (%i)
|
||||||
|
Wants=network-online.target
|
||||||
|
After=network-online.target docker.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=root
|
||||||
|
Group=root
|
||||||
|
UMask=0027
|
||||||
|
ExecStart={{ healthcheck_install_root }}/run
|
||||||
|
# The optional dispatcher is installed only by email_alert when explicitly enabled.
|
||||||
|
# This service remains healthy if alerting is intentionally unavailable.
|
||||||
|
ExecStartPost=-/usr/local/lib/vps-health/alert-dispatch
|
||||||
|
NoNewPrivileges=true
|
||||||
|
PrivateTmp=true
|
||||||
|
ProtectHome=true
|
||||||
|
ProtectSystem=full
|
||||||
|
ReadWritePaths={{ healthcheck_state_dir }} {{ healthcheck_log_dir }}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Daily read-only VPS health check
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar={{ healthcheck_timer_on_calendar }}
|
||||||
|
Persistent=true
|
||||||
|
RandomizedDelaySec={{ healthcheck_timer_randomized_delay_sec }}
|
||||||
|
Unit={{ healthcheck_service_name }}.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
---
|
||||||
|
# Helm is required by the patched matrix-stack release. Existing installs are
|
||||||
|
# reported and retained; this role only installs Helm when it is absent.
|
||||||
|
helm_client_install_enabled: false
|
||||||
|
helm_client_install_script_url: https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
|
||||||
|
helm_client_binary: /usr/local/bin/helm
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
---
|
||||||
|
- name: Check whether Helm is installed
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "{{ helm_client_binary }}"
|
||||||
|
register: helm_client_binary_state
|
||||||
|
|
||||||
|
- name: Require explicit approval to install Helm when absent
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- helm_client_install_enabled | bool
|
||||||
|
fail_msg: >-
|
||||||
|
Helm is not installed. Re-run with helm_client_install_enabled=true after
|
||||||
|
reviewing the upstream installer source and checksum policy.
|
||||||
|
when: not helm_client_binary_state.stat.exists
|
||||||
|
|
||||||
|
- name: Install Helm only when explicitly approved and absent
|
||||||
|
ansible.builtin.shell:
|
||||||
|
cmd: "curl -fsSL {{ helm_client_install_script_url }} | bash"
|
||||||
|
creates: "{{ helm_client_binary }}"
|
||||||
|
when:
|
||||||
|
- helm_client_install_enabled | bool
|
||||||
|
- not helm_client_binary_state.stat.exists
|
||||||
|
no_log: true
|
||||||
|
|
||||||
|
- name: Read Helm version
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv: ["{{ helm_client_binary }}", version, --short]
|
||||||
|
changed_when: false
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
---
|
||||||
|
# Read-only checks for a dedicated, single-node Matrix K3s host.
|
||||||
|
k3s_preflight_required_memory_mib: 6144
|
||||||
|
k3s_preflight_required_root_free_gib: 50
|
||||||
|
k3s_preflight_required_hosts:
|
||||||
|
- "{{ matrix_server_name }}"
|
||||||
|
- "{{ matrix_synapse_host }}"
|
||||||
|
- "{{ matrix_element_host }}"
|
||||||
|
- "{{ matrix_mas_host }}"
|
||||||
|
- "{{ matrix_admin_host }}"
|
||||||
|
- "{{ matrix_rtc_host }}"
|
||||||
|
k3s_preflight_required_ports:
|
||||||
|
- 80
|
||||||
|
- 443
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
---
|
||||||
|
- name: Read host memory in MiB
|
||||||
|
ansible.builtin.set_fact:
|
||||||
|
k3s_preflight_memory_mib: "{{ (ansible_memtotal_mb | int) }}"
|
||||||
|
|
||||||
|
- name: Read root filesystem capacity facts
|
||||||
|
ansible.builtin.set_fact:
|
||||||
|
k3s_preflight_root_mount: >-
|
||||||
|
{{ (ansible_mounts | selectattr('mount', 'equalto', '/') | list | first) | default({}) }}
|
||||||
|
|
||||||
|
- name: Assert host has the baseline resources for Matrix
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- k3s_preflight_memory_mib | int >= k3s_preflight_required_memory_mib | int
|
||||||
|
- (k3s_preflight_root_mount.size_available | default(0) | int) >= (k3s_preflight_required_root_free_gib | int * 1024 * 1024 * 1024)
|
||||||
|
fail_msg: >-
|
||||||
|
Matrix requires at least {{ k3s_preflight_required_memory_mib }} MiB RAM and
|
||||||
|
{{ k3s_preflight_required_root_free_gib }} GiB available on /. No change was made.
|
||||||
|
|
||||||
|
- name: Check whether K3s is already installed
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: /usr/local/bin/k3s
|
||||||
|
register: k3s_preflight_binary
|
||||||
|
|
||||||
|
- name: Read listeners on required public ports
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv: [ss, -lntH]
|
||||||
|
changed_when: false
|
||||||
|
register: k3s_preflight_listeners
|
||||||
|
|
||||||
|
- name: Assert public ports are unused before initial K3s installation
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- >-
|
||||||
|
(k3s_preflight_binary.stat.exists | bool) or
|
||||||
|
((k3s_preflight_listeners.stdout_lines | select('search', '(:|\\.)' ~ (item | string) ~ '$') | list | length) == 0)
|
||||||
|
fail_msg: "Port {{ item }} is already listening; resolve the ingress ownership conflict first."
|
||||||
|
loop: "{{ k3s_preflight_required_ports }}"
|
||||||
|
|
||||||
|
- name: Resolve Matrix hostnames over IPv4
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv: [getent, ahostsv4, "{{ item }}"]
|
||||||
|
changed_when: false
|
||||||
|
register: k3s_preflight_dns
|
||||||
|
loop: "{{ k3s_preflight_required_hosts }}"
|
||||||
|
|
||||||
|
- name: Assert every Matrix hostname resolves to the selected VPS
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- item.stdout is search(ansible_host_ipv4 | regex_escape)
|
||||||
|
fail_msg: >-
|
||||||
|
{{ item.item }} does not resolve to {{ ansible_host_ipv4 }} over IPv4. DNS must
|
||||||
|
be correct before HTTP-01 certificates can be issued.
|
||||||
|
loop: "{{ k3s_preflight_dns.results }}"
|
||||||
|
|
||||||
|
- name: Check the local backup path parent filesystem
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv: [df, -P, "{{ matrix_backup_path | dirname }}"]
|
||||||
|
changed_when: false
|
||||||
|
register: k3s_preflight_backup_filesystem
|
||||||
|
|
||||||
|
- name: Report read-only preflight state
|
||||||
|
ansible.builtin.debug:
|
||||||
|
msg:
|
||||||
|
k3s_installed: "{{ k3s_preflight_binary.stat.exists }}"
|
||||||
|
memory_mib: "{{ k3s_preflight_memory_mib }}"
|
||||||
|
root_available_bytes: "{{ k3s_preflight_root_mount.size_available | default(0) }}"
|
||||||
|
backup_path: "{{ matrix_backup_path }}"
|
||||||
|
backup_filesystem: "{{ k3s_preflight_backup_filesystem.stdout_lines[-1] }}"
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
---
|
||||||
|
# The installer downloads the current stable channel only for an uninstalled node.
|
||||||
|
# Normal reruns never auto-upgrade an existing K3s installation.
|
||||||
|
k3s_install_enabled: false
|
||||||
|
k3s_install_url: https://get.k3s.io
|
||||||
|
k3s_binary_path: /usr/local/bin/k3s
|
||||||
|
k3s_service_name: k3s
|
||||||
|
k3s_server_args:
|
||||||
|
- --disable=servicelb
|
||||||
|
- --disable=metrics-server
|
||||||
|
k3s_kubeconfig: /etc/rancher/k3s/k3s.yaml
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
---
|
||||||
|
- name: Check whether K3s is installed
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "{{ k3s_binary_path }}"
|
||||||
|
register: k3s_server_binary
|
||||||
|
|
||||||
|
- name: Require explicit approval before first K3s installation
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- k3s_install_enabled | bool
|
||||||
|
fail_msg: >-
|
||||||
|
Refusing first-time K3s installation. Re-run only after preflight succeeds
|
||||||
|
with k3s_install_enabled=true.
|
||||||
|
when: not (k3s_server_binary.stat.exists | bool)
|
||||||
|
|
||||||
|
- name: Install current stable K3s only on a new node
|
||||||
|
ansible.builtin.shell:
|
||||||
|
cmd: >-
|
||||||
|
curl -fsSL {{ k3s_install_url }} |
|
||||||
|
INSTALL_K3S_EXEC='server {{ k3s_server_args | join(' ') }}' sh -
|
||||||
|
creates: "{{ k3s_binary_path }}"
|
||||||
|
when: not (k3s_server_binary.stat.exists | bool)
|
||||||
|
no_log: true
|
||||||
|
|
||||||
|
- name: Wait for K3s API readiness
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- "{{ k3s_binary_path }}"
|
||||||
|
- kubectl
|
||||||
|
- get
|
||||||
|
- node
|
||||||
|
- --output=jsonpath={.items[0].status.conditions[?(@.type=="Ready")].status}
|
||||||
|
changed_when: false
|
||||||
|
register: k3s_server_ready
|
||||||
|
retries: 30
|
||||||
|
delay: 5
|
||||||
|
until: k3s_server_ready.stdout == 'True'
|
||||||
|
|
||||||
|
- name: Assert Traefik is installed and K3s optional components remain disabled
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- "{{ k3s_binary_path }}"
|
||||||
|
- kubectl
|
||||||
|
- get
|
||||||
|
- deployment
|
||||||
|
- traefik
|
||||||
|
- --namespace=kube-system
|
||||||
|
changed_when: false
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
---
|
||||||
|
matrix_backup_enabled: false
|
||||||
|
matrix_backup_path: /var/backups/matrix
|
||||||
|
matrix_backup_retention_days: 7
|
||||||
|
matrix_backup_warn_percent: 80
|
||||||
|
matrix_backup_stop_percent: 90
|
||||||
|
matrix_backup_script_path: /usr/local/sbin/matrix-backup
|
||||||
|
matrix_backup_service_name: matrix-backup.service
|
||||||
|
matrix_backup_timer_name: matrix-backup.timer
|
||||||
|
matrix_namespace: matrix-system
|
||||||
|
matrix_backup_postgres_pod_selector: app=matrix-postgres
|
||||||
|
matrix_backup_media_pod_selector: app=synapse,component=synapse-media-repository
|
||||||
|
matrix_backup_bootstrap_dir: /etc/matrix-bootstrap
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
---
|
||||||
|
- name: Reload systemd
|
||||||
|
ansible.builtin.systemd:
|
||||||
|
daemon_reload: true
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
---
|
||||||
|
- name: Require explicit approval before installing backup automation
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- matrix_backup_enabled | bool
|
||||||
|
fail_msg: >-
|
||||||
|
Refusing to install Matrix backup automation until matrix_backup_enabled=true
|
||||||
|
is supplied deliberately.
|
||||||
|
|
||||||
|
- name: Create root-only Matrix backup directory
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ matrix_backup_path }}"
|
||||||
|
state: directory
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0700"
|
||||||
|
|
||||||
|
- name: Install Matrix backup script
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: matrix-backup.sh.j2
|
||||||
|
dest: "{{ matrix_backup_script_path }}"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0700"
|
||||||
|
|
||||||
|
- name: Install Matrix backup systemd service
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: matrix-backup.service.j2
|
||||||
|
dest: "/etc/systemd/system/{{ matrix_backup_service_name }}"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0644"
|
||||||
|
notify: Reload systemd
|
||||||
|
|
||||||
|
- name: Install Matrix backup systemd timer
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: matrix-backup.timer.j2
|
||||||
|
dest: "/etc/systemd/system/{{ matrix_backup_timer_name }}"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0644"
|
||||||
|
notify: Reload systemd
|
||||||
|
|
||||||
|
- name: Enable Matrix backup timer
|
||||||
|
ansible.builtin.systemd:
|
||||||
|
name: "{{ matrix_backup_timer_name }}"
|
||||||
|
enabled: true
|
||||||
|
state: started
|
||||||
|
daemon_reload: true
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Create local consistent Matrix backup
|
||||||
|
Wants=network-online.target
|
||||||
|
After=network-online.target k3s.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=root
|
||||||
|
Group=root
|
||||||
|
UMask=0077
|
||||||
|
ExecStart={{ matrix_backup_script_path }}
|
||||||
|
NoNewPrivileges=true
|
||||||
|
PrivateTmp=true
|
||||||
|
ProtectHome=true
|
||||||
|
ProtectSystem=full
|
||||||
|
ReadWritePaths={{ matrix_backup_path }} /etc/matrix-bootstrap
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
umask 077
|
||||||
|
|
||||||
|
backup_root='{{ matrix_backup_path }}'
|
||||||
|
namespace='{{ matrix_namespace }}'
|
||||||
|
warn_percent='{{ matrix_backup_warn_percent }}'
|
||||||
|
stop_percent='{{ matrix_backup_stop_percent }}'
|
||||||
|
retention_days='{{ matrix_backup_retention_days }}'
|
||||||
|
postgres_selector='{{ matrix_backup_postgres_pod_selector }}'
|
||||||
|
media_selector='{{ matrix_backup_media_pod_selector }}'
|
||||||
|
bootstrap_dir='{{ matrix_backup_bootstrap_dir }}'
|
||||||
|
|
||||||
|
usage=$(df -P / | awk 'NR == 2 {gsub(/%/, "", $5); print $5}')
|
||||||
|
if (( usage >= stop_percent )); then
|
||||||
|
printf 'Refusing Matrix backup: root filesystem usage is %s%% (stop threshold %s%%).\n' "$usage" "$stop_percent" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
stamp=$(date -u +%Y%m%dT%H%M%SZ)
|
||||||
|
stage="$backup_root/.staging-$stamp"
|
||||||
|
final="$backup_root/$stamp"
|
||||||
|
mkdir -p "$stage" "$backup_root"
|
||||||
|
trap 'rm -rf "$stage"' EXIT
|
||||||
|
|
||||||
|
if ! command -v k3s >/dev/null 2>&1; then
|
||||||
|
printf 'K3s is unavailable; refusing Matrix backup.\n' >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
postgres_pod=$(k3s kubectl -n "$namespace" get pod -l "$postgres_selector" -o jsonpath='{.items[0].metadata.name}')
|
||||||
|
media_pod=$(k3s kubectl -n "$namespace" get pod -l "$media_selector" -o jsonpath='{.items[0].metadata.name}')
|
||||||
|
if [[ -z "$postgres_pod" || -z "$media_pod" ]]; then
|
||||||
|
printf 'Required Matrix PostgreSQL or media pod is unavailable.\n' >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
for database in synapse mas; do
|
||||||
|
k3s kubectl -n "$namespace" exec "$postgres_pod" -- \
|
||||||
|
pg_dump --username=postgres --format=custom --file="/tmp/$database-$stamp.dump" "$database"
|
||||||
|
k3s kubectl -n "$namespace" cp \
|
||||||
|
"$namespace/$postgres_pod:/tmp/$database-$stamp.dump" "$stage/$database.dump"
|
||||||
|
k3s kubectl -n "$namespace" exec "$postgres_pod" -- rm -f "/tmp/$database-$stamp.dump"
|
||||||
|
done
|
||||||
|
|
||||||
|
k3s kubectl -n "$namespace" exec "$media_pod" -- \
|
||||||
|
tar --create --gzip --file="/tmp/media-$stamp.tar.gz" --directory=/data media_store
|
||||||
|
k3s kubectl -n "$namespace" cp \
|
||||||
|
"$namespace/$media_pod:/tmp/media-$stamp.tar.gz" "$stage/media.tar.gz"
|
||||||
|
k3s kubectl -n "$namespace" exec "$media_pod" -- rm -f "/tmp/media-$stamp.tar.gz"
|
||||||
|
|
||||||
|
tar --create --gzip --file="$stage/bootstrap.tar.gz" --directory="$(dirname "$bootstrap_dir")" "$(basename "$bootstrap_dir")"
|
||||||
|
sha256sum "$stage"/* > "$stage/SHA256SUMS"
|
||||||
|
printf '{"created_at":"%s","root_usage_percent":%s,"warning_threshold_percent":%s}\n' \
|
||||||
|
"$stamp" "$usage" "$warn_percent" > "$stage/manifest.json"
|
||||||
|
mv "$stage" "$final"
|
||||||
|
trap - EXIT
|
||||||
|
find "$backup_root" -mindepth 1 -maxdepth 1 -type d -name '20*Z' -mtime +"$retention_days" -exec rm -rf {} +
|
||||||
|
printf 'Matrix backup created: %s\n' "$final"
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Run Matrix local backup daily
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar=*-*-* 03:15:00 UTC
|
||||||
|
Persistent=true
|
||||||
|
RandomizedDelaySec=15m
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
---
|
||||||
|
# A minimal HTTPS placeholder for the explicitly reserved RTC hostname. It also
|
||||||
|
# validates the Traefik HTTP-01 certificate path without deploying RTC services.
|
||||||
|
matrix_certificate_smoke_enabled: false
|
||||||
|
matrix_certificate_smoke_namespace: matrix-system
|
||||||
|
matrix_certificate_smoke_name: mrtc-placeholder
|
||||||
|
matrix_certificate_smoke_host: mrtc.chans.xyz
|
||||||
|
matrix_certificate_smoke_ingress_class: traefik
|
||||||
|
matrix_certificate_smoke_cluster_issuer: letsencrypt-prod
|
||||||
|
matrix_certificate_smoke_kubeconfig: /etc/rancher/k3s/k3s.yaml
|
||||||
|
matrix_certificate_smoke_cleanup: false
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
---
|
||||||
|
- name: Require explicit approval before certificate smoke test
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- matrix_certificate_smoke_enabled | bool
|
||||||
|
fail_msg: >-
|
||||||
|
Refusing certificate smoke test changes. Re-run with
|
||||||
|
matrix_certificate_smoke_enabled=true after creating the test DNS record.
|
||||||
|
|
||||||
|
- name: Render non-sensitive certificate smoke-test resources
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: smoke.yaml.j2
|
||||||
|
dest: "/etc/rancher/k3s/{{ matrix_certificate_smoke_name }}.yaml"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0600"
|
||||||
|
|
||||||
|
- name: Apply certificate smoke-test resources
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- k3s
|
||||||
|
- kubectl
|
||||||
|
- apply
|
||||||
|
- -f
|
||||||
|
- "/etc/rancher/k3s/{{ matrix_certificate_smoke_name }}.yaml"
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Wait for smoke-test deployment
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- k3s
|
||||||
|
- kubectl
|
||||||
|
- rollout
|
||||||
|
- status
|
||||||
|
- "deployment/{{ matrix_certificate_smoke_name }}"
|
||||||
|
- "--namespace={{ matrix_certificate_smoke_namespace }}"
|
||||||
|
- --timeout=180s
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Wait for smoke-test certificate
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- k3s
|
||||||
|
- kubectl
|
||||||
|
- wait
|
||||||
|
- --for=condition=Ready
|
||||||
|
- "certificate/{{ matrix_certificate_smoke_name }}-tls"
|
||||||
|
- "--namespace={{ matrix_certificate_smoke_namespace }}"
|
||||||
|
- --timeout=10m
|
||||||
|
changed_when: false
|
||||||
|
when: not (matrix_certificate_smoke_cleanup | bool)
|
||||||
|
|
||||||
|
- name: Remove smoke-test resources after verification
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- k3s
|
||||||
|
- kubectl
|
||||||
|
- delete
|
||||||
|
- -f
|
||||||
|
- "/etc/rancher/k3s/{{ matrix_certificate_smoke_name }}.yaml"
|
||||||
|
- --ignore-not-found=true
|
||||||
|
changed_when: false
|
||||||
|
when: matrix_certificate_smoke_cleanup | bool
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: {{ matrix_certificate_smoke_name }}-content
|
||||||
|
namespace: {{ matrix_certificate_smoke_namespace }}
|
||||||
|
data:
|
||||||
|
index.html: MatrixRTC is not available yet.
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: {{ matrix_certificate_smoke_name }}
|
||||||
|
namespace: {{ matrix_certificate_smoke_namespace }}
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: {{ matrix_certificate_smoke_name }}
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: {{ matrix_certificate_smoke_name }}
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: static
|
||||||
|
image: nginx:1.27.5-alpine
|
||||||
|
ports:
|
||||||
|
- containerPort: 80
|
||||||
|
volumeMounts:
|
||||||
|
- name: content
|
||||||
|
mountPath: /usr/share/nginx/html/index.html
|
||||||
|
subPath: index.html
|
||||||
|
readOnly: true
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 10m
|
||||||
|
memory: 32Mi
|
||||||
|
limits:
|
||||||
|
cpu: 50m
|
||||||
|
memory: 64Mi
|
||||||
|
volumes:
|
||||||
|
- name: content
|
||||||
|
configMap:
|
||||||
|
name: {{ matrix_certificate_smoke_name }}-content
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: {{ matrix_certificate_smoke_name }}
|
||||||
|
namespace: {{ matrix_certificate_smoke_namespace }}
|
||||||
|
spec:
|
||||||
|
selector:
|
||||||
|
app: {{ matrix_certificate_smoke_name }}
|
||||||
|
ports:
|
||||||
|
- name: http
|
||||||
|
port: 80
|
||||||
|
targetPort: 80
|
||||||
|
---
|
||||||
|
apiVersion: networking.k8s.io/v1
|
||||||
|
kind: Ingress
|
||||||
|
metadata:
|
||||||
|
name: {{ matrix_certificate_smoke_name }}
|
||||||
|
namespace: {{ matrix_certificate_smoke_namespace }}
|
||||||
|
annotations:
|
||||||
|
cert-manager.io/cluster-issuer: {{ matrix_certificate_smoke_cluster_issuer }}
|
||||||
|
spec:
|
||||||
|
ingressClassName: {{ matrix_certificate_smoke_ingress_class }}
|
||||||
|
tls:
|
||||||
|
- hosts:
|
||||||
|
- {{ matrix_certificate_smoke_host }}
|
||||||
|
secretName: {{ matrix_certificate_smoke_name }}-tls
|
||||||
|
rules:
|
||||||
|
- host: {{ matrix_certificate_smoke_host }}
|
||||||
|
http:
|
||||||
|
paths:
|
||||||
|
- path: /
|
||||||
|
pathType: Prefix
|
||||||
|
backend:
|
||||||
|
service:
|
||||||
|
name: {{ matrix_certificate_smoke_name }}
|
||||||
|
port:
|
||||||
|
name: http
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
---
|
||||||
|
# These resources are intentionally chart-independent and secret-free.
|
||||||
|
matrix_namespace: matrix-system
|
||||||
|
matrix_resource_quota:
|
||||||
|
requests.cpu: "4"
|
||||||
|
requests.memory: 6Gi
|
||||||
|
limits.cpu: "6"
|
||||||
|
limits.memory: 7Gi
|
||||||
|
persistentvolumeclaims: "4"
|
||||||
|
matrix_limit_range:
|
||||||
|
defaultRequest:
|
||||||
|
cpu: 50m
|
||||||
|
memory: 128Mi
|
||||||
|
default:
|
||||||
|
memory: 1Gi
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
---
|
||||||
|
- name: Require a working K3s kubeconfig
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: /etc/rancher/k3s/k3s.yaml
|
||||||
|
register: matrix_cluster_kubeconfig
|
||||||
|
|
||||||
|
- name: Assert K3s is ready before creating cluster resources
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- matrix_cluster_kubeconfig.stat.exists
|
||||||
|
fail_msg: Run k3s-server.yml successfully before matrix-cluster-base.yml.
|
||||||
|
|
||||||
|
- name: Install Matrix cluster base manifest
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: base-resources.yaml.j2
|
||||||
|
dest: /etc/rancher/k3s/matrix-cluster-base.yaml
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0600"
|
||||||
|
|
||||||
|
- name: Apply Matrix cluster base manifest
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv: [k3s, kubectl, apply, -f, /etc/rancher/k3s/matrix-cluster-base.yaml]
|
||||||
|
changed_when: false
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: {{ matrix_namespace }}
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ResourceQuota
|
||||||
|
metadata:
|
||||||
|
name: matrix-system-quota
|
||||||
|
namespace: {{ matrix_namespace }}
|
||||||
|
spec:
|
||||||
|
hard:
|
||||||
|
{{ matrix_resource_quota | to_nice_yaml(indent=4) | indent(4, true) }}
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: LimitRange
|
||||||
|
metadata:
|
||||||
|
name: matrix-system-defaults
|
||||||
|
namespace: {{ matrix_namespace }}
|
||||||
|
spec:
|
||||||
|
limits:
|
||||||
|
- type: Container
|
||||||
|
defaultRequest:
|
||||||
|
{{ matrix_limit_range.defaultRequest | to_nice_yaml(indent=8) | indent(8, true) }}
|
||||||
|
default:
|
||||||
|
{{ matrix_limit_range.default | to_nice_yaml(indent=8) | indent(8, true) }}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
apiVersion: cert-manager.io/v1
|
||||||
|
kind: ClusterIssuer
|
||||||
|
metadata:
|
||||||
|
name: letsencrypt-prod
|
||||||
|
spec:
|
||||||
|
acme:
|
||||||
|
server: https://acme-v02.api.letsencrypt.org/directory
|
||||||
|
privateKeySecretRef:
|
||||||
|
name: letsencrypt-prod-private-key
|
||||||
|
solvers:
|
||||||
|
- http01:
|
||||||
|
ingress:
|
||||||
|
class: traefik
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
---
|
||||||
|
matrix_healthcheck_enabled: false
|
||||||
|
matrix_healthcheck_script_path: /usr/local/lib/vps-health/matrix-k3s
|
||||||
|
matrix_healthcheck_state_path: /var/lib/vps-health/matrix-k3s.json
|
||||||
|
matrix_healthcheck_service_name: matrix-k3s-healthcheck.service
|
||||||
|
matrix_healthcheck_timer_name: matrix-k3s-healthcheck.timer
|
||||||
|
matrix_healthcheck_timer_on_calendar: '*-*-* 06:00:00 UTC'
|
||||||
|
matrix_healthcheck_timer_randomized_delay_sec: 15m
|
||||||
|
matrix_healthcheck_namespace: matrix-system
|
||||||
|
matrix_healthcheck_backup_path: /var/backups/matrix
|
||||||
|
matrix_healthcheck_backup_max_age_hours: 30
|
||||||
|
matrix_healthcheck_warn_percent: 80
|
||||||
|
matrix_healthcheck_critical_percent: 90
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
---
|
||||||
|
- name: Require explicit approval before installing Matrix health checks
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- matrix_healthcheck_enabled | bool
|
||||||
|
fail_msg: Set matrix_healthcheck_enabled=true only after the Matrix namespace and backup job exist.
|
||||||
|
|
||||||
|
- name: Create Matrix healthcheck script directory
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ matrix_healthcheck_script_path | dirname }}"
|
||||||
|
state: directory
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0755"
|
||||||
|
|
||||||
|
- name: Create Matrix healthcheck state directory
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ matrix_healthcheck_state_path | dirname }}"
|
||||||
|
state: directory
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0750"
|
||||||
|
|
||||||
|
- name: Install Matrix K3s healthcheck script
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: matrix-k3s.sh.j2
|
||||||
|
dest: "{{ matrix_healthcheck_script_path }}"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0750"
|
||||||
|
|
||||||
|
- name: Install Matrix K3s healthcheck systemd service
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: matrix-k3s.service.j2
|
||||||
|
dest: "/etc/systemd/system/{{ matrix_healthcheck_service_name }}"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0644"
|
||||||
|
|
||||||
|
- name: Install Matrix K3s healthcheck systemd timer
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: matrix-k3s.timer.j2
|
||||||
|
dest: "/etc/systemd/system/{{ matrix_healthcheck_timer_name }}"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0644"
|
||||||
|
|
||||||
|
- name: Enable Matrix K3s healthcheck timer
|
||||||
|
ansible.builtin.systemd:
|
||||||
|
name: "{{ matrix_healthcheck_timer_name }}"
|
||||||
|
enabled: true
|
||||||
|
state: started
|
||||||
|
daemon_reload: true
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Read-only Matrix K3s health check
|
||||||
|
Wants=network-online.target
|
||||||
|
After=network-online.target k3s.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=root
|
||||||
|
Group=root
|
||||||
|
UMask=0027
|
||||||
|
ExecStart={{ matrix_healthcheck_script_path }}
|
||||||
|
NoNewPrivileges=true
|
||||||
|
PrivateTmp=true
|
||||||
|
ProtectHome=true
|
||||||
|
ProtectSystem=full
|
||||||
|
ReadWritePaths={{ matrix_healthcheck_state_path | dirname }}
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
namespace='{{ matrix_healthcheck_namespace }}'
|
||||||
|
backup_root='{{ matrix_healthcheck_backup_path }}'
|
||||||
|
max_backup_age_seconds=$(({{ matrix_healthcheck_backup_max_age_hours }} * 3600))
|
||||||
|
warn_percent='{{ matrix_healthcheck_warn_percent }}'
|
||||||
|
critical_percent='{{ matrix_healthcheck_critical_percent }}'
|
||||||
|
status=ok
|
||||||
|
messages=()
|
||||||
|
|
||||||
|
set_status() {
|
||||||
|
local next="$1"
|
||||||
|
case "$next" in
|
||||||
|
critical) status=critical ;;
|
||||||
|
warning) [[ "$status" != critical ]] && status=warning ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
if ! k3s kubectl get namespace "$namespace" >/dev/null 2>&1; then
|
||||||
|
set_status critical
|
||||||
|
messages+=("namespace $namespace is unavailable")
|
||||||
|
else
|
||||||
|
unready=$(k3s kubectl -n "$namespace" get pods --no-headers 2>/dev/null | awk '$2 !~ /^[0-9]+\/[0-9]+$/ || $3 != "Running" {print $1}')
|
||||||
|
if [[ -n "$unready" ]]; then
|
||||||
|
set_status critical
|
||||||
|
messages+=("unready pods: ${unready//$'\n'/, }")
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
usage=$(df -P / | awk 'NR == 2 {gsub(/%/, "", $5); print $5}')
|
||||||
|
if (( usage >= critical_percent )); then
|
||||||
|
set_status critical
|
||||||
|
messages+=("root filesystem usage ${usage}%")
|
||||||
|
elif (( usage >= warn_percent )); then
|
||||||
|
set_status warning
|
||||||
|
messages+=("root filesystem usage ${usage}%")
|
||||||
|
fi
|
||||||
|
|
||||||
|
latest=$(find "$backup_root" -mindepth 1 -maxdepth 1 -type d -name '20*Z' -printf '%T@ %p\n' 2>/dev/null | sort -nr | awk 'NR == 1 {print $1, $2}')
|
||||||
|
if [[ -z "$latest" ]]; then
|
||||||
|
set_status critical
|
||||||
|
messages+=("no Matrix backup exists")
|
||||||
|
else
|
||||||
|
latest_epoch=${latest%% *}
|
||||||
|
latest_path=${latest#* }
|
||||||
|
age=$(( $(date +%s) - ${latest_epoch%.*} ))
|
||||||
|
if (( age > max_backup_age_seconds )); then
|
||||||
|
set_status critical
|
||||||
|
messages+=("latest Matrix backup is ${age}s old")
|
||||||
|
elif [[ ! -f "$latest_path/SHA256SUMS" || ! -f "$latest_path/manifest.json" ]]; then
|
||||||
|
set_status critical
|
||||||
|
messages+=("latest Matrix backup is incomplete")
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
printf '{"service":"matrix_k3s","status":"%s","messages":[' "$status"
|
||||||
|
for i in "${!messages[@]}"; do
|
||||||
|
(( i > 0 )) && printf ','
|
||||||
|
printf '"%s"' "${messages[$i]//\"/\\\"}"
|
||||||
|
done
|
||||||
|
printf '],"root_usage_percent":%s}\n' "$usage"
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Run Matrix K3s health check daily
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar={{ matrix_healthcheck_timer_on_calendar }}
|
||||||
|
Persistent=true
|
||||||
|
RandomizedDelaySec={{ matrix_healthcheck_timer_randomized_delay_sec }}
|
||||||
|
Unit={{ matrix_healthcheck_service_name }}
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
---
|
||||||
|
matrix_namespace: matrix-system
|
||||||
|
matrix_bootstrap_dir: /etc/matrix-bootstrap
|
||||||
|
# This role validates only names, keys, types, ownership and modes. It never reads
|
||||||
|
# bootstrap values or emits Kubernetes Secret contents.
|
||||||
|
matrix_required_bootstrap_files:
|
||||||
|
- smtp-password
|
||||||
|
- synapse-db-password
|
||||||
|
- mas-db-password
|
||||||
|
- synapse-signing-key
|
||||||
|
- synapse-macaroon-secret
|
||||||
|
- synapse-form-secret
|
||||||
|
- mas-encryption-secret
|
||||||
|
- mas-signing-key
|
||||||
|
matrix_required_kubernetes_secrets:
|
||||||
|
- name: matrix-synapse-master-config
|
||||||
|
keys: [homeserver.yaml, log.config, signing.key]
|
||||||
|
- name: matrix-authentication-config
|
||||||
|
keys: [mas-config.yaml]
|
||||||
|
- name: matrix-pgbouncer-userlist
|
||||||
|
keys: [userlist.txt]
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
---
|
||||||
|
- name: Inspect Matrix bootstrap directory without reading secrets
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "{{ matrix_bootstrap_dir }}"
|
||||||
|
register: matrix_secret_bootstrap_dir
|
||||||
|
|
||||||
|
- name: Assert Matrix bootstrap directory is root-only
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- matrix_secret_bootstrap_dir.stat.exists
|
||||||
|
- matrix_secret_bootstrap_dir.stat.isdir
|
||||||
|
- matrix_secret_bootstrap_dir.stat.pw_name == 'root'
|
||||||
|
- matrix_secret_bootstrap_dir.stat.gr_name == 'root'
|
||||||
|
- matrix_secret_bootstrap_dir.stat.mode == '0700'
|
||||||
|
fail_msg: "{{ matrix_bootstrap_dir }} must be a root:root 0700 directory."
|
||||||
|
|
||||||
|
- name: Inspect required individual bootstrap secret files without reading them
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "{{ matrix_bootstrap_dir }}/{{ item }}"
|
||||||
|
loop: "{{ matrix_required_bootstrap_files }}"
|
||||||
|
register: matrix_secret_bootstrap_files
|
||||||
|
no_log: true
|
||||||
|
|
||||||
|
- name: Assert required bootstrap secret file permissions
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- item.stat.exists
|
||||||
|
- item.stat.isreg
|
||||||
|
- item.stat.pw_name == 'root'
|
||||||
|
- item.stat.gr_name == 'root'
|
||||||
|
- item.stat.mode == '0600'
|
||||||
|
fail_msg: A required root-only Matrix bootstrap file is missing or has unsafe permissions.
|
||||||
|
loop: "{{ matrix_secret_bootstrap_files.results }}"
|
||||||
|
no_log: true
|
||||||
|
|
||||||
|
- name: Inspect Kubernetes Secret metadata without retrieving values
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv:
|
||||||
|
- k3s
|
||||||
|
- kubectl
|
||||||
|
- describe
|
||||||
|
- secret
|
||||||
|
- "{{ item.name }}"
|
||||||
|
- --namespace={{ matrix_namespace }}
|
||||||
|
changed_when: false
|
||||||
|
loop: "{{ matrix_required_kubernetes_secrets }}"
|
||||||
|
register: matrix_secret_kubernetes_metadata
|
||||||
|
no_log: true
|
||||||
|
|
||||||
|
- name: Assert required Kubernetes Secret keys exist
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- item.stdout is search('(?m)^' ~ key ~ ':')
|
||||||
|
fail_msg: Required Matrix Kubernetes Secret key is missing.
|
||||||
|
loop: "{{ matrix_secret_kubernetes_metadata.results | subelements('item.keys') }}"
|
||||||
|
loop_control:
|
||||||
|
loop_var: matrix_secret_key_check
|
||||||
|
vars:
|
||||||
|
item: "{{ matrix_secret_key_check.0 }}"
|
||||||
|
key: "{{ matrix_secret_key_check.1 }}"
|
||||||
|
no_log: true
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
---
|
||||||
|
# Matrix Stack deployment using the official ESS OCI chart
|
||||||
|
# All sensitive values are handled by the chart's built-in initSecrets
|
||||||
|
# or provided via pre-created Kubernetes Secrets.
|
||||||
|
|
||||||
|
matrix_stack_enabled: false
|
||||||
|
matrix_namespace: ess
|
||||||
|
matrix_stack_chart_ref: oci://ghcr.io/element-hq/ess-helm/matrix-stack
|
||||||
|
matrix_stack_chart_version: 26.7.2
|
||||||
|
matrix_stack_release_name: ess
|
||||||
|
# Root-only, non-secret values directory on the target host
|
||||||
|
matrix_stack_values_dir: /etc/ess
|
||||||
|
# Individual values files (must not contain secrets)
|
||||||
|
matrix_stack_values_files:
|
||||||
|
- hostnames.yaml
|
||||||
|
- tls.yaml
|
||||||
|
- single-node.yaml
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
---
|
||||||
|
- name: Require explicit approval before deploying the Matrix stack
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- matrix_stack_enabled | bool
|
||||||
|
fail_msg: >-
|
||||||
|
Refusing Matrix deployment until matrix_stack_enabled=true is set deliberately.
|
||||||
|
|
||||||
|
- name: Require Helm on the Matrix host
|
||||||
|
ansible.builtin.command:
|
||||||
|
argv: [helm, version, --short]
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Require non-secret values directory exists on the host
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "{{ matrix_stack_values_dir }}"
|
||||||
|
register: _values_dir
|
||||||
|
|
||||||
|
- name: Assert values directory exists and is root-owned
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- _values_dir.stat.exists
|
||||||
|
- _values_dir.stat.isdir
|
||||||
|
- _values_dir.stat.pw_name == 'root'
|
||||||
|
- _values_dir.stat.gr_name == 'root'
|
||||||
|
- _values_dir.stat.mode == '0700'
|
||||||
|
fail_msg: >-
|
||||||
|
Create the root-owned directory {{ matrix_stack_values_dir }}
|
||||||
|
with mode 0700 and place the non-secret values files in it.
|
||||||
|
|
||||||
|
- name: Require each non-secret values file exists on the host
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "{{ matrix_stack_values_dir }}/{{ item }}"
|
||||||
|
loop: "{{ matrix_stack_values_files }}"
|
||||||
|
register: _values_files
|
||||||
|
|
||||||
|
- name: Assert all values files exist and are root-owned
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- item.stat.exists
|
||||||
|
- item.stat.isreg
|
||||||
|
- item.stat.pw_name == 'root'
|
||||||
|
- item.stat.gr_name == 'root'
|
||||||
|
- item.stat.mode in ['0600', '0640']
|
||||||
|
fail_msg: >-
|
||||||
|
Values file {{ item.stat.path }} must be root-owned with restricted
|
||||||
|
permissions (0600 or 0640) and must not contain secrets.
|
||||||
|
loop: "{{ _values_files.results }}"
|
||||||
|
loop_control:
|
||||||
|
label: "{{ item.stat.path | default(item.item) }}"
|
||||||
|
|
||||||
|
- name: Build helm value arguments
|
||||||
|
ansible.builtin.set_fact:
|
||||||
|
_helm_values_args: >-
|
||||||
|
{%- for f in matrix_stack_values_files -%}
|
||||||
|
--values {{ matrix_stack_values_dir }}/{{ f }} {% endfor -%}
|
||||||
|
|
||||||
|
- name: Render the ESS chart without applying it (dry-run validation)
|
||||||
|
ansible.builtin.command:
|
||||||
|
cmd: >-
|
||||||
|
helm template {{ matrix_stack_release_name }}
|
||||||
|
{{ matrix_stack_chart_ref }}
|
||||||
|
--version {{ matrix_stack_chart_version }}
|
||||||
|
--namespace {{ matrix_namespace }}
|
||||||
|
--create-namespace
|
||||||
|
{{ _helm_values_args }}
|
||||||
|
environment:
|
||||||
|
KUBECONFIG: "{{ k3s_kubeconfig_path | default('/etc/rancher/k3s/k3s.yaml') }}"
|
||||||
|
changed_when: false
|
||||||
|
register: _chart_render
|
||||||
|
# No secrets in values, but render output may contain initSecrets-generated placeholders
|
||||||
|
|
||||||
|
- name: Verify the chart renders without errors
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- _chart_render.rc == 0
|
||||||
|
- _chart_render.stdout | length > 0
|
||||||
|
fail_msg: >-
|
||||||
|
Helm template rendering failed. Check values files for syntax errors.
|
||||||
|
Output: {{ _chart_render.stderr | default('(none)') }}
|
||||||
|
|
||||||
|
- name: Deploy the ESS chart via Helm upgrade --install
|
||||||
|
ansible.builtin.command:
|
||||||
|
cmd: >-
|
||||||
|
helm upgrade --install {{ matrix_stack_release_name }}
|
||||||
|
{{ matrix_stack_chart_ref }}
|
||||||
|
--version {{ matrix_stack_chart_version }}
|
||||||
|
--namespace {{ matrix_namespace }}
|
||||||
|
--create-namespace
|
||||||
|
{{ _helm_values_args }}
|
||||||
|
--wait
|
||||||
|
--timeout 15m
|
||||||
|
environment:
|
||||||
|
KUBECONFIG: "{{ k3s_kubeconfig_path | default('/etc/rancher/k3s/k3s.yaml') }}"
|
||||||
|
changed_when: true
|
||||||
|
register: _helm_deploy
|
||||||
|
# initSecrets may generate passwords at deploy time; those stay in-cluster only
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
---
|
||||||
|
restic_enabled: false
|
||||||
|
restic_binary: /usr/bin/restic
|
||||||
|
restic_config_path: /etc/vps-restic/repository.env
|
||||||
|
restic_state_dir: /var/lib/vps-restic
|
||||||
|
restic_log_dir: /var/log/vps-restic
|
||||||
|
restic_backup_on_calendar: '*-*-* 04:30:00'
|
||||||
|
restic_check_on_calendar: 'Sun *-*-* 05:30:00'
|
||||||
|
restic_forget_on_calendar: 'Sat *-*-* 05:30:00'
|
||||||
|
restic_keep_daily: 7
|
||||||
|
restic_keep_weekly: 4
|
||||||
|
restic_keep_monthly: 6
|
||||||
|
restic_sources:
|
||||||
|
vaultwarden:
|
||||||
|
- /opt/vaultwarden/backups
|
||||||
|
- /opt/vaultwarden/vw-data
|
||||||
|
pdns:
|
||||||
|
- /opt/pdns/backup
|
||||||
|
- /opt/pdns/auth/pdns.conf
|
||||||
|
# Mailcow is deliberately excluded pending its official consistency and restore
|
||||||
|
# design review. No repository/backend value is supplied by this project.
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
---
|
||||||
|
- name: Reload systemd
|
||||||
|
ansible.builtin.systemd_service:
|
||||||
|
daemon_reload: true
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
---
|
||||||
|
- name: Require explicit Restic opt-in
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- restic_enabled | bool
|
||||||
|
fail_msg: >-
|
||||||
|
Restic is disabled by default. Set restic_enabled=true and provision the
|
||||||
|
repository configuration only on the target host after backend approval.
|
||||||
|
|
||||||
|
- name: Validate supported Restic source profile
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- healthcheck_profile in restic_sources
|
||||||
|
fail_msg: "No approved Restic source profile for {{ healthcheck_profile }}."
|
||||||
|
|
||||||
|
- name: Verify Restic binary exists on target
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "{{ restic_binary }}"
|
||||||
|
register: restic_binary_stat
|
||||||
|
|
||||||
|
- name: Require target-side Restic binary
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that: restic_binary_stat.stat.exists
|
||||||
|
fail_msg: "Install Restic through an approved host maintenance change first."
|
||||||
|
|
||||||
|
- name: Verify target-side repository configuration exists
|
||||||
|
ansible.builtin.stat:
|
||||||
|
path: "{{ restic_config_path }}"
|
||||||
|
register: restic_config_stat
|
||||||
|
|
||||||
|
- name: Require root-only repository configuration
|
||||||
|
ansible.builtin.assert:
|
||||||
|
that:
|
||||||
|
- restic_config_stat.stat.exists
|
||||||
|
- restic_config_stat.stat.mode == '0600'
|
||||||
|
fail_msg: >-
|
||||||
|
Provision {{ restic_config_path }} directly on the host with mode 0600.
|
||||||
|
It must contain RESTIC_REPOSITORY, RESTIC_PASSWORD_FILE, and any backend
|
||||||
|
credentials; do not commit or pass them via Ansible.
|
||||||
|
|
||||||
|
- name: Install Restic state and log directories
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: "{{ item }}"
|
||||||
|
state: directory
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0750"
|
||||||
|
loop:
|
||||||
|
- "{{ restic_state_dir }}"
|
||||||
|
- "{{ restic_log_dir }}"
|
||||||
|
|
||||||
|
- name: Install Restic script directory
|
||||||
|
ansible.builtin.file:
|
||||||
|
path: /usr/local/lib/vps-restic
|
||||||
|
state: directory
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0755"
|
||||||
|
|
||||||
|
- name: Install Restic scripts
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: "{{ item }}.j2"
|
||||||
|
dest: "/usr/local/lib/vps-restic/{{ item }}"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0750"
|
||||||
|
loop:
|
||||||
|
- backup
|
||||||
|
- check
|
||||||
|
- forget-prune
|
||||||
|
|
||||||
|
- name: Install Restic systemd units and timers
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: "{{ item }}.j2"
|
||||||
|
dest: "/etc/systemd/system/{{ item }}"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: "0644"
|
||||||
|
loop:
|
||||||
|
- vps-restic-backup.service
|
||||||
|
- vps-restic-backup.timer
|
||||||
|
- vps-restic-check.service
|
||||||
|
- vps-restic-check.timer
|
||||||
|
- vps-restic-forget-prune.service
|
||||||
|
- vps-restic-forget-prune.timer
|
||||||
|
notify: Reload systemd
|
||||||
|
|
||||||
|
- name: Enable Restic timers
|
||||||
|
ansible.builtin.systemd_service:
|
||||||
|
name: "{{ item }}"
|
||||||
|
enabled: true
|
||||||
|
state: started
|
||||||
|
daemon_reload: true
|
||||||
|
loop:
|
||||||
|
- vps-restic-backup.timer
|
||||||
|
- vps-restic-check.timer
|
||||||
|
- vps-restic-forget-prune.timer
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
# Repository and password credentials are host-local in {{ restic_config_path }}.
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source '{{ restic_config_path }}'
|
||||||
|
exec '{{ restic_binary }}' backup --tag '{{ healthcheck_profile }}' --tag "$(hostname -s)" {% for source in restic_sources[healthcheck_profile] %}{{ source | quote }} {% endfor %}
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source '{{ restic_config_path }}'
|
||||||
|
exec '{{ restic_binary }}' check --read-data-subset=5%
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source '{{ restic_config_path }}'
|
||||||
|
exec '{{ restic_binary }}' forget --prune --keep-daily {{ restic_keep_daily }} --keep-weekly {{ restic_keep_weekly }} --keep-monthly {{ restic_keep_monthly }} --tag '{{ healthcheck_profile }}'
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Restic backup for approved {{ healthcheck_profile }} sources
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=root
|
||||||
|
Group=root
|
||||||
|
UMask=0077
|
||||||
|
ExecStart=/usr/local/lib/vps-restic/backup
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Daily Restic backup timer
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar={{ restic_backup_on_calendar }}
|
||||||
|
Persistent=true
|
||||||
|
RandomizedDelaySec=20m
|
||||||
|
Unit=vps-restic-backup.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Restic repository integrity check
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=root
|
||||||
|
Group=root
|
||||||
|
UMask=0077
|
||||||
|
ExecStart=/usr/local/lib/vps-restic/check
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Weekly Restic integrity check timer
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar={{ restic_check_on_calendar }}
|
||||||
|
Persistent=true
|
||||||
|
RandomizedDelaySec=30m
|
||||||
|
Unit=vps-restic-check.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Restic retention and prune
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=root
|
||||||
|
Group=root
|
||||||
|
UMask=0077
|
||||||
|
ExecStart=/usr/local/lib/vps-restic/forget-prune
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Weekly Restic retention timer
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar={{ restic_forget_on_calendar }}
|
||||||
|
Persistent=true
|
||||||
|
RandomizedDelaySec=30m
|
||||||
|
Unit=vps-restic-forget-prune.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
# Matrix (Element Server Suite) Upstream References
|
||||||
|
|
||||||
|
Matrix homeserver deployed via **Element Server Suite (ESS) Community** OCI chart.
|
||||||
|
|
||||||
|
## Chart
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|---|---|
|
||||||
|
| Chart name | `matrix-stack` |
|
||||||
|
| OCI registry | `oci://ghcr.io/element-hq/ess-helm/matrix-stack` |
|
||||||
|
| Version deployed | `26.7.2` |
|
||||||
|
| Chart source | <https://github.com/element-hq/ess-helm> |
|
||||||
|
| Documentation | <https://element-hq.github.io/ess-helm/> |
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
| Component | Docker image | Role |
|
||||||
|
|---|---|---|
|
||||||
|
| Synapse | `ghcr.io/element-hq/synapse:*` | Matrix homeserver |
|
||||||
|
| MAS | `ghcr.io/element-hq/matrix-authentication-service:*` | OIDC/OAuth authentication |
|
||||||
|
| Element Web | `ghcr.io/element-hq/element-web:*` | Web client |
|
||||||
|
| Element Admin | `ghcr.io/element-hq/element-admin:*` | Admin console |
|
||||||
|
| HAProxy | `haproxy:*` | Internal routing/health |
|
||||||
|
| PostgreSQL | `postgres:17` | Synapse + MAS database |
|
||||||
|
|
||||||
|
All images should be locked via digest in production; the chart's `appVersion` may not match default image tags.
|
||||||
|
|
||||||
|
## Domain configuration
|
||||||
|
|
||||||
|
| Hostname | Component |
|
||||||
|
|---|---|
|
||||||
|
| `chans.xyz` | Matrix identity (`serverName`) + well-known discovery |
|
||||||
|
| `synapse.chans.xyz` | Client-Server API, Federation API |
|
||||||
|
| `chat.chans.xyz` | Element Web |
|
||||||
|
| `account.chans.xyz` | Matrix Authentication Service (local passwords) |
|
||||||
|
| `admin.chans.xyz` | Element Admin (MAS admin auth) |
|
||||||
|
| `mrtc.chans.xyz` | Reserved for MatrixRTC — **not deployed** |
|
||||||
|
|
||||||
|
## TLS & Ingress
|
||||||
|
|
||||||
|
- Ingress controller: K3s built-in Traefik (IngressClass `traefik`)
|
||||||
|
- Certificate issuer: cert-manager + Let's Encrypt HTTP-01 (`letsencrypt-prod` ClusterIssuer)
|
||||||
|
- All TLS enabled: `global.ingress.tls.enabled=true`
|
||||||
|
- Issuer annotation: `cert-manager.io/cluster-issuer: letsencrypt-prod`
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Internet :80/:443
|
||||||
|
→ K3s Traefik (hostPort)
|
||||||
|
→ ESS Ingresses (Traefik class)
|
||||||
|
→ ClusterIP Services
|
||||||
|
→ Pods (Synapse, MAS, Element, Admin, HAProxy)
|
||||||
|
→ PostgreSQL (StatefulSet, 20Gi PVC)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
The ESS chart is deployed via Ansible (`ansible/playbooks/matrix-stack.yml`) with non-secret values at `/etc/ess/*.yaml` on the target host.
|
||||||
|
|
||||||
|
**Secrets** (signing key, database passwords, SMTP password, MAS keys) are **not** in Git. They are stored:
|
||||||
|
1. On the VPS at `/etc/matrix-bootstrap/` (root:root 0600 files)
|
||||||
|
2. In pre-created Kubernetes Secrets in the `ess` namespace
|
||||||
|
|
||||||
|
## Upgrades
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Pull latest chart version
|
||||||
|
helm pull oci://ghcr.io/element-hq/ess-helm/matrix-stack --version <new-version>
|
||||||
|
|
||||||
|
# Review changes
|
||||||
|
helm diff upgrade ess oci://ghcr.io/element-hq/ess-helm/matrix-stack \
|
||||||
|
-n ess -f /etc/ess/hostnames.yaml -f /etc/ess/tls.yaml
|
||||||
|
|
||||||
|
# Apply
|
||||||
|
helm upgrade --install ess oci://ghcr.io/element-hq/ess-helm/matrix-stack \
|
||||||
|
-n ess -f /etc/ess/hostnames.yaml -f /etc/ess/tls.yaml --wait
|
||||||
|
```
|
||||||
|
|
||||||
|
Before upgrading:
|
||||||
|
- Matrix backup automation is currently paused; establish and test a reviewed backup design before treating the deployment as recoverable.
|
||||||
|
- Review chart changelog and database migration notes
|
||||||
|
- Test in isolation if possible
|
||||||
|
|
||||||
|
## Recovery
|
||||||
|
|
||||||
|
When the planned backup automation is enabled, recovery will require the following from `/var/backups/matrix/`:
|
||||||
|
|
||||||
|
1. PostgreSQL dumps (synapse + mas)
|
||||||
|
2. Media store archive
|
||||||
|
3. Bootstrap secrets archive (`/etc/matrix-bootstrap/`)
|
||||||
|
4. Non-secret values files (`/etc/ess/*.yaml`)
|
||||||
|
5. Non-secret playbook templates (from this repo)
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy K3s + Traefik + cert-manager per Ansible
|
||||||
|
2. Restore bootstrap secrets + create Kubernetes Secrets
|
||||||
|
3. Restore PostgreSQL dumps
|
||||||
|
4. Restore media store to PVC
|
||||||
|
5. Deploy ESS chart with same values
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- ESS Helm docs: <https://element-hq.github.io/ess-helm/>
|
||||||
|
- ESS Community setup: <https://github.com/element-hq/element-server-suite>
|
||||||
|
- Synapse: <https://github.com/element-hq/synapse>
|
||||||
|
- Matrix Authentication Service: <https://github.com/element-hq/matrix-authentication-service>
|
||||||
|
- Matrix spec: <https://spec.matrix.org/>
|
||||||
|
- Federation tester: <https://federationtester.matrix.org/>
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# PowerDNS Authoritative — upstream documentation
|
||||||
|
|
||||||
|
Use these when changing `/opt/pdns` on [hk2](../hosts/hk2.chans.xyz.md). Prefer official docs over blog posts.
|
||||||
|
|
||||||
|
## Primary references
|
||||||
|
|
||||||
|
| Topic | URL |
|
||||||
|
|-------|-----|
|
||||||
|
| Settings reference | https://doc.powerdns.com/authoritative/settings.html |
|
||||||
|
| Primary / secondary modes | https://doc.powerdns.com/authoritative/modes-of-operation.html |
|
||||||
|
| Generic PostgreSQL backend | https://doc.powerdns.com/authoritative/backends/generic-postgresql.html |
|
||||||
|
| DNSSEC ops + SOA-EDIT | https://doc.powerdns.com/authoritative/dnssec/operational.html |
|
||||||
|
| DNSSEC advice | https://doc.powerdns.com/authoritative/dnssec/advice.html |
|
||||||
|
| TSIG | https://doc.powerdns.com/authoritative/tsig.html |
|
||||||
|
| HTTP API | https://doc.powerdns.com/authoritative/http-api/ |
|
||||||
|
| Security advisories | https://doc.powerdns.com/authoritative/security-advisories/ |
|
||||||
|
| Docker image (`pdns-auth-50`) | https://hub.docker.com/r/powerdns/pdns-auth-50 |
|
||||||
|
|
||||||
|
## Settings we care about on hk2 (ns1)
|
||||||
|
|
||||||
|
| Concern | Upstream guidance | Our notes (post-2026-08-01) |
|
||||||
|
|---------|-------------------|-----------------------------|
|
||||||
|
| Version | Track advisories; upgrade when Mandatory | Pinned **`powerdns/pdns-auth-50:5.0.6`** (clears 2026-06 + 2026-07) |
|
||||||
|
| Primary | `primary=yes`; zones type MASTER/PRIMARY | Set; zones `windy.me` / `wsvc.info` / `chans.xyz` are Master |
|
||||||
|
| NOTIFY | `also-notify` always notified; empty `only-notify` disables NS-based NOTIFY | `also-notify=202.91.35.141`, `only-notify=` |
|
||||||
|
| AXFR IP ACL | `allow-axfr-ips` limits **unsigned** AXFR | Global + per-zone `ALLOW-AXFR-FROM` → ns2 only |
|
||||||
|
| TSIG AXFR | **Any host with the TSIG key may AXFR**, bypassing IP ACL | Zones use `TSIG-ALLOW-AXFR=mykey.` — rotate/rename key; protect secret |
|
||||||
|
| DNSSEC | `gpgsql-dnssec=yes` + keys via `pdnsutil` | CSK ECDSAP256SHA256 on all three zones |
|
||||||
|
| SOA-EDIT | Needed so non-PDNS secondaries see signature rolls; PDNS secondaries can also compare RRSIGs | `default-soa-edit(-signed)=INCEPTION-INCREMENT` OK (ns2 is PowerDNS). Dig SOA ≠ backend serial is expected |
|
||||||
|
| API key | Prefer `pdnsutil hash-password` (plaintext still accepted) | **Deferred hardening:** live key is weak, plaintext, reused with DB password — rotate |
|
||||||
|
| Webserver | Prefer trusted clients only | Host publishes `127.0.0.1:8081`; `webserver-allow-from` private ranges |
|
||||||
|
| `gpgsql-host` | Prefer IP to avoid resolve chicken/egg | `db` (Compose DNS) is fine inside the stack |
|
||||||
|
| `domains.master` | Used for SECONDARY upstreams | Cleared on MASTER zones (was self-IP noise) |
|
||||||
|
|
||||||
|
## Intended `auth/pdns.conf` shape (no secrets)
|
||||||
|
|
||||||
|
Live file: `/opt/pdns/auth/pdns.conf`. Secrets come from `auth/templates.d/secrets.j2` → `/etc/powerdns/pdns.d/secrets.conf` at container start.
|
||||||
|
|
||||||
|
```
|
||||||
|
local-address=0.0.0.0
|
||||||
|
local-port=53
|
||||||
|
|
||||||
|
launch=gpgsql
|
||||||
|
gpgsql-host=db
|
||||||
|
gpgsql-dnssec=yes
|
||||||
|
include-dir=/etc/powerdns/pdns.d
|
||||||
|
|
||||||
|
allow-axfr-ips=202.91.35.141
|
||||||
|
also-notify=202.91.35.141
|
||||||
|
only-notify=
|
||||||
|
primary=yes
|
||||||
|
secondary=no
|
||||||
|
|
||||||
|
api=yes
|
||||||
|
webserver=yes
|
||||||
|
webserver-address=0.0.0.0
|
||||||
|
webserver-port=8081
|
||||||
|
webserver-allow-from=127.0.0.1,172.16.0.0/12,10.0.0.0/8,192.168.0.0/16
|
||||||
|
|
||||||
|
version-string=anonymous
|
||||||
|
disable-syslog=yes
|
||||||
|
loglevel=4
|
||||||
|
|
||||||
|
default-soa-edit=INCEPTION-INCREMENT
|
||||||
|
default-soa-edit-signed=INCEPTION-INCREMENT
|
||||||
|
|
||||||
|
disable-axfr=no
|
||||||
|
```
|
||||||
|
|
||||||
|
Drop-in (from `.env`, never commit): `gpgsql-dbname`, `gpgsql-user`, `gpgsql-password`, `api-key`.
|
||||||
|
|
||||||
|
## Deferred hardening (not done yet)
|
||||||
|
|
||||||
|
1. Rotate `PDNS_API_KEY` to a long random value; store hashed via `pdnsutil hash-password` if desired; update Poweradmin `.env` / recreate `auth` + `poweradmin`.
|
||||||
|
2. Rotate `DB_PASS` / Postgres role separately from the API key.
|
||||||
|
3. Rotate TSIG `mykey` (or rename); confirm ns2 still AXFRs; remember TSIG bypasses `allow-axfr-ips`.
|
||||||
|
|
||||||
|
## Related ops docs
|
||||||
|
|
||||||
|
- Host facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md)
|
||||||
|
- Health runbook: [runbooks/pdns-health.md](../runbooks/pdns-health.md)
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# Vaultwarden — upstream documentation (source of truth)
|
||||||
|
|
||||||
|
Use these when changing `/opt/vaultwarden` on [us2](../hosts/us2.wsvc.info.md). Prefer upstream over blog posts or Obsidian notes.
|
||||||
|
|
||||||
|
## Primary references
|
||||||
|
|
||||||
|
| Topic | URL |
|
||||||
|
|-------|-----|
|
||||||
|
| Configuration overview (methods + precedence) | https://github.com/dani-garcia/vaultwarden/wiki/Configuration-overview |
|
||||||
|
| Env / options template (canonical variable list) | https://github.com/dani-garcia/vaultwarden/blob/main/.env.template |
|
||||||
|
| Defaults / implementation source | https://github.com/dani-garcia/vaultwarden/blob/main/src/config.rs |
|
||||||
|
| PostgreSQL backend + SQLite migration | https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend |
|
||||||
|
| SMTP | https://github.com/dani-garcia/vaultwarden/wiki/SMTP-configuration |
|
||||||
|
| Reverse proxy examples (incl. Traefik) | https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples |
|
||||||
|
| Enabling admin page / `ADMIN_TOKEN` | https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page |
|
||||||
|
| Disable registration / invitations | https://github.com/dani-garcia/vaultwarden/wiki/Disable-registration-of-new-users |
|
||||||
|
| pgloader BIGINT login failure note | https://github.com/dani-garcia/vaultwarden/discussions/7134 |
|
||||||
|
| Project wiki home | https://github.com/dani-garcia/vaultwarden/wiki |
|
||||||
|
| Container image | https://github.com/dani-garcia/vaultwarden/pkgs/container/server |
|
||||||
|
|
||||||
|
## Configuration precedence (official)
|
||||||
|
|
||||||
|
From [Configuration overview](https://github.com/dani-garcia/vaultwarden/wiki/Configuration-overview):
|
||||||
|
|
||||||
|
1. Compile-time defaults (`src/config.rs`)
|
||||||
|
2. Vaultwarden `ENV_FILE` / `.env` read by the process (if used)
|
||||||
|
3. Process environment variables (e.g. Docker Compose `env_file` + `environment`) — override (2)
|
||||||
|
4. **`data/config.json` from `/admin` — highest precedence** (overrides env)
|
||||||
|
|
||||||
|
Official stance: prefer environment variables; `config.json` is **not** the recommended long-term method. Once admin saves `config.json`, those keys stop being changeable via `.env` until removed/edited in admin or the file.
|
||||||
|
|
||||||
|
## Settings we care about on us2
|
||||||
|
|
||||||
|
| Concern | Upstream guidance | Our notes (post-2026-08-01 migrate) |
|
||||||
|
|---------|-------------------|-------------------------------------|
|
||||||
|
| Public URL | Set `DOMAIN` e.g. `https://auth.wsvc.info` (no trailing slash) | Set in compose + `config.json` |
|
||||||
|
| SMTP | `587`+`starttls` or `465`+`force_tls` | Effective: **587/starttls** to mx2 (`465` unreachable from us2); `extra_hosts` pins A `194.163.160.244` |
|
||||||
|
| SMTP password | Same mailbox password as mailcow | Must match across `.env`, `config.json`, `.smtp-credentials`. **Admin/`config.json` wins** — stale admin password → SMTP `535` while `.env` still works |
|
||||||
|
| Database | `DATABASE_URL` for Postgres | Live: Postgres via compose `DATABASE_URL`; sqlite cold under `backups/sqlite-cold/` |
|
||||||
|
| Client IP behind proxy | Traefik → prefer `X-Forwarded-For` | `ip_header=X-Forwarded-For` in `config.json` |
|
||||||
|
| Admin | Argon2 `ADMIN_TOKEN` | On server only |
|
||||||
|
| Signups | Invite-only on public URL | `signups_allowed=false` |
|
||||||
|
|
||||||
|
## Agent checklist before edits
|
||||||
|
|
||||||
|
1. Read this page + [us2 host facts](../hosts/us2.wsvc.info.md).
|
||||||
|
2. On the server, compare `.env`, compose `environment`, and `/data/config.json` (including SMTP password fingerprint, not plaintext in chat/logs).
|
||||||
|
3. Change the layer that actually wins (often `config.json`); if rotating the mailbox password, update **all three** secret stores.
|
||||||
|
4. After recreating the `vaultwarden` container, restart Traefik if `auth.wsvc.info` 404s.
|
||||||
|
5. Do not commit secrets; do not copy passwords from Obsidian into this repo.
|
||||||
|
6. Verify SMTP with AUTH (not TCP-only) — [vaultwarden-health](../runbooks/vaultwarden-health.md) §5.
|
||||||
|
|
||||||
|
Replay migration steps: [runbooks/vaultwarden-sqlite-to-postgres.md](../runbooks/vaultwarden-sqlite-to-postgres.md).
|
||||||
@@ -0,0 +1,138 @@
|
|||||||
|
# hk2.chans.xyz
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|------|--------|
|
||||||
|
| Role | PowerDNS Authoritative (ns1) + Poweradmin / Traefik |
|
||||||
|
| SSH | `ssh -4 windy@hk2.chans.xyz` (prefer IPv4 from WSL) |
|
||||||
|
| IPv4 | `154.36.174.161` |
|
||||||
|
| Hostname (OS) | `C20210714091792` (Debian 13 trixie) |
|
||||||
|
| Also DNS | `ns1.wsvc.info` → this host |
|
||||||
|
| Secondary NS | `ns2.wsvc.info` → `202.91.35.141` (`us1.wsvc.info`, PowerDNS Auth **5.0.5**) |
|
||||||
|
| Install path | `/opt/pdns` |
|
||||||
|
| Compose | `/opt/pdns/compose.yml` |
|
||||||
|
| Env file | `/opt/pdns/.env` (secrets — **never commit**) |
|
||||||
|
| TZ | `Asia/Shanghai` |
|
||||||
|
|
||||||
|
Upstream docs / config checklist: [docs/pdns-upstream.md](../docs/pdns-upstream.md)
|
||||||
|
|
||||||
|
## PowerDNS Authoritative
|
||||||
|
|
||||||
|
**Status: operational** (auth + db healthy, API **5.0.6**, DNS on :53 — last config audit 2026-08-01 21:40 CST).
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|------|--------|
|
||||||
|
| Image | `powerdns/pdns-auth-50:5.0.6` (pinned) |
|
||||||
|
| Container | `pdns-auth` |
|
||||||
|
| Config | `/opt/pdns/auth/pdns.conf` + `auth/templates.d/secrets.j2` → `pdns.d/secrets.conf` |
|
||||||
|
| Backend | PostgreSQL 16 (`launch=gpgsql`, DB name `pdns`) |
|
||||||
|
| DNS ports | host `53/tcp` + `53/udp` |
|
||||||
|
| API | host `127.0.0.1:8081` only; `webserver-allow-from` private ranges |
|
||||||
|
| Mode | `primary=yes`; `also-notify` / AXFR ACL → `202.91.35.141`; `only-notify=` empty |
|
||||||
|
| DNSSEC | `gpgsql-dnssec=yes`; CSK ECDSAP256SHA256 |
|
||||||
|
| SOA-EDIT | `default-soa-edit(-signed)=INCEPTION-INCREMENT` |
|
||||||
|
| Poweradmin | `PDNS_VERSION=50`; Traefik Host(`pdns.wsvc.info`) |
|
||||||
|
| Advisories | 2026-06 + 2026-07 cleared by 5.0.6 |
|
||||||
|
|
||||||
|
### Zones (authoritative)
|
||||||
|
|
||||||
|
| Zone | Kind | DNSSEC | Zone metadata |
|
||||||
|
|------|------|--------|---------------|
|
||||||
|
| `windy.me` | Master | yes | `ALLOW-AXFR-FROM` + `TSIG-ALLOW-AXFR=mykey.` |
|
||||||
|
| `wsvc.info` | Master | yes | same |
|
||||||
|
| `chans.xyz` | Master | yes | same |
|
||||||
|
|
||||||
|
NS set: `ns1.wsvc.info` / `ns2.wsvc.info`. Dig SOA may differ from backend serial (SOA-EDIT) — expected.
|
||||||
|
|
||||||
|
### Intended auth config (sanitized)
|
||||||
|
|
||||||
|
See full shape in [docs/pdns-upstream.md](../docs/pdns-upstream.md). Live secrets only in `.env` / generated drop-in on the server.
|
||||||
|
|
||||||
|
### Stack
|
||||||
|
|
||||||
|
| Container | Service | Status | Image |
|
||||||
|
|-----------|---------|--------|-------|
|
||||||
|
| `pdns-auth` | auth | Up (healthy) | `powerdns/pdns-auth-50:5.0.6` |
|
||||||
|
| `pdns-db` | db | Up (healthy) | `postgres:16` |
|
||||||
|
| `poweradmin` | poweradmin | Up (healthy) | `poweradmin/poweradmin:stable` |
|
||||||
|
| `pdns_pgweb` | pgweb | Up | `sosedoff/pgweb:0.16.2` |
|
||||||
|
| `pdns-backup` | backup | Up | `postgres:16` (scheduler) |
|
||||||
|
| `powerdns-admin` | *(orphan)* | Exited | legacy PDA UI — not in active compose |
|
||||||
|
|
||||||
|
### Network model
|
||||||
|
|
||||||
|
| Network | Purpose |
|
||||||
|
|---------|---------|
|
||||||
|
| `backend` | Internal (`internal: true`) — DB + service API |
|
||||||
|
| `edge` | Host-published DNS `:53` and loopback API `:8081` |
|
||||||
|
| `frontend` (`traefik`) | External Traefik network for web UIs |
|
||||||
|
|
||||||
|
### Public / management URLs
|
||||||
|
|
||||||
|
| URL | Backend | Notes |
|
||||||
|
|-----|---------|-------|
|
||||||
|
| https://pdns.wsvc.info | `poweradmin` via Traefik | DNS admin UI |
|
||||||
|
| https://pgweb.wsvc.info | `pgweb` via Traefik | DB browser (HTTP basic) |
|
||||||
|
| DNS `ns1.wsvc.info:53` | `pdns-auth` | Public authoritative |
|
||||||
|
|
||||||
|
## Config audit (vs official docs)
|
||||||
|
|
||||||
|
**Functional primary/secondary config: OK** after 5.0.6 upgrade + `only-notify=` / `domains.master` cleanup.
|
||||||
|
|
||||||
|
| Item | Verdict |
|
||||||
|
|------|---------|
|
||||||
|
| `primary` + MASTER zones + gpgsql DNSSEC | OK |
|
||||||
|
| `also-notify` + empty `only-notify` | OK (docs-recommended pattern) |
|
||||||
|
| API bind / host port publish | OK |
|
||||||
|
| SOA-EDIT with PowerDNS secondary | OK |
|
||||||
|
| API key / DB password hygiene | **Deferred** — weak, plaintext, reused |
|
||||||
|
| TSIG `mykey` | **Deferred** — any holder can AXFR (bypasses IP ACL) |
|
||||||
|
|
||||||
|
## Backup
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|------|--------|
|
||||||
|
| Service | `pdns-backup` |
|
||||||
|
| Schedule | `CRON_SCHEDULE=0 3 * * *` (03:00 Asia/Shanghai) |
|
||||||
|
| Retention | `RETENTION_DAYS=7`, `MAX_BACKUPS=3` |
|
||||||
|
| Output | `/opt/pdns/backup/pdns_*.sql.gz` (+ optional `roles_*.sql`) |
|
||||||
|
| Scheduler | `scripts/backup-scheduler.sh` (daily `M H * * *` only) |
|
||||||
|
| Manual | `cd /opt/pdns && ./scripts/manual-backup.sh` |
|
||||||
|
|
||||||
|
**Quirk:** `backend` is internal — backup must not use Alpine + runtime `apk`/`crond`. Uses `postgres:16` + `backup-scheduler.sh` (fixed 2026-08-01).
|
||||||
|
|
||||||
|
## Other software on this host (stubs)
|
||||||
|
|
||||||
|
`/opt/traefik`, `/opt/adguard`, `/opt/remark42`, `/opt/rustdesk`, `/opt/nginx-manager`, …
|
||||||
|
|
||||||
|
## Ops / runbooks
|
||||||
|
|
||||||
|
- [pdns-health](../runbooks/pdns-health.md)
|
||||||
|
- [pdns-upstream docs](../docs/pdns-upstream.md)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@hk2.chans.xyz
|
||||||
|
cd /opt/pdns && docker compose ps -a
|
||||||
|
docker compose logs -f auth poweradmin backup
|
||||||
|
./scripts/manual-backup.sh
|
||||||
|
# after pdns.conf / secrets.j2 / auth env changes:
|
||||||
|
docker compose up -d --force-recreate auth
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dig @154.36.174.161 NS windy.me +short
|
||||||
|
dig @154.36.174.161 SOA wsvc.info +short
|
||||||
|
# compare secondary:
|
||||||
|
dig @202.91.35.141 SOA wsvc.info +short
|
||||||
|
```
|
||||||
|
|
||||||
|
On-server docs: `/opt/pdns/README.md`, `CHANGELOG.md`.
|
||||||
|
|
||||||
|
## Verified
|
||||||
|
|
||||||
|
Last checked: **2026-08-01 21:40 CST** — operational; docs audit recorded.
|
||||||
|
|
||||||
|
- Auth **5.0.6** healthy; no Mandatory security banner
|
||||||
|
- Zones Master + DNSSEC; SOA in sync with ns2 (served serial)
|
||||||
|
- `only-notify=` + `also-notify=202.91.35.141`; MASTER `domains.master` cleared
|
||||||
|
- https://pdns.wsvc.info → **302**; https://pgweb.wsvc.info → **401**
|
||||||
|
- Hardening backlog: API/DB credential rotation + TSIG rotate (see upstream doc)
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
# mx2.windy.me
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|------|--------|
|
||||||
|
| Role | Mail server (mailcow) |
|
||||||
|
| SSH | `ssh -4 windy@mx2.windy.me` (pubkey; root denied; prefer IPv4 from WSL) |
|
||||||
|
| Install path | `/opt/mail` |
|
||||||
|
| Compose project | `cow` (`COMPOSE_PROJECT_NAME=cow`) |
|
||||||
|
| Compose file | `/opt/mail/docker-compose.yml` |
|
||||||
|
| Config | `/opt/mail/mailcow.conf` (`.env` → symlink) |
|
||||||
|
| Hostname | `MAILCOW_HOSTNAME=mx2.windy.me` |
|
||||||
|
| TZ | `Asia/Shanghai` |
|
||||||
|
| IPv4 | `194.163.160.244` |
|
||||||
|
| IPv6 | `2a02:c207:2284:8258::1` |
|
||||||
|
| Docker network | `IPV4_NETWORK=172.22.1` |
|
||||||
|
| UI | https://mx2.windy.me |
|
||||||
|
|
||||||
|
## Ports (host-published)
|
||||||
|
|
||||||
|
`25`, `465`, `587` (SMTP), `110`/`995` (POP), `143`/`993` (IMAP), `4190` (Sieve), `80`/`443` (UI)
|
||||||
|
|
||||||
|
## Domain & mailboxes
|
||||||
|
|
||||||
|
- Domain: `windy.me` (primary on this host)
|
||||||
|
- MX for `windy.me`: `mx2.windy.me` (prio 20), `mx.windy.me` (prio 30)
|
||||||
|
- Mailboxes (examples): `zhiqiang@windy.me`, `claw@windy.me`, `hermes@windy.me`
|
||||||
|
- ACME / Let’s Encrypt enabled (`SKIP_LETS_ENCRYPT=n`); cert SAN includes `mx2`, `autoconfig`, `autodiscover`
|
||||||
|
- PTR (v4/v6) → `mx2.windy.me`
|
||||||
|
|
||||||
|
## Ops notes
|
||||||
|
|
||||||
|
- Manage with: `cd /opt/mail && docker compose …`
|
||||||
|
- `SKIP_CLAMD=y` may still leave `clamd` running after updates; recreate if you need it truly skipped
|
||||||
|
- Host has ~8G RAM, no swap; watch memory under load
|
||||||
|
- Unrelated stopped containers may exist (`zc-*` monitoring stack) — not part of mailcow
|
||||||
|
|
||||||
|
## Client access (SMTP / IMAP)
|
||||||
|
|
||||||
|
See [mailcow-smtp-client](../runbooks/mailcow-smtp-client.md).
|
||||||
|
|
||||||
|
- Send: `mx2.windy.me:587` STARTTLS (or `:465` SSL), auth = full email + mailbox password
|
||||||
|
- IMAP: `mx2.windy.me:993` · POP: `:995`
|
||||||
|
- Webmail: https://mx2.windy.me
|
||||||
|
|
||||||
|
## Runbooks
|
||||||
|
|
||||||
|
- [mailcow-health](../runbooks/mailcow-health.md)
|
||||||
|
- [mailcow-update](../runbooks/mailcow-update.md)
|
||||||
|
- [mailcow-smtp-client](../runbooks/mailcow-smtp-client.md)
|
||||||
|
- Script: [`../scripts/check-mx2.sh`](../scripts/check-mx2.sh)
|
||||||
|
|
||||||
|
## Verified
|
||||||
|
|
||||||
|
Last checked: 2026-08-01 — `./scripts/check-mx2.sh` exit 0 (compose up, watchdog 100%, empty queue, TLS LE OK, SMTP Postcow, A/AAAA/PTR/MX/SPF OK).
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
# synapse.chans.xyz — Matrix homeserver (ESS)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
Matrix homeserver running **Element Server Suite (ESS) Community**: Synapse + Matrix Authentication Service (MAS) + Element Web + Element Admin.
|
||||||
|
|
||||||
|
## SSH
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh -4 windy@synapse.chans.xyz
|
||||||
|
```
|
||||||
|
|
||||||
|
- User: `windy` (sudo via Ansible)
|
||||||
|
- IPv4: `169.58.86.13`
|
||||||
|
- OS: Ubuntu 26.04
|
||||||
|
|
||||||
|
## Stack
|
||||||
|
|
||||||
|
| Layer | Detail |
|
||||||
|
|-------|--------|
|
||||||
|
| Runtime | K3s v1.36.2+k3s1 (single-node) |
|
||||||
|
| Ingress | K3s built-in Traefik (hostPort `:80` / `:443`) |
|
||||||
|
| TLS | cert-manager v1.19.3 + `letsencrypt-prod` ClusterIssuer (HTTP-01) |
|
||||||
|
| DB | ESS embedded PostgreSQL 17 (PVC 20Gi, local-path) |
|
||||||
|
| Cache | ESS embedded Redis (PVC 2Gi) |
|
||||||
|
| Chart | `oci://ghcr.io/element-hq/ess-helm/matrix-stack`, version `26.7.2` |
|
||||||
|
|
||||||
|
### Matrix service endpoints
|
||||||
|
|
||||||
|
| URL | Component |
|
||||||
|
|-----|-----------|
|
||||||
|
| https://chat.chans.xyz | Element Web |
|
||||||
|
| https://synapse.chans.xyz | Synapse Client-Server / Federation API |
|
||||||
|
| https://account.chans.xyz | Matrix Authentication Service |
|
||||||
|
| https://admin.chans.xyz | Element Admin console |
|
||||||
|
| https://chans.xyz/.well-known/matrix/server | Federation discovery → `synapse.chans.xyz:443` |
|
||||||
|
| https://chans.xyz/.well-known/matrix/client | Client discovery → `https://synapse.chans.xyz` |
|
||||||
|
| `mrtc.chans.xyz` | Reserved for MatrixRTC (not deployed) |
|
||||||
|
|
||||||
|
## Public ports
|
||||||
|
|
||||||
|
| Port | Protocol | Service |
|
||||||
|
|------|----------|---------|
|
||||||
|
| 22/tcp | SSH | OpenSSH |
|
||||||
|
| 80/tcp | HTTP | Let's Encrypt HTTP-01 / Traefik redirect |
|
||||||
|
| 443/tcp | HTTPS | Matrix Client-Server, Federation, Element, MAS, Admin |
|
||||||
|
|
||||||
|
All other ports internal only (no K3s API, no database, no Redis exposed).
|
||||||
|
|
||||||
|
## Namespace layout
|
||||||
|
|
||||||
|
- `ess` — all ESS workloads (Synapse, MAS, Element, Postgres, Redis, HAProxy)
|
||||||
|
- `matrix-system` — cluster base resources (ResourceQuota, LimitRange, mrtc-placeholder)
|
||||||
|
- `cert-manager` — cert-manager
|
||||||
|
|
||||||
|
## Local backup
|
||||||
|
|
||||||
|
| Item | Detail |
|
||||||
|
|------|--------|
|
||||||
|
| Path | `/var/backups/matrix` (root:root, 0700) |
|
||||||
|
| Schedule | **Paused** — `matrix-backup.timer` is disabled pending a reviewed backup design |
|
||||||
|
| Retention | 7 days |
|
||||||
|
| Disk warning | 80% (healthcheck), 90% (backup stops) |
|
||||||
|
| Content | Planned: PostgreSQL `synapse` + `mas` logical dumps, media store archive, `/etc/matrix-bootstrap` |
|
||||||
|
| Status | **Not operational** — no current Matrix backup or recovery tier |
|
||||||
|
|
||||||
|
## Health checks
|
||||||
|
|
||||||
|
Daily via `matrix-k3s-healthcheck.timer`, outputs to `/var/lib/vps-health/latest.json`. Checks:
|
||||||
|
|
||||||
|
- K3s API / namespace health
|
||||||
|
- ESS pod restart counts
|
||||||
|
- Backup freshness
|
||||||
|
- Root disk usage
|
||||||
|
- Certificate expiry
|
||||||
|
- External HTTPS endpoints
|
||||||
|
|
||||||
|
## Bootstrap secrets
|
||||||
|
|
||||||
|
Path: `/etc/matrix-bootstrap/` (root:root, 0700) — individual files per secret key (0600). SMTP password, DB passwords, signing key etc. Included in local backup.
|
||||||
|
|
||||||
|
## Identity
|
||||||
|
|
||||||
|
- **Server name**: `chans.xyz` (user IDs: `@user:chans.xyz`)
|
||||||
|
- **Deployment type**: New instance (no recovery from previous homeserver)
|
||||||
|
- **Public registration**: Disabled
|
||||||
|
- **Authentication**: MAS local passwords
|
||||||
|
|
||||||
|
## Ansible
|
||||||
|
|
||||||
|
Inventory group: `matrix_production` (but NOT in `managed` which expects Docker Compose).
|
||||||
|
|
||||||
|
Playbooks in `ansible/playbooks/matrix-*.yml`.
|
||||||
|
|
||||||
|
## Known limitations
|
||||||
|
|
||||||
|
- MatrixRTC / Element Call / LiveKit / Coturn not deployed (`mrtc.chans.xyz` reserved only)
|
||||||
|
- SMTP email not yet configured (requires manual bootstrap + Helm upgrade)
|
||||||
|
- No off-site Restic backup
|
||||||
|
- Single-node K3s (no HA for control plane)
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
# us2.wsvc.info
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|------|--------|
|
||||||
|
| Role | Multi-service VPS (Vaultwarden, Traefik, Soft Serve, …) |
|
||||||
|
| SSH | `ssh -4 windy@us2.wsvc.info` (prefer IPv4 from WSL) |
|
||||||
|
| IPv4 | `193.9.44.165` |
|
||||||
|
| Also DNS | `auth.wsvc.info` → this host; `repo.windy.me` → this host (Soft Serve) |
|
||||||
|
| Public HTTPS | Traefik on `:80` / `:443` (`/opt/traefik`) |
|
||||||
|
|
||||||
|
## Vaultwarden (Bitwarden-compatible)
|
||||||
|
|
||||||
|
**Status: operational** (Postgres live, HTTPS 200, healthy containers, SMTP AUTH OK — last probe 2026-08-01 18:55 CST).
|
||||||
|
|
||||||
|
Upstream docs: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|------|--------|
|
||||||
|
| Install path | `/opt/vaultwarden` |
|
||||||
|
| Compose | `/opt/vaultwarden/docker-compose.yml` |
|
||||||
|
| Env file | `/opt/vaultwarden/.env` |
|
||||||
|
| Admin overrides | `/opt/vaultwarden/vw-data/config.json` (**wins over env**) |
|
||||||
|
| Public URL / `DOMAIN` | `https://auth.wsvc.info` |
|
||||||
|
| Image | `vaultwarden/server:1.37.1` (pinned) |
|
||||||
|
| Live DB | **Postgres 16** (`vw-db` / service `pg`) via compose `DATABASE_URL` |
|
||||||
|
| Data (probe) | users=1, ciphers=1327 |
|
||||||
|
| Cold SQLite | `backups/sqlite-cold/db.sqlite3.pre-pg-20260801` (not used live) |
|
||||||
|
| Pre-migrate backup | `backups/pre-pg-migrate-20260801_161204/` |
|
||||||
|
| Data dir | `./vw-data` → `/data` (attachments, rsa keys, `config.json`) |
|
||||||
|
| DB backups | `vaultwarden-backup` → `pg_dump` daily 02:00 (`scripts/backup.sh`); latest e.g. `backups/vaultwarden_2026-08-01_16-23-03.sql.gz` |
|
||||||
|
| Reverse proxy | Traefik (`vw-net`) → Host(`auth.wsvc.info`), LE (`auth.wsvc.info`, valid → 2026-10-08) |
|
||||||
|
| SMTP | `mx2.windy.me:587` STARTTLS (`extra_hosts` → 194.163.160.244); from `zhiqiang@windy.me` |
|
||||||
|
| SMTP secrets | Keep **`.env`**, **`vw-data/config.json`**, and **`.smtp-credentials`** in sync — `config.json` wins; drift breaks mail |
|
||||||
|
| IP header | `X-Forwarded-For` |
|
||||||
|
| Signups | disabled (`signups_allowed=false`); invitations allowed |
|
||||||
|
| pgweb | compose profile `debug` (stopped by default) |
|
||||||
|
| Secrets | `.env`, `.admin-token`, `.smtp-credentials` — **never commit** |
|
||||||
|
|
||||||
|
### Official documentation
|
||||||
|
|
||||||
|
| Topic | Link |
|
||||||
|
|-------|------|
|
||||||
|
| Config overview | https://github.com/dani-garcia/vaultwarden/wiki/Configuration-overview |
|
||||||
|
| Postgres / migration | https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend |
|
||||||
|
| SMTP | https://github.com/dani-garcia/vaultwarden/wiki/SMTP-configuration |
|
||||||
|
| Proxy / Traefik | https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples |
|
||||||
|
| Full index | [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md) |
|
||||||
|
|
||||||
|
### Stack
|
||||||
|
|
||||||
|
| Container | Status |
|
||||||
|
|-----------|--------|
|
||||||
|
| `vaultwarden` | Up (healthy), `vaultwarden/server:1.37.1` |
|
||||||
|
| `vw-db` | Up (healthy) — **live** Postgres |
|
||||||
|
| `vaultwarden-backup` | Up (`pg_dump`) |
|
||||||
|
| `vaultwarden-pgweb` | Exited (profile `debug`) |
|
||||||
|
|
||||||
|
Effective `config.json`: domain `https://auth.wsvc.info`, SMTP 587/starttls to `mx2.windy.me` as `zhiqiang@windy.me`, `ip_header=X-Forwarded-For`. Password matched to `.env` (fixed 2026-08-01).
|
||||||
|
|
||||||
|
### Client use
|
||||||
|
|
||||||
|
- Server URL: `https://auth.wsvc.info`
|
||||||
|
- Admin: `https://auth.wsvc.info/admin`
|
||||||
|
|
||||||
|
### Ops / runbooks
|
||||||
|
|
||||||
|
- [vaultwarden-health](../runbooks/vaultwarden-health.md)
|
||||||
|
- [vaultwarden-sqlite-to-postgres](../runbooks/vaultwarden-sqlite-to-postgres.md)
|
||||||
|
- [vaultwarden-upstream docs](../docs/vaultwarden-upstream.md)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@us2.wsvc.info
|
||||||
|
cd /opt/vaultwarden && docker compose ps -a
|
||||||
|
# after recreating vaultwarden, if auth 404s:
|
||||||
|
docker restart traefik
|
||||||
|
```
|
||||||
|
|
||||||
|
## Other software on this host (stubs)
|
||||||
|
|
||||||
|
`/opt/traefik`, `soft-serve`, `authelia`, `dendrite`/`conduit`, `mastodon`, `rustdesk`, `zitadel`, etc.
|
||||||
|
|
||||||
|
## Verified
|
||||||
|
|
||||||
|
Last checked: **2026-08-01 18:55 CST** — operational.
|
||||||
|
|
||||||
|
- `vaultwarden` + `vw-db` healthy; `DATABASE_URL` → `pg:5432/vaultwarden`
|
||||||
|
- `https://auth.wsvc.info/` **200**, `/admin` **200**, `/api/config` OK (`disableUserRegistration: true`)
|
||||||
|
- Identity wrong-password → **400** business error (DB readable, not 500)
|
||||||
|
- SMTP: container → `mx2:587` OK; STARTTLS cert CN=`mx2.windy.me`; **AUTH OK** with effective `config.json` password (synced with `.env` / `.smtp-credentials`)
|
||||||
|
- LE cert CN=`auth.wsvc.info`
|
||||||
|
- PG counts: users=1, ciphers=1327
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# Host inventory
|
||||||
|
|
||||||
|
| Host | Role | SSH | IPv4 | Status | Facts |
|
||||||
|
|------|------|-----|------|--------|-------|
|
||||||
|
| mx2.windy.me | mailcow (primary MX prio 20) | `ssh -4 windy@mx2.windy.me` | 194.163.160.244 | active | [hosts/mx2.windy.me.md](../hosts/mx2.windy.me.md) |
|
||||||
|
| us2.wsvc.info | Vaultwarden/Postgres (+ Traefik, Soft Serve, …) | `ssh -4 windy@us2.wsvc.info` | 193.9.44.165 | active | [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md) |
|
||||||
|
| mx.windy.me | mail (secondary MX prio 30) | TBD | see AAAA/A | stub | — |
|
||||||
|
| repo.windy.me | Soft Serve git (on us2) | `ssh -p 2222 windy@repo.windy.me` | 193.9.44.165 | stub | see us2 |
|
||||||
|
| auth.wsvc.info | Vaultwarden public hostname | — (HTTPS) | → us2 | active | see us2 |
|
||||||
|
| us1.wsvc.info | PowerDNS secondary (ns2 host) | TBD | 202.91.35.141 | stub | Auth 5.0.5; see hk2 |
|
||||||
|
| us4.wsvc.info | Traefik / services | TBD | TBD | stub | — |
|
||||||
|
| hk2.chans.xyz | PowerDNS auth (ns1) | `ssh -4 windy@hk2.chans.xyz` | 154.36.174.161 | active | [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) |
|
||||||
|
| ns1.wsvc.info | PowerDNS public NS name | — (DNS) | → hk2 `154.36.174.161` | active | see hk2 |
|
||||||
|
| ns2.wsvc.info | Secondary NS (AXFR/NOTIFY peer) | — (DNS) | → us1 `202.91.35.141` | active | see hk2 |
|
||||||
|
| pdns.wsvc.info | Poweradmin UI | — (HTTPS) | → hk2 | active | see hk2 |
|
||||||
|
| pgweb.wsvc.info | PowerDNS Postgres UI | — (HTTPS) | → hk2 | active | see hk2 |
|
||||||
|
| **synapse.chans.xyz** | Matrix homeserver (ESS: Synapse + MAS + Element) | `ssh -4 windy@synapse.chans.xyz` | `169.58.86.13` | **active** | [hosts/synapse.chans.xyz.md](../hosts/synapse.chans.xyz.md) |
|
||||||
|
|
||||||
|
`status: stub` = known to exist; fill `hosts/<name>.md` when next touched.
|
||||||
|
|
||||||
|
### Matrix services (synapse.chans.xyz)
|
||||||
|
|
||||||
|
| URL | Service | Notes |
|
||||||
|
|-----|---------|-------|
|
||||||
|
| https://chat.chans.xyz | Element Web | Matrix web client |
|
||||||
|
| https://synapse.chans.xyz | Synapse API | Client-Server + Federation API |
|
||||||
|
| https://account.chans.xyz | MAS | Matrix Authentication Service (local passwords) |
|
||||||
|
| https://admin.chans.xyz | Element Admin | Admin console (MAS admin auth) |
|
||||||
|
| `mrtc.chans.xyz` | MatrixRTC | **Reserved** – not deployed |
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# VPS 維護專案重構：Ansible + systemd timers + Restic
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
將現有 VPS 維運知識庫漸進重構為一個以 Ansible 管理一致性、以 systemd timers 執行每日唯讀檢查、以 Restic 建立加密異機備份的個人 SRE 維運專案。第一期僅納管 mx2、us2、hk2；自動化只能觀測與告警，任何更新、重啟、修復、DNS 或機密變更都必須人工確認。
|
||||||
|
|
||||||
|
## Confirmed Decisions
|
||||||
|
|
||||||
|
- [x] 定位為個人 SRE／維運手冊庫，而非全自動修復平台。
|
||||||
|
- [x] 第一階段完整納管 mx2、us2、hk2；mx、us1、us4 待完成盤點與緊急存取驗證後才納管。
|
||||||
|
- [x] 每日自動做唯讀檢查與 Email 通知；每週人工審查、每月人工維護、每季復原演練與權限盤點。
|
||||||
|
- [x] 以 Email 作為唯一告警與摘要通道。
|
||||||
|
- [x] 採用本機快速復原加密異機副本的 3-2-1 最小可行策略。
|
||||||
|
- [x] 真實機密不進 Git；repo 只保存去敏設定結構與操作／驗證紀錄。
|
||||||
|
- [x] 日常存取使用非 root SSH key；禁止 root/password SSH，維護 provider/recovery console 緊急存取。
|
||||||
|
- [x] 採用 Ansible + systemd timers + Restic 作為技術基線。
|
||||||
|
|
||||||
|
## Implementation Plan
|
||||||
|
|
||||||
|
- [x] **Status: Done — 建立去敏 Ansible inventory。** 為 mx2、us2、hk2 建立主機與服務群組，將非機密主機資料與現有人類可讀的 `inventory/hosts.md` 對應；理由是提供可審查的機器可讀編排層，且不取代既有事實來源。
|
||||||
|
- [x] **Status: Done — 實作 audit-only Ansible playbook。** 只收集連線、OS、磁碟、systemd、Docker Compose 和依賴命令狀態，不做設定或服務改動；理由是先確認控制面與真實現況。
|
||||||
|
- [x] **Status: Done — 定義跨服務健康檢查結果合約。** 統一去敏結構化結果、日誌位置、嚴重度及 exit code，讓人工執行、systemd 和 Ansible audit 可共同消費；理由是避免各服務告警語意漂移。
|
||||||
|
- [x] **Status: Done — 保留並模組化 Mailcow 健康檢查。** 持續檢查 Compose、watchdog、queue、listeners、HTTP/HTTPS、TLS、SMTP、DNS/PTR/MX/SPF；理由是現有腳本已覆蓋關鍵郵件服務面向。
|
||||||
|
- [x] **Status: Done — 實作 Vaultwarden 唯讀健康檢查。** 驗證 Compose/Postgres、HTTPS、有效設定、SMTP AUTH、備份新鮮度及機密指紋一致性，禁止輸出機密；理由是 `config.json` 優先於 `.env`，且 SMTP 漂移已有已知風險。
|
||||||
|
- [x] **Status: Done — 實作 PowerDNS 唯讀健康檢查。** 驗證容器、版本與安全公告、ns1/ns2 served SOA、API、Web UI、備份新鮮度與必要設定；理由是同時覆蓋公開 DNS 與 secondary 同步。
|
||||||
|
- [x] **Status: Done — 使用 Ansible 部署 systemd healthcheck service/timer。** 每台主機本機執行每日檢查，具 persistent 排程、權限、logrotate 和一致錯誤處理；理由是控制端離線不應阻止巡檢。
|
||||||
|
- [x] **Status: Done — 部署 Email 告警與每日摘要。** Critical/unknown 立即通知、健康狀態每日摘要、重複失敗抑制；SMTP 真實認證僅在各主機受限路徑保存；理由是已選定 Email 為唯一通知通道。
|
||||||
|
- [!] **Status: Blocked — 決定 Restic 異機 repository 與存取隔離。** 選擇具加密傳輸、權限隔離與可承受保留需求的 S3/B2/SFTP 或等價目的地；理由是無異機 repository 即無法滿足已確認的 3-2-1 政策。阻塞：本地實作依要求未虛構 backend、repository 或 credentials。
|
||||||
|
- [x] **Status: Done (templates gated) — 部署 Vaultwarden 與 PowerDNS 的 Restic 備份。** 將既有一致性資料庫 dump、必要資料目錄及復原元資料加密同步，並加入 snapshot 年齡和 `restic check` 驗證；理由是兩者已有本機備份但缺少異機保護。啟用被 repository 決策和主機端受限設定檔阻擋。
|
||||||
|
- [!] **Status: Blocked — 完成 Mailcow 備份設計審查。** 官方流程已確認：以 `/opt/mail/helper-scripts/backup_and_restore.sh backup all`（或經明確核准的元件集）先產生一致性備份，再由 Restic 同步該輸出；不得直接複製 Docker volumes。阻塞：仍需決定本機備份位置、保留期、排程及異機 Restic repository。
|
||||||
|
- [x] **Status: Done (templates gated) — 部署 Restic backup、retention、forget/prune 與 check timers。** 與現有資料庫 dump 時段錯開，所有 repository 認證與密碼留在伺服器端受限檔案；理由是保持備份可用與成本可控。啟用被 repository 決策和主機端受限設定檔阻擋。
|
||||||
|
- [!] **Status: Blocked — 補齊服務級復原 runbook。** 阻塞：需在選定 Restic repository、建立實際 snapshot 並確認各服務的實際備份輸出後，才能編寫可驗證的 restore 前置條件、順序與 rollback；禁止臆造 backend/credentials 或未驗證還原命令。
|
||||||
|
- [!] **Status: Blocked — 執行隔離式復原演練。** 阻塞：尚未選定／配置 Restic 異機 repository，亦尚無可供還原的異機 snapshot；演練不得對現有生產資料執行。
|
||||||
|
- [x] **Status: Done — 實作受控 common baseline 與 maintenance playbook。** 僅在 audit 穩定後納入 SSH 稽核、時間同步、logrotate、更新預覽與人工確認的維護操作；理由是避免工具導入期間同時改變服務狀態。
|
||||||
|
- [x] **Status: Done (separate change defined) — 將 PowerDNS API key、DB password 與 TSIG 輪替列為分離變更。** 已明確保持 API key、DB password 與 TSIG 為三項獨立、需人工核准的變更，並要求逐步驗證 Auth、Poweradmin、AXFR/NOTIFY、DNSSEC 與 ns2 同步；未輪替任何機密。
|
||||||
|
- [!] **Status: Blocked — 盤點 mx、us1、us4。** 阻塞：inventory 僅記錄 `TBD` SSH／角色或缺少完整事實，尚未提供可驗證的存取方式與 provider/recovery console 資訊；不得猜測或嘗試未授權存取。
|
||||||
|
|
||||||
|
## Verification Criteria
|
||||||
|
|
||||||
|
- [ ] 三台 active 主機能被 Ansible 正確解析及執行 audit，且 audit 不產生主機變更。
|
||||||
|
- [ ] 三台主機每日產生去敏健康結果；服務、容量、TLS、公開端點和備份新鮮度異常均能被偵測。
|
||||||
|
- [ ] 正常狀態寄送 Email 摘要；受控測試異常能觸發一次可讀且不含機密的 Email 告警。
|
||||||
|
- [ ] 每個核心服務至少有一份加密、異機、可列出且通過完整性檢查的 Restic snapshot。
|
||||||
|
- [ ] 至少完成一次隔離式實際還原，並確認其結果符合或明確量化偏離服務 RPO/RTO。
|
||||||
|
- [ ] 自動化排程不包含更新、重啟、修復、秘密輪替或 DNS 變更。
|
||||||
|
- [ ] Repo、Ansible vars、產出日誌及 Email 均不包含任何真實機密。
|
||||||
|
|
||||||
|
## Potential Risks and Mitigations
|
||||||
|
|
||||||
|
1. **Mailcow 備份未保持資料一致性。**
|
||||||
|
Mitigation: 先依官方方式完成備份範圍與還原設計，再上線排程；以隔離還原驗證作為完成門檻。
|
||||||
|
|
||||||
|
2. **Ansible 設定錯誤改動生產系統。**
|
||||||
|
Mitigation: 先導入 audit-only；變更 playbook 必須使用 dry-run/diff、明確 tag 與人工確認，且先限制單一 host。
|
||||||
|
|
||||||
|
3. **機密出現在 Git、Ansible output 或 Email。**
|
||||||
|
Mitigation: 真實值只存主機受限檔案；去敏輸出、no-log 機制和測試用機密掃描為強制要求。
|
||||||
|
|
||||||
|
4. **Timer 成功但公開服務實際不可用。**
|
||||||
|
Mitigation: 同時執行本機 Compose/DB 檢查與外部 HTTP、SMTP、TLS、DNS 檢查。
|
||||||
|
|
||||||
|
5. **PowerDNS 密鑰輪替中斷 DNS 管理或 zone transfer。**
|
||||||
|
Mitigation: API key、DB password、TSIG 分批處理；每步都驗證 primary/secondary 同步、DNSSEC 和管理 UI。
|
||||||
|
|
||||||
|
## Alternative Approaches
|
||||||
|
|
||||||
|
1. **純 shell + systemd，不使用 Ansible**：初期較快，但設定與排程容易跨主機漂移，且難以審查或擴充；不建議作為長期方案。
|
||||||
|
2. **Ansible + 集中式監控平台**：可增加儀表板與趨勢，但平台本身需額外維護；等 Email 摘要無法滿足需求時再評估。
|
||||||
|
3. **立即導入 SOPS + age**：可使加密設定進 Git，但需先完成密鑰生命週期與緊急存取設計；建議列為後續階段。
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
# Runbook: mailcow health (mx2)
|
||||||
|
|
||||||
|
Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
|
||||||
|
Path: `/opt/mail`
|
||||||
|
Prefer: `./scripts/check-mx2.sh` from this repo, or the steps below over SSH.
|
||||||
|
|
||||||
|
## 1. Containers
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose ps -a'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect mailcow (`cow-*`) services **Up**. Note any Exit/Restarting.
|
||||||
|
|
||||||
|
## 2. Watchdog
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose logs --tail=40 watchdog-mailcow'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect recent lines at **100%** for Nginx, Postfix, Dovecot, MySQL, Redis, SOGo, Rspamd, ACME, etc.
|
||||||
|
|
||||||
|
## 3. Queue
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose exec -T postfix-mailcow postqueue -p'
|
||||||
|
```
|
||||||
|
|
||||||
|
Healthy: empty / `Mail queue is empty`.
|
||||||
|
|
||||||
|
## 4. Local listeners
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh windy@mx2.windy.me 'ss -tlnp | grep -E ":(25|465|587|143|993|995|80|443|4190)\\s"'
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. External checks (from laptop / agent host)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sI --max-time 10 http://mx2.windy.me/ | head -5
|
||||||
|
curl -skI --max-time 10 https://mx2.windy.me/ | head -15
|
||||||
|
echo | openssl s_client -connect mx2.windy.me:443 -servername mx2.windy.me 2>/dev/null \
|
||||||
|
| openssl x509 -noout -subject -issuer -dates
|
||||||
|
timeout 8 bash -c 'exec 3<>/dev/tcp/mx2.windy.me/25; echo -e "EHLO test.local\r\nQUIT\r\n" >&3; cat <&3' | head -20
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect: HTTP 301 → HTTPS; HTTPS 200; LE cert for `mx2.windy.me`; SMTP banner `Postcow`.
|
||||||
|
|
||||||
|
## 6. DNS summary
|
||||||
|
|
||||||
|
Use a public resolver if the local stub is flaky (WSL `127.0.0.53`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dig @1.1.1.1 +short mx2.windy.me A
|
||||||
|
dig @1.1.1.1 +short mx2.windy.me AAAA
|
||||||
|
dig @1.1.1.1 +short -x 194.163.160.244
|
||||||
|
dig @1.1.1.1 +short windy.me MX
|
||||||
|
dig @1.1.1.1 +short windy.me TXT | grep -i spf
|
||||||
|
```
|
||||||
|
|
||||||
|
## Scheduled local check
|
||||||
|
|
||||||
|
The sanitized Ansible health profile is `mailcow` (`ansible/playbooks/healthchecks.yml`). It preserves the existing standalone `scripts/check-mx2.sh` as the richer operator check, while the server-local timer emits a sanitized result at `/var/lib/vps-health/latest.json`. Neither check changes Mailcow configuration or service state.
|
||||||
|
|
||||||
|
## Pass criteria
|
||||||
|
|
||||||
|
- Compose stack up; watchdog ~100%
|
||||||
|
- Queue empty (or explain backlog)
|
||||||
|
- TLS valid; mail ports reachable externally
|
||||||
|
- A/AAAA/PTR/MX consistent with [host facts](../hosts/mx2.windy.me.md)
|
||||||
|
|
||||||
|
Update the **Verified** line on the host page after a successful check.
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
# Runbook: use mailcow SMTP / IMAP (client)
|
||||||
|
|
||||||
|
Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
|
||||||
|
Prerequisite: a mailbox on `windy.me` (password from mailcow UI, not the admin account unless it is that mailbox).
|
||||||
|
|
||||||
|
Clients should use **submission** to send. Do **not** use port 25 as a desktop/app outbound port.
|
||||||
|
|
||||||
|
## Send (SMTP)
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|------|--------|
|
||||||
|
| Server | `mx2.windy.me` |
|
||||||
|
| Port | **587** (STARTTLS) — preferred |
|
||||||
|
| Alt port | **465** (SSL/TLS) |
|
||||||
|
| Username | Full address, e.g. `zhiqiang@windy.me` |
|
||||||
|
| Password | Mailbox password |
|
||||||
|
| Auth | Required (LOGIN / PLAIN) |
|
||||||
|
|
||||||
|
Match encryption to the port: 587 → STARTTLS; 465 → SSL/TLS (not STARTTLS).
|
||||||
|
|
||||||
|
From address should be that mailbox or an allowed alias on the same domain.
|
||||||
|
|
||||||
|
## Receive (IMAP / POP)
|
||||||
|
|
||||||
|
| Protocol | Server | Port |
|
||||||
|
|----------|--------|------|
|
||||||
|
| IMAPS | `mx2.windy.me` | **993** |
|
||||||
|
| POP3S | `mx2.windy.me` | **995** |
|
||||||
|
|
||||||
|
Username = full email address; same mailbox password.
|
||||||
|
|
||||||
|
## Webmail
|
||||||
|
|
||||||
|
- UI / SOGo: https://mx2.windy.me
|
||||||
|
- Log in with the same mailbox credentials.
|
||||||
|
|
||||||
|
## CLI smoke test (optional)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
swaks --to someone@example.com \
|
||||||
|
--from YOU@windy.me \
|
||||||
|
--server mx2.windy.me --port 587 -tls \
|
||||||
|
--auth LOGIN --auth-user YOU@windy.me --auth-password 'MAILBOX_PASSWORD'
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not commit or paste real passwords into this repo.
|
||||||
|
|
||||||
|
## Auth failures — quick checks
|
||||||
|
|
||||||
|
- Username is the **full** email, not a local part alone
|
||||||
|
- Password is the **mailbox** password (not unrelated host/root secrets)
|
||||||
|
- Port/TLS mode mismatch (587 vs 465)
|
||||||
|
- Account active in mailcow; not rate-limited / fail2banned after bad attempts
|
||||||
|
- Apps that store SMTP in their own config (e.g. Vaultwarden `config.json`) may keep a **stale** password even when `.env` is correct — verify AUTH against the effective config ([vaultwarden-health](vaultwarden-health.md) §5)
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Runbook: mailcow update (mx2)
|
||||||
|
|
||||||
|
Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
|
||||||
|
Path: `/opt/mail`
|
||||||
|
**Confirm with the user before running an update.**
|
||||||
|
|
||||||
|
## Before
|
||||||
|
|
||||||
|
1. Run [mailcow-health](mailcow-health.md) (or `./scripts/check-mx2.sh`). Record baseline.
|
||||||
|
2. Note config quirks in `mailcow.conf`, especially:
|
||||||
|
- `SKIP_CLAMD=y` — after update, `clamd` may still be running until services are recreated per mailcow docs.
|
||||||
|
- `SKIP_LETS_ENCRYPT`, ports, hostname — should stay `mx2.windy.me`.
|
||||||
|
3. Disk/memory: `df -h /` and `free -h` on the host.
|
||||||
|
|
||||||
|
## Update
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -t windy@mx2.windy.me 'cd /opt/mail && ./update.sh'
|
||||||
|
```
|
||||||
|
|
||||||
|
- Prefer interactive SSH (`-t`) so prompts work.
|
||||||
|
- Do not pass secrets into the chat log; do not commit `mailcow.conf`.
|
||||||
|
|
||||||
|
If update asks to stop/recreate containers, allow it unless the user said otherwise.
|
||||||
|
|
||||||
|
## After
|
||||||
|
|
||||||
|
1. Re-run health checks ([mailcow-health](mailcow-health.md)).
|
||||||
|
2. Confirm UI: `https://mx2.windy.me`
|
||||||
|
3. Confirm SMTP banner and queue empty.
|
||||||
|
4. If `SKIP_CLAMD=y` but `cow-clamd-mailcow-1` is still Up and that is unwanted:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose up -d --force-recreate'
|
||||||
|
```
|
||||||
|
|
||||||
|
Only after user OK; recreate briefly interrupts mail.
|
||||||
|
|
||||||
|
5. Update **Verified** on [hosts/mx2.windy.me.md](../hosts/mx2.windy.me.md) with date and outcome.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
Mailcow updates are image/git based; there is no one-click rollback in this runbook. If something fails, capture `docker compose ps` and `docker compose logs --tail=100` for the failing service and stop before further changes.
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
# Matrix Health Check
|
||||||
|
|
||||||
|
Monitor the Matrix homeserver running on `synapse.chans.xyz` (ESS chart `26.7.2`, K3s node).
|
||||||
|
|
||||||
|
## Basic checks
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Overall status
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods,svc,ingress,certificate -o wide'
|
||||||
|
|
||||||
|
# Pod restarts
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods -o wide | grep -E "CrashLoop|Error|Evicted" || echo "No unhealthy pods"'
|
||||||
|
|
||||||
|
# Disk
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'df -hP /'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Certificate verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get certificate'
|
||||||
|
```
|
||||||
|
|
||||||
|
All certificates should show `READY=True`. Domains:
|
||||||
|
- `chans.xyz`
|
||||||
|
- `synapse.chans.xyz`
|
||||||
|
- `chat.chans.xyz`
|
||||||
|
- `account.chans.xyz`
|
||||||
|
- `admin.chans.xyz`
|
||||||
|
- `mrtc.chans.xyz` (placeholder only)
|
||||||
|
|
||||||
|
## External endpoint verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Client API
|
||||||
|
curl -4 -s https://synapse.chans.xyz/_matrix/client/versions | python3 -m json.tool | head -5
|
||||||
|
|
||||||
|
# Well-known server discovery
|
||||||
|
curl -4 -s https://chans.xyz/.well-known/matrix/server
|
||||||
|
|
||||||
|
# Well-known client discovery
|
||||||
|
curl -4 -s https://chans.xyz/.well-known/matrix/client
|
||||||
|
|
||||||
|
# Federation tester (manual)
|
||||||
|
open https://federationtester.matrix.org/#chans.xyz
|
||||||
|
```
|
||||||
|
|
||||||
|
## Local health-check automation
|
||||||
|
|
||||||
|
The node has a systemd health-check timer that runs daily:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-k3s-healthcheck.timer'
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'cat /var/lib/vps-health/latest.json 2>/dev/null || echo "No data yet"'
|
||||||
|
```
|
||||||
|
|
||||||
|
Checks cover:
|
||||||
|
- K3s node Ready
|
||||||
|
- Pods in `ess` namespace (no CrashLoopBackOff / Evicted)
|
||||||
|
- Root disk usage (warning ≥80%, critical ≥90%)
|
||||||
|
- Backup freshness
|
||||||
|
|
||||||
|
## Backup status
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'systemctl status matrix-backup.timer'
|
||||||
|
ssh -4 windy@synapse.chans.xyz 'ls -la /var/backups/matrix/'
|
||||||
|
```
|
||||||
|
|
||||||
|
Backup automation is currently paused. `/var/backups/matrix/` is retained for a future, reviewed design and must not be treated as a recovery source.
|
||||||
|
- PostgreSQL dumps (synapse + mas)
|
||||||
|
- Media archive
|
||||||
|
- Bootstrap secrets archive
|
||||||
|
- Backup manifest with SHA-256
|
||||||
|
- Retention: 7 days
|
||||||
|
|
||||||
|
## Common pitfalls
|
||||||
|
|
||||||
|
| Symptom | Likely cause |
|
||||||
|
|---|---|
|
||||||
|
| `/_matrix/client/versions` fails | Traefik ingress not routing to Synapse |
|
||||||
|
| HTTPS returns non-LE cert | cert-manager `letsencrypt-prod` not ready |
|
||||||
|
| Well-known returns 404/redirect | Root `chans.xyz` ingress missing or misconfigured |
|
||||||
|
| 502 Bad Gateway | Synapse pod restarting or DB down |
|
||||||
|
| SMTP emails not sent | MAS SMTP config incomplete; TCP reachable but AUTH failing — see `runbooks/vaultwarden-health.md` |
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
# PowerDNS health (hk2)
|
||||||
|
|
||||||
|
Read-only checks for the `/opt/pdns` stack on **hk2.chans.xyz** (`ns1.wsvc.info`).
|
||||||
|
|
||||||
|
Facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) · Upstream: [docs/pdns-upstream.md](../docs/pdns-upstream.md)
|
||||||
|
|
||||||
|
## Quick remote
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose ps -a'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect `auth`, `db`, `poweradmin` healthy; `backup` Up; `pgweb` Up. Ignore stopped orphan `powerdns-admin` unless cleaning orphans.
|
||||||
|
|
||||||
|
## Version / security poll
|
||||||
|
|
||||||
|
Expect Authoritative **5.0.6** (or newer). Startup logs should **not** show a Mandatory `Security Update` banner. If they do, check https://doc.powerdns.com/authoritative/security-advisories/ and bump the `powerdns/pdns-auth-50` image pin in `/opt/pdns/compose.yml`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@hk2.chans.xyz 'docker logs pdns-auth 2>&1 | grep -i "Security Update" | tail -5'
|
||||||
|
ssh -4 windy@hk2.chans.xyz 'docker inspect pdns-auth --format "{{.Config.Image}}"'
|
||||||
|
```
|
||||||
|
|
||||||
|
## DNS (from laptop / WSL)
|
||||||
|
|
||||||
|
Prefer querying auth/secondary IPs directly (local stub / public resolvers may be flaky or cached):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
NS1=154.36.174.161
|
||||||
|
NS2=202.91.35.141
|
||||||
|
for z in windy.me wsvc.info chans.xyz; do
|
||||||
|
echo "== $z =="
|
||||||
|
dig @$NS1 +short NS "$z"
|
||||||
|
dig @$NS1 +short SOA "$z"
|
||||||
|
dig @$NS2 +short SOA "$z"
|
||||||
|
done
|
||||||
|
dig @$NS1 +short A ns1.wsvc.info # expect 154.36.174.161
|
||||||
|
dig @$NS1 +short A ns2.wsvc.info # expect 202.91.35.141
|
||||||
|
```
|
||||||
|
|
||||||
|
Served SOA serial can differ from the backend/API serial when `SOA-EDIT=INCEPTION-INCREMENT` is active — compare **ns1 vs ns2 served SOA**, not dig vs `pdnsutil`/API.
|
||||||
|
|
||||||
|
## API (on host)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose exec -T auth \
|
||||||
|
python3 -c "import json,os,urllib.request; print(json.load(urllib.request.urlopen(urllib.request.Request(\"http://127.0.0.1:8081/api/v1/servers/localhost\", headers={\"X-API-Key\": os.environ[\"PDNS_API_KEY\"]})))[\"version\")"'
|
||||||
|
```
|
||||||
|
|
||||||
|
Or use host loopback with the key from `.env` (do not paste the key into chat/logs).
|
||||||
|
|
||||||
|
## Web UIs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# If local stub DNS is flaky, pin SNI to the auth IP:
|
||||||
|
curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pdns.wsvc.info:443:154.36.174.161 https://pdns.wsvc.info/ # 302 → login
|
||||||
|
curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pgweb.wsvc.info:443:154.36.174.161 https://pgweb.wsvc.info/ # 401 without basic auth
|
||||||
|
```
|
||||||
|
|
||||||
|
## Backup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && ./scripts/manual-backup.sh && ls -lt backup/*.sql.gz | head'
|
||||||
|
ssh -4 windy@hk2.chans.xyz 'tail -30 /opt/pdns/backup/backup.log'
|
||||||
|
```
|
||||||
|
|
||||||
|
Scheduler should log `scheduler started … (daily 03:00)`. If you see `can't execute 'bash'` or Alpine `apk` errors, the backup image/entrypoint was reverted — see [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md).
|
||||||
|
|
||||||
|
## Config sanity (docs-aligned)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@hk2.chans.xyz 'grep -E "^(primary|also-notify|only-notify|allow-axfr-ips|gpgsql-dnssec|default-soa-edit)" /opt/pdns/auth/pdns.conf'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect: `primary=yes`, `also-notify=202.91.35.141`, `only-notify=` empty, `gpgsql-dnssec=yes`, SOA-EDIT `INCEPTION-INCREMENT`.
|
||||||
|
|
||||||
|
## Scheduled local check
|
||||||
|
|
||||||
|
The sanitized Ansible health profile is `pdns` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses the API key only inside the PowerDNS container. It does not modify PowerDNS, DNS records, or secrets.
|
||||||
|
|
||||||
|
|
||||||
|
## After config changes
|
||||||
|
|
||||||
|
- `auth/pdns.conf`, `auth/templates.d/secrets.j2`, or auth-related `.env` → `docker compose up -d --force-recreate auth`
|
||||||
|
- Poweradmin env / Traefik labels → `docker compose up -d --no-deps --force-recreate poweradmin` (restart Traefik if UI 404s)
|
||||||
|
- backup schedule / retention env → `docker compose up -d --force-recreate --no-deps backup`
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# Runbook: Vaultwarden health (us2)
|
||||||
|
|
||||||
|
Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)
|
||||||
|
Path: `/opt/vaultwarden`
|
||||||
|
URL: https://auth.wsvc.info/
|
||||||
|
Upstream: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
|
||||||
|
|
||||||
|
## 1. Containers
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose ps -a'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect `vaultwarden` + `vw-db` **Up (healthy)**. `pgweb` should be stopped unless started with `--profile debug`.
|
||||||
|
|
||||||
|
## 2. Live DB is Postgres
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden sh -c "printenv DATABASE_URL" | sed -E "s#://[^@]+@#://REDACTED@#"'
|
||||||
|
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T pg psql -U vaultwarden -d vaultwarden -c "SELECT COUNT(*) AS users FROM users; SELECT COUNT(*) AS ciphers FROM ciphers;"'
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. Effective config (config.json wins)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden cat /data/config.json' \
|
||||||
|
| python3 -c "import sys,json;d=json.load(sys.stdin);print({k:d.get(k) for k in ['domain','smtp_host','smtp_port','smtp_security','smtp_username','smtp_from','ip_header','signups_allowed']});print('smtp_password_set',bool(d.get('smtp_password')))"
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect SMTP **mx2.windy.me** / **587** / **starttls**, `ip_header` **X-Forwarded-For**.
|
||||||
|
|
||||||
|
Also confirm `.env` and `config.json` SMTP passwords match (lengths/hashes only — never print secrets):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
|
||||||
|
import json, re, pathlib, hashlib, subprocess
|
||||||
|
env = pathlib.Path(".env").read_text()
|
||||||
|
def grab(k):
|
||||||
|
m = re.search(rf"^{re.escape(k)}=(.*)$", env, re.M)
|
||||||
|
return m.group(1).strip().strip("\"'\''") if m else None
|
||||||
|
cfg = json.loads(subprocess.check_output(
|
||||||
|
["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
|
||||||
|
ep, cp = grab("SMTP_PASSWORD"), cfg.get("smtp_password") or ""
|
||||||
|
def fp(s): return {"len": len(s), "sha256_8": hashlib.sha256(s.encode()).hexdigest()[:8]}
|
||||||
|
print("env_pass", fp(ep or ""), "cfg_pass", fp(cp), "match", ep == cp)
|
||||||
|
EOF'
|
||||||
|
```
|
||||||
|
|
||||||
|
If they diverge, **auth uses `config.json`** — sync password there (or via `/admin`) and update `.smtp-credentials`.
|
||||||
|
|
||||||
|
## 4. External HTTPS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -4 -sS -I --max-time 15 https://auth.wsvc.info/ | head -15
|
||||||
|
```
|
||||||
|
|
||||||
|
If **404** right after recreating VW: `ssh -4 windy@us2.wsvc.info 'docker restart traefik'` then retry.
|
||||||
|
|
||||||
|
## 5. SMTP reachability + AUTH
|
||||||
|
|
||||||
|
TCP (expect `587:0`; `465` often times out from us2):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@us2.wsvc.info 'docker compose -f /opt/vaultwarden/docker-compose.yml exec -T vaultwarden sh -c "timeout 5 bash -c \"cat </dev/null >/dev/tcp/mx2.windy.me/587\"; echo 587:\$?"'
|
||||||
|
```
|
||||||
|
|
||||||
|
AUTH with the **effective** `config.json` password (run on us2; do not print the password):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && python3 - << "EOF"
|
||||||
|
import json, smtplib, ssl, subprocess
|
||||||
|
cfg = json.loads(subprocess.check_output(
|
||||||
|
["docker","compose","exec","-T","vaultwarden","cat","/data/config.json"]))
|
||||||
|
host, port = cfg["smtp_host"], int(cfg["smtp_port"])
|
||||||
|
user, pw = cfg["smtp_username"], cfg["smtp_password"]
|
||||||
|
ctx = ssl.create_default_context()
|
||||||
|
with smtplib.SMTP(host, port, timeout=15) as s:
|
||||||
|
s.ehlo(); s.starttls(context=ctx); s.ehlo(); s.login(user, pw)
|
||||||
|
print("AUTH_OK", host, port, user)
|
||||||
|
EOF'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expect `AUTH_OK`. `535` usually means stale password in `config.json` (see step 3).
|
||||||
|
|
||||||
|
## Scheduled local check
|
||||||
|
|
||||||
|
The sanitized Ansible health profile is `vaultwarden` (`ansible/playbooks/healthchecks.yml`). It runs locally through `vps-healthcheck.timer`, writes a sanitized JSON result to `/var/lib/vps-health/latest.json`, and uses only server-side credentials for the SMTP AUTH probe. It does not modify Vaultwarden, Traefik, SMTP, or secrets.
|
||||||
|
|
||||||
|
|
||||||
|
## Pass criteria
|
||||||
|
|
||||||
|
- Healthy compose; `DATABASE_URL` points at `pg`
|
||||||
|
- HTTPS 200 + `/admin` 200; PG counts sane
|
||||||
|
- Effective SMTP 587/starttls; `.env` ↔ `config.json` password match; **AUTH_OK**
|
||||||
|
- Wrong-password login returns 400 business error (not 500)
|
||||||
|
- Update **Verified** on [hosts/us2.wsvc.info.md](../hosts/us2.wsvc.info.md)
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Runbook: Vaultwarden SQLite → Postgres (us2)
|
||||||
|
|
||||||
|
Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)
|
||||||
|
Upstream: [Using the PostgreSQL Backend](https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend) · [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
|
||||||
|
|
||||||
|
**Done on 2026-08-01** for this host. Keep this as the replay checklist.
|
||||||
|
|
||||||
|
## Summary of what we did
|
||||||
|
|
||||||
|
1. Backup `vw-data` + `.env` → `backups/pre-pg-migrate-<timestamp>/`
|
||||||
|
2. Baseline counts from SQLite (then: users=1, ciphers=1327)
|
||||||
|
3. Drop/recreate empty Postgres DB; pin image `vaultwarden/server:1.37.1`
|
||||||
|
4. Set compose `DATABASE_URL=postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:${DB_PORT}/${DB_NAME}`
|
||||||
|
5. Start VW once → diesel schema → stop
|
||||||
|
6. `pgloader` **data only** (exclude `__diesel_schema_migrations`) with CAST bigint→integer
|
||||||
|
7. Remaining intentional bigints: `attachments.file_size`, `twofactor.last_used`, `twofactor_duo_ctx.exp`
|
||||||
|
8. Align `config.json` SMTP **587/starttls**, `ip_header=X-Forwarded-For`, `DOMAIN` without trailing slash
|
||||||
|
9. Rotate `DB_PASS` / `PGWEB_PASS`; `pgweb` behind compose `profiles: [debug]`
|
||||||
|
10. Move live sqlite file to `backups/sqlite-cold/` (cold only)
|
||||||
|
11. After VW recreate, **restart Traefik** so Docker provider picks up the router
|
||||||
|
|
||||||
|
## Critical pitfalls
|
||||||
|
|
||||||
|
- `config.json` overrides `.env` (admin wins). Fix SMTP there or via `/admin`.
|
||||||
|
- SMTP password drift: if `.env` AUTH works but VW mail fails with `535`, sync `vw-data/config.json` (+ `.smtp-credentials`) to the mailbox password — do not assume env alone is enough.
|
||||||
|
- From this network, **mx2:465 times out**; use **587 + starttls**.
|
||||||
|
- After recreating `vaultwarden`, Traefik may 404 until `docker restart traefik`.
|
||||||
|
- pgloader needs a **writable** directory for the sqlite file (CANTOPEN if mount is awkward).
|
||||||
|
- After pgloader, wrong integer types → silent “wrong password”; see [Discussion #7134](https://github.com/dani-garcia/vaultwarden/discussions/7134).
|
||||||
|
- Do not commit `.env` or load files with passwords into git.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
1. `docker compose stop vaultwarden`
|
||||||
|
2. Restore `vw-data` from `backups/pre-pg-migrate-*` (including sqlite)
|
||||||
|
3. Remove `DATABASE_URL` from compose `environment`
|
||||||
|
4. `docker compose up -d vaultwarden` && `docker restart traefik`
|
||||||
Executable
+195
@@ -0,0 +1,195 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Read-only health check for mx2.windy.me mailcow.
|
||||||
|
# Usage: ./scripts/check-mx2.sh
|
||||||
|
# Env:
|
||||||
|
# MX2_HOST default mx2.windy.me
|
||||||
|
# MX2_SSH_USER default windy
|
||||||
|
# MX2_ALLOW_IPV6 set to 1 to allow IPv6 (default forces IPv4; WSL often lacks v6 route)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
HOST="${MX2_HOST:-mx2.windy.me}"
|
||||||
|
SSH_USER="${MX2_SSH_USER:-windy}"
|
||||||
|
|
||||||
|
# Default -4: host has AAAA but many clients (WSL) have no IPv6 route → "Network is unreachable"
|
||||||
|
SSH_AF_OPTS=(-4)
|
||||||
|
CURL_AF_OPTS=(-4)
|
||||||
|
OPENSSL_AF_OPTS=(-4)
|
||||||
|
if [[ "${MX2_ALLOW_IPV6:-0}" == "1" ]]; then
|
||||||
|
SSH_AF_OPTS=()
|
||||||
|
CURL_AF_OPTS=()
|
||||||
|
OPENSSL_AF_OPTS=()
|
||||||
|
fi
|
||||||
|
|
||||||
|
SSH=(ssh "${SSH_AF_OPTS[@]}" -o BatchMode=yes -o ConnectTimeout=10 "${SSH_USER}@${HOST}")
|
||||||
|
# Prefer public resolvers: local stub (127.0.0.53) is flaky from some WSL setups
|
||||||
|
DIG_OPTS=(+time=3 +tries=2 +short)
|
||||||
|
DIG_SERVERS=(1.1.1.1 8.8.8.8)
|
||||||
|
FAILURES=0
|
||||||
|
|
||||||
|
section() { printf '\n===== %s =====\n' "$1"; }
|
||||||
|
|
||||||
|
fail() {
|
||||||
|
printf 'FAIL: %s\n' "$1" >&2
|
||||||
|
FAILURES=$((FAILURES + 1))
|
||||||
|
}
|
||||||
|
|
||||||
|
# Strip dig commentary (timeouts are printed on stdout as ";; ...")
|
||||||
|
dig_answers_only() {
|
||||||
|
grep -v '^;;' | grep -v '^$' || true
|
||||||
|
}
|
||||||
|
|
||||||
|
# dig_short <name> [type] — tries each public resolver; skips timeout noise
|
||||||
|
dig_short() {
|
||||||
|
local name="$1"
|
||||||
|
local typ="${2:-A}"
|
||||||
|
local server out
|
||||||
|
for server in "${DIG_SERVERS[@]}"; do
|
||||||
|
out="$(dig @"${server}" "${DIG_OPTS[@]}" "$name" "$typ" 2>/dev/null | dig_answers_only)" || out=""
|
||||||
|
if [[ -n "$out" ]]; then
|
||||||
|
printf '%s\n' "$out"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# dig_ptr <ip>
|
||||||
|
dig_ptr() {
|
||||||
|
local ip="$1"
|
||||||
|
local server out
|
||||||
|
for server in "${DIG_SERVERS[@]}"; do
|
||||||
|
out="$(dig @"${server}" "${DIG_OPTS[@]}" -x "$ip" 2>/dev/null | dig_answers_only)" || out=""
|
||||||
|
if [[ -n "$out" ]]; then
|
||||||
|
printf '%s\n' "$out"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
check_ptrs() {
|
||||||
|
local ip ptr
|
||||||
|
while read -r ip; do
|
||||||
|
[[ -z "$ip" ]] && continue
|
||||||
|
ptr="$(dig_ptr "$ip" || true)"
|
||||||
|
printf 'PTR %s -> %s\n' "$ip" "${ptr:-<none>}"
|
||||||
|
grep -Fiq "${HOST}" <<<"$ptr" || fail "PTR for ${ip} does not mention ${HOST}"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
need_cmd() {
|
||||||
|
command -v "$1" >/dev/null 2>&1 || {
|
||||||
|
echo "missing required command: $1" >&2
|
||||||
|
exit 127
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
need_cmd ssh
|
||||||
|
need_cmd curl
|
||||||
|
need_cmd openssl
|
||||||
|
need_cmd dig
|
||||||
|
need_cmd timeout
|
||||||
|
|
||||||
|
section "SSH + compose ps"
|
||||||
|
PS_OUT="$("${SSH[@]}" 'cd /opt/mail && docker compose ps -a')" || {
|
||||||
|
fail "ssh or docker compose ps"
|
||||||
|
echo "Cannot continue without SSH/compose." >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
printf '%s\n' "$PS_OUT"
|
||||||
|
if grep -qiE 'Exited|Restarting|[[:space:]]Dead[[:space:]]' <<<"$PS_OUT"; then
|
||||||
|
fail "compose has Exited/Restarting/Dead containers"
|
||||||
|
fi
|
||||||
|
for svc in nginx-mailcow postfix-mailcow dovecot-mailcow mysql-mailcow; do
|
||||||
|
grep -Fq "$svc" <<<"$PS_OUT" || fail "missing service ${svc}"
|
||||||
|
done
|
||||||
|
|
||||||
|
section "Watchdog (tail)"
|
||||||
|
if ! "${SSH[@]}" 'cd /opt/mail && docker compose logs --tail=40 watchdog-mailcow'; then
|
||||||
|
fail "watchdog logs"
|
||||||
|
fi
|
||||||
|
|
||||||
|
section "Mail queue"
|
||||||
|
QUEUE_OUT="$("${SSH[@]}" 'cd /opt/mail && docker compose exec -T postfix-mailcow postqueue -p' 2>&1)" || {
|
||||||
|
fail "postqueue"
|
||||||
|
QUEUE_OUT=""
|
||||||
|
}
|
||||||
|
printf '%s\n' "$QUEUE_OUT"
|
||||||
|
if [[ -n "$QUEUE_OUT" ]] && ! grep -Fiq 'Mail queue is empty' <<<"$QUEUE_OUT"; then
|
||||||
|
fail "mail queue not empty"
|
||||||
|
fi
|
||||||
|
|
||||||
|
section "Listeners"
|
||||||
|
LISTEN_OUT="$("${SSH[@]}" 'ss -tlnp 2>/dev/null | grep -E ":(25|465|587|110|143|993|995|80|443|4190)[[:space:]]" || true')"
|
||||||
|
printf '%s\n' "$LISTEN_OUT"
|
||||||
|
for p in 25 465 587 110 143 993 995 80 443 4190; do
|
||||||
|
if ! grep -qE ":${p}[[:space:]]" <<<"$LISTEN_OUT"; then
|
||||||
|
fail "port ${p} not listening on host"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
section "HTTP / HTTPS"
|
||||||
|
HTTP_OUT="$(curl "${CURL_AF_OPTS[@]}" -sS -I --max-time 10 "http://${HOST}/" 2>&1 | head -8)" || true
|
||||||
|
printf '%s\n' "$HTTP_OUT"
|
||||||
|
echo "---"
|
||||||
|
# No -k: surface TLS trust problems
|
||||||
|
HTTPS_OUT="$(curl "${CURL_AF_OPTS[@]}" -sS -I --max-time 10 "https://${HOST}/" 2>&1 | head -15)" || true
|
||||||
|
printf '%s\n' "$HTTPS_OUT"
|
||||||
|
grep -qE '^HTTP/[0-9.]+ 301' <<<"$HTTP_OUT" || fail "HTTP did not redirect (expect 301)"
|
||||||
|
grep -qE '^HTTP/[0-9.]+ 200' <<<"$HTTPS_OUT" || fail "HTTPS did not return 200"
|
||||||
|
|
||||||
|
section "TLS cert (443)"
|
||||||
|
CERT_OUT="$(
|
||||||
|
timeout 15 openssl s_client "${OPENSSL_AF_OPTS[@]}" -connect "${HOST}:443" -servername "${HOST}" </dev/null 2>/dev/null \
|
||||||
|
| openssl x509 -noout -subject -issuer -dates 2>/dev/null
|
||||||
|
)" || true
|
||||||
|
printf '%s\n' "$CERT_OUT"
|
||||||
|
if ! grep -Fq "CN=${HOST}" <<<"$CERT_OUT" && ! grep -Fq "CN = ${HOST}" <<<"$CERT_OUT"; then
|
||||||
|
fail "TLS subject missing ${HOST}"
|
||||||
|
fi
|
||||||
|
grep -Fiq "Let's Encrypt" <<<"$CERT_OUT" || fail "TLS issuer not Let's Encrypt"
|
||||||
|
|
||||||
|
section "SMTP banner"
|
||||||
|
# Connect via A record so /dev/tcp does not pick unreachable AAAA first
|
||||||
|
SMTP_IP="$(dig_short "${HOST}" A 2>/dev/null | head -n1 || true)"
|
||||||
|
if [[ -z "$SMTP_IP" ]]; then
|
||||||
|
fail "cannot resolve A for SMTP check"
|
||||||
|
SMTP_OUT=""
|
||||||
|
else
|
||||||
|
SMTP_OUT="$(
|
||||||
|
timeout 8 bash -c "exec 3<>/dev/tcp/${SMTP_IP}/25; printf 'EHLO test.local\r\nQUIT\r\n' >&3; cat <&3" 2>/dev/null \
|
||||||
|
| head -20
|
||||||
|
)" || true
|
||||||
|
fi
|
||||||
|
printf 'connect %s:25\n' "${SMTP_IP:-?}"
|
||||||
|
printf '%s\n' "$SMTP_OUT"
|
||||||
|
grep -qiE 'Postcow|ESMTP' <<<"$SMTP_OUT" || fail "SMTP banner unexpected"
|
||||||
|
|
||||||
|
section "DNS"
|
||||||
|
A_REC="$(dig_short "${HOST}" A || true)"
|
||||||
|
AAAA_REC="$(dig_short "${HOST}" AAAA || true)"
|
||||||
|
printf 'A: %s\n' "${A_REC:-<none>}"
|
||||||
|
printf 'AAAA: %s\n' "${AAAA_REC:-<none>}"
|
||||||
|
[[ -n "$A_REC" ]] || fail "missing A record"
|
||||||
|
[[ -n "$AAAA_REC" ]] || fail "missing AAAA record"
|
||||||
|
|
||||||
|
check_ptrs <<<"$A_REC"
|
||||||
|
check_ptrs <<<"$AAAA_REC"
|
||||||
|
|
||||||
|
MX_REC="$(dig_short windy.me MX || true)"
|
||||||
|
printf 'MX:\n%s\n' "${MX_REC:-<none>}"
|
||||||
|
grep -Fiq "${HOST}" <<<"$MX_REC" || fail "windy.me MX missing ${HOST}"
|
||||||
|
|
||||||
|
SPF="$(dig_short windy.me TXT || true)"
|
||||||
|
printf 'SPF:\n'
|
||||||
|
if ! grep -Fi 'v=spf1' <<<"$SPF"; then
|
||||||
|
fail "no SPF TXT on windy.me"
|
||||||
|
fi
|
||||||
|
|
||||||
|
section "Done"
|
||||||
|
if [[ "$FAILURES" -gt 0 ]]; then
|
||||||
|
echo "Health check finished with ${FAILURES} failure(s)."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Health check OK. Update hosts/mx2.windy.me.md Verified line if desired."
|
||||||
|
exit 0
|
||||||
Reference in New Issue
Block a user