windy/vps
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Manage routine operations through Ansible

Browse Source
This commit is contained in:
windyboy
2026-08-03 16:01:35 +08:00
parent b73125e5bc
commit 7a9acb4f7c
25 changed files with 353 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files
runbooks/ansible-operations.md
+61
View File
@@ -0,0 +1,61 @@
# Runbook: routine operations through Ansible
Run commands from `ansible/`. The inventory forces IPv4 and uses the `windy`
account with sudo. Do a read-only health pass before any reconciliation.
## Health report (read-only)
```bash
cd ansible
ansible-playbook playbooks/health-report.yml
ansible-playbook playbooks/health-report.yml --limit mailcow
ansible-playbook playbooks/health-report.yml --limit matrix
```
This invokes the existing server-local checks and prints sanitized output; it
does not expose secrets or alter application configuration. A warning or
critical health result makes the play fail deliberately.
If the local health check has not been installed yet, deploy it first:
```bash
ansible-playbook playbooks/healthchecks.yml
ansible-playbook playbooks/matrix-healthchecks.yml
```
## Reconcile a known Compose service (mutating)
Only use this after a reviewed configuration change that requires a recreate.
The allowlist and flags are in inventory; arbitrary commands and service names
are rejected. Run one host at a time and re-run the health report afterwards.
```bash
# PowerDNS auth after its configuration changed
ansible-playbook playbooks/compose-reconcile.yml --limit powerdns \
-e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["auth"]}'
# Vaultwarden, including the documented Traefik refresh for a post-recreate 404
ansible-playbook playbooks/compose-reconcile.yml --limit vaultwarden \
-e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["vaultwarden"], "service_reconcile_restart_traefik": true}'
# Poweradmin after changing its environment or Traefik labels; restart its
# Traefik container only when the UI is returning 404
ansible-playbook playbooks/compose-reconcile.yml --limit powerdns \
-e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["poweradmin"], "service_reconcile_restart_traefik": true}'
```
Do not use this playbook for a Mailcow update, database migration, DNS record
change, or secret rotation. Those operations require their dedicated reviewed
and, where appropriate, interactive procedures.
## Host-level maintenance
These playbooks cover every inventory host, including the Matrix K3s node:
```bash
# Read-only pending package and reboot report
ansible-playbook playbooks/maintenance-preview.yml
# Baseline observations; logrotate management remains opt-in
ansible-playbook playbooks/baseline.yml
```
runbooks/mailcow-health.md
+11 -5
View File
@@ -2,9 +2,15 @@
Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
Path: `/opt/mail`
Prefer: `./scripts/check-mx2.sh` from this repo, or the steps below over SSH.
Prefer: the Ansible health report (`ansible/playbooks/health-report.yml`), or
`./scripts/check-mx2.sh` from this repo. The SSH commands below remain useful
for focused, interactive diagnosis after the report identifies a problem.
## 1. Containers
## Focused SSH diagnostics
Use these only after the Ansible health report needs investigation.
### 1. Containers
```bash
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose ps -a'
@@ -12,7 +18,7 @@ ssh windy@mx2.windy.me 'cd /opt/mail && docker compose ps -a'
Expect mailcow (`cow-*`) services **Up**. Note any Exit/Restarting.
## 2. Watchdog
### 2. Watchdog
```bash
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose logs --tail=40 watchdog-mailcow'
@@ -20,7 +26,7 @@ ssh windy@mx2.windy.me 'cd /opt/mail && docker compose logs --tail=40 watchdog-m
Expect recent lines at **100%** for Nginx, Postfix, Dovecot, MySQL, Redis, SOGo, Rspamd, ACME, etc.
## 3. Queue
### 3. Queue
```bash
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose exec -T postfix-mailcow postqueue -p'
@@ -28,7 +34,7 @@ ssh windy@mx2.windy.me 'cd /opt/mail && docker compose exec -T postfix-mailcow p
Healthy: empty / `Mail queue is empty`.
## 4. Local listeners
### 4. Local listeners
```bash
ssh windy@mx2.windy.me 'ss -tlnp | grep -E ":(25|465|587|143|993|995|80|443|4190)\\s"'
runbooks/mailcow-update.md
+3 -1
View File
@@ -31,7 +31,9 @@ If update asks to stop/recreate containers, allow it unless the user said otherw
4. If `SKIP_CLAMD=y` but `cow-clamd-mailcow-1` is still Up and that is unwanted:
```bash
ssh windy@mx2.windy.me 'cd /opt/mail && docker compose up -d --force-recreate'
cd ansible
ansible-playbook playbooks/compose-reconcile.yml --limit mailcow \
-e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["all"]}'
```
Only after user OK; recreate briefly interrupts mail.
runbooks/matrix-health.md
+10 -3
View File
@@ -2,7 +2,14 @@
Monitor the Matrix homeserver running on `synapse.chans.xyz` (ESS chart `26.7.2`, K3s node).
## Basic checks
Prefer `cd ansible && ansible-playbook playbooks/health-report.yml --limit matrix`
for the routine report. The SSH commands below remain diagnostic detail.
## Focused SSH diagnostics
Use these only after the Ansible health report needs investigation.
### Basic checks
```bash
# Overall status
@@ -15,7 +22,7 @@ ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get pods -o wide | grep
ssh -4 windy@synapse.chans.xyz 'df -hP /'
```
## Certificate verification
### Certificate verification
```bash
ssh -4 windy@synapse.chans.xyz 'sudo k3s kubectl -n ess get certificate'
@@ -29,7 +36,7 @@ All certificates should show `READY=True`. Domains:
- `admin.chans.xyz`
- `mrtc.chans.xyz` (placeholder only)
## External endpoint verification
### External endpoint verification
```bash
# Client API
runbooks/pdns-health.md
+18 -10
View File
@@ -4,7 +4,15 @@ Read-only checks for the `/opt/pdns` stack on **hk2.chans.xyz** (`ns1.wsvc.info`
Facts: [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md) · Upstream: [docs/pdns-upstream.md](../docs/pdns-upstream.md)
## Quick remote
Prefer the Ansible health report (`ansible/playbooks/health-report.yml` with
`--limit powerdns`) for routine checks. Keep the SSH commands below for focused
diagnosis, especially direct authoritative DNS queries.
## Focused SSH diagnostics
Use these only after the Ansible health report needs investigation.
### Quick remote
```bash
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose ps -a'
@@ -12,7 +20,7 @@ ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose ps -a'
Expect `auth`, `db`, `poweradmin` healthy; `backup` Up; `pgweb` Up. Ignore stopped orphan `powerdns-admin` unless cleaning orphans.
## Version / security poll
### Version / security poll
Expect Authoritative **5.0.6** (or newer). Startup logs should **not** show a Mandatory `Security Update` banner. If they do, check https://doc.powerdns.com/authoritative/security-advisories/ and bump the `powerdns/pdns-auth-50` image pin in `/opt/pdns/compose.yml`.
@@ -21,7 +29,7 @@ ssh -4 windy@hk2.chans.xyz 'docker logs pdns-auth 2>&1 | grep -i "Security Updat
ssh -4 windy@hk2.chans.xyz 'docker inspect pdns-auth --format "{{.Config.Image}}"'
```
## DNS (from laptop / WSL)
### DNS (from laptop / WSL)
Prefer querying auth/secondary IPs directly (local stub / public resolvers may be flaky or cached):
@@ -40,7 +48,7 @@ dig @$NS1 +short A ns2.wsvc.info # expect 202.91.35.141
Served SOA serial can differ from the backend/API serial when `SOA-EDIT=INCEPTION-INCREMENT` is active — compare **ns1 vs ns2 served SOA**, not dig vs `pdnsutil`/API.
## API (on host)
### API (on host)
```bash
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose exec -T auth \
@@ -49,7 +57,7 @@ ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && docker compose exec -T auth \
Or use host loopback with the key from `.env` (do not paste the key into chat/logs).
## Web UIs
### Web UIs
```bash
# If local stub DNS is flaky, pin SNI to the auth IP:
@@ -57,7 +65,7 @@ curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pdns.wsvc.info:443:154.36
curl -4 -sS -o /dev/null -w "%{http_code}\n" --resolve pgweb.wsvc.info:443:154.36.174.161 https://pgweb.wsvc.info/ # 401 without basic auth
```
## Backup
### Backup
```bash
ssh -4 windy@hk2.chans.xyz 'cd /opt/pdns && ./scripts/manual-backup.sh && ls -lt backup/*.sql.gz | head'
@@ -66,7 +74,7 @@ ssh -4 windy@hk2.chans.xyz 'tail -30 /opt/pdns/backup/backup.log'
Scheduler should log `scheduler started … (daily 03:00)`. If you see `can't execute 'bash'` or Alpine `apk` errors, the backup image/entrypoint was reverted — see [hosts/hk2.chans.xyz.md](../hosts/hk2.chans.xyz.md).
## Config sanity (docs-aligned)
### Config sanity (docs-aligned)
```bash
ssh -4 windy@hk2.chans.xyz 'grep -E "^(primary|also-notify|only-notify|allow-axfr-ips|gpgsql-dnssec|default-soa-edit)" /opt/pdns/auth/pdns.conf'
@@ -81,6 +89,6 @@ The sanitized Ansible health profile is `pdns` (`ansible/playbooks/healthchecks.
## After config changes
- `auth/pdns.conf`, `auth/templates.d/secrets.j2`, or auth-related `.env` → `docker compose up -d --force-recreate auth`
- Poweradmin env / Traefik labels → `docker compose up -d --no-deps --force-recreate poweradmin` (restart Traefik if UI 404s)
- backup schedule / retention env → `docker compose up -d --force-recreate --no-deps backup`
- `auth/pdns.conf`, `auth/templates.d/secrets.j2`, or auth-related `.env` → use the Ansible Compose reconcile playbook with target `auth`
- Poweradmin env / Traefik labels → use the Ansible Compose reconcile playbook with target `poweradmin`; add `service_reconcile_restart_traefik=true` only if its UI returns 404
- backup schedule / retention env → use the Ansible Compose reconcile playbook with target `backup`
runbooks/vaultwarden-health.md
+15 -6
View File
@@ -5,7 +5,15 @@ Path: `/opt/vaultwarden`
URL: https://auth.wsvc.info/
Upstream: [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
## 1. Containers
Prefer the Ansible health report (`ansible/playbooks/health-report.yml` with
`--limit vaultwarden`) for the routine check. The SSH commands below are
focused diagnostics for an already identified problem.
## Focused SSH diagnostics
Use these only after the Ansible health report needs investigation.
### 1. Containers
```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose ps -a'
@@ -13,14 +21,14 @@ ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose ps -a'
Expect `vaultwarden` + `vw-db` **Up (healthy)**. `pgweb` should be stopped unless started with `--profile debug`.
## 2. Live DB is Postgres
### 2. Live DB is Postgres
```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden sh -c "printenv DATABASE_URL" | sed -E "s#://[^@]+@#://REDACTED@#"'
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T pg psql -U vaultwarden -d vaultwarden -c "SELECT COUNT(*) AS users FROM users; SELECT COUNT(*) AS ciphers FROM ciphers;"'
```
## 3. Effective config (config.json wins)
### 3. Effective config (config.json wins)
```bash
ssh -4 windy@us2.wsvc.info 'cd /opt/vaultwarden && docker compose exec -T vaultwarden cat /data/config.json' \
@@ -48,15 +56,16 @@ EOF'
If they diverge, **auth uses `config.json`** — sync password there (or via `/admin`) and update `.smtp-credentials`.
## 4. External HTTPS
### 4. External HTTPS
```bash
curl -4 -sS -I --max-time 15 https://auth.wsvc.info/ | head -15
```
If **404** right after recreating VW: `ssh -4 windy@us2.wsvc.info 'docker restart traefik'` then retry.
If **404** right after recreating VW: use the Ansible Compose reconcile playbook
with `service_reconcile_restart_traefik=true`, then retry.
## 5. SMTP reachability + AUTH
### 5. SMTP reachability + AUTH
TCP (expect `587:0`; `465` often times out from us2):
runbooks/vaultwarden-sqlite-to-postgres.md
+3 -1
View File
@@ -3,7 +3,9 @@
Target: [us2.wsvc.info](../hosts/us2.wsvc.info.md)
Upstream: [Using the PostgreSQL Backend](https://github.com/dani-garcia/vaultwarden/wiki/Using-the-PostgreSQL-Backend) · [docs/vaultwarden-upstream.md](../docs/vaultwarden-upstream.md)
**Done on 2026-08-01** for this host. Keep this as the replay checklist.
**Done on 2026-08-01** for this host. Keep this as the replay checklist. This
is an exceptional, destructive migration procedure: it is intentionally not
automated by the routine Ansible reconciliation playbook.
## Summary of what we did