vps/runbooks/ansible-operations.md

# Runbook: routine operations through Ansible

## Purpose

Routine operations (health, reconcile, maintenance) through the Ansible playbooks.

## Scope

- Applicable: every inventory host, run from `ansible/`.
- Not applicable: arbitrary remote commands — the reconcile playbook is allowlisted and gated.

Run commands from `ansible/`. The inventory forces IPv4 and uses the `windy`
account with sudo. Do a read-only health pass before any reconciliation.

## Safety

- Read-only health pass before any reconciliation.
- Mutating playbooks require explicit confirmation variables; do not bypass them.
- If a reconcile target or service name is not allowlisted, `STOP` — do not invent one.

## Ownership

- Owner: personal ops (Windy)
- Last reviewed: 2026-08-17
- Related systems: Ansible control-plane + all inventory hosts

## Health report (read-only)

```bash
cd ansible
ansible-playbook playbooks/health-report.yml
ansible-playbook playbooks/health-report.yml --limit mailcow
ansible-playbook playbooks/health-report.yml --limit matrix
```

This invokes the existing server-local checks and prints sanitized output; it
does not expose secrets or alter application configuration. A warning or
critical health result makes the play fail deliberately.

If the local health check has not been installed yet, deploy it first:

```bash
ansible-playbook playbooks/healthchecks.yml
ansible-playbook playbooks/matrix-healthchecks.yml
```

## Reconcile a known Compose service (mutating)

Only use this after a reviewed configuration change that requires a recreate.
The allowlist and flags are in inventory; arbitrary commands and service names
are rejected. Run one host at a time and re-run the health report afterwards.

```bash
# PowerDNS auth after its configuration changed
ansible-playbook playbooks/compose-reconcile.yml --limit powerdns \
  -e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["auth"]}'

# Vaultwarden, including the documented Traefik refresh for a post-recreate 404
ansible-playbook playbooks/compose-reconcile.yml --limit vaultwarden \
  -e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["vaultwarden"], "service_reconcile_restart_traefik": true}'

# Poweradmin after changing its environment or Traefik labels; restart its
# Traefik container only when the UI is returning 404
ansible-playbook playbooks/compose-reconcile.yml --limit powerdns \
  -e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["poweradmin"], "service_reconcile_restart_traefik": true}'
```

Do not use this playbook for a Mailcow update, database migration, DNS record
change, or secret rotation. Those operations require their dedicated reviewed
and, where appropriate, interactive procedures.

## Deploy repo-owned Compose (static projects)

Repo source: `compose/<project>/compose.yml` (non-secret; secrets come from the
server-local `.env` via `${VAR}`). Mechanism and per-project status:
[`compose/README.md`](../compose/README.md).

```bash
# Read-only: staged-file diff + allowlist/confirmation asserts, no writes
ansible-playbook playbooks/compose-deploy.yml --limit vaultwarden --check --diff
ansible-playbook playbooks/compose-deploy.yml --limit powerdns --check --diff

# Apply: stage repo file → validate `docker compose config -q` against the
# server .env → backup current file (*.bak-<ts>) → promote → `up -d` (gated)
ansible-playbook playbooks/compose-deploy.yml --limit vaultwarden \
  -e '{"compose_deploy_confirm": true}'
```

The playbook never writes, reads, or transfers the server `.env`. A failed
validation never touches the live compose file. Hosts without an allowlisted
`compose_repo_project` fail the assert — do not invent targets.

## Host-level maintenance

These playbooks cover every inventory host, including the Matrix K3s node:

```bash
# Read-only pending package and reboot report
ansible-playbook playbooks/maintenance-preview.yml

# Baseline observations; logrotate management remains opt-in
ansible-playbook playbooks/baseline.yml
```

## us4 firewalld reconciliation

The us4 playbook owns only the audited `public` zone allowlist. It fails closed
on unknown services or ports, never reloads/restarts firewalld, and does not
manage Docker-published ports.

```bash
cd ansible
ansible-galaxy collection install -r requirements.yml

# Read-only preview
ansible-playbook playbooks/us4-firewalld.yml --limit us4 --check --diff

# Apply only after testing the provider console and retaining an independent
# SSH rollback session.
ansible-playbook playbooks/us4-firewalld.yml --limit us4 \
  -e '{"us4_firewalld_confirm": true, "us4_console_confirm": true}'
```

Apply creates a protected server-local backup and schedules a 15-minute
automatic rollback before changing rules. The rollback is cancelled only after
the playbook verifies fresh SSH/sudo access, public HTTPS routes, SMTP, Docker,
Fail2ban, and WireGuard. Do not bypass either confirmation variable.

## UniFi SSO login setting (mutating)

Reconciles `super_sdn.sso_login_enabled` on the UniFi controller (host `ubnt`,
group `unifi`). Idempotent and gated: without `unifi_sso_confirm=true` the
playbook only reports the current state and refuses to change anything.

```bash
# Read-only status report
ansible-playbook playbooks/unifi-sso.yml --limit unifi --check

# Apply (disable SSO login; local accounts use local passwords, no MFA)
ansible-playbook playbooks/unifi-sso.yml --limit unifi \
  -e '{"unifi_sso_confirm": true, "unifi_sso_target_value": false}'

# Rollback (re-enable SSO login)
ansible-playbook playbooks/unifi-sso.yml --limit unifi \
  -e '{"unifi_sso_confirm": true, "unifi_sso_target_value": true}'
```

The controller container is not restarted by default; the UI path applies the
setting immediately. Set `unifi_sso_restart_controller: true` only when the
setting was changed directly in the DB while the controller was running.