vps/AGENTS.md

# VPS Agent Ops Hub

This repo is the **agent ops handbook + fact source** for maintaining personal VPS hosts. Prefer verifying live state over assuming docs are complete.

Also readable as `agent.md` (symlink → this file).

## How to work

1. Read [`inventory/hosts.md`](inventory/hosts.md) for the machine list.
2. Open the matching [`hosts/<name>.md`](hosts/) for SSH, roles, paths, and quirks.
3. For common tasks, follow a runbook under [`runbooks/`](runbooks/). Pick the
   most specific applicable one from [`runbooks/README.md`](runbooks/README.md);
   the spec is [`RUNBOOKS.md`](RUNBOOKS.md) and new runbooks start from
   [`runbooks/_template.md`](runbooks/_template.md).
4. Prefer read-only checks first; change only after confirming current state.
5. For routine checks and approved service reconciliation, run the matching
   Ansible playbook from `ansible/`; see [routine Ansible operations](runbooks/ansible-operations.md).
6. Default SSH access (`ssh -4 windy@<host>`) is for focused diagnostics,
   imperative upstream procedures, and incident work. Prefer **IPv4** from this
   WSL client (AAAA often exists but IPv6 route does not).

   > **Agent sandbox SSH quirk (verified 2026-08-20):** the agent shell runs in
   > a sandboxed user namespace — system files such as
   > `/etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf` appear owned by `nobody`,
   > so plain `ssh` aborts with `Bad owner or permissions on ...`. Always use
   > `ssh -F /dev/null` from the agent shell and pass options explicitly
   > (`~/.ssh/config` is skipped; e.g. `ssh -F /dev/null -p 2222
   > -i ~/.ssh/id_ed25519 windy@repo.windy.me`). `sudo` never works in the
   > sandbox (`NoNewPrivs`, no capabilities, `/` read-only). The host itself is
   > healthy — to inspect or act on the real host from the sandbox use
   > `/mnt/c/WINDOWS/system32/wsl.exe -u root -- <cmd>` (real root: keep
   > read-only unless a change is approved).

7. Record each material VPS operation, incident, configuration change, or
   verification outcome in the corresponding **Linear `vps` project**. Include
   scope, action, verification, and remaining follow-up; never put passwords,
   tokens, private keys, recovery keys, or private room IDs in Linear.

### Runbook execution rules

Before operational work: inspect `runbooks/`, select the most specific
applicable runbook, follow its steps in order, do not skip verification steps,
and respect its STOP and approval conditions. If no runbook applies, diagnose
only — do not mutate production state. When live state conflicts with a
runbook's assumptions, `STOP` and report; never invent missing parameters or
bypass failed checks. The spec is [`RUNBOOKS.md`](RUNBOOKS.md).

## Active hosts

The canonical machine list (roles, SSH endpoints, Ansible coverage, status) is
[`inventory/hosts.md`](inventory/hosts.md) — the single human-readable source
of truth. Per-host facts live in [`hosts/`](hosts/). The Ansible execution
inventory is [`ansible/inventory/hosts.yml`](ansible/inventory/hosts.yml). Do
not maintain a second copy of the machine table here.

### Public services

| URL / endpoint | Backend host | Notes |
|----------------|--------------|--------|
| https://mx2.windy.me | mx2 | mailcow UI / SOGo |
| SMTP `mx2.windy.me:587` (STARTTLS) or `:465` | mx2 | client submission; full email + mailbox password — [runbook](runbooks/mailcow-smtp-client.md) |
| IMAP `mx2.windy.me:993` | mx2 | same mailbox credentials |
| https://auth.wsvc.info | us2 (`/opt/vaultwarden`) | Vaultwarden (Postgres, **operational**) — client Server URL |
| `repo.windy.me:2222` | us2 (`/opt/soft-serve`) | Soft Serve (stub details) |
| DNS `ns1.wsvc.info:53` | hk2 (`/opt/pdns`, Auth **5.0.6**) | PowerDNS auth — zones `windy.me`, `wsvc.info`, `chans.xyz` |
| https://pdns.wsvc.info | hk2 (`poweradmin`) | Poweradmin UI |
| https://pgweb.wsvc.info | hk2 (`pgweb`) | PowerDNS Postgres browser |
| https://chat.chans.xyz | synapse (`ess`, ESS chart `26.7.2`) | Element Web (Matrix client) |
| https://synapse.chans.xyz | synapse | Synapse Client-Server + Federation API |
| https://account.chans.xyz | synapse | Matrix Authentication Service (local passwords) |
| https://admin.chans.xyz | synapse | Element Admin console (MAS admin auth) |

### Upstream docs

**Vaultwarden:** Before changing config, read [docs/vaultwarden-upstream.md](docs/vaultwarden-upstream.md). Live DB is **Postgres** (`DATABASE_URL`); `vw-data/config.json` from `/admin` overrides `.env` (including SMTP password — keep it synced with `.env` / `.smtp-credentials`). After recreating the VW container, restart Traefik if `https://auth.wsvc.info` returns 404. SMTP health needs **AUTH**, not TCP-only — see [vaultwarden-health](runbooks/vaultwarden-health.md).

**PowerDNS:** Before changing `/opt/pdns` on hk2, read [docs/pdns-upstream.md](docs/pdns-upstream.md). Auth pin **5.0.6**; primary + `also-notify`/`only-notify=` + DNSSEC OK. Deferred: rotate weak API/DB credentials and TSIG (`TSIG-ALLOW-AXFR` bypasses IP ACL). Health: [pdns-health](runbooks/pdns-health.md).

**Matrix (ESS on synapse):** Matrix homeserver running on `synapse.chans.xyz` via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone `chans.xyz` managed by hk2 PowerDNS. Before changing config, read [docs/matrix-upstream.md](docs/matrix-upstream.md) and [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: [matrix-health](runbooks/matrix-health.md).

**RustDesk:** Self-hosted RustDesk server on `hk2.chans.xyz` (`/opt/rustdesk`, containers `hbbs`/`hbbr`, image pinned `1.1.14`). The `hbbs -r` relay hostname must resolve to the host's public IP `154.36.174.161` — use `hk2.chans.xyz` (never `hk2.wsvc.info`, which has no DNS record). Health: [rustdesk-health](runbooks/rustdesk-health.md).

## Runbooks & scripts

| Task | Path |
|------|------|
| mailcow health | [runbooks/mailcow-health.md](runbooks/mailcow-health.md) |
| mailcow update | [runbooks/mailcow-update.md](runbooks/mailcow-update.md) |
| mailcow SMTP/IMAP client | [runbooks/mailcow-smtp-client.md](runbooks/mailcow-smtp-client.md) |
| Vaultwarden health | [runbooks/vaultwarden-health.md](runbooks/vaultwarden-health.md) |
| Vaultwarden SQLite→PG migrate | [runbooks/vaultwarden-sqlite-to-postgres.md](runbooks/vaultwarden-sqlite-to-postgres.md) |
| Vaultwarden upstream refs | [docs/vaultwarden-upstream.md](docs/vaultwarden-upstream.md) |
| PowerDNS health (hk2) | [runbooks/pdns-health.md](runbooks/pdns-health.md) |
| PowerDNS upstream refs | [docs/pdns-upstream.md](docs/pdns-upstream.md) |
| Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) |
| RustDesk health (hk2) | [runbooks/rustdesk-health.md](runbooks/rustdesk-health.md) |
| AdGuard Home health | [runbooks/adguard-home-health.md](runbooks/adguard-home-health.md) |
| Home Assistant maintenance | [runbooks/home-assistant-maintenance.md](runbooks/home-assistant-maintenance.md) + [scripts/ha-maintenance.sh](runbooks/scripts/ha-maintenance.sh) |
| matrix_e2ee update (hass.windy.lan) | [runbooks/matrix-e2ee-update.md](runbooks/matrix-e2ee-update.md) |
| Matrix upstream refs | [docs/matrix-upstream.md](docs/matrix-upstream.md) |
| Hermes Agent Matrix channel | [docs/hermes-matrix.md](docs/hermes-matrix.md) |
| UniFi local-service proxy bypass | [docs/unifi-openclash-localhost.md](docs/unifi-openclash-localhost.md) |
| UniFi SSO login setting (Ansible) | `cd ansible && ansible-playbook playbooks/unifi-sso.yml --limit unifi` |
| Routine Ansible operations | [runbooks/ansible-operations.md](runbooks/ansible-operations.md) |
| Issue → mergeable change | [runbooks/issue-to-merge.md](runbooks/issue-to-merge.md) |
| Fix failing health/playbook run | [runbooks/fix-ci.md](runbooks/fix-ci.md) |
| Release a reviewed change | [runbooks/release.md](runbooks/release.md) |
| Roll back a change | [runbooks/rollback.md](runbooks/rollback.md) |
| Controlled network change | [runbooks/network-change.md](runbooks/network-change.md) |
| Network outage recovery | [runbooks/network-recovery.md](runbooks/network-recovery.md) |

Full index: [runbooks/README.md](runbooks/README.md). Spec: [RUNBOOKS.md](RUNBOOKS.md).

Routine mailcow health: `cd ansible && ansible-playbook playbooks/health-report.yml --limit mailcow`. The local stub resolver is flaky; DNS probes use `1.1.1.1` / `8.8.8.8`.

## Agent skills

### Issue tracker

Issues are tracked in Linear and created/updated via the Linear MCP (`vps` project). See `docs/agents/issue-tracker.md`.

### Triage labels

Default triage labels: needs-triage, needs-info, ready-for-agent, ready-for-human, wontfix. See `docs/agents/triage-labels.md`.

### Domain docs

Single-context layout: `CONTEXT.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.

## Safety

- Never commit secrets: passwords, API keys, private keys, `.env`, `mailcow.conf` DB passwords, Vaultwarden `ADMIN_TOKEN` / `.smtp-credentials`.
- Secrets live **only on the servers** (e.g. `/opt/vaultwarden/.env`). Do not copy them from Obsidian into this repo.
- Ask before destructive actions: `docker compose down -v`, DB drops, DNS changes, force-push, disk wipes.
- Ansible reconciliation is intentionally allowlisted and requires an explicit
  confirmation variable; it is not a general remote-command facility.
- Do not modify local `~/.ssh/config` or git config unless explicitly asked.

## SSH access safety — mandatory change procedure

**Treat any SSH authentication or authorization change as a lockout-risk change.** This includes `sshd_config` / drop-in edits, `PasswordAuthentication`, `PermitRootLogin`, `AuthorizedKeysFile`, `Match` blocks, SSH keys, accounts, groups, chroots, firewalls, and SSH ports.

Before disabling an existing access method or reloading `sshd`, all of the following are mandatory:

1. Keep the current verified SSH management session open as the rollback path; do not use it to run the validation connection.
2. Install the intended management public key from a **verified local source**. Never assume a public-key file exists on the target host or copy a remote path without checking it first.
3. Validate the candidate configuration with `sshd -t` before reloading the service.
4. From a separate local terminal, prove the new key-only login works using the exact target account, hostname, and key (`BatchMode=yes`, password authentication disabled), and verify needed sudo access.
5. Only after step 4 succeeds may password login, root login, the old key, or another existing access route be disabled.
6. Keep the original management session open until the final effective SSH configuration and a fresh key-only login have both been verified.

If the new key-based login cannot be proven first, stop immediately: do **not** reload an access-restricting configuration. Provider web/serial console access must be confirmed before any SSH access-policy change, but it is a recovery path—not a substitute for the validation steps above.

## Obsidian

Bills, rough notes, and personal clutter stay in the Obsidian vault. This repo holds **actionable facts and runbooks** only.

## Layout

```
AGENTS.md / agent.md   # this entry (agent.md → AGENTS.md)
RUNBOOKS.md            # runbook spec (six-field model, naming, review rules)
inventory/hosts.md     # machine index (human-readable source of truth)
ansible/               # playbooks, roles, sanitized control-plane inventory
compose/               # repo-owned non-secret Compose sources (+ .env.example)
hosts/                 # per-host facts
runbooks/              # step-by-step ops (README.md = index, _template.md = template)
docs/                  # upstream doc indexes / design notes
scripts/validate-repo.sh  # repo-wide validation (run before merging)
```