# VPS Agent Ops Hub

This repo is the **agent ops handbook + fact source** for maintaining personal VPS hosts. Prefer verifying live state over assuming docs are complete.

Also readable as `agent.md` (symlink → this file).

## How to work

1. Read [`inventory/hosts.md`](inventory/hosts.md) for the machine list.
2. Open the matching [`hosts/<name>.md`](hosts/) for SSH, roles, paths, and quirks.
3. For common tasks, follow a runbook under [`runbooks/`](runbooks/).
4. Prefer read-only checks first; change only after confirming current state.
5. For routine checks and approved service reconciliation, run the matching
   Ansible playbook from `ansible/`; see [routine Ansible operations](runbooks/ansible-operations.md).
6. Default SSH access (`ssh -4 windy@<host>`) is for focused diagnostics,
   imperative upstream procedures, and incident work. Prefer **IPv4** from this
   WSL client (AAAA often exists but IPv6 route does not).
7. Record each material VPS operation, incident, configuration change, or
   verification outcome in the corresponding **Linear `vps` project**. Include
   scope, action, verification, and remaining follow-up; never put passwords,
   tokens, private keys, recovery keys, or private room IDs in Linear.

## Active hosts (quick map)

| Host | Role | SSH | Facts |
|------|------|-----|--------|
| **mx2.windy.me** | mailcow (`/opt/mail`, project `cow`) | `ssh -4 windy@mx2.windy.me` | [hosts/mx2.windy.me.md](hosts/mx2.windy.me.md) |
| **us2.wsvc.info** | Vaultwarden + Traefik (+ Soft Serve, …) | `ssh -4 windy@us2.wsvc.info` | [hosts/us2.wsvc.info.md](hosts/us2.wsvc.info.md) |
| **hk2.chans.xyz** | PowerDNS auth ns1 (`/opt/pdns`) | `ssh -4 windy@hk2.chans.xyz` | [hosts/hk2.chans.xyz.md](hosts/hk2.chans.xyz.md) |
| **synapse.chans.xyz** | Matrix ESS (Synapse + MAS + Element) on K3s | `ssh -4 windy@synapse.chans.xyz` | [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md) |
| **dns.windy.lan** | AdGuard Home LAN DNS + Mihomo explicit proxy | `ssh -4 windy@192.168.66.36` | [hosts/dns.windy.lan.md](hosts/dns.windy.lan.md) |
| **gfw.windy.lan** | OpenWrt LAN gateway / OpenClash | `ssh -4 root@192.168.66.1` | [hosts/gfw.windy.lan.md](hosts/gfw.windy.lan.md) |
| **gw** | EdgeRouter X primary LAN gateway | `ssh -4 zhiqiang@192.168.66.254` | [hosts/gw.md](hosts/gw.md) |
| **ubnt** | UniFi Network Controller | `ssh -4 windy@192.168.66.46` | [hosts/ubnt.md](hosts/ubnt.md) |

### Public services

| URL / endpoint | Backend host | Notes |
|----------------|--------------|--------|
| https://mx2.windy.me | mx2 | mailcow UI / SOGo |
| SMTP `mx2.windy.me:587` (STARTTLS) or `:465` | mx2 | client submission; full email + mailbox password — [runbook](runbooks/mailcow-smtp-client.md) |
| IMAP `mx2.windy.me:993` | mx2 | same mailbox credentials |
| https://auth.wsvc.info | us2 (`/opt/vaultwarden`) | Vaultwarden (Postgres, **operational**) — client Server URL |
| `repo.windy.me:2222` | us2 (`/opt/soft-serve`) | Soft Serve (stub details) |
| DNS `ns1.wsvc.info:53` | hk2 (`/opt/pdns`, Auth **5.0.6**) | PowerDNS auth — zones `windy.me`, `wsvc.info`, `chans.xyz` |
| https://pdns.wsvc.info | hk2 (`poweradmin`) | Poweradmin UI |
| https://pgweb.wsvc.info | hk2 (`pgweb`) | PowerDNS Postgres browser |
| https://chat.chans.xyz | synapse (`ess`, ESS chart `26.7.2`) | Element Web (Matrix client) |
| https://synapse.chans.xyz | synapse | Synapse Client-Server + Federation API |
| https://account.chans.xyz | synapse | Matrix Authentication Service (local passwords) |
| https://admin.chans.xyz | synapse | Element Admin console (MAS admin auth) |

### Upstream docs

**Vaultwarden:** Before changing config, read [docs/vaultwarden-upstream.md](docs/vaultwarden-upstream.md). Live DB is **Postgres** (`DATABASE_URL`); `vw-data/config.json` from `/admin` overrides `.env` (including SMTP password — keep it synced with `.env` / `.smtp-credentials`). After recreating the VW container, restart Traefik if `https://auth.wsvc.info` returns 404. SMTP health needs **AUTH**, not TCP-only — see [vaultwarden-health](runbooks/vaultwarden-health.md).

**PowerDNS:** Before changing `/opt/pdns` on hk2, read [docs/pdns-upstream.md](docs/pdns-upstream.md). Auth pin **5.0.6**; primary + `also-notify`/`only-notify=` + DNSSEC OK. Deferred: rotate weak API/DB credentials and TSIG (`TSIG-ALLOW-AXFR` bypasses IP ACL). Health: [pdns-health](runbooks/pdns-health.md).

**Matrix (ESS on synapse):** Matrix homeserver running on `synapse.chans.xyz` via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone `chans.xyz` managed by hk2 PowerDNS. Before changing config, read [docs/matrix-upstream.md](docs/matrix-upstream.md) and [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md). K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: [matrix-health](runbooks/matrix-health.md).

## Runbooks & scripts

| Task | Path |
|------|------|
| mailcow health | [runbooks/mailcow-health.md](runbooks/mailcow-health.md) |
| mailcow update | [runbooks/mailcow-update.md](runbooks/mailcow-update.md) |
| mailcow SMTP/IMAP client | [runbooks/mailcow-smtp-client.md](runbooks/mailcow-smtp-client.md) |
| Vaultwarden health | [runbooks/vaultwarden-health.md](runbooks/vaultwarden-health.md) |
| Vaultwarden SQLite→PG migrate | [runbooks/vaultwarden-sqlite-to-postgres.md](runbooks/vaultwarden-sqlite-to-postgres.md) |
| Vaultwarden upstream refs | [docs/vaultwarden-upstream.md](docs/vaultwarden-upstream.md) |
| PowerDNS health (hk2) | [runbooks/pdns-health.md](runbooks/pdns-health.md) |
| PowerDNS upstream refs | [docs/pdns-upstream.md](docs/pdns-upstream.md) |
| Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) |
| AdGuard Home health | [runbooks/adguard-home-health.md](runbooks/adguard-home-health.md) |
| Matrix upstream refs | [docs/matrix-upstream.md](docs/matrix-upstream.md) |
| Hermes Agent Matrix channel | [docs/hermes-matrix.md](docs/hermes-matrix.md) |
| UniFi local-service proxy bypass | [docs/unifi-openclash-localhost.md](docs/unifi-openclash-localhost.md) |
| Routine Ansible operations | [runbooks/ansible-operations.md](runbooks/ansible-operations.md) |

Routine mailcow health: `cd ansible && ansible-playbook playbooks/health-report.yml --limit mailcow`. The local stub resolver is flaky; DNS probes use `1.1.1.1` / `8.8.8.8`.

## Agent skills

### Issue tracker

Issues are tracked in Linear and created/updated via the Linear MCP (`vps` project). See `docs/agents/issue-tracker.md`.

### Triage labels

Default triage labels: needs-triage, needs-info, ready-for-agent, ready-for-human, wontfix. See `docs/agents/triage-labels.md`.

## Safety

- Never commit secrets: passwords, API keys, private keys, `.env`, `mailcow.conf` DB passwords, Vaultwarden `ADMIN_TOKEN` / `.smtp-credentials`.
- Secrets live **only on the servers** (e.g. `/opt/vaultwarden/.env`). Do not copy them from Obsidian into this repo.
- Ask before destructive actions: `docker compose down -v`, DB drops, DNS changes, force-push, disk wipes.
- Ansible reconciliation is intentionally allowlisted and requires an explicit
  confirmation variable; it is not a general remote-command facility.
- Do not modify local `~/.ssh/config` or git config unless explicitly asked.

## SSH access safety — mandatory change procedure

**Treat any SSH authentication or authorization change as a lockout-risk change.** This includes `sshd_config` / drop-in edits, `PasswordAuthentication`, `PermitRootLogin`, `AuthorizedKeysFile`, `Match` blocks, SSH keys, accounts, groups, chroots, firewalls, and SSH ports.

Before disabling an existing access method or reloading `sshd`, all of the following are mandatory:

1. Keep the current verified SSH management session open as the rollback path; do not use it to run the validation connection.
2. Install the intended management public key from a **verified local source**. Never assume a public-key file exists on the target host or copy a remote path without checking it first.
3. Validate the candidate configuration with `sshd -t` before reloading the service.
4. From a separate local terminal, prove the new key-only login works using the exact target account, hostname, and key (`BatchMode=yes`, password authentication disabled), and verify needed sudo access.
5. Only after step 4 succeeds may password login, root login, the old key, or another existing access route be disabled.
6. Keep the original management session open until the final effective SSH configuration and a fresh key-only login have both been verified.

If the new key-based login cannot be proven first, stop immediately: do **not** reload an access-restricting configuration. Provider web/serial console access must be confirmed before any SSH access-policy change, but it is a recovery path—not a substitute for the validation steps above.

## Obsidian

Bills, rough notes, and personal clutter stay in the Obsidian vault. This repo holds **actionable facts and runbooks** only.

## Layout

```
AGENTS.md / agent.md   # this entry (agent.md → AGENTS.md)
inventory/hosts.md     # machine index
hosts/                 # per-host facts
runbooks/              # step-by-step ops
docs/                  # upstream doc indexes / design notes
```