vps/AGENTS.md

VPS Agent Ops Hub

This repo is the agent ops handbook + fact source for maintaining personal VPS hosts. Prefer verifying live state over assuming docs are complete.

Also readable as agent.md (symlink → this file).

How to work

1. Read inventory/hosts.md for the machine list.

2. Open the matching hosts/<name>.md for SSH, roles, paths, and quirks.

3. For common tasks, follow a runbook under runbooks/. Pick the most specific applicable one from runbooks/README.md; the spec is RUNBOOKS.md and new runbooks start from runbooks/_template.md.

4. Prefer read-only checks first; change only after confirming current state.

5. For routine checks and approved service reconciliation, run the matching Ansible playbook from ansible/; see routine Ansible operations.

6. Default SSH access (ssh -4 windy@<host>) is for focused diagnostics, imperative upstream procedures, and incident work. Prefer IPv4 from this WSL client (AAAA often exists but IPv6 route does not).

   Agent sandbox SSH quirk (verified 2026-08-20): the agent shell runs in a sandboxed user namespace — system files such as /etc/ssh/ssh_config.d/20-systemd-ssh-proxy.conf appear owned by nobody, so plain ssh aborts with Bad owner or permissions on .... Always use ssh -F /dev/null from the agent shell and pass options explicitly (~/.ssh/config is skipped; e.g. ssh -F /dev/null -p 2222 -i ~/.ssh/id_ed25519 windy@repo.windy.me). sudo never works in the sandbox (NoNewPrivs, no capabilities, / read-only). The host itself is healthy — to inspect or act on the real host from the sandbox use /mnt/c/WINDOWS/system32/wsl.exe -u root -- <cmd> (real root: keep read-only unless a change is approved).

7. Record each material VPS operation, incident, configuration change, or verification outcome in the corresponding Linear vps project. Include scope, action, verification, and remaining follow-up; never put passwords, tokens, private keys, recovery keys, or private room IDs in Linear.

Runbook execution rules

Before operational work: inspect runbooks/, select the most specific applicable runbook, follow its steps in order, do not skip verification steps, and respect its STOP and approval conditions. If no runbook applies, diagnose only — do not mutate production state. When live state conflicts with a runbook's assumptions, STOP and report; never invent missing parameters or bypass failed checks. The spec is RUNBOOKS.md.

Active hosts

The canonical machine list (roles, SSH endpoints, Ansible coverage, status) is inventory/hosts.md — the single human-readable source of truth. Per-host facts live in hosts/. The Ansible execution inventory is ansible/inventory/hosts.yml. Do not maintain a second copy of the machine table here.

Public services

URL / endpoint	Backend host	Notes
https://mx2.windy.me	mx2	mailcow UI / SOGo
SMTP mx2.windy.me:587 (STARTTLS) or :465	mx2	client submission; full email + mailbox password — runbook
IMAP mx2.windy.me:993	mx2	same mailbox credentials
https://auth.wsvc.info	us2 (/opt/vaultwarden)	Vaultwarden (Postgres, operational) — client Server URL
repo.windy.me:2222	us2 (/opt/soft-serve)	Soft Serve (stub details)
DNS ns1.wsvc.info:53	hk2 (/opt/pdns, Auth 5.0.6)	PowerDNS auth — zones windy.me, wsvc.info, chans.xyz
https://pdns.wsvc.info	hk2 (poweradmin)	Poweradmin UI
https://pgweb.wsvc.info	hk2 (pgweb)	PowerDNS Postgres browser
https://chat.chans.xyz	synapse (ess, ESS chart 26.7.2)	Element Web (Matrix client)
https://synapse.chans.xyz	synapse	Synapse Client-Server + Federation API
https://account.chans.xyz	synapse	Matrix Authentication Service (local passwords)
https://admin.chans.xyz	synapse	Element Admin console (MAS admin auth)

Upstream docs

Vaultwarden: Before changing config, read docs/vaultwarden-upstream.md. Live DB is Postgres (DATABASE_URL); vw-data/config.json from /admin overrides .env (including SMTP password — keep it synced with .env / .smtp-credentials). After recreating the VW container, restart Traefik if https://auth.wsvc.info returns 404. SMTP health needs AUTH, not TCP-only — see vaultwarden-health.

PowerDNS: Before changing /opt/pdns on hk2, read docs/pdns-upstream.md. Auth pin 5.0.6; primary + also-notify/only-notify= + DNSSEC OK. Deferred: rotate weak API/DB credentials and TSIG (TSIG-ALLOW-AXFR bypasses IP ACL). Health: pdns-health.

Matrix (ESS on synapse): Matrix homeserver running on synapse.chans.xyz via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone chans.xyz managed by hk2 PowerDNS. Before changing config, read docs/matrix-upstream.md and hosts/synapse.chans.xyz.md. K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: matrix-health.

RustDesk: Self-hosted RustDesk server on hk2.chans.xyz (/opt/rustdesk, containers hbbs/hbbr, image pinned 1.1.14). The hbbs -r relay hostname must resolve to the host's public IP 154.36.174.161 — use hk2.chans.xyz (never hk2.wsvc.info, which has no DNS record). Health: rustdesk-health.

Runbooks & scripts

Task	Path
mailcow health	runbooks/mailcow-health.md
mailcow update	runbooks/mailcow-update.md
mailcow SMTP/IMAP client	runbooks/mailcow-smtp-client.md
Vaultwarden health	runbooks/vaultwarden-health.md
Vaultwarden SQLite→PG migrate	runbooks/vaultwarden-sqlite-to-postgres.md
Vaultwarden upstream refs	docs/vaultwarden-upstream.md
PowerDNS health (hk2)	runbooks/pdns-health.md
PowerDNS upstream refs	docs/pdns-upstream.md
Matrix health	runbooks/matrix-health.md
RustDesk health (hk2)	runbooks/rustdesk-health.md
AdGuard Home health	runbooks/adguard-home-health.md
Home Assistant maintenance	runbooks/home-assistant-maintenance.md + scripts/ha-maintenance.sh
matrix_e2ee update (hass.windy.lan)	runbooks/matrix-e2ee-update.md
Matrix upstream refs	docs/matrix-upstream.md
Hermes Agent Matrix channel	docs/hermes-matrix.md
UniFi local-service proxy bypass	docs/unifi-openclash-localhost.md
UniFi SSO login setting (Ansible)	cd ansible && ansible-playbook playbooks/unifi-sso.yml --limit unifi
Routine Ansible operations	runbooks/ansible-operations.md
Issue → mergeable change	runbooks/issue-to-merge.md
Fix failing health/playbook run	runbooks/fix-ci.md
Release a reviewed change	runbooks/release.md
Roll back a change	runbooks/rollback.md
Controlled network change	runbooks/network-change.md
Network outage recovery	runbooks/network-recovery.md

Full index: runbooks/README.md. Spec: RUNBOOKS.md.

Routine mailcow health: cd ansible && ansible-playbook playbooks/health-report.yml --limit mailcow. The local stub resolver is flaky; DNS probes use 1.1.1.1 / 8.8.8.8.

Agent skills

Issue tracker

Issues are tracked in Linear and created/updated via the Linear MCP (vps project). See docs/agents/issue-tracker.md.

Triage labels

Default triage labels: needs-triage, needs-info, ready-for-agent, ready-for-human, wontfix. See docs/agents/triage-labels.md.

Domain docs

Single-context layout: CONTEXT.md + docs/adr/ at the repo root. See docs/agents/domain.md.

Safety

• Never commit secrets: passwords, API keys, private keys, .env, mailcow.conf DB passwords, Vaultwarden ADMIN_TOKEN / .smtp-credentials.
• Secrets live only on the servers (e.g. /opt/vaultwarden/.env). Do not copy them from Obsidian into this repo.
• Ask before destructive actions: docker compose down -v, DB drops, DNS changes, force-push, disk wipes.
• Ansible reconciliation is intentionally allowlisted and requires an explicit confirmation variable; it is not a general remote-command facility.
• Do not modify local ~/.ssh/config or git config unless explicitly asked.

SSH access safety — mandatory change procedure

Treat any SSH authentication or authorization change as a lockout-risk change. This includes sshd_config / drop-in edits, PasswordAuthentication, PermitRootLogin, AuthorizedKeysFile, Match blocks, SSH keys, accounts, groups, chroots, firewalls, and SSH ports.

Before disabling an existing access method or reloading sshd, all of the following are mandatory:

1. Keep the current verified SSH management session open as the rollback path; do not use it to run the validation connection.
2. Install the intended management public key from a verified local source. Never assume a public-key file exists on the target host or copy a remote path without checking it first.
3. Validate the candidate configuration with sshd -t before reloading the service.
4. From a separate local terminal, prove the new key-only login works using the exact target account, hostname, and key (BatchMode=yes, password authentication disabled), and verify needed sudo access.
5. Only after step 4 succeeds may password login, root login, the old key, or another existing access route be disabled.
6. Keep the original management session open until the final effective SSH configuration and a fresh key-only login have both been verified.

If the new key-based login cannot be proven first, stop immediately: do not reload an access-restricting configuration. Provider web/serial console access must be confirmed before any SSH access-policy change, but it is a recovery path—not a substitute for the validation steps above.

Obsidian

Bills, rough notes, and personal clutter stay in the Obsidian vault. This repo holds actionable facts and runbooks only.

Layout

AGENTS.md / agent.md   # this entry (agent.md → AGENTS.md)
RUNBOOKS.md            # runbook spec (six-field model, naming, review rules)
inventory/hosts.md     # machine index (human-readable source of truth)
ansible/               # playbooks, roles, sanitized control-plane inventory
compose/               # repo-owned non-secret Compose sources (+ .env.example)
hosts/                 # per-host facts
runbooks/              # step-by-step ops (README.md = index, _template.md = template)
docs/                  # upstream doc indexes / design notes
scripts/validate-repo.sh  # repo-wide validation (run before merging)