vps/AGENTS.md

VPS Agent Ops Hub

This repo is the agent ops handbook + fact source for maintaining personal VPS hosts. Prefer verifying live state over assuming docs are complete.

Also readable as agent.md (symlink → this file).

How to work

1. Read inventory/hosts.md for the machine list.
2. Open the matching hosts/<name>.md for SSH, roles, paths, and quirks.
3. For common tasks, follow a runbook under runbooks/. Pick the most specific applicable one from runbooks/README.md; the spec is RUNBOOKS.md and new runbooks start from runbooks/_template.md.
4. Prefer read-only checks first; change only after confirming current state.
5. For routine checks and approved service reconciliation, run the matching Ansible playbook from ansible/; see routine Ansible operations.
6. Default SSH access (ssh -4 windy@<host>) is for focused diagnostics, imperative upstream procedures, and incident work. Prefer IPv4 from this WSL client (AAAA often exists but IPv6 route does not).
7. Record each material VPS operation, incident, configuration change, or verification outcome in the corresponding Linear vps project. Include scope, action, verification, and remaining follow-up; never put passwords, tokens, private keys, recovery keys, or private room IDs in Linear.

Runbook execution rules

Before operational work: inspect runbooks/, select the most specific applicable runbook, follow its steps in order, do not skip verification steps, and respect its STOP and approval conditions. If no runbook applies, diagnose only — do not mutate production state. When live state conflicts with a runbook's assumptions, STOP and report; never invent missing parameters or bypass failed checks. The spec is RUNBOOKS.md.

Active hosts

The canonical machine list (roles, SSH endpoints, Ansible coverage, status) is inventory/hosts.md — the single human-readable source of truth. Per-host facts live in hosts/. The Ansible execution inventory is ansible/inventory/hosts.yml. Do not maintain a second copy of the machine table here.

Public services

URL / endpoint	Backend host	Notes
https://mx2.windy.me	mx2	mailcow UI / SOGo
SMTP mx2.windy.me:587 (STARTTLS) or :465	mx2	client submission; full email + mailbox password — runbook
IMAP mx2.windy.me:993	mx2	same mailbox credentials
https://auth.wsvc.info	us2 (/opt/vaultwarden)	Vaultwarden (Postgres, operational) — client Server URL
repo.windy.me:2222	us2 (/opt/soft-serve)	Soft Serve (stub details)
DNS ns1.wsvc.info:53	hk2 (/opt/pdns, Auth 5.0.6)	PowerDNS auth — zones windy.me, wsvc.info, chans.xyz
https://pdns.wsvc.info	hk2 (poweradmin)	Poweradmin UI
https://pgweb.wsvc.info	hk2 (pgweb)	PowerDNS Postgres browser
https://chat.chans.xyz	synapse (ess, ESS chart 26.7.2)	Element Web (Matrix client)
https://synapse.chans.xyz	synapse	Synapse Client-Server + Federation API
https://account.chans.xyz	synapse	Matrix Authentication Service (local passwords)
https://admin.chans.xyz	synapse	Element Admin console (MAS admin auth)

Upstream docs

Vaultwarden: Before changing config, read docs/vaultwarden-upstream.md. Live DB is Postgres (DATABASE_URL); vw-data/config.json from /admin overrides .env (including SMTP password — keep it synced with .env / .smtp-credentials). After recreating the VW container, restart Traefik if https://auth.wsvc.info returns 404. SMTP health needs AUTH, not TCP-only — see vaultwarden-health.

PowerDNS: Before changing /opt/pdns on hk2, read docs/pdns-upstream.md. Auth pin 5.0.6; primary + also-notify/only-notify= + DNSSEC OK. Deferred: rotate weak API/DB credentials and TSIG (TSIG-ALLOW-AXFR bypasses IP ACL). Health: pdns-health.

Matrix (ESS on synapse): Matrix homeserver running on synapse.chans.xyz via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone chans.xyz managed by hk2 PowerDNS. Before changing config, read docs/matrix-upstream.md and hosts/synapse.chans.xyz.md. K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: matrix-health.

RustDesk: Self-hosted RustDesk server on hk2.chans.xyz (/opt/rustdesk, containers hbbs/hbbr, image pinned 1.1.14). The hbbs -r relay hostname must resolve to the host's public IP 154.36.174.161 — use hk2.chans.xyz (never hk2.wsvc.info, which has no DNS record). Health: rustdesk-health.

Runbooks & scripts

Task	Path
mailcow health	runbooks/mailcow-health.md
mailcow update	runbooks/mailcow-update.md
mailcow SMTP/IMAP client	runbooks/mailcow-smtp-client.md
Vaultwarden health	runbooks/vaultwarden-health.md
Vaultwarden SQLite→PG migrate	runbooks/vaultwarden-sqlite-to-postgres.md
Vaultwarden upstream refs	docs/vaultwarden-upstream.md
PowerDNS health (hk2)	runbooks/pdns-health.md
PowerDNS upstream refs	docs/pdns-upstream.md
Matrix health	runbooks/matrix-health.md
RustDesk health (hk2)	runbooks/rustdesk-health.md
AdGuard Home health	runbooks/adguard-home-health.md
Home Assistant maintenance	runbooks/home-assistant-maintenance.md + scripts/ha-maintenance.sh
Matrix upstream refs	docs/matrix-upstream.md
Hermes Agent Matrix channel	docs/hermes-matrix.md
UniFi local-service proxy bypass	docs/unifi-openclash-localhost.md
UniFi SSO login setting (Ansible)	cd ansible && ansible-playbook playbooks/unifi-sso.yml --limit unifi
Routine Ansible operations	runbooks/ansible-operations.md
Issue → mergeable change	runbooks/issue-to-merge.md
Fix failing health/playbook run	runbooks/fix-ci.md
Release a reviewed change	runbooks/release.md
Roll back a change	runbooks/rollback.md
Controlled network change	runbooks/network-change.md
Network outage recovery	runbooks/network-recovery.md

Full index: runbooks/README.md. Spec: RUNBOOKS.md.

Routine mailcow health: cd ansible && ansible-playbook playbooks/health-report.yml --limit mailcow. The local stub resolver is flaky; DNS probes use 1.1.1.1 / 8.8.8.8.

Agent skills

Issue tracker

Issues are tracked in Linear and created/updated via the Linear MCP (vps project). See docs/agents/issue-tracker.md.

Triage labels

Default triage labels: needs-triage, needs-info, ready-for-agent, ready-for-human, wontfix. See docs/agents/triage-labels.md.

Domain docs

Single-context layout: CONTEXT.md + docs/adr/ at the repo root. See docs/agents/domain.md.

Safety

• Never commit secrets: passwords, API keys, private keys, .env, mailcow.conf DB passwords, Vaultwarden ADMIN_TOKEN / .smtp-credentials.
• Secrets live only on the servers (e.g. /opt/vaultwarden/.env). Do not copy them from Obsidian into this repo.
• Ask before destructive actions: docker compose down -v, DB drops, DNS changes, force-push, disk wipes.
• Ansible reconciliation is intentionally allowlisted and requires an explicit confirmation variable; it is not a general remote-command facility.
• Do not modify local ~/.ssh/config or git config unless explicitly asked.

SSH access safety — mandatory change procedure

Treat any SSH authentication or authorization change as a lockout-risk change. This includes sshd_config / drop-in edits, PasswordAuthentication, PermitRootLogin, AuthorizedKeysFile, Match blocks, SSH keys, accounts, groups, chroots, firewalls, and SSH ports.

Before disabling an existing access method or reloading sshd, all of the following are mandatory:

1. Keep the current verified SSH management session open as the rollback path; do not use it to run the validation connection.
2. Install the intended management public key from a verified local source. Never assume a public-key file exists on the target host or copy a remote path without checking it first.
3. Validate the candidate configuration with sshd -t before reloading the service.
4. From a separate local terminal, prove the new key-only login works using the exact target account, hostname, and key (BatchMode=yes, password authentication disabled), and verify needed sudo access.
5. Only after step 4 succeeds may password login, root login, the old key, or another existing access route be disabled.
6. Keep the original management session open until the final effective SSH configuration and a fresh key-only login have both been verified.

If the new key-based login cannot be proven first, stop immediately: do not reload an access-restricting configuration. Provider web/serial console access must be confirmed before any SSH access-policy change, but it is a recovery path—not a substitute for the validation steps above.

Obsidian

Bills, rough notes, and personal clutter stay in the Obsidian vault. This repo holds actionable facts and runbooks only.

Layout

AGENTS.md / agent.md   # this entry (agent.md → AGENTS.md)
RUNBOOKS.md            # runbook spec (six-field model, naming, review rules)
inventory/hosts.md     # machine index (human-readable source of truth)
ansible/               # playbooks, roles, sanitized control-plane inventory
compose/               # repo-owned non-secret Compose sources (+ .env.example)
hosts/                 # per-host facts
runbooks/              # step-by-step ops (README.md = index, _template.md = template)
docs/                  # upstream doc indexes / design notes
scripts/validate-repo.sh  # repo-wide validation (run before merging)