vps/AGENTS.md

VPS Agent Ops Hub

This repo is the agent ops handbook + fact source for maintaining personal VPS hosts. Prefer verifying live state over assuming docs are complete.

Also readable as agent.md (symlink → this file).

How to work

1. Read inventory/hosts.md for the machine list.
2. Open the matching hosts/<name>.md for SSH, roles, paths, and quirks.
3. For common tasks, follow a runbook under runbooks/.
4. Prefer read-only checks first; change only after confirming current state.
5. For routine checks and approved service reconciliation, run the matching Ansible playbook from ansible/; see routine Ansible operations.
6. Default SSH access (ssh -4 windy@<host>) is for focused diagnostics, imperative upstream procedures, and incident work. Prefer IPv4 from this WSL client (AAAA often exists but IPv6 route does not).
7. Record each material VPS operation, incident, configuration change, or verification outcome in the corresponding Linear vps project. Include scope, action, verification, and remaining follow-up; never put passwords, tokens, private keys, recovery keys, or private room IDs in Linear.

Active hosts (quick map)

Host	Role	SSH	Facts
mx2.windy.me	mailcow (/opt/mail, project cow)	ssh -4 windy@mx2.windy.me	hosts/mx2.windy.me.md
us2.wsvc.info	Vaultwarden + Traefik (+ Soft Serve, …)	ssh -4 windy@us2.wsvc.info	hosts/us2.wsvc.info.md
hk2.chans.xyz	PowerDNS auth ns1 (/opt/pdns)	ssh -4 windy@hk2.chans.xyz	hosts/hk2.chans.xyz.md
synapse.chans.xyz	Matrix ESS (Synapse + MAS + Element) on K3s	ssh -4 windy@synapse.chans.xyz	hosts/synapse.chans.xyz.md
dns.windy.lan	AdGuard Home LAN DNS + Mihomo explicit proxy	ssh -4 windy@192.168.66.36	hosts/dns.windy.lan.md
gfw.windy.lan	OpenWrt LAN gateway / OpenClash	ssh -4 root@192.168.66.1	hosts/gfw.windy.lan.md
gw	EdgeRouter X primary LAN gateway	ssh -4 zhiqiang@192.168.66.254	hosts/gw.md
ubnt	UniFi Network Controller	ssh -4 windy@192.168.66.46	hosts/ubnt.md

Public services

URL / endpoint	Backend host	Notes
https://mx2.windy.me	mx2	mailcow UI / SOGo
SMTP mx2.windy.me:587 (STARTTLS) or :465	mx2	client submission; full email + mailbox password — runbook
IMAP mx2.windy.me:993	mx2	same mailbox credentials
https://auth.wsvc.info	us2 (/opt/vaultwarden)	Vaultwarden (Postgres, operational) — client Server URL
repo.windy.me:2222	us2 (/opt/soft-serve)	Soft Serve (stub details)
DNS ns1.wsvc.info:53	hk2 (/opt/pdns, Auth 5.0.6)	PowerDNS auth — zones windy.me, wsvc.info, chans.xyz
https://pdns.wsvc.info	hk2 (poweradmin)	Poweradmin UI
https://pgweb.wsvc.info	hk2 (pgweb)	PowerDNS Postgres browser
https://chat.chans.xyz	synapse (ess, ESS chart 26.7.2)	Element Web (Matrix client)
https://synapse.chans.xyz	synapse	Synapse Client-Server + Federation API
https://account.chans.xyz	synapse	Matrix Authentication Service (local passwords)
https://admin.chans.xyz	synapse	Element Admin console (MAS admin auth)

Upstream docs

Vaultwarden: Before changing config, read docs/vaultwarden-upstream.md. Live DB is Postgres (DATABASE_URL); vw-data/config.json from /admin overrides .env (including SMTP password — keep it synced with .env / .smtp-credentials). After recreating the VW container, restart Traefik if https://auth.wsvc.info returns 404. SMTP health needs AUTH, not TCP-only — see vaultwarden-health.

PowerDNS: Before changing /opt/pdns on hk2, read docs/pdns-upstream.md. Auth pin 5.0.6; primary + also-notify/only-notify= + DNSSEC OK. Deferred: rotate weak API/DB credentials and TSIG (TSIG-ALLOW-AXFR bypasses IP ACL). Health: pdns-health.

Matrix (ESS on synapse): Matrix homeserver running on synapse.chans.xyz via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone chans.xyz managed by hk2 PowerDNS. Before changing config, read docs/matrix-upstream.md and hosts/synapse.chans.xyz.md. K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: matrix-health.

Runbooks & scripts

Task	Path
mailcow health	runbooks/mailcow-health.md
mailcow update	runbooks/mailcow-update.md
mailcow SMTP/IMAP client	runbooks/mailcow-smtp-client.md
Vaultwarden health	runbooks/vaultwarden-health.md
Vaultwarden SQLite→PG migrate	runbooks/vaultwarden-sqlite-to-postgres.md
Vaultwarden upstream refs	docs/vaultwarden-upstream.md
PowerDNS health (hk2)	runbooks/pdns-health.md
PowerDNS upstream refs	docs/pdns-upstream.md
Matrix health	runbooks/matrix-health.md
AdGuard Home health	runbooks/adguard-home-health.md
Matrix upstream refs	docs/matrix-upstream.md
Hermes Agent Matrix channel	docs/hermes-matrix.md
UniFi local-service proxy bypass	docs/unifi-openclash-localhost.md
UniFi SSO login setting (Ansible)	cd ansible && ansible-playbook playbooks/unifi-sso.yml --limit unifi
Routine Ansible operations	runbooks/ansible-operations.md

Routine mailcow health: cd ansible && ansible-playbook playbooks/health-report.yml --limit mailcow. The local stub resolver is flaky; DNS probes use 1.1.1.1 / 8.8.8.8.

Agent skills

Issue tracker

Issues are tracked in Linear and created/updated via the Linear MCP (vps project). See docs/agents/issue-tracker.md.

Triage labels

Default triage labels: needs-triage, needs-info, ready-for-agent, ready-for-human, wontfix. See docs/agents/triage-labels.md.

Safety

• Never commit secrets: passwords, API keys, private keys, .env, mailcow.conf DB passwords, Vaultwarden ADMIN_TOKEN / .smtp-credentials.
• Secrets live only on the servers (e.g. /opt/vaultwarden/.env). Do not copy them from Obsidian into this repo.
• Ask before destructive actions: docker compose down -v, DB drops, DNS changes, force-push, disk wipes.
• Ansible reconciliation is intentionally allowlisted and requires an explicit confirmation variable; it is not a general remote-command facility.
• Do not modify local ~/.ssh/config or git config unless explicitly asked.

SSH access safety — mandatory change procedure

Treat any SSH authentication or authorization change as a lockout-risk change. This includes sshd_config / drop-in edits, PasswordAuthentication, PermitRootLogin, AuthorizedKeysFile, Match blocks, SSH keys, accounts, groups, chroots, firewalls, and SSH ports.

Before disabling an existing access method or reloading sshd, all of the following are mandatory:

1. Keep the current verified SSH management session open as the rollback path; do not use it to run the validation connection.
2. Install the intended management public key from a verified local source. Never assume a public-key file exists on the target host or copy a remote path without checking it first.
3. Validate the candidate configuration with sshd -t before reloading the service.
4. From a separate local terminal, prove the new key-only login works using the exact target account, hostname, and key (BatchMode=yes, password authentication disabled), and verify needed sudo access.
5. Only after step 4 succeeds may password login, root login, the old key, or another existing access route be disabled.
6. Keep the original management session open until the final effective SSH configuration and a fresh key-only login have both been verified.

If the new key-based login cannot be proven first, stop immediately: do not reload an access-restricting configuration. Provider web/serial console access must be confirmed before any SSH access-policy change, but it is a recovery path—not a substitute for the validation steps above.

Obsidian

Bills, rough notes, and personal clutter stay in the Obsidian vault. This repo holds actionable facts and runbooks only.

Layout

AGENTS.md / agent.md   # this entry (agent.md → AGENTS.md)
inventory/hosts.md     # machine index
hosts/                 # per-host facts
runbooks/              # step-by-step ops
docs/                  # upstream doc indexes / design notes