vps/AGENTS.md

VPS Agent Ops Hub

This repo is the agent ops handbook + fact source for maintaining personal VPS hosts. Prefer verifying live state over assuming docs are complete.

Also readable as agent.md (symlink → this file).

How to work

1. Read inventory/hosts.md for the machine list.
2. Open the matching hosts/<name>.md for SSH, roles, paths, and quirks.
3. For common tasks, follow a runbook under runbooks/.
4. Prefer read-only checks first; change only after confirming current state.
5. Default access: ssh -4 windy@<host> (pubkey). Prefer IPv4 from this WSL client (AAAA often exists but IPv6 route does not).

Active hosts (quick map)

Host	Role	SSH	Facts
mx2.windy.me	mailcow (/opt/mail, project cow)	ssh -4 windy@mx2.windy.me	hosts/mx2.windy.me.md
us2.wsvc.info	Vaultwarden + Traefik (+ Soft Serve, …)	ssh -4 windy@us2.wsvc.info	hosts/us2.wsvc.info.md
hk2.chans.xyz	PowerDNS auth ns1 (/opt/pdns)	ssh -4 windy@hk2.chans.xyz	hosts/hk2.chans.xyz.md

Public services

URL / endpoint	Backend host	Notes
https://mx2.windy.me	mx2	mailcow UI / SOGo
SMTP mx2.windy.me:587 (STARTTLS) or :465	mx2	client submission; full email + mailbox password — runbook
IMAP mx2.windy.me:993	mx2	same mailbox credentials
https://auth.wsvc.info	us2 (/opt/vaultwarden)	Vaultwarden (Postgres, operational) — client Server URL
repo.windy.me:2222	us2 (/opt/soft-serve)	Soft Serve (stub details)
DNS ns1.wsvc.info:53	hk2 (/opt/pdns, Auth 5.0.6)	PowerDNS auth — zones windy.me, wsvc.info, chans.xyz
https://pdns.wsvc.info	hk2 (poweradmin)	Poweradmin UI
https://pgweb.wsvc.info	hk2 (pgweb)	PowerDNS Postgres browser
https://chat.chans.xyz	synapse (ess, ESS chart 26.7.2)	Element Web (Matrix client)
https://synapse.chans.xyz	synapse	Synapse Client-Server + Federation API
https://account.chans.xyz	synapse	Matrix Authentication Service (local passwords)
https://admin.chans.xyz	synapse	Element Admin console (MAS admin auth)

Upstream docs

Vaultwarden: Before changing config, read docs/vaultwarden-upstream.md. Live DB is Postgres (DATABASE_URL); vw-data/config.json from /admin overrides .env (including SMTP password — keep it synced with .env / .smtp-credentials). After recreating the VW container, restart Traefik if https://auth.wsvc.info returns 404. SMTP health needs AUTH, not TCP-only — see vaultwarden-health.

PowerDNS: Before changing /opt/pdns on hk2, read docs/pdns-upstream.md. Auth pin 5.0.6; primary + also-notify/only-notify= + DNSSEC OK. Deferred: rotate weak API/DB credentials and TSIG (TSIG-ALLOW-AXFR bypasses IP ACL). Health: pdns-health.

Matrix (ESS on synapse): Matrix homeserver running on synapse.chans.xyz via the official ESS (Element Server Suite) Helm chart with Synapse + MAS + Element Web + Admin. DNS zone chans.xyz managed by hk2 PowerDNS. Before changing config, read docs/matrix-upstream.md and hosts/synapse.chans.xyz.md. K3s cluster on this node has hostPort 80/443 for Traefik (no ServiceLB). Health: matrix-health.

Runbooks & scripts

Task	Path
mailcow health	runbooks/mailcow-health.md · ./scripts/check-mx2.sh
mailcow update	runbooks/mailcow-update.md
mailcow SMTP/IMAP client	runbooks/mailcow-smtp-client.md
Vaultwarden health	runbooks/vaultwarden-health.md
Vaultwarden SQLite→PG migrate	runbooks/vaultwarden-sqlite-to-postgres.md
Vaultwarden upstream refs	docs/vaultwarden-upstream.md
PowerDNS health (hk2)	runbooks/pdns-health.md
PowerDNS upstream refs	docs/pdns-upstream.md
Matrix health	runbooks/matrix-health.md
Matrix upstream refs	docs/matrix-upstream.md

./scripts/check-mx2.sh

check-mx2.sh defaults to IPv4; set MX2_ALLOW_IPV6=1 to allow IPv6. DNS checks use 1.1.1.1 / 8.8.8.8 (local stub is flaky).

Safety

• Never commit secrets: passwords, API keys, private keys, .env, mailcow.conf DB passwords, Vaultwarden ADMIN_TOKEN / .smtp-credentials.
• Secrets live only on the servers (e.g. /opt/vaultwarden/.env). Do not copy them from Obsidian into this repo.
• Ask before destructive actions: docker compose down -v, DB drops, DNS changes, force-push, disk wipes.
• Do not modify local ~/.ssh/config or git config unless explicitly asked.

SSH access safety — mandatory change procedure

Treat any SSH authentication or authorization change as a lockout-risk change. This includes sshd_config / drop-in edits, PasswordAuthentication, PermitRootLogin, AuthorizedKeysFile, Match blocks, SSH keys, accounts, groups, chroots, firewalls, and SSH ports.

Before disabling an existing access method or reloading sshd, all of the following are mandatory:

1. Keep the current verified SSH management session open as the rollback path; do not use it to run the validation connection.
2. Install the intended management public key from a verified local source. Never assume a public-key file exists on the target host or copy a remote path without checking it first.
3. Validate the candidate configuration with sshd -t before reloading the service.
4. From a separate local terminal, prove the new key-only login works using the exact target account, hostname, and key (BatchMode=yes, password authentication disabled), and verify needed sudo access.
5. Only after step 4 succeeds may password login, root login, the old key, or another existing access route be disabled.
6. Keep the original management session open until the final effective SSH configuration and a fresh key-only login have both been verified.

If the new key-based login cannot be proven first, stop immediately: do not reload an access-restricting configuration. Provider web/serial console access must be confirmed before any SSH access-policy change, but it is a recovery path—not a substitute for the validation steps above.

Obsidian

Bills, rough notes, and personal clutter stay in the Obsidian vault. This repo holds actionable facts and runbooks only.

Layout

AGENTS.md / agent.md   # this entry (agent.md → AGENTS.md)
inventory/hosts.md     # machine index
hosts/                 # per-host facts
runbooks/              # step-by-step ops
docs/                  # upstream doc indexes / design notes
scripts/               # read-only helper scripts