vps/hosts/us4.wsvc.info.md

us4.wsvc.info

Item	Value
Role	WireGuard VPN server (LinuxServer Docker image)
SSH	ssh -4 windy@us4.wsvc.info (prefer IPv4 from WSL)
IPv4	185.201.226.122
Compose project	/opt/wireguard
Compose file	/opt/wireguard/compose.yml
Container	wireguard
Image policy	Immutable digest, updated only in an approved maintenance window
Public endpoint	us4.wsvc.info:51820/udp; DNS publishes only A 185.201.226.122 (no native AAAA)
Tunnel subnet	10.13.13.0/24
Routing policy	IPv4-only full tunnel (ALLOWEDIPS=0.0.0.0/0); IPv6 traffic is not guaranteed to use the VPN

Upstream image documentation: LinuxServer.io WireGuard.

Deployment configuration

The repository-owned, non-secret Compose declaration is rendered from ansible/templates/wireguard-compose.yml.j2. The live declaration was verified on 2026-08-12 with these core settings:

Setting	Live value / intent
Image	lscr.io/linuxserver/wireguard@sha256:ac43e1226878d2611315172d6ea357a95cb326ee73124b91108118efc8666889
Image version	1.0.20260223-r0-ls119 (build 2026-07-30)
Required capability	NET_ADMIN only; host kernel already supplies WireGuard/iptables, so SYS_MODULE and /lib/modules are not granted
Filesystem	Read-only container root; executable tmpfs at /run; writable bind mount /opt/wireguard/config:/config
Restart	unless-stopped
Server mode	Named peers ha, phone, mbp; runtime and configured peer counts both 3
Client DNS	1.1.1.1
Tunnel routing	IPv4 full tunnel, 0.0.0.0/0; no client IPv6 tunnel
Runtime interface	wg0, server address 10.13.13.1/32, listen port 51820
Forwarding/NAT	IPv4 forwarding enabled in the container namespace; wg0 forwarding allowed and egress masqueraded on eth+; IPv6 forwarding disabled

Docker binds UDP 51820 on both host socket families, but the public hostname has no AAAA record. Clients using us4.wsvc.info therefore reach the server over IPv4.

Other host services and firewall (2026-08-12)

This host also carries the windy.me secondary MX and several web applications; do not build its firewall allowlist from the WireGuard role alone.

Port	Owner / purpose	Effective public state
TCP 22	SSH management	Open
TCP 25	Postfix, mx.windy.me (MX priority 30)	Open; retain until the secondary-MX role is explicitly retired
TCP 80, 443	Traefik for update.wsvc.info, us4-gate.wsvc.info, and trlm.wsvc.info	Open
TCP 3000	Semaphore UI direct Docker publish	Open; redundant with the Traefik route and should be removed or bound to loopback
TCP 8080	Traefik direct Docker publish	Open; redundant with the authenticated dashboard route and should be removed or bound to loopback
UDP 51820	WireGuard	Required public endpoint
TCP 6443	Traefik proxyin entrypoint → sing-box HTTP proxy (/opt/sing-box)	Open (Docker publish); added 2026-10-01, VPS-99
TCP 9443	Host nghttpx-to-Squid proxy — legacy, retirement pending (see HTTPS forward proxy below)	Listening but blocked by the current firewall
UDP 123	ntpsec	Listening but blocked by the current firewall

PostgreSQL (5433/5434/5435), MariaDB (3306), and the host Squid TCP listener (3128) are loopback-only. Squid also owns wildcard UDP sockets, which are not allowed by the current public zone.

UFW is not installed. Firewalld 2.3.1 is active with nftables. On 2026-08-12, the reviewed ansible/playbooks/us4-firewalld.yml reconciliation removed the stale imap, imaps, smtp-submission, and smtps services plus TCP 24, 6443, and 8443 without reloading or restarting firewalld. Runtime and permanent public-zone state now match exactly: services dhcpv6-client, http, https, smtp, and ssh, with no explicit ports.

Docker-published ports are accepted through Docker's DNAT/FORWARD chains, so the public-zone cleanup does not close 3000 or 8080. Their Compose bindings remain a separate, staged follow-up after the required observation window. Firewalld logged Docker chain/policy conflicts during the 2026-08-10 boots; treat any firewall reload or service restart as a maintenance-window operation and reverify Docker routing. Tracking: Linear W1N-60.

mx.windy.me also publishes AAAA 2602:f9f3:0:2::878, while the host currently has no global IPv6 address or IPv6 default route. Treat that as a separate secondary-MX reachability issue.

HTTPS forward proxy (verified 2026-09-26)

The canonical HTTPS forward proxy on this host is:

Item	Value
Endpoint	https://us4.wsvc.info:443 (Traefik TLS, ACME-managed)
Chain	Traefik :443 HostSNI(us4.wsvc.info) → container nghttpx-proxy:3000 (no-TLS) → squid-backend:3128
Config source	/opt/nghttpx/compose.yml, nghttpx.conf, squid.conf (server files are authoritative; not yet repo-owned)
Credential	/opt/nghttpx/passwords, owner root:proxy (0:13) mode 0640, bcrypt ($2y$, cost 12). Squid reads it through a read-only bind mount; the helper runs as 13:13, so any permission change must keep group proxy read access.
Username	windy
Password	Not stored in this repository or Plane. The plaintext lives only in the operator's password manager; the server holds the bcrypt hash. Rotate with htpasswd -B -C 12, validate against basic_ncsa_auth before replacing the file, then docker stop squid-backend → write in place → docker start squid-backend (a rename would break the bind mount).
Auth behaviour	No/incorrect credentials → 407; correct credentials → tunnel. Verified egress IP 185.201.226.122
Destination policy	HTTPS-tunnel only: Safe_ports/SSL_ports = 443; CONNECT to other ports and destinations in 127/8, 10/8, 172.16/12, 192.168/16, 169.254/16 (metadata), 100.64/10, 224/4, ::1, fc00::/7, fe80::/10, ff00::/8 return 403 ERR_ACCESS_DENIED
Limitation	Plain-HTTP (absolute-URI) proxying is not supported; non-CONNECT requests fail with 400. Clients must use HTTPS URLs.
Credential scope	No longer shared with us2.wsvc.info (the two hosts had an identical /opt/nghttpx/passwords hash before 2026-09-26; us4 now has its own)

Client settings (username windy, password from the password manager):

Proxy type: HTTP/HTTPS proxy (CONNECT)
Host:       us4.wsvc.info
Port:       443
Username:   windy
Password:   <from password manager>

curl equivalent (replace $PROXY_PASSWORD from the password manager; never inline it in shell history on a shared host):

curl -x https://us4.wsvc.info:443 -U "windy:$PROXY_PASSWORD" https://api.ipify.org
# expected: 185.201.226.122

The 9443 path (nghttpx.service → host squid.service on 127.0.0.1:3128) is a legacy stack: its certbot-issued certificate expired 2026-01-11, the host Squid has no authentication at all, and no historical client traffic was found in the nghttpx access log. It is blocked by the firewall and reachable only from the host itself. Retirement (stop + disable both units, keep config for rollback) is pending.

Rollback material from the 2026-09-26 change: /opt/nghttpx/backups/20260926T091907+0800 (mode 0700, includes the pre-change password file). Keep it protected and delete it once the change is accepted.

Independent sing-box proxy on :6443 (added 2026-10-01, VPS-99)

A second, independent HTTPS forward proxy on 6443. It does not touch the :443 chain above, and the :443/:9443 stacks keep running.

Item	Value
Endpoint	https://us4.wsvc.info:6443 (Traefik TLS, ACME-managed)
Chain	Traefik TCP router singbox-proxy (entrypoint proxyin, HostSNI(us4.wsvc.info), tls=true, resolver letsencrypt) → sing-box:8080
Config source	/opt/sing-box/compose.yml + config.json (server authoritative); repo copy compose/sing-box/compose.yml
Image	ghcr.io/sagernet/sing-box:v1.14.2@sha256:afbffd294c5eb3519cc7b4587299ef189bb0a2ca2f667cb6301fdb6b9bce9558
Credential	sing-box users in config.json (mode 0600); operator copy in /opt/sing-box/.proxy-password. Not in this repo or Plane.
Capability	Supports both CONNECT tunnels and absolute-URI plain-HTTP requests, unlike the :443 path
Host port	The container publishes no port; 6443 is bound by Traefik

Traefik changes live in /opt/traefik/compose.yml: --entrypoints.proxyin.address=:6443 plus the 6443:6443 publish. Changing either requires recreating the Traefik container (docker compose up -d), not docker compose restart.

6443 was free before the change (nothing listening; nginx inactive, certbot/apache2 failed). Docker-published ports land in the firewalld docker zone, not public, so a public-zone --add-port=6443 is neither needed nor effective; source restriction belongs in the DOCKER-USER chain or a Traefik TCP middleware. 6443 is a public TCP port beyond the target exposure recorded in Plane VPS-47 and still needs an explicit amendment there.

Client notes — full detail in runbooks/proxy-client-setup.md:

• The client's proxy type must be HTTPS (TLS to the proxy). A plaintext HTTP proxy entry reaches Traefik in cleartext and gets 404, not a proxied request.
• Browser extensions (Zero Omega / SwitchyOmega) must have the username and password saved in the proxy profile, and the routing conditions must include the site's CDN hostnames (for x.com: twimg.com, api.x.com, t.co), not only the landing domain — otherwise the page loads incompletely while the server-side log shows no inbound connection to line for those hosts.
• python3 urllib and wget cannot speak TLS to a proxy and fail with Tunnel connection failed: 404 Not Found.
• Client-side symptoms are diagnosed from the server with docker logs -f sing-box | grep 'inbound connection to'; no Proxy-Authorization header means the client sent no credentials.

Safety

• Private keys, preshared keys, peer configuration files, and QR codes remain only in /opt/wireguard/config; do not copy them to this repository or Linear.
• Local rollback archives are stored in /opt/wireguard/backups (directory mode 0700, archives mode 0600). They contain private keys, are not an off-host disaster-recovery backup, and must never leave the server.
• Live private keys, preshared keys, generated peer configs, QR images, and wg0.conf are mode 0600. Template-only peer.conf and server.conf files are mode 0644 and do not contain generated key material.
• /opt/wireguard/config is mode 0755, but its sensitive files are 0600. The current files are owned by the image's numeric UID/GID rather than the declared PUID=1000 / PGID=1000; the root-run WireGuard processes can use them, but reconcile ownership only after a protected backup and maintenance review.
• LOG_CONFS is currently unset and the inspected container log contained no QR-code/config banners. Do not enable config logging; generated QR images are credentials.
• Do not delete, move, or regenerate /opt/wireguard/config during maintenance.
• Before a container recreation, validate docker compose config and retain a verified SSH session for rollback. Do not run docker compose down -v.

Routine operations

Run read-only checks through Ansible:

cd ansible
ansible-playbook playbooks/health-report.yml --limit wireguard

Preview the narrow, fail-closed public-zone reconciliation:

ansible-galaxy collection install -r requirements.yml
ansible-playbook playbooks/us4-firewalld.yml --limit us4 --check --diff

Apply it only after testing the provider console and keeping an independent SSH rollback session open. The playbook creates a protected server-local backup and a 15-minute automatic rollback before changing rules; it cancels that rollback only after SSH, HTTPS, SMTP, Docker, Fail2ban, and WireGuard checks pass:

ansible-playbook playbooks/us4-firewalld.yml --limit us4 \
  -e '{"us4_firewalld_confirm": true, "us4_console_confirm": true}'

The image update and recreate procedure is deliberately separate and requires an immutable image digest in the server-side Compose file plus an explicit maintenance-window confirmation:

cd ansible
ansible-playbook playbooks/wireguard-maintenance.yml --limit wireguard \
  -e '{"wireguard_maintenance_confirm": true}'

Before that recreate, apply the reviewed Compose hardening from the repository:

ansible-playbook playbooks/wireguard-harden.yml --limit wireguard \
  -e '{"wireguard_harden_confirm": true}'

Forward-proxy health for both endpoints (:443 and :6443) is read-only: follow runbooks/proxy-health.md. No proxy credential is stored in this repository.

Verification

• Container wireguard is running and interface wg0 exists.
• UDP 51820 is listening on the host.
• Validate a known client can handshake and sends IPv4 traffic through the VPN.
• Do not treat inactive mobile peers as a failure solely because their latest handshake is old.

Live audit snapshot (2026-08-12)

The WireGuard service itself is healthy and its installation is broadly reasonable:

• The sanitized Ansible health report returned status=ok; Compose is valid, the container is running with zero restarts, wg0 exists, and UDP 51820 is listening.
• One of three peers had a current handshake during the audit. Two peers had not handshaken since the current container/interface start; confirm those clients only if they are expected to be active.
• The image is immutable-digest pinned, key-bearing files are protected, the container root is read-only, and the container has NET_ADMIN without the broader SYS_MODULE capability.
• Debian 13.6, kernel 6.12.101+deb13-amd64, Docker Engine 29.7.2, and Docker Compose v5.4.0 were observed. No Debian package updates or reboot requirement were pending.

Open host-level follow-up (do not conflate these with a WireGuard outage):

1. Disk capacity: / was 90% used with about 3.4 GiB free. Docker reported about 2.48 GB of reclaimable images and the system journal used about 1.9 GB, but do not prune or vacuum without reviewing retention and rollback needs first. — Resolved 2026-09-26 (91% → 43%); see Root disk recovery below.
2. Docker exposure: the firewalld public-zone cleanup is complete, but Docker still publishes 3000 and 8080 outside the ordinary host INPUT path. Remove those redundant Compose bindings in separate maintenance units after the observation window, and confirm provider firewall rules first.
3. Image maintenance: the upstream latest amd64 image had advanced to 1.0.20260223-r0-ls120 (build 2026-08-06). Review and pin its immutable digest in a maintenance window rather than updating unattended.
4. Host hygiene: apache2.service, certbot.service, and postgresql@9.6-main.service were in a failed state while unrelated Docker workloads remained active. Establish ownership and remove or repair stale units separately.
5. Resource/log limits: the WireGuard container has no memory, CPU, or PID limit and uses Docker's json-file log driver without a per-container rotation setting. Current log size was small, but limits/rotation should be considered during a reviewed Compose update.

Root disk recovery (2026-09-26, Plane VPS-100)

/ went from 91% used (3.0G free) to 43% (18G free) — about 14G reclaimed, following runbooks/host-disk-cleanup.md.

Removed: ~/.nvm 2.6G, ~/.go 1.9G, ~/.rustup 1.4G, ~/.cargo 505M, ~/.npm 78M, stale rootless-podman storage under ~/.local/share/containers 1.7G, ~/sigstore 953M, ~/softz 910M, ~/.cache 295M, apt archives 613M, the container JSON logs (594M → 320K), and six unreferenced Docker images (postgres:17, postgres:18, traefik:v3.4, zadam/trilium:0.60.4, linuxserver/wireguard:latest, traefik/whoami:latest — all re-pullable).

Durable changes, not just deletions:

• journalctl --vacuum-size=100M plus drop-in /etc/systemd/journald.conf.d/00-disk-us4.conf (SystemMaxUse=200M); journald was restarted so the cap is effective (journal back to 83M). Without this the journal regrows toward its default 10%-of-filesystem limit.
• Log rotation max-size: 10m / max-file: 3 added to nghttpx-proxy, squid-backend (/opt/nghttpx/compose.yml) and traefik (/opt/traefik/compose.yml); both stacks were recreated and verified. This closes the unbounded json.log growth for these containers.
• ~/.zshenv, ~/.bashrc, ~/.profile and ~/.zshrc had their ~/.cargo/env and nvm references commented out after those trees were deleted; backups are ~/.<file>.bak-20260926-disk. Also note ~/.go/bin/cosign was removed and is re-installable.

Deliberately kept: ~/.local/share/trilium-data (88M) is a 2022 Trilium database, not a cache. Still removable if unneeded: /opt/zeroclaw/target 1.4G, /opt/trilium-linux-x64-server 592M (2023 host install, unit disabled), /opt/go 167M (2017).