vps/hosts/us4.wsvc.info.md

# us4.wsvc.info

| Item | Value |
|------|-------|
| Role | WireGuard VPN server (LinuxServer Docker image) |
| SSH | `ssh -4 windy@us4.wsvc.info` (prefer IPv4 from WSL) |
| IPv4 | `185.201.226.122` |
| Compose project | `/opt/wireguard` |
| Compose file | `/opt/wireguard/compose.yml` |
| Container | `wireguard` |
| Image policy | Immutable digest, updated only in an approved maintenance window |
| Public endpoint | `us4.wsvc.info:51820/udp`; DNS publishes only A `185.201.226.122` (no native AAAA) |
| Tunnel subnet | `10.13.13.0/24` |
| Routing policy | IPv4-only full tunnel (`ALLOWEDIPS=0.0.0.0/0`); IPv6 traffic is not guaranteed to use the VPN |

Upstream image documentation:
[LinuxServer.io WireGuard](https://docs.linuxserver.io/images/docker-wireguard/).

## Deployment configuration

The repository-owned, non-secret Compose declaration is rendered from
`ansible/templates/wireguard-compose.yml.j2`. The live declaration was verified
on 2026-08-12 with these core settings:

| Setting | Live value / intent |
|---------|---------------------|
| Image | `lscr.io/linuxserver/wireguard@sha256:ac43e1226878d2611315172d6ea357a95cb326ee73124b91108118efc8666889` |
| Image version | `1.0.20260223-r0-ls119` (build 2026-07-30) |
| Required capability | `NET_ADMIN` only; host kernel already supplies WireGuard/iptables, so `SYS_MODULE` and `/lib/modules` are not granted |
| Filesystem | Read-only container root; executable tmpfs at `/run`; writable bind mount `/opt/wireguard/config:/config` |
| Restart | `unless-stopped` |
| Server mode | Named peers `ha`, `phone`, `mbp`; runtime and configured peer counts both `3` |
| Client DNS | `1.1.1.1` |
| Tunnel routing | IPv4 full tunnel, `0.0.0.0/0`; no client IPv6 tunnel |
| Runtime interface | `wg0`, server address `10.13.13.1/32`, listen port `51820` |
| Forwarding/NAT | IPv4 forwarding enabled in the container namespace; `wg0` forwarding allowed and egress masqueraded on `eth+`; IPv6 forwarding disabled |

Docker binds UDP `51820` on both host socket families, but the public hostname
has no AAAA record. Clients using `us4.wsvc.info` therefore reach the server over
IPv4.

## Other host services and firewall (2026-08-12)

This host also carries the `windy.me` secondary MX and several web applications;
do not build its firewall allowlist from the WireGuard role alone.

| Port | Owner / purpose | Effective public state |
|------|-----------------|------------------------|
| TCP `22` | SSH management | Open |
| TCP `25` | Postfix, `mx.windy.me` (MX priority 30) | Open; retain until the secondary-MX role is explicitly retired |
| TCP `80`, `443` | Traefik for `update.wsvc.info`, `us4-gate.wsvc.info`, and `trlm.wsvc.info` | Open |
| TCP `3000` | Semaphore UI direct Docker publish | Open; redundant with the Traefik route and should be removed or bound to loopback |
| TCP `8080` | Traefik direct Docker publish | Open; redundant with the authenticated dashboard route and should be removed or bound to loopback |
| UDP `51820` | WireGuard | Required public endpoint |
| TCP `6443` | Traefik `proxyin` entrypoint → `sing-box` HTTP proxy (`/opt/sing-box`) | Open (Docker publish); added 2026-10-01, VPS-99 |
| TCP `9443` | Host nghttpx-to-Squid proxy — **legacy, retirement pending** (see HTTPS forward proxy below) | Listening but blocked by the current firewall |
| UDP `123` | ntpsec | Listening but blocked by the current firewall |

PostgreSQL (`5433`/`5434`/`5435`), MariaDB (`3306`), and the host Squid TCP
listener (`3128`) are loopback-only. Squid also owns wildcard UDP sockets, which
are not allowed by the current public zone.

UFW is not installed. Firewalld `2.3.1` is active with nftables. On 2026-08-12,
the reviewed `ansible/playbooks/us4-firewalld.yml` reconciliation removed the
stale `imap`, `imaps`, `smtp-submission`, and `smtps` services plus TCP `24`,
`6443`, and `8443` without reloading or restarting firewalld. Runtime and
permanent public-zone state now match exactly: services `dhcpv6-client`, `http`,
`https`, `smtp`, and `ssh`, with no explicit ports.

Docker-published ports are accepted through Docker's DNAT/FORWARD chains, so
the public-zone cleanup does not close `3000` or `8080`. Their Compose bindings
remain a separate, staged follow-up after the required observation window.
Firewalld logged Docker chain/policy conflicts during the 2026-08-10 boots;
treat any firewall reload or service restart as a maintenance-window operation
and reverify Docker routing. Tracking: Linear `W1N-60`.

`mx.windy.me` also publishes AAAA `2602:f9f3:0:2::878`, while the host currently
has no global IPv6 address or IPv6 default route. Treat that as a separate
secondary-MX reachability issue.

## HTTPS forward proxy (verified 2026-09-26)

The canonical HTTPS forward proxy on this host is:

| Item | Value |
|------|-------|
| Endpoint | `https://us4.wsvc.info:443` (Traefik TLS, ACME-managed) |
| Chain | Traefik `:443` `HostSNI(us4.wsvc.info)` → container `nghttpx-proxy:3000` (no-TLS) → `squid-backend:3128` |
| Config source | `/opt/nghttpx/compose.yml`, `nghttpx.conf`, `squid.conf` (server files are authoritative; not yet repo-owned) |
| Credential | `/opt/nghttpx/passwords`, owner `root:proxy` (`0:13`) mode `0640`, bcrypt (`$2y$`, cost 12). Squid reads it through a read-only bind mount; the helper runs as `13:13`, so any permission change must keep group `proxy` read access. |
| Username | `windy` |
| Password | **Not stored in this repository or Plane.** The plaintext lives only in the operator's password manager; the server holds the bcrypt hash. Rotate with `htpasswd -B -C 12`, validate against `basic_ncsa_auth` before replacing the file, then `docker stop squid-backend` → write in place → `docker start squid-backend` (a rename would break the bind mount). |
| Auth behaviour | No/incorrect credentials → `407`; correct credentials → tunnel. Verified egress IP `185.201.226.122` |
| Destination policy | HTTPS-tunnel only: `Safe_ports`/`SSL_ports` = `443`; CONNECT to other ports and destinations in `127/8`, `10/8`, `172.16/12`, `192.168/16`, `169.254/16` (metadata), `100.64/10`, `224/4`, `::1`, `fc00::/7`, `fe80::/10`, `ff00::/8` return `403` `ERR_ACCESS_DENIED` |
| Limitation | Plain-HTTP (absolute-URI) proxying is **not** supported; non-CONNECT requests fail with `400`. Clients must use HTTPS URLs. |
| Credential scope | No longer shared with `us2.wsvc.info` (the two hosts had an identical `/opt/nghttpx/passwords` hash before 2026-09-26; us4 now has its own) |

Client settings (username `windy`, password from the password manager):

```text
Proxy type: HTTP/HTTPS proxy (CONNECT)
Host:       us4.wsvc.info
Port:       443
Username:   windy
Password:   <from password manager>
```

`curl` equivalent (replace `$PROXY_PASSWORD` from the password manager; never inline it in shell history on a shared host):

```bash
curl -x https://us4.wsvc.info:443 -U "windy:$PROXY_PASSWORD" https://api.ipify.org
# expected: 185.201.226.122
```

The `9443` path (`nghttpx.service` → host `squid.service` on `127.0.0.1:3128`) is a legacy
stack: its certbot-issued certificate expired 2026-01-11, the host Squid has no
authentication at all, and no historical client traffic was found in the nghttpx access
log. It is blocked by the firewall and reachable only from the host itself. Retirement
(stop + disable both units, keep config for rollback) is pending.

Rollback material from the 2026-09-26 change: `/opt/nghttpx/backups/20260926T091907+0800`
(mode `0700`, includes the pre-change password file). Keep it protected and delete it once
the change is accepted.

### Independent `sing-box` proxy on `:6443` (added 2026-10-01, VPS-99)

A second, independent HTTPS forward proxy on `6443`. It does not touch the `:443`
chain above, and the `:443`/`:9443` stacks keep running.

| Item | Value |
|------|-------|
| Endpoint | `https://us4.wsvc.info:6443` (Traefik TLS, ACME-managed) |
| Chain | Traefik TCP router `singbox-proxy` (entrypoint `proxyin`, `HostSNI(us4.wsvc.info)`, `tls=true`, resolver `letsencrypt`) → `sing-box:8080` |
| Config source | `/opt/sing-box/compose.yml` + `config.json` (server authoritative); repo copy `compose/sing-box/compose.yml` |
| Image | `ghcr.io/sagernet/sing-box:v1.14.2@sha256:afbffd294c5eb3519cc7b4587299ef189bb0a2ca2f667cb6301fdb6b9bce9558` |
| Credential | sing-box `users` in `config.json` (mode `0600`); operator copy in `/opt/sing-box/.proxy-password`. Not in this repo or Plane. |
| Capability | Supports **both** CONNECT tunnels and absolute-URI plain-HTTP requests, unlike the `:443` path |
| Host port | The container publishes no port; `6443` is bound by Traefik |

Traefik changes live in `/opt/traefik/compose.yml`:
`--entrypoints.proxyin.address=:6443` plus the `6443:6443` publish. Changing either
requires recreating the Traefik container (`docker compose up -d`), not
`docker compose restart`.

`6443` was free before the change (nothing listening; `nginx` inactive,
`certbot`/`apache2` failed). Docker-published ports land in the firewalld `docker`
zone, not `public`, so a public-zone `--add-port=6443` is neither needed nor
effective; source restriction belongs in the `DOCKER-USER` chain or a Traefik TCP
middleware. `6443` is a public TCP port beyond the target exposure recorded in
Plane VPS-47 and still needs an explicit amendment there.

Client notes — full detail in
[runbooks/proxy-client-setup.md](../runbooks/proxy-client-setup.md):

- The client's proxy type must be **HTTPS** (TLS to the proxy). A plaintext HTTP
  proxy entry reaches Traefik in cleartext and gets `404`, not a proxied request.
- Browser extensions (Zero Omega / SwitchyOmega) must have the username and
  password saved in the proxy profile, and the routing conditions must include
  the site's CDN hostnames (for `x.com`: `twimg.com`, `api.x.com`, `t.co`), not
  only the landing domain — otherwise the page loads incompletely while the
  server-side log shows no `inbound connection to` line for those hosts.
- `python3 urllib` and `wget` cannot speak TLS to a proxy and fail with
  `Tunnel connection failed: 404 Not Found`.
- Client-side symptoms are diagnosed from the server with
  `docker logs -f sing-box | grep 'inbound connection to'`; `no
  Proxy-Authorization header` means the client sent no credentials.

## Safety

- Private keys, preshared keys, peer configuration files, and QR codes remain
  only in `/opt/wireguard/config`; do not copy them to this repository or
  Linear.
- Local rollback archives are stored in `/opt/wireguard/backups` (directory
  mode `0700`, archives mode `0600`). They contain private keys, are not an
  off-host disaster-recovery backup, and must never leave the server.
- Live private keys, preshared keys, generated peer configs, QR images, and
  `wg0.conf` are mode `0600`. Template-only `peer.conf` and `server.conf` files
  are mode `0644` and do not contain generated key material.
- `/opt/wireguard/config` is mode `0755`, but its sensitive files are `0600`.
  The current files are owned by the image's numeric UID/GID rather than the
  declared `PUID=1000` / `PGID=1000`; the root-run WireGuard processes can use
  them, but reconcile ownership only after a protected backup and maintenance
  review.
- `LOG_CONFS` is currently unset and the inspected container log contained no
  QR-code/config banners. Do not enable config logging; generated QR images are
  credentials.
- Do not delete, move, or regenerate `/opt/wireguard/config` during
  maintenance.
- Before a container recreation, validate `docker compose config` and retain a
  verified SSH session for rollback. Do not run `docker compose down -v`.

## Routine operations

Run read-only checks through Ansible:

```bash
cd ansible
ansible-playbook playbooks/health-report.yml --limit wireguard
```

Preview the narrow, fail-closed public-zone reconciliation:

```bash
ansible-galaxy collection install -r requirements.yml
ansible-playbook playbooks/us4-firewalld.yml --limit us4 --check --diff
```

Apply it only after testing the provider console and keeping an independent SSH
rollback session open. The playbook creates a protected server-local backup and
a 15-minute automatic rollback before changing rules; it cancels that rollback
only after SSH, HTTPS, SMTP, Docker, Fail2ban, and WireGuard checks pass:

```bash
ansible-playbook playbooks/us4-firewalld.yml --limit us4 \
  -e '{"us4_firewalld_confirm": true, "us4_console_confirm": true}'
```

The image update and recreate procedure is deliberately separate and requires
an immutable image digest in the server-side Compose file plus an explicit
maintenance-window confirmation:

```bash
cd ansible
ansible-playbook playbooks/wireguard-maintenance.yml --limit wireguard \
  -e '{"wireguard_maintenance_confirm": true}'
```

Before that recreate, apply the reviewed Compose hardening from the repository:

```bash
ansible-playbook playbooks/wireguard-harden.yml --limit wireguard \
  -e '{"wireguard_harden_confirm": true}'
```

Forward-proxy health for both endpoints (`:443` and `:6443`) is read-only:
follow [runbooks/proxy-health.md](../runbooks/proxy-health.md). No proxy
credential is stored in this repository.

## Verification

- Container `wireguard` is running and interface `wg0` exists.
- UDP `51820` is listening on the host.
- Validate a known client can handshake and sends IPv4 traffic through the VPN.
- Do not treat inactive mobile peers as a failure solely because their latest
  handshake is old.

## Live audit snapshot (2026-08-12)

The WireGuard service itself is healthy and its installation is broadly
reasonable:

- The sanitized Ansible health report returned `status=ok`; Compose is valid,
  the container is running with zero restarts, `wg0` exists, and UDP `51820` is
  listening.
- One of three peers had a current handshake during the audit. Two peers had
  not handshaken since the current container/interface start; confirm those
  clients only if they are expected to be active.
- The image is immutable-digest pinned, key-bearing files are protected, the
  container root is read-only, and the container has `NET_ADMIN` without the
  broader `SYS_MODULE` capability.
- Debian `13.6`, kernel `6.12.101+deb13-amd64`, Docker Engine `29.7.2`, and
  Docker Compose `v5.4.0` were observed. No Debian package updates or reboot
  requirement were pending.

Open host-level follow-up (do not conflate these with a WireGuard outage):

1. **Disk capacity:** `/` was 90% used with about 3.4 GiB free. Docker reported
   about 2.48 GB of reclaimable images and the system journal used about 1.9
   GB, but do not prune or vacuum without reviewing retention and rollback
   needs first. — **Resolved 2026-09-26 (91% → 43%); see *Root disk recovery*
   below.**
2. **Docker exposure:** the firewalld public-zone cleanup is complete, but
   Docker still publishes `3000` and `8080` outside the ordinary host INPUT
   path. Remove those redundant Compose bindings in separate maintenance units
   after the observation window, and confirm provider firewall rules first.
3. **Image maintenance:** the upstream `latest` amd64 image had advanced to
   `1.0.20260223-r0-ls120` (build 2026-08-06). Review and pin its immutable
   digest in a maintenance window rather than updating unattended.
4. **Host hygiene:** `apache2.service`, `certbot.service`, and
   `postgresql@9.6-main.service` were in a failed state while unrelated Docker
   workloads remained active. Establish ownership and remove or repair stale
   units separately.
5. **Resource/log limits:** the WireGuard container has no memory, CPU, or PID
   limit and uses Docker's `json-file` log driver without a per-container
   rotation setting. Current log size was small, but limits/rotation should be
   considered during a reviewed Compose update.

## Root disk recovery (2026-09-26, Plane VPS-100)

`/` went from **91% used (3.0G free)** to **43% (18G free)** — about 14G
reclaimed, following [runbooks/host-disk-cleanup.md](../runbooks/host-disk-cleanup.md).

Removed: `~/.nvm` 2.6G, `~/.go` 1.9G, `~/.rustup` 1.4G, `~/.cargo` 505M,
`~/.npm` 78M, stale rootless-podman storage under `~/.local/share/containers`
1.7G, `~/sigstore` 953M, `~/softz` 910M, `~/.cache` 295M, apt archives 613M,
the container JSON logs (594M → 320K), and six unreferenced Docker images
(`postgres:17`, `postgres:18`, `traefik:v3.4`, `zadam/trilium:0.60.4`,
`linuxserver/wireguard:latest`, `traefik/whoami:latest` — all re-pullable).

Durable changes, not just deletions:

- `journalctl --vacuum-size=100M` plus drop-in
  `/etc/systemd/journald.conf.d/00-disk-us4.conf` (`SystemMaxUse=200M`);
  journald was restarted so the cap is effective (journal back to 83M). Without
  this the journal regrows toward its default 10%-of-filesystem limit.
- Log rotation `max-size: 10m` / `max-file: 3` added to `nghttpx-proxy`,
  `squid-backend` (`/opt/nghttpx/compose.yml`) and `traefik`
  (`/opt/traefik/compose.yml`); both stacks were recreated and verified. This
  closes the unbounded `json.log` growth for these containers.
- `~/.zshenv`, `~/.bashrc`, `~/.profile` and `~/.zshrc` had their `~/.cargo/env`
  and nvm references commented out after those trees were deleted; backups are
  `~/.<file>.bak-20260926-disk`. Also note `~/.go/bin/cosign` was removed and is
  re-installable.

Deliberately kept: `~/.local/share/trilium-data` (88M) is a 2022 Trilium
database, not a cache. Still removable if unneeded: `/opt/zeroclaw/target`
1.4G, `/opt/trilium-linux-x64-server` 592M (2023 host install, unit disabled),
`/opt/go` 167M (2017).