From 7ec29627fb281cae37dffdb2742ec1502d4dca20 Mon Sep 17 00:00:00 2001 From: windyboy Date: Sat, 26 Sep 2026 10:34:09 +0800 Subject: [PATCH] =?UTF-8?q?docs(us4):=20sing-box=20:6443=20=E4=BB=A3?= =?UTF-8?q?=E7=90=86=20+=20=E6=A0=B9=E7=9B=98=E5=9B=9E=E6=94=B6=2091%?= =?UTF-8?q?=E2=86=9243%=20(Plane=20VPS-99/100)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增独立 HTTPS forward proxy：Traefik 新 entrypoint proxyin(:6443) 终止 TLS， 后端 sing-box（HTTP inbound + users 认证 + direct 出站），与既有 :443 nghttpx/squid 链路并存。宿主机只新增 6443/tcp；容器不发布 host 端口。 新增 compose/sing-box/（镜像 digest 钉版）与两份 runbook： - proxy-health.md：:443/:6443 只读健康检查，含开放代理负向测试 - proxy-client-setup.md：curl / env / 浏览器扩展配置与三类失败排查 （407 无认证头、页面加载不全=CDN 域名漏配、urllib/wget 不支持 TLS-to-proxy） 同批根盘回收 91%→43%（约 14G）：rust/node/.go/podman 孤儿存储/apt/journal/ 未用镜像/容器日志；journald 加 SystemMaxUse=200M；nghttpx+squid+traefik 加 json-file 轮转 10m×3；删除 ~/.cargo 后修好 zshenv/bashrc/profile 的失效 source。 验证：:6443 LE 证书、匿名 407、CONNECT 与绝对 URI 均返回 185.201.226.122； us4-gate 401 / update 200 / trlm 302；8 容器全部 Up；validate-repo.sh PASS。 --- compose/README.md | 1 + compose/sing-box/compose.yml | 47 ++++++ compose/sing-box/config.example.json | 29 ++++ hosts/us4.wsvc.info.md | 129 ++++++++++++++- runbooks/README.md | 2 + runbooks/proxy-client-setup.md | 142 +++++++++++++++++ runbooks/proxy-health.md | 226 +++++++++++++++++++++++++++ 7 files changed, 574 insertions(+), 2 deletions(-) create mode 100644 compose/sing-box/compose.yml create mode 100644 compose/sing-box/config.example.json create mode 100644 runbooks/proxy-client-setup.md create mode 100644 runbooks/proxy-health.md diff --git a/compose/README.md b/compose/README.md index 946d69d..d556bde 100644 --- a/compose/README.md +++ b/compose/README.md @@ -17,6 +17,7 @@ files: every secret is a `${VAR}` reference resolved from the **server-local | `unifi` | ubnt (`/home/windy/unifi-9`) | — (待从 LAN 提取) | static file (pending) | | `wireguard` | us4 (`/opt/wireguard`) | `ansible/templates/wireguard-compose.yml.j2` | role-rendered (inventory vars) | | `rustdesk` | hk2 (`/opt/rustdesk`) | `ansible/roles/rustdesk/templates/compose.yml.j2` | role-rendered (inventory vars) | +| `sing-box` | us4 (`/opt/sing-box`) | `compose/sing-box/compose.yml` (+ `config.example.json`) | static file（参考镜像, 未接入 compose-deploy; 服务器文件为准; 凭据仅存服务器 `config.json` 0600, VPS-99） | | `mailcow` | mx2 (`/opt/mail`) | — (mailcow update generator owns it) | excluded by design | Mechanism rule: **static** `compose//compose.yml` for declarations that diff --git a/compose/sing-box/compose.yml b/compose/sing-box/compose.yml new file mode 100644 index 0000000..d9a0db7 --- /dev/null +++ b/compose/sing-box/compose.yml @@ -0,0 +1,47 @@ +# sing-box HTTP forward proxy on us4.wsvc.info, published through Traefik :6443. +# +# TLS terminates in Traefik (entrypoint `proxyin`); sing-box only speaks plain +# HTTP-proxy on the private `proxy` Docker network, so it publishes NO host port. +# The credential lives in the server-side ./config.json (mode 0600) and is never +# committed here. +# +# Repo-owned copy of /opt/sing-box/compose.yml (static mechanism, not yet wired +# to compose-deploy). Change record: Plane VPS-99. Health: runbooks/proxy-health.md. +services: + sing-box: + image: ghcr.io/sagernet/sing-box:v1.14.2@sha256:afbffd294c5eb3519cc7b4587299ef189bb0a2ca2f667cb6301fdb6b9bce9558 + container_name: sing-box + restart: unless-stopped + security_opt: + - no-new-privileges:true + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + command: + - run + - -c + - /etc/sing-box/config.json + volumes: + - ./config.json:/etc/sing-box/config.json:ro + networks: + - proxy + labels: + - "traefik.enable=true" + - "traefik.docker.network=proxy" + + # TCP router: TLS terminates here, decrypted bytes go to sing-box. + - "traefik.tcp.routers.singbox-proxy.rule=HostSNI(`us4.wsvc.info`)" + - "traefik.tcp.routers.singbox-proxy.entrypoints=proxyin" + - "traefik.tcp.routers.singbox-proxy.tls=true" + - "traefik.tcp.routers.singbox-proxy.tls.certresolver=letsencrypt" + + # TCP service: in-container HTTP proxy port. + - "traefik.tcp.routers.singbox-proxy.service=singbox-proxy" + - "traefik.tcp.services.singbox-proxy.loadbalancer.server.port=8080" + +networks: + proxy: + name: proxy + external: true diff --git a/compose/sing-box/config.example.json b/compose/sing-box/config.example.json new file mode 100644 index 0000000..60b0a37 --- /dev/null +++ b/compose/sing-box/config.example.json @@ -0,0 +1,29 @@ +{ + "log": { + "level": "info", + "timestamp": true + }, + "inbounds": [ + { + "type": "http", + "tag": "http-proxy", + "listen": "0.0.0.0", + "listen_port": 8080, + "users": [ + { + "username": "windy", + "password": "REPLACE_WITH_openssl_rand_hex_32" + } + ] + } + ], + "outbounds": [ + { + "type": "direct", + "tag": "direct" + } + ], + "route": { + "final": "direct" + } +} diff --git a/hosts/us4.wsvc.info.md b/hosts/us4.wsvc.info.md index 6511e49..867cd23 100644 --- a/hosts/us4.wsvc.info.md +++ b/hosts/us4.wsvc.info.md @@ -52,7 +52,8 @@ do not build its firewall allowlist from the WireGuard role alone. | TCP `3000` | Semaphore UI direct Docker publish | Open; redundant with the Traefik route and should be removed or bound to loopback | | TCP `8080` | Traefik direct Docker publish | Open; redundant with the authenticated dashboard route and should be removed or bound to loopback | | UDP `51820` | WireGuard | Required public endpoint | -| TCP `9443` | Host nghttpx-to-Squid proxy | Listening but blocked by the current firewall | +| TCP `6443` | Traefik `proxyin` entrypoint → `sing-box` HTTP proxy (`/opt/sing-box`) | Open (Docker publish); added 2026-10-01, VPS-99 | +| TCP `9443` | Host nghttpx-to-Squid proxy — **legacy, retirement pending** (see HTTPS forward proxy below) | Listening but blocked by the current firewall | | UDP `123` | ntpsec | Listening but blocked by the current firewall | PostgreSQL (`5433`/`5434`/`5435`), MariaDB (`3306`), and the host Squid TCP @@ -77,6 +78,93 @@ and reverify Docker routing. Tracking: Linear `W1N-60`. has no global IPv6 address or IPv6 default route. Treat that as a separate secondary-MX reachability issue. +## HTTPS forward proxy (verified 2026-09-26) + +The canonical HTTPS forward proxy on this host is: + +| Item | Value | +|------|-------| +| Endpoint | `https://us4.wsvc.info:443` (Traefik TLS, ACME-managed) | +| Chain | Traefik `:443` `HostSNI(us4.wsvc.info)` → container `nghttpx-proxy:3000` (no-TLS) → `squid-backend:3128` | +| Config source | `/opt/nghttpx/compose.yml`, `nghttpx.conf`, `squid.conf` (server files are authoritative; not yet repo-owned) | +| Credential | `/opt/nghttpx/passwords`, owner `root:proxy` (`0:13`) mode `0640`, bcrypt (`$2y$`, cost 12). Squid reads it through a read-only bind mount; the helper runs as `13:13`, so any permission change must keep group `proxy` read access. | +| Username | `windy` | +| Password | **Not stored in this repository or Plane.** The plaintext lives only in the operator's password manager; the server holds the bcrypt hash. Rotate with `htpasswd -B -C 12`, validate against `basic_ncsa_auth` before replacing the file, then `docker stop squid-backend` → write in place → `docker start squid-backend` (a rename would break the bind mount). | +| Auth behaviour | No/incorrect credentials → `407`; correct credentials → tunnel. Verified egress IP `185.201.226.122` | +| Destination policy | HTTPS-tunnel only: `Safe_ports`/`SSL_ports` = `443`; CONNECT to other ports and destinations in `127/8`, `10/8`, `172.16/12`, `192.168/16`, `169.254/16` (metadata), `100.64/10`, `224/4`, `::1`, `fc00::/7`, `fe80::/10`, `ff00::/8` return `403` `ERR_ACCESS_DENIED` | +| Limitation | Plain-HTTP (absolute-URI) proxying is **not** supported; non-CONNECT requests fail with `400`. Clients must use HTTPS URLs. | +| Credential scope | No longer shared with `us2.wsvc.info` (the two hosts had an identical `/opt/nghttpx/passwords` hash before 2026-09-26; us4 now has its own) | + +Client settings (username `windy`, password from the password manager): + +```text +Proxy type: HTTP/HTTPS proxy (CONNECT) +Host: us4.wsvc.info +Port: 443 +Username: windy +Password: +``` + +`curl` equivalent (replace `$PROXY_PASSWORD` from the password manager; never inline it in shell history on a shared host): + +```bash +curl -x https://us4.wsvc.info:443 -U "windy:$PROXY_PASSWORD" https://api.ipify.org +# expected: 185.201.226.122 +``` + +The `9443` path (`nghttpx.service` → host `squid.service` on `127.0.0.1:3128`) is a legacy +stack: its certbot-issued certificate expired 2026-01-11, the host Squid has no +authentication at all, and no historical client traffic was found in the nghttpx access +log. It is blocked by the firewall and reachable only from the host itself. Retirement +(stop + disable both units, keep config for rollback) is pending. + +Rollback material from the 2026-09-26 change: `/opt/nghttpx/backups/20260926T091907+0800` +(mode `0700`, includes the pre-change password file). Keep it protected and delete it once +the change is accepted. + +### Independent `sing-box` proxy on `:6443` (added 2026-10-01, VPS-99) + +A second, independent HTTPS forward proxy on `6443`. It does not touch the `:443` +chain above, and the `:443`/`:9443` stacks keep running. + +| Item | Value | +|------|-------| +| Endpoint | `https://us4.wsvc.info:6443` (Traefik TLS, ACME-managed) | +| Chain | Traefik TCP router `singbox-proxy` (entrypoint `proxyin`, `HostSNI(us4.wsvc.info)`, `tls=true`, resolver `letsencrypt`) → `sing-box:8080` | +| Config source | `/opt/sing-box/compose.yml` + `config.json` (server authoritative); repo copy `compose/sing-box/compose.yml` | +| Image | `ghcr.io/sagernet/sing-box:v1.14.2@sha256:afbffd294c5eb3519cc7b4587299ef189bb0a2ca2f667cb6301fdb6b9bce9558` | +| Credential | sing-box `users` in `config.json` (mode `0600`); operator copy in `/opt/sing-box/.proxy-password`. Not in this repo or Plane. | +| Capability | Supports **both** CONNECT tunnels and absolute-URI plain-HTTP requests, unlike the `:443` path | +| Host port | The container publishes no port; `6443` is bound by Traefik | + +Traefik changes live in `/opt/traefik/compose.yml`: +`--entrypoints.proxyin.address=:6443` plus the `6443:6443` publish. Changing either +requires recreating the Traefik container (`docker compose up -d`), not +`docker compose restart`. + +`6443` was free before the change (nothing listening; `nginx` inactive, +`certbot`/`apache2` failed). Docker-published ports land in the firewalld `docker` +zone, not `public`, so a public-zone `--add-port=6443` is neither needed nor +effective; source restriction belongs in the `DOCKER-USER` chain or a Traefik TCP +middleware. `6443` is a public TCP port beyond the target exposure recorded in +Plane VPS-47 and still needs an explicit amendment there. + +Client notes — full detail in +[runbooks/proxy-client-setup.md](../runbooks/proxy-client-setup.md): + +- The client's proxy type must be **HTTPS** (TLS to the proxy). A plaintext HTTP + proxy entry reaches Traefik in cleartext and gets `404`, not a proxied request. +- Browser extensions (Zero Omega / SwitchyOmega) must have the username and + password saved in the proxy profile, and the routing conditions must include + the site's CDN hostnames (for `x.com`: `twimg.com`, `api.x.com`, `t.co`), not + only the landing domain — otherwise the page loads incompletely while the + server-side log shows no `inbound connection to` line for those hosts. +- `python3 urllib` and `wget` cannot speak TLS to a proxy and fail with + `Tunnel connection failed: 404 Not Found`. +- Client-side symptoms are diagnosed from the server with + `docker logs -f sing-box | grep 'inbound connection to'`; `no + Proxy-Authorization header` means the client sent no credentials. + ## Safety - Private keys, preshared keys, peer configuration files, and QR codes remain @@ -144,6 +232,10 @@ ansible-playbook playbooks/wireguard-harden.yml --limit wireguard \ -e '{"wireguard_harden_confirm": true}' ``` +Forward-proxy health for both endpoints (`:443` and `:6443`) is read-only: +follow [runbooks/proxy-health.md](../runbooks/proxy-health.md). No proxy +credential is stored in this repository. + ## Verification - Container `wireguard` is running and interface `wg0` exists. @@ -175,7 +267,8 @@ Open host-level follow-up (do not conflate these with a WireGuard outage): 1. **Disk capacity:** `/` was 90% used with about 3.4 GiB free. Docker reported about 2.48 GB of reclaimable images and the system journal used about 1.9 GB, but do not prune or vacuum without reviewing retention and rollback - needs first. + needs first. — **Resolved 2026-09-26 (91% → 43%); see *Root disk recovery* + below.** 2. **Docker exposure:** the firewalld public-zone cleanup is complete, but Docker still publishes `3000` and `8080` outside the ordinary host INPUT path. Remove those redundant Compose bindings in separate maintenance units @@ -191,3 +284,35 @@ Open host-level follow-up (do not conflate these with a WireGuard outage): limit and uses Docker's `json-file` log driver without a per-container rotation setting. Current log size was small, but limits/rotation should be considered during a reviewed Compose update. + +## Root disk recovery (2026-09-26, Plane VPS-100) + +`/` went from **91% used (3.0G free)** to **43% (18G free)** — about 14G +reclaimed, following [runbooks/host-disk-cleanup.md](../runbooks/host-disk-cleanup.md). + +Removed: `~/.nvm` 2.6G, `~/.go` 1.9G, `~/.rustup` 1.4G, `~/.cargo` 505M, +`~/.npm` 78M, stale rootless-podman storage under `~/.local/share/containers` +1.7G, `~/sigstore` 953M, `~/softz` 910M, `~/.cache` 295M, apt archives 613M, +the container JSON logs (594M → 320K), and six unreferenced Docker images +(`postgres:17`, `postgres:18`, `traefik:v3.4`, `zadam/trilium:0.60.4`, +`linuxserver/wireguard:latest`, `traefik/whoami:latest` — all re-pullable). + +Durable changes, not just deletions: + +- `journalctl --vacuum-size=100M` plus drop-in + `/etc/systemd/journald.conf.d/00-disk-us4.conf` (`SystemMaxUse=200M`); + journald was restarted so the cap is effective (journal back to 83M). Without + this the journal regrows toward its default 10%-of-filesystem limit. +- Log rotation `max-size: 10m` / `max-file: 3` added to `nghttpx-proxy`, + `squid-backend` (`/opt/nghttpx/compose.yml`) and `traefik` + (`/opt/traefik/compose.yml`); both stacks were recreated and verified. This + closes the unbounded `json.log` growth for these containers. +- `~/.zshenv`, `~/.bashrc`, `~/.profile` and `~/.zshrc` had their `~/.cargo/env` + and nvm references commented out after those trees were deleted; backups are + `~/..bak-20260926-disk`. Also note `~/.go/bin/cosign` was removed and is + re-installable. + +Deliberately kept: `~/.local/share/trilium-data` (88M) is a 2022 Trilium +database, not a cache. Still removable if unneeded: `/opt/zeroclaw/target` +1.4G, `/opt/trilium-linux-x64-server` 592M (2023 host install, unit disabled), +`/opt/go` 167M (2017). diff --git a/runbooks/README.md b/runbooks/README.md index e8f2961..501a655 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -18,6 +18,8 @@ runbooks start from [`_template.md`](_template.md). | Gitea health check | [gitea-health.md](gitea-health.md) | read-only | | Matrix health check | [matrix-health.md](matrix-health.md) | read-only | | Plane health check | [plane-health.md](plane-health.md) | read-only | +| us4 forward proxy health check (`:443` + `:6443`) | [proxy-health.md](proxy-health.md) | read-only | +| us4 forward proxy client setup (curl / env / browser) | [proxy-client-setup.md](proxy-client-setup.md) | reference | | pgdb health check | [pgdb-health.md](pgdb-health.md) | read-only | | pgdb DB restore (pg_restore) | [pgdb-restore.md](pgdb-restore.md) | change (procedure) | | pgdb image/compose update | [pgdb-update.md](pgdb-update.md) | change (gated) | diff --git a/runbooks/proxy-client-setup.md b/runbooks/proxy-client-setup.md new file mode 100644 index 0000000..b94428a --- /dev/null +++ b/runbooks/proxy-client-setup.md @@ -0,0 +1,142 @@ +# Runbook: us4 forward proxy — client configuration + +## Purpose + +Configure a client (curl, environment variables, browser, or a proxy-switching +extension) to use the authenticated HTTPS forward proxies on `us4.wsvc.info`, +and triage the three failures seen in practice: `407` responses, pages that load +incompletely, and clients that cannot speak TLS to a proxy. + +## Scope + +- Applicable: the two authenticated HTTPS forward proxies on `us4.wsvc.info` + (`:6443` sing-box, `:443` nghttpx → Squid). +- Applicable: curl, `https_proxy`/`HTTPS_PROXY` environment variables, browser + proxy-switching extensions (Zero Omega / SwitchyOmega), and OS proxy settings. +- Not applicable: server-side proxy health — use [proxy-health.md](proxy-health.md). +- Not applicable: changing proxy, Traefik, or firewall configuration — use + [network-change.md](network-change.md) with a change record. +- Not applicable: WireGuard client setup. + +## Ownership + +- Owner: personal ops (Windy) +- Last reviewed: 2026-09-26 +- Related systems: us4 (`/opt/sing-box`, `/opt/nghttpx`, `/opt/traefik`) + +## Safety + +### Non-negotiable rules + +- Never put the proxy credential in this repository, in Plane, or in a shell + command that lands in shell history on a shared host. +- Read it from its server-side source only: + `ssh -4 windy@us4.wsvc.info 'sudo cat /opt/sing-box/.proxy-password'`. +- Never "fix" a client problem by removing proxy authentication or by publishing + the backend port. + +### Stop conditions + +- The client sends no `Proxy-Authorization` and the server answers `407`: that is + the server behaving correctly. `STOP` — fix the client, do not change the server. +- A plaintext HTTP request to either port is answered by Traefik with `404`: the + client is not speaking TLS to the proxy. `STOP` and change the client's proxy + protocol; do not add a plaintext entrypoint. +- A known-good curl command from [proxy-health.md](proxy-health.md) Step 4 also + fails: `STOP` and run the health runbook — the fault is server-side. + +## Endpoint facts + +| | `:6443` (sing-box) | `:443` (nghttpx → Squid) | +|---|---|---| +| Proxy type to the proxy | **HTTPS / TLS** | **HTTPS / TLS** | +| Username | `windy` | `windy` | +| Credential source | `/opt/sing-box/.proxy-password` (server-side, mode 0600) | operator password manager; server holds a bcrypt hash | +| CONNECT (`https://` targets) | yes | yes | +| Absolute-URI (`http://` targets) | yes | **no** — nghttpx answers `400` | + +Both endpoints terminate TLS in Traefik. The client's proxy type is therefore +HTTPS (TLS to the proxy), not "HTTP proxy on port 6443". + +## Client configuration + +### curl + +```bash +PW=$(ssh -4 windy@us4.wsvc.info 'sudo cat /opt/sing-box/.proxy-password') +curl -sS --proxy https://us4.wsvc.info:6443 --proxy-user "windy:$PW" https://ipinfo.io/ip +# expected: 185.201.226.122 +``` + +### Environment variables + +curl prefers the **lowercase** `https_proxy` over `HTTPS_PROXY`; a pre-existing +lowercase value wins even when the uppercase one is overridden. On the WSL +client both are pre-set to the LAN proxy `http://192.168.66.99:7890`, so +overriding only `HTTPS_PROXY` silently keeps using the LAN proxy (observed: +returns `152.175.29.59` instead of us4's `185.201.226.122`). Set the lowercase +variable, or pass `--proxy` explicitly. + +```bash +https_proxy="https://windy:$PW@us4.wsvc.info:6443" curl -sS https://ipinfo.io/ip +``` + +### Browser proxy-switching extension (Zero Omega / SwitchyOmega) + +1. Proxy profile: protocol **HTTPS**, server `us4.wsvc.info`, port `6443`, + username `windy`, password from the server-side file. Save **and** click + *Apply changes* — an unapplied profile is ignored. +2. Routing conditions must cover every hostname the site needs, not only the + landing domain. For `x.com` that means `x.com`, `twitter.com`, `api.x.com`, + `t.co`, `twimg.com`, `abs.twimg.com`, `pbs.twimg.com`, `video.twimg.com`. + Missing CDN domains are the usual cause of a page that loads incompletely. +3. Set `chrome://flags/#enable-quic` to **Disabled** if individual resources + stall: QUIC is UDP and cannot traverse a CONNECT proxy. +4. Watch the server side while the browser loads the page: + +```bash +ssh -4 windy@us4.wsvc.info "docker logs -f sing-box" | grep --line-buffered 'inbound connection to' +``` + +## Troubleshooting + +### A — `407 Proxy Authentication Required` + +Server log on us4: `authentication failed, no Proxy-Authorization header` — the +client connected but sent no credentials (extension profile missing the +username/password, or not applied). Add them and re-apply. +`authentication failed, incorrect password` instead means the value itself is +wrong; re-read it from `/opt/sing-box/.proxy-password`. + +### B — Page loads but is incomplete + +Some hostnames are not routed through the proxy (see the condition list above), +or QUIC is stalling. Confirm with the live log: if the site's CDN hosts never +appear as an `inbound connection to` line, they are not being proxied at all. + +### C — Client cannot speak TLS to a proxy + +`python3 urllib` and `wget` treat an `https://` proxy as a plaintext HTTP proxy. +The request reaches Traefik in cleartext and is answered `404 Not Found` +(observed: `Tunnel connection failed: 404 Not Found`). Use curl, or a client that +supports HTTPS proxies. + +### D — `400 Bad Request` on the `:443` endpoint + +The `:443` path is CONNECT-only. Absolute-URI plain-HTTP proxying is available +only on `:6443`. + +## Final Verification + +- `curl --proxy https://us4.wsvc.info:6443 --proxy-user "windy:" https://ipinfo.io/ip` + returns `185.201.226.122`. +- The browser loads the target site completely, and each expected CDN host + appears in the sing-box log as an `inbound connection to` line. +- No credential was written to the repository, Plane, or a shared transcript. + +## References + +- [proxy-health.md](proxy-health.md) — read-only server-side health check +- [hosts/us4.wsvc.info.md](../hosts/us4.wsvc.info.md) — endpoint and credential facts +- [network-change.md](network-change.md), [rollback.md](rollback.md) +- Plane VPS-99 (proxy deployment), VPS-100 (root disk recovery) diff --git a/runbooks/proxy-health.md b/runbooks/proxy-health.md new file mode 100644 index 0000000..a2931ec --- /dev/null +++ b/runbooks/proxy-health.md @@ -0,0 +1,226 @@ +# Runbook: us4 forward proxy health check + +## Purpose + +Read-only health check for the HTTPS forward proxies on `us4.wsvc.info`: the +canonical `:443` path (Traefik TCP SNI → nghttpx → Squid) and the independent +`:6443` path (Traefik TCP SNI → sing-box). Confirms TLS, authentication, egress +identity, and that neither endpoint is an open proxy. + +## Scope + +- Applicable: `us4.wsvc.info` only — Traefik containers, the + `/opt/nghttpx` project (`:443` path) and the `/opt/sing-box` project (`:6443`). +- Applicable: verifying that no unauthenticated or wrong-credential request is + proxied, and that private/metadata destinations are refused. +- Not applicable: any change to Traefik, nghttpx, Squid, or sing-box + configuration — this runbook is read-only; changes go through + [network-change.md](network-change.md) with a change record. +- Not applicable: client-side proxy configuration and browser symptoms + (missing credentials, incomplete page loads, TLS-to-proxy support) — use + [proxy-client-setup.md](proxy-client-setup.md). +- Not applicable: host firewall, WireGuard, or the `:9443` legacy host path + retirement (see Plane VPS-98). + +## Ownership + +- Owner: personal ops (Windy) +- Last reviewed: 2026-10-01 +- Related systems: us4 (`/opt/traefik`, `/opt/nghttpx`, `/opt/sing-box`) + +## Safety + +### Stop conditions + +- Live state contradicts the expected topology below → `STOP`, record evidence, + change nothing in this runbook. +- Any request without a credential, or with a wrong credential, returns proxied + content instead of `407` → `STOP` immediately: treat the endpoint as an open + proxy, keep the evidence, and escalate before any other action. +- A private, loopback, link-local, or metadata destination is reachable through + the proxy → `STOP` and escalate. +- An expected listener (`:6443`, `:443`, or sing-box `:8080` inside its network) + is missing, or a container is restart-looping → `STOP`; do not "fix it in + place" from this runbook. + +## Expected topology + +| Endpoint | Chain | Credential | +|---|---|---| +| `us4.wsvc.info:443` | Traefik TCP router `ng-gateway` (entrypoint `websecure`, `HostSNI(us4.wsvc.info)`, `tls=true`, resolver `letsencrypt`) → `nghttpx-proxy:3000` → `squid-backend:3128` | Squid bcrypt (`/opt/nghttpx/passwords`) | +| `us4.wsvc.info:6443` | Traefik TCP router `singbox-proxy` (entrypoint `proxyin`, `HostSNI(us4.wsvc.info)`, `tls=true`, resolver `letsencrypt`) → `sing-box:8080` | sing-box `users` in `/opt/sing-box/config.json` | +| host `:9443` | host `nghttpx.service` → host Squid `:3128` | none (legacy, firewall-blocked) | + +Known limitation of the `:443` path: nghttpx is CONNECT-only and answers `400` +to absolute-URI plain-HTTP requests. The `:6443` sing-box path supports both. + +## Pass criteria + +1. `docker ps` shows `traefik`, `sing-box`, `nghttpx-proxy`, `squid-backend` + running, with `sing-box` and `traefik` restart counts not increasing. +2. `docker inspect traefik --format '{{json .Config.Cmd}}'` contains + `--entrypoints.proxyin.address=:6443`. +3. TLS on `:6443` serves a valid Let's Encrypt certificate for + `us4.wsvc.info` (not the self-signed default certificate). +4. No credential → `407`; wrong credential → `407`; plain HTTP to `:6443` → + TLS handshake failure. +5. Authenticated request through `:6443` returns egress IP `185.201.226.122` + for both an HTTPS (CONNECT) target and an HTTP (absolute-URI) target. +6. Authenticated request through `:443` returns the same egress IP (CONNECT + target only). +7. `https://us4-gate.wsvc.info` still answers (dashboard, basic auth → `401`), + and the `update`/`trlm` sites are unaffected. + +## Procedure + +### Step 1 — Container and listener state + +**Action** + +```bash +ssh -4 windy@us4.wsvc.info +docker ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}' +docker inspect traefik --format '{{json .Config.Cmd}}' +sudo ss -ltnp | grep -E ':(6443|443|9443)\b' +``` + +**Expected** + +`traefik` publishing `6443`, `sing-box` running with no host port, and the +`proxyin` entrypoint present in Traefik's command. + +**Decision** + +- All present → Step 2. +- Anything missing or restart-looping → `STOP` and record the evidence. + +### Step 2 — TLS certificate + +**Action** + +```bash +openssl s_client -connect us4.wsvc.info:6443 -servername us4.wsvc.info /dev/null \ + | openssl x509 -noout -subject -issuer -dates -ext subjectAltName +``` + +**Expected** + +`subject=CN=us4.wsvc.info`, issuer Let's Encrypt, `notAfter` in the future. + +**Decision** + +- Valid LE certificate → Step 3. +- Self-signed default certificate, expired certificate, or handshake failure → + `STOP`; the certificate is issued on demand, so retry once before concluding. + +### Step 3 — Negative authentication tests (run from an external client) + +**Action** + +```bash +# no credential +curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 \ + --proxy https://us4.wsvc.info:6443 https://example.com + +# wrong credential +curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 \ + --proxy https://us4.wsvc.info:6443 --proxy-user 'windy:definitely-wrong' https://example.com + +# plain HTTP to the TLS port +curl -sS -o /dev/null -w '%{http_code}\n' --max-time 8 http://us4.wsvc.info:6443/ +``` + +**Expected** + +The two proxy attempts fail authentication (`407`, or a curl CONNECT abort), and +the plain-HTTP attempt fails the TLS handshake. + +**Decision** + +- All rejected → Step 4. +- Any of them returns proxied content → `STOP` immediately (open proxy). + +### Step 4 — Positive path and egress identity + +**Action** — supply the credential from its server-side source; never paste it +into Plane, the repo, or a shared transcript. + +```bash +# on us4, for :6443 (CONNECT target and absolute-URI HTTP target) +PW="$(sudo cat /opt/sing-box/.proxy-password)" +curl -sS --proxy https://us4.wsvc.info:6443 --proxy-user "windy:$PW" https://ipinfo.io/ip +curl -sS --proxy https://us4.wsvc.info:6443 --proxy-user "windy:$PW" http://ipinfo.io/ip + +# on us4, for the :443 path +curl -sS --proxy https://us4.wsvc.info:443 --proxy-user 'windy:' https://ipinfo.io/ip +``` + +**Expected** + +`185.201.226.122` on all successful requests. + +**Decision** + +- Expected egress IP → Step 5. +- `407` with the known-good credential → `STOP`; check `docker logs sing-box` + for the inbound user list before touching configuration. + +### Step 5 — Regression on existing HTTPS sites + +**Action** + +```bash +for h in us4-gate.wsvc.info update.wsvc.info trlm.wsvc.info; do + printf '%s ' "$h" + curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 "https://$h" +done +``` + +**Expected** + +`us4-gate` → `401` (dashboard basic auth), `update` → `200`, `trlm` → `302` +redirecting to `/login`. A `302` is a healthy application response, not a +failure. + +**Decision** + +- Expected codes → health check passes. +- Any other code → `STOP`; a Traefik recreate may have failed, check + `docker logs traefik`. + +## Troubleshooting + +### Troubleshooting A — `407` with the correct credential + +- Evidence: `docker logs sing-box` shows the HTTP inbound rejecting the user. +- Allowed: read-only inspection of `/opt/sing-box/config.json` (mode 0600) to + confirm the username matches; compare with the client's username. +- Next: if the password itself is wrong, rotate it as a change (regenerate + on-host, update `config.json`, `docker compose up -d`), then re-run Step 4. + Do not paste the new password into Plane or the repo. + +### Troubleshooting B — tunnel establishes but no traffic flows + +- Evidence: `docker logs sing-box`, and `docker exec sing-box nslookup `. +- Allowed: read-only checks. Confirm the container resolves and reaches the + internet, and that `route.final` points at the `direct` outbound. +- Next: if DNS resolution fails, compare against the other containers on the + `proxy` network; do not add `internal: true` to the network (it removes + egress and the proxy will accept connections but never reach destinations). + +## Final Verification + +- All five steps pass. +- No credential appears in Plane, the repo, or the terminal transcript. +- Result and any deviation recorded on Plane VPS-99. + +## References + +- [hosts/us4.wsvc.info.md](../hosts/us4.wsvc.info.md) +- [proxy-client-setup.md](proxy-client-setup.md) — client-side setup and triage +- [compose/sing-box/compose.yml](../compose/sing-box/compose.yml) +- [network-change.md](network-change.md), [rollback.md](rollback.md) +- Plane VPS-99 (this proxy), VPS-98 (`:443` path audit), VPS-47 (public port + exposure target) +- Traefik TCP TLS: +- sing-box HTTP inbound: