windy/vps
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(us4): sing-box :6443 代理 + 根盘回收 91%→43% (Plane VPS-99/100)

Browse Source 
新增独立 HTTPS forward proxy：Traefik 新 entrypoint proxyin(:6443) 终止 TLS，
后端 sing-box（HTTP inbound + users 认证 + direct 出站），与既有 :443
nghttpx/squid 链路并存。宿主机只新增 6443/tcp；容器不发布 host 端口。

新增 compose/sing-box/（镜像 digest 钉版）与两份 runbook：
- proxy-health.md：:443/:6443 只读健康检查，含开放代理负向测试
- proxy-client-setup.md：curl / env / 浏览器扩展配置与三类失败排查
  （407 无认证头、页面加载不全=CDN 域名漏配、urllib/wget 不支持 TLS-to-proxy）

同批根盘回收 91%→43%（约 14G）：rust/node/.go/podman 孤儿存储/apt/journal/
未用镜像/容器日志；journald 加 SystemMaxUse=200M；nghttpx+squid+traefik 加
json-file 轮转 10m×3；删除 ~/.cargo 后修好 zshenv/bashrc/profile 的失效 source。

验证：:6443 LE 证书、匿名 407、CONNECT 与绝对 URI 均返回 185.201.226.122；
us4-gate 401 / update 200 / trlm 302；8 容器全部 Up；validate-repo.sh PASS。
This commit is contained in:
windyboy
2026-09-26 10:34:09 +08:00
parent 1512dde3ec
commit 7ec29627fb
7 changed files with 574 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
compose/README.md
+1
View File
@@ -17,6 +17,7 @@ files: every secret is a `${VAR}` reference resolved from the **server-local
| `unifi` | ubnt (`/home/windy/unifi-9`) | — (待从 LAN 提取) | static file (pending) | | `unifi` | ubnt (`/home/windy/unifi-9`) | — (待从 LAN 提取) | static file (pending) |
| `wireguard` | us4 (`/opt/wireguard`) | `ansible/templates/wireguard-compose.yml.j2` | role-rendered (inventory vars) | | `wireguard` | us4 (`/opt/wireguard`) | `ansible/templates/wireguard-compose.yml.j2` | role-rendered (inventory vars) |
| `rustdesk` | hk2 (`/opt/rustdesk`) | `ansible/roles/rustdesk/templates/compose.yml.j2` | role-rendered (inventory vars) | | `rustdesk` | hk2 (`/opt/rustdesk`) | `ansible/roles/rustdesk/templates/compose.yml.j2` | role-rendered (inventory vars) |
| `sing-box` | us4 (`/opt/sing-box`) | `compose/sing-box/compose.yml` (+ `config.example.json`) | static file（参考镜像, 未接入 compose-deploy; 服务器文件为准; 凭据仅存服务器 `config.json` 0600, VPS-99） |
| `mailcow` | mx2 (`/opt/mail`) | — (mailcow update generator owns it) | excluded by design | | `mailcow` | mx2 (`/opt/mail`) | — (mailcow update generator owns it) | excluded by design |
Mechanism rule: **static** `compose/<project>/compose.yml` for declarations that Mechanism rule: **static** `compose/<project>/compose.yml` for declarations that
compose/sing-box/compose.yml
+47
View File
@@ -0,0 +1,47 @@
# sing-box HTTP forward proxy on us4.wsvc.info, published through Traefik :6443.
#
# TLS terminates in Traefik (entrypoint `proxyin`); sing-box only speaks plain
# HTTP-proxy on the private `proxy` Docker network, so it publishes NO host port.
# The credential lives in the server-side ./config.json (mode 0600) and is never
# committed here.
#
# Repo-owned copy of /opt/sing-box/compose.yml (static mechanism, not yet wired
# to compose-deploy). Change record: Plane VPS-99. Health: runbooks/proxy-health.md.
services:
sing-box:
image: ghcr.io/sagernet/sing-box:v1.14.2@sha256:afbffd294c5eb3519cc7b4587299ef189bb0a2ca2f667cb6301fdb6b9bce9558
container_name: sing-box
restart: unless-stopped
security_opt:
- no-new-privileges:true
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
command:
- run
- -c
- /etc/sing-box/config.json
volumes:
- ./config.json:/etc/sing-box/config.json:ro
networks:
- proxy
labels:
- "traefik.enable=true"
- "traefik.docker.network=proxy"
# TCP router: TLS terminates here, decrypted bytes go to sing-box.
- "traefik.tcp.routers.singbox-proxy.rule=HostSNI(`us4.wsvc.info`)"
- "traefik.tcp.routers.singbox-proxy.entrypoints=proxyin"
- "traefik.tcp.routers.singbox-proxy.tls=true"
- "traefik.tcp.routers.singbox-proxy.tls.certresolver=letsencrypt"
# TCP service: in-container HTTP proxy port.
- "traefik.tcp.routers.singbox-proxy.service=singbox-proxy"
- "traefik.tcp.services.singbox-proxy.loadbalancer.server.port=8080"
networks:
proxy:
name: proxy
external: true
compose/sing-box/config.example.json
+29
View File
@@ -0,0 +1,29 @@
{
"log": {
"level": "info",
"timestamp": true
},
"inbounds": [
{
"type": "http",
"tag": "http-proxy",
"listen": "0.0.0.0",
"listen_port": 8080,
"users": [
{
"username": "windy",
"password": "REPLACE_WITH_openssl_rand_hex_32"
}
]
}
],
"outbounds": [
{
"type": "direct",
"tag": "direct"
}
],
"route": {
"final": "direct"
}
}
hosts/us4.wsvc.info.md
+127 -2
View File
@@ -52,7 +52,8 @@ do not build its firewall allowlist from the WireGuard role alone.
| TCP `3000` | Semaphore UI direct Docker publish | Open; redundant with the Traefik route and should be removed or bound to loopback | | TCP `3000` | Semaphore UI direct Docker publish | Open; redundant with the Traefik route and should be removed or bound to loopback |
| TCP `8080` | Traefik direct Docker publish | Open; redundant with the authenticated dashboard route and should be removed or bound to loopback | | TCP `8080` | Traefik direct Docker publish | Open; redundant with the authenticated dashboard route and should be removed or bound to loopback |
| UDP `51820` | WireGuard | Required public endpoint | | UDP `51820` | WireGuard | Required public endpoint |
| TCP `9443` | Host nghttpx-to-Squid proxy | Listening but blocked by the current firewall | | TCP `6443` | Traefik `proxyin` entrypoint → `sing-box` HTTP proxy (`/opt/sing-box`) | Open (Docker publish); added 2026-10-01, VPS-99 |
| TCP `9443` | Host nghttpx-to-Squid proxy — **legacy, retirement pending** (see HTTPS forward proxy below) | Listening but blocked by the current firewall |
| UDP `123` | ntpsec | Listening but blocked by the current firewall | | UDP `123` | ntpsec | Listening but blocked by the current firewall |
PostgreSQL (`5433`/`5434`/`5435`), MariaDB (`3306`), and the host Squid TCP PostgreSQL (`5433`/`5434`/`5435`), MariaDB (`3306`), and the host Squid TCP
@@ -77,6 +78,93 @@ and reverify Docker routing. Tracking: Linear `W1N-60`.
has no global IPv6 address or IPv6 default route. Treat that as a separate has no global IPv6 address or IPv6 default route. Treat that as a separate
secondary-MX reachability issue. secondary-MX reachability issue.
## HTTPS forward proxy (verified 2026-09-26)
The canonical HTTPS forward proxy on this host is:
| Item | Value |
|------|-------|
| Endpoint | `https://us4.wsvc.info:443` (Traefik TLS, ACME-managed) |
| Chain | Traefik `:443` `HostSNI(us4.wsvc.info)` → container `nghttpx-proxy:3000` (no-TLS) → `squid-backend:3128` |
| Config source | `/opt/nghttpx/compose.yml`, `nghttpx.conf`, `squid.conf` (server files are authoritative; not yet repo-owned) |
| Credential | `/opt/nghttpx/passwords`, owner `root:proxy` (`0:13`) mode `0640`, bcrypt (`$2y$`, cost 12). Squid reads it through a read-only bind mount; the helper runs as `13:13`, so any permission change must keep group `proxy` read access. |
| Username | `windy` |
| Password | **Not stored in this repository or Plane.** The plaintext lives only in the operator's password manager; the server holds the bcrypt hash. Rotate with `htpasswd -B -C 12`, validate against `basic_ncsa_auth` before replacing the file, then `docker stop squid-backend` → write in place → `docker start squid-backend` (a rename would break the bind mount). |
| Auth behaviour | No/incorrect credentials → `407`; correct credentials → tunnel. Verified egress IP `185.201.226.122` |
| Destination policy | HTTPS-tunnel only: `Safe_ports`/`SSL_ports` = `443`; CONNECT to other ports and destinations in `127/8`, `10/8`, `172.16/12`, `192.168/16`, `169.254/16` (metadata), `100.64/10`, `224/4`, `::1`, `fc00::/7`, `fe80::/10`, `ff00::/8` return `403` `ERR_ACCESS_DENIED` |
| Limitation | Plain-HTTP (absolute-URI) proxying is **not** supported; non-CONNECT requests fail with `400`. Clients must use HTTPS URLs. |
| Credential scope | No longer shared with `us2.wsvc.info` (the two hosts had an identical `/opt/nghttpx/passwords` hash before 2026-09-26; us4 now has its own) |
Client settings (username `windy`, password from the password manager):
```text
Proxy type: HTTP/HTTPS proxy (CONNECT)
Host: us4.wsvc.info
Port: 443
Username: windy
Password: <from password manager>
```
`curl` equivalent (replace `$PROXY_PASSWORD` from the password manager; never inline it in shell history on a shared host):
```bash
curl -x https://us4.wsvc.info:443 -U "windy:$PROXY_PASSWORD" https://api.ipify.org
# expected: 185.201.226.122
```
The `9443` path (`nghttpx.service` → host `squid.service` on `127.0.0.1:3128`) is a legacy
stack: its certbot-issued certificate expired 2026-01-11, the host Squid has no
authentication at all, and no historical client traffic was found in the nghttpx access
log. It is blocked by the firewall and reachable only from the host itself. Retirement
(stop + disable both units, keep config for rollback) is pending.
Rollback material from the 2026-09-26 change: `/opt/nghttpx/backups/20260926T091907+0800`
(mode `0700`, includes the pre-change password file). Keep it protected and delete it once
the change is accepted.
### Independent `sing-box` proxy on `:6443` (added 2026-10-01, VPS-99)
A second, independent HTTPS forward proxy on `6443`. It does not touch the `:443`
chain above, and the `:443`/`:9443` stacks keep running.
| Item | Value |
|------|-------|
| Endpoint | `https://us4.wsvc.info:6443` (Traefik TLS, ACME-managed) |
| Chain | Traefik TCP router `singbox-proxy` (entrypoint `proxyin`, `HostSNI(us4.wsvc.info)`, `tls=true`, resolver `letsencrypt`) → `sing-box:8080` |
| Config source | `/opt/sing-box/compose.yml` + `config.json` (server authoritative); repo copy `compose/sing-box/compose.yml` |
| Image | `ghcr.io/sagernet/sing-box:v1.14.2@sha256:afbffd294c5eb3519cc7b4587299ef189bb0a2ca2f667cb6301fdb6b9bce9558` |
| Credential | sing-box `users` in `config.json` (mode `0600`); operator copy in `/opt/sing-box/.proxy-password`. Not in this repo or Plane. |
| Capability | Supports **both** CONNECT tunnels and absolute-URI plain-HTTP requests, unlike the `:443` path |
| Host port | The container publishes no port; `6443` is bound by Traefik |
Traefik changes live in `/opt/traefik/compose.yml`:
`--entrypoints.proxyin.address=:6443` plus the `6443:6443` publish. Changing either
requires recreating the Traefik container (`docker compose up -d`), not
`docker compose restart`.
`6443` was free before the change (nothing listening; `nginx` inactive,
`certbot`/`apache2` failed). Docker-published ports land in the firewalld `docker`
zone, not `public`, so a public-zone `--add-port=6443` is neither needed nor
effective; source restriction belongs in the `DOCKER-USER` chain or a Traefik TCP
middleware. `6443` is a public TCP port beyond the target exposure recorded in
Plane VPS-47 and still needs an explicit amendment there.
Client notes — full detail in
[runbooks/proxy-client-setup.md](../runbooks/proxy-client-setup.md):
- The client's proxy type must be **HTTPS** (TLS to the proxy). A plaintext HTTP
proxy entry reaches Traefik in cleartext and gets `404`, not a proxied request.
- Browser extensions (Zero Omega / SwitchyOmega) must have the username and
password saved in the proxy profile, and the routing conditions must include
the site's CDN hostnames (for `x.com`: `twimg.com`, `api.x.com`, `t.co`), not
only the landing domain — otherwise the page loads incompletely while the
server-side log shows no `inbound connection to` line for those hosts.
- `python3 urllib` and `wget` cannot speak TLS to a proxy and fail with
`Tunnel connection failed: 404 Not Found`.
- Client-side symptoms are diagnosed from the server with
`docker logs -f sing-box | grep 'inbound connection to'`; `no
Proxy-Authorization header` means the client sent no credentials.
## Safety ## Safety
- Private keys, preshared keys, peer configuration files, and QR codes remain - Private keys, preshared keys, peer configuration files, and QR codes remain
@@ -144,6 +232,10 @@ ansible-playbook playbooks/wireguard-harden.yml --limit wireguard \
-e '{"wireguard_harden_confirm": true}' -e '{"wireguard_harden_confirm": true}'
``` ```
Forward-proxy health for both endpoints (`:443` and `:6443`) is read-only:
follow [runbooks/proxy-health.md](../runbooks/proxy-health.md). No proxy
credential is stored in this repository.
## Verification ## Verification
- Container `wireguard` is running and interface `wg0` exists. - Container `wireguard` is running and interface `wg0` exists.
@@ -175,7 +267,8 @@ Open host-level follow-up (do not conflate these with a WireGuard outage):
1. **Disk capacity:** `/` was 90% used with about 3.4 GiB free. Docker reported 1. **Disk capacity:** `/` was 90% used with about 3.4 GiB free. Docker reported
about 2.48 GB of reclaimable images and the system journal used about 1.9 about 2.48 GB of reclaimable images and the system journal used about 1.9
GB, but do not prune or vacuum without reviewing retention and rollback GB, but do not prune or vacuum without reviewing retention and rollback
needs first. needs first. — **Resolved 2026-09-26 (91% → 43%); see *Root disk recovery*
below.**
2. **Docker exposure:** the firewalld public-zone cleanup is complete, but 2. **Docker exposure:** the firewalld public-zone cleanup is complete, but
Docker still publishes `3000` and `8080` outside the ordinary host INPUT Docker still publishes `3000` and `8080` outside the ordinary host INPUT
path. Remove those redundant Compose bindings in separate maintenance units path. Remove those redundant Compose bindings in separate maintenance units
@@ -191,3 +284,35 @@ Open host-level follow-up (do not conflate these with a WireGuard outage):
limit and uses Docker's `json-file` log driver without a per-container limit and uses Docker's `json-file` log driver without a per-container
rotation setting. Current log size was small, but limits/rotation should be rotation setting. Current log size was small, but limits/rotation should be
considered during a reviewed Compose update. considered during a reviewed Compose update.
## Root disk recovery (2026-09-26, Plane VPS-100)
`/` went from **91% used (3.0G free)** to **43% (18G free)** — about 14G
reclaimed, following [runbooks/host-disk-cleanup.md](../runbooks/host-disk-cleanup.md).
Removed: `~/.nvm` 2.6G, `~/.go` 1.9G, `~/.rustup` 1.4G, `~/.cargo` 505M,
`~/.npm` 78M, stale rootless-podman storage under `~/.local/share/containers`
1.7G, `~/sigstore` 953M, `~/softz` 910M, `~/.cache` 295M, apt archives 613M,
the container JSON logs (594M → 320K), and six unreferenced Docker images
(`postgres:17`, `postgres:18`, `traefik:v3.4`, `zadam/trilium:0.60.4`,
`linuxserver/wireguard:latest`, `traefik/whoami:latest` — all re-pullable).
Durable changes, not just deletions:
- `journalctl --vacuum-size=100M` plus drop-in
`/etc/systemd/journald.conf.d/00-disk-us4.conf` (`SystemMaxUse=200M`);
journald was restarted so the cap is effective (journal back to 83M). Without
this the journal regrows toward its default 10%-of-filesystem limit.
- Log rotation `max-size: 10m` / `max-file: 3` added to `nghttpx-proxy`,
`squid-backend` (`/opt/nghttpx/compose.yml`) and `traefik`
(`/opt/traefik/compose.yml`); both stacks were recreated and verified. This
closes the unbounded `json.log` growth for these containers.
- `~/.zshenv`, `~/.bashrc`, `~/.profile` and `~/.zshrc` had their `~/.cargo/env`
and nvm references commented out after those trees were deleted; backups are
`~/.<file>.bak-20260926-disk`. Also note `~/.go/bin/cosign` was removed and is
re-installable.
Deliberately kept: `~/.local/share/trilium-data` (88M) is a 2022 Trilium
database, not a cache. Still removable if unneeded: `/opt/zeroclaw/target`
1.4G, `/opt/trilium-linux-x64-server` 592M (2023 host install, unit disabled),
`/opt/go` 167M (2017).
runbooks/README.md
+2
View File
@@ -18,6 +18,8 @@ runbooks start from [`_template.md`](_template.md).
| Gitea health check | [gitea-health.md](gitea-health.md) | read-only | | Gitea health check | [gitea-health.md](gitea-health.md) | read-only |
| Matrix health check | [matrix-health.md](matrix-health.md) | read-only | | Matrix health check | [matrix-health.md](matrix-health.md) | read-only |
| Plane health check | [plane-health.md](plane-health.md) | read-only | | Plane health check | [plane-health.md](plane-health.md) | read-only |
| us4 forward proxy health check (`:443` + `:6443`) | [proxy-health.md](proxy-health.md) | read-only |
| us4 forward proxy client setup (curl / env / browser) | [proxy-client-setup.md](proxy-client-setup.md) | reference |
| pgdb health check | [pgdb-health.md](pgdb-health.md) | read-only | | pgdb health check | [pgdb-health.md](pgdb-health.md) | read-only |
| pgdb DB restore (pg_restore) | [pgdb-restore.md](pgdb-restore.md) | change (procedure) | | pgdb DB restore (pg_restore) | [pgdb-restore.md](pgdb-restore.md) | change (procedure) |
| pgdb image/compose update | [pgdb-update.md](pgdb-update.md) | change (gated) | | pgdb image/compose update | [pgdb-update.md](pgdb-update.md) | change (gated) |
runbooks/proxy-client-setup.md
+142
View File
@@ -0,0 +1,142 @@
# Runbook: us4 forward proxy — client configuration
## Purpose
Configure a client (curl, environment variables, browser, or a proxy-switching
extension) to use the authenticated HTTPS forward proxies on `us4.wsvc.info`,
and triage the three failures seen in practice: `407` responses, pages that load
incompletely, and clients that cannot speak TLS to a proxy.
## Scope
- Applicable: the two authenticated HTTPS forward proxies on `us4.wsvc.info`
(`:6443` sing-box, `:443` nghttpx → Squid).
- Applicable: curl, `https_proxy`/`HTTPS_PROXY` environment variables, browser
proxy-switching extensions (Zero Omega / SwitchyOmega), and OS proxy settings.
- Not applicable: server-side proxy health — use [proxy-health.md](proxy-health.md).
- Not applicable: changing proxy, Traefik, or firewall configuration — use
[network-change.md](network-change.md) with a change record.
- Not applicable: WireGuard client setup.
## Ownership
- Owner: personal ops (Windy)
- Last reviewed: 2026-09-26
- Related systems: us4 (`/opt/sing-box`, `/opt/nghttpx`, `/opt/traefik`)
## Safety
### Non-negotiable rules
- Never put the proxy credential in this repository, in Plane, or in a shell
command that lands in shell history on a shared host.
- Read it from its server-side source only:
`ssh -4 windy@us4.wsvc.info 'sudo cat /opt/sing-box/.proxy-password'`.
- Never "fix" a client problem by removing proxy authentication or by publishing
the backend port.
### Stop conditions
- The client sends no `Proxy-Authorization` and the server answers `407`: that is
the server behaving correctly. `STOP` — fix the client, do not change the server.
- A plaintext HTTP request to either port is answered by Traefik with `404`: the
client is not speaking TLS to the proxy. `STOP` and change the client's proxy
protocol; do not add a plaintext entrypoint.
- A known-good curl command from [proxy-health.md](proxy-health.md) Step 4 also
fails: `STOP` and run the health runbook — the fault is server-side.
## Endpoint facts
| | `:6443` (sing-box) | `:443` (nghttpx → Squid) |
|---|---|---|
| Proxy type to the proxy | **HTTPS / TLS** | **HTTPS / TLS** |
| Username | `windy` | `windy` |
| Credential source | `/opt/sing-box/.proxy-password` (server-side, mode 0600) | operator password manager; server holds a bcrypt hash |
| CONNECT (`https://` targets) | yes | yes |
| Absolute-URI (`http://` targets) | yes | **no** — nghttpx answers `400` |
Both endpoints terminate TLS in Traefik. The client's proxy type is therefore
HTTPS (TLS to the proxy), not "HTTP proxy on port 6443".
## Client configuration
### curl
```bash
PW=$(ssh -4 windy@us4.wsvc.info 'sudo cat /opt/sing-box/.proxy-password')
curl -sS --proxy https://us4.wsvc.info:6443 --proxy-user "windy:$PW" https://ipinfo.io/ip
# expected: 185.201.226.122
```
### Environment variables
curl prefers the **lowercase** `https_proxy` over `HTTPS_PROXY`; a pre-existing
lowercase value wins even when the uppercase one is overridden. On the WSL
client both are pre-set to the LAN proxy `http://192.168.66.99:7890`, so
overriding only `HTTPS_PROXY` silently keeps using the LAN proxy (observed:
returns `152.175.29.59` instead of us4's `185.201.226.122`). Set the lowercase
variable, or pass `--proxy` explicitly.
```bash
https_proxy="https://windy:$PW@us4.wsvc.info:6443" curl -sS https://ipinfo.io/ip
```
### Browser proxy-switching extension (Zero Omega / SwitchyOmega)
1. Proxy profile: protocol **HTTPS**, server `us4.wsvc.info`, port `6443`,
username `windy`, password from the server-side file. Save **and** click
*Apply changes* — an unapplied profile is ignored.
2. Routing conditions must cover every hostname the site needs, not only the
landing domain. For `x.com` that means `x.com`, `twitter.com`, `api.x.com`,
`t.co`, `twimg.com`, `abs.twimg.com`, `pbs.twimg.com`, `video.twimg.com`.
Missing CDN domains are the usual cause of a page that loads incompletely.
3. Set `chrome://flags/#enable-quic` to **Disabled** if individual resources
stall: QUIC is UDP and cannot traverse a CONNECT proxy.
4. Watch the server side while the browser loads the page:
```bash
ssh -4 windy@us4.wsvc.info "docker logs -f sing-box" | grep --line-buffered 'inbound connection to'
```
## Troubleshooting
### A — `407 Proxy Authentication Required`
Server log on us4: `authentication failed, no Proxy-Authorization header` — the
client connected but sent no credentials (extension profile missing the
username/password, or not applied). Add them and re-apply.
`authentication failed, incorrect password` instead means the value itself is
wrong; re-read it from `/opt/sing-box/.proxy-password`.
### B — Page loads but is incomplete
Some hostnames are not routed through the proxy (see the condition list above),
or QUIC is stalling. Confirm with the live log: if the site's CDN hosts never
appear as an `inbound connection to` line, they are not being proxied at all.
### C — Client cannot speak TLS to a proxy
`python3 urllib` and `wget` treat an `https://` proxy as a plaintext HTTP proxy.
The request reaches Traefik in cleartext and is answered `404 Not Found`
(observed: `Tunnel connection failed: 404 Not Found`). Use curl, or a client that
supports HTTPS proxies.
### D — `400 Bad Request` on the `:443` endpoint
The `:443` path is CONNECT-only. Absolute-URI plain-HTTP proxying is available
only on `:6443`.
## Final Verification
- `curl --proxy https://us4.wsvc.info:6443 --proxy-user "windy:<pw>" https://ipinfo.io/ip`
returns `185.201.226.122`.
- The browser loads the target site completely, and each expected CDN host
appears in the sing-box log as an `inbound connection to` line.
- No credential was written to the repository, Plane, or a shared transcript.
## References
- [proxy-health.md](proxy-health.md) — read-only server-side health check
- [hosts/us4.wsvc.info.md](../hosts/us4.wsvc.info.md) — endpoint and credential facts
- [network-change.md](network-change.md), [rollback.md](rollback.md)
- Plane VPS-99 (proxy deployment), VPS-100 (root disk recovery)
runbooks/proxy-health.md
+226
View File
@@ -0,0 +1,226 @@
# Runbook: us4 forward proxy health check
## Purpose
Read-only health check for the HTTPS forward proxies on `us4.wsvc.info`: the
canonical `:443` path (Traefik TCP SNI → nghttpx → Squid) and the independent
`:6443` path (Traefik TCP SNI → sing-box). Confirms TLS, authentication, egress
identity, and that neither endpoint is an open proxy.
## Scope
- Applicable: `us4.wsvc.info` only — Traefik containers, the
`/opt/nghttpx` project (`:443` path) and the `/opt/sing-box` project (`:6443`).
- Applicable: verifying that no unauthenticated or wrong-credential request is
proxied, and that private/metadata destinations are refused.
- Not applicable: any change to Traefik, nghttpx, Squid, or sing-box
configuration — this runbook is read-only; changes go through
[network-change.md](network-change.md) with a change record.
- Not applicable: client-side proxy configuration and browser symptoms
(missing credentials, incomplete page loads, TLS-to-proxy support) — use
[proxy-client-setup.md](proxy-client-setup.md).
- Not applicable: host firewall, WireGuard, or the `:9443` legacy host path
retirement (see Plane VPS-98).
## Ownership
- Owner: personal ops (Windy)
- Last reviewed: 2026-10-01
- Related systems: us4 (`/opt/traefik`, `/opt/nghttpx`, `/opt/sing-box`)
## Safety
### Stop conditions
- Live state contradicts the expected topology below → `STOP`, record evidence,
change nothing in this runbook.
- Any request without a credential, or with a wrong credential, returns proxied
content instead of `407` → `STOP` immediately: treat the endpoint as an open
proxy, keep the evidence, and escalate before any other action.
- A private, loopback, link-local, or metadata destination is reachable through
the proxy → `STOP` and escalate.
- An expected listener (`:6443`, `:443`, or sing-box `:8080` inside its network)
is missing, or a container is restart-looping → `STOP`; do not "fix it in
place" from this runbook.
## Expected topology
| Endpoint | Chain | Credential |
|---|---|---|
| `us4.wsvc.info:443` | Traefik TCP router `ng-gateway` (entrypoint `websecure`, `HostSNI(us4.wsvc.info)`, `tls=true`, resolver `letsencrypt`) → `nghttpx-proxy:3000` → `squid-backend:3128` | Squid bcrypt (`/opt/nghttpx/passwords`) |
| `us4.wsvc.info:6443` | Traefik TCP router `singbox-proxy` (entrypoint `proxyin`, `HostSNI(us4.wsvc.info)`, `tls=true`, resolver `letsencrypt`) → `sing-box:8080` | sing-box `users` in `/opt/sing-box/config.json` |
| host `:9443` | host `nghttpx.service` → host Squid `:3128` | none (legacy, firewall-blocked) |
Known limitation of the `:443` path: nghttpx is CONNECT-only and answers `400`
to absolute-URI plain-HTTP requests. The `:6443` sing-box path supports both.
## Pass criteria
1. `docker ps` shows `traefik`, `sing-box`, `nghttpx-proxy`, `squid-backend`
running, with `sing-box` and `traefik` restart counts not increasing.
2. `docker inspect traefik --format '{{json .Config.Cmd}}'` contains
`--entrypoints.proxyin.address=:6443`.
3. TLS on `:6443` serves a valid Let's Encrypt certificate for
`us4.wsvc.info` (not the self-signed default certificate).
4. No credential → `407`; wrong credential → `407`; plain HTTP to `:6443` →
TLS handshake failure.
5. Authenticated request through `:6443` returns egress IP `185.201.226.122`
for both an HTTPS (CONNECT) target and an HTTP (absolute-URI) target.
6. Authenticated request through `:443` returns the same egress IP (CONNECT
target only).
7. `https://us4-gate.wsvc.info` still answers (dashboard, basic auth → `401`),
and the `update`/`trlm` sites are unaffected.
## Procedure
### Step 1 — Container and listener state
**Action**
```bash
ssh -4 windy@us4.wsvc.info
docker ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}'
docker inspect traefik --format '{{json .Config.Cmd}}'
sudo ss -ltnp | grep -E ':(6443|443|9443)\b'
```
**Expected**
`traefik` publishing `6443`, `sing-box` running with no host port, and the
`proxyin` entrypoint present in Traefik's command.
**Decision**
- All present → Step 2.
- Anything missing or restart-looping → `STOP` and record the evidence.
### Step 2 — TLS certificate
**Action**
```bash
openssl s_client -connect us4.wsvc.info:6443 -servername us4.wsvc.info </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates -ext subjectAltName
```
**Expected**
`subject=CN=us4.wsvc.info`, issuer Let's Encrypt, `notAfter` in the future.
**Decision**
- Valid LE certificate → Step 3.
- Self-signed default certificate, expired certificate, or handshake failure →
`STOP`; the certificate is issued on demand, so retry once before concluding.
### Step 3 — Negative authentication tests (run from an external client)
**Action**
```bash
# no credential
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 \
--proxy https://us4.wsvc.info:6443 https://example.com
# wrong credential
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 \
--proxy https://us4.wsvc.info:6443 --proxy-user 'windy:definitely-wrong' https://example.com
# plain HTTP to the TLS port
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 8 http://us4.wsvc.info:6443/
```
**Expected**
The two proxy attempts fail authentication (`407`, or a curl CONNECT abort), and
the plain-HTTP attempt fails the TLS handshake.
**Decision**
- All rejected → Step 4.
- Any of them returns proxied content → `STOP` immediately (open proxy).
### Step 4 — Positive path and egress identity
**Action** — supply the credential from its server-side source; never paste it
into Plane, the repo, or a shared transcript.
```bash
# on us4, for :6443 (CONNECT target and absolute-URI HTTP target)
PW="$(sudo cat /opt/sing-box/.proxy-password)"
curl -sS --proxy https://us4.wsvc.info:6443 --proxy-user "windy:$PW" https://ipinfo.io/ip
curl -sS --proxy https://us4.wsvc.info:6443 --proxy-user "windy:$PW" http://ipinfo.io/ip
# on us4, for the :443 path
curl -sS --proxy https://us4.wsvc.info:443 --proxy-user 'windy:<password>' https://ipinfo.io/ip
```
**Expected**
`185.201.226.122` on all successful requests.
**Decision**
- Expected egress IP → Step 5.
- `407` with the known-good credential → `STOP`; check `docker logs sing-box`
for the inbound user list before touching configuration.
### Step 5 — Regression on existing HTTPS sites
**Action**
```bash
for h in us4-gate.wsvc.info update.wsvc.info trlm.wsvc.info; do
printf '%s ' "$h"
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 "https://$h"
done
```
**Expected**
`us4-gate` → `401` (dashboard basic auth), `update` → `200`, `trlm` → `302`
redirecting to `/login`. A `302` is a healthy application response, not a
failure.
**Decision**
- Expected codes → health check passes.
- Any other code → `STOP`; a Traefik recreate may have failed, check
`docker logs traefik`.
## Troubleshooting
### Troubleshooting A — `407` with the correct credential
- Evidence: `docker logs sing-box` shows the HTTP inbound rejecting the user.
- Allowed: read-only inspection of `/opt/sing-box/config.json` (mode 0600) to
confirm the username matches; compare with the client's username.
- Next: if the password itself is wrong, rotate it as a change (regenerate
on-host, update `config.json`, `docker compose up -d`), then re-run Step 4.
Do not paste the new password into Plane or the repo.
### Troubleshooting B — tunnel establishes but no traffic flows
- Evidence: `docker logs sing-box`, and `docker exec sing-box nslookup <host>`.
- Allowed: read-only checks. Confirm the container resolves and reaches the
internet, and that `route.final` points at the `direct` outbound.
- Next: if DNS resolution fails, compare against the other containers on the
`proxy` network; do not add `internal: true` to the network (it removes
egress and the proxy will accept connections but never reach destinations).
## Final Verification
- All five steps pass.
- No credential appears in Plane, the repo, or the terminal transcript.
- Result and any deviation recorded on Plane VPS-99.
## References
- [hosts/us4.wsvc.info.md](../hosts/us4.wsvc.info.md)
- [proxy-client-setup.md](proxy-client-setup.md) — client-side setup and triage
- [compose/sing-box/compose.yml](../compose/sing-box/compose.yml)
- [network-change.md](network-change.md), [rollback.md](rollback.md)
- Plane VPS-99 (this proxy), VPS-98 (`:443` path audit), VPS-47 (public port
exposure target)
- Traefik TCP TLS: <https://doc.traefik.io/traefik/v3.4/reference/routing-configuration/tcp/tls/>
- sing-box HTTP inbound: <https://sing-box.sagernet.org/configuration/inbound/http/>