vps/runbooks/proxy-health.md

Runbook: us4 forward proxy health check

Purpose

Read-only health check for the HTTPS forward proxies on us4.wsvc.info: the canonical :443 path (Traefik TCP SNI → nghttpx → Squid) and the independent :6443 path (Traefik TCP SNI → sing-box). Confirms TLS, authentication, egress identity, and that neither endpoint is an open proxy.

Scope

• Applicable: us4.wsvc.info only — Traefik containers, the /opt/nghttpx project (:443 path) and the /opt/sing-box project (:6443).
• Applicable: verifying that no unauthenticated or wrong-credential request is proxied, and that private/metadata destinations are refused.
• Not applicable: any change to Traefik, nghttpx, Squid, or sing-box configuration — this runbook is read-only; changes go through network-change.md with a change record.
• Not applicable: client-side proxy configuration and browser symptoms (missing credentials, incomplete page loads, TLS-to-proxy support) — use proxy-client-setup.md.
• Not applicable: host firewall, WireGuard, or the :9443 legacy host path retirement (see Plane VPS-98).

Ownership

• Owner: personal ops (Windy)
• Last reviewed: 2026-10-01
• Related systems: us4 (/opt/traefik, /opt/nghttpx, /opt/sing-box)

Safety

Stop conditions

• Live state contradicts the expected topology below → STOP, record evidence, change nothing in this runbook.
• Any request without a credential, or with a wrong credential, returns proxied content instead of 407 → STOP immediately: treat the endpoint as an open proxy, keep the evidence, and escalate before any other action.
• A private, loopback, link-local, or metadata destination is reachable through the proxy → STOP and escalate.
• An expected listener (:6443, :443, or sing-box :8080 inside its network) is missing, or a container is restart-looping → STOP; do not "fix it in place" from this runbook.

Expected topology

Endpoint	Chain	Credential
us4.wsvc.info:443	Traefik TCP router ng-gateway (entrypoint websecure, HostSNI(us4.wsvc.info), tls=true, resolver letsencrypt) → nghttpx-proxy:3000 → squid-backend:3128	Squid bcrypt (/opt/nghttpx/passwords)
us4.wsvc.info:6443	Traefik TCP router singbox-proxy (entrypoint proxyin, HostSNI(us4.wsvc.info), tls=true, resolver letsencrypt) → sing-box:8080	sing-box users in /opt/sing-box/config.json
host :9443	host nghttpx.service → host Squid :3128	none (legacy, firewall-blocked)

Known limitation of the :443 path: nghttpx is CONNECT-only and answers 400 to absolute-URI plain-HTTP requests. The :6443 sing-box path supports both.

Pass criteria

1. docker ps shows traefik, sing-box, nghttpx-proxy, squid-backend running, with sing-box and traefik restart counts not increasing.
2. docker inspect traefik --format '{{json .Config.Cmd}}' contains --entrypoints.proxyin.address=:6443.
3. TLS on :6443 serves a valid Let's Encrypt certificate for us4.wsvc.info (not the self-signed default certificate).
4. No credential → 407; wrong credential → 407; plain HTTP to :6443 → TLS handshake failure.
5. Authenticated request through :6443 returns egress IP 185.201.226.122 for both an HTTPS (CONNECT) target and an HTTP (absolute-URI) target.
6. Authenticated request through :443 returns the same egress IP (CONNECT target only).
7. https://us4-gate.wsvc.info still answers (dashboard, basic auth → 401), and the update/trlm sites are unaffected.

Procedure

Step 1 — Container and listener state

Action

ssh -4 windy@us4.wsvc.info
docker ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}'
docker inspect traefik --format '{{json .Config.Cmd}}'
sudo ss -ltnp | grep -E ':(6443|443|9443)\b'

Expected

traefik publishing 6443, sing-box running with no host port, and the proxyin entrypoint present in Traefik's command.

Decision

• All present → Step 2.
• Anything missing or restart-looping → STOP and record the evidence.

Step 2 — TLS certificate

Action

openssl s_client -connect us4.wsvc.info:6443 -servername us4.wsvc.info </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Expected

subject=CN=us4.wsvc.info, issuer Let's Encrypt, notAfter in the future.

Decision

• Valid LE certificate → Step 3.
• Self-signed default certificate, expired certificate, or handshake failure → STOP; the certificate is issued on demand, so retry once before concluding.

Step 3 — Negative authentication tests (run from an external client)

Action

# no credential
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 \
  --proxy https://us4.wsvc.info:6443 https://example.com

# wrong credential
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 \
  --proxy https://us4.wsvc.info:6443 --proxy-user 'windy:definitely-wrong' https://example.com

# plain HTTP to the TLS port
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 8 http://us4.wsvc.info:6443/

Expected

The two proxy attempts fail authentication (407, or a curl CONNECT abort), and the plain-HTTP attempt fails the TLS handshake.

Decision

• All rejected → Step 4.
• Any of them returns proxied content → STOP immediately (open proxy).

Step 4 — Positive path and egress identity

Action — supply the credential from its server-side source; never paste it into Plane, the repo, or a shared transcript.

# on us4, for :6443 (CONNECT target and absolute-URI HTTP target)
PW="$(sudo cat /opt/sing-box/.proxy-password)"
curl -sS --proxy https://us4.wsvc.info:6443 --proxy-user "windy:$PW" https://ipinfo.io/ip
curl -sS --proxy https://us4.wsvc.info:6443 --proxy-user "windy:$PW" http://ipinfo.io/ip

# on us4, for the :443 path
curl -sS --proxy https://us4.wsvc.info:443 --proxy-user 'windy:<password>' https://ipinfo.io/ip

Expected

185.201.226.122 on all successful requests.

Decision

• Expected egress IP → Step 5.
• 407 with the known-good credential → STOP; check docker logs sing-box for the inbound user list before touching configuration.

Step 5 — Regression on existing HTTPS sites

Action

for h in us4-gate.wsvc.info update.wsvc.info trlm.wsvc.info; do
  printf '%s ' "$h"
  curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 "https://$h"
done

Expected

us4-gate → 401 (dashboard basic auth), update → 200, trlm → 302 redirecting to /login. A 302 is a healthy application response, not a failure.

Decision

• Expected codes → health check passes.
• Any other code → STOP; a Traefik recreate may have failed, check docker logs traefik.

Troubleshooting

Troubleshooting A — 407 with the correct credential

• Evidence: docker logs sing-box shows the HTTP inbound rejecting the user.
• Allowed: read-only inspection of /opt/sing-box/config.json (mode 0600) to confirm the username matches; compare with the client's username.
• Next: if the password itself is wrong, rotate it as a change (regenerate on-host, update config.json, docker compose up -d), then re-run Step 4. Do not paste the new password into Plane or the repo.

Troubleshooting B — tunnel establishes but no traffic flows

• Evidence: docker logs sing-box, and docker exec sing-box nslookup <host>.
• Allowed: read-only checks. Confirm the container resolves and reaches the internet, and that route.final points at the direct outbound.
• Next: if DNS resolution fails, compare against the other containers on the proxy network; do not add internal: true to the network (it removes egress and the proxy will accept connections but never reach destinations).

Final Verification

• All five steps pass.
• No credential appears in Plane, the repo, or the terminal transcript.
• Result and any deviation recorded on Plane VPS-99.

References

• hosts/us4.wsvc.info.md
• proxy-client-setup.md — client-side setup and triage
• compose/sing-box/compose.yml
• network-change.md, rollback.md
• Plane VPS-99 (this proxy), VPS-98 (:443 path audit), VPS-47 (public port exposure target)
• Traefik TCP TLS: https://doc.traefik.io/traefik/v3.4/reference/routing-configuration/tcp/tls/
• sing-box HTTP inbound: https://sing-box.sagernet.org/configuration/inbound/http/