# Runbook: us4 forward proxy health check

## Purpose

Read-only health check for the HTTPS forward proxies on `us4.wsvc.info`: the
canonical `:443` path (Traefik TCP SNI → nghttpx → Squid) and the independent
`:6443` path (Traefik TCP SNI → sing-box). Confirms TLS, authentication, egress
identity, and that neither endpoint is an open proxy.

## Scope

- Applicable: `us4.wsvc.info` only — Traefik containers, the
  `/opt/nghttpx` project (`:443` path) and the `/opt/sing-box` project (`:6443`).
- Applicable: verifying that no unauthenticated or wrong-credential request is
  proxied, and that private/metadata destinations are refused.
- Not applicable: any change to Traefik, nghttpx, Squid, or sing-box
  configuration — this runbook is read-only; changes go through
  [network-change.md](network-change.md) with a change record.
- Not applicable: client-side proxy configuration and browser symptoms
  (missing credentials, incomplete page loads, TLS-to-proxy support) — use
  [proxy-client-setup.md](proxy-client-setup.md).
- Not applicable: host firewall, WireGuard, or the `:9443` legacy host path
  retirement (see Plane VPS-98).

## Ownership

- Owner: personal ops (Windy)
- Last reviewed: 2026-10-01
- Related systems: us4 (`/opt/traefik`, `/opt/nghttpx`, `/opt/sing-box`)

## Safety

### Stop conditions

- Live state contradicts the expected topology below → `STOP`, record evidence,
  change nothing in this runbook.
- Any request without a credential, or with a wrong credential, returns proxied
  content instead of `407` → `STOP` immediately: treat the endpoint as an open
  proxy, keep the evidence, and escalate before any other action.
- A private, loopback, link-local, or metadata destination is reachable through
  the proxy → `STOP` and escalate.
- An expected listener (`:6443`, `:443`, or sing-box `:8080` inside its network)
  is missing, or a container is restart-looping → `STOP`; do not "fix it in
  place" from this runbook.

## Expected topology

| Endpoint | Chain | Credential |
|---|---|---|
| `us4.wsvc.info:443` | Traefik TCP router `ng-gateway` (entrypoint `websecure`, `HostSNI(us4.wsvc.info)`, `tls=true`, resolver `letsencrypt`) → `nghttpx-proxy:3000` → `squid-backend:3128` | Squid bcrypt (`/opt/nghttpx/passwords`) |
| `us4.wsvc.info:6443` | Traefik TCP router `singbox-proxy` (entrypoint `proxyin`, `HostSNI(us4.wsvc.info)`, `tls=true`, resolver `letsencrypt`) → `sing-box:8080` | sing-box `users` in `/opt/sing-box/config.json` |
| host `:9443` | host `nghttpx.service` → host Squid `:3128` | none (legacy, firewall-blocked) |

Known limitation of the `:443` path: nghttpx is CONNECT-only and answers `400`
to absolute-URI plain-HTTP requests. The `:6443` sing-box path supports both.

## Pass criteria

1. `docker ps` shows `traefik`, `sing-box`, `nghttpx-proxy`, `squid-backend`
   running, with `sing-box` and `traefik` restart counts not increasing.
2. `docker inspect traefik --format '{{json .Config.Cmd}}'` contains
   `--entrypoints.proxyin.address=:6443`.
3. TLS on `:6443` serves a valid Let's Encrypt certificate for
   `us4.wsvc.info` (not the self-signed default certificate).
4. No credential → `407`; wrong credential → `407`; plain HTTP to `:6443` →
   TLS handshake failure.
5. Authenticated request through `:6443` returns egress IP `185.201.226.122`
   for both an HTTPS (CONNECT) target and an HTTP (absolute-URI) target.
6. Authenticated request through `:443` returns the same egress IP (CONNECT
   target only).
7. `https://us4-gate.wsvc.info` still answers (dashboard, basic auth → `401`),
   and the `update`/`trlm` sites are unaffected.

## Procedure

### Step 1 — Container and listener state

**Action**

```bash
ssh -4 windy@us4.wsvc.info
docker ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}'
docker inspect traefik --format '{{json .Config.Cmd}}'
sudo ss -ltnp | grep -E ':(6443|443|9443)\b'
```

**Expected**

`traefik` publishing `6443`, `sing-box` running with no host port, and the
`proxyin` entrypoint present in Traefik's command.

**Decision**

- All present → Step 2.
- Anything missing or restart-looping → `STOP` and record the evidence.

### Step 2 — TLS certificate

**Action**

```bash
openssl s_client -connect us4.wsvc.info:6443 -servername us4.wsvc.info </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName
```

**Expected**

`subject=CN=us4.wsvc.info`, issuer Let's Encrypt, `notAfter` in the future.

**Decision**

- Valid LE certificate → Step 3.
- Self-signed default certificate, expired certificate, or handshake failure →
  `STOP`; the certificate is issued on demand, so retry once before concluding.

### Step 3 — Negative authentication tests (run from an external client)

**Action**

```bash
# no credential
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 \
  --proxy https://us4.wsvc.info:6443 https://example.com

# wrong credential
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 \
  --proxy https://us4.wsvc.info:6443 --proxy-user 'windy:definitely-wrong' https://example.com

# plain HTTP to the TLS port
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 8 http://us4.wsvc.info:6443/
```

**Expected**

The two proxy attempts fail authentication (`407`, or a curl CONNECT abort), and
the plain-HTTP attempt fails the TLS handshake.

**Decision**

- All rejected → Step 4.
- Any of them returns proxied content → `STOP` immediately (open proxy).

### Step 4 — Positive path and egress identity

**Action** — supply the credential from its server-side source; never paste it
into Plane, the repo, or a shared transcript.

```bash
# on us4, for :6443 (CONNECT target and absolute-URI HTTP target)
PW="$(sudo cat /opt/sing-box/.proxy-password)"
curl -sS --proxy https://us4.wsvc.info:6443 --proxy-user "windy:$PW" https://ipinfo.io/ip
curl -sS --proxy https://us4.wsvc.info:6443 --proxy-user "windy:$PW" http://ipinfo.io/ip

# on us4, for the :443 path
curl -sS --proxy https://us4.wsvc.info:443 --proxy-user 'windy:<password>' https://ipinfo.io/ip
```

**Expected**

`185.201.226.122` on all successful requests.

**Decision**

- Expected egress IP → Step 5.
- `407` with the known-good credential → `STOP`; check `docker logs sing-box`
  for the inbound user list before touching configuration.

### Step 5 — Regression on existing HTTPS sites

**Action**

```bash
for h in us4-gate.wsvc.info update.wsvc.info trlm.wsvc.info; do
  printf '%s ' "$h"
  curl -sS -o /dev/null -w '%{http_code}\n' --max-time 15 "https://$h"
done
```

**Expected**

`us4-gate` → `401` (dashboard basic auth), `update` → `200`, `trlm` → `302`
redirecting to `/login`. A `302` is a healthy application response, not a
failure.

**Decision**

- Expected codes → health check passes.
- Any other code → `STOP`; a Traefik recreate may have failed, check
  `docker logs traefik`.

## Troubleshooting

### Troubleshooting A — `407` with the correct credential

- Evidence: `docker logs sing-box` shows the HTTP inbound rejecting the user.
- Allowed: read-only inspection of `/opt/sing-box/config.json` (mode 0600) to
  confirm the username matches; compare with the client's username.
- Next: if the password itself is wrong, rotate it as a change (regenerate
  on-host, update `config.json`, `docker compose up -d`), then re-run Step 4.
  Do not paste the new password into Plane or the repo.

### Troubleshooting B — tunnel establishes but no traffic flows

- Evidence: `docker logs sing-box`, and `docker exec sing-box nslookup <host>`.
- Allowed: read-only checks. Confirm the container resolves and reaches the
  internet, and that `route.final` points at the `direct` outbound.
- Next: if DNS resolution fails, compare against the other containers on the
  `proxy` network; do not add `internal: true` to the network (it removes
  egress and the proxy will accept connections but never reach destinations).

## Final Verification

- All five steps pass.
- No credential appears in Plane, the repo, or the terminal transcript.
- Result and any deviation recorded on Plane VPS-99.

## References

- [hosts/us4.wsvc.info.md](../hosts/us4.wsvc.info.md)
- [proxy-client-setup.md](proxy-client-setup.md) — client-side setup and triage
- [compose/sing-box/compose.yml](../compose/sing-box/compose.yml)
- [network-change.md](network-change.md), [rollback.md](rollback.md)
- Plane VPS-99 (this proxy), VPS-98 (`:443` path audit), VPS-47 (public port
  exposure target)
- Traefik TCP TLS: <https://doc.traefik.io/traefik/v3.4/reference/routing-configuration/tcp/tls/>
- sing-box HTTP inbound: <https://sing-box.sagernet.org/configuration/inbound/http/>