docs: Matter packet-capture runbook v2 — hass end0 commissioner capture point, corrected AP-point scope (no wired↔wired unicast), UAP-AC-Lite model fix, SE5420 live status; fix hass interface end1→end0 (W1N-207)

2026-08-22 10:14:29 +08:00
parent 32631e2996
commit 6b298491a8
6 changed files with 328 additions and 7 deletions
@@ -17,6 +17,7 @@ runbooks start from [`_template.md`](_template.md).
 | RustDesk health check | [rustdesk-health.md](rustdesk-health.md) | read-only |
 | Matrix health check | [matrix-health.md](matrix-health.md) | read-only |
 | AdGuard Home health check | [adguard-home-health.md](adguard-home-health.md) | read-only |
+| Matter packet capture | [matter-packet-capture.md](matter-packet-capture.md) | read-only |
 | Home Assistant maintenance | [home-assistant-maintenance.md](home-assistant-maintenance.md) | change (gated) |
 | Routine Ansible operations | [ansible-operations.md](ansible-operations.md) | change (allowlisted) |
 | Linear issue → mergeable change | [issue-to-merge.md](issue-to-merge.md) | delivery |
@@ -359,7 +359,7 @@ full backup `pre-maintenance-20260813` (slug `411a4ba5`). Remaining:
 - `host info` reports `disk_life_time: 10` (boot eMMC ~10% life left) —
  monitor on each snapshot; plan disk replacement / data-disk migration.
 - **Home PPPoE IPv4 to CSG is blackholed** (`curl -4` to
-  `218.19.148.218:443` times out). `end1` IPv6 works (`curl -6
+  `218.19.148.218:443` times out). `end0` IPv6 works (`curl -6
  https://95598.csg.cn` → HTTP 200). Entry `ip_family: ipv4` still
  matches the stored option; first post-restart poll can still time out
  — reload the config entry rather than reinstalling.
@@ -0,0 +1,298 @@
+# Matter packet capture (read-only)
+
+## Purpose
+
+Capture Matter-related traffic on LAN55 (mDNS discovery + PASE/CASE commissioning +
+operational traffic) to determine whether a device is on the network, is in
+commissioning mode, and whether the commissioning handshake completes. Capture
+is read-only and changes no device or network state.
+
+## Scope
+
+- Environment: LAN55 (`hass.windy.lan`, Aqara M3, ESP32-C2 Matter bulbs,
+  phone / HA matter-server all on the 55 subnet).
+- Subject: Matter over Wi-Fi and Thread relay nodes. The Thread 802.15.4 air
+  side itself is not capturable — only IPv6 forwarding by a Thread relay such
+  as the M3 is visible.
+- Not applicable: BLE commissioning, Thread 802.15.4 frames, cross-subnet
+  multicast (66-subnet hosts cannot see the 55 subnet's mDNS — link-local
+  multicast does not cross the routed 55/66 boundary, there is no reflector).
+- Read-only: no AP/device/network config is modified; state returns to normal
+  when tcpdump exits.
+
+### Capture-point selection
+
+Matter commissioning is a two-party conversation and the commissioner
+participates in every message of it, so capturing on the commissioner host
+equals capturing the whole flow.
+
+| Capture point | Sees | Blind spot | Notes |
+|---|---|---|---|
+| **hass `end0` — commissioner side (recommended)** | The full HA-driven commissioning conversation: all mDNS queries/announcements (segment multicast) + the complete TCP 5540 PASE/CASE session | Phone-as-commissioner flows (the phone's session to the device does not pass through hass) | `core_matter_server` uses **host networking**, so tcpdump on `end0` sees the add-on's traffic directly; `/` is overlay with ~42 GB free — no 60 MB tmpfs rotation needed |
+| **UAP-AC-Lite `br0` (192.168.55.5)** | All mDNS multicast (flooded; igmp snooping off) + all wireless-client unicast + unicast to/from the AP | Wired↔wired unicast — e.g. HA↔M3 TCP 5540 while a Thread device commissions via the M3 (wired, observed) — is switched locally and never traverses the AP | AP `/tmp` is a ~60 MB tmpfs → rotating capture is **mandatory** |
+
+For the common "add device" case with HA matter-server as the commissioner,
+capture on hass `end0`. Use the AP `br0` point for wireless-device or
+phone-driven flows (a wireless client's unicast to/from its AP is only visible
+there).
+
+A third candidate point, `gw` `switch0`, is topology-dependent and is **not yet
+a documented capture point** — see
+[Conditional capture point: gw switch0](#conditional-capture-point-gw-switch0).
+
+## Ownership
+
+- Owner: personal ops (Windy)
+- Last reviewed: 2026-08-22
+- Related systems: UAP-AC-Lite AP `192.168.55.5` (br0), `core_matter_server` on
+  `hass.windy.lan` (`end0`), Aqara M3, ESP32-C2 Matter bulbs
+
+## Preconditions
+
+- SSH to the capture point:
+  - AP: `ssh zhiqiangf@192.168.55.5` (key-only, `BatchMode=yes` verified).
+  - hass: `ssh hassio@hass.windy.lan`. Non-interactive SSH does **not** source
+    `.zprofile`, so run tcpdump as `sudo -n -i tcpdump …` (verified 2026-08-22).
+- tcpdump available:
+  - AP: full 4.9.2 / libpcap 1.8.1 (verified 2026-08-22).
+  - hass: `/usr/bin/tcpdump` via `sudo -n -i` (verified 2026-08-22).
+- Trigger source ready: put the Matter device into commissioning mode, or have
+  HA/phone perform discovery/commissioning — otherwise no relevant packets.
+- AP `/tmp` is a ~60 MB tmpfs (61.3 M total, 60.4 M free): rotating capture
+  (`-C`/`-W`) is mandatory on the AP. hass `/` is overlay — rotation optional
+  but keep the habit for long captures.
+
+## Safety
+
+### Non-negotiable rules
+
+- Read-only diagnosis: no installs, config changes, or service restarts on the
+  AP, hass, devices, or network.
+- pcap files are limited to `/tmp`; pull them off and delete them afterwards
+  (mandatory on the AP; same hygiene on hass).
+- Never write captured content (including any plaintext key material) into this
+  repository or Linear.
+
+### Stop conditions
+
+- Capture point unreachable (ssh fails) → `STOP`, fix the network first.
+- tcpdump reports "Permission denied" or cannot listen → `STOP` (admin needed;
+  on hass verify `sudo -n -i` works).
+- Filter expression syntax error → `STOP`, use only expressions verified in
+  this document.
+- AP `/tmp` nearly full (rotation file count × single-file size ≈ 60 MB) →
+  `STOP` and clean old pcaps.
+
+## Procedure
+
+### Step 1 — Choose the capture point
+
+**Action**
+
+- HA matter-server is the commissioner (the "add device" case) → hass `end0`.
+- Wireless device or phone-driven flow → AP `br0`.
+
+**Expected**
+
+- The chosen point is reachable and tcpdump starts listening.
+
+**Decision**
+
+- Capture point chosen and reachable → Step 2.
+- Neither applies or the choice is unclear → `STOP` and record why.
+
+### Step 2 — Realtime observation (quick confirm traffic appears)
+
+**Action**
+
+AP:
+
+```bash
+ssh zhiqiangf@192.168.55.5 "tcpdump -ni br0 -s 0 -tt 'udp port 5353 or tcp port 5540 or tcp port 5552'"
+```
+
+hass (commissioner side):
+
+```bash
+ssh hassio@hass.windy.lan "sudo -n -i tcpdump -ni end0 -s 0 -tt 'udp port 5353 or tcp port 5540 or tcp port 5552'"
+```
+
+Keep the window open, trigger the device behavior (enter commissioning mode /
+start commissioning / send a command), `Ctrl+C` to stop.
+
+**Expected**
+
+- `_matterc._udp` / `_matter._tcp` mDNS announcements (UDP 5353, multicast
+  `224.0.0.251` / `ff02::fb`).
+- During commissioning: TCP **5540** (PASE/CASE) SYN/SYN-ACK between the device
+  IP and HA/M3.
+- If the target device's MAC is known, add `and ether host <mac>` to keep only
+  that device (see variants).
+- `5552` is not a standard Matter port; it is an observed port for the Aqara M3
+  Thread-relay node (see `docs/matter-pairing-troubleshoot.md`).
+
+**Decision**
+
+- Expected packets present → Step 3 to save evidence, or judge directly against
+  the stage table (`docs/matter-pairing-troubleshoot.md` §4).
+- No packets at all → `STOP`: fix device online / commissioning-mode first; the
+  network side is repeatedly verified healthy (see troubleshooting doc).
+- mDNS present but no 5540 → see troubleshooting doc decision tree, item 4
+  (§5).
+
+### Step 3 — Rotating capture + pull to WSL
+
+**Action** (`-C 5` = rotate every 5 MB, `-W 12` = max 12 files, ≈ 60 MB ≤ AP tmpfs)
+
+AP:
+
+```bash
+ssh zhiqiangf@192.168.55.5 "tcpdump -ni br0 -s 0 -C 5 -W 12 -w /tmp/matter.pcap 'udp port 5353 or tcp port 5540 or tcp port 5552'"
+```
+
+hass (rotation optional — overlay disk):
+
+```bash
+ssh hassio@hass.windy.lan "sudo -n -i tcpdump -ni end0 -s 0 -C 5 -W 12 -w /tmp/matter.pcap 'udp port 5353 or tcp port 5540 or tcp port 5552'"
+```
+
+Trigger the traffic, then `Ctrl+C`. Files are `/tmp/matter.pcap`,
+`/tmp/matter.pcap1`, …
+
+**Expected**
+
+- tcpdump prints capture statistics (`N packets captured`).
+- `ls -la /tmp/matter.pcap*` shows the files; total stays < 60 MB on the AP.
+
+**Verification**
+
+```bash
+ssh zhiqiangf@192.168.55.5 "ls -la /tmp/matter.pcap*"
+# or
+ssh hassio@hass.windy.lan "ls -la /tmp/matter.pcap*"
+```
+
+**Pull to WSL for analysis and clean up afterwards**
+
+```bash
+scp zhiqiangf@192.168.55.5:/tmp/matter.pcap* .
+# or
+scp hassio@hass.windy.lan:/tmp/matter.pcap* .
+
+# clean up on the capture point
+ssh zhiqiangf@192.168.55.5 "rm -f /tmp/matter.pcap*"
+ssh hassio@hass.windy.lan "sudo -n -i rm -f /tmp/matter.pcap*"
+```
+
+### Step 4 — Wireshark analysis (optional)
+
+**Action**
+
+Open the pcap in Wireshark. mDNS (UDP 5353) is plaintext and directly
+readable; Matter payloads on TCP/UDP 5540 show only the handshake by default —
+plaintext needs the dissector plus session keys (Step 5).
+
+**Expected**
+
+- `mDNS` filter shows all discovery records; `tcp.port==5540` shows the
+  commissioning handshake.
+
+### Step 5 — Decrypt Matter plaintext (optional, needs session keys)
+
+Matter payloads are encrypted (AES-CCM); mDNS plaintext contains no keys. To
+decrypt, one of:
+
+1. **Capture-side key leak with a chip tool (most common)**: the commissioner
+   (HA matter-server / chip-tool) prints or exports session keys during
+   commissioning; enter them in Wireshark → Preferences → Protocols → Matter.
+   See [matter-dissector README](https://github.com/project-chip/matter-dissector#security-features).
+2. **well-known CASE keys**: both sides compiled with
+   `MATTER_CONFIG_SECURITY_TEST_MODE` / `CASEUseKnownECDHKey`; not enabled in
+   this environment (ESP32-C2 + HA official matter-server).
+
+**Expected**
+
+- Matter dissector expands protocol headers, IM commands, and cluster content.
+
+**Stop condition (decryption)**: with no session keys or test keys obtainable,
+do not fabricate keys to force a decrypt — plaintext mDNS + TCP handshake
+still resolves most troubleshooting; for plaintext payloads, upgrade to
+exporting keys on the commissioner side, then return to this runbook.
+
+## Targeted capture variants
+
+### One device only (known MAC)
+
+```bash
+ssh zhiqiangf@192.168.55.5 "tcpdump -ni br0 -s 0 -tt 'ether host 34:98:7a:27:7f:08 and (udp port 5353 or tcp port 5540 or tcp port 5552)'"
+```
+
+MACs from `docs/matter-pairing-troubleshoot.md` §3 (working bulb
+`34:98:7a:25:a1:f0`, broken bulb `34:98:7a:27:7f:08`).
+
+### mDNS announcements only (no 5540 noise)
+
+```bash
+ssh zhiqiangf@192.168.55.5 "tcpdump -ni br0 -s 0 -tt 'udp port 5353'"
+```
+
+### Rotating capture with timestamped filename (multiple runs)
+
+```bash
+ssh zhiqiangf@192.168.55.5 "tcpdump -ni br0 -s 0 -C 5 -W 12 -w /tmp/matter-\$(date +%H%M%S).pcap 'udp port 5353 or tcp port 5540 or tcp port 5552'"
+```
+
+> The nested `$(date)` must be escaped as `\$(...)` inside the double-quoted ssh
+> command so the remote shell expands it.
+
+For the hass point, prefix the same commands with
+`ssh hassio@hass.windy.lan "sudo -n -i tcpdump -ni end0 …"`.
+
+## Conditional capture point: gw switch0
+
+**Status: NOT yet a documented capture point.** Its coverage depends on the
+live topology; verify before relying on it.
+
+- `gw` `switch0` (`eth1`–`eth3`, `192.168.55.254/24`) is LAN55's L2 aggregation
+  point only while devices plug directly into the ER-X. EdgeOS ships tcpdump,
+  and `tcpdump -ni switch0` follows Linux bridge semantics.
+- **Live state (verified 2026-08-22): the SE5420 core switch is deployed**
+  (management `192.168.66.253` reachable; TP-Link "Web Switch" on :80/:443).
+  With the flat-VLAN55 single-uplink design, same-segment traffic switches
+  locally on the SE5420 and never reaches `switch0`. In that state `switch0`
+  sees only: cross-subnet (66↔55) unicast, gateway-bound traffic, and LAN55
+  mDNS multicast (flooded) — **not a full mirror**. The full-mirror point
+  becomes the SE5420 itself, which cannot run tcpdump (port mirroring only).
+- **Before using this point**, run these read-only checks from a machine with
+  gw SSH access:
+  ```bash
+  ssh -4 zhiqiang@192.168.66.254 'show interfaces ethernet'      # which switch0 member ports have link
+  ssh -4 zhiqiang@192.168.66.254 'show ethernet-switch port all' # per-port MAC table
+  ssh -4 zhiqiang@192.168.66.254 'show mac-address-table'        # where LAN55 wired devices land
+  ```
+  - LAN55 wired devices all on a single member port → single uplink confirmed;
+    `tcpdump -ni switch0 …` is then valid with the limited coverage above.
+  - Devices spread across `eth1`–`eth3` → SE5420 not in the LAN55 path;
+    `switch0` is the full mirror point.
+  - Until one of these is confirmed, do not treat `switch0` as a capture point.
+
+## Pass criteria
+
+- Realtime capture consistently shows the target device's mDNS announcements
+  (`_matterc` / `_matter._tcp`) on the chosen point.
+- Commissioning shows the TCP 5540 handshake (SYN/SYN-ACK/ACK); on the hass
+  `end0` point this includes wired Thread-relay commissioning (HA↔M3), which
+  the AP point cannot see.
+- Saved pcap opens in Wireshark and filters by `mDNS` / `tcp.port==5540`.
+
+## References
+
+- [docs/matter-pairing-troubleshoot.md](../docs/matter-pairing-troubleshoot.md) —
+  troubleshooting decision tree, stage table, device MAC/fabric facts
+- [docs/unifi-network.md](../docs/unifi-network.md) — UniFi network/IPv6/SSID records
+- [hosts/hass.windy.lan.md](../hosts/hass.windy.lan.md) — matter-server host
+  networking + `sudo -n -i` non-interactive note
+- [hosts/gw.md](../hosts/gw.md) — DHCP `matter` reservation MAC mismatch (pending, W1N-207)
+- [matter-dissector](https://github.com/project-chip/matter-dissector) —
+  Wireshark Matter dissector (incl. decryption)
+- [Silabs: Using Wireshark to Capture Network Traffic in Matter](https://docs.silabs.com/matter/2.9.1/matter-references/matter-wireshark)