vps/runbooks/matter-packet-capture.md

# Matter packet capture (read-only)

## Purpose

Capture Matter-related traffic on LAN55 (mDNS discovery + PASE/CASE commissioning +
operational traffic) to determine whether a device is on the network, is in
commissioning mode, and whether the commissioning handshake completes. Capture
is read-only and changes no device or network state.

## Scope

- Environment: LAN55 (`hass.windy.lan`, Aqara M3, ESP32-C2 Matter bulbs,
  phone / HA matter-server all on the 55 subnet).
- Subject: Matter over Wi-Fi and Thread relay nodes. The Thread 802.15.4 air
  side itself is not capturable — only IPv6 forwarding by a Thread relay such
  as the M3 is visible.
- Not applicable: BLE commissioning, Thread 802.15.4 frames, cross-subnet
  multicast (66-subnet hosts cannot see the 55 subnet's mDNS — link-local
  multicast does not cross the routed 55/66 boundary, there is no reflector).
- Read-only: no AP/device/network config is modified; state returns to normal
  when tcpdump exits.

### Capture-point selection

Matter commissioning is a two-party conversation and the commissioner
participates in every message of it, so capturing on the commissioner host
equals capturing the whole flow.

| Capture point | Sees | Blind spot | Notes |
|---|---|---|---|
| **hass `end0` — commissioner side (recommended)** | The full HA-driven commissioning conversation: all mDNS queries/announcements (segment multicast) + the complete TCP 5540 PASE/CASE session | Phone-as-commissioner flows (the phone's session to the device does not pass through hass) | `core_matter_server` uses **host networking**, so tcpdump on `end0` sees the add-on's traffic directly; `/` is overlay with ~42 GB free — no 60 MB tmpfs rotation needed |
| **UAP-AC-Lite `br0` (192.168.55.5)** | All mDNS multicast (flooded; igmp snooping off) + all wireless-client unicast + unicast to/from the AP | Wired↔wired unicast — e.g. HA↔M3 TCP 5540 while a Thread device commissions via the M3 (wired, observed) — is switched locally and never traverses the AP | AP `/tmp` is a ~60 MB tmpfs → rotating capture is **mandatory** |

For the common "add device" case with HA matter-server as the commissioner,
capture on hass `end0`. Use the AP `br0` point for wireless-device or
phone-driven flows (a wireless client's unicast to/from its AP is only visible
there).

A third point, `gw` `switch0`, is **verified as a limited capture point**
(cross-subnet/gateway/mDNS flows only — not a full mirror of LAN55) — see
[Capture point: gw switch0](#capture-point-gw-switch0).

## Ownership

- Owner: personal ops (Windy)
- Last reviewed: 2026-08-22
- Related systems: UAP-AC-Lite AP `192.168.55.5` (br0), `core_matter_server` on
  `hass.windy.lan` (`end0`), Aqara M3, ESP32-C2 Matter bulbs

## Preconditions

- SSH to the capture point:
  - AP: `ssh zhiqiangf@192.168.55.5` (key-only, `BatchMode=yes` verified).
  - hass: `ssh hassio@hass.windy.lan`. Non-interactive SSH does **not** source
    `.zprofile`, so run tcpdump as `sudo -n -i tcpdump …` (verified 2026-08-22).
- tcpdump available:
  - AP: full 4.9.2 / libpcap 1.8.1 (verified 2026-08-22).
  - hass: `/usr/bin/tcpdump` via `sudo -n -i` (verified 2026-08-22).
- Trigger source ready: put the Matter device into commissioning mode, or have
  HA/phone perform discovery/commissioning — otherwise no relevant packets.
- AP `/tmp` is a ~60 MB tmpfs (61.3 M total, 60.4 M free): rotating capture
  (`-C`/`-W`) is mandatory on the AP. hass `/` is overlay — rotation optional
  but keep the habit for long captures.

## Safety

### Non-negotiable rules

- Read-only diagnosis: no installs, config changes, or service restarts on the
  AP, hass, devices, or network.
- pcap files are limited to `/tmp`; pull them off and delete them afterwards
  (mandatory on the AP; same hygiene on hass).
- Never write captured content (including any plaintext key material) into this
  repository or Linear.

### Stop conditions

- Capture point unreachable (ssh fails) → `STOP`, fix the network first.
- tcpdump reports "Permission denied" or cannot listen → `STOP` (admin needed;
  on hass verify `sudo -n -i` works).
- Filter expression syntax error → `STOP`, use only expressions verified in
  this document.
- AP `/tmp` nearly full (rotation file count × single-file size ≈ 60 MB) →
  `STOP` and clean old pcaps.

## Procedure

### Step 1 — Choose the capture point

**Action**

- HA matter-server is the commissioner (the "add device" case) → hass `end0`.
- Wireless device or phone-driven flow → AP `br0`.

**Expected**

- The chosen point is reachable and tcpdump starts listening.

**Decision**

- Capture point chosen and reachable → Step 2.
- Neither applies or the choice is unclear → `STOP` and record why.

### Step 2 — Realtime observation (quick confirm traffic appears)

**Action**

AP:

```bash
ssh zhiqiangf@192.168.55.5 "tcpdump -ni br0 -s 0 -tt 'udp port 5353 or tcp port 5540 or tcp port 5552'"
```

hass (commissioner side):

```bash
ssh hassio@hass.windy.lan "sudo -n -i tcpdump -ni end0 -s 0 -tt 'udp port 5353 or tcp port 5540 or tcp port 5552'"
```

Keep the window open, trigger the device behavior (enter commissioning mode /
start commissioning / send a command), `Ctrl+C` to stop.

**Expected**

- `_matterc._udp` / `_matter._tcp` mDNS announcements (UDP 5353, multicast
  `224.0.0.251` / `ff02::fb`).
- During commissioning: TCP **5540** (PASE/CASE) SYN/SYN-ACK between the device
  IP and HA/M3.
- If the target device's MAC is known, add `and ether host <mac>` to keep only
  that device (see variants).
- `5552` is not a standard Matter port; it is an observed port for the Aqara M3
  Thread-relay node (see `docs/matter-pairing-troubleshoot.md`).

**Decision**

- Expected packets present → Step 3 to save evidence, or judge directly against
  the stage table (`docs/matter-pairing-troubleshoot.md` §4).
- No packets at all → `STOP`: fix device online / commissioning-mode first; the
  network side is repeatedly verified healthy (see troubleshooting doc).
- mDNS present but no 5540 → see troubleshooting doc decision tree, item 4
  (§5).

### Step 3 — Rotating capture + pull to WSL

**Action** (`-C 5` = rotate every 5 MB, `-W 12` = max 12 files, ≈ 60 MB ≤ AP tmpfs)

AP:

```bash
ssh zhiqiangf@192.168.55.5 "tcpdump -ni br0 -s 0 -C 5 -W 12 -w /tmp/matter.pcap 'udp port 5353 or tcp port 5540 or tcp port 5552'"
```

hass (rotation optional — overlay disk):

```bash
ssh hassio@hass.windy.lan "sudo -n -i tcpdump -ni end0 -s 0 -C 5 -W 12 -w /tmp/matter.pcap 'udp port 5353 or tcp port 5540 or tcp port 5552'"
```

Trigger the traffic, then `Ctrl+C`. Files are `/tmp/matter.pcap`,
`/tmp/matter.pcap1`, …

**Expected**

- tcpdump prints capture statistics (`N packets captured`).
- `ls -la /tmp/matter.pcap*` shows the files; total stays < 60 MB on the AP.

**Verification**

```bash
ssh zhiqiangf@192.168.55.5 "ls -la /tmp/matter.pcap*"
# or
ssh hassio@hass.windy.lan "ls -la /tmp/matter.pcap*"
```

**Pull to WSL for analysis and clean up afterwards**

```bash
scp zhiqiangf@192.168.55.5:/tmp/matter.pcap* .
# or
scp hassio@hass.windy.lan:/tmp/matter.pcap* .

# clean up on the capture point
ssh zhiqiangf@192.168.55.5 "rm -f /tmp/matter.pcap*"
ssh hassio@hass.windy.lan "sudo -n -i rm -f /tmp/matter.pcap*"
```

### Step 4 — Wireshark analysis (optional)

**Action**

Open the pcap in Wireshark. mDNS (UDP 5353) is plaintext and directly
readable; Matter payloads on TCP/UDP 5540 show only the handshake by default —
plaintext needs the dissector plus session keys (Step 5).

**Expected**

- `mDNS` filter shows all discovery records; `tcp.port==5540` shows the
  commissioning handshake.

### Step 5 — Decrypt Matter plaintext (optional, needs session keys)

Matter payloads are encrypted (AES-CCM); mDNS plaintext contains no keys. To
decrypt, one of:

1. **Capture-side key leak with a chip tool (most common)**: the commissioner
   (HA matter-server / chip-tool) prints or exports session keys during
   commissioning; enter them in Wireshark → Preferences → Protocols → Matter.
   See [matter-dissector README](https://github.com/project-chip/matter-dissector#security-features).
2. **well-known CASE keys**: both sides compiled with
   `MATTER_CONFIG_SECURITY_TEST_MODE` / `CASEUseKnownECDHKey`; not enabled in
   this environment (ESP32-C2 + HA official matter-server).

**Expected**

- Matter dissector expands protocol headers, IM commands, and cluster content.

**Stop condition (decryption)**: with no session keys or test keys obtainable,
do not fabricate keys to force a decrypt — plaintext mDNS + TCP handshake
still resolves most troubleshooting; for plaintext payloads, upgrade to
exporting keys on the commissioner side, then return to this runbook.

## Targeted capture variants

### One device only (known MAC)

```bash
ssh zhiqiangf@192.168.55.5 "tcpdump -ni br0 -s 0 -tt 'ether host 34:98:7a:27:7f:08 and (udp port 5353 or tcp port 5540 or tcp port 5552)'"
```

MACs from `docs/matter-pairing-troubleshoot.md` §3 (working bulb
`34:98:7a:25:a1:f0`, broken bulb `34:98:7a:27:7f:08`).

### mDNS announcements only (no 5540 noise)

```bash
ssh zhiqiangf@192.168.55.5 "tcpdump -ni br0 -s 0 -tt 'udp port 5353'"
```

### Rotating capture with timestamped filename (multiple runs)

```bash
ssh zhiqiangf@192.168.55.5 "tcpdump -ni br0 -s 0 -C 5 -W 12 -w /tmp/matter-\$(date +%H%M%S).pcap 'udp port 5353 or tcp port 5540 or tcp port 5552'"
```

> The nested `$(date)` must be escaped as `\$(...)` inside the double-quoted ssh
> command so the remote shell expands it.

For the hass point, prefix the same commands with
`ssh hassio@hass.windy.lan "sudo -n -i tcpdump -ni end0 …"`.

## Capture point: gw switch0

**Status: verified 2026-08-22 — limited capture point; NOT a full mirror of
LAN55.**

- `gw` `switch0` (`eth1`–`eth3`, `192.168.55.254/24`) is LAN55's L2 aggregation
  only while devices plug directly into the ER-X. EdgeOS ships tcpdump;
  `tcpdump -ni switch0` follows Linux bridge semantics.
- **Live topology (verified 2026-08-22): the SE5420 core switch is deployed**
  (management `192.168.66.253` up — TP-Link OUI `f8:c9:03`, web UI on
  :80/:443) and the ER-X uplink is a **single switch0 member port**: `eth1`
  link up, `eth2`/`eth3` down. All LAN55 wired devices (hass `.11`, Aqara M3
  `.248`, SmartThings `.48`, UAP-AC-Lite `.5`) are reached via `switch0`
  behind that one uplink. Same-segment wired↔wired unicast switches locally on
  the SE5420 and never reaches `switch0`.
- **What `switch0` still sees:** cross-subnet (66↔55) unicast, traffic to/from
  the gateway itself (DHCP, DNS forwarding, port-forwards), and LAN55 mDNS
  multicast (flooded up the uplink). Use it only for those flows; for a full
  commissioning conversation use the hass `end0` or AP `br0` point instead.
- **Full mirror:** only via SE5420 port mirroring (the switch cannot run
  tcpdump). Not configured; out of scope here.
- **Verification commands (EdgeOS v3.0.1 build 5862409):**
  - Interactive: `ssh ubnt@192.168.66.254` (or `zhiqiang`), then
    `show interfaces ethernet` — port link states are the decisive check
    (`eth1` up + `eth2`/`eth3` down = single uplink). `configure` (config
    mode) also accepts `show ...`.
  - Non-interactive (agent/script): `show`/`configure` are interactive-only
    aliases on this build; use the op wrapper:
    ```bash
    ssh ubnt@192.168.66.254 '/opt/vyatta/bin/vyatta-op-cmd-wrapper show interfaces ethernet'
    ```
  - `show ethernet-switch port all` and `show mac-address-table` are NOT
    available on this build; the switch FDB is hardware-offloaded
    (`brctl showmacs switch0` → "Operation not supported"). Port link state
    + ARP (`show arp`) are the reliable checks.
  - SE5420 liveness: `ping 192.168.66.253` and `:80/:443`.
- Sample capture at this point (cross-segment/gateway/mDNS flows only;
  tcpdump needs root — `zhiqiang` has passwordless sudo):
  ```bash
  ssh zhiqiang@192.168.66.254 "sudo -n tcpdump -ni switch0 -s 0 'udp port 5353 or tcp port 5540 or tcp port 5552'"
  ```

## Pass criteria

- Realtime capture consistently shows the target device's mDNS announcements
  (`_matterc` / `_matter._tcp`) on the chosen point.
- Commissioning shows the TCP 5540 handshake (SYN/SYN-ACK/ACK); on the hass
  `end0` point this includes wired Thread-relay commissioning (HA↔M3), which
  the AP point cannot see.
- Saved pcap opens in Wireshark and filters by `mDNS` / `tcp.port==5540`.

## References

- [docs/matter-pairing-troubleshoot.md](../docs/matter-pairing-troubleshoot.md) —
  troubleshooting decision tree, stage table, device MAC/fabric facts
- [docs/unifi-network.md](../docs/unifi-network.md) — UniFi network/IPv6/SSID records
- [hosts/hass.windy.lan.md](../hosts/hass.windy.lan.md) — matter-server host
  networking + `sudo -n -i` non-interactive note
- [hosts/gw.md](../hosts/gw.md) — DHCP `matter` reservation MAC mismatch (pending, W1N-207)
- [matter-dissector](https://github.com/project-chip/matter-dissector) —
  Wireshark Matter dissector (incl. decryption)
- [Silabs: Using Wireshark to Capture Network Traffic in Matter](https://docs.silabs.com/matter/2.9.1/matter-references/matter-wireshark)