docs: onboard AdGuard Home to Ansible; retire check-mx2; migrate plan to Linear

- Add dns.windy.lan to Ansible inventory (adguardhome group/profile) with a
  read-only healthcheck template, systemd timer, and on-demand report parity
  with the other active hosts; verified green on the host.
- Add synapse.chans.xyz to the AGENTS.md active-host quick map.
- Remove plans/; migrate decisions/verification to a Linear vps project doc and
  file the remaining gaps as W1N-46..49 with blocking edges.
- Retire scripts/check-mx2.sh; point mailcow health/update runbooks and the
  mx2 host file at the Ansible health report.
- Trim us2 stub software to verified running services; note dormant dirs.
- Normalize repo docs to English; drop the non-existent CONTEXT.md/ADR claim
  and remove docs/agents/domain.md.
This commit is contained in:
windyboy
2026-08-06 15:48:48 +08:00
parent 29bdf06383
commit b03d7019c4
16 changed files with 484 additions and 280 deletions
+14 -6
View File
@@ -27,6 +27,8 @@ Also readable as `agent.md` (symlink → this file).
| **mx2.windy.me** | mailcow (`/opt/mail`, project `cow`) | `ssh -4 windy@mx2.windy.me` | [hosts/mx2.windy.me.md](hosts/mx2.windy.me.md) | | **mx2.windy.me** | mailcow (`/opt/mail`, project `cow`) | `ssh -4 windy@mx2.windy.me` | [hosts/mx2.windy.me.md](hosts/mx2.windy.me.md) |
| **us2.wsvc.info** | Vaultwarden + Traefik (+ Soft Serve, …) | `ssh -4 windy@us2.wsvc.info` | [hosts/us2.wsvc.info.md](hosts/us2.wsvc.info.md) | | **us2.wsvc.info** | Vaultwarden + Traefik (+ Soft Serve, …) | `ssh -4 windy@us2.wsvc.info` | [hosts/us2.wsvc.info.md](hosts/us2.wsvc.info.md) |
| **hk2.chans.xyz** | PowerDNS auth ns1 (`/opt/pdns`) | `ssh -4 windy@hk2.chans.xyz` | [hosts/hk2.chans.xyz.md](hosts/hk2.chans.xyz.md) | | **hk2.chans.xyz** | PowerDNS auth ns1 (`/opt/pdns`) | `ssh -4 windy@hk2.chans.xyz` | [hosts/hk2.chans.xyz.md](hosts/hk2.chans.xyz.md) |
| **synapse.chans.xyz** | Matrix ESS (Synapse + MAS + Element) on K3s | `ssh -4 windy@synapse.chans.xyz` | [hosts/synapse.chans.xyz.md](hosts/synapse.chans.xyz.md) |
| **dns.windy.lan** | AdGuard Home LAN DNS + Mihomo explicit proxy | `ssh -4 windy@192.168.66.36` | [hosts/dns.windy.lan.md](hosts/dns.windy.lan.md) |
| **gfw.windy.lan** | OpenWrt LAN gateway / OpenClash | `ssh -4 root@192.168.66.1` | [hosts/gfw.windy.lan.md](hosts/gfw.windy.lan.md) | | **gfw.windy.lan** | OpenWrt LAN gateway / OpenClash | `ssh -4 root@192.168.66.1` | [hosts/gfw.windy.lan.md](hosts/gfw.windy.lan.md) |
| **gw** | EdgeRouter X primary LAN gateway | `ssh -4 zhiqiang@192.168.66.254` | [hosts/gw.md](hosts/gw.md) | | **gw** | EdgeRouter X primary LAN gateway | `ssh -4 zhiqiang@192.168.66.254` | [hosts/gw.md](hosts/gw.md) |
| **ubnt** | UniFi Network Controller | `ssh -4 windy@192.168.66.46` | [hosts/ubnt.md](hosts/ubnt.md) | | **ubnt** | UniFi Network Controller | `ssh -4 windy@192.168.66.46` | [hosts/ubnt.md](hosts/ubnt.md) |
@@ -60,7 +62,7 @@ Also readable as `agent.md` (symlink → this file).
| Task | Path | | Task | Path |
|------|------| |------|------|
| mailcow health | [runbooks/mailcow-health.md](runbooks/mailcow-health.md) · `./scripts/check-mx2.sh` | | mailcow health | [runbooks/mailcow-health.md](runbooks/mailcow-health.md) |
| mailcow update | [runbooks/mailcow-update.md](runbooks/mailcow-update.md) | | mailcow update | [runbooks/mailcow-update.md](runbooks/mailcow-update.md) |
| mailcow SMTP/IMAP client | [runbooks/mailcow-smtp-client.md](runbooks/mailcow-smtp-client.md) | | mailcow SMTP/IMAP client | [runbooks/mailcow-smtp-client.md](runbooks/mailcow-smtp-client.md) |
| Vaultwarden health | [runbooks/vaultwarden-health.md](runbooks/vaultwarden-health.md) | | Vaultwarden health | [runbooks/vaultwarden-health.md](runbooks/vaultwarden-health.md) |
@@ -69,16 +71,23 @@ Also readable as `agent.md` (symlink → this file).
| PowerDNS health (hk2) | [runbooks/pdns-health.md](runbooks/pdns-health.md) | | PowerDNS health (hk2) | [runbooks/pdns-health.md](runbooks/pdns-health.md) |
| PowerDNS upstream refs | [docs/pdns-upstream.md](docs/pdns-upstream.md) | | PowerDNS upstream refs | [docs/pdns-upstream.md](docs/pdns-upstream.md) |
| Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) | | Matrix health | [runbooks/matrix-health.md](runbooks/matrix-health.md) |
| AdGuard Home health | [runbooks/adguard-home-health.md](runbooks/adguard-home-health.md) |
| Matrix upstream refs | [docs/matrix-upstream.md](docs/matrix-upstream.md) | | Matrix upstream refs | [docs/matrix-upstream.md](docs/matrix-upstream.md) |
| Hermes Agent Matrix channel | [docs/hermes-matrix.md](docs/hermes-matrix.md) | | Hermes Agent Matrix channel | [docs/hermes-matrix.md](docs/hermes-matrix.md) |
| UniFi local-service proxy bypass | [docs/unifi-openclash-localhost.md](docs/unifi-openclash-localhost.md) | | UniFi local-service proxy bypass | [docs/unifi-openclash-localhost.md](docs/unifi-openclash-localhost.md) |
| Routine Ansible operations | [runbooks/ansible-operations.md](runbooks/ansible-operations.md) | | Routine Ansible operations | [runbooks/ansible-operations.md](runbooks/ansible-operations.md) |
```bash Routine mailcow health: `cd ansible && ansible-playbook playbooks/health-report.yml --limit mailcow`. The local stub resolver is flaky; DNS probes use `1.1.1.1` / `8.8.8.8`.
./scripts/check-mx2.sh
```
`check-mx2.sh` defaults to IPv4; set `MX2_ALLOW_IPV6=1` to allow IPv6. DNS checks use `1.1.1.1` / `8.8.8.8` (local stub is flaky). ## Agent skills
### Issue tracker
Issues are tracked in Linear and created/updated via the Linear MCP (`vps` project). See `docs/agents/issue-tracker.md`.
### Triage labels
Default triage labels: needs-triage, needs-info, ready-for-agent, ready-for-human, wontfix. See `docs/agents/triage-labels.md`.
## Safety ## Safety
@@ -116,5 +125,4 @@ inventory/hosts.md # machine index
hosts/ # per-host facts hosts/ # per-host facts
runbooks/ # step-by-step ops runbooks/ # step-by-step ops
docs/ # upstream doc indexes / design notes docs/ # upstream doc indexes / design notes
scripts/ # read-only helper scripts
``` ```
+13
View File
@@ -56,6 +56,15 @@ all:
service_reconcile_services: service_reconcile_services:
wireguard: wireguard:
compose_args: [--no-deps, --force-recreate] compose_args: [--no-deps, --force-recreate]
dns_windy_lan:
ansible_host: 192.168.66.36
ansible_host_ipv4: 192.168.66.36
service_role: adguardhome
compose_project_dir: /opt/adguardhome
healthcheck_profile: adguardhome
service_reconcile_services:
adguardhome:
compose_args: [--no-deps, --force-recreate]
mailcow: mailcow:
hosts: hosts:
mx2: mx2:
@@ -68,12 +77,16 @@ all:
wireguard: wireguard:
hosts: hosts:
us4: us4:
adguardhome:
hosts:
dns_windy_lan:
docker_hosts: docker_hosts:
children: children:
mailcow: mailcow:
vaultwarden: vaultwarden:
powerdns: powerdns:
wireguard: wireguard:
adguardhome:
# Matrix is a dedicated K3s node and intentionally remains outside the # Matrix is a dedicated K3s node and intentionally remains outside the
# Docker-oriented managed group. # Docker-oriented managed group.
k3s_servers: k3s_servers:
@@ -12,3 +12,4 @@ healthcheck_profiles:
vaultwarden: vaultwarden.sh vaultwarden: vaultwarden.sh
pdns: pdns.sh pdns: pdns.sh
wireguard: wireguard.sh wireguard: wireguard.sh
adguardhome: adguardhome.sh
@@ -0,0 +1,75 @@
#!/usr/bin/env bash
set -uo pipefail
source '{{ healthcheck_install_root }}/health-common.sh'
require_command docker
require_command curl
require_command dig
require_command ss
require_command df
check_compose
container='adguardhome'
if ! docker inspect "$container" >/dev/null 2>&1; then
record critical 'adguardhome_container_missing'
else
running="$(docker inspect "$container" --format '{{ '{{' }}.State.Running{{ '}}' }}' 2>/dev/null)"
[[ "$running" == true ]] || record critical 'adguardhome_container_not_running'
fi
config_check="$(docker exec "$container" /opt/adguardhome/AdGuardHome --check-config \
-c /opt/adguardhome/conf/AdGuardHome.yaml 2>&1 || true)"
if grep -qiE 'error|invalid|fatal' <<<"$config_check"; then
record critical 'adguardhome_config_invalid'
else
record ok 'adguardhome_config_valid'
fi
if ss -H -ltn 2>/dev/null | awk '{print $4}' | grep -Eq '(^|:)53$'; then
record ok 'dns_tcp_53_listening'
else
record critical 'dns_tcp_53_missing'
fi
if ss -H -uln 2>/dev/null | awk '{print $4}' | grep -Eq '(^|:)53$'; then
record ok 'dns_udp_53_listening'
else
record critical 'dns_udp_53_missing'
fi
if ss -H -ltn 2>/dev/null | awk '{print $4}' | grep -Eq '(^|:)80$'; then
record ok 'ui_http_80_listening'
else
record critical 'ui_http_80_missing'
fi
dns_a="$(dig +time=5 +tries=1 @192.168.66.36 example.com A +noall +answer 2>/dev/null)"
[[ -n "$dns_a" ]] && record ok 'dns_example_a' || record critical 'dns_example_a_failed'
dns_aaaa="$(dig +time=5 +tries=1 @192.168.66.36 example.com AAAA +noall +answer 2>/dev/null)"
[[ -n "$dns_aaaa" ]] && record ok 'dns_example_aaaa' || record warning 'dns_example_aaaa_none'
dns_local="$(dig +time=5 +tries=1 @192.168.66.36 hass.windy.lan A +noall +answer 2>/dev/null)"
[[ -n "$dns_local" ]] && record ok 'dns_local_name' || record warning 'dns_local_name_unresolved'
ui_code="$(curl --silent --show-error --output /dev/null --write-out '%{http_code}' --max-time 10 http://192.168.66.36/ 2>/dev/null || true)"
case "$ui_code" in
200|301|302) record ok "ui_http_${ui_code}" ;;
'') record critical 'ui_unreachable' ;;
*) record critical "ui_http_${ui_code}" ;;
esac
usage="$(df -P /opt/adguardhome/work 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%')"
if [[ "$usage" =~ ^[0-9]+$ ]]; then
if (( usage >= 90 )); then record critical 'disk_usage_high'
elif (( usage >= 80 )); then record warning 'disk_usage_elevated'
else record ok 'disk_usage_ok'; fi
else
record unknown 'disk_usage_unknown'
fi
logs="$(docker logs --since 30m "$container" 2>&1 || true)"
if grep -qiE '\[(error|warn|fatal)\]|level=(error|warn|fatal)' <<<"$logs"; then
record warning 'adguardhome_log_errors'
else
record ok 'adguardhome_logs_clean'
fi
emit_result
exit "$EXIT_CODE"
@@ -0,0 +1,164 @@
# AdGuard Home official configuration review basis (2026-08)
This page is the **official source basis and recommendation baseline** used
during the on-site review of AdGuard Home (AGH) on `192.168.66.36`. It does not
claim to have read that host's configuration and contains no credentials.
On-site conclusions must be based on the live `AdGuardHome.yaml`, OpenWrt
`dnsmasq`/DHCP, firewall, routing, and actual client leases.
Official sources (AdGuard only):
- [Configuration](https://adguard-dns.io/kb/adguard-home/configuration/)
- [Setting up AdGuard Home securely](https://adguard-dns.io/kb/adguard-home/running-securely/)
- [Getting started](https://adguard-dns.io/kb/adguard-home/getting-started/)
- [DHCP](https://adguard-dns.io/kb/adguard-home/dhcp/)
- [DNS encryption](https://adguard-dns.io/kb/adguard-home/encryption/)
## Recommended role split (for the existing OpenWrt gateway)
Preferred: keep **OpenWrt/dnsmasq as the single DHCP and local-domain
authority, with AGH as the LAN DNS filter/forwarder**: DHCP option 6 hands
clients `.36`; AGH sets the router's local DNS (usually the gateway LAN IP) as
`local_ptr_upstreams` and, where needed, uses domain-specific upstreams to route
the LAN search domain / local names back to dnsmasq. Clients are filtered while
still keeping DHCP lease names, static hosts, local reverse resolution, and
gateway-internal service discovery.
Rationale: AGH supports dnsmasq-style domain-specific upstreams for internal
private name servers; private PTR should be configured in **Private reverse DNS
servers** / `local_ptr_upstreams` rather than regular upstreams; when not
explicitly configured, it falls back to the OS default resolver, which shifts as
host networking changes. [Official config: domain-specific upstreams](https://adguard-dns.io/kb/adguard-home/configuration/#configuring-upstreams);
[private rDNS](https://adguard-dns.io/kb/adguard-home/configuration/#private-addresses).
Do not have OpenWrt dnsmasq and AGH both listen on `:53` at the same LAN IP;
port conflicts, or pointing dnsmasq's upstream at AGH while AGH points back at
dnsmasq, cause failures/forwarding loops. When AGH runs on a separate `.36`
host, the simplest topology is clients → AGH `.36:53` → public/encrypted
upstreams, forwarding only local resolution (PTR, LAN domain) → gateway dnsmasq.
This is an operational inference based on the official capabilities above and
must be validated against the live `ss -lntup`, dnsmasq config, and query path.
AGH also has built-in DHCP, but it should only be enabled after **disabling the
existing gateway DHCP, explicitly migrating leases/reservations and verifying
options 3/6**. The official docs describe it as an alternative when the router
cannot hand out custom DNS, not as a dual-DHCP setup; the DHCP docs also state
leases/static leases are managed by AGH. [Getting started: router DHCP/DNS
handout and alternative DHCP](https://adguard-dns.io/kb/adguard-home/getting-started/#router);
[DHCP configuration](https://adguard-dns.io/kb/adguard-home/dhcp/).
## Exposure and access control
1. `dns.bind_hosts` should bind only loopback and the actual LAN address (here
prefer `.36`'s LAN address); do not use `0.0.0.0`/public interfaces for a
home recursive resolver. The official docs note full-interface binding
exposes DDoS risk; a small isolated network should pick local service
interfaces and may add loopback alongside. [Secure deployment: server
addresses](https://adguard-dns.io/kb/adguard-home/running-securely/#choosing-server-addresses);
[`bind_hosts` definition](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
2. In the firewall, allow only managed LAN/VLAN → `.36` TCP/UDP 53; bind the web
admin UI to the management network/loopback and avoid WAN forwarding. At the
app layer, further restrict with `allowed_clients` to the local subnets; when
non-empty, only the CIDR/IP/ClientIDs in it are accepted, and it overrides
the effect of `disallowed_clients`. [Official access
control](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
3. Keep `refuse_any: true` and a non-zero `ratelimit` (default 20 qps), unless
on-site evidence shows the service can never be reached from the internet and
LAN high-concurrency needs are data-backed; the official docs list both as
protection against reflection/DDoS. [Configuration
notes](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
4. The web UI must have a user with login-failure lockout enabled;
`users: []` disables authentication, and `auth_attempts` or `block_auth_min`
of 0 disables the corresponding lockout logic. [Web/auth
parameters](https://adguard-dns.io/kb/adguard-home/configuration/#users).
## Upstreams, encryption, and DNSSEC
- Use two trusted, differently-networked DoH/DoT/DoQ upstreams for normal public
resolution; AGH supports these protocols and DNS Stamps. If an upstream
URL/hostname needs resolving, `bootstrap_dns` is its initial resolver; do not
point bootstrap at a name that requires AGH itself to resolve, and use
directly reachable, stable IPs. A `fallback_dns` can be set separately; it is
used only when primary upstreams do not respond. [Upstream protocols and
bootstrap/fallback](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
- If local IPv6's actual egress path is unstable, do not enable
`bootstrap_prefer_ipv6`; the official docs note it makes DoH/DoQ/DoT bootstrap
prefer IPv6. Decide on-site based on each upstream's IPv4/IPv6 reachability,
latency, and failure rate. [Same reference](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
- Consider enabling `enable_dnssec`, choosing recursive upstreams that actually
handle DNSSEC; but understand precisely: officially AGH sets the DO flag on
requests to upstreams; it does not perform recursive validation itself. On
site, query a known-bad-signature domain and observe AD/failure responses to
confirm upstream validation behavior. [DNSSEC parameter
definition](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
- Prefer `upstream_mode: load_balance` unless measurement shows `parallel` first
answer latency is clearly worth the privacy/traffic cost of querying all
upstreams. The former picks by failure rate and average latency weighting; the
latter queries all upstreams concurrently; this is an operational judgment
based on documented behavior. [Upstream
modes](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
## Cache, ECS, and privacy
- Keep DNS cache enabled; use `cache_ttl_min`/`cache_ttl_max` overrides
cautiously to avoid stale or unnecessarily short TTLs. `cache_optimistic`
answers with a 10-second TTL after expiry and attempts refresh — good for
low-latency scenarios that tolerate briefly stale results, not to be enabled
blindly on internal domains sensitive to immediate changes. [Cache
parameters](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
- ECS is off by default: when enabled, AGH adds EDNS Client Subnet to upstream
requests; if CDN geo-scheduling is truly needed, use `use_custom: true` with a
coarse, non-real-client-subnet `custom_ip`, then compare hit rate vs. privacy
impact. [ECS parameters](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
- Review log retention and who can read it. If device identity is not needed,
consider `anonymize_client_ip: true`; this anonymizes client addresses in logs
and stats. [Anonymization
parameter](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
## Proxy / OpenClash coexistence
If OpenClash/transparent proxy redirects DNS, validate hop by hop to ensure:
client DNS requests reach AGH uniquely; AGH's outbound traffic to public
encrypted upstreams is not hijacked back to AGH; and the router's local dnsmasq
PTR/local-domain requests return to dnsmasq. Create explicit proxy/hijack
exceptions for `.36` and its upstreams, or explicitly design the proxy to take
over AGH's outbound resolution; do not rely on "it seems to resolve" to conclude
there is no loop.
This is not an AGH-official OpenClash instruction (the official docs do not
maintain that integration), but an on-site verification requirement derived from
AGH's independent listening, upstream forwarding, and local-PTR-split
capabilities. The officially supported mechanisms are referenced at
[`bind_hosts`/upstreams/PTR](https://adguard-dns.io/kb/adguard-home/configuration/#dns).
If serving DoH via a reverse proxy, `trusted_proxies` must list only the actual
proxy IP/CIDR; AGH will only trust `X-Real-IP`, `X-Forwarded-For`, etc. from
those sources. Do not write the whole LAN/WAN for convenience;
`http.doh.insecure_enabled` should only be enabled when a reverse proxy really
terminates TLS in front. [Proxy and
DoH](https://adguard-dns.io/kb/adguard-home/configuration/#dns); [encrypted DNS
docs](https://adguard-dns.io/kb/adguard-home/encryption/). For a pure-LAN
plaintext-53 service, publishing DoH/DoT/DoQ outward is usually unnecessary.
## On-site review checklist (read-only)
1. Record the version, service method, and the full effective configuration
(secrets redacted); run the AGH config check. Officially provided via
`--check-config`. [Command-line
arguments](https://adguard-dns.io/kb/adguard-home/configuration/#command-line-arguments).
2. On `.36`, the gateway, and a wired/wireless client, record `ss` listeners,
DHCP option 6, default route, `/etc/config/dhcp`, and OpenClash
DNS/firewall-redirect rules.
3. From a client test: public A/AAAA, ad-rule hits, DHCP hostnames, forward LAN
domain, private PTR, DNSSEC bad-signature domain, and the expected failure
mode when AGH stops. Then check whether client IPs in AGH query logs are
still genuinely identifiable.
4. From the AGH host, test each upstream and bootstrap over the actual protocol
for IPv4 and IPv6 reachability; compare with transparent proxy on/off to rule
out loops and leaks.
5. Verify the web UI/DNS ports have no WAN exposure; confirm allowed clients and
firewall match actual LAN/VLAN, guest network, and VPN subnets.
Before changing anything, export the current configuration and DHCP/firewall
state; adjust one layer at a time (DHCP handout, AGH listening, upstreams, or
DNS hijack) and keep a known-good management and DNS fallback path.
+47
View File
@@ -0,0 +1,47 @@
# Issue tracker: Linear
Issues and PRDs for this repo are tracked in Linear, driven entirely through the
Linear MCP configured in the current runtime. VPS operation records use the
Linear **`vps`** project (team `W1ndy`).
## Conventions
- **Create**: `linear_save_issue` (requires `title` + `team`); description as
real-newline Markdown.
- **Read**: `linear_get_issue <identifier>` (e.g. LIN-123), optionally with
`includeRelations`.
- **List**: `linear_list_issues`, filtered by `query` / `state` / `assignee` /
`team` / `project`.
- **Comment**: `linear_save_comment` (pass `issueId` + `body`).
- **Labels**: `linear_save_issue` `labels` field (replaces the whole label set).
- **Status**: `linear_save_issue` `state` field (type / name / ID).
Resolve team, project, and cycle names through the Linear MCP rather than
hard-coding them.
## Cross-issue constraint baseline
The `plans/` directory is retired (removed 2026-08-06). Plans and baselines now
live in the vps project document "VPS 运维重构:已确认决策与验收标准". Open work
items are tracked as individual issues with dependencies expressed via
`blockedBy` / `blocks`. Read that document before adding new tasks.
## When a skill says "publish to the issue tracker"
Create a Linear issue with `linear_save_issue` (`project: vps`).
## When a skill says "fetch the relevant ticket"
Use `linear_get_issue <identifier>`.
## Wayfinding operations
- **Map**: one issue as the map; child tickets link via `parentId`.
- **Child ticket**: `linear_save_issue` with `parentId` set to the map; labels
`wayfinder:<type>` (research / prototype / grilling / task).
- **Blocking**: `linear_save_issue` `blockedBy` / `blocks` (append-only).
- **Frontier query**: list the map's open children, excluding those blocked or
already assigned; take the first in map order.
- **Claim**: `linear_save_issue` `assignee: me`.
- **Resolve**: `linear_save_comment` the answer → update `state` → append a
pointer under the map's decisions-so-far.
+15
View File
@@ -0,0 +1,15 @@
# Triage Labels
The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker.
| Label in mattpocock/skills | Label in our tracker | Meaning |
| -------------------------- | -------------------- | ---------------------------------------- |
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
| `needs-info` | `needs-info` | Waiting on reporter for more information |
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
| `ready-for-human` | `ready-for-human` | Requires human implementation |
| `wontfix` | `wontfix` | Will not be actioned |
When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
Edit the right-hand column to match whatever vocabulary you actually use.
+75
View File
@@ -0,0 +1,75 @@
# dns.windy.lan — AdGuard Home LAN DNS
## Role and access
| Item | Value |
|---|---|
| IPv4 | `192.168.66.36` |
| SSH | `ssh -4 windy@192.168.66.36` |
| OS | Debian 12 |
| Roles | AdGuard Home LAN DNS; Mihomo explicit-proxy service |
Use `windy` for routine inspection. Do not store credentials, AdGuard Home
user hashes, proxy subscriptions, or query-log contents in this repository.
## AdGuard Home
AdGuard Home runs as the root-owned `adguardhome` Docker container in host
network mode. The persistent deployment is `/opt/adguardhome`:
| Item | Path / setting |
|---|---|
| Compose file | `/opt/adguardhome/docker-compose.yml` |
| Configuration | `/opt/adguardhome/conf/AdGuardHome.yaml` |
| Mutable data and query logs | `/opt/adguardhome/work` |
| DNS | TCP/UDP `53`; intentionally no source ACL so both LAN55 and LAN66 work |
| HTTP UI | `192.168.66.36:80`; LAN66 only |
| Image policy | Pin the reviewed image digest; do not use `latest` for a rebuild |
The EdgeRouter DHCP service advertises this host as DNS for LAN55 and LAN66.
Do not enable AdGuard Home DHCP unless the existing EdgeRouter DHCP service is
explicitly migrated and disabled first.
`agh-ui-access.service` loads `/etc/nftables-agh-ui-access.nft`. It permits
only `192.168.66.0/24` to TCP/80 and drops other TCP/80 input. It deliberately
does **not** restrict DNS, SSH, Docker, or Mihomo ports. Keep it isolated from
Docker-managed nftables tables.
Current query-log policy is 14 days with anonymized client IPs. Check free
space before increasing retention. DNSSEC is disabled because the selected
upstream path did not pass the known-bad-signature validation check; do not
enable it without re-testing validated upstreams.
The compatible names `hass.windy.lan` and legacy `hass.local` currently point
to the same Home Assistant address. Migrate clients to `hass.windy.lan`; keep
the legacy rewrite until its planned retirement.
## Mihomo and routing boundary
Mihomo listens for explicit proxy traffic on this host. Read-only checks found
no host TUN policy routing or local nftables/iptables transparent-redirection
rules. A Mihomo listener alone does not mean it intercepts AdGuard Home
traffic. The default gateway (`gfw.windy.lan`) can still affect outbound
traffic, so inspect its OpenClash rules separately before changing upstream
routing.
## Safety
- Back up `/opt/adguardhome/conf/AdGuardHome.yaml` and `docker-compose.yml`
before changing either, then run the official `--check-config` command.
- Recreate only the AdGuard Home container and verify DNS A/AAAA, local names,
the UI from LAN66, and the UI denial from LAN55 afterwards.
- Private PTR currently has no local authoritative source: direct EdgeRouter
PTR queries return NXDOMAIN. Do not point `local_ptr_upstreams` at the
router until that source exists.
See [the health runbook](../runbooks/adguard-home-health.md) and
[the official configuration review](../docs/adguard-home-official-review-2026-08.md).
Routine health through Ansible (`--limit adguardhome`), with a daily timer +
email alerting like the other active hosts:
```bash
cd ansible
ansible-playbook playbooks/health-report.yml --limit adguardhome
```
+3 -2
View File
@@ -50,8 +50,9 @@ See [mailcow-smtp-client](../runbooks/mailcow-smtp-client.md).
- [mailcow-health](../runbooks/mailcow-health.md) - [mailcow-health](../runbooks/mailcow-health.md)
- [mailcow-update](../runbooks/mailcow-update.md) - [mailcow-update](../runbooks/mailcow-update.md)
- [mailcow-smtp-client](../runbooks/mailcow-smtp-client.md) - [mailcow-smtp-client](../runbooks/mailcow-smtp-client.md)
- Script: [`../scripts/check-mx2.sh`](../scripts/check-mx2.sh)
Routine health: `cd ansible && ansible-playbook playbooks/health-report.yml --limit mailcow`.
## Verified ## Verified
Last checked: 2026-08-01 — `./scripts/check-mx2.sh` exit 0 (compose up, watchdog 100%, empty queue, TLS LE OK, SMTP Postcow, A/AAAA/PTR/MX/SPF OK). Last checked: 2026-08-01 — Ansible mailcow health profile OK (compose up, watchdog 100%, empty queue, TLS LE OK, SMTP Postcow, A/AAAA/PTR/MX/SPF OK).
+8 -2
View File
@@ -76,9 +76,15 @@ ansible-playbook playbooks/compose-reconcile.yml --limit vaultwarden \
-e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["vaultwarden"], "service_reconcile_restart_traefik": true}' -e '{"service_reconcile_confirm": true, "service_reconcile_targets": ["vaultwarden"], "service_reconcile_restart_traefik": true}'
``` ```
## Other software on this host (stubs) ## Other running services on this host
`/opt/traefik`, `soft-serve`, `authelia`, `dendrite`/`conduit`, `mastodon`, `rustdesk`, `zitadel`, etc. | Container | Status | Image / notes |
|-----------|--------|---------------|
| `soft-serve` | Up | `ghcr.io/charmbracelet/soft-serve:latest` (`repo.windy.me:2222`) |
| `traefik` | Up | `traefik:v3.6.2` (`/opt/traefik`, public `:80`/`:443`) |
| `nghttpx-proxy` + `squid-backend` | Up | HTTP forward-proxy stack (`/opt/nghttpx`), network `nghttpx_internal-net`; details TBD |
Directories for `authelia`, `conduit`, `dendrite`, `mastodon`, `rustdesk`, `zitadel`, etc. exist under `/opt` but have no running containers; treat them as dormant, not documented services.
## Verified ## Verified
+1
View File
@@ -21,6 +21,7 @@ diagnosis and procedures that are deliberately interactive or destructive; see
| pgweb.wsvc.info | PowerDNS Postgres UI | — (HTTPS) | → hk2 | active | see hk2 | | pgweb.wsvc.info | PowerDNS Postgres UI | — (HTTPS) | → hk2 | active | see hk2 |
| **synapse.chans.xyz** | Matrix homeserver (ESS: Synapse + MAS + Element) | `ssh -4 windy@synapse.chans.xyz` | `169.58.86.13` | **active** | [hosts/synapse.chans.xyz.md](../hosts/synapse.chans.xyz.md) | | **synapse.chans.xyz** | Matrix homeserver (ESS: Synapse + MAS + Element) | `ssh -4 windy@synapse.chans.xyz` | `169.58.86.13` | **active** | [hosts/synapse.chans.xyz.md](../hosts/synapse.chans.xyz.md) |
| **gfw.windy.lan** | OpenWrt LAN gateway / OpenClash | `ssh -4 root@192.168.66.1` | `192.168.66.1` | **active** | [hosts/gfw.windy.lan.md](../hosts/gfw.windy.lan.md) | | **gfw.windy.lan** | OpenWrt LAN gateway / OpenClash | `ssh -4 root@192.168.66.1` | `192.168.66.1` | **active** | [hosts/gfw.windy.lan.md](../hosts/gfw.windy.lan.md) |
| **dns.windy.lan** | AdGuard Home LAN DNS + Mihomo explicit proxy | `ssh -4 windy@192.168.66.36` | `192.168.66.36` | **active** | [hosts/dns.windy.lan.md](../hosts/dns.windy.lan.md) |
| **gw** | EdgeRouter X primary LAN gateway | `ssh -4 zhiqiang@192.168.66.254` | `192.168.66.254` | **active** | [hosts/gw.md](../hosts/gw.md) | | **gw** | EdgeRouter X primary LAN gateway | `ssh -4 zhiqiang@192.168.66.254` | `192.168.66.254` | **active** | [hosts/gw.md](../hosts/gw.md) |
| **ubnt** | UniFi Network Controller | `ssh -4 windy@192.168.66.46` | `192.168.66.46` | **active** | [hosts/ubnt.md](../hosts/ubnt.md) | | **ubnt** | UniFi Network Controller | `ssh -4 windy@192.168.66.46` | `192.168.66.46` | **active** | [hosts/ubnt.md](../hosts/ubnt.md) |
@@ -1,70 +0,0 @@
# VPS 維護專案重構:Ansible + systemd timers + Restic
## Objective
將現有 VPS 維運知識庫漸進重構為一個以 Ansible 管理一致性、以 systemd timers 執行每日唯讀檢查、以 Restic 建立加密異機備份的個人 SRE 維運專案。現納管 mx2、us2、hk2 與 Matrix K3s 主機。自動化預設只能觀測與告警;受 allowlist 和顯式確認保護的 Compose reconciliation 可處理已審查的服務重建,DNS、機密與破壞性資料操作仍需人工確認。
## Confirmed Decisions
- [x] 定位為個人 SRE/維運手冊庫,而非全自動修復平台。
- [x] 已納管 mx2、us2、hk2 與 Matrix K3smx、us1、us4 待完成盤點與緊急存取驗證後才納管。
- [x] 每日自動做唯讀檢查與 Email 通知;每週人工審查、每月人工維護、每季復原演練與權限盤點。
- [x] 以 Email 作為唯一告警與摘要通道。
- [x] 採用本機快速復原加密異機副本的 3-2-1 最小可行策略。
- [x] 真實機密不進 Git;repo 只保存去敏設定結構與操作/驗證紀錄。
- [x] 日常存取使用非 root SSH key;禁止 root/password SSH,維護 provider/recovery console 緊急存取。
- [x] 採用 Ansible + systemd timers + Restic 作為技術基線。
## Implementation Plan
- [x] **Status: Done — 建立去敏 Ansible inventory。** 為 mx2、us2、hk2 與 Matrix 建立主機與服務群組,將非機密主機資料與現有人類可讀的 `inventory/hosts.md` 對應;理由是提供可審查的機器可讀編排層,且不取代既有事實來源。
- [x] **Status: Done — 實作 audit-only Ansible playbook。** 只收集連線、OS、磁碟、systemd、Docker Compose 和依賴命令狀態,不做設定或服務改動;理由是先確認控制面與真實現況。
- [x] **Status: Done — 定義跨服務健康檢查結果合約。** 統一去敏結構化結果、日誌位置、嚴重度及 exit code,讓人工執行、systemd 和 Ansible audit 可共同消費;理由是避免各服務告警語意漂移。
- [x] **Status: Done — 保留並模組化 Mailcow 健康檢查。** 持續檢查 Compose、watchdog、queue、listeners、HTTP/HTTPS、TLS、SMTP、DNS/PTR/MX/SPF;理由是現有腳本已覆蓋關鍵郵件服務面向。
- [x] **Status: Done — 實作 Vaultwarden 唯讀健康檢查。** 驗證 Compose/Postgres、HTTPS、有效設定、SMTP AUTH、備份新鮮度及機密指紋一致性,禁止輸出機密;理由是 `config.json` 優先於 `.env`,且 SMTP 漂移已有已知風險。
- [x] **Status: Done — 實作 PowerDNS 唯讀健康檢查。** 驗證容器、版本與安全公告、ns1/ns2 served SOA、API、Web UI、備份新鮮度與必要設定;理由是同時覆蓋公開 DNS 與 secondary 同步。
- [x] **Status: Done — 使用 Ansible 部署 systemd healthcheck service/timer。** 每台主機本機執行每日檢查,具 persistent 排程、權限、logrotate 和一致錯誤處理;理由是控制端離線不應阻止巡檢。
- [x] **Status: Done — 部署 Email 告警與每日摘要。** Critical/unknown 立即通知、健康狀態每日摘要、重複失敗抑制;SMTP 真實認證僅在各主機受限路徑保存;理由是已選定 Email 為唯一通知通道。
- [!] **Status: Blocked — 決定 Restic 異機 repository 與存取隔離。** 選擇具加密傳輸、權限隔離與可承受保留需求的 S3/B2/SFTP 或等價目的地;理由是無異機 repository 即無法滿足已確認的 3-2-1 政策。阻塞:本地實作依要求未虛構 backend、repository 或 credentials。
- [x] **Status: Done (templates gated) — 部署 Vaultwarden 與 PowerDNS 的 Restic 備份。** 將既有一致性資料庫 dump、必要資料目錄及復原元資料加密同步,並加入 snapshot 年齡和 `restic check` 驗證;理由是兩者已有本機備份但缺少異機保護。啟用被 repository 決策和主機端受限設定檔阻擋。
- [!] **Status: Blocked — 完成 Mailcow 備份設計審查。** 官方流程已確認:以 `/opt/mail/helper-scripts/backup_and_restore.sh backup all`(或經明確核准的元件集)先產生一致性備份,再由 Restic 同步該輸出;不得直接複製 Docker volumes。阻塞:仍需決定本機備份位置、保留期、排程及異機 Restic repository。
- [x] **Status: Done (templates gated) — 部署 Restic backup、retention、forget/prune 與 check timers。** 與現有資料庫 dump 時段錯開,所有 repository 認證與密碼留在伺服器端受限檔案;理由是保持備份可用與成本可控。啟用被 repository 決策和主機端受限設定檔阻擋。
- [!] **Status: Blocked — 補齊服務級復原 runbook。** 阻塞:需在選定 Restic repository、建立實際 snapshot 並確認各服務的實際備份輸出後,才能編寫可驗證的 restore 前置條件、順序與 rollback;禁止臆造 backend/credentials 或未驗證還原命令。
- [!] **Status: Blocked — 執行隔離式復原演練。** 阻塞:尚未選定/配置 Restic 異機 repository,亦尚無可供還原的異機 snapshot;演練不得對現有生產資料執行。
- [x] **Status: Done — 實作受控 common baseline 與 maintenance playbook。** 僅在 audit 穩定後納入 SSH 稽核、時間同步、logrotate、更新預覽與人工確認的維護操作;理由是避免工具導入期間同時改變服務狀態。
- [x] **Status: Done (2026-08-03) — 將例行操作收斂為 Ansible 入口。** 新增 on-demand health report 與 allowlisted Compose reconciliationhealth、maintenance preview、baseline 現可涵蓋 Matrix。互動式 Mailcow 更新、資料遷移、DNS 與機密操作保留人工程序。
- [x] **Status: Done (separate change defined) — 將 PowerDNS API key、DB password 與 TSIG 輪替列為分離變更。** 已明確保持 API key、DB password 與 TSIG 為三項獨立、需人工核准的變更,並要求逐步驗證 Auth、Poweradmin、AXFR/NOTIFY、DNSSEC 與 ns2 同步;未輪替任何機密。
- [!] **Status: Blocked — 盤點 mx、us1、us4。** 阻塞:inventory 僅記錄 `TBD` SSH/角色或缺少完整事實,尚未提供可驗證的存取方式與 provider/recovery console 資訊;不得猜測或嘗試未授權存取。
## Verification Criteria
- [ ] 四台 active 主機能被 Ansible 正確解析;Compose 主機可執行 audit,且 audit 不產生主機變更。
- [ ] 四台主機每日產生去敏健康結果;服務、容量、TLS、公開端點和備份新鮮度異常均能被偵測。
- [ ] 正常狀態寄送 Email 摘要;受控測試異常能觸發一次可讀且不含機密的 Email 告警。
- [ ] 每個核心服務至少有一份加密、異機、可列出且通過完整性檢查的 Restic snapshot。
- [ ] 至少完成一次隔離式實際還原,並確認其結果符合或明確量化偏離服務 RPO/RTO。
- [ ] 自動化排程不包含更新、重啟、修復、秘密輪替或 DNS 變更。
- [ ] Repo、Ansible vars、產出日誌及 Email 均不包含任何真實機密。
## Potential Risks and Mitigations
1. **Mailcow 備份未保持資料一致性。**
Mitigation: 先依官方方式完成備份範圍與還原設計,再上線排程;以隔離還原驗證作為完成門檻。
2. **Ansible 設定錯誤改動生產系統。**
Mitigation: 先導入 audit-only;變更 playbook 必須使用 dry-run/diff、明確 tag 與人工確認,且先限制單一 host。
3. **機密出現在 Git、Ansible output 或 Email。**
Mitigation: 真實值只存主機受限檔案;去敏輸出、no-log 機制和測試用機密掃描為強制要求。
4. **Timer 成功但公開服務實際不可用。**
Mitigation: 同時執行本機 Compose/DB 檢查與外部 HTTP、SMTP、TLS、DNS 檢查。
5. **PowerDNS 密鑰輪替中斷 DNS 管理或 zone transfer。**
Mitigation: API key、DB password、TSIG 分批處理;每步都驗證 primary/secondary 同步、DNSSEC 和管理 UI。
## Alternative Approaches
1. **純 shell + systemd,不使用 Ansible**:初期較快,但設定與排程容易跨主機漂移,且難以審查或擴充;不建議作為長期方案。
2. **Ansible + 集中式監控平台**:可增加儀表板與趨勢,但平台本身需額外維護;等 Email 摘要無法滿足需求時再評估。
3. **立即導入 SOPS + age**:可使加密設定進 Git,但需先完成密鑰生命週期與緊急存取設計;建議列為後續階段。
+60
View File
@@ -0,0 +1,60 @@
# AdGuard Home health — dns.windy.lan
This runbook is read-only. It does not expose query-log contents or secrets.
Routine checks run through Ansible on demand:
```bash
cd ansible
ansible-playbook playbooks/health-report.yml --limit adguardhome
```
The same sanitized profile is installed as a daily server-local timer
(`vps-healthcheck.timer`, output at `/var/lib/vps-health/latest.json`) via
`playbooks/healthchecks.yml`; the manual checks below are for focused diagnosis.
## Verify service and configuration
```bash
ssh -4 -o BatchMode=yes windy@192.168.66.36
sudo docker ps --filter name=^/adguardhome$
sudo docker exec adguardhome /opt/adguardhome/AdGuardHome --check-config \
-c /opt/adguardhome/conf/AdGuardHome.yaml
sudo ss -lntup | grep -E '(:53|:80)'
```
Expected: DNS listens on TCP/UDP 53, and the HTTP UI listens only on
`192.168.66.36:80`.
## Verify DNS and storage
```bash
dig +time=5 +tries=1 @192.168.66.36 example.com A +noall +comments +answer
dig +time=5 +tries=1 @192.168.66.36 example.com AAAA +noall +comments +answer
dig +time=5 +tries=1 @192.168.66.36 hass.windy.lan A +noall +comments +answer
sudo docker logs --since 30m adguardhome 2>&1 | grep -Ei '\[(error|warn|fatal)\]|level=(error|warn|fatal)' || true
df -h /opt/adguardhome/work
```
The host intentionally accepts DNS from LAN55 and LAN66. Do not infer a
broken DNS policy merely because `allowed_clients` is empty.
## Verify management UI boundary
From LAN66, the UI should return a login redirect:
```bash
curl -sS -o /dev/null -w '%{http_code}\n' http://192.168.66.36/
```
From a LAN55 source, TCP/80 should time out or be refused by the dedicated
`agh-ui-access.service` nftables rule. Do not alter this rule to restrict
port 53 without an explicit network-access review.
## DNSSEC and PTR caveats
DNSSEC is currently disabled. Before enabling it, select upstreams that pass
a known-bad-signature test; an enabled DO bit alone is not validation.
Private PTR forwarding is intentionally absent because the EdgeRouter does
not currently answer private PTR requests.
+7 -4
View File
@@ -2,9 +2,10 @@
Target: [mx2.windy.me](../hosts/mx2.windy.me.md) Target: [mx2.windy.me](../hosts/mx2.windy.me.md)
Path: `/opt/mail` Path: `/opt/mail`
Prefer: the Ansible health report (`ansible/playbooks/health-report.yml`), or Prefer: the Ansible health report (`ansible/playbooks/health-report.yml --limit mailcow`),
`./scripts/check-mx2.sh` from this repo. The SSH commands below remain useful or the server-local daily timer result at `/var/lib/vps-health/latest.json`. The
for focused, interactive diagnosis after the report identifies a problem. SSH commands below remain useful for focused, interactive diagnosis after the
report identifies a problem.
## Focused SSH diagnostics ## Focused SSH diagnostics
@@ -66,7 +67,9 @@ dig @1.1.1.1 +short windy.me TXT | grep -i spf
## Scheduled local check ## Scheduled local check
The sanitized Ansible health profile is `mailcow` (`ansible/playbooks/healthchecks.yml`). It preserves the existing standalone `scripts/check-mx2.sh` as the richer operator check, while the server-local timer emits a sanitized result at `/var/lib/vps-health/latest.json`. Neither check changes Mailcow configuration or service state. The sanitized Ansible health profile is `mailcow` (`ansible/playbooks/healthchecks.yml`).
The server-local timer emits a sanitized result at `/var/lib/vps-health/latest.json`.
It does not change Mailcow configuration or service state.
## Pass criteria ## Pass criteria
+1 -1
View File
@@ -6,7 +6,7 @@ Path: `/opt/mail`
## Before ## Before
1. Run [mailcow-health](mailcow-health.md) (or `./scripts/check-mx2.sh`). Record baseline. 1. Run [mailcow-health](mailcow-health.md) (Ansible health report). Record baseline.
2. Note config quirks in `mailcow.conf`, especially: 2. Note config quirks in `mailcow.conf`, especially:
- `SKIP_CLAMD=y` — after update, `clamd` may still be running until services are recreated per mailcow docs. - `SKIP_CLAMD=y` — after update, `clamd` may still be running until services are recreated per mailcow docs.
- `SKIP_LETS_ENCRYPT`, ports, hostname — should stay `mx2.windy.me`. - `SKIP_LETS_ENCRYPT`, ports, hostname — should stay `mx2.windy.me`.
-195
View File
@@ -1,195 +0,0 @@
#!/usr/bin/env bash
# Read-only health check for mx2.windy.me mailcow.
# Usage: ./scripts/check-mx2.sh
# Env:
# MX2_HOST default mx2.windy.me
# MX2_SSH_USER default windy
# MX2_ALLOW_IPV6 set to 1 to allow IPv6 (default forces IPv4; WSL often lacks v6 route)
set -euo pipefail
HOST="${MX2_HOST:-mx2.windy.me}"
SSH_USER="${MX2_SSH_USER:-windy}"
# Default -4: host has AAAA but many clients (WSL) have no IPv6 route → "Network is unreachable"
SSH_AF_OPTS=(-4)
CURL_AF_OPTS=(-4)
OPENSSL_AF_OPTS=(-4)
if [[ "${MX2_ALLOW_IPV6:-0}" == "1" ]]; then
SSH_AF_OPTS=()
CURL_AF_OPTS=()
OPENSSL_AF_OPTS=()
fi
SSH=(ssh "${SSH_AF_OPTS[@]}" -o BatchMode=yes -o ConnectTimeout=10 "${SSH_USER}@${HOST}")
# Prefer public resolvers: local stub (127.0.0.53) is flaky from some WSL setups
DIG_OPTS=(+time=3 +tries=2 +short)
DIG_SERVERS=(1.1.1.1 8.8.8.8)
FAILURES=0
section() { printf '\n===== %s =====\n' "$1"; }
fail() {
printf 'FAIL: %s\n' "$1" >&2
FAILURES=$((FAILURES + 1))
}
# Strip dig commentary (timeouts are printed on stdout as ";; ...")
dig_answers_only() {
grep -v '^;;' | grep -v '^$' || true
}
# dig_short <name> [type] — tries each public resolver; skips timeout noise
dig_short() {
local name="$1"
local typ="${2:-A}"
local server out
for server in "${DIG_SERVERS[@]}"; do
out="$(dig @"${server}" "${DIG_OPTS[@]}" "$name" "$typ" 2>/dev/null | dig_answers_only)" || out=""
if [[ -n "$out" ]]; then
printf '%s\n' "$out"
return 0
fi
done
return 1
}
# dig_ptr <ip>
dig_ptr() {
local ip="$1"
local server out
for server in "${DIG_SERVERS[@]}"; do
out="$(dig @"${server}" "${DIG_OPTS[@]}" -x "$ip" 2>/dev/null | dig_answers_only)" || out=""
if [[ -n "$out" ]]; then
printf '%s\n' "$out"
return 0
fi
done
return 1
}
check_ptrs() {
local ip ptr
while read -r ip; do
[[ -z "$ip" ]] && continue
ptr="$(dig_ptr "$ip" || true)"
printf 'PTR %s -> %s\n' "$ip" "${ptr:-<none>}"
grep -Fiq "${HOST}" <<<"$ptr" || fail "PTR for ${ip} does not mention ${HOST}"
done
}
need_cmd() {
command -v "$1" >/dev/null 2>&1 || {
echo "missing required command: $1" >&2
exit 127
}
}
need_cmd ssh
need_cmd curl
need_cmd openssl
need_cmd dig
need_cmd timeout
section "SSH + compose ps"
PS_OUT="$("${SSH[@]}" 'cd /opt/mail && docker compose ps -a')" || {
fail "ssh or docker compose ps"
echo "Cannot continue without SSH/compose." >&2
exit 1
}
printf '%s\n' "$PS_OUT"
if grep -qiE 'Exited|Restarting|[[:space:]]Dead[[:space:]]' <<<"$PS_OUT"; then
fail "compose has Exited/Restarting/Dead containers"
fi
for svc in nginx-mailcow postfix-mailcow dovecot-mailcow mysql-mailcow; do
grep -Fq "$svc" <<<"$PS_OUT" || fail "missing service ${svc}"
done
section "Watchdog (tail)"
if ! "${SSH[@]}" 'cd /opt/mail && docker compose logs --tail=40 watchdog-mailcow'; then
fail "watchdog logs"
fi
section "Mail queue"
QUEUE_OUT="$("${SSH[@]}" 'cd /opt/mail && docker compose exec -T postfix-mailcow postqueue -p' 2>&1)" || {
fail "postqueue"
QUEUE_OUT=""
}
printf '%s\n' "$QUEUE_OUT"
if [[ -n "$QUEUE_OUT" ]] && ! grep -Fiq 'Mail queue is empty' <<<"$QUEUE_OUT"; then
fail "mail queue not empty"
fi
section "Listeners"
LISTEN_OUT="$("${SSH[@]}" 'ss -tlnp 2>/dev/null | grep -E ":(25|465|587|110|143|993|995|80|443|4190)[[:space:]]" || true')"
printf '%s\n' "$LISTEN_OUT"
for p in 25 465 587 110 143 993 995 80 443 4190; do
if ! grep -qE ":${p}[[:space:]]" <<<"$LISTEN_OUT"; then
fail "port ${p} not listening on host"
fi
done
section "HTTP / HTTPS"
HTTP_OUT="$(curl "${CURL_AF_OPTS[@]}" -sS -I --max-time 10 "http://${HOST}/" 2>&1 | head -8)" || true
printf '%s\n' "$HTTP_OUT"
echo "---"
# No -k: surface TLS trust problems
HTTPS_OUT="$(curl "${CURL_AF_OPTS[@]}" -sS -I --max-time 10 "https://${HOST}/" 2>&1 | head -15)" || true
printf '%s\n' "$HTTPS_OUT"
grep -qE '^HTTP/[0-9.]+ 301' <<<"$HTTP_OUT" || fail "HTTP did not redirect (expect 301)"
grep -qE '^HTTP/[0-9.]+ 200' <<<"$HTTPS_OUT" || fail "HTTPS did not return 200"
section "TLS cert (443)"
CERT_OUT="$(
timeout 15 openssl s_client "${OPENSSL_AF_OPTS[@]}" -connect "${HOST}:443" -servername "${HOST}" </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates 2>/dev/null
)" || true
printf '%s\n' "$CERT_OUT"
if ! grep -Fq "CN=${HOST}" <<<"$CERT_OUT" && ! grep -Fq "CN = ${HOST}" <<<"$CERT_OUT"; then
fail "TLS subject missing ${HOST}"
fi
grep -Fiq "Let's Encrypt" <<<"$CERT_OUT" || fail "TLS issuer not Let's Encrypt"
section "SMTP banner"
# Connect via A record so /dev/tcp does not pick unreachable AAAA first
SMTP_IP="$(dig_short "${HOST}" A 2>/dev/null | head -n1 || true)"
if [[ -z "$SMTP_IP" ]]; then
fail "cannot resolve A for SMTP check"
SMTP_OUT=""
else
SMTP_OUT="$(
timeout 8 bash -c "exec 3<>/dev/tcp/${SMTP_IP}/25; printf 'EHLO test.local\r\nQUIT\r\n' >&3; cat <&3" 2>/dev/null \
| head -20
)" || true
fi
printf 'connect %s:25\n' "${SMTP_IP:-?}"
printf '%s\n' "$SMTP_OUT"
grep -qiE 'Postcow|ESMTP' <<<"$SMTP_OUT" || fail "SMTP banner unexpected"
section "DNS"
A_REC="$(dig_short "${HOST}" A || true)"
AAAA_REC="$(dig_short "${HOST}" AAAA || true)"
printf 'A: %s\n' "${A_REC:-<none>}"
printf 'AAAA: %s\n' "${AAAA_REC:-<none>}"
[[ -n "$A_REC" ]] || fail "missing A record"
[[ -n "$AAAA_REC" ]] || fail "missing AAAA record"
check_ptrs <<<"$A_REC"
check_ptrs <<<"$AAAA_REC"
MX_REC="$(dig_short windy.me MX || true)"
printf 'MX:\n%s\n' "${MX_REC:-<none>}"
grep -Fiq "${HOST}" <<<"$MX_REC" || fail "windy.me MX missing ${HOST}"
SPF="$(dig_short windy.me TXT || true)"
printf 'SPF:\n'
if ! grep -Fi 'v=spf1' <<<"$SPF"; then
fail "no SPF TXT on windy.me"
fi
section "Done"
if [[ "$FAILURES" -gt 0 ]]; then
echo "Health check finished with ${FAILURES} failure(s)."
exit 1
fi
echo "Health check OK. Update hosts/mx2.windy.me.md Verified line if desired."
exit 0