my-vault/01_Projects/Infrastructure/Services/Matrix Ess Server Install.md

# Matrix ESS (Community) — Single‑Node Install on Debian 13 (K3s + Traefik + cert‑manager)
_Last updated: 2025-09-25 08:45 UTC_

This guide installs **Element Server Suite (ESS) Community** (Synapse + MAS + Element Web + Matrix RTC) on a **single Debian 13** node using **K3s**, **Traefik** (default in K3s), and **cert‑manager** with **Let’s Encrypt**. It is tailored to your domain choices:

- **serverName**: `chans.xyz`
- **Hosts**: `synapse.chans.xyz`, `account.chans.xyz`, `chat.chans.xyz`, `mrtc.chans.xyz`

> Tip: if you already have K3s and cert‑manager installed and working, you can jump to **5. Values files** and **6. Install ESS**.

---

## 0) Requirements & Ports

- Debian 13 (root/sudo), public IPv4 (and optional IPv6).
- DNS control for `chans.xyz`.
- Open/forward these ports to this node:
  - **80/tcp**, **443/tcp** (ACME + HTTPS + federation)
  - **30881/tcp**, **30882/udp** (Matrix RTC SFU)
- Time in sync (`systemd-timesyncd` or equivalent).

---

## 1) DNS Setup

Create A/AAAA records that point to your node’s public IP(s):

```
chans.xyz             A / AAAA -> <YOUR_IP>
synapse.chans.xyz     A / AAAA -> <YOUR_IP>
account.chans.xyz     A / AAAA -> <YOUR_IP>
chat.chans.xyz        A / AAAA -> <YOUR_IP>
mrtc.chans.xyz        A / AAAA -> <YOUR_IP>
```

Notes:

- **Do not** use a `CNAME` at the **apex** (`chans.xyz`)—use `A/AAAA`. Subdomains can be `CNAME`s if you prefer.
- Federation relies on `https://chans.xyz/.well-known/matrix/server` which the chart serves for you.

---

## 2) (Optional) Cloud‑Init (without firewalld)

If you build the node via cloud‑init, this minimal config installs K3s & Helm and disables swap:

```yaml
#cloud-config
package_update: true
package_upgrade: true
packages: [curl, ca-certificates, gnupg, lsb-release]

runcmd:
  - swapoff -a
  - sed -ri 's/^[^#].*\sswap\s/## &/g' /etc/fstab
  - curl -sfL https://get.k3s.io | sh -s - server
  - mkdir -p /home/windy/.kube
  - cp /etc/rancher/k3s/k3s.yaml /home/windy/.kube/config
  - chown windy:windy /home/windy/.kube/config && chmod 600 /home/windy/.kube/config
  - bash -lc 'echo export KUBECONFIG=$HOME/.kube/config >> /home/windy/.bashrc'
  - su - windy -c "curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash"
```

You can manage ports at the cloud firewall or your router (no `firewalld` required).

---

## 3) Manual K3s + Helm (if not using cloud‑init)

```bash
# Install latest K3s
curl -sfL https://get.k3s.io | sh -s - server

# kubeconfig for your user (replace 'windy' if needed)
mkdir -p ~windy/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~windy/.kube/config
sudo chown windy:windy ~windy/.kube/config
chmod 600 ~windy/.kube/config
echo 'export KUBECONFIG=$HOME/.kube/config' | sudo tee -a ~windy/.bashrc

# Helm
sudo -iu windy bash -lc 'curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash'
```

Verify:
```bash
kubectl get nodes -o wide
kubectl get pods -A
```

You should see the node `Ready` and `traefik` running in `kube-system`.

---

## 4) cert‑manager + Let’s Encrypt (ClusterIssuer)

If you haven’t installed cert‑manager yet:

```bash
helm repo add jetstack https://charts.jetstack.io --force-update
kubectl create namespace cert-manager 2>/dev/null || true
helm install cert-manager jetstack/cert-manager -n cert-manager --set crds.enabled=true
```

Create a production ClusterIssuer (`letsencrypt-prod`):

```yaml
# clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-private-key
    solvers:
    - http01:
        ingress:
          class: traefik
```

Apply:
```bash
kubectl apply -f clusterissuer.yaml
kubectl get clusterissuer
```

You should see `letsencrypt-prod  READY=True`.

---

## 5) Values files (hosts + TLS)

Create the directory and values files:

```bash
mkdir -p ~/ess-config-values
```

**`~/ess-config-values/hostnames.yaml`**
```yaml
serverName: chans.xyz

elementWeb:
  ingress:
    host: chat.chans.xyz

synapse:
  ingress:
    host: synapse.chans.xyz

matrixAuthenticationService:
  ingress:
    host: account.chans.xyz

matrixRTC:
  ingress:
    host: mrtc.chans.xyz
```

**`~/ess-config-values/tls.yaml`**
```yaml
global:
  ingress:
    className: traefik
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    tls:
      enabled: true
      issuer: letsencrypt-prod
```

> The TLS values ensure your Ingresses are annotated for cert‑manager and include TLS host entries so Certificates are created automatically.

---

## 6) Install ESS (matrix‑stack chart)

```bash
kubectl create namespace ess 2>/dev/null || true

helm upgrade --install ess   oci://ghcr.io/element-hq/ess-helm/matrix-stack   -n ess   -f ~/ess-config-values/hostnames.yaml   -f ~/ess-config-values/tls.yaml   --wait
```

Check status:
```bash
kubectl get pods -n ess
kubectl get ingress -n ess
```

You should see ingresses for `synapse`, `account`, `chat`, `mrtc`, and `well-known` with `CLASS=traefik`.

---

## 7) Certificates issuance

Confirm the ingresses have TLS + issuer:
```bash
kubectl -n ess get ingress -o jsonpath='{range .items[*]}{.metadata.name}{"  issuer="}{.metadata.annotations.cert-manager\.io/cluster-issuer}{"  tlsHosts="}{range .spec.tls[*].hosts}{.}{" "}{end}{"\n"}{end}'
```

Then watch certs:
```bash
kubectl get certificate -n ess
kubectl get order,challenge -n ess
```

When ready, confirm live certs:
```bash
for h in synapse.chans.xyz account.chans.xyz chat.chans.xyz mrtc.chans.xyz chans.xyz; do
  echo "=== $h ==="
  openssl s_client -connect "$h:443" -servername "$h" </dev/null 2>/dev/null |     openssl x509 -noout -issuer -subject -dates
done
```

---

## 8) Well‑Known verification (federation & clients)

```bash
curl -s https://chans.xyz/.well-known/matrix/server | jq .
curl -s https://chans.xyz/.well-known/matrix/client | jq .
```

Expected:
- `server` → `{ "m.server": "synapse.chans.xyz:443" }`
- `client` → `{ "m.homeserver": { "base_url": "https://synapse.chans.xyz" }, ... }`

Optional federation tester: <https://federationtester.matrix.org/#chans.xyz>

---

## 9) Create the first admin account

Interactive:
```bash
kubectl exec -n ess -it deploy/ess-matrix-authentication-service --   mas-cli manage register-user --admin
```

Non‑interactive example:
```bash
kubectl exec -n ess deploy/ess-matrix-authentication-service --   mas-cli manage register-user --yes --admin   --username admin --password 'CHANGE_ME_strong_password'
```

Login at **https://chat.chans.xyz**.

---

## 10) Enable self‑registration (optional)

```yaml
# ~/ess-config-values/mas-registration.yaml
matrixAuthenticationService:
  additional:
    registration.yaml:
      config: |
        account:
          password_registration_enabled: true
          password_recovery_enabled: true
          login_with_email_allowed: true
```

Apply (include this file):
```bash
helm upgrade --install ess   oci://ghcr.io/element-hq/ess-helm/matrix-stack   -n ess   -f ~/ess-config-values/hostnames.yaml   -f ~/ess-config-values/tls.yaml   -f ~/ess-config-values/mas-registration.yaml   --wait
```

---

## 11) Outbound email (MAS required, Synapse optional)

### 11.1 MAS SMTP (required for signup/reset)

**Option A — inline values (simple):**
```yaml
# ~/ess-config-values/mas-email.yaml
matrixAuthenticationService:
  additional:
    user-config.yaml:
      config: |
        email:
          from: '"Matrix @ chans.xyz" <noreply@chans.xyz>'
          reply_to: '"Support" <support@chans.xyz>'
          transport: smtp
          mode: starttls
          hostname: smtp.windy.me
          port: 587
          username: noreply@chans.xyz        # authenticate as the sender
          password: "MAILBOX_PASSWORD"
        account:
          password_registration_enabled: true
          password_recovery_enabled: true
          login_with_email_allowed: true
```

**Option B — secret ref (keeps password out of Git):**
```bash
cat > /tmp/mas-user-config.yaml <<'YAML'
email:
  from: '"Matrix @ chans.xyz" <noreply@chans.xyz>'
  reply_to: '"Support" <support@chans.xyz>'
  transport: smtp
  mode: starttls
  hostname: smtp.windy.me
  port: 587
  username: noreply@chans.xyz
  password: "MAILBOX_PASSWORD"
account:
  password_registration_enabled: true
  password_recovery_enabled: true
  login_with_email_allowed: true
YAML

kubectl -n ess create secret generic mas-extra-config   --from-file=user-config.yaml=/tmp/mas-user-config.yaml
```

Then reference it:
```yaml
# ~/ess-config-values/mas-email-secretref.yaml
matrixAuthenticationService:
  additional:
    user-config.yaml:
      configSecret: mas-extra-config
      configSecretKey: user-config.yaml
```

Apply (include one of the two files above):
```bash
helm upgrade --install ess oci://ghcr.io/element-hq/ess-helm/matrix-stack   -n ess   -f ~/ess-config-values/hostnames.yaml   -f ~/ess-config-values/tls.yaml   -f ~/ess-config-values/mas-email.yaml   --wait
# or replace mas-email.yaml with mas-email-secretref.yaml if you used a Secret
```

> **Mailcow 553 fix**: If authenticating as `zhiqiang@windy.me` and sending as `noreply@chans.xyz`, Mailcow rejects with `553 5.7.1 Sender address rejected`. Either (a) **authenticate as** `noreply@chans.xyz` by creating that mailbox in Mailcow and publishing SPF/DKIM/DMARC for `chans.xyz`; or (b) allow “send as” in Mailcow’s **Sender ACL** for `zhiqiang@windy.me`. Hosting the `chans.xyz` mailbox gives best deliverability (DKIM/DMARC alignment).

Monitor while testing:
```bash
kubectl -n ess logs deploy/ess-matrix-authentication-service -f | grep -iE 'smtp|email|send'
```

### 11.2 Synapse email notifications (optional)
```yaml
# ~/ess-config-values/synapse-email.yaml
synapse:
  additional:
    email.yaml:
      config: |
        email:
          smtp_host: "smtp.windy.me"
          smtp_port: 587
          smtp_user: "noreply@chans.xyz"
          smtp_pass: "MAILBOX_PASSWORD"
          require_transport_security: true
          notif_from: "Matrix on chans.xyz <noreply@chans.xyz>"
          enable_notifs: true
```

Include this file in your next Helm upgrade.

---

## 12) Health checks & troubleshooting

**Basic:**
```bash
kubectl get pods,svc,ingress,certificate -n ess -o wide
```

**Certs flow:**
```bash
kubectl get certificate,order,challenge -n ess
kubectl describe challenge -n ess <name>
kubectl logs -n kube-system deploy/traefik --tail=200
```

**Well‑known + federation:**
```bash
curl -s https://chans.xyz/.well-known/matrix/server | jq .
curl -s https://chans.xyz/.well-known/matrix/client | jq .
```

**Common pitfalls:**
- Ingresses lack TLS + `cert-manager.io/cluster-issuer` → fix `tls.yaml`.
- `553 Sender address rejected` from Mailcow → align SMTP auth user with sender or allow “send as”, and set SPF/DKIM/DMARC for `chans.xyz`.
- Port 80 blocked → Let’s Encrypt HTTP‑01 fails (check challenges).
- Apex `chans.xyz` not pointing at the node → `.well-known` fails → federation fails.

---

## 13) Upgrades / Uninstall

Upgrade to latest chart:
```bash
helm repo update  # if using repos
helm upgrade --install ess oci://ghcr.io/element-hq/ess-helm/matrix-stack -n ess   -f ~/ess-config-values/hostnames.yaml -f ~/ess-config-values/tls.yaml --wait
```

Uninstall ESS (keeps PVCs unless you delete them):
```bash
helm uninstall ess -n ess
kubectl delete namespace ess
```

Reset K3s (if ever needed):
```bash
sudo /usr/local/bin/k3s-uninstall.sh
```

---

## 14) Quick copy‑paste checklist

1. DNS A/AAAA for: `chans.xyz`, `synapse.`, `account.`, `chat.`, `mrtc.` → your IP.  
2. K3s running with Traefik; cert‑manager installed; `ClusterIssuer letsencrypt-prod` **Ready**.  
3. `hostnames.yaml` with `*.ingress.host` set to your subdomains.  
4. `tls.yaml` with `global.ingress.annotations.cert-manager.io/cluster-issuer=letsencrypt-prod` and TLS enabled.  
5. `helm upgrade --install ess …` with both files.  
6. `kubectl get certificate -n ess` → `READY=True`.  
7. `/.well-known` returns correct JSON; federation tester OK.  
8. Create admin via MAS CLI; log in at `https://chat.chans.xyz`.  
9. Configure SMTP for MAS (and optionally Synapse), fix Mailcow sender policy if needed.