restructure 04-Reference: resource map for evaluation engineering (archive old drafts)

2026-08-21 17:12:45 +08:00
parent 0f25594088
commit 4523ce67b3
20 changed files with 277 additions and 3 deletions
@@ -0,0 +1,35 @@
+---
+type: reference
+tags:
+  - llm-evaluation
+  - continuous-evaluation
+  - ci
+status: active
+created: 2026-08-21
+---
+
+# 03 · Continuous Evaluation
+
+主题：把评测从「手工跑一次」升级为「每次变更都会跑」
+
+## A 级资源
+
+### 1. CircleCI + RAGAS Continuous Eval
+- 价值不在 RAGAS 本身，而在：
+  Eval → CI → Threshold → Gate
+- 真正要学习的是：
+  Metric → Baseline → Threshold → Action
+  例如：
+  - security_failure > 0 → fail
+  - overall_pass_rate ↓ → review
+  - latency regression → review
+- RAGAS 只是 grader/metric 工具
+- 适合阶段：项目已有 regression set 后
+
+### 2. DeepLearning.AI + Weights & Biases
+- 真正适合学习的：
+  Experiment Tracking / Run / Artifact / Version / Trace / Comparison
+- 建议用途：当你的目录开始出现
+  runs/、runs-v2/、runs-final/、runs-final-new/
+  说明该学 experiment tracking 了
+- 不建议：一开始就为了"专业"搭 W&B