36 lines
961 B
Markdown
36 lines
961 B
Markdown
|
---
|
||
|
|
type: reference
|
||
|
|
tags:
|
||
|
|
- llm-evaluation
|
||
|
|
- continuous-evaluation
|
||
|
|
- ci
|
||
|
|
status: active
|
||
|
|
created: 2026-08-21
|
||
|
|
---
|
||
|
|
|
||
|
|
# 03 · Continuous Evaluation
|
||
|
|
|
||
|
|
主题：把评测从「手工跑一次」升级为「每次变更都会跑」
|
||
|
|
|
||
|
|
## A 级资源
|
||
|
|
|
||
|
|
### 1. CircleCI + RAGAS Continuous Eval
|
||
|
|
- 价值不在 RAGAS 本身，而在：
|
||
|
|
Eval → CI → Threshold → Gate
|
||
|
|
- 真正要学习的是：
|
||
|
|
Metric → Baseline → Threshold → Action
|
||
|
|
例如：
|
||
|
|
- security_failure > 0 → fail
|
||
|
|
- overall_pass_rate ↓ → review
|
||
|
|
- latency regression → review
|
||
|
|
- RAGAS 只是 grader/metric 工具
|
||
|
|
- 适合阶段：项目已有 regression set 后
|
||
|
|
|
||
|
|
### 2. DeepLearning.AI + Weights & Biases
|
||
|
|
- 真正适合学习的：
|
||
|
|
Experiment Tracking / Run / Artifact / Version / Trace / Comparison
|
||
|
|
- 建议用途：当你的目录开始出现
|
||
|
|
runs/、runs-v2/、runs-final/、runs-final-new/
|
||
|
|
说明该学 experiment tracking 了
|
||
|
|
- 不建议：一开始就为了"专业"搭 W&B
|