2026-08-21 17:12:45 +08:00
|
|
|
|
---
|
|
|
|
|
|
type: moc
|
|
|
|
|
|
tags:
|
|
|
|
|
|
- llm-evaluation
|
|
|
|
|
|
- learning-zone
|
|
|
|
|
|
- moc
|
|
|
|
|
|
status: active
|
|
|
|
|
|
created: 2026-08-21
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
# 04-Reference · LLM Evaluation 资源地图
|
|
|
|
|
|
|
|
|
|
|
|
本目录不是"收藏夹"，而是**评测工程能力地图**。
|
|
|
|
|
|
每个资源都对应明确的能力点，并标明「为什么看、什么时候看、重点看什么」。
|
|
|
|
|
|
|
|
|
|
|
|
## 五层能力框架
|
|
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
|
1. Evaluation Methodology
|
|
|
|
|
|
↓
|
|
|
|
|
|
2. Benchmark & Reproducibility
|
|
|
|
|
|
↓
|
|
|
|
|
|
3. System / Agent Evaluation
|
|
|
|
|
|
↓
|
|
|
|
|
|
4. Continuous Evaluation / CI
|
|
|
|
|
|
↓
|
|
|
|
|
|
5. Safety / Sandbox / Environment
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
对应到本仓库：
|
|
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
|
00-Foundations ← 方法论基础
|
|
|
|
|
|
01-Getting-Started ← 为什么做评测
|
|
|
|
|
|
02-Practical-Roadmap ← 工程路线
|
|
|
|
|
|
03-Practice ← 自己动手的 Lab
|
|
|
|
|
|
04-Reference ← 本目录（深度参考）
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 推荐学习顺序（只精读 4 个）
|
|
|
|
|
|
|
|
|
|
|
|
1. **AWS Generative AI Evaluations Workshop**
|
|
|
|
|
|
→ 先看实际 Eval 长什么样（RAG / Tool Calling / Multi-Agent / Red Teaming）
|
|
|
|
|
|
|
|
|
|
|
|
2. **EleutherAI lm-evaluation-harness**
|
|
|
|
|
|
→ 理解标准化 Task、Prompt 固定、Metric 配置、去污染机制
|
|
|
|
|
|
|
|
|
|
|
|
3. **Inspect AI + AISI Engineering Playbook**
|
|
|
|
|
|
→ 现代 Evaluation Framework 的抽象（Task / Solver / Scorer / Sandbox / Trace）
|
|
|
|
|
|
|
|
|
|
|
|
4. **Ai2 OLMES / olmo-eval**
|
|
|
|
|
|
→ 如何把 Evaluation Protocol 真正冻结，实现可复现比较
|
|
|
|
|
|
|
|
|
|
|
|
后续按需：
|
|
|
|
|
|
- CircleCI + RAGAS → Continuous Evaluation
|
|
|
|
|
|
- BenchFlow → Environment-based Agent Evaluation
|
|
|
|
|
|
|
|
|
|
|
|
## 文件索引
|
|
|
|
|
|
|
|
|
|
|
|
| 文件 | 对应能力 | 核心资源 |
|
|
|
|
|
|
|------|----------|----------|
|
|
|
|
|
|
| [[01-Evaluation-Infrastructure\|01-Evaluation-Infrastructure]] | Harness / Sandbox / Scaling | Inspect AI, AISI Playbook, AWS Workshop |
|
|
|
|
|
|
| [[02-Benchmark-and-Reproducibility\|02-Benchmark-and-Reproducibility]] | 标准化 / 去污染 / 可复现 | lm-evaluation-harness, OLMES |
|
|
|
|
|
|
| [[03-Continuous-Evaluation\|03-Continuous-Evaluation]] | CI Gate / Experiment Tracking | RAGAS + CircleCI, W&B |
|
|
|
|
|
|
| [[04-Agent-Safety-and-Environments\|04-Agent-Safety-and-Environments]] | Tool Use / Sandbox / Trajectory | Inspect Sandbox, BenchFlow |
|
|
|
|
|
|
| [[05-Source-Reading-Checklist\|05-Source-Reading-Checklist]] | 统一阅读方法论 | 固定的 8 个问题 |
|
|
|
|
|
|
| [[04-Reference/evaluation-guidebook/00-Overview\|evaluation-guidebook]] | 方法论补充 | Hugging Face 官方 Guidebook |
|
2026-08-21 17:25:12 +08:00
|
|
|
|
| [[04-Reference/archive/00-Material-List\|archive/]] | 早期草案与外部资源归档 | 索引见 00-Material-List |
|
2026-08-21 17:12:45 +08:00
|
|
|
|
|
|
|
|
|
|
## 原则
|
|
|
|
|
|
|
|
|
|
|
|
- 优先官方 docs、源码、Design Doc
|
|
|
|
|
|
- 二手博客 / 课程宣传只做导航，不作为主线
|
|
|
|
|
|
- 每读一个框架，强制用 [[05-Source-Reading-Checklist\|05-Source-Reading-Checklist]] 的 8 个问题对照
|
