33 lines
874 B
Markdown
33 lines
874 B
Markdown
|
---
|
||
|
|
type: reference
|
||
|
|
tags:
|
||
|
|
- llm-evaluation
|
||
|
|
- methodology
|
||
|
|
- source-reading
|
||
|
|
status: active
|
||
|
|
created: 2026-08-21
|
||
|
|
---
|
||
|
|
|
||
|
|
# 05 · 源码阅读检查清单
|
||
|
|
|
||
|
|
每看到一个新框架，强制回答以下 8 个问题：
|
||
|
|
|
||
|
|
1. Test case 怎么表示？ → Dataset schema
|
||
|
|
2. Task 怎么定义？ → Task abstraction
|
||
|
|
3. 被测系统怎么调用？ → Model / Agent adapter
|
||
|
|
4. 多次运行如何表达？ → Trial / seed / sampling
|
||
|
|
5. Trace 怎么保存？ → Logging / trajectory
|
||
|
|
6. Success 怎么定义？ → Grader / Scorer
|
||
|
|
7. 结果怎么聚合？ → Metrics / Report
|
||
|
|
8. Regression 怎么做？ → Suite / CI / Version
|
||
|
|
|
||
|
|
最终你会发现：
|
||
|
|
Inspect、lm-eval、OLMES、AWS Workshop、RAGAS
|
||
|
|
表面 API 完全不同，但底层都绕不开：
|
||
|
|
|
||
|
|
```text
|
||
|
|
Task → Dataset → Runner → Output / Trace → Grader → Metrics → Report
|
||
|
|
```
|
||
|
|
|
||
|
|
这才是最值得学的东西。
|