reconcile LLM_Evaluation zone: align stage numbering and scopes, standardize full-path wikilinks, slim old guidebook notes, dedupe templates
- fix stage-numbering conflict (README vs 05-Progress) and unify stage-1 reading scope - resolve AWS workshop prerequisite contradiction in 04-Reference/01 - convert medium-path wikilinks to vault-root paths (~30 links), fix .pyy typos, annotate ragas fork, unify archive status, add 01-/02- README hubs - compress old-version guidebook notes (01, 05) into pointers; add 2026 reading guidance to 00-Overview - dedupe project templates and remove embedded template copy in 03-Practice/README
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
---
|
||||
type: project
|
||||
status: active
|
||||
---
|
||||
|
||||
# 03 · 运行与盲评
|
||||
|
||||
> 为什么盲评、最小记录格式见 [[02-Why-Guide]] 第 4 步;run 的 JSONL schema 见 [[01-LLM-Evaluation-Roadmap]] 第五节。
|
||||
|
||||
## 候选生成条件(对应工作表第 3 节)
|
||||
|
||||
| 项目 | A | B |
|
||||
|---|---|---|
|
||||
| 模型 / 来源 | | |
|
||||
| 系统提示词版本 | | |
|
||||
| 温度 / 生成设置 | | |
|
||||
| 运行日期 | | |
|
||||
| dataset_version | | |
|
||||
|
||||
> 重要:先把来源隐藏、随机标 A/B;完成全部判断前不要看真实来源。
|
||||
|
||||
## 盲评记录(对应工作表第 4 节)
|
||||
|
||||
| Case | A:有据性 | A:资料不足 | A:完成度 | B:有据性 | B:资料不足 | B:完成度 | 更优/平局 | 一句话理由 | 置信度 |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| 01 | | | | | | | | | | high/low |
|
||||
| 02 | | | | | | | | | | high/low |
|
||||
| 03 | | | | | | | | | | high/low |
|
||||
| 04 | | | | | | | | | | high/low |
|
||||
| 05 | | | | | | | | | | high/low |
|
||||
| 06 | | | | | | | | | | high/low |
|
||||
| 07 | | | | | | | | | | high/low |
|
||||
| 08 | | | | | | | | | | high/low |
|
||||
| 09 | | | | | | | | | | high/low |
|
||||
| 10 | | | | | | | | | | high/low |
|
||||
|
||||
### 低置信度归因(`low` 不是坏事,是最有价值的发现)
|
||||
|
||||
| Case | 为什么难判 | 下一步动作 |
|
||||
|---|---|---|
|
||||
| | rubric 模糊 / 文档不完整 / 问题有歧义 / 自己标错 | 改规则 / 补证据 / 移出自动评分 / 请人复核 |
|
||||
|
||||
## Run 记录(可选,脚本自动生成)
|
||||
|
||||
> 每次运行保存:case_id / system_version / model / temperature / trial / timestamp / output / grading。大输入只存 case_id + 版本号,不复制全文。
|
||||
|
||||
```json
|
||||
{
|
||||
"case_id": "rag-0001",
|
||||
"run": { "system_version": "", "model": "", "temperature": 0, "trial": 1, "timestamp": "" },
|
||||
"output": "",
|
||||
"grading": { "groundedness": "", "completeness": 0, "grader_version": "human-v0.1" }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
返回 [[01_Projects/Personal-Tech/LLM_Evaluation/03-Practice/README|项目实践入口]]。
|
||||
Reference in New Issue
Block a user