435 lines
13 KiB
Markdown
435 lines
13 KiB
Markdown
|
---
|
|||
|
|
title: 混合搜索系统
|
|||
|
|
created: 2026-04-13
|
|||
|
|
updated: 2026-04-13
|
|||
|
|
type: concept
|
|||
|
|
tags: [knowledge-management, hybrid-search, bm25, vector-search, knowledge-graph]
|
|||
|
|
confidence: 0.9
|
|||
|
|
sources_count: 3
|
|||
|
|
last_confirmed: 2026-04-13
|
|||
|
|
status: active
|
|||
|
|
relationships:
|
|||
|
|
- target: SCHEMA.md
|
|||
|
|
type: defines
|
|||
|
|
detail: "LLM Wiki v2 搜索架构"
|
|||
|
|
confidence: 0.95
|
|||
|
|
- target: entities/index.md
|
|||
|
|
type: uses
|
|||
|
|
detail: "知识图谱遍历"
|
|||
|
|
confidence: 0.9
|
|||
|
|
- target: knowledge-management/wiki-operations.md
|
|||
|
|
type: integrates_with
|
|||
|
|
detail: "自动化搜索触发"
|
|||
|
|
confidence: 0.85
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 🎯 混合搜索系统
|
|||
|
|
|
|||
|
|
基于 **LLM Wiki v2** 的可扩展搜索架构，专为机场智能化工程 wiki（当前 89 页，预计增长至 200+ 页）设计。当传统 `index.md` 目录变得不可行时，混合搜索提供三层次检索融合。
|
|||
|
|
|
|||
|
|
> **核心理念**：单一检索方法无法覆盖所有查询场景。关键词匹配精准但缺乏语义理解，向量搜索理解语义但可能缺乏精确匹配，图谱遍历发现隐含关系但需要结构化数据。
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 🏗️ 三层检索架构
|
|||
|
|
|
|||
|
|
### 1️⃣ BM25 关键词检索
|
|||
|
|
**算法**：Okapi BM25（TF-IDF 的现代改进版）
|
|||
|
|
**用途**：精确术语匹配、技术参数查找、缩写搜索
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
# 伪代码实现
|
|||
|
|
def bm25_search(query: str, documents: List[str], k1=1.5, b=0.75):
|
|||
|
|
"""
|
|||
|
|
参数:
|
|||
|
|
- k1: 术语频率饱和度 (通常 1.2-2.0)
|
|||
|
|
- b: 文档长度归一化 (0-1, 通常 0.75)
|
|||
|
|
"""
|
|||
|
|
# 1. 分词 + 词干提取
|
|||
|
|
terms = stem(tokenize(query))
|
|||
|
|
|
|||
|
|
# 2. 计算每个文档的 BM25 分数
|
|||
|
|
scores = []
|
|||
|
|
for doc in documents:
|
|||
|
|
score = sum(
|
|||
|
|
idf(term) * (tf(term, doc) * (k1 + 1)) /
|
|||
|
|
(tf(term, doc) + k1 * (1 - b + b * len(doc)/avg_doc_len))
|
|||
|
|
for term in terms
|
|||
|
|
)
|
|||
|
|
scores.append(score)
|
|||
|
|
|
|||
|
|
return ranked_documents(scores)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**优势**：
|
|||
|
|
- ✅ 精确匹配技术术语（如 "InfiniBand NDR 400G"）
|
|||
|
|
- ✅ 支持同义词扩展（如 "GPU" → "图形处理器"）
|
|||
|
|
- ✅ 快速响应（毫秒级）
|
|||
|
|
- ✅ 可解释性强（高亮匹配术语）
|
|||
|
|
|
|||
|
|
**局限**：
|
|||
|
|
- ❌ 无法理解语义相似性（"智算中心" ≠ "数据中心"）
|
|||
|
|
- ❌ 对拼写错误敏感
|
|||
|
|
- ❌ 无法处理复杂概念组合
|
|||
|
|
|
|||
|
|
**机场场景示例**：
|
|||
|
|
```
|
|||
|
|
查询: "Tier IV 数据中心 PUE"
|
|||
|
|
BM25 匹配:
|
|||
|
|
- tier-iv-design.md (PUE < 1.2)
|
|||
|
|
- power-and-cooling.md (PUE 计算方式)
|
|||
|
|
- 机场智算中心技术方案.md (Tier IV 章节)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 2️⃣ 向量语义检索
|
|||
|
|
**模型**：`text-embedding-3-small` (OpenAI) 或 `BGE-M3` (开源)
|
|||
|
|
**维度**：1536 维向量空间
|
|||
|
|
**用途**：概念搜索、相似文档发现、跨语言检索
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
# 伪代码实现
|
|||
|
|
def vector_search(query: str, embeddings: Dict[str, List[float]], top_k=10):
|
|||
|
|
"""
|
|||
|
|
参数:
|
|||
|
|
- embeddings: {page_path: [vector]}
|
|||
|
|
- top_k: 返回 top K 结果
|
|||
|
|
"""
|
|||
|
|
# 1. 查询编码
|
|||
|
|
query_vec = embed_model.encode(query)
|
|||
|
|
|
|||
|
|
# 2. 计算余弦相似度
|
|||
|
|
similarities = []
|
|||
|
|
for page_path, page_vec in embeddings.items():
|
|||
|
|
sim = cosine_similarity(query_vec, page_vec)
|
|||
|
|
similarities.append((page_path, sim))
|
|||
|
|
|
|||
|
|
# 3. 返回 top K
|
|||
|
|
return sorted(similarities, key=lambda x: x[1], reverse=True)[:top_k]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**嵌入生成策略**：
|
|||
|
|
```python
|
|||
|
|
# 页面内容预处理
|
|||
|
|
def prepare_for_embedding(page_content: str) -> str:
|
|||
|
|
"""
|
|||
|
|
优化嵌入质量的预处理：
|
|||
|
|
1. 提取 frontmatter 关键字段 (title, tags, type)
|
|||
|
|
2. 保留正文前 2000 tokens（最重要的内容）
|
|||
|
|
3. 移除代码块、表格格式（保留纯文本）
|
|||
|
|
4. 标准化术语（统一缩写/全称）
|
|||
|
|
"""
|
|||
|
|
return processed_text
|
|||
|
|
|
|||
|
|
# 批量嵌入生成（每周更新）
|
|||
|
|
def regenerate_embeddings():
|
|||
|
|
for page in all_wiki_pages:
|
|||
|
|
content = read_page(page)
|
|||
|
|
text = prepare_for_embedding(content)
|
|||
|
|
embedding = embed_model.encode(text)
|
|||
|
|
save_embedding(page, embedding)
|
|||
|
|
|
|||
|
|
log("嵌入更新完成", timestamp=now())
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**优势**：
|
|||
|
|
- ✅ 理解语义相似性（"AI训练集群" ≈ "GPU计算农场"）
|
|||
|
|
- ✅ 支持模糊查询（拼写容错）
|
|||
|
|
- ✅ 发现相关但无关键词重叠的内容
|
|||
|
|
- ✅ 跨语言检索潜力
|
|||
|
|
|
|||
|
|
**局限**：
|
|||
|
|
- ❌ 无法精确匹配特定参数（如 "H100 功耗 700W"）
|
|||
|
|
- ❌ 需要定期重新计算嵌入（内容更新时）
|
|||
|
|
- ❌ 计算成本较高（API 调用或本地推理）
|
|||
|
|
|
|||
|
|
**机场场景示例**：
|
|||
|
|
```
|
|||
|
|
查询: "如何降低数据中心能耗"
|
|||
|
|
向量匹配:
|
|||
|
|
- liquid-cooling.md (液冷节能 40%)
|
|||
|
|
- tier-iv-design.md (PUE 优化)
|
|||
|
|
- modern-airport-trends.md (绿色机场趋势)
|
|||
|
|
- prefab-modular-dc.md (模块化节能)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 3️⃣ 知识图谱遍历检索
|
|||
|
|
**数据源**：`entities/index.md` + 页面 `relationships` 字段
|
|||
|
|
**算法**：图遍历（BFS/DFS）、路径查询、社区发现
|
|||
|
|
**用途**：关系发现、影响分析、生态系统查询
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
# 伪代码实现
|
|||
|
|
def graph_traversal_search(start_entity: str,
|
|||
|
|
relation_type: Optional[str] = None,
|
|||
|
|
max_depth: int = 3):
|
|||
|
|
"""
|
|||
|
|
从起点实体开始遍历知识图谱
|
|||
|
|
"""
|
|||
|
|
visited = set()
|
|||
|
|
results = []
|
|||
|
|
|
|||
|
|
def dfs(entity: str, depth: int, path: List[str]):
|
|||
|
|
if depth > max_depth or entity in visited:
|
|||
|
|
return
|
|||
|
|
|
|||
|
|
visited.add(entity)
|
|||
|
|
path.append(entity)
|
|||
|
|
|
|||
|
|
# 获取实体的所有关系
|
|||
|
|
relationships = get_relationships(entity)
|
|||
|
|
|
|||
|
|
for rel in relationships:
|
|||
|
|
if relation_type and rel.type != relation_type:
|
|||
|
|
continue
|
|||
|
|
|
|||
|
|
# 记录发现的关系路径
|
|||
|
|
results.append({
|
|||
|
|
"path": path.copy() + [rel.target],
|
|||
|
|
"relation": rel.type,
|
|||
|
|
"confidence": rel.confidence,
|
|||
|
|
"depth": depth + 1
|
|||
|
|
})
|
|||
|
|
|
|||
|
|
# 递归遍历
|
|||
|
|
dfs(rel.target, depth + 1, path.copy() + [rel.target])
|
|||
|
|
|
|||
|
|
dfs(start_entity, 0, [])
|
|||
|
|
return results
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**图谱查询类型**：
|
|||
|
|
1. **直接关系查询**：`find_related("郑州航空港区机场", relation_type="deploys")`
|
|||
|
|
2. **路径查找**：`find_path("NVIDIA", "华为", max_depth=3)`
|
|||
|
|
3. **社区发现**：`find_community("aodb-core", min_confidence=0.8)`
|
|||
|
|
4. **影响力分析**：`find_influencers("liquid-cooling", direction="upstream")`
|
|||
|
|
|
|||
|
|
**优势**：
|
|||
|
|
- ✅ 发现隐含关系（间接连接）
|
|||
|
|
- ✅ 理解系统依赖和影响链
|
|||
|
|
- ✅ 支持推理查询（"如果X故障，影响什么？"）
|
|||
|
|
- ✅ 可视化展示（关系图）
|
|||
|
|
|
|||
|
|
**局限**：
|
|||
|
|
- ❌ 依赖结构化数据质量
|
|||
|
|
- ❌ 需要手动维护关系（或自动提取）
|
|||
|
|
- ❌ 无法处理非实体内容（概念解释）
|
|||
|
|
|
|||
|
|
**机场场景示例**：
|
|||
|
|
```
|
|||
|
|
查询: "哪些机场使用ADB SAFEGATE的AODB"
|
|||
|
|
图谱遍历:
|
|||
|
|
起点: ADB SAFEGATE → provides → aodb-core
|
|||
|
|
遍历: aodb-core ← deploys ← [shenzhen-airport, jfk-airport, ...]
|
|||
|
|
结果: [深圳机场, 纽约肯尼迪机场, ...]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 🔄 结果融合策略
|
|||
|
|
|
|||
|
|
### 倒数排名融合（RRF）
|
|||
|
|
```python
|
|||
|
|
def reciprocal_rank_fusion(bm25_results: List[str],
|
|||
|
|
vector_results: List[str],
|
|||
|
|
graph_results: List[str],
|
|||
|
|
k: int = 60):
|
|||
|
|
"""
|
|||
|
|
RRF 公式: score = Σ(1 / (k + rank))
|
|||
|
|
- k: 平滑参数，通常 60
|
|||
|
|
- rank: 在单个列表中的排名 (1-based)
|
|||
|
|
"""
|
|||
|
|
|
|||
|
|
# 初始化得分字典
|
|||
|
|
scores = defaultdict(float)
|
|||
|
|
|
|||
|
|
# 处理 BM25 结果
|
|||
|
|
for rank, doc in enumerate(bm25_results, 1):
|
|||
|
|
scores[doc] += 1 / (k + rank)
|
|||
|
|
|
|||
|
|
# 处理向量结果
|
|||
|
|
for rank, doc in enumerate(vector_results, 1):
|
|||
|
|
scores[doc] += 1 / (k + rank)
|
|||
|
|
|
|||
|
|
# 处理图谱结果（可能需要转换实体→页面）
|
|||
|
|
for rank, entity_path in enumerate(graph_results, 1):
|
|||
|
|
# 将实体路径转换为相关页面
|
|||
|
|
pages = entity_path_to_pages(entity_path)
|
|||
|
|
for page in pages:
|
|||
|
|
scores[page] += 1 / (k + rank) / len(pages)
|
|||
|
|
|
|||
|
|
# 按总得分排序
|
|||
|
|
return sorted(scores.items(), key=lambda x: x[1], reverse=True)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 查询类型自适应权重
|
|||
|
|
| 查询类型 | BM25权重 | 向量权重 | 图谱权重 | 说明 |
|
|||
|
|
|----------|----------|----------|----------|------|
|
|||
|
|
| **技术参数** | 0.6 | 0.3 | 0.1 | 精确数字、规格、型号 |
|
|||
|
|
| **概念解释** | 0.3 | 0.6 | 0.1 | 定义、原理、背景 |
|
|||
|
|
| **关系查询** | 0.1 | 0.2 | 0.7 | 依赖、影响、连接 |
|
|||
|
|
| **综合搜索** | 0.4 | 0.4 | 0.2 | 默认权重分配 |
|
|||
|
|
|
|||
|
|
### 去重与多样化
|
|||
|
|
```python
|
|||
|
|
def diversify_results(merged_results: List[Tuple[str, float]],
|
|||
|
|
max_similar: float = 0.8):
|
|||
|
|
"""
|
|||
|
|
确保结果多样性，避免同质化
|
|||
|
|
"""
|
|||
|
|
diversified = []
|
|||
|
|
seen_content = set()
|
|||
|
|
|
|||
|
|
for doc, score in merged_results:
|
|||
|
|
# 计算与已选结果的相似度
|
|||
|
|
max_sim = 0
|
|||
|
|
for selected in diversified[:5]: # 与前5个比较
|
|||
|
|
sim = content_similarity(doc, selected)
|
|||
|
|
max_sim = max(max_sim, sim)
|
|||
|
|
|
|||
|
|
# 如果太相似，降低权重
|
|||
|
|
if max_sim > max_similar:
|
|||
|
|
adjusted_score = score * (1 - max_sim)
|
|||
|
|
else:
|
|||
|
|
adjusted_score = score
|
|||
|
|
|
|||
|
|
diversified.append((doc, adjusted_score))
|
|||
|
|
|
|||
|
|
return sorted(diversified, key=lambda x: x[1], reverse=True)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 🚀 实施路线图
|
|||
|
|
|
|||
|
|
### 阶段 1：基础 BM25 + 简易向量（当前）
|
|||
|
|
- ✅ Ripgrep 实现关键词搜索
|
|||
|
|
- ✅ 同义词词典扩展（`search/synonyms.txt`）
|
|||
|
|
- 🔄 OpenAI embeddings API 调用（按需）
|
|||
|
|
- 📊 搜索日志记录与分析
|
|||
|
|
|
|||
|
|
### 阶段 2：本地向量库 + 基础图谱（1-2周）
|
|||
|
|
- 🔄 本地嵌入模型部署（`BGE-M3` 或 `text-embedding-3-small`）
|
|||
|
|
- 🔄 每周批量嵌入更新
|
|||
|
|
- 🔄 实体关系图谱基础遍历
|
|||
|
|
- 📊 搜索结果质量评估框架
|
|||
|
|
|
|||
|
|
### 阶段 3：完整混合搜索 + 自动化（1个月）
|
|||
|
|
- 🔄 RRF 融合算法实现
|
|||
|
|
- 🔄 查询分类器（自动识别查询类型）
|
|||
|
|
- 🔄 图谱嵌入（Node2Vec 或 GraphSAGE）
|
|||
|
|
- 🔄 自动化搜索优化（基于用户反馈）
|
|||
|
|
|
|||
|
|
### 阶段 4：高级功能（未来）
|
|||
|
|
- 🔄 多语言检索支持
|
|||
|
|
- 🔄 时序搜索（基于 `updated` 日期）
|
|||
|
|
- 🔄 个性化排名（基于用户历史）
|
|||
|
|
- 🔄 可视化搜索界面
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 📋 技术栈建议
|
|||
|
|
|
|||
|
|
### 轻量级方案（Python 优先）
|
|||
|
|
```yaml
|
|||
|
|
bm25:
|
|||
|
|
- whoosh 或 tantivy (Python)
|
|||
|
|
- 同义词: pywsd 或 nltk.wordnet
|
|||
|
|
vector:
|
|||
|
|
- sentence-transformers (BGE-M3)
|
|||
|
|
- 或 OpenAI API (text-embedding-3-small)
|
|||
|
|
graph:
|
|||
|
|
- networkx (内存图)
|
|||
|
|
- 或 redisgraph (持久化)
|
|||
|
|
fusion:
|
|||
|
|
- 自定义 RRF 实现
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 生产级方案
|
|||
|
|
```yaml
|
|||
|
|
bm25:
|
|||
|
|
- Elasticsearch 或 Typesense
|
|||
|
|
vector:
|
|||
|
|
- Qdrant 或 Weaviate (向量数据库)
|
|||
|
|
graph:
|
|||
|
|
- Neo4j 或 Amazon Neptune
|
|||
|
|
fusion:
|
|||
|
|
- 自定义微服务或 LangChain
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 📊 性能指标与监控
|
|||
|
|
|
|||
|
|
### 搜索质量指标
|
|||
|
|
| 指标 | 计算方法 | 目标值 |
|
|||
|
|
|------|----------|--------|
|
|||
|
|
| **MRR** | Mean Reciprocal Rank | >0.6 |
|
|||
|
|
| **NDCG@10** | 归一化折损累计增益 | >0.7 |
|
|||
|
|
| **点击率** | 结果点击/展示 | >25% |
|
|||
|
|
| **查询分类准确率** | 自动分类准确率 | >85% |
|
|||
|
|
|
|||
|
|
### 性能指标
|
|||
|
|
| 指标 | 计算方法 | 目标值 |
|
|||
|
|
|------|----------|--------|
|
|||
|
|
| **P95 延迟** | 95% 查询响应时间 | <2s |
|
|||
|
|
| **吞吐量** | QPS (查询/秒) | >10 |
|
|||
|
|
| **缓存命中率** | 缓存结果/总查询 | >40% |
|
|||
|
|
| **嵌入新鲜度** | 嵌入更新延迟 | <7天 |
|
|||
|
|
|
|||
|
|
### 监控仪表板
|
|||
|
|
```python
|
|||
|
|
# 搜索日志格式
|
|||
|
|
search_log = {
|
|||
|
|
"query": "Tier IV PUE 标准",
|
|||
|
|
"query_type": "technical", # 自动分类
|
|||
|
|
"results_count": 15,
|
|||
|
|
"fusion_method": "rrf_k60",
|
|||
|
|
"response_time_ms": 1240,
|
|||
|
|
"components_timing": {
|
|||
|
|
"bm25": 120,
|
|||
|
|
"vector": 980,
|
|||
|
|
"graph": 140
|
|||
|
|
},
|
|||
|
|
"user_feedback": None, # 点击或评分
|
|||
|
|
"timestamp": "2026-04-13T10:30:00Z"
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 🔧 维护指南
|
|||
|
|
|
|||
|
|
### 每周维护任务
|
|||
|
|
1. **嵌入更新**：重新计算所有页面的向量嵌入
|
|||
|
|
2. **同义词更新**：根据搜索日志添加新同义词
|
|||
|
|
3. **图谱验证**：检查关系一致性和置信度衰减
|
|||
|
|
4. **性能分析**：分析慢查询，优化索引
|
|||
|
|
|
|||
|
|
### 每月优化任务
|
|||
|
|
1. **权重调整**：基于用户反馈调整融合权重
|
|||
|
|
2. **模型评估**：评估嵌入模型效果，考虑升级
|
|||
|
|
3. **查询分析**：识别常见查询模式，优化处理
|
|||
|
|
4. **容量规划**：预测增长，规划扩容
|
|||
|
|
|
|||
|
|
### 故障恢复
|
|||
|
|
```bash
|
|||
|
|
# 搜索系统故障恢复流程
|
|||
|
|
1. 降级到纯 BM25 搜索
|
|||
|
|
2. 禁用向量和图谱组件
|
|||
|
|
3. 检查嵌入存储完整性
|
|||
|
|
4. 逐步恢复各组件
|
|||
|
|
5. 验证搜索结果质量
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 📚 相关文档
|
|||
|
|
|
|||
|
|
- [[SCHEMA.md]] - LLM Wiki v2 架构定义
|
|||
|
|
- [[entities/index.md]] - 知识图谱数据源
|
|||
|
|
- [[knowledge-management/wiki-operations.md]] - 自动化维护
|
|||
|
|
- [[search-logs-analysis.md]] - 搜索日志分析报告
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
> **实施状态**：当前处于阶段 1（基础 BM25 + API 向量）。下一步：部署本地嵌入模型，实现每周批量更新。最后更新：2026-04-13。
|