vault backup: 2026-02-24 07:55:02
This commit is contained in:
windyboy
+533
@@ -0,0 +1,533 @@
|
||||
---
|
||||
title: "tobi/qmd: mini cli search engine for your docs, knowledge bases, meeting notes, whatever. Tracking current sota approaches while being all local"
|
||||
source: "https://github.com/tobi/qmd"
|
||||
author:
|
||||
- "[[Agents]]"
|
||||
- "[[dgilperez]]"
|
||||
published:
|
||||
created: 2026-02-02
|
||||
description: "mini cli search engine for your docs, knowledge bases, meeting notes, whatever. Tracking current sota approaches while being all local - tobi/qmd"
|
||||
tags:
|
||||
- "clippings"
|
||||
- "webclipper"
|
||||
---
|
||||
> [!info] Source
|
||||
> URL: https://github.com/tobi/qmd
|
||||
> Title: tobi/qmd: mini cli search engine for your docs, knowledge bases, meeting notes, whatever. Tracking current sota approaches while being all local
|
||||
> Clipped:
|
||||
|
||||
**[qmd](https://github.com/tobi/qmd)** Public
|
||||
|
||||
mini cli search engine for your docs, knowledge bases, meeting notes, whatever. Tracking current sota approaches while being all local
|
||||
|
||||
[Open in github.dev](https://github.dev/) [Open in a new github.dev tab](https://github.dev/) [Open in codespace](https://github.com/codespaces/new/tobi/qmd?resume=1)
|
||||
|
||||
<table><thead><tr><th colspan="2"><span>Name</span></th><th colspan="1"><span>Name</span></th><th><p><span>Last commit message</span></p></th><th colspan="1"><p><span>Last commit date</span></p></th></tr></thead><tbody><tr><td colspan="3"><p><span><a href="https://github.com/tobi/qmd/commit/47b705409eb1427e574ce82c16e1860b216869ed">fix: BM25 score normalization - use Math.abs instead of Math.max (</a><a href="https://github.com/tobi/qmd/pull/76">#76</a><a href="https://github.com/tobi/qmd/commit/47b705409eb1427e574ce82c16e1860b216869ed">)</a></span></p><p><span><a href="https://github.com/tobi/qmd/commit/47b705409eb1427e574ce82c16e1860b216869ed">47b7054</a> ·</span></p><p><a href="https://github.com/tobi/qmd/commits/main/"><span><span><span>172 Commits</span></span></span></a></p></td></tr><tr><td colspan="2"><p><a href="https://github.com/tobi/qmd/tree/main/finetune">finetune</a></p></td><td colspan="1"><p><a href="https://github.com/tobi/qmd/tree/main/finetune">finetune</a></p></td><td></td><td></td></tr><tr><td colspan="2"><p><a href="https://github.com/tobi/qmd/tree/main/skills/qmd"><span>skills/</span> <span>qmd</span></a></p></td><td colspan="1"><p><a href="https://github.com/tobi/qmd/tree/main/skills/qmd"><span>skills/</span> <span>qmd</span></a></p></td><td><p><a href="https://github.com/tobi/qmd/commit/f6a987a642fccd2a3c6585811fdf92b6b61be2e2">Add skills.sh integration for AI agent discovery (</a><a href="https://github.com/tobi/qmd/pull/64">#64</a><a href="https://github.com/tobi/qmd/commit/f6a987a642fccd2a3c6585811fdf92b6b61be2e2">)</a></p></td><td></td></tr><tr><td colspan="2"><p><a href="https://github.com/tobi/qmd/tree/main/src">src</a></p></td><td colspan="1"><p><a href="https://github.com/tobi/qmd/tree/main/src">src</a></p></td><td><p><a href="https://github.com/tobi/qmd/commit/47b705409eb1427e574ce82c16e1860b216869ed">fix: BM25 score normalization - use Math.abs instead of Math.max (</a><a href="https://github.com/tobi/qmd/pull/76">#76</a><a href="https://github.com/tobi/qmd/commit/47b705409eb1427e574ce82c16e1860b216869ed">)</a></p></td><td></td></tr><tr><td colspan="2"><p><a href="https://github.com/tobi/qmd/tree/main/test">test</a></p></td><td colspan="1"><p><a href="https://github.com/tobi/qmd/tree/main/test">test</a></p></td><td><p><a href="https://github.com/tobi/qmd/commit/431f6e505ba2fc53f196da03b0580f3f7be59269">Fix qmd embed crash and resolve all TypeScript errors</a></p></td><td></td></tr><tr><td colspan="2"><p><a href="https://github.com/tobi/qmd/blob/main/.gitattributes">.gitattributes</a></p></td><td colspan="1"><p><a href="https://github.com/tobi/qmd/blob/main/.gitattributes">.gitattributes</a></p></td><td><p><a href="https://github.com/tobi/qmd/commit/99aee7190387483079358cf50b4cd152e607c2c3">Update get and multi-get commands for virtual paths</a></p></td><td></td></tr><tr><td colspan="2"><p><a href="https://github.com/tobi/qmd/blob/main/.gitignore">.gitignore</a></p></td><td colspan="1"><p><a href="https://github.com/tobi/qmd/blob/main/.gitignore">.gitignore</a></p></td><td><p><a href="https://github.com/tobi/qmd/commit/533f0eed372f03a07d46569ba3fcbca0b1d8cf4e">docs: add finetune CLAUDE.md and update training workflow</a></p></td><td></td></tr><tr><td colspan="2"><p><a href="https://github.com/tobi/qmd/blob/main/CLAUDE.md">CLAUDE.md</a></p></td><td colspan="1"><p><a href="https://github.com/tobi/qmd/blob/main/CLAUDE.md">CLAUDE.md</a></p></td><td><p><a href="https://github.com/tobi/qmd/commit/17c201ea8173e90962bfb64c4d3758f48cefb224">fix: correct QMD acronym to Query Markup Documents</a></p></td><td></td></tr><tr><td colspan="2"><p><a href="https://github.com/tobi/qmd/blob/main/README.md">README.md</a></p></td><td colspan="1"><p><a href="https://github.com/tobi/qmd/blob/main/README.md">README.md</a></p></td><td><p><a href="https://github.com/tobi/qmd/commit/17c201ea8173e90962bfb64c4d3758f48cefb224">fix: correct QMD acronym to Query Markup Documents</a></p></td><td></td></tr><tr><td colspan="2"><p><a href="https://github.com/tobi/qmd/blob/main/bun.lock">bun.lock</a></p></td><td colspan="1"><p><a href="https://github.com/tobi/qmd/blob/main/bun.lock">bun.lock</a></p></td><td><p><a href="https://github.com/tobi/qmd/commit/c85889df12d59a090e21bbe41a02144a7e251191">fixes</a></p></td><td></td></tr><tr><td colspan="2"><p><a href="https://github.com/tobi/qmd/blob/main/example-index.yml">example-index.yml</a></p></td><td colspan="1"><p><a href="https://github.com/tobi/qmd/blob/main/example-index.yml">example-index.yml</a></p></td><td><p><a href="https://github.com/tobi/qmd/commit/c85889df12d59a090e21bbe41a02144a7e251191">fixes</a></p></td><td></td></tr><tr><td colspan="3"></td></tr></tbody></table>
|
||||
|
||||
An on-device search engine for everything you need to remember. Index your markdown notes, meeting transcripts, documentation, and knowledge bases. Search with keywords or natural language. Ideal for your agentic flows.
|
||||
|
||||
QMD combines BM25 full-text search, vector semantic search, and LLM re-ranking—all running locally via node-llama-cpp with GGUF models.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```
|
||||
# Install globally
|
||||
bun install -g https://github.com/tobi/qmd
|
||||
|
||||
# Create collections for your notes, docs, and meeting transcripts
|
||||
qmd collection add ~/notes --name notes
|
||||
qmd collection add ~/Documents/meetings --name meetings
|
||||
qmd collection add ~/work/docs --name docs
|
||||
|
||||
# Add context to help with search results
|
||||
qmd context add qmd://notes "Personal notes and ideas"
|
||||
qmd context add qmd://meetings "Meeting transcripts and notes"
|
||||
qmd context add qmd://docs "Work documentation"
|
||||
|
||||
# Generate embeddings for semantic search
|
||||
qmd embed
|
||||
|
||||
# Search across everything
|
||||
qmd search "project timeline" # Fast keyword search
|
||||
qmd vsearch "how to deploy" # Semantic search
|
||||
qmd query "quarterly planning process" # Hybrid + reranking (best quality)
|
||||
|
||||
# Get a specific document
|
||||
qmd get "meetings/2024-01-15.md"
|
||||
|
||||
# Get a document by docid (shown in search results)
|
||||
qmd get "#abc123"
|
||||
|
||||
# Get multiple documents by glob pattern
|
||||
qmd multi-get "journals/2025-05*.md"
|
||||
|
||||
# Search within a specific collection
|
||||
qmd search "API" -c notes
|
||||
|
||||
# Export all matches for an agent
|
||||
qmd search "API" --all --files --min-score 0.3
|
||||
```
|
||||
|
||||
QMD's `--json` and `--files` output formats are designed for agentic workflows:
|
||||
|
||||
```
|
||||
# Get structured results for an LLM
|
||||
qmd search "authentication" --json -n 10
|
||||
|
||||
# List all relevant files above a threshold
|
||||
qmd query "error handling" --all --files --min-score 0.4
|
||||
|
||||
# Retrieve full document content
|
||||
qmd get "docs/api-reference.md" --full
|
||||
```
|
||||
|
||||
### MCP Server
|
||||
|
||||
Although the tool works perfectly fine when you just tell your agent to use it on the command line, it also exposes an MCP (Model Context Protocol) server for tighter integration.
|
||||
|
||||
**Tools exposed:**
|
||||
|
||||
- `qmd_search` - Fast BM25 keyword search (supports collection filter)
|
||||
- `qmd_vsearch` - Semantic vector search (supports collection filter)
|
||||
- `qmd_query` - Hybrid search with reranking (supports collection filter)
|
||||
- `qmd_get` - Retrieve document by path or docid (with fuzzy matching suggestions)
|
||||
- `qmd_multi_get` - Retrieve multiple documents by glob pattern, list, or docids
|
||||
- `qmd_status` - Index health and collection info
|
||||
|
||||
**Claude Desktop configuration** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
|
||||
|
||||
```
|
||||
{
|
||||
"mcpServers": {
|
||||
"qmd": {
|
||||
"command": "qmd",
|
||||
"args": ["mcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Claude Code configuration** (`~/.claude/settings.json`):
|
||||
|
||||
```
|
||||
{
|
||||
"mcpServers": {
|
||||
"qmd": {
|
||||
"command": "qmd",
|
||||
"args": ["mcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ QMD Hybrid Search Pipeline │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────┐
|
||||
│ User Query │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌──────────────┴──────────────┐
|
||||
▼ ▼
|
||||
┌────────────────┐ ┌────────────────┐
|
||||
│ Query Expansion│ │ Original Query│
|
||||
│ (fine-tuned) │ │ (×2 weight) │
|
||||
└───────┬────────┘ └───────┬────────┘
|
||||
│ │
|
||||
│ 2 alternative queries │
|
||||
└──────────────┬──────────────┘
|
||||
│
|
||||
┌───────────────────────┼───────────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Original Query │ │ Expanded Query 1│ │ Expanded Query 2│
|
||||
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
|
||||
│ │ │
|
||||
┌───────┴───────┐ ┌───────┴───────┐ ┌───────┴───────┐
|
||||
▼ ▼ ▼ ▼ ▼ ▼
|
||||
┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐
|
||||
│ BM25 │ │Vector │ │ BM25 │ │Vector │ │ BM25 │ │Vector │
|
||||
│(FTS5) │ │Search │ │(FTS5) │ │Search │ │(FTS5) │ │Search │
|
||||
└───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘
|
||||
│ │ │ │ │ │
|
||||
└───────┬───────┘ └──────┬──────┘ └──────┬──────┘
|
||||
│ │ │
|
||||
└────────────────────────┼───────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────┐
|
||||
│ RRF Fusion + Bonus │
|
||||
│ Original query: ×2 │
|
||||
│ Top-rank bonus: +0.05│
|
||||
│ Top 30 Kept │
|
||||
└───────────┬───────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────┐
|
||||
│ LLM Re-ranking │
|
||||
│ (qwen3-reranker) │
|
||||
│ Yes/No + logprobs │
|
||||
└───────────┬───────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────┐
|
||||
│ Position-Aware Blend │
|
||||
│ Top 1-3: 75% RRF │
|
||||
│ Top 4-10: 60% RRF │
|
||||
│ Top 11+: 40% RRF │
|
||||
└───────────────────────┘
|
||||
```
|
||||
|
||||
### Search Backends
|
||||
|
||||
| Backend | Raw Score | Conversion | Range |
|
||||
| --- | --- | --- | --- |
|
||||
| **FTS (BM25)** | SQLite FTS5 BM25 | `Math.abs(score)` | 0 to ~25+ |
|
||||
| **Vector** | Cosine distance | `1 / (1 + distance)` | 0.0 to 1.0 |
|
||||
| **Reranker** | LLM 0-10 rating | `score / 10` | 0.0 to 1.0 |
|
||||
|
||||
### Fusion Strategy
|
||||
|
||||
The `query` command uses **Reciprocal Rank Fusion (RRF)** with position-aware blending:
|
||||
|
||||
1. **Query Expansion**: Original query (×2 for weighting) + 1 LLM variation
|
||||
2. **Parallel Retrieval**: Each query searches both FTS and vector indexes
|
||||
3. **RRF Fusion**: Combine all result lists using `score = Σ(1/(k+rank+1))` where k=60
|
||||
4. **Top-Rank Bonus**: Documents ranking #1 in any list get +0.05, #2-3 get +0.02
|
||||
5. **Top-K Selection**: Take top 30 candidates for reranking
|
||||
6. **Re-ranking**: LLM scores each document (yes/no with logprobs confidence)
|
||||
7. **Position-Aware Blending**:
|
||||
- RRF rank 1-3: 75% retrieval, 25% reranker (preserves exact matches)
|
||||
- RRF rank 4-10: 60% retrieval, 40% reranker
|
||||
- RRF rank 11+: 40% retrieval, 60% reranker (trust reranker more)
|
||||
|
||||
**Why this approach**: Pure RRF can dilute exact matches when expanded queries don't match. The top-rank bonus preserves documents that score #1 for the original query. Position-aware blending prevents the reranker from destroying high-confidence retrieval results.
|
||||
|
||||
### Score Interpretation
|
||||
|
||||
| Score | Meaning |
|
||||
| --- | --- |
|
||||
| 0.8 - 1.0 | Highly relevant |
|
||||
| 0.5 - 0.8 | Moderately relevant |
|
||||
| 0.2 - 0.5 | Somewhat relevant |
|
||||
| 0.0 - 0.2 | Low relevance |
|
||||
|
||||
## Requirements
|
||||
|
||||
### System Requirements
|
||||
|
||||
- **Bun** >= 1.0.0
|
||||
- **macOS**: Homebrew SQLite (for extension support)
|
||||
```
|
||||
brew install sqlite
|
||||
```
|
||||
|
||||
QMD uses three local GGUF models (auto-downloaded on first use):
|
||||
|
||||
| Model | Purpose | Size |
|
||||
| --- | --- | --- |
|
||||
| `embeddinggemma-300M-Q8_0` | Vector embeddings | ~300MB |
|
||||
| `qwen3-reranker-0.6b-q8_0` | Re-ranking | ~640MB |
|
||||
| `qmd-query-expansion-1.7B-q4_k_m` | Query expansion (fine-tuned) | ~1.1GB |
|
||||
|
||||
Models are downloaded from HuggingFace and cached in `~/.cache/qmd/models/`.
|
||||
|
||||
## Installation
|
||||
|
||||
```
|
||||
bun install -g github:tobi/qmd
|
||||
```
|
||||
|
||||
Make sure `~/.bun/bin` is in your PATH.
|
||||
|
||||
### Development
|
||||
|
||||
```
|
||||
git clone https://github.com/tobi/qmd
|
||||
cd qmd
|
||||
bun install
|
||||
bun link
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Collection Management
|
||||
|
||||
```
|
||||
# Create a collection from current directory
|
||||
qmd collection add . --name myproject
|
||||
|
||||
# Create a collection with explicit path and custom glob mask
|
||||
qmd collection add ~/Documents/notes --name notes --mask "**/*.md"
|
||||
|
||||
# List all collections
|
||||
qmd collection list
|
||||
|
||||
# Remove a collection
|
||||
qmd collection remove myproject
|
||||
|
||||
# Rename a collection
|
||||
qmd collection rename myproject my-project
|
||||
|
||||
# List files in a collection
|
||||
qmd ls notes
|
||||
qmd ls notes/subfolder
|
||||
```
|
||||
```
|
||||
# Embed all indexed documents (800 tokens/chunk, 15% overlap)
|
||||
qmd embed
|
||||
|
||||
# Force re-embed everything
|
||||
qmd embed -f
|
||||
```
|
||||
|
||||
### Context Management
|
||||
|
||||
Context adds descriptive metadata to collections and paths, helping search understand your content.
|
||||
|
||||
### Search Commands
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ Search Modes │
|
||||
├──────────┬───────────────────────────────────────────────────────┤
|
||||
│ search │ BM25 full-text search only │
|
||||
│ vsearch │ Vector semantic search only │
|
||||
│ query │ Hybrid: FTS + Vector + Query Expansion + Re-ranking │
|
||||
└──────────┴───────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```
|
||||
# Search options
|
||||
-n <num> # Number of results (default: 5, or 20 for --files/--json)
|
||||
-c, --collection # Restrict search to a specific collection
|
||||
--all # Return all matches (use with --min-score to filter)
|
||||
--min-score <num> # Minimum score threshold (default: 0)
|
||||
--full # Show full document content
|
||||
--line-numbers # Add line numbers to output
|
||||
--index <name> # Use named index
|
||||
|
||||
# Output formats (for search and multi-get)
|
||||
--files # Output: docid,score,filepath,context
|
||||
--json # JSON output with snippets
|
||||
--csv # CSV output
|
||||
--md # Markdown output
|
||||
--xml # XML output
|
||||
|
||||
# Get options
|
||||
qmd get <file>[:line] # Get document, optionally starting at line
|
||||
-l <num> # Maximum lines to return
|
||||
--from <num> # Start from line number
|
||||
|
||||
# Multi-get options
|
||||
-l <num> # Maximum lines per file
|
||||
--max-bytes <num> # Skip files larger than N bytes (default: 10KB)
|
||||
```
|
||||
|
||||
### Output Format
|
||||
|
||||
Default output is colorized CLI format (respects `NO_COLOR` env):
|
||||
|
||||
```
|
||||
docs/guide.md:42 #a1b2c3
|
||||
Title: Software Craftsmanship
|
||||
Context: Work documentation
|
||||
Score: 93%
|
||||
|
||||
This section covers the **craftsmanship** of building
|
||||
quality software with attention to detail.
|
||||
See also: engineering principles
|
||||
|
||||
notes/meeting.md:15 #d4e5f6
|
||||
Title: Q4 Planning
|
||||
Context: Personal notes and ideas
|
||||
Score: 67%
|
||||
|
||||
Discussion about code quality and craftsmanship
|
||||
in the development process.
|
||||
```
|
||||
|
||||
- **Path**: Collection-relative path (e.g., `docs/guide.md`)
|
||||
- **Docid**: Short hash identifier (e.g., `#a1b2c3`) - use with `qmd get #a1b2c3`
|
||||
- **Title**: Extracted from document (first heading or filename)
|
||||
- **Context**: Path context if configured via `qmd context add`
|
||||
- **Score**: Color-coded (green >70%, yellow >40%, dim otherwise)
|
||||
- **Snippet**: Context around match with query terms highlighted
|
||||
|
||||
### Examples
|
||||
|
||||
```
|
||||
# Get 10 results with minimum score 0.3
|
||||
qmd query -n 10 --min-score 0.3 "API design patterns"
|
||||
|
||||
# Output as markdown for LLM context
|
||||
qmd search --md --full "error handling"
|
||||
|
||||
# JSON output for scripting
|
||||
qmd query --json "quarterly reports"
|
||||
|
||||
# Use separate index for different knowledge base
|
||||
qmd --index work search "quarterly reports"
|
||||
```
|
||||
|
||||
### Index Maintenance
|
||||
|
||||
```
|
||||
# Show index status and collections with contexts
|
||||
qmd status
|
||||
|
||||
# Re-index all collections
|
||||
qmd update
|
||||
|
||||
# Re-index with git pull first (for remote repos)
|
||||
qmd update --pull
|
||||
|
||||
# Get document by filepath (with fuzzy matching suggestions)
|
||||
qmd get notes/meeting.md
|
||||
|
||||
# Get document by docid (from search results)
|
||||
qmd get "#abc123"
|
||||
|
||||
# Get document starting at line 50, max 100 lines
|
||||
qmd get notes/meeting.md:50 -l 100
|
||||
|
||||
# Get multiple documents by glob pattern
|
||||
qmd multi-get "journals/2025-05*.md"
|
||||
|
||||
# Get multiple documents by comma-separated list (supports docids)
|
||||
qmd multi-get "doc1.md, doc2.md, #abc123"
|
||||
|
||||
# Limit multi-get to files under 20KB
|
||||
qmd multi-get "docs/*.md" --max-bytes 20480
|
||||
|
||||
# Output multi-get as JSON for agent processing
|
||||
qmd multi-get "docs/*.md" --json
|
||||
|
||||
# Clean up cache and orphaned data
|
||||
qmd cleanup
|
||||
```
|
||||
|
||||
## Data Storage
|
||||
|
||||
Index stored in: `~/.cache/qmd/index.sqlite`
|
||||
|
||||
### Schema
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `XDG_CACHE_HOME` | `~/.cache` | Cache directory location |
|
||||
|
||||
### Indexing Flow
|
||||
|
||||
```
|
||||
Collection ──► Glob Pattern ──► Markdown Files ──► Parse Title ──► Hash Content
|
||||
│ │ │
|
||||
│ │ ▼
|
||||
│ │ Generate docid
|
||||
│ │ (6-char hash)
|
||||
│ │ │
|
||||
└──────────────────────────────────────────────────►└──► Store in SQLite
|
||||
│
|
||||
▼
|
||||
FTS5 Index
|
||||
```
|
||||
|
||||
### Embedding Flow
|
||||
|
||||
Documents are chunked into 800-token pieces with 15% overlap:
|
||||
|
||||
```
|
||||
Document ──► Chunk (800 tokens) ──► Format each chunk ──► node-llama-cpp ──► Store Vectors
|
||||
│ "title | text" embedBatch()
|
||||
│
|
||||
└─► Chunks stored with:
|
||||
- hash: document hash
|
||||
- seq: chunk sequence (0, 1, 2...)
|
||||
- pos: character position in original
|
||||
```
|
||||
|
||||
```
|
||||
Query ──► LLM Expansion ──► [Original, Variant 1, Variant 2]
|
||||
│
|
||||
┌─────────┴─────────┐
|
||||
▼ ▼
|
||||
For each query: FTS (BM25)
|
||||
│ │
|
||||
▼ ▼
|
||||
Vector Search Ranked List
|
||||
│
|
||||
▼
|
||||
Ranked List
|
||||
│
|
||||
└─────────┬─────────┘
|
||||
▼
|
||||
RRF Fusion (k=60)
|
||||
Original query ×2 weight
|
||||
Top-rank bonus: +0.05/#1, +0.02/#2-3
|
||||
│
|
||||
▼
|
||||
Top 30 candidates
|
||||
│
|
||||
▼
|
||||
LLM Re-ranking
|
||||
(yes/no + logprob confidence)
|
||||
│
|
||||
▼
|
||||
Position-Aware Blend
|
||||
Rank 1-3: 75% RRF / 25% reranker
|
||||
Rank 4-10: 60% RRF / 40% reranker
|
||||
Rank 11+: 40% RRF / 60% reranker
|
||||
│
|
||||
▼
|
||||
Final Results
|
||||
```
|
||||
|
||||
## Model Configuration
|
||||
|
||||
Models are configured in `src/llm.ts` as HuggingFace URIs:
|
||||
|
||||
```
|
||||
const DEFAULT_EMBED_MODEL = "hf:ggml-org/embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf";
|
||||
const DEFAULT_RERANK_MODEL = "hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf";
|
||||
const DEFAULT_GENERATE_MODEL = "hf:tobil/qmd-query-expansion-1.7B-gguf/qmd-query-expansion-1.7B-q4_k_m.gguf";
|
||||
```
|
||||
|
||||
```
|
||||
// For queries
|
||||
"task: search result | query: {query}"
|
||||
|
||||
// For documents
|
||||
"title: {title} | text: {content}"
|
||||
```
|
||||
|
||||
### Qwen3-Reranker
|
||||
|
||||
Uses node-llama-cpp's `createRankingContext()` and `rankAndSort()` API for cross-encoder reranking. Returns documents sorted by relevance score (0.0 - 1.0).
|
||||
|
||||
Used for generating query variations via `LlamaChatSession`.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
## Releases
|
||||
|
||||
No releases published
|
||||
|
||||
## Packages
|
||||
|
||||
No packages published
|
||||
|
||||
## Languages
|
||||
|
||||
- [TypeScript 65.8%](https://github.com/tobi/qmd/search?l=typescript)
|
||||
- [Python 33.5%](https://github.com/tobi/qmd/search?l=python)
|
||||
- Other 0.7%
|
||||
@@ -0,0 +1,110 @@
|
||||
# 改进计划（基于评审意见，排除 CLAUDE-BOOTSTRAP.md 相关问题）
|
||||
|
||||
**计划日期**: 2026-02-03
|
||||
**适用范围**: README、QUICK_REFERENCE、WORKFLOWS、WEEKLY_REVIEW、package.json、06_Metadata 模板与附件管理文档
|
||||
**目标**: 统一入口与命令规范，明确工具调用方式，强化文档主线导航与可追溯性。
|
||||
|
||||
---
|
||||
|
||||
## 阶段 0：基线校准（1 天）
|
||||
|
||||
**目标**: 修正评审报告中的硬性数据与证据链。
|
||||
|
||||
- 更新 `06_Metadata/REVIEW_REPORT_2026-02-03.md` 的“索引概况”为实际值（当前索引为 639 个 Markdown 文档）。
|
||||
- 在“关键问题与风险”每条后追加“证据”行（文件路径 + 简短说明）。
|
||||
- 在“结论”中注明评审日期与索引日期一致。
|
||||
|
||||
**验收标准**:
|
||||
- 报告中所有关键问题具备可追溯证据。
|
||||
|
||||
---
|
||||
|
||||
## 阶段 1：入口统一（1-2 天）
|
||||
|
||||
**目标**: 明确权威入口，消除“并行主线”的认知冲突。
|
||||
|
||||
- 在 `README.md` 增加“权威入口”段落，明确：
|
||||
- `README.md` = 项目入口与导航
|
||||
- `QUICK_REFERENCE.md` = 速查
|
||||
- `06_Metadata/WORKFLOWS.md` = 详细工作流
|
||||
- 在 `QUICK_REFERENCE.md` 顶部增加“入口说明 + 返回导航”。
|
||||
- 在 `06_Metadata/WORKFLOWS.md` 顶部增加“与 README/Quick Reference 的关系说明”。
|
||||
|
||||
**验收标准**:
|
||||
- 三者之间互相可达，且入口职责清晰一致。
|
||||
|
||||
---
|
||||
|
||||
## 阶段 2：命令规范一致性（2-3 天）
|
||||
|
||||
**目标**: 统一文档命令示例与脚本规则的边界。
|
||||
|
||||
- 明确规则（选其一并写入文档）：
|
||||
- 方案 A：文档示例不使用管道、grep、find 等；脚本亦保持简化。
|
||||
- 方案 B：脚本允许复杂命令，文档中的“手动命令”禁止复杂命令，并明确例外。
|
||||
- 同步修订 `WEEKLY_REVIEW.md` 的示例命令，使其与规则一致。
|
||||
- 对 `package.json` 脚本在 README 或 WORKFLOWS 增加说明，避免规则冲突。
|
||||
|
||||
**验收标准**:
|
||||
- 文档中的命令示例与规则完全一致。
|
||||
- 不再出现“禁止管道但示例有管道”的矛盾。
|
||||
|
||||
---
|
||||
|
||||
## 阶段 3：工具调用统一（1 天）
|
||||
|
||||
**目标**: 统一 `pnpm`/`npm` 的使用方式。
|
||||
|
||||
- 统一所有文档的脚本调用为 `pnpm`。
|
||||
- 若保留兼容说明：增加一段“未安装 pnpm 可用 npm run”的规范化说明。
|
||||
- 修订 README 中出现的 `npm run` 示例，保持一致。
|
||||
|
||||
**验收标准**:
|
||||
- 文档中的脚本调用方式保持一致。
|
||||
|
||||
---
|
||||
|
||||
## 阶段 4：文档主线导航（1 天）
|
||||
|
||||
**目标**: 提供清晰的阅读与执行路径。
|
||||
|
||||
- 在 `README.md` 增加“新用户路径”和“维护者路径”。
|
||||
- 在 `QUICK_REFERENCE.md` 增加“常用路径导航”。
|
||||
|
||||
**验收标准**:
|
||||
- 新用户可以在 2-3 次跳转内找到完整工作流说明。
|
||||
|
||||
---
|
||||
|
||||
## 阶段 5：复审与维护（0.5-1 天）
|
||||
|
||||
**目标**: 验证改动有效、评分依据可复用。
|
||||
|
||||
- 对 README、QUICK_REFERENCE、WORKFLOWS、WEEKLY_REVIEW、package.json 做一次一致性复查。
|
||||
- 更新评审报告评分依据（补充评分标准或示例）。
|
||||
|
||||
**验收标准**:
|
||||
- 评审报告可被他人复核。
|
||||
|
||||
---
|
||||
|
||||
## 执行顺序建议
|
||||
|
||||
1. 阶段 0（校准）
|
||||
2. 阶段 1（入口）
|
||||
3. 阶段 2（命令规范）
|
||||
4. 阶段 3（工具统一）
|
||||
5. 阶段 4（导航）
|
||||
6. 阶段 5（复审）
|
||||
|
||||
---
|
||||
|
||||
## 产出文件清单
|
||||
|
||||
- `06_Metadata/IMPROVEMENT_PLAN_2026-02-03.md`（本计划）
|
||||
- 更新：`06_Metadata/REVIEW_REPORT_2026-02-03.md`
|
||||
- 更新：`README.md`
|
||||
- 更新：`QUICK_REFERENCE.md`
|
||||
- 更新：`06_Metadata/WORKFLOWS.md`
|
||||
- 更新：`WEEKLY_REVIEW.md`
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# {{date:YYYY-MM-DD}}
|
||||
|
||||
## Capture
|
||||
<!-- Quick thoughts, links, ideas throughout the day -->
|
||||
获取所有文件名和大小
|
||||
```
|
||||
Get-ChildItem -Recurse -File | Select-Object DirectoryName, Name, @{N='Size(MB)';E={"{0:N2}" -f ($_.Length/1MB)}}, LastWriteTime | Export-Csv -Path "all_files_list.csv" -NoTypeInformation -Encoding UTF8
|
||||
```
|
||||
|
||||
## Questions
|
||||
<!-- What am I curious about today? -->
|
||||
-
|
||||
|
||||
## Insights
|
||||
<!-- What did I learn or realize? -->
|
||||
-
|
||||
|
||||
## Connections
|
||||
<!-- Links to other notes or ideas -->
|
||||
-
|
||||
|
||||
## For Tomorrow
|
||||
<!-- What needs follow-up? -->
|
||||
-
|
||||
|
||||
---
|
||||
*End of day: Ask Claude Code to review and find connections*
|
||||
Reference in New Issue
Block a user