add incremental enrich controls with progress and retry limits

This commit is contained in:
windyboy
2026-02-09 23:55:13 +08:00
parent 259e7506d7
commit 59a3b52fee
15 changed files with 1653 additions and 142 deletions
+85 -10
View File
@@ -9,6 +9,8 @@ A Python-based CLI tool for managing personal video collections with a safety-fi
- **Comprehensive Analysis**: Detect episode gaps and duplicate files
- **Rich Metadata**: Extract video resolution, codec, duration, and bitrate
- **Flexible Organization**: Customizable directory structure and naming templates
- **Metadata Enrichment**: Add bilingual titles and reputation signals (TMDB + optional Douban + optional AI fallback)
- **Incremental Performance**: SQLite-backed cache avoids repeated metadata lookups
- **State Tracking**: Track file status throughout the workflow
- **Detailed Reporting**: Generate inventory, completeness, and duplicate reports
@@ -72,7 +74,26 @@ This creates `identities.json` with parsed information.
- **Series**: Title, season, and episode numbers (e.g., "Breaking Bad S01E01")
- **Confidence scores**: Indicates parsing reliability
### 4. Analyze Your Library
### 4. Enrich Titles and Reputation (Optional but Recommended)
Add translation and reputation metadata to `identities.json`:
```bash
vlm enrich
```
This updates `identities.json` in place and adds fields like:
- `title_zh`, `title_en`, `display_title`
- `reputation_score`, `reputation_votes`, `reputation_source`
- `review_status`, `enrichment_confidence`
To refresh all records instead of using incremental cache:
```bash
vlm enrich --refresh-all
```
### 5. Analyze Your Library
Detect episode gaps and duplicates:
@@ -84,7 +105,7 @@ This creates `analysis.json` with:
- Series with missing episodes
- Duplicate files with quality comparison
### 5. Generate Execution Plan
### 6. Generate Execution Plan
Create a reviewable plan of file operations:
@@ -96,7 +117,7 @@ This creates `plan.json` with proposed operations (move, rename, quarantine).
**Review the plan** by opening `plan.json` in your editor. You can edit it if needed.
### 6. Execute (Dry-Run First)
### 7. Execute (Dry-Run First)
Preview what will happen without making changes:
@@ -112,7 +133,7 @@ vlm execute --confirm
**Important**: This creates a rollback log in `~/.vlm/rollback/` for reverting changes.
### 7. Rollback (If Needed)
### 8. Rollback (If Needed)
If you need to undo the operations:
@@ -139,27 +160,31 @@ vlm scan
vlm parse
# Output: identities.json with parsed titles and episodes
# 4. Analyze for gaps and duplicates
# 4. Enrich identities (translation + reputation)
vlm enrich
# Output: identities.json updated in place (incremental cache enabled)
# 5. Analyze for gaps and duplicates
vlm analyze
# Output: analysis.json with 5 series with gaps, 12 duplicate groups
# 5. Generate execution plan
# 6. Generate execution plan
vlm plan
# Output: plan.json with 456 operations proposed
# 6. Review the plan
# 7. Review the plan
cat plan.json | less
# or open in your editor
# 7. Dry-run to preview
# 8. Dry-run to preview
vlm execute
# Shows what would happen without making changes
# 8. Execute with confirmation
# 9. Execute with confirmation
vlm execute --confirm
# Actually performs the file operations
# 9. If needed, rollback
# 10. If needed, rollback
vlm rollback
```
@@ -204,6 +229,25 @@ vlm parse
vlm parse --input my_inventory.csv --output my_identities.json
```
### Enrichment
```bash
# Enrich identities in place (default: identities.json)
vlm enrich
# Enrich custom file and write to another file
vlm enrich --input my_identities.json --output enriched_identities.json
# Refresh changed records only (explicit incremental mode)
vlm enrich --refresh-changed-only
# Force full refresh (ignore cache for all records)
vlm enrich --refresh-all
# Tune request behavior
vlm enrich --timeout 6 --retries 2
```
### Analysis
```bash
@@ -337,6 +381,30 @@ categories:
movie: [movie, movies, films]
series: [series, tv, shows]
anime: [anime]
# Enrichment settings
enrichment:
enabled: true
incremental: true
refresh_mode: "manual"
providers: [tmdb, douban]
cache_db: "~/.vlm/enrichment_cache.db"
max_concurrency: 6
min_match_score: 0.75
douban_endpoint: null
translation:
mode: "bidirectional"
fallback_machine: true
api_keys:
tmdb: null
douban: null
openai: null
reputation:
min_votes: 50
low_score_threshold: 6.0
policy: "flag_for_review"
naming:
title_format: "{title_zh} {title_en}"
```
### Template Variables
@@ -414,6 +482,7 @@ vlm report summary
# 1. Scan and parse
vlm scan
vlm parse
vlm enrich
# 2. Analyze completeness
vlm analyze
@@ -428,6 +497,7 @@ vlm report completeness
# 1. Scan and parse
vlm scan
vlm parse
vlm enrich
# 2. Analyze for duplicates
vlm analyze
@@ -452,6 +522,7 @@ vlm execute
# 2. Scan and parse
vlm scan
vlm parse
vlm enrich
# 3. Generate plan
vlm plan
@@ -580,6 +651,9 @@ src/vlm/
├── cli.py # Click-based CLI interface
├── scanner.py # File discovery and metadata extraction
├── parser.py # Filename parsing (titles, years, episodes)
├── enrichment.py # Title/reputation enrichment pipeline
├── cache.py # SQLite cache for incremental enrichment
├── providers/ # External metadata providers (TMDB/Douban)
├── analysis.py # Completeness and duplicate detection
├── planner.py # Execution plan generation
├── executor.py # File operations and rollback
@@ -593,6 +667,7 @@ src/vlm/
tests/
├── test_scanner.py # Scanner tests
├── test_parser.py # Parser tests
├── test_enrichment.py # Enrichment tests
├── test_analysis.py # Analysis tests
├── test_planner.py # Planner tests
└── ... # More test files