add incremental enrich controls with progress and retry limits
This commit is contained in:
@@ -9,6 +9,8 @@ A Python-based CLI tool for managing personal video collections with a safety-fi
|
||||
- **Comprehensive Analysis**: Detect episode gaps and duplicate files
|
||||
- **Rich Metadata**: Extract video resolution, codec, duration, and bitrate
|
||||
- **Flexible Organization**: Customizable directory structure and naming templates
|
||||
- **Metadata Enrichment**: Add bilingual titles and reputation signals (TMDB + optional Douban + optional AI fallback)
|
||||
- **Incremental Performance**: SQLite-backed cache avoids repeated metadata lookups
|
||||
- **State Tracking**: Track file status throughout the workflow
|
||||
- **Detailed Reporting**: Generate inventory, completeness, and duplicate reports
|
||||
|
||||
@@ -72,7 +74,26 @@ This creates `identities.json` with parsed information.
|
||||
- **Series**: Title, season, and episode numbers (e.g., "Breaking Bad S01E01")
|
||||
- **Confidence scores**: Indicates parsing reliability
|
||||
|
||||
### 4. Analyze Your Library
|
||||
### 4. Enrich Titles and Reputation (Optional but Recommended)
|
||||
|
||||
Add translation and reputation metadata to `identities.json`:
|
||||
|
||||
```bash
|
||||
vlm enrich
|
||||
```
|
||||
|
||||
This updates `identities.json` in place and adds fields like:
|
||||
- `title_zh`, `title_en`, `display_title`
|
||||
- `reputation_score`, `reputation_votes`, `reputation_source`
|
||||
- `review_status`, `enrichment_confidence`
|
||||
|
||||
To refresh all records instead of using incremental cache:
|
||||
|
||||
```bash
|
||||
vlm enrich --refresh-all
|
||||
```
|
||||
|
||||
### 5. Analyze Your Library
|
||||
|
||||
Detect episode gaps and duplicates:
|
||||
|
||||
@@ -84,7 +105,7 @@ This creates `analysis.json` with:
|
||||
- Series with missing episodes
|
||||
- Duplicate files with quality comparison
|
||||
|
||||
### 5. Generate Execution Plan
|
||||
### 6. Generate Execution Plan
|
||||
|
||||
Create a reviewable plan of file operations:
|
||||
|
||||
@@ -96,7 +117,7 @@ This creates `plan.json` with proposed operations (move, rename, quarantine).
|
||||
|
||||
**Review the plan** by opening `plan.json` in your editor. You can edit it if needed.
|
||||
|
||||
### 6. Execute (Dry-Run First)
|
||||
### 7. Execute (Dry-Run First)
|
||||
|
||||
Preview what will happen without making changes:
|
||||
|
||||
@@ -112,7 +133,7 @@ vlm execute --confirm
|
||||
|
||||
**Important**: This creates a rollback log in `~/.vlm/rollback/` for reverting changes.
|
||||
|
||||
### 7. Rollback (If Needed)
|
||||
### 8. Rollback (If Needed)
|
||||
|
||||
If you need to undo the operations:
|
||||
|
||||
@@ -139,27 +160,31 @@ vlm scan
|
||||
vlm parse
|
||||
# Output: identities.json with parsed titles and episodes
|
||||
|
||||
# 4. Analyze for gaps and duplicates
|
||||
# 4. Enrich identities (translation + reputation)
|
||||
vlm enrich
|
||||
# Output: identities.json updated in place (incremental cache enabled)
|
||||
|
||||
# 5. Analyze for gaps and duplicates
|
||||
vlm analyze
|
||||
# Output: analysis.json with 5 series with gaps, 12 duplicate groups
|
||||
|
||||
# 5. Generate execution plan
|
||||
# 6. Generate execution plan
|
||||
vlm plan
|
||||
# Output: plan.json with 456 operations proposed
|
||||
|
||||
# 6. Review the plan
|
||||
# 7. Review the plan
|
||||
cat plan.json | less
|
||||
# or open in your editor
|
||||
|
||||
# 7. Dry-run to preview
|
||||
# 8. Dry-run to preview
|
||||
vlm execute
|
||||
# Shows what would happen without making changes
|
||||
|
||||
# 8. Execute with confirmation
|
||||
# 9. Execute with confirmation
|
||||
vlm execute --confirm
|
||||
# Actually performs the file operations
|
||||
|
||||
# 9. If needed, rollback
|
||||
# 10. If needed, rollback
|
||||
vlm rollback
|
||||
```
|
||||
|
||||
@@ -204,6 +229,25 @@ vlm parse
|
||||
vlm parse --input my_inventory.csv --output my_identities.json
|
||||
```
|
||||
|
||||
### Enrichment
|
||||
|
||||
```bash
|
||||
# Enrich identities in place (default: identities.json)
|
||||
vlm enrich
|
||||
|
||||
# Enrich custom file and write to another file
|
||||
vlm enrich --input my_identities.json --output enriched_identities.json
|
||||
|
||||
# Refresh changed records only (explicit incremental mode)
|
||||
vlm enrich --refresh-changed-only
|
||||
|
||||
# Force full refresh (ignore cache for all records)
|
||||
vlm enrich --refresh-all
|
||||
|
||||
# Tune request behavior
|
||||
vlm enrich --timeout 6 --retries 2
|
||||
```
|
||||
|
||||
### Analysis
|
||||
|
||||
```bash
|
||||
@@ -337,6 +381,30 @@ categories:
|
||||
movie: [movie, movies, films]
|
||||
series: [series, tv, shows]
|
||||
anime: [anime]
|
||||
|
||||
# Enrichment settings
|
||||
enrichment:
|
||||
enabled: true
|
||||
incremental: true
|
||||
refresh_mode: "manual"
|
||||
providers: [tmdb, douban]
|
||||
cache_db: "~/.vlm/enrichment_cache.db"
|
||||
max_concurrency: 6
|
||||
min_match_score: 0.75
|
||||
douban_endpoint: null
|
||||
translation:
|
||||
mode: "bidirectional"
|
||||
fallback_machine: true
|
||||
api_keys:
|
||||
tmdb: null
|
||||
douban: null
|
||||
openai: null
|
||||
reputation:
|
||||
min_votes: 50
|
||||
low_score_threshold: 6.0
|
||||
policy: "flag_for_review"
|
||||
naming:
|
||||
title_format: "{title_zh} {title_en}"
|
||||
```
|
||||
|
||||
### Template Variables
|
||||
@@ -414,6 +482,7 @@ vlm report summary
|
||||
# 1. Scan and parse
|
||||
vlm scan
|
||||
vlm parse
|
||||
vlm enrich
|
||||
|
||||
# 2. Analyze completeness
|
||||
vlm analyze
|
||||
@@ -428,6 +497,7 @@ vlm report completeness
|
||||
# 1. Scan and parse
|
||||
vlm scan
|
||||
vlm parse
|
||||
vlm enrich
|
||||
|
||||
# 2. Analyze for duplicates
|
||||
vlm analyze
|
||||
@@ -452,6 +522,7 @@ vlm execute
|
||||
# 2. Scan and parse
|
||||
vlm scan
|
||||
vlm parse
|
||||
vlm enrich
|
||||
|
||||
# 3. Generate plan
|
||||
vlm plan
|
||||
@@ -580,6 +651,9 @@ src/vlm/
|
||||
├── cli.py # Click-based CLI interface
|
||||
├── scanner.py # File discovery and metadata extraction
|
||||
├── parser.py # Filename parsing (titles, years, episodes)
|
||||
├── enrichment.py # Title/reputation enrichment pipeline
|
||||
├── cache.py # SQLite cache for incremental enrichment
|
||||
├── providers/ # External metadata providers (TMDB/Douban)
|
||||
├── analysis.py # Completeness and duplicate detection
|
||||
├── planner.py # Execution plan generation
|
||||
├── executor.py # File operations and rollback
|
||||
@@ -593,6 +667,7 @@ src/vlm/
|
||||
tests/
|
||||
├── test_scanner.py # Scanner tests
|
||||
├── test_parser.py # Parser tests
|
||||
├── test_enrichment.py # Enrichment tests
|
||||
├── test_analysis.py # Analysis tests
|
||||
├── test_planner.py # Planner tests
|
||||
└── ... # More test files
|
||||
|
||||
Reference in New Issue
Block a user