Fix identity reconstruction metadata loss with v2 schema
Implements identities.json v2 schema with embedded video metadata to fix duplicate resolution by quality, which previously failed due to VideoFile objects being reconstructed with hardcoded defaults (size_bytes=0, resolution=None, codec=None). Changes: - Add --inventory flag to vlm parse command to embed video metadata - Update _video_file_from_record() to extract embedded metadata if present - Add vlm_schema_version field to identities.json (v1.0 or v2.0) - Maintain backward compatibility with v1 files (no metadata) Schema v2 format: - Embeds video_metadata object in each record (movies/series) - Contains: size_bytes, modified_timestamp, resolution, codec, duration_seconds, bitrate_kbps - Enables accurate quality comparison during duplicate analysis Testing: - Added comprehensive unit tests for io.py functions - Added CLI integration tests for parse command - Added end-to-end tests for duplicate quality comparison - All 437 existing tests still pass (1 pre-existing failure in executor) Documentation: - Updated README.md with --inventory usage examples - Updated CLAUDE.md with schema versioning details - Added workflow examples showing metadata embedding This fix resolves the critical P0 issue where duplicate resolution by_quality strategy failed completely due to missing video metadata in reconstructed VideoFile objects. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 4.5
parent
1f55eab304
commit
065195b83b
@@ -39,12 +39,31 @@ vlm config init
|
||||
|
||||
# Common workflow
|
||||
vlm scan # Discover files
|
||||
vlm parse # Extract identities
|
||||
vlm parse # Extract identities (v1 schema)
|
||||
vlm parse --inventory inventory.csv # Extract identities with embedded metadata (v2 schema, recommended)
|
||||
vlm enrich # (Optional) Enrich titles/reputation via TMDB
|
||||
vlm enrich --refresh-all # Force full refresh (ignore cache)
|
||||
vlm analyze # Detect gaps/duplicates
|
||||
vlm plan # Generate execution plan
|
||||
vlm plan --analysis analysis.json # Generate plan with duplicate resolution
|
||||
vlm execute # Dry-run (default)
|
||||
vlm execute --confirm # Actually execute
|
||||
vlm rollback # Undo executed operations
|
||||
|
||||
# Reporting
|
||||
vlm report summary # Overview statistics
|
||||
vlm report inventory # File inventory
|
||||
vlm report completeness # Series with missing episodes
|
||||
vlm report duplicates # Duplicate files with quality comparison
|
||||
|
||||
# Quarantine management
|
||||
vlm quarantine list # List quarantined files
|
||||
vlm quarantine add <file> --reason "duplicate"
|
||||
vlm quarantine restore <file>
|
||||
|
||||
# State management
|
||||
vlm state show <file>
|
||||
vlm state set <file> --status reviewed
|
||||
```
|
||||
|
||||
## Architecture
|
||||
@@ -53,6 +72,8 @@ vlm execute --confirm # Actually execute
|
||||
VLM follows a read-first, multi-stage pipeline:
|
||||
1. **Scan** → discovers video files, extracts metadata via ffprobe (optional), saves to inventory.csv
|
||||
2. **Parse** → extracts titles/years/seasons/episodes from filenames, saves to identities.json
|
||||
- Use `--inventory inventory.csv` to embed video metadata (v2 schema) for accurate duplicate resolution by quality
|
||||
- Without `--inventory`, produces v1 schema (lightweight, no embedded metadata)
|
||||
3. **Enrich** (optional) → adds bilingual titles and reputation (TMDB); updates identities.json in place; uses SQLite cache for incremental runs
|
||||
4. **Analyze** → detects episode gaps and duplicates, saves to analysis.json
|
||||
5. **Plan** → generates reviewable execution plan (plan.json) with file operations
|
||||
@@ -143,6 +164,27 @@ This allows files in `/library/movies/` or `/library/films/` to be recognized as
|
||||
|
||||
**Migration Note:** If you have existing directories with non-standard names (like "movies" or "tv"), update your config.yaml and re-run `vlm scan` to fix categorization. No files will be moved.
|
||||
|
||||
### Schema Versioning
|
||||
|
||||
**identities.json Schema Versions:**
|
||||
|
||||
- **v1** (default without `--inventory`): Lightweight schema without embedded metadata
|
||||
- Records contain: path, filename, category, title, year/season/episodes, confidence, needs_review
|
||||
- VideoFile objects reconstructed with defaults: size_bytes=0, resolution=None, codec=None
|
||||
- Suitable for basic organization workflows
|
||||
|
||||
- **v2** (with `--inventory`): Enhanced schema with embedded video metadata
|
||||
- All v1 fields plus `video_metadata` object containing:
|
||||
- size_bytes, modified_timestamp, resolution, codec, duration_seconds, bitrate_kbps
|
||||
- Enables accurate duplicate resolution by quality (compare resolution, codec, file size)
|
||||
- Required for `by_quality` duplicate resolution strategy
|
||||
- Backward compatible: v1 files load without errors
|
||||
|
||||
**Implementation Details:**
|
||||
- `_video_file_from_record()` in io.py extracts embedded metadata if present
|
||||
- Parse command with `--inventory` flag loads inventory.csv and embeds metadata in output
|
||||
- Schema version stored in `vlm_schema_version` field at root level of identities.json
|
||||
|
||||
### File Discovery
|
||||
Uses system `find` command for speed, falls back to Python recursion if unavailable. Hidden paths (starting with `.`) are skipped automatically.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user