windy/dl-organizer

Fork 0

Files

T

windyboy 0be802eac5 chore: snapshot current project updates

2026-02-13 13:36:39 +08:00

23 KiB

Raw Blame History

Video Library Manager

A Python-based CLI tool for managing personal video collections with a safety-first, human-in-the-loop approach.

Features

Safety-First Design: All file operations are reversible with rollback support
Human-in-the-Loop: Explicit confirmation required before making any changes
Comprehensive Analysis: Detect episode gaps and duplicate files
Rich Metadata: Extract video resolution, codec, duration, and bitrate
Flexible Organization: Customizable directory structure and naming templates
Metadata Enrichment: Add bilingual titles and reputation signals (TMDB + optional AI fallback)
Incremental Performance: SQLite-backed cache avoids repeated metadata lookups
State Tracking: Track file status throughout the workflow
Detailed Reporting: Generate inventory, completeness, and duplicate reports (reports can include plan content summary via --plan)
Plan–Analysis Integration: vlm plan --analysis applies duplicate resolution (keep by reputation, quarantine rest) and adds a Chinese human summary to the plan for quick review

Installation

This project uses uv for Python package management. To install:

# Install dependencies
uv pip install -e .

# Install with development dependencies
uv pip install -e ".[dev]"

Quick Start

1. Initialize Configuration

First, create a configuration file:

vlm config init

This creates ~/.vlm/config.yaml. Edit it to set your library root:

library_root: "/mnt/Downloads"  # Change this to your video library path

2. Scan Your Library

Discover all video files in your library:

vlm scan

This creates inventory.csv with all discovered files and their metadata.

What happens:

Discovers video files using the system find command
Extracts file metadata (size, modification time)
Categorizes files based on directory structure (movie/series/anime/other)
Extracts video metadata using ffprobe (if available)
Saves results to inventory.csv

3. Parse Filenames

Extract titles, years, seasons, and episodes from filenames:

vlm parse

For accurate duplicate resolution by quality, embed video metadata:

vlm parse --inventory inventory.csv

This creates identities.json with parsed information. When using --inventory, video metadata (size, resolution, codec) is embedded, enabling accurate quality comparison during duplicate analysis.

What it extracts:

Movies: Title and year (e.g., "Inception (2010)")
Series: Title, season, and episode numbers (e.g., "Breaking Bad S01E01")
Confidence scores: Indicates parsing reliability
Video metadata (with --inventory): Size, resolution, codec, duration, bitrate

4. Enrich Titles and Reputation (Optional but Recommended)

Add translation and reputation metadata to identities.json:

vlm enrich

This updates identities.json in place and adds fields like:

title_zh, title_en, display_title
reputation_score, reputation_votes, reputation_source
review_status, enrichment_confidence
summary metrics including api_calls, cache_hits, and skip_reasons

To refresh all records instead of using incremental cache:

vlm enrich --refresh-all

If enrichment cannot run for some records, CLI shows grouped reasons, for example:

Skip reasons: no_key=4632

5. Analyze Your Library

Detect episode gaps and duplicates:

vlm analyze

This creates analysis.json with:

Series with missing episodes
Duplicate files with quality comparison

6. Generate Execution Plan

Create a reviewable plan of file operations:

vlm plan

To let the plan automatically resolve duplicate groups (keep one file per group by reputation, quarantine the rest), pass the analysis file:

vlm plan --analysis analysis.json

This creates plan.json with:

Proposed operations (move, rename, quarantine, no-op)
Summary: counts by operation type and by reason
Human summary (中文): short narrative for quick review
Metadata: when using --analysis, duplicate groups considered and completeness gaps

Safety guards in planning:

Titles used in path templates are sanitized (path separators/control chars/.. stripped)
Any destination outside library_root is rejected as no-op

Duplicate keep strategy is configurable in ~/.vlm/config.yaml under plan.duplicate_keep:

by_quality - Prefer highest quality (resolution > source > codec > file size). Best for automatic duplicate resolution.
by_reputation - Prefer external rating (TMDB); when rating ties/missing, fallback to quality (source-first: BluRay > WEB-DL), then first-seen. Default.
by_reputation_quality_time - Prefer external rating first, then quality, then newer modified time.
first_seen - Keep the first file in each duplicate group.
manual - Do not generate quarantine operations; duplicates are listed in analysis only.

Review the plan by opening plan.json in your editor, or read the human summary when you run vlm execute. You can edit the plan JSON if needed.

7. Execute (Dry-Run First)

Preview what will happen without making changes:

vlm execute

Before running, the CLI prints the plan’s human summary (or a short summary from counts) so you can confirm at a glance. When ready to actually move/rename/quarantine files:

vlm execute --confirm

Important: This creates a rollback log in ~/.vlm/rollback/ for reverting changes.

Execution safeguards:

Even if a manually edited plan.json contains an unsafe destination, execution rejects paths outside library_root
Summary counters treat conflict skips separately from real failures (failed/skipped are mutually exclusive)

8. Rollback (If Needed)

If you need to undo the operations:

vlm rollback

This uses the most recent rollback log to restore files to their original locations.

Complete Workflow Example

Here's a complete workflow from start to finish:

# 1. Initialize configuration
vlm config init
# Edit ~/.vlm/config.yaml to set your library_root

# 2. Scan your library
vlm scan
# Output: inventory.csv with 1234 files discovered

# 3. Parse filenames with metadata embedding (recommended for duplicate resolution)
vlm parse --inventory inventory.csv
# Output: identities.json with parsed titles, episodes, and embedded video metadata (v2 schema)

# 4. Enrich identities (translation + reputation)
vlm enrich
# Output: identities.json updated in place (incremental cache enabled)

# 5. Analyze for gaps and duplicates
vlm analyze
# Output: analysis.json with 5 series with gaps, 12 duplicate groups (accurate quality comparison)

# 6. Generate execution plan (optionally use analysis for duplicate handling)
vlm plan --analysis analysis.json
# Output: plan.json with operations, human summary, and duplicate quarantine decisions

# 7. Review the plan
cat plan.json | less
# or open in your editor

# 8. Dry-run to preview
vlm execute
# Shows what would happen without making changes

# 9. Execute with confirmation
vlm execute --confirm
# Actually performs the file operations

# 10. If needed, rollback
vlm rollback

Command Reference

Configuration

# Initialize config file
vlm config init

# Show current configuration
vlm config show

# Validate configuration
vlm config validate

Scanning

# Scan with default output (inventory.csv)
vlm scan

# Scan with custom output file
vlm scan --output my_library.csv

Scan behavior:

Hidden paths are skipped (any component starting with .)
File matching uses configured video_extensions (default: .mp4, .mkv, .avi, .mov, .wmv, .flv, .webm, .m4v)
If ffprobe is available, resolution/codec/duration/bitrate are included
ffprobe extraction uses controlled concurrency (enrichment.max_concurrency)
Timestamps stored in UTC format (YYYY-MM-DDTHH:MM:SS)

Parsing

# Parse with default files (v1 schema - no metadata embedding)
vlm parse

# Parse with metadata embedding (v2 schema - enables quality comparison)
vlm parse --inventory inventory.csv

# Parse with custom input/output
vlm parse --input my_inventory.csv --output my_identities.json

# Parse with metadata from custom inventory
vlm parse --input my_inventory.csv --output my_identities.json --inventory my_inventory.csv

Schema Versions:

v1 (without --inventory): Lightweight identities, suitable for basic organization
v2 (with --inventory): Embeds video metadata, required for accurate duplicate resolution by quality

Enrichment

# Enrich identities in place (default: identities.json)
vlm enrich

# Enrich custom file and write to another file
vlm enrich --input my_identities.json --output enriched_identities.json

# Refresh changed records only (explicit incremental mode)
vlm enrich --refresh-changed-only

# Force full refresh (ignore cache for all records)
vlm enrich --refresh-all

# Tune request behavior
vlm enrich --timeout 6 --retries 2

Analysis

# Analyze with default files
vlm analyze

# Analyze with custom files
vlm analyze --input my_identities.json --output my_analysis.json

Planning

# Generate plan with default files
vlm plan

# Use analysis so duplicate groups become "keep one + quarantine rest" (by_reputation by default)
vlm plan --analysis analysis.json

# Custom input/output
vlm plan --input my_identities.json --output my_plan.json
vlm plan --input my_identities.json --analysis my_analysis.json --output my_plan.json

Execution

# Dry-run (preview only, no changes)
vlm execute

# Execute with confirmation (actually makes changes)
vlm execute --confirm

# Use custom plan file
vlm execute --plan my_plan.json --confirm

Rollback

# Rollback using most recent log
vlm rollback

# Rollback using specific log
vlm rollback --log ~/.vlm/rollback/rollback_<uuid>.json

Quarantine Management

# List quarantined files
vlm quarantine list

# List by category
vlm quarantine list --category movie

# Quarantine a file
vlm quarantine add /path/to/file.mkv --reason "duplicate"

# Restore from quarantine
vlm quarantine restore /path/to/.quarantine/file.mkv

Reporting

# Generate inventory report (text format)
vlm report inventory

# Generate in different formats
vlm report inventory --format csv
vlm report inventory --format json --output inventory_report.json

# Generate completeness report (series with gaps)
vlm report completeness

# Include plan content summary in the report (human_summary from plan.json)
vlm report completeness --plan plan.json

# Generate duplicates report
vlm report duplicates
vlm report duplicates --plan plan.json

# Generate summary statistics
vlm report summary

State Management

# Show state for a file
vlm state show /path/to/file.mkv

# Set state for a file
vlm state set /path/to/file.mkv --status reviewed
vlm state set /path/to/file.mkv --status ignored --reason "duplicate"

# Query files by status
vlm state query --status ignored

# Clear state for a file
vlm state clear /path/to/file.mkv

Configuration File

The configuration file (~/.vlm/config.yaml) controls VLM's behavior:

# Required: Root directory of your video library
library_root: "/mnt/nas/videos"

# Video file extensions to recognize
video_extensions:
  - .mp4
  - .mkv
  - .avi
  - .mov
  - .wmv
  - .flv
  - .webm
  - .m4v

# Directory structure templates
templates:
  movie_dir: "movie/{title} ({year})/"
  series_dir: "series/{title}/Season {season:02d}/"
  movie_filename: "{title} ({year}){ext}"
  series_filename: "S{season:02d}E{episode:02d}{ext}"

# Quarantine directory name (relative to category root)
quarantine_dir: ".quarantine"

# Logging level (DEBUG, INFO, WARNING, ERROR)
log_level: "INFO"

# Plan behavior (e.g. when using vlm plan --analysis)
plan:
  # Duplicate keep strategy: "by_quality", "by_reputation" (default),
  # "by_reputation_quality_time", "first_seen", or "manual"
  # by_quality: resolution > source (BluRay > WEB-DL) > codec (x265 > x264) > file size
  duplicate_keep: "by_reputation"

# Category mappings (directory name to category)
categories:
  movie: [movie, movies, films]
  series: [series, tv, shows]
  anime: [anime]

# Enrichment settings (`enrich` alias is also supported)
enrichment:
  enabled: true
  incremental: true
  refresh_mode: "manual"
  providers: [tmdb]
  cache_db: "~/.vlm/enrichment_cache.db"
  max_concurrency: 6  # used by enrichment requests and scan-time ffprobe workers
  min_match_score: 0.75
  translation:
    mode: "bidirectional"
    fallback_machine: true
  api_keys:
    # Preferred: TMDB v4 Bearer token
    tmdb_bearer: null
    # Backward-compatible fallback (legacy query api_key)
    tmdb: null
    openai: null
  tmdb:
    language: "zh-CN"
    region: null
    include_adult: false
  reputation:
    min_votes: 50
    low_score_threshold: 6.0
    policy: "flag_for_review"
  naming:
    title_format: "{title_zh} {title_en}"

TMDB Enrichment Setup

vlm enrich works best with TMDB Bearer auth (recommended by TMDB). Legacy tmdb api key is still supported for compatibility.

Minimal config:

enrichment:
  providers: [tmdb]
  api_keys:
    tmdb_bearer: "YOUR_TMDB_BEARER_TOKEN"

Optional TMDB query tuning:

enrichment:
  tmdb:
    language: "zh-CN"   # localized title language
    region: "US"        # affects regional release/search behavior
    include_adult: false

Validation flow:

# 1) run small incremental pass
vlm enrich --input identities.json --refresh-changed-only

# 2) then full refresh if output looks correct
vlm enrich --input identities.json --refresh-all

Template Variables

Movies:

{title} - Movie title
{year} - Release year
{ext} - File extension

Series:

{title} - Series title
{season} - Season number
{episode} - Episode number
{ext} - File extension

Format specifiers are supported (e.g., {season:02d} for zero-padded numbers).

File Categorization

VLM categorizes files based on the top-level directory in your library, matched against configured category mappings. By default:

movie/ → "movie"
series/ → "series"
anime/ → "anime"
Anything else → "other"

You can customize these mappings to support multiple directory names per category:

categories:
  movie: [movie, movies, films]
  series: [series, tv, shows, television]
  anime: [anime]

This allows directories like /library/movies/ or /library/tv/ to be correctly categorized. Directory matching is case-insensitive.

Example:

/mnt/Downloads/
├── movies/                          # Category: movie (plural form)
│   └── Inception (2010).mkv
├── tv/                              # Category: series (tv variant)
│   └── Breaking Bad S01E01.mkv
├── anime/
│   └── Attack on Titan E01.mkv     # Category: anime
└── documentaries/
    └── Planet Earth.mkv             # Category: other

Migration Note: If you have existing directories with non-standard names (like "movies" or "tv"), update your categories configuration in config.yaml and re-run vlm scan to fix categorization. No files will be moved.

Common Scenarios

Scenario 1: First Time Setup

# 1. Install
uv pip install -e .

# 2. Initialize config
vlm config init
# Edit ~/.vlm/config.yaml

# 3. Scan library
vlm scan

# 4. Check what was found
vlm report summary

Scenario 2: Finding Missing Episodes

# 1. Scan and parse
vlm scan
vlm parse --inventory inventory.csv
vlm enrich

# 2. Analyze completeness
vlm analyze

# 3. View report (optionally include plan summary if you have a plan)
vlm report completeness
vlm report completeness --plan plan.json

Scenario 3: Finding and Removing Duplicates

# 1. Scan and parse (with --inventory for accurate quality comparison)
vlm scan
vlm parse --inventory inventory.csv
vlm enrich

# 2. Analyze for duplicates
vlm analyze

# 3. View duplicates with quality comparison (now shows actual resolution/codec/size)
vlm report duplicates

# 4. Generate plan with analysis: VLM keeps one file per duplicate group (by reputation) and quarantines the rest
vlm plan --analysis analysis.json

# 5. Review plan (human summary in plan.json and when you run execute)
vlm execute
vlm execute --confirm

# Alternatively: manual quarantine without plan
vlm quarantine add /path/to/lower/quality/file.mkv --reason "duplicate - lower quality"

# Report with plan context
vlm report duplicates --plan plan.json

Scenario 4: Reorganizing Your Library

# 1. Customize templates in config
# Edit ~/.vlm/config.yaml to set your preferred structure

# 2. Scan and parse
vlm scan
vlm parse --inventory inventory.csv
vlm enrich

# 3. Generate plan
vlm plan

# 4. Review plan.json carefully

# 5. Dry-run to preview
vlm execute

# 6. Execute when satisfied
vlm execute --confirm

# 7. If something goes wrong, rollback
vlm rollback

Logging

VLM logs to ~/.vlm/vlm.log by default. If the log directory is unwritable, it falls back to console logging and continues running.

Troubleshooting

ffprobe Not Found

If you see warnings about ffprobe:

# Install ffmpeg (includes ffprobe)
# Ubuntu/Debian
sudo apt install ffmpeg

# macOS
brew install ffmpeg

# Windows
# Download from https://ffmpeg.org/download.html

Without ffprobe, VLM still works but won't extract video metadata (resolution, codec, etc.).

Configuration Errors

If VLM can't find your config:

# Check config location
ls -la ~/.vlm/config.yaml

# Reinitialize if needed
vlm config init

# Validate current config
vlm config validate

TMDB Auth / Rate Limit / Zero Enriched

Common enrichment outcomes:

Error during enrichment: TMDB authentication failed (401/403)
Cause: invalid/missing tmdb_bearer (or tmdb) key.
Action: update ~/.vlm/config.yaml and rerun.
Skip reasons: no_key=...
Cause: no TMDB credentials configured for provider.
Action: set enrichment.api_keys.tmdb_bearer (recommended) or tmdb.
Skip reasons: rate_limited=...
Cause: TMDB rate limit hit (429).
Action: retry later; VLM already applies bounded retry/backoff.
Enriched now: 0 with non-zero records
Cause: often no_key, no_match, or provider errors.
Action: check Skip reasons and Failure sample in CLI output.

Permission Errors

If you can't access certain files:

# Check permissions
ls -la /path/to/library

# VLM logs errors but continues scanning other files
# Check log file for details
cat ~/.vlm/vlm.log

Rollback Fails

Rollback is best-effort and may fail if:

Files have been moved/deleted since execution
Destination paths are occupied
Permissions have changed

Always keep backups of important files!

Safety Features

No Permanent Deletion: VLM never deletes files. Use quarantine instead.
Dry-Run Default: Execute command defaults to dry-run mode
Explicit Confirmation: --confirm flag required to make actual changes
Rollback Logs: All operations logged for reversal
Conflict Detection: Plan generation detects destination conflicts
Best-Effort Rollback: Attempt to restore files to original locations

Limitations

Version 1.0

Anime parsing not implemented: Anime files are discovered and categorized but not parsed
Movie/Series only for quarantine: Only movie and series categories can be quarantined
Hardcoded parsing patterns: Filename patterns are not user-configurable
Local files only: Designed for local or mounted network storage
Best-effort rollback: Rollback may not succeed if files have been modified

Gemini CLI Agent Skills

This project includes built-in expert guidance for the Gemini CLI. These skills co-locate project knowledge with the source code.

Activation

To activate the expert guidance in Gemini CLI, use:

# General expert guidance and knowledge base
activate_skill vlm-expert

# Step-by-step library organization workflow
activate_skill vlm-library-workflow

Features

Context-Aware Guidance: The Agent understands the VLM safety-first workflow and configuration.
Risk Warnings: Automatic alerts if a plan contains high-risk operations (>20% quarantine).
Command Recipes: Instant access to complex command sequences and troubleshooting steps.
Developer Verification: Built-in test execution recipes for verifying core logic changes.

Development

Running Tests

# Run all tests
pytest

# Run specific test file
pytest tests/test_scanner.py

# Run with verbose output
pytest -v

# Run with coverage
pytest --cov=vlm tests/

Code Style

Python 3.10+ idioms
PEP 8 naming conventions
Type hints for public functions
4-space indentation

Project Structure

src/vlm/
├── cli.py              # Click-based CLI interface, global options
├── context.py          # CLIContext and pass_context for commands
├── commands/           # Command implementations
│   ├── scan.py         # Scan command
│   ├── analyze.py      # Analyze command
│   └── plan.py         # Plan command
├── scanner.py          # File discovery and metadata extraction
├── parser.py           # Filename parsing (titles, years, episodes)
├── enrichment.py       # Title/reputation enrichment pipeline
├── cache.py            # SQLite cache for incremental enrichment
├── providers/          # External metadata providers (TMDB, etc.)
│   ├── base.py         # Provider interface
│   └── tmdb.py         # TMDB API client
├── io.py               # JSON/CSV load/save, load_analysis_json, plan/analysis input helpers
├── utils.py            # UTC time, format_size, etc.
├── analysis.py         # Completeness and duplicate detection
├── duplicate_resolve.py # Duplicate group keep-index (by_quality, by_reputation, by_reputation_quality_time, first_seen, manual)
├── planner.py          # Execution plan generation (optionally consumes analysis)
├── executor.py         # File operations and rollback
├── quarantine.py       # Quarantine management
├── state.py            # File state tracking
├── reports.py          # Report generation
├── config.py           # Configuration management
├── models.py           # Data structures
└── logging_config.py   # Logging setup

tests/
├── test_scanner.py     # Scanner tests
├── test_parser.py      # Parser tests
├── test_enrichment.py  # Enrichment tests
├── test_analysis.py    # Analysis tests
├── test_planner.py     # Planner tests
└── ...                 # More test files

Requirements

Python >= 3.10
uv (Python package manager)
ffmpeg (optional, for video metadata extraction)
find command (standard on Unix-like systems)

Contributing

Contributions are welcome! Please:

Write tests for new features
Follow existing code style
Update documentation
Add clear commit messages

License

[Add your license here]

Acknowledgments

Built with:

Click - CLI framework
PyYAML - Configuration parsing
pytest - Testing framework
hypothesis - Property-based testing

Support

Report issues: GitHub Issues
Documentation: See CLAUDE.md for architecture details
Logs: Check ~/.vlm/vlm.log for detailed information

23 KiB Raw Blame History Unescape Escape

Video Library Manager

Features

Installation

Quick Start

1. Initialize Configuration

2. Scan Your Library

3. Parse Filenames

4. Enrich Titles and Reputation (Optional but Recommended)

5. Analyze Your Library

6. Generate Execution Plan

7. Execute (Dry-Run First)

8. Rollback (If Needed)

Complete Workflow Example

Command Reference

Configuration

Scanning

Parsing

Enrichment

Analysis

Planning

Execution

Rollback

Quarantine Management

Reporting

State Management

Configuration File

TMDB Enrichment Setup

Template Variables

File Categorization

Common Scenarios

Scenario 1: First Time Setup

Scenario 2: Finding Missing Episodes

Scenario 3: Finding and Removing Duplicates

Scenario 4: Reorganizing Your Library

Logging

Troubleshooting

ffprobe Not Found

Configuration Errors

TMDB Auth / Rate Limit / Zero Enriched

Permission Errors

Rollback Fails

Safety Features

Limitations

Version 1.0

Gemini CLI Agent Skills

Activation

Features

Development

Running Tests

Code Style

Project Structure

Requirements

Contributing

License

Acknowledgments

Support

23 KiB

Raw Blame History