dl-organizer/README.md

# Video Library Manager

A Python-based CLI tool for managing personal video collections with a safety-first, human-in-the-loop approach.

## Features

- **Safety-First Design**: All file operations are reversible with rollback support
- **Human-in-the-Loop**: Explicit confirmation required before making any changes
- **Comprehensive Analysis**: Detect episode gaps and duplicate files
- **Rich Metadata**: Extract video resolution, codec, duration, and bitrate
- **Flexible Organization**: Customizable directory structure and naming templates
- **Metadata Enrichment**: Add bilingual titles and reputation signals (TMDB + optional AI fallback)
- **Incremental Performance**: SQLite-backed cache avoids repeated metadata lookups
- **State Tracking**: Track file status throughout the workflow
- **Detailed Reporting**: Generate inventory, completeness, and duplicate reports (reports can include plan content summary via `--plan`)
- **Plan–Analysis Integration**: `vlm plan --analysis` applies duplicate resolution (keep by reputation, quarantine rest) and adds a Chinese human summary to the plan for quick review

## Installation

This project uses `uv` for Python package management. To install:

```bash
# Install dependencies
uv pip install -e .

# Install with development dependencies
uv pip install -e ".[dev]"
```

## Quick Start

### 1. Initialize Configuration

First, create a configuration file:

```bash
vlm config init
```

This creates `~/.vlm/config.yaml`. Edit it to set your library root:

```yaml
library_root: "/mnt/Downloads"  # Change this to your video library path
```

### 2. Scan Your Library

Discover all video files in your library:

```bash
vlm scan
```

This creates `inventory.csv` with all discovered files and their metadata.

**What happens:**
- Discovers video files using the system `find` command
- Extracts file metadata (size, modification time)
- Categorizes files based on directory structure (movie/series/anime/other)
- Extracts video metadata using ffprobe (if available)
- Saves results to `inventory.csv`

### 3. Parse Filenames

Extract titles, years, seasons, and episodes from filenames:

```bash
vlm parse
```

This creates `identities.json` with parsed information.

**What it extracts:**
- **Movies**: Title and year (e.g., "Inception (2010)")
- **Series**: Title, season, and episode numbers (e.g., "Breaking Bad S01E01")
- **Confidence scores**: Indicates parsing reliability

### 4. Enrich Titles and Reputation (Optional but Recommended)

Add translation and reputation metadata to `identities.json`:

```bash
vlm enrich
```

This updates `identities.json` in place and adds fields like:
- `title_zh`, `title_en`, `display_title`
- `reputation_score`, `reputation_votes`, `reputation_source`
- `review_status`, `enrichment_confidence`
- summary metrics including `api_calls`, `cache_hits`, and `skip_reasons`

To refresh all records instead of using incremental cache:

```bash
vlm enrich --refresh-all
```

If enrichment cannot run for some records, CLI shows grouped reasons, for example:

```text
Skip reasons: no_key=4632
```

### 5. Analyze Your Library

Detect episode gaps and duplicates:

```bash
vlm analyze
```

This creates `analysis.json` with:
- Series with missing episodes
- Duplicate files with quality comparison

### 6. Generate Execution Plan

Create a reviewable plan of file operations:

```bash
vlm plan
```

To let the plan automatically resolve duplicate groups (keep one file per group by reputation, quarantine the rest), pass the analysis file:

```bash
vlm plan --analysis analysis.json
```

This creates `plan.json` with:
- Proposed operations (move, rename, quarantine, no-op)
- **Summary**: counts by operation type and by reason
- **Human summary** (中文): short narrative for quick review
- **Metadata**: when using `--analysis`, duplicate groups considered and completeness gaps

Duplicate keep strategy is configurable in `~/.vlm/config.yaml` under `plan.duplicate_keep` (`by_reputation`, `first_seen`, or `manual`). Default is `by_reputation` (prefer external rating; fallback to first-seen).

**Review the plan** by opening `plan.json` in your editor, or read the human summary when you run `vlm execute`. You can edit the plan JSON if needed.

### 7. Execute (Dry-Run First)

Preview what will happen without making changes:

```bash
vlm execute
```

Before running, the CLI prints the plan’s **human summary** (or a short summary from counts) so you can confirm at a glance. When ready to actually move/rename/quarantine files:

```bash
vlm execute --confirm
```

**Important**: This creates a rollback log in `~/.vlm/rollback/` for reverting changes.

### 8. Rollback (If Needed)

If you need to undo the operations:

```bash
vlm rollback
```

This uses the most recent rollback log to restore files to their original locations.

## Complete Workflow Example

Here's a complete workflow from start to finish:

```bash
# 1. Initialize configuration
vlm config init
# Edit ~/.vlm/config.yaml to set your library_root

# 2. Scan your library
vlm scan
# Output: inventory.csv with 1234 files discovered

# 3. Parse filenames
vlm parse
# Output: identities.json with parsed titles and episodes

# 4. Enrich identities (translation + reputation)
vlm enrich
# Output: identities.json updated in place (incremental cache enabled)

# 5. Analyze for gaps and duplicates
vlm analyze
# Output: analysis.json with 5 series with gaps, 12 duplicate groups

# 6. Generate execution plan (optionally use analysis for duplicate handling)
vlm plan --analysis analysis.json
# Output: plan.json with operations, human summary, and duplicate quarantine decisions

# 7. Review the plan
cat plan.json | less
# or open in your editor

# 8. Dry-run to preview
vlm execute
# Shows what would happen without making changes

# 9. Execute with confirmation
vlm execute --confirm
# Actually performs the file operations

# 10. If needed, rollback
vlm rollback
```

## Command Reference

### Configuration

```bash
# Initialize config file
vlm config init

# Show current configuration
vlm config show

# Validate configuration
vlm config validate
```

### Scanning

```bash
# Scan with default output (inventory.csv)
vlm scan

# Scan with custom output file
vlm scan --output my_library.csv
```

**Scan behavior:**
- Hidden paths are skipped (any component starting with `.`)
- File matching uses configured `video_extensions` (default: `.mp4`, `.mkv`, `.avi`, `.mov`, `.wmv`, `.flv`, `.webm`, `.m4v`)
- If `ffprobe` is available, resolution/codec/duration/bitrate are included
- Timestamps stored in UTC format (`YYYY-MM-DDTHH:MM:SS`)

### Parsing

```bash
# Parse with default files
vlm parse

# Parse with custom input/output
vlm parse --input my_inventory.csv --output my_identities.json
```

### Enrichment

```bash
# Enrich identities in place (default: identities.json)
vlm enrich

# Enrich custom file and write to another file
vlm enrich --input my_identities.json --output enriched_identities.json

# Refresh changed records only (explicit incremental mode)
vlm enrich --refresh-changed-only

# Force full refresh (ignore cache for all records)
vlm enrich --refresh-all

# Tune request behavior
vlm enrich --timeout 6 --retries 2
```

### Analysis

```bash
# Analyze with default files
vlm analyze

# Analyze with custom files
vlm analyze --input my_identities.json --output my_analysis.json
```

### Planning

```bash
# Generate plan with default files
vlm plan

# Use analysis so duplicate groups become "keep one + quarantine rest" (by_reputation by default)
vlm plan --analysis analysis.json

# Custom input/output
vlm plan --input my_identities.json --output my_plan.json
vlm plan --input my_identities.json --analysis my_analysis.json --output my_plan.json
```

### Execution

```bash
# Dry-run (preview only, no changes)
vlm execute

# Execute with confirmation (actually makes changes)
vlm execute --confirm

# Use custom plan file
vlm execute --plan my_plan.json --confirm
```

### Rollback

```bash
# Rollback using most recent log
vlm rollback

# Rollback using specific log
vlm rollback --log ~/.vlm/rollback/rollback_<uuid>.json
```

### Quarantine Management

```bash
# List quarantined files
vlm quarantine list

# List by category
vlm quarantine list --category movie

# Quarantine a file
vlm quarantine add /path/to/file.mkv --reason "duplicate"

# Restore from quarantine
vlm quarantine restore /path/to/.quarantine/file.mkv
```

### Reporting

```bash
# Generate inventory report (text format)
vlm report inventory

# Generate in different formats
vlm report inventory --format csv
vlm report inventory --format json --output inventory_report.json

# Generate completeness report (series with gaps)
vlm report completeness

# Include plan content summary in the report (human_summary from plan.json)
vlm report completeness --plan plan.json

# Generate duplicates report
vlm report duplicates
vlm report duplicates --plan plan.json

# Generate summary statistics
vlm report summary
```

### State Management

```bash
# Show state for a file
vlm state show /path/to/file.mkv

# Set state for a file
vlm state set /path/to/file.mkv --status reviewed
vlm state set /path/to/file.mkv --status ignored --reason "duplicate"

# Query files by status
vlm state query --status ignored

# Clear state for a file
vlm state clear /path/to/file.mkv
```

## Configuration File

The configuration file (`~/.vlm/config.yaml`) controls VLM's behavior:

```yaml
# Required: Root directory of your video library
library_root: "/mnt/nas/videos"

# Video file extensions to recognize
video_extensions:
  - .mp4
  - .mkv
  - .avi
  - .mov
  - .wmv
  - .flv
  - .webm
  - .m4v

# Directory structure templates
templates:
  movie_dir: "movie/{title} ({year})/"
  series_dir: "series/{title}/Season {season:02d}/"
  movie_filename: "{title} ({year}){ext}"
  series_filename: "S{season:02d}E{episode:02d}{ext}"

# Quarantine directory name (relative to category root)
quarantine_dir: ".quarantine"

# Logging level (DEBUG, INFO, WARNING, ERROR)
log_level: "INFO"

# Plan behavior (e.g. when using vlm plan --analysis)
plan:
  # Duplicate keep strategy: "by_reputation" (default), "first_seen", or "manual"
  duplicate_keep: "by_reputation"

# Category mappings (directory name to category)
categories:
  movie: [movie, movies, films]
  series: [series, tv, shows]
  anime: [anime]

# Enrichment settings (`enrich` alias is also supported)
enrichment:
  enabled: true
  incremental: true
  refresh_mode: "manual"
  providers: [tmdb]
  cache_db: "~/.vlm/enrichment_cache.db"
  max_concurrency: 6
  min_match_score: 0.75
  translation:
    mode: "bidirectional"
    fallback_machine: true
  api_keys:
    # Preferred: TMDB v4 Bearer token
    tmdb_bearer: null
    # Backward-compatible fallback (legacy query api_key)
    tmdb: null
    openai: null
  tmdb:
    language: "zh-CN"
    region: null
    include_adult: false
  reputation:
    min_votes: 50
    low_score_threshold: 6.0
    policy: "flag_for_review"
  naming:
    title_format: "{title_zh} {title_en}"
```

### TMDB Enrichment Setup

`vlm enrich` works best with TMDB Bearer auth (recommended by TMDB). Legacy `tmdb` api key is still supported for compatibility.

Minimal config:

```yaml
enrichment:
  providers: [tmdb]
  api_keys:
    tmdb_bearer: "YOUR_TMDB_BEARER_TOKEN"
```

Optional TMDB query tuning:

```yaml
enrichment:
  tmdb:
    language: "zh-CN"   # localized title language
    region: "US"        # affects regional release/search behavior
    include_adult: false
```

Validation flow:

```bash
# 1) run small incremental pass
vlm enrich --input identities.json --refresh-changed-only

# 2) then full refresh if output looks correct
vlm enrich --input identities.json --refresh-all
```

### Template Variables

**Movies:**
- `{title}` - Movie title
- `{year}` - Release year
- `{ext}` - File extension

**Series:**
- `{title}` - Series title
- `{season}` - Season number
- `{episode}` - Episode number
- `{ext}` - File extension

Format specifiers are supported (e.g., `{season:02d}` for zero-padded numbers).

## File Categorization

VLM categorizes files based on the top-level directory in your library, matched against configured category mappings. By default:

- `movie/` → "movie"
- `series/` → "series"
- `anime/` → "anime"
- Anything else → "other"

You can customize these mappings to support multiple directory names per category:

```yaml
categories:
  movie: [movie, movies, films]
  series: [series, tv, shows, television]
  anime: [anime]
```

This allows directories like `/library/movies/` or `/library/tv/` to be correctly categorized. Directory matching is case-insensitive.

Example:
```
/mnt/Downloads/
├── movies/                          # Category: movie (plural form)
│   └── Inception (2010).mkv
├── tv/                              # Category: series (tv variant)
│   └── Breaking Bad S01E01.mkv
├── anime/
│   └── Attack on Titan E01.mkv     # Category: anime
└── documentaries/
    └── Planet Earth.mkv             # Category: other
```

**Migration Note:** If you have existing directories with non-standard names (like "movies" or "tv"), update your `categories` configuration in `config.yaml` and re-run `vlm scan` to fix categorization. No files will be moved.

## Common Scenarios

### Scenario 1: First Time Setup

```bash
# 1. Install
uv pip install -e .

# 2. Initialize config
vlm config init
# Edit ~/.vlm/config.yaml

# 3. Scan library
vlm scan

# 4. Check what was found
vlm report summary
```

### Scenario 2: Finding Missing Episodes

```bash
# 1. Scan and parse
vlm scan
vlm parse
vlm enrich

# 2. Analyze completeness
vlm analyze

# 3. View report (optionally include plan summary if you have a plan)
vlm report completeness
vlm report completeness --plan plan.json
```

### Scenario 3: Finding and Removing Duplicates

```bash
# 1. Scan and parse
vlm scan
vlm parse
vlm enrich

# 2. Analyze for duplicates
vlm analyze

# 3. View duplicates with quality comparison
vlm report duplicates

# 4. Generate plan with analysis: VLM keeps one file per duplicate group (by reputation) and quarantines the rest
vlm plan --analysis analysis.json

# 5. Review plan (human summary in plan.json and when you run execute)
vlm execute
vlm execute --confirm

# Alternatively: manual quarantine without plan
vlm quarantine add /path/to/lower/quality/file.mkv --reason "duplicate - lower quality"

# Report with plan context
vlm report duplicates --plan plan.json
```

### Scenario 4: Reorganizing Your Library

```bash
# 1. Customize templates in config
# Edit ~/.vlm/config.yaml to set your preferred structure

# 2. Scan and parse
vlm scan
vlm parse
vlm enrich

# 3. Generate plan
vlm plan

# 4. Review plan.json carefully

# 5. Dry-run to preview
vlm execute

# 6. Execute when satisfied
vlm execute --confirm

# 7. If something goes wrong, rollback
vlm rollback
```

## Logging

VLM logs to `~/.vlm/vlm.log` by default. If the log directory is unwritable, it falls back to console logging and continues running.

## Troubleshooting

### ffprobe Not Found

If you see warnings about ffprobe:

```bash
# Install ffmpeg (includes ffprobe)
# Ubuntu/Debian
sudo apt install ffmpeg

# macOS
brew install ffmpeg

# Windows
# Download from https://ffmpeg.org/download.html
```

Without ffprobe, VLM still works but won't extract video metadata (resolution, codec, etc.).

### Configuration Errors

If VLM can't find your config:

```bash
# Check config location
ls -la ~/.vlm/config.yaml

# Reinitialize if needed
vlm config init

# Validate current config
vlm config validate
```

### TMDB Auth / Rate Limit / Zero Enriched

Common enrichment outcomes:

- `Error during enrichment: TMDB authentication failed (401/403)`  
  Cause: invalid/missing `tmdb_bearer` (or `tmdb`) key.  
  Action: update `~/.vlm/config.yaml` and rerun.

- `Skip reasons: no_key=...`  
  Cause: no TMDB credentials configured for provider.  
  Action: set `enrichment.api_keys.tmdb_bearer` (recommended) or `tmdb`.

- `Skip reasons: rate_limited=...`  
  Cause: TMDB rate limit hit (`429`).  
  Action: retry later; VLM already applies bounded retry/backoff.

- `Enriched now: 0` with non-zero records  
  Cause: often `no_key`, `no_match`, or provider errors.  
  Action: check `Skip reasons` and `Failure sample` in CLI output.

### Permission Errors

If you can't access certain files:

```bash
# Check permissions
ls -la /path/to/library

# VLM logs errors but continues scanning other files
# Check log file for details
cat ~/.vlm/vlm.log
```

### Rollback Fails

Rollback is **best-effort** and may fail if:
- Files have been moved/deleted since execution
- Destination paths are occupied
- Permissions have changed

Always keep backups of important files!

## Safety Features

- **No Permanent Deletion**: VLM never deletes files. Use quarantine instead.
- **Dry-Run Default**: Execute command defaults to dry-run mode
- **Explicit Confirmation**: `--confirm` flag required to make actual changes
- **Rollback Logs**: All operations logged for reversal
- **Conflict Detection**: Plan generation detects destination conflicts
- **Best-Effort Rollback**: Attempt to restore files to original locations

## Limitations

### Version 1.0

- **Anime parsing not implemented**: Anime files are discovered and categorized but not parsed
- **Movie/Series only for quarantine**: Only movie and series categories can be quarantined
- **Hardcoded parsing patterns**: Filename patterns are not user-configurable
- **Local files only**: Designed for local or mounted network storage
- **Best-effort rollback**: Rollback may not succeed if files have been modified

## Development

### Running Tests

```bash
# Run all tests
pytest

# Run specific test file
pytest tests/test_scanner.py

# Run with verbose output
pytest -v

# Run with coverage
pytest --cov=vlm tests/
```

### Code Style

- Python 3.10+ idioms
- PEP 8 naming conventions
- Type hints for public functions
- 4-space indentation

### Project Structure

```
src/vlm/
├── cli.py              # Click-based CLI interface, global options
├── context.py          # CLIContext and pass_context for commands
├── commands/           # Command implementations
│   ├── scan.py         # Scan command
│   ├── analyze.py      # Analyze command
│   └── plan.py         # Plan command
├── scanner.py          # File discovery and metadata extraction
├── parser.py           # Filename parsing (titles, years, episodes)
├── enrichment.py       # Title/reputation enrichment pipeline
├── cache.py            # SQLite cache for incremental enrichment
├── providers/          # External metadata providers (TMDB, etc.)
│   ├── base.py         # Provider interface
│   └── tmdb.py         # TMDB API client
├── io.py               # JSON/CSV load/save, load_analysis_json, plan/analysis input helpers
├── utils.py            # UTC time, format_size, etc.
├── analysis.py         # Completeness and duplicate detection
├── duplicate_resolve.py # Duplicate group keep-index (by_reputation, first_seen, manual)
├── planner.py          # Execution plan generation (optionally consumes analysis)
├── executor.py         # File operations and rollback
├── quarantine.py       # Quarantine management
├── state.py            # File state tracking
├── reports.py          # Report generation
├── config.py           # Configuration management
├── models.py           # Data structures
└── logging_config.py   # Logging setup

tests/
├── test_scanner.py     # Scanner tests
├── test_parser.py      # Parser tests
├── test_enrichment.py  # Enrichment tests
├── test_analysis.py    # Analysis tests
├── test_planner.py     # Planner tests
└── ...                 # More test files
```

## Requirements

- **Python** >= 3.10
- **uv** (Python package manager)
- **ffmpeg** (optional, for video metadata extraction)
- **find** command (standard on Unix-like systems)

## Contributing

Contributions are welcome! Please:

1. Write tests for new features
2. Follow existing code style
3. Update documentation
4. Add clear commit messages

## License

[Add your license here]

## Acknowledgments

Built with:
- [Click](https://click.palletsprojects.com/) - CLI framework
- [PyYAML](https://pyyaml.org/) - Configuration parsing
- [pytest](https://pytest.org/) - Testing framework
- [hypothesis](https://hypothesis.readthedocs.io/) - Property-based testing

## Support

- Report issues: [GitHub Issues](https://github.com/yourusername/video-library-manager/issues)
- Documentation: See CLAUDE.md for architecture details
- Logs: Check `~/.vlm/vlm.log` for detailed information