dl-organizer/README.md

# Video Library Manager

A Python-based CLI tool for managing personal video collections with a safety-first, human-in-the-loop approach.

## Features

- **Safety-First Design**: All file operations are reversible with rollback support
- **Human-in-the-Loop**: Explicit confirmation required before making any changes
- **Comprehensive Analysis**: Detect episode gaps and duplicate files
- **Rich Metadata**: Extract video resolution, codec, duration, and bitrate
- **Flexible Organization**: Customizable directory structure and naming templates
- **Metadata Enrichment**: Add bilingual titles and reputation signals (TMDB + optional Douban + optional AI fallback)
- **Incremental Performance**: SQLite-backed cache avoids repeated metadata lookups
- **State Tracking**: Track file status throughout the workflow
- **Detailed Reporting**: Generate inventory, completeness, and duplicate reports

## Installation

This project uses `uv` for Python package management. To install:

```bash
# Install dependencies
uv pip install -e .

# Install with development dependencies
uv pip install -e ".[dev]"
```

## Quick Start

### 1. Initialize Configuration

First, create a configuration file:

```bash
vlm config init
```

This creates `~/.vlm/config.yaml`. Edit it to set your library root:

```yaml
library_root: "/mnt/Downloads"  # Change this to your video library path
```

### 2. Scan Your Library

Discover all video files in your library:

```bash
vlm scan
```

This creates `inventory.csv` with all discovered files and their metadata.

**What happens:**
- Discovers video files using the system `find` command
- Extracts file metadata (size, modification time)
- Categorizes files based on directory structure (movie/series/anime/other)
- Extracts video metadata using ffprobe (if available)
- Saves results to `inventory.csv`

### 3. Parse Filenames

Extract titles, years, seasons, and episodes from filenames:

```bash
vlm parse
```

This creates `identities.json` with parsed information.

**What it extracts:**
- **Movies**: Title and year (e.g., "Inception (2010)")
- **Series**: Title, season, and episode numbers (e.g., "Breaking Bad S01E01")
- **Confidence scores**: Indicates parsing reliability

### 4. Enrich Titles and Reputation (Optional but Recommended)

Add translation and reputation metadata to `identities.json`:

```bash
vlm enrich
```

This updates `identities.json` in place and adds fields like:
- `title_zh`, `title_en`, `display_title`
- `reputation_score`, `reputation_votes`, `reputation_source`
- `review_status`, `enrichment_confidence`

To refresh all records instead of using incremental cache:

```bash
vlm enrich --refresh-all
```

### 5. Analyze Your Library

Detect episode gaps and duplicates:

```bash
vlm analyze
```

This creates `analysis.json` with:
- Series with missing episodes
- Duplicate files with quality comparison

### 6. Generate Execution Plan

Create a reviewable plan of file operations:

```bash
vlm plan
```

This creates `plan.json` with proposed operations (move, rename, quarantine).

**Review the plan** by opening `plan.json` in your editor. You can edit it if needed.

### 7. Execute (Dry-Run First)

Preview what will happen without making changes:

```bash
vlm execute
```

When ready to actually move/rename files:

```bash
vlm execute --confirm
```

**Important**: This creates a rollback log in `~/.vlm/rollback/` for reverting changes.

### 8. Rollback (If Needed)

If you need to undo the operations:

```bash
vlm rollback
```

This uses the most recent rollback log to restore files to their original locations.

## Complete Workflow Example

Here's a complete workflow from start to finish:

```bash
# 1. Initialize configuration
vlm config init
# Edit ~/.vlm/config.yaml to set your library_root

# 2. Scan your library
vlm scan
# Output: inventory.csv with 1234 files discovered

# 3. Parse filenames
vlm parse
# Output: identities.json with parsed titles and episodes

# 4. Enrich identities (translation + reputation)
vlm enrich
# Output: identities.json updated in place (incremental cache enabled)

# 5. Analyze for gaps and duplicates
vlm analyze
# Output: analysis.json with 5 series with gaps, 12 duplicate groups

# 6. Generate execution plan
vlm plan
# Output: plan.json with 456 operations proposed

# 7. Review the plan
cat plan.json | less
# or open in your editor

# 8. Dry-run to preview
vlm execute
# Shows what would happen without making changes

# 9. Execute with confirmation
vlm execute --confirm
# Actually performs the file operations

# 10. If needed, rollback
vlm rollback
```

## Command Reference

### Configuration

```bash
# Initialize config file
vlm config init

# Show current configuration
vlm config show

# Validate configuration
vlm config validate
```

### Scanning

```bash
# Scan with default output (inventory.csv)
vlm scan

# Scan with custom output file
vlm scan --output my_library.csv
```

**Scan behavior:**
- Hidden paths are skipped (any component starting with `.`)
- File matching uses configured `video_extensions` (default: `.mp4`, `.mkv`, `.avi`, `.mov`, `.wmv`, `.flv`, `.webm`, `.m4v`)
- If `ffprobe` is available, resolution/codec/duration/bitrate are included
- Timestamps stored in UTC format (`YYYY-MM-DDTHH:MM:SS`)

### Parsing

```bash
# Parse with default files
vlm parse

# Parse with custom input/output
vlm parse --input my_inventory.csv --output my_identities.json
```

### Enrichment

```bash
# Enrich identities in place (default: identities.json)
vlm enrich

# Enrich custom file and write to another file
vlm enrich --input my_identities.json --output enriched_identities.json

# Refresh changed records only (explicit incremental mode)
vlm enrich --refresh-changed-only

# Force full refresh (ignore cache for all records)
vlm enrich --refresh-all

# Tune request behavior
vlm enrich --timeout 6 --retries 2
```

### Analysis

```bash
# Analyze with default files
vlm analyze

# Analyze with custom files
vlm analyze --input my_identities.json --output my_analysis.json
```

### Planning

```bash
# Generate plan with default files
vlm plan

# Generate plan with custom files
vlm plan --input my_identities.json --output my_plan.json
```

### Execution

```bash
# Dry-run (preview only, no changes)
vlm execute

# Execute with confirmation (actually makes changes)
vlm execute --confirm

# Use custom plan file
vlm execute --plan my_plan.json --confirm
```

### Rollback

```bash
# Rollback using most recent log
vlm rollback

# Rollback using specific log
vlm rollback --log ~/.vlm/rollback/rollback_<uuid>.json
```

### Quarantine Management

```bash
# List quarantined files
vlm quarantine list

# List by category
vlm quarantine list --category movie

# Quarantine a file
vlm quarantine add /path/to/file.mkv --reason "duplicate"

# Restore from quarantine
vlm quarantine restore /path/to/.quarantine/file.mkv
```

### Reporting

```bash
# Generate inventory report (text format)
vlm report inventory

# Generate in different formats
vlm report inventory --format csv
vlm report inventory --format json --output inventory_report.json

# Generate completeness report (series with gaps)
vlm report completeness

# Generate duplicates report
vlm report duplicates

# Generate summary statistics
vlm report summary
```

### State Management

```bash
# Show state for a file
vlm state show /path/to/file.mkv

# Set state for a file
vlm state set /path/to/file.mkv --status reviewed
vlm state set /path/to/file.mkv --status ignored --reason "duplicate"

# Query files by status
vlm state query --status ignored

# Clear state for a file
vlm state clear /path/to/file.mkv
```

## Configuration File

The configuration file (`~/.vlm/config.yaml`) controls VLM's behavior:

```yaml
# Required: Root directory of your video library
library_root: "/mnt/nas/videos"

# Video file extensions to recognize
video_extensions:
  - .mp4
  - .mkv
  - .avi
  - .mov
  - .wmv
  - .flv
  - .webm
  - .m4v

# Directory structure templates
templates:
  movie_dir: "movie/{title} ({year})/"
  series_dir: "series/{title}/Season {season:02d}/"
  movie_filename: "{title} ({year}){ext}"
  series_filename: "S{season:02d}E{episode:02d}{ext}"

# Quarantine directory name (relative to category root)
quarantine_dir: ".quarantine"

# Logging level (DEBUG, INFO, WARNING, ERROR)
log_level: "INFO"

# Category mappings (directory name to category)
categories:
  movie: [movie, movies, films]
  series: [series, tv, shows]
  anime: [anime]

# Enrichment settings
enrichment:
  enabled: true
  incremental: true
  refresh_mode: "manual"
  providers: [tmdb]
  cache_db: "~/.vlm/enrichment_cache.db"
  max_concurrency: 6
  min_match_score: 0.75
  translation:
    mode: "bidirectional"
    fallback_machine: true
  api_keys:
    tmdb: null
    openai: null
  reputation:
    min_votes: 50
    low_score_threshold: 6.0
    policy: "flag_for_review"
  naming:
    title_format: "{title_zh} {title_en}"
```

### Template Variables

**Movies:**
- `{title}` - Movie title
- `{year}` - Release year
- `{ext}` - File extension

**Series:**
- `{title}` - Series title
- `{season}` - Season number
- `{episode}` - Episode number
- `{ext}` - File extension

Format specifiers are supported (e.g., `{season:02d}` for zero-padded numbers).

## File Categorization

VLM categorizes files based on the top-level directory in your library, matched against configured category mappings. By default:

- `movie/` → "movie"
- `series/` → "series"
- `anime/` → "anime"
- Anything else → "other"

You can customize these mappings to support multiple directory names per category:

```yaml
categories:
  movie: [movie, movies, films]
  series: [series, tv, shows, television]
  anime: [anime]
```

This allows directories like `/library/movies/` or `/library/tv/` to be correctly categorized. Directory matching is case-insensitive.

Example:
```
/mnt/Downloads/
├── movies/                          # Category: movie (plural form)
│   └── Inception (2010).mkv
├── tv/                              # Category: series (tv variant)
│   └── Breaking Bad S01E01.mkv
├── anime/
│   └── Attack on Titan E01.mkv     # Category: anime
└── documentaries/
    └── Planet Earth.mkv             # Category: other
```

**Migration Note:** If you have existing directories with non-standard names (like "movies" or "tv"), update your `categories` configuration in `config.yaml` and re-run `vlm scan` to fix categorization. No files will be moved.

## Common Scenarios

### Scenario 1: First Time Setup

```bash
# 1. Install
uv pip install -e .

# 2. Initialize config
vlm config init
# Edit ~/.vlm/config.yaml

# 3. Scan library
vlm scan

# 4. Check what was found
vlm report summary
```

### Scenario 2: Finding Missing Episodes

```bash
# 1. Scan and parse
vlm scan
vlm parse
vlm enrich

# 2. Analyze completeness
vlm analyze

# 3. View report
vlm report completeness
```

### Scenario 3: Finding and Removing Duplicates

```bash
# 1. Scan and parse
vlm scan
vlm parse
vlm enrich

# 2. Analyze for duplicates
vlm analyze

# 3. View duplicates with quality comparison
vlm report duplicates

# 4. Manually quarantine lower quality files
vlm quarantine add /path/to/lower/quality/file.mkv --reason "duplicate - lower quality"

# 5. Or generate plan and let VLM suggest operations
vlm plan
vlm execute
```

### Scenario 4: Reorganizing Your Library

```bash
# 1. Customize templates in config
# Edit ~/.vlm/config.yaml to set your preferred structure

# 2. Scan and parse
vlm scan
vlm parse
vlm enrich

# 3. Generate plan
vlm plan

# 4. Review plan.json carefully

# 5. Dry-run to preview
vlm execute

# 6. Execute when satisfied
vlm execute --confirm

# 7. If something goes wrong, rollback
vlm rollback
```

## Logging

VLM logs to `~/.vlm/vlm.log` by default. If the log directory is unwritable, it falls back to console logging and continues running.

## Troubleshooting

### ffprobe Not Found

If you see warnings about ffprobe:

```bash
# Install ffmpeg (includes ffprobe)
# Ubuntu/Debian
sudo apt install ffmpeg

# macOS
brew install ffmpeg

# Windows
# Download from https://ffmpeg.org/download.html
```

Without ffprobe, VLM still works but won't extract video metadata (resolution, codec, etc.).

### Configuration Errors

If VLM can't find your config:

```bash
# Check config location
ls -la ~/.vlm/config.yaml

# Reinitialize if needed
vlm config init

# Validate current config
vlm config validate
```

### Permission Errors

If you can't access certain files:

```bash
# Check permissions
ls -la /path/to/library

# VLM logs errors but continues scanning other files
# Check log file for details
cat ~/.vlm/vlm.log
```

### Rollback Fails

Rollback is **best-effort** and may fail if:
- Files have been moved/deleted since execution
- Destination paths are occupied
- Permissions have changed

Always keep backups of important files!

## Safety Features

- **No Permanent Deletion**: VLM never deletes files. Use quarantine instead.
- **Dry-Run Default**: Execute command defaults to dry-run mode
- **Explicit Confirmation**: `--confirm` flag required to make actual changes
- **Rollback Logs**: All operations logged for reversal
- **Conflict Detection**: Plan generation detects destination conflicts
- **Best-Effort Rollback**: Attempt to restore files to original locations

## Limitations

### Version 1.0

- **Anime parsing not implemented**: Anime files are discovered and categorized but not parsed
- **Movie/Series only for quarantine**: Only movie and series categories can be quarantined
- **Hardcoded parsing patterns**: Filename patterns are not user-configurable
- **Local files only**: Designed for local or mounted network storage
- **Best-effort rollback**: Rollback may not succeed if files have been modified

## Development

### Running Tests

```bash
# Run all tests
pytest

# Run specific test file
pytest tests/test_scanner.py

# Run with verbose output
pytest -v

# Run with coverage
pytest --cov=vlm tests/
```

### Code Style

- Python 3.10+ idioms
- PEP 8 naming conventions
- Type hints for public functions
- 4-space indentation

### Project Structure

```
src/vlm/
├── cli.py              # Click-based CLI interface
├── scanner.py          # File discovery and metadata extraction
├── parser.py           # Filename parsing (titles, years, episodes)
├── enrichment.py       # Title/reputation enrichment pipeline
├── cache.py            # SQLite cache for incremental enrichment
├── providers/          # External metadata providers (TMDB/Douban)
├── analysis.py         # Completeness and duplicate detection
├── planner.py          # Execution plan generation
├── executor.py         # File operations and rollback
├── quarantine.py       # Quarantine management
├── state.py            # File state tracking
├── reports.py          # Report generation
├── config.py           # Configuration management
├── models.py           # Data structures
└── logging_config.py   # Logging setup

tests/
├── test_scanner.py     # Scanner tests
├── test_parser.py      # Parser tests
├── test_enrichment.py  # Enrichment tests
├── test_analysis.py    # Analysis tests
├── test_planner.py     # Planner tests
└── ...                 # More test files
```

## Requirements

- **Python** >= 3.10
- **uv** (Python package manager)
- **ffmpeg** (optional, for video metadata extraction)
- **find** command (standard on Unix-like systems)

## Contributing

Contributions are welcome! Please:

1. Write tests for new features
2. Follow existing code style
3. Update documentation
4. Add clear commit messages

## License

[Add your license here]

## Acknowledgments

Built with:
- [Click](https://click.palletsprojects.com/) - CLI framework
- [PyYAML](https://pyyaml.org/) - Configuration parsing
- [pytest](https://pytest.org/) - Testing framework
- [hypothesis](https://hypothesis.readthedocs.io/) - Property-based testing

## Support

- Report issues: [GitHub Issues](https://github.com/yourusername/video-library-manager/issues)
- Documentation: See CLAUDE.md for architecture details
- Logs: Check `~/.vlm/vlm.log` for detailed information