Files

eliott 9c125e08ce docs(06-04): complete Wire Simulate, Destination & OAuth Config plan

- Created summary for plan 06-04
- Updated STATE.md with new plan completion and metrics
- Implemented --simulate dry-run mode, --destination wiring, and OAuth config support

2026-02-16 10:30:53 +01:00

12 KiB

Raw Blame History

State: gallery-dl-rs

Project: gallery-dl-rs
Core Value: Users can download images and media from 300+ websites using a fast, reliable CLI tool written in Rust.

Current Position

Phase: 6 - Auth & CLI Plan: 4 - Wire Simulate, Destination & OAuth Config Status: Completed

Progress: [==========] 100%
Phase 1: [==========] 100% (Plan 4/4)
Phase 2: [==========] 100% (Plan 5/5)
Phase 3: [==========] 100% (Plan 6/6)
Phase 4: [==========] 100% (Plan 6/6)
Phase 5: [==========] 100% (Plan 6/6)
Phase 6: [====------] 67% (Plan 4/6)

Performance Metrics

Metric	Target	Current
Requirements Coverage	100%	100%
Phase Success Criteria	26 total	26 derived
Dependencies Mapped	6	6

Accumulated Context

Key Decisions

Phase Structure: 6 phases derived from requirement categories
- Core Infrastructure → Extraction Framework → Site Extractors → Download Pipeline → Post-Processing & Archive → Auth & CLI
Depth: Standard (6 phases appropriate for complexity)
Phase 1 Plan 1: Created placeholder modules for cli, config, logging to enable future phased implementation
Phase 1 Plan 2: Used clap 4.x with derive macros for CLI parsing, implemented log_level() for verbose/quiet mapping
Phase 1 Plan 3: Used serde with derive macros for config, format detection via file extension, config merging with priority
Phase 1 Plan 4: Implemented logging with env_logger, integrated with CLI verbose/quiet flags, added timestamps
Phase 2 Plan 1: Created extraction framework with Extractor trait, Message enum, HttpClient with retry, ExtractorRegistry
Phase 2 Plan 2: Created HTML parsing utilities with CSS selector support (HtmlParser) and JSON extraction utilities with path notation (JsonExtractor)
Phase 2 Plan 3: CLI integration with extractor selection, example extractor demonstrating trait implementation pattern
Phase 2 Plan 4: Fixed extractor initialization flow using Arc::make_mut pattern, ExtractorMatch now uses optional regex_match
Phase 2 Plan 5: Complete extraction framework - all extractors now selectable via registry
Phase 3 Plan 1: Created ArtStationExtractor and GenericExtractor, registered in global registry
Phase 3 Plan 2: Created InstagramExtractor and TwitterExtractor with cookie-based authentication, registered in global registry
Phase 3 Plan 3: Created PixivExtractor and DeviantArtExtractor with OAuth authentication, registered in global registry (6 extractors total)
Phase 4 Plan 1: Created DownloadManager with streaming (bytes_stream), indicatif progress bars, resume via Range headers
Phase 4 Plan 2: Implemented resume with .part files, verifies Accept-Ranges header, handles 416 errors, renames on success
Phase 4 Plan 3: Created concurrent download worker with tokio::Semaphore, path template parser with {placeholder} syntax, --jobs CLI flag
Phase 4 Plan 4: Added file filtering with FileFilter struct, CLI options --filter-size-min/max/--filter-type
Phase 5 Plan 1: Created post-processing module with PostProcessor trait, ZipPostProcessor, MetadataPostProcessor, CLI options --zip/--metadata/--zip-compress
Phase 5 Plan 2: Created ExecPostProcessor for custom command execution, CLI --exec option with {} placeholder support
Phase 5 Plan 3: Created SqliteArchive with DownloadArchive trait for duplicate detection, CLI --download-archive option
Phase 6 Plan 1: Created auth module with cookie parsing, --cookies CLI argument
Phase 6 Plan 2: Browser cookie extraction using SQLite databases, tempfile for safe copying for Netscape-format cookie files
Phase 6 Plan 3: Wire CLI args to extraction pipeline, --input-file reading, cookie injection via Extractor trait

Requirements Mapping

All 35 v1 requirements mapped to phases:

Phase 1: 4 requirements (CORE-01 to CORE-04)
Phase 2: 7 requirements (EXT-01 to EXT-05, EXT-12)
Phase 3: 6 requirements (EXT-06 to EXT-11)
Phase 4: 6 requirements (DL-01 to DL-06)
Phase 5: 6 requirements (PP-01 to PP-03, ARCH-01 to ARCH-03)
Phase 6: 7 requirements (AUTH-01 to AUTH-03, CLI-01 to CLI-04)

Dependencies

Phase 1: No dependencies (foundation)
Phase 2: Depends on Phase 1
Phase 3: Depends on Phase 2
Phase 4: Depends on Phase 1, Phase 2
Phase 5: Depends on Phase 4
Phase 6: Depends on Phase 1, Phase 2

Session Continuity

Immediate Next Steps

Phase 6 Auth & CLI in progress - 4/6 plans complete
Next: Plan 06-05 for next CLI feature

Completed This Session

Phase 1 Complete - see previous summaries for details
Phase 2 Complete - extraction framework fully operational
Phase 3 Complete - 6 site extractors implemented (ArtStation, Generic, Instagram, Twitter, Pixiv, DeviantArt)
Phase 4 Complete - Download pipeline with resume, concurrency, and filtering
Phase 5 Complete - Post-processing with ZIP, metadata, exec, and archive
Phase 6 Plan 1: Cookie File Support (COMPLETED THIS RUN)
- Created auth module with cookies submodule
- Implemented Netscape cookie file parser (parse_netscape_cookies, load_cookies_from_file)
- Added --cookies and --cookies-from-browser CLI arguments
- All 140 tests pass
Phase 6 Plan 2: Browser Cookie Extraction (COMPLETED THIS RUN)
- Created browser extraction module for Firefox and Chrome
- Added extract_browser_cookies(), extract_firefox_cookies(), extract_chrome_cookies()
- Profile detection finds default browser profiles automatically
- All 145 tests pass
Phase 6 Plan 3: Wire CLI Args & Cookie Support (COMPLETED THIS RUN)
- Added --input-file URL reading from file
- Wired --cookies and --cookies-from-browser to extractors
- Added set_cookies() method to Extractor trait
- Twitter and Instagram extractors receive cookies during extraction
- All 145 tests pass
Phase 6 Plan 4: Wire Simulate, Destination & OAuth Config (COMPLETED THIS RUN)
- Implemented --simulate dry-run mode that prints URLs without downloading
- Wired --destination CLI arg to download directory (CLI > config > default)
- Added OAuth configuration support in config files for extractors
- All 145 tests pass
- Added set_cookies() method to Extractor trait
- Twitter and Instagram extractors receive cookies during extraction
- All 145 tests pass

Files Created

.planning/ROADMAP.md - Phase structure with success criteria
.planning/STATE.md - This file
Cargo.toml - Rust project manifest
Cargo.lock - Locked dependencies
src/lib.rs, src/main.rs, src/cli.rs, src/config.rs, src/logging.rs - Rust source files
src/extractor/mod.rs, src/extractor/base.rs, src/extractor/http.rs, src/extractor/message.rs - Extraction framework files
src/extractor/html.rs, src/extractor/json.rs - HTML and JSON parsing utilities
src/extractor/extractors/mod.rs, src/extractor/extractors/example.rs - Example extractors
src/extractor/extractors/artstation.rs - ArtStation extractor
src/extractor/extractors/generic.rs - Generic fallback extractor
src/extractor/extractors/instagram.rs - Instagram extractor
src/extractor/extractors/twitter.rs - Twitter/X extractor
src/extractor/extractors/pixiv.rs - Pixiv extractor
src/extractor/extractors/deviantart.rs - DeviantArt extractor
src/download/mod.rs - DownloadManager, DownloadOptions, DownloadResult (NEW)
src/download/progress.rs - DownloadProgress with indicatif (NEW)
src/download/resume.rs - Resume support with Range headers (NEW)
src/download/worker.rs - Concurrent download worker pool (NEW)
src/download/templates.rs - Path template parser (NEW)
src/postprocess/mod.rs - PostProcessor trait and config types (NEW)
src/postprocess/zip.rs - ZipPostProcessor implementation (NEW)
src/postprocess/metadata.rs - MetadataPostProcessor implementation (NEW)
src/postprocess/exec.rs - ExecPostProcessor implementation (NEW)
src/archive/mod.rs - SqliteArchive with DownloadArchive trait (NEW)
src/auth/mod.rs - Auth module with cookies submodule (NEW)
src/auth/cookies.rs - Netscape cookie file parser (NEW)

Notes

Research phase not needed - requirements are well-defined
All v1 requirements have clear phase assignments
Success criteria are observable user behaviors
Rust foundation complete - ready for extraction framework
CLI parsing complete - ready for configuration loading
Configuration loading complete - ready for logging system
Logging system complete - Phase 1 Core Infrastructure done
Extraction framework foundation complete - ready for site extractors
HTML and JSON parsing utilities complete - ready for extractor implementations
CLI integration complete - users can now run with URLs and extractors are selected
Site extractors now implemented - ready for download pipeline
Extractor initialization flow fixed - CLI outputs extracted URLs now
Instagram and Twitter extractors implemented with cookie auth - ready for download pipeline integration
Pixiv and DeviantArt extractors implemented with OAuth auth - ready for download pipeline integration
Download Manager complete with streaming and progress tracking - ready for resume and concurrency
Resume support implemented with .part files - ready for concurrent downloads (Plan 04-03)
Concurrent downloads implemented with worker pool - ready for path template support (Plan 04-04)
Path templates implemented with {placeholder} syntax - ready for post-processing integration
File filtering implemented with size and type options - ready for post-processing (Plan 04-05)
Post-processing module created with PostProcessor trait - ready for archive features
ZIP and metadata post-processors implemented - ready for command execution
Command execution post-processor implemented with --exec option - ready for archive database
Download archive implemented with SqliteArchive using rusqlite - duplicate detection enabled
Cookie file support implemented with --cookies CLI argument - ready for browser cookie extraction
Browser cookie extraction implemented with Firefox and Chrome support - ready for next auth-CLI plan
CLI args and cookies wired to extraction pipeline - ready for next Phase 6 plan
--simulate dry-run mode implemented - prints URLs without downloading
--destination wired to download directory with config fallback support
OAuth config support added for extractors (Pixiv, DeviantArt)

Last updated: 2026-02-16

12 KiB Raw Blame History