- Created worker pool with tokio::sync::Semaphore
- Created path template parser with {placeholder} support
- Added --jobs CLI flag
- Integrated both into DownloadManager
- All 105 tests pass
- Exported PathTemplate and TemplateError for template parsing
- Exported DownloadItem, DownloadWorker, DownloadWorkerResult for batch downloads
- Exported download_batch function for concurrent downloads
- All 105 tests pass
- Created DownloadWorker struct with bounded semaphore pattern
- Added download_batch() function for concurrent downloads
- Uses tokio::sync::Semaphore to limit concurrent downloads
- DownloadItem and DownloadWorkerResult structs for batch operations
- All tests pass
- Add pub mod resume to expose resume functionality
- Add resume::ResumeError to DownloadError enum
- Update DownloadManager.download() to use .part files:
- Uses resume module's get_resume_offset() to check server support
- Creates .part file during download, renames on success
- Handles 416 Range Not Satisfiable and other resume errors
- Properly integrates Accept-Ranges header checking
- Add type annotation for File variable
- Fix missing PathBuf import in resume.rs
- All tests pass (92 tests)
- Add pub mod resume to export resume functionality
- resume.rs already exists with Range header support
- Provides check_resume_support() and get_resume_offset()
- Created src/download/resume.rs with ResumeSupport struct
- Implemented check_resume_support() to verify server Accept-Ranges header
- Added get_resume_offset() to combine server check with partial file detection
- Added PART_EXTENSION constant (.part)
- Handles 416 Range Not Satisfiable errors
- Verifies 206 Partial Content response before claiming resume works
- Tests for path utilities pass
- Added 'pub mod download' to src/lib.rs
- Exported DownloadManager, DownloadOptions, DownloadResult, DownloadError
- Users can now access via gallery_dl::DownloadManager
- Created src/download/mod.rs with DownloadOptions, DownloadResult, DownloadManager
- Uses reqwest streaming (bytes_stream()) to avoid buffering entire file
- Uses tokio::fs for async file I/O with resume support via Range headers
- Created src/download/progress.rs with DownloadProgress using indicatif
- Batches progress updates every 100ms to avoid flickering
- Added 'stream' feature to reqwest for streaming support
- Created src/download/progress.rs with DownloadProgress struct
- Uses indicatif for professional-looking progress bars
- Batches updates every 100ms to avoid flickering (per research)
- Provides update(), finish(), set_message() methods
- Template: spinner + bar + position + ETA
- Created src/download/mod.rs with DownloadOptions, DownloadResult structs
- Implemented DownloadManager that wraps HttpClient for HTTP operations
- Uses reqwest streaming (bytes_stream()) to avoid buffering entire file
- Uses tokio::fs for async file I/O
- Supports resume from existing file using Range headers
- Provides builder-style API for download options
- Created DeviantArtExtractor implementing Extractor trait
- Supports user subdomains and artwork URL patterns
- OAuth token handling with client credentials support
- Uses DeviantArt API v1 and Eclipse API
- Pattern: https://username.deviantart.com, /username/art/title
- 7 unit tests for pattern matching, URL parsing, auth checks
- Fixed regex to properly distinguish subdomain vs artwork URLs
- Created PixivExtractor implementing Extractor trait
- Supports user, artwork, and series URL patterns
- OAuth token handling with refresh token support
- Uses Pixiv mobile App API (app-api.pixiv.net)
- Pattern: https://www.pixiv.net/users/{id}, /artworks/{id}, /series/{id}
- 10 unit tests for pattern matching, URL parsing, auth checks
- Created 03-02-SUMMARY.md with execution details
- Created 03-02-USER-SETUP.md with cookie configuration instructions
- Updated STATE.md with plan completion status (2/6)
- Add mod declarations for instagram and twitter modules
- Register InstagramExtractor and TwitterExtractor in register_all()
- Extractors registered before generic fallback for proper routing
- All 72 tests pass
- Implements Extractor trait for Twitter/X URLs
- Supports user profile and tweet status URL patterns
- Cookie-based authentication via auth_token cookie
- CSRF token extraction from ct0 cookie
- GraphQL API response structures for parsing tweets
- Regex pattern matching for both twitter.com and x.com
- Implements Extractor trait for Instagram URLs
- Supports profile, post, story, and highlight URL patterns
- Cookie-based authentication via sessionid cookie
- GraphQL API response structures for parsing media
- Regex pattern matching for URL routing
- Rate limiting support (6-12 second intervals recommended)
- Add module declarations for artstation and generic
- Register artstation extractor before generic for pattern priority
- Generic extractor registered last as catch-all fallback
- Add HttpClientError to ExtractorError conversion
- Implements catch-all Extractor for any HTTP/HTTPS URL
- Extracts images from img src, srcset, and link hrefs
- Converts relative URLs to absolute using base URL
- Filters by common image extensions and URL patterns
- Serves as last-resort fallback in registry
- Implements Extractor trait for artstation.com URLs
- Supports profile, project, and artwork URL patterns
- Extracts images via HTML parsing with src, srcset, and data-src
- Includes 2-second rate limiting delay
- Adds HttpClientError conversion to ExtractorError
- Plan 1: ArtStation + Generic Fallback (no auth)
- Plan 2: Instagram + Twitter/X (cookie auth)
- Plan 3: Pixiv + DeviantArt (OAuth auth)
Each plan implements site-specific extractors following the existing
Extractor trait pattern. ArtStation is simplest (no auth required),
others require varying levels of authentication.
- Created SUMMARY.md with task details and metrics
- Updated STATE.md to reflect Plan 4 completion (4/5 complete)
- Added key decision: Arc::make_mut pattern for mutable extractor access
- Fixed ExtractorMatch to use optional regex_match (removes 'static lifetime issue)
- Added Arc::make_mut to get mutable extractor access
- Added initialize() call with ExtractorMatch containing URL
- Added items() call to extract messages after initialization
- CLI now outputs extracted URLs when run with matching URL
Verification: cargo run -- https://example.com/gallery/123 outputs 3 sample URLs
- Created SUMMARY.md documenting 4 tasks completed
- Updated STATE.md to reflect 3/5 plans complete in Phase 2
- Added key decisions and completed work to context
- Built release binary successfully
- Tested CLI with example URL: extractor found correctly
- Tested CLI with unknown URL: shows helpful error with supported patterns
- All tests pass
- Added Extractor trait clone_extractor() method for Box<dyn Extractor> clone support
- Added Clone and clone_extractor to ExampleExtractor
- Fixed extractor registry to return shared extractors via Arc
- Updated main.rs to register extractors and find extractor by URL
- Added CLI integration showing found extractor or available patterns
- Added extract_url_array standalone function for tests
- Changed RegisteredExtractor to use Arc<Box<dyn Extractor>> for shared access
- Added SharedExtractor type alias
- Updated find() to return Result<SharedExtractor, ExtractorError>
- Added get_extractor() as convenience function
- Updated lib.rs exports to include SharedExtractor and get_extractor
- Created HtmlParser with CSS selector support
- Created JsonExtractor with path-based extraction
- Both modules exported from extractor crate
- Updated STATE.md with progress (2/5 plans in Phase 2)
