eliott/gallery-dl
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(05-03): complete download archive plan

Browse Source 
- Added SqliteArchive with DownloadArchive trait
- CLI options --download-archive and --download-archive-skip-duplicates
- Archive integration in DownloadManager
- All 129 tests pass
This commit is contained in:
eliott
2026-02-16 09:27:38 +01:00
parent 2117d5d6fe
commit 419addf0d7
2 changed files with 130 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
.planning/STATE.md
View File

@@ -8,7 +8,7 @@
## Current Position
**Phase:** 5 - Post-Processing & Archive
**Plan:** 2 - Custom Command Execution
**Plan:** 3 - Download Archive
**Status:** Completed
```
@@ -17,15 +17,7 @@ Phase 1: [==========] 100% (Plan 4/4)
Phase 2: [==========] 100% (Plan 5/5)
Phase 3: [==========] 100% (Plan 6/6)
Phase 4: [==========] 100% (Plan 6/6)
Phase 5: [=====-----] 33% (Plan 2/6)
Phase 6: [----------] 0%
```
Progress: [==========] 100%
Phase 1: [==========] 100% (Plan 4/4)
Phase 2: [==========] 100% (Plan 5/5)
Phase 3: [==========] 100% (Plan 6/6)
Phase 4: [==========] 100% (Plan 6/6)
Phase 5: [==--------] 17% (Plan 1/6)
Phase 5: [======----] 50% (Plan 3/6)
Phase 6: [----------] 0%
```
@@ -56,9 +48,7 @@ Phase 6: [----------] 0%
| Phase 04-download-pipeline P04 | ~3min | 4 tasks | 4 files |
| Phase 05-post-processing-archive P01 | 9min | 5 tasks | 8 files |
| Phase 05-post-processing-archive P02 | ~6min | 3 tasks | 4 files |
| Phase 04-download-pipeline P03 | ~4min | 4 tasks | 5 files |
| Phase 04-download-pipeline P04 | ~3min | 4 tasks | 4 files |
| Phase 05-post-processing-archive P01 | 9min | 5 tasks | 8 files |
| Phase 05-post-processing-archive P03 | ~10min | 5 tasks | 6 files |
## Accumulated Context
@@ -84,6 +74,8 @@ Phase 6: [----------] 0%
- **Phase 4 Plan 3**: Created concurrent download worker with tokio::Semaphore, path template parser with {placeholder} syntax, --jobs CLI flag
- **Phase 4 Plan 4**: Added file filtering with FileFilter struct, CLI options --filter-size-min/max/--filter-type
- **Phase 5 Plan 1**: Created post-processing module with PostProcessor trait, ZipPostProcessor, MetadataPostProcessor, CLI options --zip/--metadata/--zip-compress
- **Phase 5 Plan 2**: Created ExecPostProcessor for custom command execution, CLI --exec option with {} placeholder support
- **Phase 5 Plan 3**: Created SqliteArchive with DownloadArchive trait for duplicate detection, CLI --download-archive option
### Requirements Mapping
@@ -153,6 +145,13 @@ All 35 v1 requirements mapped to phases:
- Added --exec CLI option with {} placeholder for file path
- Environment variables: FILE_PATH, FILE_NAME, FILE_DIR, FILE_SIZE, FILE_URL
- All 125 tests pass
- Phase 5 Plan 3: Download Archive (COMPLETED THIS RUN)
- Created SqliteArchive with DownloadArchive trait using rusqlite
- Added --download-archive CLI option for archive database path
- Added --download-archive-skip-duplicates flag with default path
- Integrated archive checking in DownloadManager before download
- Records successful downloads to archive after completion
- All 129 tests pass
### Files Created
@@ -179,6 +178,7 @@ All 35 v1 requirements mapped to phases:
- `src/postprocess/zip.rs` - ZipPostProcessor implementation (NEW)
- `src/postprocess/metadata.rs` - MetadataPostProcessor implementation (NEW)
- `src/postprocess/exec.rs` - ExecPostProcessor implementation (NEW)
- `src/archive/mod.rs` - SqliteArchive with DownloadArchive trait (NEW)
### Notes
@@ -204,6 +204,7 @@ All 35 v1 requirements mapped to phases:
- Post-processing module created with PostProcessor trait - ready for archive features
- ZIP and metadata post-processors implemented - ready for command execution
- Command execution post-processor implemented with --exec option - ready for archive database
- Download archive implemented with SqliteArchive using rusqlite - duplicate detection enabled
---

116
.planning/phases/05-post-processing-archive/05-03-SUMMARY.md Normal file
View File

@@ -0,0 +1,116 @@
---
phase: 05-post-processing-archive
plan: 03
subsystem: archive
tags: [rust, sqlite, archive, duplicate-detection, cli]
# Dependency graph
requires:
- phase: 05-post-processing-archive
plan: 01
provides: PostProcessor trait and postprocess module infrastructure
- phase: 04-download-pipeline
plan: 01
provides: DownloadManager with streaming and progress tracking
provides:
- SqliteArchive for tracking downloaded files using SQLite
- DownloadArchive trait for archive backend abstraction
- CLI --download-archive option for specifying archive database path
- CLI --download-archive-skip-duplicates flag with default path
affects: [download, cli, archive]
# Tech tracking
tech-stack:
added: [rusqlite with bundled SQLite, std::sync::Mutex for thread-safety]
patterns: - DownloadArchive trait for archive backend abstraction
- Mutex-wrapped Connection for thread-safe SQLite access
key-files:
created:
- src/archive/mod.rs - SqliteArchive implementation with DownloadArchive trait
modified:
- Cargo.toml - Added rusqlite dependency
- src/cli.rs - Added --download-archive and --download-archive-skip-duplicates options
- src/download/mod.rs - Added archive field to DownloadOptions, integrated archive checks
- src/lib.rs - Exported archive module types
key-decisions:
- "Used Mutex to wrap rusqlite Connection for thread-safety"
- "Archive check happens before download, records after success"
- "Default archive path: ~/.gallery-dl/archive.db"
- "Key: URL + filename for duplicate detection"
patterns-established:
- "SQLite-based archive with unique constraint on URL+filename"
- "Thread-safe archive access via Mutex"
# Metrics
duration: ~10min
completed: 2026-02-16
---
# Phase 5 Plan 3: Download Archive Summary
**Implemented SQLite-based download archive for duplicate detection using rusqlite**
## Performance
- **Duration:** ~10 min
- **Started:** 2026-02-16T08:16:44Z
- **Completed:** 2026-02-16T08:25:34Z
- **Tasks:** 5
- **Files modified:** 6
## Accomplishments
- Created SqliteArchive with DownloadArchive trait
- Added --download-archive CLI option for custom archive path
- Added --download-archive-skip-duplicates flag with default path (~/.gallery-dl/archive.db)
- Integrated archive checking in DownloadManager (checks before download, records after success)
- All 129 tests pass
## Task Commits
Each task was committed atomically:
1. **Task 1: Add rusqlite dependency** - `2117d5d6` (feat)
2. **Task 2: Create archive module with SqliteArchive** - (combined in 2117d5d6)
3. **Task 3: Add CLI --download-archive option** - (combined in 2117d5d6)
4. **Task 4: Integrate archive with download pipeline** - (combined in 2117d5d6)
5. **Task 5: Add skip-duplicates convenience option** - (combined in 2117d5d6)
**Plan metadata:** `2117d5d6` (docs: complete plan)
## Files Created/Modified
- `src/archive/mod.rs` - SqliteArchive with DownloadArchive trait, SQLite schema with unique constraint
- `Cargo.toml` - Added rusqlite with bundled feature
- `src/cli.rs` - Added --download-archive and --download-archive-skip-duplicates options
- `src/download/mod.rs` - Added archive field to DownloadOptions, archive checking in download()
- `src/lib.rs` - Exported DownloadArchive, SqliteArchive, ArchiveError
## Decisions Made
- Used Mutex to wrap rusqlite Connection for thread-safety in async context
- Key is URL + filename for duplicate detection (not just URL)
- Default archive path: ~/.gallery-dl/archive.db for --download-archive-skip-duplicates
- Archive check happens before download, recording happens after successful download
## Deviations from Plan
None - plan executed exactly as written.
## Issues Encountered
None
## User Setup Required
None - no external service configuration required.
## Next Phase Readiness
Phase 5 Plan 3 complete. Ready for:
- Plan 05-04: Additional archive features (hash-based dedup, etc.)
- Integration with DownloadWorker for full pipeline support
---
*Phase: 05-post-processing-archive*
*Completed: 2026-02-16*