* feat(docker): enable Redis RDB persistence, add seed-orchestrator to supervisord, copy scripts into image
* feat(orchestrator): add prefixed logger utility
* feat(orchestrator): add seed-meta read/write helpers
* feat(orchestrator): add child process runner with timeout support
* feat(orchestrator): add central seed catalog with 42 seeders
* feat(orchestrator): implement seed orchestrator with tiered scheduling
Adds scripts/seed-orchestrator.mjs with:
- classifySeeders: classify seeders into active/skipped by env vars
- buildStartupSummary: human-readable startup report
- Tiered cold start (hot/warm/cold/frozen with per-tier concurrency)
- Freshness check via seed-meta keys before running stale seeders
- Steady-state scheduling with setTimeout-based recurring timers
- Overlap protection, retry-after-60s, consecutive failure demotion
- Global concurrency cap of 5 with queue-based overflow
- Graceful shutdown on SIGTERM/SIGINT (15s drain timeout)
- Meta writing for null-metaKey seeders to seed-meta:orchestrator:{name}
* fix(seeds): use local API for warm-ping seeders in Docker mode
* fix(orchestrator): allow ACLED email+password as alternative to access token
* feat(wmsm): add seed manager CLI scaffold with help, catalog, and checks
* feat(wmsm): implement status command with freshness display
* feat(wmsm): implement schedule command with next-run estimates
* feat(wmsm): implement refresh command with single and --all modes
* feat(wmsm): implement flush and logs commands
* fix(wmsm): auto-detect docker vs podman runtime
* feat(orchestrator): extract pure scheduling functions and add test harness
* feat(orchestrator): add SEED_TURBO=real|dry mode with compressed intervals
* feat(orchestrator): add SEED_TURBO env passthrough and fix retry log message
* feat: self-hosted Docker stack with nginx, Redis REST proxy, and seeders
Multi-stage Docker build: esbuild TS handler compilation, vite frontend
build, nginx + Node.js API under supervisord. Upstash-compatible Redis
REST proxy with command allowlist for security. AIS relay WebSocket
sidecar. Seeder wrapper script with auto-sourced env vars from
docker-compose.override.yml. Self-hosting guide with architecture
diagram, API key setup, and troubleshooting.
Security: Redis proxy command allowlist (blocks FLUSHALL/CONFIG/EVAL),
nginx security headers (X-Content-Type-Options, X-Frame-Options,
Referrer-Policy), non-root container user.
* feat(docker): add Docker secrets support for API keys
Entrypoint reads /run/secrets/* files and exports as env vars at
startup. Secrets take priority over environment block values and
stay out of docker inspect / process metadata.
Both methods (env vars and secrets) work simultaneously.
* fix(docker): point supervisord at templated nginx config
The entrypoint runs envsubst on nginx.conf.template and writes
the result to /tmp/nginx.conf (with LOCAL_API_PORT substituted
and listening on port 8080 for non-root). But supervisord was
still launching nginx with /etc/nginx/nginx.conf — the default
Alpine config that listens on port 80, which fails with
"Permission denied" under the non-root appuser.
* fix(docker): remove KEYS from Redis allowlist, fix nginx header inheritance, add LLM vars to seeders
- Remove KEYS from redis-rest-proxy allowlist (O(N) blocking, Redis DoS risk)
- Move security headers into each nginx location block to prevent add_header
inheritance suppression
- Add LLM_API_URL / LLM_API_KEY / LLM_MODEL to run-seeders.sh grep filter
so LLM API keys set in docker-compose.override.yml are forwarded to seed scripts
* fix(docker): add path-based POST to Redis proxy, expand allowlist, add missing seeder secrets
- Add POST /{command}/{args...} handler to redis-rest-proxy so Upstash-style
path POSTs work (setCachedJson uses POST /set/<key>/<value>/EX/<ttl>)
- Expand allowlist: HLEN, LTRIM (seed-military-bases, seed-forecasts),
ZREVRANGE (premium-stock-store), ZRANDMEMBER (seed-military-bases)
- Add ACLED_EMAIL, ACLED_PASSWORD, OPENROUTER_API_KEY, OLLAMA_API_URL,
OLLAMA_MODEL to run-seeders.sh so override keys reach host-run seeders
---------
Co-authored-by: Elie Habib <elie.habib@gmail.com>
