Files
worldmonitor/docker-compose.yml
Jon Torrez bb79386d24 feat: seed orchestrator with auto-seeding, persistence, and management CLI (#1940)
* feat(docker): enable Redis RDB persistence, add seed-orchestrator to supervisord, copy scripts into image

* feat(orchestrator): add prefixed logger utility

* feat(orchestrator): add seed-meta read/write helpers

* feat(orchestrator): add child process runner with timeout support

* feat(orchestrator): add central seed catalog with 42 seeders

* feat(orchestrator): implement seed orchestrator with tiered scheduling

Adds scripts/seed-orchestrator.mjs with:
- classifySeeders: classify seeders into active/skipped by env vars
- buildStartupSummary: human-readable startup report
- Tiered cold start (hot/warm/cold/frozen with per-tier concurrency)
- Freshness check via seed-meta keys before running stale seeders
- Steady-state scheduling with setTimeout-based recurring timers
- Overlap protection, retry-after-60s, consecutive failure demotion
- Global concurrency cap of 5 with queue-based overflow
- Graceful shutdown on SIGTERM/SIGINT (15s drain timeout)
- Meta writing for null-metaKey seeders to seed-meta:orchestrator:{name}

* fix(seeds): use local API for warm-ping seeders in Docker mode

* fix(orchestrator): allow ACLED email+password as alternative to access token

* feat(wmsm): add seed manager CLI scaffold with help, catalog, and checks

* feat(wmsm): implement status command with freshness display

* feat(wmsm): implement schedule command with next-run estimates

* feat(wmsm): implement refresh command with single and --all modes

* feat(wmsm): implement flush and logs commands

* fix(wmsm): auto-detect docker vs podman runtime

* feat(orchestrator): extract pure scheduling functions and add test harness

* feat(orchestrator): add SEED_TURBO=real|dry mode with compressed intervals

* feat(orchestrator): add SEED_TURBO env passthrough and fix retry log message
2026-03-22 19:51:03 +04:00

135 lines
4.5 KiB
YAML

# =============================================================================
# World Monitor — Docker / Podman Compose
# =============================================================================
# Self-contained stack: app + Redis + AIS relay.
#
# Quick start:
# cp .env.example .env # add your API keys
# docker compose up -d --build
#
# The app will be available at http://localhost:3000
# =============================================================================
services:
worldmonitor:
build:
context: .
dockerfile: Dockerfile
image: worldmonitor:latest
container_name: worldmonitor
ports:
- "${WM_PORT:-3000}:8080"
environment:
UPSTASH_REDIS_REST_URL: "http://redis-rest:80"
UPSTASH_REDIS_REST_TOKEN: "${REDIS_TOKEN:-wm-local-token}"
LOCAL_API_PORT: "46123"
LOCAL_API_MODE: "docker"
LOCAL_API_CLOUD_FALLBACK: "false"
WS_RELAY_URL: "http://ais-relay:3004"
# LLM provider (any OpenAI-compatible endpoint)
LLM_API_URL: "${LLM_API_URL:-}"
LLM_API_KEY: "${LLM_API_KEY:-}"
LLM_MODEL: "${LLM_MODEL:-}"
GROQ_API_KEY: "${GROQ_API_KEY:-}"
# Data source API keys (optional — features degrade gracefully)
AISSTREAM_API_KEY: "${AISSTREAM_API_KEY:-}"
FINNHUB_API_KEY: "${FINNHUB_API_KEY:-}"
EIA_API_KEY: "${EIA_API_KEY:-}"
FRED_API_KEY: "${FRED_API_KEY:-}"
ACLED_ACCESS_TOKEN: "${ACLED_ACCESS_TOKEN:-}"
NASA_FIRMS_API_KEY: "${NASA_FIRMS_API_KEY:-}"
CLOUDFLARE_API_TOKEN: "${CLOUDFLARE_API_TOKEN:-}"
AVIATIONSTACK_API: "${AVIATIONSTACK_API:-}"
# Seeder API keys — all optional, features degrade gracefully without them
WINDY_API_KEY: "${WINDY_API_KEY:-}"
COINGECKO_API_KEY: "${COINGECKO_API_KEY:-}"
ICAO_API_KEY: "${ICAO_API_KEY:-}"
WTO_API_KEY: "${WTO_API_KEY:-}"
WINGBITS_API_KEY: "${WINGBITS_API_KEY:-}"
OPENSKY_CLIENT_ID: "${OPENSKY_CLIENT_ID:-}"
OPENSKY_CLIENT_SECRET: "${OPENSKY_CLIENT_SECRET:-}"
OPENSKY_PROXY_AUTH: "${OPENSKY_PROXY_AUTH:-}"
OREF_PROXY_AUTH: "${OREF_PROXY_AUTH:-}"
UCDP_ACCESS_TOKEN: "${UCDP_ACCESS_TOKEN:-}"
ACLED_EMAIL: "${ACLED_EMAIL:-}"
ACLED_PASSWORD: "${ACLED_PASSWORD:-}"
ABUSEIPDB_API_KEY: "${ABUSEIPDB_API_KEY:-}"
OTX_API_KEY: "${OTX_API_KEY:-}"
URLHAUS_AUTH_KEY: "${URLHAUS_AUTH_KEY:-}"
OLLAMA_API_KEY: "${OLLAMA_API_KEY:-}"
OLLAMA_MODEL: "${OLLAMA_MODEL:-}"
CLOUDFLARE_R2_ACCOUNT_ID: "${CLOUDFLARE_R2_ACCOUNT_ID:-}"
CLOUDFLARE_R2_TOKEN: "${CLOUDFLARE_R2_TOKEN:-}"
# Testing
SEED_TURBO: "${SEED_TURBO:-}"
# Docker secrets (recommended for API keys — keeps them out of docker inspect).
# Create secrets/ dir with one file per key, then uncomment below.
# See SELF_HOSTING.md or docker-compose.override.yml for details.
# secrets:
# - GROQ_API_KEY
# - AISSTREAM_API_KEY
# - FINNHUB_API_KEY
# - FRED_API_KEY
# - NASA_FIRMS_API_KEY
# - LLM_API_KEY
depends_on:
redis-rest:
condition: service_started
ais-relay:
condition: service_started
restart: unless-stopped
ais-relay:
build:
context: .
dockerfile: Dockerfile.relay
image: worldmonitor-ais-relay:latest
container_name: worldmonitor-ais-relay
environment:
AISSTREAM_API_KEY: "${AISSTREAM_API_KEY:-}"
PORT: "3004"
restart: unless-stopped
redis:
image: docker.io/redis:7-alpine
container_name: worldmonitor-redis
command: redis-server --maxmemory ${REDIS_MAXMEMORY:-512mb} --maxmemory-policy allkeys-lru --save 300 10 --save 900 1
volumes:
- redis-data:/data
restart: unless-stopped
redis-rest:
build:
context: docker
dockerfile: Dockerfile.redis-rest
image: worldmonitor-redis-rest:latest
container_name: worldmonitor-redis-rest
ports:
- "127.0.0.1:8079:80"
environment:
SRH_TOKEN: "${REDIS_TOKEN:-wm-local-token}"
SRH_CONNECTION_STRING: "redis://redis:6379"
depends_on:
- redis
restart: unless-stopped
# Docker secrets — uncomment and point to your secret files.
# Example: echo "gsk_abc123" > secrets/groq_api_key.txt
# secrets:
# GROQ_API_KEY:
# file: ./secrets/groq_api_key.txt
# AISSTREAM_API_KEY:
# file: ./secrets/aisstream_api_key.txt
# FINNHUB_API_KEY:
# file: ./secrets/finnhub_api_key.txt
# FRED_API_KEY:
# file: ./secrets/fred_api_key.txt
# NASA_FIRMS_API_KEY:
# file: ./secrets/nasa_firms_api_key.txt
# LLM_API_KEY:
# file: ./secrets/llm_api_key.txt
volumes:
redis-data: