From 6d923108d87d437e159c0684a0ed27137dd157eb Mon Sep 17 00:00:00 2001 From: Elie Habib Date: Fri, 10 Apr 2026 11:50:32 +0400 Subject: [PATCH] =?UTF-8?q?refactor(seeds):=20bundle=20orchestrator=20to?= =?UTF-8?q?=20consolidate=20Railway=20cron=20services=20(100=E2=86=9265)?= =?UTF-8?q?=20(#2891)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * refactor(seeds): add bundle orchestrator to consolidate Railway cron services Railway is at the 100-service limit. This adds a shared _bundle-runner.mjs orchestrator and 11 bundle scripts that group related seed cron services, reducing the count from 100 to ~65 when deployed. Each bundle spawns sub-seeds via child_process.execFile (proven pattern from ais-relay.cjs), with freshness-gated skipping so monthly seeds in a daily bundle only run when due. Original scripts are unchanged and independently runnable. Bundles: ecb-eu (4→1), portwatch (4→1), climate (5→1), energy-sources (6→1), macro (6→1), health (4→1), static-ref (3→1), resilience (2→1), derived-signals (2→1), market-backup (5→1), relay-backup (4→1). * refactor(seeds): deduplicate time constants across bundle scripts Export MIN/HOUR/DAY/WEEK from _bundle-runner.mjs so all 11 bundle scripts import shared constants instead of re-declaring them locally. Eliminates inconsistent computation styles (24*60*60*1000 vs 24*HOUR). * fix(seeds): correct wb-indicators seedMetaKey in relay-backup bundle The seed writes to seed-meta:economic:worldbank-techreadiness:v1 but the bundle config was missing the :v1 suffix, causing the freshness gate to always return null and the seed to run every cycle instead of daily. Found by architecture-strategist review agent. * fix(seeds): address review findings in bundle runner - Remove em dashes from comment and log line (project convention) - Read Redis creds directly instead of via getRedisCredentials() which calls process.exit(1) on missing env vars, bypassing try/catch and silently killing the entire bundle before any seed runs - Missing creds now gracefully skip freshness check (seeds still run) * fix(seeds): correct intervalMs values and exit code in bundle runner P1 fixes from external review: 1. process.exit(0) on failure now exits non-zero (exit 1 when failed > 0) so Railway cron monitoring detects degraded runs. 2. Corrected intervalMs to match actual cron cadences (was using TTL values): - crypto-quotes: 15min -> 5min (actual cron is 5min) - stablecoin-markets: 15min -> 10min (actual cron is 10min) - gulf-quotes: 15min -> 10min (actual cron is 10min) - health-air-quality: 3h -> 1h (actual cron is 1h) - bls-series: 3d -> 1d (actual cron is daily) - eurostat: 3d -> 1d (actual cron is daily) - fao-ffpi: 30d -> 1d (runs daily to catch monthly release window) - imf-macro: 35d -> 30d (monthly data) - national-debt: 35d -> 30d (monthly data) * docs: add Railway seed consolidation runbook Complete migration checklist with: - 46 services to delete (with Railway UUIDs) - 11 bundle services to create (with cron, start cmd, watch paths) - 43 standalone services that stay (with reasons) - Execution order, verification checklist, env var guidance - Watch paths: scripts/** + shared/** (covers loadSharedConfig resolution) - Inventory checksum: 4+4+3+46+43 = 100 --- docs/railway-seed-consolidation-runbook.md | 420 +++++++++++++++++++++ scripts/_bundle-runner.mjs | 119 ++++++ scripts/seed-bundle-climate.mjs | 10 + scripts/seed-bundle-derived-signals.mjs | 7 + scripts/seed-bundle-ecb-eu.mjs | 9 + scripts/seed-bundle-energy-sources.mjs | 11 + scripts/seed-bundle-health.mjs | 9 + scripts/seed-bundle-macro.mjs | 11 + scripts/seed-bundle-market-backup.mjs | 10 + scripts/seed-bundle-portwatch.mjs | 9 + scripts/seed-bundle-relay-backup.mjs | 9 + scripts/seed-bundle-resilience.mjs | 7 + scripts/seed-bundle-static-ref.mjs | 8 + 13 files changed, 639 insertions(+) create mode 100644 docs/railway-seed-consolidation-runbook.md create mode 100644 scripts/_bundle-runner.mjs create mode 100644 scripts/seed-bundle-climate.mjs create mode 100644 scripts/seed-bundle-derived-signals.mjs create mode 100644 scripts/seed-bundle-ecb-eu.mjs create mode 100644 scripts/seed-bundle-energy-sources.mjs create mode 100644 scripts/seed-bundle-health.mjs create mode 100644 scripts/seed-bundle-macro.mjs create mode 100644 scripts/seed-bundle-market-backup.mjs create mode 100644 scripts/seed-bundle-portwatch.mjs create mode 100644 scripts/seed-bundle-relay-backup.mjs create mode 100644 scripts/seed-bundle-resilience.mjs create mode 100644 scripts/seed-bundle-static-ref.mjs diff --git a/docs/railway-seed-consolidation-runbook.md b/docs/railway-seed-consolidation-runbook.md new file mode 100644 index 000000000..d86671254 --- /dev/null +++ b/docs/railway-seed-consolidation-runbook.md @@ -0,0 +1,420 @@ +# Railway Seed Consolidation Runbook + +**Date:** 2026-04-10 +**PR:** #2891 +**Current services:** 100 (at Railway limit) +**Target services:** 65 (~35 slots freed) + +--- + +## Prerequisites + +1. Merge PR #2891 to `main` +2. Verify the bundle scripts are in the deployed branch +3. Have Railway dashboard access and `gh` CLI authenticated + +--- + +## How It Works + +Each "bundle" is a single Railway cron service that replaces N individual services. The bundle script spawns each member seed sequentially via `child_process.execFile`, checking Redis `seed-meta:` timestamps to skip seeds that ran recently. Original seed scripts are unchanged. + +**Per-bundle migration:** + +1. Delete ONE old member first (to free a slot under the 100 limit) +2. Create the bundle service on Railway +3. Wait 2-3 cron cycles, verify `/api/health` shows OK for all member seeds +4. Delete remaining old member services +5. Monitor 24h before proceeding to next bundle + +**Rollback:** Delete the bundle service, re-create individual services. Scripts are unchanged in the repo. + +--- + +## Services to DELETE (46 total) + +### Standalone delete (no bundle replacement needed) + +| # | Service Name | Service ID | Reason | +|---|---|---|---| +| 1 | seed-defense-patents (DISABLED) | `6f8bfd1b-7ccc-4db5-b03c-a2075b173e91` | Already disabled, no data flowing | + +### Replaced by seed-bundle-ecb-eu + +| # | Service Name | Service ID | Original Cron | +|---|---|---|---| +| 2 | seed-ecb-fx-rates | `9cc81d27-745f-4925-a956-d9e0acacc8a2` | daily | +| 3 | seed-ecb-short-rates | `b695dd14-12fd-4493-a41b-30d50a9519d5` | daily | +| 4 | seed-yield-curve-eu | `b372da1c-e67d-44c0-ae23-4e391e75709b` | daily | +| 5 | seed-fsi-eu | `9c67552d-0a0a-409a-bf4f-571ac3f741c3` | weekly | + +### Replaced by seed-bundle-portwatch + +| # | Service Name | Service ID | Original Cron | +|---|---|---|---| +| 6 | seed-portwatch | `72b553c9-bf63-4905-ab47-706b0cc674e8` | every 6h | +| 7 | seed-portwatch-disruptions | `cb0aea5d-806b-49f9-85f3-b0a0e1372a26` | hourly | +| 8 | seed-portwatch-chokepoints-ref | `7907937c-5730-4768-a3cc-f4a3f555a9c5` | weekly | +| 9 | seed-portwatch-port-activity | `334303bb-41a2-4e66-9add-b1762fda9a1a` | every 12h | + +### Replaced by seed-bundle-static-ref + +| # | Service Name | Service ID | Original Cron | +|---|---|---|---| +| 10 | seed-submarine-cables | `fde66e2c-e542-47e0-8ff5-49026b229949` | weekly | +| 11 | seed-chokepoint-baselines | `de51db71-3492-4521-873c-90b9c08dd8b4` | infrequent (400d TTL) | +| 12 | seed-military-bases | `54b44749-c318-4392-aebe-aaf8308db1e9` | infrequent (one-time) | + +### Replaced by seed-bundle-resilience + +| # | Service Name | Service ID | Original Cron | +|---|---|---|---| +| 13 | seed-resilience-scores | `e87c212a-eab6-4a85-9e43-b855ca207823` | every 6h | +| 14 | seed-resilience-static | `e0709305-0270-4f53-b133-7d74e8260400` | annual window | + +### Replaced by seed-bundle-derived-signals + +| # | Service Name | Service ID | Original Cron | +|---|---|---|---| +| 15 | seed-correlation | `6cb62419-f354-419a-835c-67f494347680` | every 5min | +| 16 | seed-cross-source-signals | `57708db4-37a9-490e-98ee-dcdc783ce0f9` | every 15min | + +### Replaced by seed-bundle-climate + +| # | Service Name | Service ID | Original Cron | +|---|---|---|---| +| 17 | seed-climate-zone-normals | `01d57359-bccd-46f7-8b78-351040058f5f` | monthly | +| 18 | seed-climate-anomalies | `90095ed3-c9a8-4e42-b955-3b66fe288edb` | every 3h | +| 19 | seed-climate-disasters | `7a8e2384-925a-42c3-9767-c4cf14822985` | every 6h | +| 20 | seed-climate-ocean-ice | `05c54150-226f-471d-9938-90fde67a8f11` | daily | +| 21 | seed-co2-monitoring | `2a1cd437-fed3-4f74-b327-f2336ffcbb3f` | every 3 days | + +### Replaced by seed-bundle-energy-sources + +| # | Service Name | Service ID | Original Cron | +|---|---|---|---| +| 22 | seed-gie-gas-storage | `70a43803-f91e-4306-973c-b99ce29fb055` | daily | +| 23 | seed-gas-storage-countries | `a8dd33d5-ed2a-4462-97ef-3e9654920e19` | daily | +| 24 | seed-jodi-gas | `7b7c7198-60e0-48b4-8f9c-33036d530586` | monthly | +| 25 | seed-jodi-oil | `c0d829a5-42ce-4644-bd7d-94f93bf92e26` | monthly | +| 26 | seed-owid-energy-mix | `31303e69-ec86-4fa0-b956-0c5524f038a1` | monthly | +| 27 | seed-iea-oil-stocks | `8a05aaa6-8802-4221-ab3b-59001a4df5d3` | monthly | + +### Replaced by seed-bundle-macro + +| # | Service Name | Service ID | Original Cron | +|---|---|---|---| +| 28 | seed-bis-data | `8a2896ea-207e-4bef-8cd0-c6871df09a1d` | every 12h | +| 29 | seed-bls-series | `cf6f0bd4-3b09-4e77-b720-f2d08cb2c04f` | daily | +| 30 | seed-eurostat-country-data | `9314f05a-c9d6-4d5a-8af6-575da09174b0` | daily | +| 31 | seed-imf-macro | `5634de02-83ff-4ab1-8b88-aef73c4055e7` | monthly | +| 32 | seed-national-debt | `7ca57c8b-5d26-4a47-ba76-ae8f465eb0f3` | monthly | +| 33 | seed-fao-food-price-index | `c923b38f-3a52-4933-96d1-89443c8deda1` | daily | + +### Replaced by seed-bundle-health + +| # | Service Name | Service ID | Original Cron | +|---|---|---|---| +| 34 | seed-health-air-quality | `7be8c278-1c00-4761-adb5-85336ee4661b` | hourly | +| 35 | seed-disease-outbreaks | `12c8681b-6e82-464d-b6e5-6b397123643d` | daily | +| 36 | seed-vpd-tracker | `bd286f94-39f2-4341-895d-4ea6ea4d1905` | daily | +| 37 | seed-displacement-summary | `fed916c2-97bc-434b-ad2d-636121bcd70d` | daily | + +### Replaced by seed-bundle-market-backup + +| # | Service Name | Service ID | Original Cron | Also in ais-relay? | +|---|---|---|---|---| +| 38 | seed-crypto-quotes | `3bf34a40-e4dc-4fac-9fa6-8438118d0f53` | every 5min | Yes (Market loop) | +| 39 | seed-stablecoin-markets | `0410d0eb-81ee-46e0-a50f-8fd9de334ef8` | every 10min | Yes (Market loop) | +| 40 | seed-etf-flows | `6d907720-b274-4b4c-a2e5-a37e9161f349` | every 15min | Yes (Market loop) | +| 41 | seed-gulf-quotes | `ba1ad92b-1813-412d-b6e5-6c37f3f741c2` | every 10min | Yes (Market loop) | +| 42 | seed-token-panels | `a975dc1a-6ac3-4db0-89bf-bdcdecb92fde` | every 30min | Yes (Market loop) | + +### Replaced by seed-bundle-relay-backup + +| # | Service Name | Service ID | Original Cron | Also in ais-relay? | +|---|---|---|---|---| +| 43 | seed-climate-news | `c4875401-90b5-4738-ba64-6f27496d41a0` | every 30min | Yes (child spawn) | +| 44 | seed-usa-spending | `f420ca72-c41d-46aa-a151-0315ce45df2d` | hourly | Yes (Spending loop) | +| 45 | seed-ucdp-events | `6bce510f-d3a9-4252-b896-45aef3521cac` | every 6h | Yes (UCDP loop) | +| 46 | seed-wb-indicators | `ad9df8af-f27c-41db-a89d-f68f2fab2cf6` | daily | Yes (WB loop) | + +--- + +## Services to CREATE (11 total) + +All new services share these settings: + +- **Root directory:** `.` (repo root, so `npm ci` installs all deps) +- **Build command:** (default nixpacks, uses `scripts/nixpacks.toml`) +- **Source branch:** `main` +- **Resources:** 1 vCPU / 1 GB RAM +- **NODE_OPTIONS:** `--dns-result-order=ipv4first` + +**Watch paths:** Use `scripts/**`, `shared/**` for all bundles. `scripts/**` covers all seed scripts and their helpers. `shared/**` is needed because `loadSharedConfig()` in `_seed-utils.mjs` resolves `../shared/` (repo root) before `./shared/` (scripts dir), so config JSON files like `country-names.json`, `iso3-to-iso2.json`, and others live at the repo root `shared/` directory. Without `shared/**`, config-only edits won't trigger redeploys. + +### Bundle 1: seed-bundle-ecb-eu + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-ecb-eu` | +| **Start command** | `node scripts/seed-bundle-ecb-eu.mjs` | +| **Cron schedule** | `0 6 * * *` (daily 06:00 UTC) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 4 services (ecb-fx-rates, ecb-short-rates, yield-curve-eu, fsi-eu) | +| **Net savings** | 3 slots | +| **Members** | ECB FX Rates (daily), ECB Short Rates (daily), Yield Curve EU (daily), FSI EU (weekly, skips 6/7 runs) | + +### Bundle 2: seed-bundle-portwatch + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-portwatch` | +| **Start command** | `node scripts/seed-bundle-portwatch.mjs` | +| **Cron schedule** | `0 */1 * * *` (hourly) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 4 services | +| **Net savings** | 3 slots | +| **Members** | Disruptions (hourly), Main (6h), Port Activity (12h), Chokepoints Ref (weekly) | + +### Bundle 3: seed-bundle-static-ref + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-static-ref` | +| **Start command** | `node scripts/seed-bundle-static-ref.mjs` | +| **Cron schedule** | `0 3 * * 0` (weekly, Sunday 03:00 UTC) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 3 services | +| **Net savings** | 2 slots | +| **Members** | Submarine Cables (weekly), Chokepoint Baselines (400d, runs rarely), Military Bases (30d, runs rarely) | + +### Bundle 4: seed-bundle-resilience + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-resilience` | +| **Start command** | `node scripts/seed-bundle-resilience.mjs` | +| **Cron schedule** | `0 */6 * * *` (every 6h) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 2 services | +| **Net savings** | 1 slot | +| **Members** | Resilience Scores (6h), Resilience Static (annual window Oct 1-3, skips most runs) | + +### Bundle 5: seed-bundle-derived-signals + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-derived-signals` | +| **Start command** | `node scripts/seed-bundle-derived-signals.mjs` | +| **Cron schedule** | `*/5 * * * *` (every 5 min) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 2 services | +| **Net savings** | 1 slot | +| **Members** | Correlation (5min), Cross-Source Signals (15min, runs every 3rd invocation) | +| **Note** | Both are Redis-derived (no external API calls), fast execution | + +### Bundle 6: seed-bundle-climate + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-climate` | +| **Start command** | `node scripts/seed-bundle-climate.mjs` | +| **Cron schedule** | `0 */3 * * *` (every 3h) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 5 services | +| **Net savings** | 4 slots | +| **Members** | Zone Normals (monthly, skips ~359/360), Anomalies (3h, depends on zone-normals), Disasters (6h), Ocean Ice (daily), CO2 Monitoring (3 days) | +| **Note** | Zone-normals runs before anomalies (dependency ordering) | + +### Bundle 7: seed-bundle-energy-sources + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-energy-sources` | +| **Start command** | `node scripts/seed-bundle-energy-sources.mjs` | +| **Cron schedule** | `30 7 * * *` (daily 07:30 UTC) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 6 services | +| **Net savings** | 5 slots | +| **Members** | GIE Gas Storage (daily), Gas Storage Countries (daily), JODI Gas (monthly), JODI Oil (monthly), OWID Energy Mix (monthly), IEA Oil Stocks (monthly) | + +### Bundle 8: seed-bundle-macro + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-macro` | +| **Start command** | `node scripts/seed-bundle-macro.mjs` | +| **Cron schedule** | `0 8 * * *` (daily 08:00 UTC) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 6 services | +| **Net savings** | 5 slots | +| **Members** | BIS Data (12h), BLS Series (daily), Eurostat (daily), IMF Macro (monthly), National Debt (monthly), FAO FFPI (daily, catches monthly release window) | + +### Bundle 9: seed-bundle-health + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-health` | +| **Start command** | `node scripts/seed-bundle-health.mjs` | +| **Cron schedule** | `0 */1 * * *` (hourly) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 4 services | +| **Net savings** | 3 slots | +| **Members** | Air Quality (hourly), Disease Outbreaks (daily), VPD Tracker (daily), Displacement (daily) | + +### Bundle 10: seed-bundle-market-backup + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-market-backup` | +| **Start command** | `node scripts/seed-bundle-market-backup.mjs` | +| **Cron schedule** | `*/5 * * * *` (every 5 min) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 5 services | +| **Net savings** | 4 slots | +| **Members** | Crypto Quotes (5min), Stablecoin Markets (10min), ETF Flows (15min), Gulf Quotes (10min), Token Panels (30min) | +| **Note** | These are BACKUP for ais-relay inline loops. ais-relay is the primary seeder. The bundle provides redundancy if relay goes down. Gulf Quotes uses Alpha Vantage (richer than relay's Yahoo-only). | + +### Bundle 11: seed-bundle-relay-backup + +| Setting | Value | +|---|---| +| **Service name** | `seed-bundle-relay-backup` | +| **Start command** | `node scripts/seed-bundle-relay-backup.mjs` | +| **Cron schedule** | `*/30 * * * *` (every 30 min) | +| **Watch paths** | `scripts/**`, `shared/**` | +| **Replaces** | 4 services | +| **Net savings** | 3 slots | +| **Members** | Climate News (30min), USA Spending (hourly), UCDP Events (6h), WB Indicators (daily) | +| **Note** | These are BACKUP for ais-relay inline loops/child spawns. Each seed's freshness gate skips if the relay already refreshed the data recently. | + +--- + +## Services that STAY unchanged (54 total) + +### Infrastructure (4) + +| Service | ID | Type | +|---|---|---| +| Postgres | `8a5871b9-5ca9-4551-8343-aef7fa67b8a4` | Database | +| Postgres-azIG | `3ea8ae20-44f4-49bd-a363-76b0adec8dcd` | Database | +| Valkey | `651a4b62-e224-47c2-9f7c-64e35908c44a` | Cache | +| umami | `d7620480-e05a-4c09-b210-05166c3c0e59` | Analytics | + +### Long-running services (4) + +| Service | ID | Type | +|---|---|---| +| worldmonitor (ais-relay) | `a5f66d97-217f-44a0-a42d-5f3b67752223` | AIS relay + inline seeds | +| notification-relay | `aa37bd8e-c28d-4e9b-9d1e-0961f1b63d97` | Notification dispatch | +| simulation-worker | `67264e35-0b51-457b-984f-4ef20e36a117` | Forecast simulations | +| deep-forecast-worker | `750bc68f-9840-49a3-95eb-7c8bcc060485` | Deep forecast tasks | + +### Consumer prices pipeline (3) + +| Service | ID | Type | +|---|---|---| +| seed-consumer-prices | `2a369c41-cc5c-486a-a8d7-f0ca552e27a8` | Scraper | +| seed-consumer-prices-publish | `4492a338-cb37-40da-9e98-95a8d67e49c9` | Redis publisher | +| seed-consumer-aggregate | `4fdd1078-7884-48f8-92fc-06b390d0fdc4` | Index calculator | + +### Standalone seed crons (43, not bundled) + +| # | Service | ID | Why not bundled | +|---|---|---|---| +| 1 | digest-notifications | `01d644b8-057f-4040-a50e-500bd684daa8` | Notification dispatch, not a data seed | +| 2 | seed-airport-delays | `444e9cc0-4eb2-4820-b430-3228e6ce9568` | Unique aviation domain | +| 3 | seed-aviation | `a8e49386-64c1-4e1e-9f82-4eb69a55fce3` | Different keys from relay's aviation loop | +| 4 | seed-bigmac | `e8269317-c717-498b-adcf-be693a2bb8d3` | Weekly, web scraping via Exa | +| 5 | seed-chokepoint-exposure | `12e8e87d-1214-4ba3-a813-709f279a5ba9` | Derived from Comtrade flows | +| 6 | seed-conflict-intel | `e4188e09-ae3b-4398-bb24-04f4b4b48b52` | Fast cadence (15min), notifications | +| 7 | seed-cot | `23b2597f-1989-4904-9018-b3722a9e1bc2` | Weekly CFTC data | +| 8 | seed-cyber-threats | `fd27928b-0b9b-45d6-b056-92fa2f5d60a6` | Relay disabled its loop, cron is sole source | +| 9 | seed-earnings-calendar | `cd07f48e-6433-4847-9f7b-1f05d062e619` | Finnhub, different domain | +| 10 | seed-earthquakes | `5a953848-0678-4946-8ea0-b2269914ea12` | Independent seismology | +| 11 | seed-economic-calendar | `555fc987-a043-4f64-bfa3-c827157ec706` | FRED + Eurostat + Fed/ECB scrape | +| 12 | seed-economy | `565a66c1-662d-4a3a-b8e2-83b79d75dbe4` | Already multi-section (11+ keys) | +| 13 | seed-electricity-prices | `1aee77cd-3af9-4640-a78d-e957c322adc0` | ENTSO-E + EIA, large dataset | +| 14 | seed-ember-electricity | `67e01a64-d3cb-4b53-bf7d-cd5d223323b3` | Large CSV download | +| 15 | seed-energy-intelligence | `9c2135c6-d638-4137-955a-8819c4d969f6` | RSS parsing | +| 16 | seed-energy-spine | `a6c1d05f-a639-4470-829d-9337ffbdcbbe` | Composite from other seeds | +| 17 | seed-fear-greed | `fcff514b-7b32-46c2-9413-0a48bcf4968e` | Composite index, unique sources | +| 18 | seed-fire-detections | `1ebe342b-074b-4fb5-b012-c1dbfdef1971` | Feeds thermal-escalation | +| 19 | seed-forecasts | `9bcbf89e-2785-452b-b59f-144b4863bd95` | LLM-heavy, long runtime | +| 20 | seed-fuel-prices | `8d966e58-e01c-42cf-8d28-b85fd5d45460` | EU XLSX download | +| 21 | seed-fx-rates | `5221253d-a22e-4560-a3db-ea4634c2049a` | Shared dependency for other seeds | +| 22 | seed-gdelt-intel | `3472577e-dff4-49f9-bc17-f32c2f366f75` | 6 topics with 20s delays | +| 23 | seed-gpsjam | `16949dc7-b908-4740-bfbe-74a213db7c0b` | GPS interference monitoring | +| 24 | seed-grocery-basket | `c8438692-843d-46ae-bee7-8c19e6847fa4` | Web scraping via Exa | +| 25 | seed-hormuz | `e6156007-e917-4139-90bd-71b6333a6d0e` | Power BI scraping | +| 26 | seed-infra | `c615c211-1237-47cc-8d90-e23657437838` | Warm-ping to Vercel | +| 27 | seed-insights | `d1e092bb-6a5b-4225-8043-8ed93ccff268` | LLM-dependent | +| 28 | seed-internet-outages | `5a07e099-14d8-42aa-ad6e-e66631fdd19f` | Cloudflare Radar | +| 29 | seed-iran-events | `5d294bd6-7943-4454-aa9c-eb90bd9d9124` | Iran-focused aggregation | +| 30 | seed-military-flights | `7953a066-0627-4550-b72c-d2aceb33fbd3` | Real-time tracking, live/stale keys | +| 31 | seed-military-maritime | `88768189-f80b-4615-87d1-dbc7803a6a28` | USNI warm-ping | +| 32 | seed-natural-events | `7119c932-05f5-4727-a54f-e4e2de2a907f` | NASA EONET + GDACS + NHC | +| 33 | seed-prediction-markets | `96fabace-d56d-4854-8096-3f5bcfe0d88a` | Polymarket anti-bot measures | +| 34 | seed-radiation-watch | `3b76bb85-637c-43b7-ab90-5dee288f8bca` | EPA + Safecast | +| 35 | seed-regulatory-actions | `249ae8df-5746-4cdb-9978-ec61dce9121f` | Financial regulator RSS | +| 36 | seed-research | `ab850199-4d48-4af8-9681-aafbe2f31b8e` | arXiv + HN + GitHub | +| 37 | seed-sanctions-pressure | `e1686cdf-980f-426d-b5f2-a7757729fe9b` | 120MB+ XML streaming | +| 38 | seed-security-advisories | `8fb9c6b7-0ae9-441b-ae02-0f31baa3aed6` | 22 advisory feeds | +| 39 | seed-supply-chain-trade | `d7cc29f0-691b-40fd-84f2-ce8e8f12b567` | Already multi-section | +| 40 | seed-thermal-escalation | `71d124d5-a4fb-42c3-9c5b-2fb0e5645e5b` | Derived from fire detections | +| 41 | seed-trade-flows | `dd3097f7-df65-4b0e-89ca-86a5fac7d558` | UN Comtrade, 6 reporters | +| 42 | seed-unrest-events | `33c8c2a1-ad66-45ec-ac7e-609d69a59455` | GDELT + ACLED | +| 43 | seed-webcams | `2bf93afa-1922-4f9c-936d-f5054051b8a5` | Paginated across 8 regions | + +**Inventory check:** 4 infra + 4 long-running + 3 consumer + 46 delete + 43 standalone = **100** + +--- + +## Execution Order (recommended) + +Start with lowest-risk, highest-savings bundles. + +| Order | Bundle | Slots Freed | Risk | Cron Frequency | +|---|---|---|---|---| +| 1 | seed-bundle-ecb-eu | 3 | Low (daily, same API) | Daily | +| 2 | seed-bundle-static-ref | 2 | Low (weekly, static data) | Weekly | +| 3 | seed-bundle-resilience | 1 | Low (6h, annual window) | 6h | +| 4 | seed-bundle-portwatch | 3 | Medium (hourly, 4 members) | Hourly | +| 5 | seed-bundle-climate | 4 | Medium (3h, 5 members) | 3h | +| 6 | seed-bundle-energy-sources | 5 | Medium (daily, 6 members) | Daily | +| 7 | seed-bundle-macro | 5 | Medium (daily, 6 members) | Daily | +| 8 | seed-bundle-health | 3 | Medium (hourly, 4 members) | Hourly | +| 9 | seed-bundle-derived-signals | 1 | Low (5min, Redis-only) | 5min | +| 10 | seed-bundle-market-backup | 4 | Low (backup for relay) | 5min | +| 11 | seed-bundle-relay-backup | 3 | Low (backup for relay) | 30min | +| - | seed-defense-patents | 1 | None (already disabled) | - | + +**Running total:** 3 + 2 + 1 + 3 + 4 + 5 + 5 + 3 + 1 + 4 + 3 + 1 = **35 slots freed** + +--- + +## Verification Checklist (per bundle) + +After deploying each bundle and before deleting old services: + +- [ ] Bundle service shows "Active" in Railway dashboard +- [ ] First cron fire produced logs (check Railway logs) +- [ ] Logs show expected `[Bundle:X] Starting (N sections)` and `Finished` lines +- [ ] Each member seed shows `Done` or `Skipped` (not all `failed`) +- [ ] `/api/health` shows OK for all member seed-meta keys (not STALE_SEED) +- [ ] Wait at least 2 full cron cycles before deleting old services +- [ ] After deleting old services, verify health still shows OK on next cycle + +--- + +## Env Vars + +Each bundle service inherits the same env vars as the individual seeds it replaces. Copy these from any existing seed service in Railway: + +- `UPSTASH_REDIS_REST_URL` +- `UPSTASH_REDIS_REST_TOKEN` +- `NODE_OPTIONS=--dns-result-order=ipv4first` +- Plus any API keys used by member seeds (GIE_API_KEY, ICAO_API_KEY, etc.) + +The simplest approach: use Railway's "shared variables" or copy all env vars from the `worldmonitor` (ais-relay) service, which has a superset of all API keys. diff --git a/scripts/_bundle-runner.mjs b/scripts/_bundle-runner.mjs new file mode 100644 index 000000000..0359f61fa --- /dev/null +++ b/scripts/_bundle-runner.mjs @@ -0,0 +1,119 @@ +#!/usr/bin/env node +/** + * Bundle orchestrator: spawns multiple seed scripts sequentially + * via child_process.execFile, with freshness-gated skipping. + * + * Pattern matches ais-relay.cjs:5645-5695 (ClimateNews/ChokepointFlows spawns). + * + * Usage from a bundle script: + * import { runBundle } from './_bundle-runner.mjs'; + * await runBundle('ecb-eu', [ { label, script, seedMetaKey, intervalMs, timeoutMs } ]); + */ + +import { execFile } from 'node:child_process'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { loadEnvFile } from './_seed-utils.mjs'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); + +export const MIN = 60_000; +export const HOUR = 3_600_000; +export const DAY = 86_400_000; +export const WEEK = 604_800_000; + +loadEnvFile(import.meta.url); + +const REDIS_URL = process.env.UPSTASH_REDIS_REST_URL; +const REDIS_TOKEN = process.env.UPSTASH_REDIS_REST_TOKEN; + +async function readSeedMeta(seedMetaKey) { + if (!REDIS_URL || !REDIS_TOKEN) return null; + try { + const resp = await fetch(`${REDIS_URL}/get/${encodeURIComponent(`seed-meta:${seedMetaKey}`)}`, { + headers: { Authorization: `Bearer ${REDIS_TOKEN}` }, + signal: AbortSignal.timeout(5_000), + }); + if (!resp.ok) return null; + const data = await resp.json(); + return data.result ? JSON.parse(data.result) : null; + } catch { + return null; + } +} + +function spawnSeed(scriptPath, { timeoutMs, label }) { + return new Promise((resolve, reject) => { + const t0 = Date.now(); + execFile(process.execPath, [scriptPath], { + env: process.env, + timeout: timeoutMs, + maxBuffer: 2 * 1024 * 1024, + }, (err, stdout, stderr) => { + const elapsed = ((Date.now() - t0) / 1000).toFixed(1); + if (stdout) { + for (const line of String(stdout).trim().split('\n')) { + if (line) console.log(` [${label}] ${line}`); + } + } + if (stderr) { + for (const line of String(stderr).trim().split('\n')) { + if (line) console.warn(` [${label}] ${line}`); + } + } + if (err) { + const reason = err.killed ? 'timeout' : (err.code || err.message); + reject(new Error(`${label} failed after ${elapsed}s: ${reason}`)); + } else { + resolve({ elapsed }); + } + }); + }); +} + +/** + * @param {string} label - Bundle name for logging + * @param {Array<{ + * label: string, + * script: string, + * seedMetaKey: string, + * intervalMs: number, + * timeoutMs?: number, + * }>} sections + */ +export async function runBundle(label, sections) { + const t0 = Date.now(); + console.log(`[Bundle:${label}] Starting (${sections.length} sections)`); + + let ran = 0, skipped = 0, failed = 0; + + for (const section of sections) { + const scriptPath = join(__dirname, section.script); + const timeout = section.timeoutMs || 300_000; + + const meta = await readSeedMeta(section.seedMetaKey); + if (meta?.fetchedAt) { + const elapsed = Date.now() - meta.fetchedAt; + if (elapsed < section.intervalMs * 0.8) { + const agoMin = Math.round(elapsed / 60_000); + const intervalMin = Math.round(section.intervalMs / 60_000); + console.log(` [${section.label}] Skipped, last seeded ${agoMin}min ago (interval: ${intervalMin}min)`); + skipped++; + continue; + } + } + + try { + const result = await spawnSeed(scriptPath, { timeoutMs: timeout, label: section.label }); + console.log(` [${section.label}] Done (${result.elapsed}s)`); + ran++; + } catch (err) { + console.error(` [${section.label}] ${err.message}`); + failed++; + } + } + + const totalSec = ((Date.now() - t0) / 1000).toFixed(1); + console.log(`[Bundle:${label}] Finished in ${totalSec}s, ran:${ran} skipped:${skipped} failed:${failed}`); + process.exit(failed > 0 ? 1 : 0); +} diff --git a/scripts/seed-bundle-climate.mjs b/scripts/seed-bundle-climate.mjs new file mode 100644 index 000000000..0d4b08e6e --- /dev/null +++ b/scripts/seed-bundle-climate.mjs @@ -0,0 +1,10 @@ +#!/usr/bin/env node +import { runBundle, HOUR, DAY } from './_bundle-runner.mjs'; + +await runBundle('climate', [ + { label: 'Zone-Normals', script: 'seed-climate-zone-normals.mjs', seedMetaKey: 'climate:zone-normals', intervalMs: 30 * DAY, timeoutMs: 600_000 }, + { label: 'Anomalies', script: 'seed-climate-anomalies.mjs', seedMetaKey: 'climate:anomalies', intervalMs: 3 * HOUR, timeoutMs: 300_000 }, + { label: 'Disasters', script: 'seed-climate-disasters.mjs', seedMetaKey: 'climate:disasters', intervalMs: 6 * HOUR, timeoutMs: 180_000 }, + { label: 'Ocean-Ice', script: 'seed-climate-ocean-ice.mjs', seedMetaKey: 'climate:ocean-ice', intervalMs: DAY, timeoutMs: 300_000 }, + { label: 'CO2-Monitoring', script: 'seed-co2-monitoring.mjs', seedMetaKey: 'climate:co2-monitoring', intervalMs: 3 * DAY, timeoutMs: 180_000 }, +]); diff --git a/scripts/seed-bundle-derived-signals.mjs b/scripts/seed-bundle-derived-signals.mjs new file mode 100644 index 000000000..4a483f7aa --- /dev/null +++ b/scripts/seed-bundle-derived-signals.mjs @@ -0,0 +1,7 @@ +#!/usr/bin/env node +import { runBundle, MIN } from './_bundle-runner.mjs'; + +await runBundle('derived-signals', [ + { label: 'Correlation', script: 'seed-correlation.mjs', seedMetaKey: 'correlation:cards', intervalMs: 5 * MIN, timeoutMs: 60_000 }, + { label: 'Cross-Source-Signals', script: 'seed-cross-source-signals.mjs', seedMetaKey: 'intelligence:cross-source-signals', intervalMs: 15 * MIN, timeoutMs: 120_000 }, +]); diff --git a/scripts/seed-bundle-ecb-eu.mjs b/scripts/seed-bundle-ecb-eu.mjs new file mode 100644 index 000000000..e1e4e3c2e --- /dev/null +++ b/scripts/seed-bundle-ecb-eu.mjs @@ -0,0 +1,9 @@ +#!/usr/bin/env node +import { runBundle, DAY, WEEK } from './_bundle-runner.mjs'; + +await runBundle('ecb-eu', [ + { label: 'ECB-FX-Rates', script: 'seed-ecb-fx-rates.mjs', seedMetaKey: 'economic:ecb-fx-rates', intervalMs: DAY, timeoutMs: 120_000 }, + { label: 'ECB-Short-Rates', script: 'seed-ecb-short-rates.mjs', seedMetaKey: 'economic:ecb-short-rates', intervalMs: DAY, timeoutMs: 120_000 }, + { label: 'Yield-Curve-EU', script: 'seed-yield-curve-eu.mjs', seedMetaKey: 'economic:yield-curve-eu', intervalMs: DAY, timeoutMs: 120_000 }, + { label: 'FSI-EU', script: 'seed-fsi-eu.mjs', seedMetaKey: 'economic:fsi-eu', intervalMs: WEEK, timeoutMs: 120_000 }, +]); diff --git a/scripts/seed-bundle-energy-sources.mjs b/scripts/seed-bundle-energy-sources.mjs new file mode 100644 index 000000000..ef321c79c --- /dev/null +++ b/scripts/seed-bundle-energy-sources.mjs @@ -0,0 +1,11 @@ +#!/usr/bin/env node +import { runBundle, DAY } from './_bundle-runner.mjs'; + +await runBundle('energy-sources', [ + { label: 'GIE-Gas-Storage', script: 'seed-gie-gas-storage.mjs', seedMetaKey: 'economic:eu-gas-storage', intervalMs: DAY, timeoutMs: 180_000 }, + { label: 'Gas-Storage-Countries', script: 'seed-gas-storage-countries.mjs', seedMetaKey: 'energy:gas-storage-countries', intervalMs: DAY, timeoutMs: 600_000 }, + { label: 'JODI-Gas', script: 'seed-jodi-gas.mjs', seedMetaKey: 'energy:jodi-gas', intervalMs: 35 * DAY, timeoutMs: 600_000 }, + { label: 'JODI-Oil', script: 'seed-jodi-oil.mjs', seedMetaKey: 'energy:jodi-oil', intervalMs: 35 * DAY, timeoutMs: 600_000 }, + { label: 'OWID-Energy-Mix', script: 'seed-owid-energy-mix.mjs', seedMetaKey: 'economic:owid-energy-mix', intervalMs: 35 * DAY, timeoutMs: 600_000 }, + { label: 'IEA-Oil-Stocks', script: 'seed-iea-oil-stocks.mjs', seedMetaKey: 'energy:iea-oil-stocks', intervalMs: 40 * DAY, timeoutMs: 300_000 }, +]); diff --git a/scripts/seed-bundle-health.mjs b/scripts/seed-bundle-health.mjs new file mode 100644 index 000000000..27bbdf569 --- /dev/null +++ b/scripts/seed-bundle-health.mjs @@ -0,0 +1,9 @@ +#!/usr/bin/env node +import { runBundle, HOUR, DAY } from './_bundle-runner.mjs'; + +await runBundle('health', [ + { label: 'Air-Quality', script: 'seed-health-air-quality.mjs', seedMetaKey: 'health:air-quality', intervalMs: HOUR, timeoutMs: 600_000 }, + { label: 'Disease-Outbreaks', script: 'seed-disease-outbreaks.mjs', seedMetaKey: 'health:disease-outbreaks', intervalMs: DAY, timeoutMs: 300_000 }, + { label: 'VPD-Tracker', script: 'seed-vpd-tracker.mjs', seedMetaKey: 'health:vpd-tracker', intervalMs: DAY, timeoutMs: 300_000 }, + { label: 'Displacement', script: 'seed-displacement-summary.mjs', seedMetaKey: 'displacement:summary', intervalMs: DAY, timeoutMs: 300_000 }, +]); diff --git a/scripts/seed-bundle-macro.mjs b/scripts/seed-bundle-macro.mjs new file mode 100644 index 000000000..de7cef955 --- /dev/null +++ b/scripts/seed-bundle-macro.mjs @@ -0,0 +1,11 @@ +#!/usr/bin/env node +import { runBundle, HOUR, DAY } from './_bundle-runner.mjs'; + +await runBundle('macro', [ + { label: 'BIS-Data', script: 'seed-bis-data.mjs', seedMetaKey: 'economic:bis', intervalMs: 12 * HOUR, timeoutMs: 300_000 }, + { label: 'BLS-Series', script: 'seed-bls-series.mjs', seedMetaKey: 'economic:bls-series', intervalMs: DAY, timeoutMs: 120_000 }, + { label: 'Eurostat', script: 'seed-eurostat-country-data.mjs', seedMetaKey: 'economic:eurostat-country-data', intervalMs: DAY, timeoutMs: 300_000 }, + { label: 'IMF-Macro', script: 'seed-imf-macro.mjs', seedMetaKey: 'economic:imf-macro', intervalMs: 30 * DAY, timeoutMs: 300_000 }, + { label: 'National-Debt', script: 'seed-national-debt.mjs', seedMetaKey: 'economic:national-debt', intervalMs: 30 * DAY, timeoutMs: 300_000 }, + { label: 'FAO-FFPI', script: 'seed-fao-food-price-index.mjs', seedMetaKey: 'economic:fao-ffpi', intervalMs: DAY, timeoutMs: 120_000 }, +]); diff --git a/scripts/seed-bundle-market-backup.mjs b/scripts/seed-bundle-market-backup.mjs new file mode 100644 index 000000000..67834d90e --- /dev/null +++ b/scripts/seed-bundle-market-backup.mjs @@ -0,0 +1,10 @@ +#!/usr/bin/env node +import { runBundle, MIN } from './_bundle-runner.mjs'; + +await runBundle('market-backup', [ + { label: 'Crypto-Quotes', script: 'seed-crypto-quotes.mjs', seedMetaKey: 'market:crypto', intervalMs: 5 * MIN, timeoutMs: 120_000 }, + { label: 'Stablecoin-Markets', script: 'seed-stablecoin-markets.mjs', seedMetaKey: 'market:stablecoins', intervalMs: 10 * MIN, timeoutMs: 120_000 }, + { label: 'ETF-Flows', script: 'seed-etf-flows.mjs', seedMetaKey: 'market:etf-flows', intervalMs: 15 * MIN, timeoutMs: 120_000 }, + { label: 'Gulf-Quotes', script: 'seed-gulf-quotes.mjs', seedMetaKey: 'market:gulf-quotes', intervalMs: 10 * MIN, timeoutMs: 120_000 }, + { label: 'Token-Panels', script: 'seed-token-panels.mjs', seedMetaKey: 'market:token-panels', intervalMs: 30 * MIN, timeoutMs: 120_000 }, +]); diff --git a/scripts/seed-bundle-portwatch.mjs b/scripts/seed-bundle-portwatch.mjs new file mode 100644 index 000000000..193772b80 --- /dev/null +++ b/scripts/seed-bundle-portwatch.mjs @@ -0,0 +1,9 @@ +#!/usr/bin/env node +import { runBundle, HOUR, WEEK } from './_bundle-runner.mjs'; + +await runBundle('portwatch', [ + { label: 'PW-Disruptions', script: 'seed-portwatch-disruptions.mjs', seedMetaKey: 'portwatch:disruptions', intervalMs: HOUR, timeoutMs: 120_000 }, + { label: 'PW-Main', script: 'seed-portwatch.mjs', seedMetaKey: 'supply_chain:portwatch', intervalMs: 6 * HOUR, timeoutMs: 300_000 }, + { label: 'PW-Port-Activity', script: 'seed-portwatch-port-activity.mjs', seedMetaKey: 'supply_chain:portwatch-ports', intervalMs: 12 * HOUR, timeoutMs: 600_000 }, + { label: 'PW-Chokepoints-Ref', script: 'seed-portwatch-chokepoints-ref.mjs', seedMetaKey: 'portwatch:chokepoints-ref', intervalMs: WEEK, timeoutMs: 120_000 }, +]); diff --git a/scripts/seed-bundle-relay-backup.mjs b/scripts/seed-bundle-relay-backup.mjs new file mode 100644 index 000000000..504d39840 --- /dev/null +++ b/scripts/seed-bundle-relay-backup.mjs @@ -0,0 +1,9 @@ +#!/usr/bin/env node +import { runBundle, MIN, HOUR, DAY } from './_bundle-runner.mjs'; + +await runBundle('relay-backup', [ + { label: 'Climate-News', script: 'seed-climate-news.mjs', seedMetaKey: 'climate:news-intelligence', intervalMs: 30 * MIN, timeoutMs: 240_000 }, + { label: 'USA-Spending', script: 'seed-usa-spending.mjs', seedMetaKey: 'economic:spending', intervalMs: HOUR, timeoutMs: 120_000 }, + { label: 'UCDP-Events', script: 'seed-ucdp-events.mjs', seedMetaKey: 'conflict:ucdp-events', intervalMs: 6 * HOUR, timeoutMs: 300_000 }, + { label: 'WB-Indicators', script: 'seed-wb-indicators.mjs', seedMetaKey: 'economic:worldbank-techreadiness:v1', intervalMs: DAY, timeoutMs: 300_000 }, +]); diff --git a/scripts/seed-bundle-resilience.mjs b/scripts/seed-bundle-resilience.mjs new file mode 100644 index 000000000..b7ae59b3e --- /dev/null +++ b/scripts/seed-bundle-resilience.mjs @@ -0,0 +1,7 @@ +#!/usr/bin/env node +import { runBundle, HOUR, DAY } from './_bundle-runner.mjs'; + +await runBundle('resilience', [ + { label: 'Resilience-Scores', script: 'seed-resilience-scores.mjs', seedMetaKey: 'resilience:intervals', intervalMs: 6 * HOUR, timeoutMs: 600_000 }, + { label: 'Resilience-Static', script: 'seed-resilience-static.mjs', seedMetaKey: 'resilience:static', intervalMs: 90 * DAY, timeoutMs: 900_000 }, +]); diff --git a/scripts/seed-bundle-static-ref.mjs b/scripts/seed-bundle-static-ref.mjs new file mode 100644 index 000000000..09238c655 --- /dev/null +++ b/scripts/seed-bundle-static-ref.mjs @@ -0,0 +1,8 @@ +#!/usr/bin/env node +import { runBundle, DAY, WEEK } from './_bundle-runner.mjs'; + +await runBundle('static-ref', [ + { label: 'Submarine-Cables', script: 'seed-submarine-cables.mjs', seedMetaKey: 'infrastructure:submarine-cables', intervalMs: WEEK, timeoutMs: 300_000 }, + { label: 'Chokepoint-Baselines', script: 'seed-chokepoint-baselines.mjs', seedMetaKey: 'energy:chokepoint-baselines', intervalMs: 400 * DAY, timeoutMs: 60_000 }, + { label: 'Military-Bases', script: 'seed-military-bases.mjs', seedMetaKey: 'military:bases', intervalMs: 30 * DAY, timeoutMs: 120_000 }, +]);