mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
64c906a406
* feat(eia): move /api/eia/petroleum to gold-standard (Railway seed → Redis → Vercel reads only)
Live api.eia.gov fetches from the Vercel edge function were causing
FUNCTION_INVOCATION_TIMEOUT 504s on /api/eia/petroleum (Sydney edge →
US origin with no timeout, no cache, no stale fallback — one EIA blip
blew the 25s budget).
- New seeder scripts/seed-eia-petroleum.mjs — fetches WTI/Brent/
production/inventory from api.eia.gov with per-fetch 15s timeouts,
writes energy:eia-petroleum:v1 with the {_seed, data} envelope.
Accepts 1-of-4 series; 0-of-4 routes to contract-mode RETRY so
seed-meta stays stale and the bundle retries on next cron.
- Bundled into seed-bundle-energy-sources.mjs (daily, 90s timeout) —
no new Railway service needed.
- Rewrote api/eia/[[...path]].js as a Redis-only reader via
readJsonFromUpstash. Same response shape for backward compat with
widgets/MCP/external callers. 503 + Retry-After on miss (never 504).
- Registered eiaPetroleum in api/health.js STANDALONE_KEYS + gated as
ON_DEMAND_KEYS for the deploy window; promote to SEED_META
(maxStaleMin: 4320) in a follow-up after ~7 days of clean cron.
- Tests: 14 seeder unit tests + 9 edge handler tests.
Audit result: /api/eia/petroleum was the only Vercel route fetching
dashboard data live. Every other fetch(https://…) in api/ is
auth/payments/notifications/user-initiated enrichment.
* fix(eia): close silent-stale window — add SEED_META + seed-health registration
Review finding on PR #3161: without a SEED_META entry, readSeedMeta
returns seedStale: null and classifyKey never reaches STALE_SEED.
That meant a broken Railway cron or missing EIA_API_KEY after the first
successful seed would keep /api/eia/petroleum serving stale data for
up to 7 days (TTL) while /api/health reported OK.
- api/health.js: add SEED_META.eiaPetroleum with maxStaleMin=4320
(72h = 3× daily bundle cadence). Keep eiaPetroleum in ON_DEMAND_KEYS
so the Vercel-instant / Railway-delayed deploy window doesn't CRIT
on first seed, but stale-after-seed now properly fires STALE_SEED.
- api/seed-health.js: register energy:eia-petroleum in SEED_DOMAINS
(intervalMin=1440) so the secondary health endpoint reports it too.
- Updated ON_DEMAND_KEYS comment to reflect freshness is now enforced.
20 lines
2.0 KiB
JavaScript
20 lines
2.0 KiB
JavaScript
#!/usr/bin/env node
|
|
import { runBundle, DAY } from './_bundle-runner.mjs';
|
|
|
|
await runBundle('energy-sources', [
|
|
{ label: 'GIE-Gas-Storage', script: 'seed-gie-gas-storage.mjs', seedMetaKey: 'economic:eu-gas-storage', canonicalKey: 'economic:eu-gas-storage:v1', intervalMs: DAY, timeoutMs: 180_000 },
|
|
{ label: 'Gas-Storage-Countries', script: 'seed-gas-storage-countries.mjs', seedMetaKey: 'energy:gas-storage-countries', intervalMs: DAY, timeoutMs: 600_000 },
|
|
{ label: 'JODI-Gas', script: 'seed-jodi-gas.mjs', seedMetaKey: 'energy:jodi-gas', canonicalKey: 'energy:jodi-gas:v1:_countries', intervalMs: 35 * DAY, timeoutMs: 600_000 },
|
|
{ label: 'JODI-Oil', script: 'seed-jodi-oil.mjs', seedMetaKey: 'energy:jodi-oil', canonicalKey: 'energy:jodi-oil:v1:_countries', intervalMs: 35 * DAY, timeoutMs: 600_000 },
|
|
{ label: 'OWID-Energy-Mix', script: 'seed-owid-energy-mix.mjs', seedMetaKey: 'economic:owid-energy-mix', intervalMs: 35 * DAY, timeoutMs: 600_000 },
|
|
{ label: 'IEA-Oil-Stocks', script: 'seed-iea-oil-stocks.mjs', seedMetaKey: 'energy:iea-oil-stocks', canonicalKey: 'energy:iea-oil-stocks:v1:index', intervalMs: 40 * DAY, timeoutMs: 300_000 },
|
|
{ label: 'EIA-Petroleum', script: 'seed-eia-petroleum.mjs', seedMetaKey: 'energy:eia-petroleum', canonicalKey: 'energy:eia-petroleum:v1', intervalMs: DAY, timeoutMs: 90_000 },
|
|
{ label: 'IEA-Crisis-Policies', script: 'seed-energy-crisis-policies.mjs', seedMetaKey: 'energy:crisis-policies', canonicalKey: 'energy:crisis-policies:v1', intervalMs: 7 * DAY, timeoutMs: 120_000 },
|
|
// SPR-Policies: static registry (data lives in scripts/data/spr-policies.json), TTL 400d
|
|
// in api/health.js (maxStaleMin: 576000). Weekly cadence is generous — only needs to run
|
|
// once after deploys + restarts to populate energy:spr-policies:v1. No prior Railway
|
|
// service exists for it, so health has been EMPTY (seedAgeMin: null) since the seeder
|
|
// was added.
|
|
{ label: 'SPR-Policies', script: 'seed-spr-policies.mjs', seedMetaKey: 'energy:spr-policies', canonicalKey: 'energy:spr-policies:v1', intervalMs: 7 * DAY, timeoutMs: 60_000 },
|
|
]);
|