eliott/worldmonitor
1
0
Fork 0
mirror of https://github.com/koala73/worldmonitor.git synced 2026-04-25 17:14:57 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
worldmonitor/api/seed-health.js
Elie Habib 8cca8d19e3 feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read (#3385) 
* feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup

Prereq for the re-export-share Comtrade seeder (plan 2026-04-24-003),
usable by any cohort seeder whose consumer needs bundle-level freshness.

Two coupled changes:

1. `_bundle-runner.mjs` injects `BUNDLE_RUN_STARTED_AT_MS` into every
   spawned child. All siblings in a single bundle run share one value
   (captured at `runBundle` start, not spawn time). Consumers use this
   to detect stale peer keys — if a peer's seed-meta predates the
   current bundle run, fall back to a hard default rather than read
   a cohort-peer's last-week output.

2. `_seed-utils.mjs::runSeed` registers a `process.once('SIGTERM')`
   handler that releases the acquired lock and extends existing-data
   TTL before exiting 143. `_bundle-runner.mjs` sends SIGTERM on
   section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without
   this handler the `finally` path never runs on SIGKILL, leaving
   the 30-min acquireLock reservation in place until its own TTL
   expires — the next cron tick silently skips the resource.

Regression guard memory: `bundle-runner-sigkill-leaks-child-lock` (PR
#3128 root cause).

Tests added:
- bundle-runner env injection (value within run bounds)
- sibling sections share the same timestamp (critical for the
  consumer freshness guard)
- runSeed SIGTERM path: exit 143 + cleanup log
- process.once contract: second SIGTERM does not re-enter handler

* fix(seed): address P1/P2 review findings on SIGTERM + bundle contracts

Addresses PR #3384 review findings (todos 256, 257, 259, 260):

#256 (P1) — SIGTERM handler narrowed to fetch phase only. Was installed
at runSeed entry and armed through every `process.exit` path; could
race `emptyDataIsFailure: true` strict-floor exits (IMF-External,
WB-bulk) and extend seed-meta TTL when the contract forbids it —
silently re-masking 30-day outages. Now the handler is attached
immediately before `withRetry(fetchFn)` and removed in a try/finally
that covers all fetch-phase exit branches.

#257 (P1) — `BUNDLE_RUN_STARTED_AT_MS` now has a first-class helper.
Exported `getBundleRunStartedAtMs()` from `_seed-utils.mjs` with JSDoc
describing the bundle-freshness contract. Fleet-wide helper so the
next consumer seeder imports instead of rediscovering the idiom.

#259 (P2) — SIGTERM cleanup runs `Promise.allSettled` on disjoint-key
ops (`releaseLock` + `extendExistingTtl`). Serialising compounded
Upstash latency during the exact failure mode (Redis degraded) this
handler exists to handle, risking breach of the 5s SIGKILL grace.

#260 (P2) — `_bundle-runner.mjs` asserts topological order on
optional `dependsOn` section field. Throws on unknown-label refs and
on deps appearing at a later index. Fleet-wide contract replacing
the previous prose-comment ordering guarantee.

Tests added/updated:
- New: SIGTERM handler removed after fetchFn completes (narrowed-scope
  contract — post-fetch SIGTERM must NOT trigger TTL extension)
- New: dependsOn unknown-label + out-of-order + happy-path (3 tests)

Full test suite: 6,866 tests pass (+4 net).

* fix(seed): getBundleRunStartedAtMs returns null outside a bundle run

Review follow-up: the earlier `Math.floor(Date.now()/1000)*1000` fallback
regressed standalone (non-bundle) runs. A consumer seeder invoked
manually just after its peer wrote `fetchedAt = (now - 5s)` would see
`bundleStartMs = Date.now()`, reject the perfectly-fresh peer envelope
as "stale", and fall back to defaults — defeating the point of the
peer-read path outside the bundle.

Returning null when `BUNDLE_RUN_STARTED_AT_MS` is unset/invalid keeps
the freshness gate scoped to its real purpose (across-bundle-tick
staleness) and lets standalone runs skip the gate entirely. Consumers
check `bundleStartMs != null` before applying the comparison; see the
companion `seed-sovereign-wealth.mjs` change on the stacked PR.

* test(seed): SIGTERM cleanup test now verifies Redis DEL + EXPIRE calls

Greptile review P2 on PR #3384: the existing test only asserted exit
code + log line, not that the Redis ops were actually issued. The
log claim was ahead of the test.

Fixture now logs every Upstash fetch call's shape (EVAL / pipeline-
EXPIRE / other) to stderr. Test asserts:

- >=1 EVAL op was issued during SIGTERM cleanup (releaseLock Lua
  script on the lock key)
- >=1 pipeline-EXPIRE op was issued (extendExistingTtl on canonical
  + seed-meta keys)
- The EVAL body carries the runSeed-generated runId (proves it's
  THIS run's release, not a phantom op)
- The EXPIRE pipeline touches both the canonicalKey AND the
  seed-meta key (proves the keys[] array was built correctly
  including the extraKeys merge path)

Full test suite: 6,866 tests pass, typecheck clean.

* feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read

Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md

Motivating case. Before this PR, the SWF `rawMonths` denominator for
the `sovereignFiscalBuffer` dimension used GROSS annual imports for
every country. For re-export hubs (goods transiting without domestic
settlement), this structurally under-reports resilience: UAE's 2023
$941B of imports include $334B of transit flow that never represents
domestic consumption. Net imports = gross × (1 − reexport_share).

The previous (PR 3A) design flattened a hand-curated YAML into Redis;
the YAML shipped empty and never populated, so the correction never
applied and the cohort audit showed no movement.

Gap #2 (this PR). Two coupled changes to make the correction actually
apply:

1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`).
   Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and
   `flowCode=M` (imports) per cohort member, compute share = RX/M at
   the latest co-populated year, clamp to [0.05, 0.95], publish the
   envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription
   key never reaches URL/logs/Redis. `maxRecords=250000` cap with
   truncation detection. Sequential + retry-on-429 with backoff.

   Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0):
   ['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200
   with zero RX rows — Comtrade doesn't expose RX for those reporters.

2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`).
   Swaps `loadReexportShareByCountry()` (YAML) for
   `loadReexportShareFromRedis()` (Redis key written by #1). Guarded
   by bundle-run freshness: if the sibling Reexport-Share seeder's
   `seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the
   prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback
   to gross imports rather than apply last-month's stale share.

Health registries. Both new keys registered in BOTH `api/health.js`
SEED_META (60-day alert threshold) and `api/seed-health.js`
SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match.

Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share
timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min
worst-case). Ordering preserved: Reexport-Share before Sovereign-
Wealth so the SWF seeder reads a freshly-written key in the same
cron tick.

Deletions. YAML + loader + 7 obsolete loader tests removed; single
source of truth is now Comtrade → Redis.

Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm)
which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed
SIGTERM cleanup. This PR's bundle-freshness guard depends on
that env variable.

Tests (19 new, 7 deleted, +12 net):
- Pure math: parseComtradeFlowResponse, computeShareFromFlows,
  clampShare, declareRecords + credential-leak source scan (15)
- Integration (Gap #2 regression guards): SWF seeder loadReexport
  ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5)
- Health registry dual-registry drift guard — scoped to this PR's
  keys, respecting pre-existing asymmetry (4)
- Bundle-ordering + timeout assertions (2)

Phase 0 cohort validation committed to plan. Full test suite
passes: 6,881 tests.

* fix(resilience): address P1/P2 review findings — adopt shared helpers, pin freshness boundary

Addresses PR #3385 review findings:

#257 (P1) consumer — `seed-sovereign-wealth.mjs` imports the shared
`getBundleRunStartedAtMs` helper from `_seed-utils.mjs` (added in the
prereq commit) instead of its own `getBundleStartMs`. Single source of
truth for the bundle-freshness contract.

#258 (P2) — `seed-recovery-reexport-share.mjs` isMain guard uses the
canonical `pathToFileURL(process.argv[1]).href === import.meta.url`
form instead of basename-suffix matching. Handles symlinks, case-
different paths on macOS HFS+, and Windows path separators without
string munging.

#260 (P2) consumer — Sovereign-Wealth declares `dependsOn:
['Reexport-Share']` in the bundle spec. `_bundle-runner.mjs` (prereq
commit) now enforces topological order on load and throws on
violation — replaces the previous prose-comment ordering contract.

#261 (P2) — added a test to `tests/seed-sovereign-wealth-reads-redis-
reexport-share.test.mts` pinning the inclusive-boundary semantic:
`fetchedAtMs === bundleStartMs` must be treated as FRESH. Guards
against a future refactor to `<=` that would silently reject peers
writing at the very first millisecond of the bundle run.

Rebased onto updated prereq. Full test suite: 6,886 tests pass (+5 net).

* fix(resilience): freshness gate skipped in standalone mode; meta still required

Review catch: the previous `bundleStartMs = Date.now()` fallback made
standalone/manual `seed-sovereign-wealth.mjs` runs ALWAYS reject any
previously-seeded re-export-share meta as "stale" — even when the
operator ran the Reexport seeder milliseconds beforehand. Defeated
the point of the peer-read path outside the bundle.

With `getBundleRunStartedAtMs()` now returning null outside a bundle
(companion commit on the prereq branch), the consumer only applies
the freshness gate when `bundleStartMs != null`. Standalone runs
accept any `fetchedAt` — the operator is responsible for ordering.

Two guards survive the change:
- Meta MUST exist (absence = peer-outage fail-safe, both modes)
- In-bundle: meta MUST be at or after `BUNDLE_RUN_STARTED_AT_MS`

Two new tests pin both modes:
- standalone: accepts meta written 10 min before this process started
- standalone: still rejects missing meta (peer-outage fail-safe
  survives gate bypass)

Rebased onto updated prereq. Full test suite: 6,888 tests (+2 net).

* fix(resilience): filter world-aggregate Comtrade rows + skip final-retry sleep

Greptile review of PR #3385 flagged two P2s in the Comtrade seeder.

Finding #3 (parseComtradeFlowResponse double-count risk):
`cmdCode=TOTAL` without a partner filter currently returns only
world-aggregate rows in practice — but `parseComtradeFlowResponse`
summed every row unconditionally. A future refactor adding per-
partner querying would silently double-count (world-aggregate row +
partner-level rows for the same year), cutting the derived share in
half with no test signal.

Fix: explicit `partnerCode ∈ {'0', 0, null/undefined}` filter. Matches
current empirical behavior (aggregate-only responses) and makes the
construct robust to a future partner-level query.

Finding #4 (wasted backoff on final retry):
429 and 5xx branches slept `backoffMs` before `continue`, but on
`attempt === RETRY_MAX_ATTEMPTS` the loop condition fails immediately
after — the sleep was pure waste. Added early-return (parallel to the
existing pattern in the network-error catch branch) so the final
attempt exits the retry loop at the first non-success response
without extra latency.

Tests:
- 3 new `parseComtradeFlowResponse` variants: world-only filter,
  numeric-0 partnerCode shape, rows without partnerCode field
- Existing tests updated: the double-count assertion replaced with
  a "per-partner rows must NOT sum into the world-aggregate total"
  assertion that pins the new contract

Rebased onto updated prereq. Full test suite: 6,890 tests (+2 net).
2026-04-25 00:14:17 +04:00

185 lines
14 KiB
JavaScript
Raw Permalink Blame History

import { getCorsHeaders, isDisallowedOrigin } from './_cors.js';
import { validateApiKey } from './_api-key.js';
import { jsonResponse } from './_json-response.js';
// @ts-expect-error — JS module, no declaration file
import { redisPipeline } from './_upstash-json.js';
export const config = { runtime: 'edge' };
const SEED_DOMAINS = {
// Phase 1 — Snapshot endpoints
'seismology:earthquakes': { key: 'seed-meta:seismology:earthquakes', intervalMin: 15 },
'wildfire:fires': { key: 'seed-meta:wildfire:fires', intervalMin: 60 },
'infra:outages': { key: 'seed-meta:infra:outages', intervalMin: 15 },
'climate:anomalies': { key: 'seed-meta:climate:anomalies', intervalMin: 120 },
'climate:disasters': { key: 'seed-meta:climate:disasters', intervalMin: 360 },
'climate:zone-normals': { key: 'seed-meta:climate:zone-normals', intervalMin: 44640 },
'climate:co2-monitoring': { key: 'seed-meta:climate:co2-monitoring', intervalMin: 1440 }, // daily cron; health.js maxStaleMin:4320 (3x) is intentionally higher — it's an alarm threshold, not the cron cadence
'climate:ocean-ice': { key: 'seed-meta:climate:ocean-ice', intervalMin: 1440 }, // daily cron; health.js maxStaleMin:2880 (2x) tolerates one missed run
'climate:news-intelligence': { key: 'seed-meta:climate:news-intelligence', intervalMin: 30 },
// Phase 2 — Parameterized endpoints
'unrest:events': { key: 'seed-meta:unrest:events', intervalMin: 15 },
'cyber:threats': { key: 'seed-meta:cyber:threats', intervalMin: 240 },
'market:crypto': { key: 'seed-meta:market:crypto', intervalMin: 15 },
'market:hyperliquid-flow': { key: 'seed-meta:market:hyperliquid-flow', intervalMin: 5 }, // Railway cron 5min via seed-bundle-market-backup
'market:etf-flows': { key: 'seed-meta:market:etf-flows', intervalMin: 30 },
'market:gulf-quotes': { key: 'seed-meta:market:gulf-quotes', intervalMin: 15 },
'market:stablecoins': { key: 'seed-meta:market:stablecoins', intervalMin: 30 },
// Phase 3 — Hybrid endpoints
'natural:events': { key: 'seed-meta:natural:events', intervalMin: 60 },
'displacement:summary': { key: 'seed-meta:displacement:summary', intervalMin: 360 },
// Aligned with health.js SEED_META (intervalMin = maxStaleMin / 2)
'market:stocks': { key: 'seed-meta:market:stocks', intervalMin: 15 },
'market:commodities': { key: 'seed-meta:market:commodities', intervalMin: 15 },
'market:gold-extended': { key: 'seed-meta:market:gold-extended', intervalMin: 15 },
'market:gold-etf-flows': { key: 'seed-meta:market:gold-etf-flows', intervalMin: 1440 },
// maxStaleMin in health.js is 44640 (~31 days; IMF IFS is monthly w/ 2-3mo lag).
// This endpoint flags stale at intervalMin*2, so keep intervalMin = 22320 to match.
'market:gold-cb-reserves': { key: 'seed-meta:market:gold-cb-reserves', intervalMin: 22320 },
'market:sectors': { key: 'seed-meta:market:sectors', intervalMin: 15 },
'aviation:faa': { key: 'seed-meta:aviation:faa', intervalMin: 45 },
'news:insights': { key: 'seed-meta:news:insights', intervalMin: 15 },
'positive-events:geo': { key: 'seed-meta:positive-events:geo', intervalMin: 30 },
'risk:scores:sebuf': { key: 'seed-meta:risk:scores:sebuf', intervalMin: 15 },
'conflict:iran-events': { key: 'seed-meta:conflict:iran-events', intervalMin: 5040 },
'conflict:ucdp-events': { key: 'seed-meta:conflict:ucdp-events', intervalMin: 210 },
'weather:alerts': { key: 'seed-meta:weather:alerts', intervalMin: 15 },
'economic:spending': { key: 'seed-meta:economic:spending', intervalMin: 60 },
'intelligence:gpsjam': { key: 'seed-meta:intelligence:gpsjam', intervalMin: 360 },
'intelligence:satellites': { key: 'seed-meta:intelligence:satellites', intervalMin: 90 },
'military:flights': { key: 'seed-meta:military:flights', intervalMin: 8 },
'military-forecast-inputs': { key: 'seed-meta:military-forecast-inputs', intervalMin: 8 },
'infra:service-statuses': { key: 'seed-meta:infra:service-statuses', intervalMin: 60 },
'supply_chain:shipping': { key: 'seed-meta:supply_chain:shipping', intervalMin: 120 },
'supply_chain:chokepoints': { key: 'seed-meta:supply_chain:chokepoints', intervalMin: 30 },
'cable-health': { key: 'seed-meta:cable-health', intervalMin: 30 },
'prediction:markets': { key: 'seed-meta:prediction:markets', intervalMin: 8 },
'aviation:intl': { key: 'seed-meta:aviation:intl', intervalMin: 15 },
'theater-posture': { key: 'seed-meta:theater-posture', intervalMin: 8 },
'economic:worldbank-techreadiness': { key: 'seed-meta:economic:worldbank-techreadiness:v1', intervalMin: 5040 },
'economic:worldbank-progress': { key: 'seed-meta:economic:worldbank-progress:v1', intervalMin: 5040 },
'economic:worldbank-renewable': { key: 'seed-meta:economic:worldbank-renewable:v1', intervalMin: 5040 },
'economic:bis-extended': { key: 'seed-meta:economic:bis-extended', intervalMin: 720 }, // 12h Railway cron; "seeder ran" aggregate — per-dataset freshness lives below
'economic:bis-dsr': { key: 'seed-meta:economic:bis-dsr', intervalMin: 720 }, // 12h cron; only written when DSR slice fetched fresh entries
'economic:bis-property-residential': { key: 'seed-meta:economic:bis-property-residential', intervalMin: 720 }, // 12h cron; only written when SPP slice fetched fresh entries
'economic:bis-property-commercial': { key: 'seed-meta:economic:bis-property-commercial', intervalMin: 720 }, // 12h cron; only written when CPP slice fetched fresh entries
'research:tech-events': { key: 'seed-meta:research:tech-events', intervalMin: 240 },
'intelligence:gdelt-intel': { key: 'seed-meta:intelligence:gdelt-intel', intervalMin: 210 }, // 420min maxStaleMin / 2 — aligned with health.js (6h cron + 1h grace)
'correlation:cards': { key: 'seed-meta:correlation:cards', intervalMin: 5 },
'intelligence:advisories': { key: 'seed-meta:intelligence:advisories', intervalMin: 60 },
'intelligence:social-reddit': { key: 'seed-meta:intelligence:social-reddit', intervalMin: 90 }, // 60min relay loop (hourly, bumped from 10min to reduce Reddit IP blocking); intervalMin = maxStaleMin / 2 (180 / 2)
'intelligence:wsb-tickers': { key: 'seed-meta:intelligence:wsb-tickers', intervalMin: 90 }, // 60min relay loop (hourly, bumped from 10min); intervalMin = maxStaleMin / 2 (180 / 2)
'trade:customs-revenue': { key: 'seed-meta:trade:customs-revenue', intervalMin: 720 },
'thermal:escalation': { key: 'seed-meta:thermal:escalation', intervalMin: 180 },
'radiation:observations': { key: 'seed-meta:radiation:observations', intervalMin: 15 },
'sanctions:pressure': { key: 'seed-meta:sanctions:pressure', intervalMin: 360 },
'health:air-quality': { key: 'seed-meta:health:air-quality', intervalMin: 60 }, // hourly cron (shared seeder writes health + climate keys)
'economic:grocery-basket': { key: 'seed-meta:economic:grocery-basket', intervalMin: 5040 }, // weekly seed; intervalMin = maxStaleMin / 2
'economic:bigmac': { key: 'seed-meta:economic:bigmac', intervalMin: 5040 }, // weekly seed; intervalMin = maxStaleMin / 2
'resilience:static': { key: 'seed-meta:resilience:static', intervalMin: 288000 }, // annual October snapshot; intervalMin = health.js maxStaleMin / 2 (400d alert threshold)
'resilience:intervals': { key: 'seed-meta:resilience:intervals', intervalMin: 10080 }, // weekly cron; intervalMin = health.js maxStaleMin / 2 (20160 / 2)
'regulatory:actions': { key: 'seed-meta:regulatory:actions', intervalMin: 120 }, // 2h cron; intervalMin = maxStaleMin / 3
'economic:owid-energy-mix': { key: 'seed-meta:economic:owid-energy-mix', intervalMin: 25200 }, // monthly cron on 1st; intervalMin = health.js maxStaleMin / 2 (50400 / 2)
'economic:fao-ffpi': { key: 'seed-meta:economic:fao-ffpi', intervalMin: 43200 }, // monthly seed; intervalMin = health.js maxStaleMin / 2 (86400 / 2)
'economic:imf-growth': { key: 'seed-meta:economic:imf-growth', intervalMin: 50400 }, // monthly WEO seed; intervalMin = health.js maxStaleMin / 2 (100800 / 2)
'economic:imf-labor': { key: 'seed-meta:economic:imf-labor', intervalMin: 50400 }, // monthly WEO seed; intervalMin = health.js maxStaleMin / 2 (100800 / 2)
'economic:imf-external': { key: 'seed-meta:economic:imf-external', intervalMin: 50400 }, // monthly WEO seed; intervalMin = health.js maxStaleMin / 2 (100800 / 2)
'product-catalog': { key: 'seed-meta:product-catalog', intervalMin: 360 }, // relay loop every 6h; intervalMin = health.js maxStaleMin / 3 (1080 / 3)
'portwatch:chokepoints-ref': { key: 'seed-meta:portwatch:chokepoints-ref', intervalMin: 10080 }, // seed-bundle-portwatch runs this at WEEK cadence; intervalMin*2 = 14d matches api/health.js SEED_META.portwatchChokepointsRef
'supply_chain:portwatch-ports': { key: 'seed-meta:supply_chain:portwatch-ports', intervalMin: 720 }, // 12h cron (0 */12 * * *); intervalMin = maxStaleMin / 3 (2160 / 3)
'energy:chokepoint-flows': { key: 'seed-meta:energy:chokepoint-flows', intervalMin: 360 }, // 6h relay loop; intervalMin = maxStaleMin / 2 (720 / 2)
'energy:eia-petroleum': { key: 'seed-meta:energy:eia-petroleum', intervalMin: 1440 }, // daily bundle cron; intervalMin*3 = health.js maxStaleMin (4320)
'energy:spine': { key: 'seed-meta:energy:spine', intervalMin: 1440 }, // daily cron (0 6 * * *); intervalMin = maxStaleMin / 2 (2880 / 2)
'energy:ember': { key: 'seed-meta:energy:ember', intervalMin: 1440 }, // daily cron (0 8 * * *); intervalMin = maxStaleMin / 2 (2880 / 2)
'energy:spr-policies': { key: 'seed-meta:energy:spr-policies', intervalMin: 288000 }, // annual static registry; intervalMin = health.js maxStaleMin / 2 (576000 / 2)
'energy:pipelines-gas': { key: 'seed-meta:energy:pipelines-gas', intervalMin: 10080 }, // weekly cron (7d); intervalMin = health.js maxStaleMin / 2 (20160 / 2)
'energy:pipelines-oil': { key: 'seed-meta:energy:pipelines-oil', intervalMin: 10080 }, // weekly cron; same seeder writes both keys
'energy:storage-facilities': { key: 'seed-meta:energy:storage-facilities', intervalMin: 10080 }, // weekly cron (7d); intervalMin = health.js maxStaleMin / 2 (20160 / 2)
'energy:fuel-shortages': { key: 'seed-meta:energy:fuel-shortages', intervalMin: 1440 }, // daily cron; intervalMin = health.js maxStaleMin / 2 (2880 / 2)
'energy:disruptions': { key: 'seed-meta:energy:disruptions', intervalMin: 10080 }, // weekly cron; intervalMin = health.js maxStaleMin / 2 (20160 / 2)
'market:aaii-sentiment': { key: 'seed-meta:market:aaii-sentiment', intervalMin: 10080 }, // weekly cron; intervalMin = maxStaleMin / 2 (20160 / 2)
'intelligence:regional-briefs': { key: 'seed-meta:intelligence:regional-briefs', intervalMin: 10080 }, // weekly cron; intervalMin = health.js maxStaleMin / 2 (20160 / 2)
'economic:eurostat-house-prices': { key: 'seed-meta:economic:eurostat-house-prices', intervalMin: 36000 }, // weekly cron, annual data; intervalMin = health.js maxStaleMin / 2 (72000 / 2)
'economic:eurostat-gov-debt-q': { key: 'seed-meta:economic:eurostat-gov-debt-q', intervalMin: 10080 }, // 2d cron, quarterly data; intervalMin = health.js maxStaleMin / 2 (20160 / 2)
'economic:eurostat-industrial-production': { key: 'seed-meta:economic:eurostat-industrial-production', intervalMin: 3600 }, // daily cron, monthly data; intervalMin = health.js maxStaleMin / 2 (7200 / 2)
'resilience:recovery:reexport-share': { key: 'seed-meta:resilience:recovery:reexport-share', intervalMin: 43200 }, // monthly bundle cron (30d); intervalMin*2 = 60d matches health.js maxStaleMin
'resilience:recovery:sovereign-wealth': { key: 'seed-meta:resilience:recovery:sovereign-wealth', intervalMin: 43200 }, // monthly bundle cron (30d); intervalMin*2 = 60d matches health.js maxStaleMin
};
async function getMetaBatch(keys) {
const pipeline = keys.map((k) => ['GET', k]);
const data = await redisPipeline(pipeline, 3000);
if (!data) throw new Error('Redis not configured');
const result = new Map();
for (let i = 0; i < keys.length; i++) {
const raw = data[i]?.result;
if (raw) {
try { result.set(keys[i], JSON.parse(raw)); } catch { /* skip */ }
}
}
return result;
}
export default async function handler(req) {
if (isDisallowedOrigin(req))
return new Response('Forbidden', { status: 403 });
const cors = getCorsHeaders(req);
if (req.method === 'OPTIONS')
return new Response(null, { status: 204, headers: cors });
const apiKeyResult = validateApiKey(req);
if (apiKeyResult.required && !apiKeyResult.valid)
return jsonResponse({ error: apiKeyResult.error }, 401, cors);
const now = Date.now();
const entries = Object.entries(SEED_DOMAINS);
const metaKeys = entries.map(([, v]) => v.key);
let metaMap;
try {
metaMap = await getMetaBatch(metaKeys);
} catch {
return jsonResponse({ error: 'Redis unavailable' }, 503, cors);
}
const seeds = {};
let staleCount = 0;
let missingCount = 0;
for (const [domain, cfg] of entries) {
const meta = metaMap.get(cfg.key);
const maxStalenessMs = cfg.intervalMin * 2 * 60 * 1000;
if (!meta) {
seeds[domain] = { status: 'missing', fetchedAt: null, recordCount: null, stale: true };
missingCount++;
continue;
}
const ageMs = now - (meta.fetchedAt || 0);
const isError = meta.status === 'error';
const stale = ageMs > maxStalenessMs || isError;
if (stale) staleCount++;
seeds[domain] = {
status: stale ? (isError ? 'error' : 'stale') : 'ok',
fetchedAt: meta.fetchedAt,
recordCount: meta.recordCount ?? null,
sourceVersion: meta.sourceVersion || null,
ageMinutes: Math.round(ageMs / 60000),
stale,
};
}
const overall = missingCount > 0 ? 'degraded' : staleCount > 0 ? 'warning' : 'healthy';
const httpStatus = overall === 'healthy' ? 200 : overall === 'warning' ? 200 : 503;
return jsonResponse({ overall, seeds, checkedAt: now }, httpStatus, {
...cors,
'Cache-Control': 'no-cache',
});
}
Reference in New Issue View Git Blame Copy Permalink