eliott/worldmonitor
1
0
Fork 0
mirror of https://github.com/koala73/worldmonitor.git synced 2026-04-25 17:14:57 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
46d17efe551693e728aa9e19fc1889bd34147363
worldmonitor/api/health.js
Elie Habib 46d17efe55 fix(resilience): wider FX YoY upstream + sanctions absolute threshold (#3071) 
* fix(resilience): wider FX YoY upstream + sanctions absolute threshold

Two backtest families consistently failed Outcome-Backtest gates because
the detectors were reading the wrong shape of upstream data, not because
the upstream seeders were missing.

FX Stress (was AUC=0.500):
- BIS WS_EER (`economic:bis:eer:v1`) only covers 12 G10/major-EM countries
  — Argentina, Egypt, Turkey, Pakistan, Nigeria etc. are absent, so the
  detector had no positive events to score against
- Add `seed-fx-yoy.mjs` fetching Yahoo Finance 2-year monthly history
  across 45 single-country currencies, computing YoY % and 24-month
  peak-to-trough drawdown
- Switch detector to read drawdown24m with -15% threshold (matches
  methodology spec); falls back to yoyChange/realChange for back-compat
- Why drawdown not just YoY: rolling 12-month windows slice through
  historic crises (Egypt's March 2024 devaluation falls outside an
  April→April window by 2026); drawdown captures actual stress magnitude
  regardless of crisis timing
- Verified locally: flags AR (-38%), TR (-28%), NG (-21%), MX (-18%)

Sanctions Shocks (was AUC=0.624):
- Detector previously used top-quartile (Q3) of country-counts which
  conflated genuine comprehensive-sanctions targets (RU, IR, KP, CU,
  SY, VE, BY, MM) with financial hubs (UK, CH, DE, US) hosting many
  sanctioned entities
- Replace with absolute threshold of 100 entities — the OFAC
  distribution is heavy-tailed enough that this cleanly separates
  targets from hubs

Both fixes use existing seeded data (or new seeded data via
seed-fx-yoy.mjs) — no hardcoded country curation.

api/health.js: register `economic:fx:yoy:v1` in STANDALONE_KEYS +
SEED_META so the validation cron monitors freshness.

Railway: deploy `seed-fx-yoy` as a daily cron service (NIXPACKS
builder, startCommand `node scripts/seed-fx-yoy.mjs`, schedule
`30 6 * * *`).

* fix(seed-fx-yoy): use running-peak max-drawdown instead of global peak

PR #3071 review (P1): the original drawdown calculation found the global
maximum across the entire window, then the lowest point AFTER that peak.
This silently erased earlier crashes when the currency later recovered to
a new high — exactly the class of events the FX Stress family is trying
to capture.

Example series [5, 10, 7, 9, 6, 11, 10]: true worst drawdown is 10 → 6 =
-40%, but the broken implementation picked the later global peak 11 and
reported only 11 → 10 = -9.1%.

Fix: sweep forward tracking the running peak; for each subsequent bar
compute the drop from that running peak; keep the largest such drop.
This is the standard max-drawdown computation and correctly handles
recover-then-fall-again sequences.

Live data verification:
- BR (Brazilian real) was missing from the flagged set under the broken
  algorithm because BRL recovered above its 2024-04 peak. With the fix it
  correctly surfaces at drawdown=-15.8% (peak 2024-04, trough 2024-12).
- KR / CO peaks now resolve to mid-series dates instead of end-of-window,
  proving the running-peak scan is finding intermediate peaks.

Tests added covering: reviewer's regression case, peak-at-start (NGN
style), pure appreciation, multi-trough series, yoyChange anchor.

* fix(health): gate fxYoy as on-demand to avoid post-merge CRIT alarm

PR #3071 review (P1): registering `fxYoy` as a required standalone
seeded key creates an operational hazard during the deploy gap. After
merge, Vercel auto-deploys `api/health.js` immediately, but the
`seed-fx-yoy` Railway cron lives in a separate deployment surface that
must be triggered manually. Any gap (deploy-order race, first-cron
failure, env var typo) flips health to DEGRADED/UNHEALTHY because
`classifyKey()` marks the check as `EMPTY` without an on-demand or
empty-data-OK exemption.

Add `fxYoy` to ON_DEMAND_KEYS as a transitional safety net (matches the
pattern recovery* uses for "stub seeders not yet deployed"). The key is
still monitored — freshness via seed-meta — but missing data downgrades
from CRIT to WARN, which won't page anyone. Once the Railway cron has
fired cleanly for ~7 days in production we can remove this entry and
let it be a hard-required key like the rest of the FRED family.

Note: the Railway service IS already provisioned (cron `30 6 * * *`,
0.5 vCPU / 0.5 GB, NIXPACKS, watchPatterns scoped to the seeder + utils)
and the `economic:fx:yoy:v1` key is currently fresh in production from
local test runs. The gating here is defense-in-depth against the
operational coupling, not against a known absent key.
2026-04-13 21:57:11 +04:00

671 lines
44 KiB
JavaScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { jsonResponse } from './_json-response.js';
// @ts-expect-error — JS module, no declaration file
import { redisPipeline, getRedisCredentials } from './_upstash-json.js';
export const config = { runtime: 'edge' };
const BOOTSTRAP_KEYS = {
earthquakes: 'seismology:earthquakes:v1',
outages: 'infra:outages:v1',
sectors: 'market:sectors:v2',
etfFlows: 'market:etf-flows:v1',
climateAnomalies: 'climate:anomalies:v2',
climateDisasters: 'climate:disasters:v1',
climateAirQuality: 'climate:air-quality:v1',
co2Monitoring: 'climate:co2-monitoring:v1',
oceanIce: 'climate:ocean-ice:v1',
wildfires: 'wildfire:fires:v1',
marketQuotes: 'market:stocks-bootstrap:v1',
commodityQuotes: 'market:commodities-bootstrap:v1',
cyberThreats: 'cyber:threats-bootstrap:v2',
techReadiness: 'economic:worldbank-techreadiness:v1',
progressData: 'economic:worldbank-progress:v1',
renewableEnergy: 'economic:worldbank-renewable:v1',
positiveGeoEvents: 'positive_events:geo-bootstrap:v1',
riskScores: 'risk:scores:sebuf:stale:v1',
naturalEvents: 'natural:events:v1',
flightDelays: 'aviation:delays-bootstrap:v1',
newsInsights: 'news:insights:v1',
predictionMarkets: 'prediction:markets-bootstrap:v1',
cryptoQuotes: 'market:crypto:v1',
gulfQuotes: 'market:gulf-quotes:v1',
stablecoinMarkets: 'market:stablecoins:v1',
unrestEvents: 'unrest:events:v1',
iranEvents: 'conflict:iran-events:v1',
ucdpEvents: 'conflict:ucdp-events:v1',
weatherAlerts: 'weather:alerts:v1',
spending: 'economic:spending:v1',
techEvents: 'research:tech-events-bootstrap:v1',
gdeltIntel: 'intelligence:gdelt-intel:v1',
correlationCards: 'correlation:cards-bootstrap:v1',
forecasts: 'forecast:predictions:v2',
securityAdvisories: 'intelligence:advisories-bootstrap:v1',
customsRevenue: 'trade:customs-revenue:v1',
comtradeFlows: 'comtrade:flows:v1',
blsSeries: 'bls:series:v1',
sanctionsPressure: 'sanctions:pressure:v1',
crossSourceSignals: 'intelligence:cross-source-signals:v1',
sanctionsEntities: 'sanctions:entities:v1',
radiationWatch: 'radiation:observations:v1',
consumerPricesOverview: 'consumer-prices:overview:ae',
consumerPricesCategories: 'consumer-prices:categories:ae:30d',
consumerPricesMovers: 'consumer-prices:movers:ae:30d',
consumerPricesSpread: 'consumer-prices:retailer-spread:ae:essentials-ae',
consumerPricesFreshness: 'consumer-prices:freshness:ae',
groceryBasket: 'economic:grocery-basket:v1',
bigmac: 'economic:bigmac:v1',
fuelPrices: 'economic:fuel-prices:v1',
faoFoodPriceIndex: 'economic:fao-ffpi:v1',
nationalDebt: 'economic:national-debt:v1',
defiTokens: 'market:defi-tokens:v1',
aiTokens: 'market:ai-tokens:v1',
otherTokens: 'market:other-tokens:v1',
fredBatch: 'economic:fred:v1:FEDFUNDS:0',
ecbEstr: 'economic:fred:v1:ESTR:0',
ecbEuribor3m: 'economic:fred:v1:EURIBOR3M:0',
ecbEuribor6m: 'economic:fred:v1:EURIBOR6M:0',
ecbEuribor1y: 'economic:fred:v1:EURIBOR1Y:0',
fearGreedIndex: 'market:fear-greed:v1',
breadthHistory: 'market:breadth-history:v1',
euYieldCurve: 'economic:yield-curve-eu:v1',
earningsCalendar: 'market:earnings-calendar:v1',
econCalendar: 'economic:econ-calendar:v1',
cotPositioning: 'market:cot:v1',
crudeInventories: 'economic:crude-inventories:v1',
natGasStorage: 'economic:nat-gas-storage:v1',
spr: 'economic:spr:v1',
refineryInputs: 'economic:refinery-inputs:v1',
ecbFxRates: 'economic:ecb-fx-rates:v1',
eurostatCountryData: 'economic:eurostat-country-data:v1',
eurostatHousePrices: 'economic:eurostat:house-prices:v1',
eurostatGovDebtQ: 'economic:eurostat:gov-debt-q:v1',
eurostatIndProd: 'economic:eurostat:industrial-production:v1',
euGasStorage: 'economic:eu-gas-storage:v1',
euFsi: 'economic:fsi-eu:v1',
shippingStress: 'supply_chain:shipping_stress:v1',
diseaseOutbreaks: 'health:disease-outbreaks:v1',
healthAirQuality: 'health:air-quality:v1',
socialVelocity: 'intelligence:social:reddit:v1',
wsbTickers: 'intelligence:wsb-tickers:v1',
vpdTrackerRealtime: 'health:vpd-tracker:realtime:v1',
vpdTrackerHistorical: 'health:vpd-tracker:historical:v1',
electricityPrices: 'energy:electricity:v1:index',
gasStorageCountries: 'energy:gas-storage:v1:_countries',
aaiiSentiment: 'market:aaii-sentiment:v1',
cryptoSectors: 'market:crypto-sectors:v1',
ddosAttacks: 'cf:radar:ddos:v1',
economicStress: 'economic:stress-index:v1',
trafficAnomalies: 'cf:radar:traffic-anomalies:v1',
};
const STANDALONE_KEYS = {
serviceStatuses: 'infra:service-statuses:v1',
macroSignals: 'economic:macro-signals:v1',
bisPolicy: 'economic:bis:policy:v1',
bisExchange: 'economic:bis:eer:v1',
fxYoy: 'economic:fx:yoy:v1',
bisCredit: 'economic:bis:credit:v1',
bisDsr: 'economic:bis:dsr:v1',
bisPropertyResidential: 'economic:bis:property-residential:v1',
bisPropertyCommercial: 'economic:bis:property-commercial:v1',
imfMacro: 'economic:imf:macro:v2',
imfGrowth: 'economic:imf:growth:v1',
imfLabor: 'economic:imf:labor:v1',
imfExternal: 'economic:imf:external:v1',
climateZoneNormals: 'climate:zone-normals:v1',
shippingRates: 'supply_chain:shipping:v2',
chokepoints: 'supply_chain:chokepoints:v4',
minerals: 'supply_chain:minerals:v2',
giving: 'giving:summary:v1',
gpsjam: 'intelligence:gpsjam:v2',
theaterPosture: 'theater_posture:sebuf:stale:v1',
theaterPostureLive: 'theater-posture:sebuf:v1',
theaterPostureBackup: 'theater-posture:sebuf:backup:v1',
riskScoresLive: 'risk:scores:sebuf:v1',
usniFleet: 'usni-fleet:sebuf:v1',
usniFleetStale: 'usni-fleet:sebuf:stale:v1',
faaDelays: 'aviation:delays:faa:v1',
intlDelays: 'aviation:delays:intl:v3',
notamClosures: 'aviation:notam:closures:v2',
positiveEventsLive: 'positive-events:geo:v1',
cableHealth: 'cable-health-v1',
cyberThreatsRpc: 'cyber:threats:v2',
militaryBases: 'military:bases:active',
militaryFlights: 'military:flights:v1',
militaryFlightsStale: 'military:flights:stale:v1',
temporalAnomalies: 'temporal:anomalies:v1',
displacement: `displacement:summary:v1:${new Date().getUTCFullYear()}`,
displacementPrev: `displacement:summary:v1:${new Date().getUTCFullYear() - 1}`,
satellites: 'intelligence:satellites:tle:v1',
portwatch: 'supply_chain:portwatch:v1',
portwatchPortActivity: 'supply_chain:portwatch-ports:v1:_countries',
corridorrisk: 'supply_chain:corridorrisk:v1',
chokepointTransits: 'supply_chain:chokepoint_transits:v1',
transitSummaries: 'supply_chain:transit-summaries:v1',
thermalEscalation: 'thermal:escalation:v1',
tariffTrendsUs: 'trade:tariffs:v1:840:all:10',
militaryForecastInputs: 'military:forecast-inputs:stale:v1',
gscpi: 'economic:fred:v1:GSCPI:0',
marketImplications: 'intelligence:market-implications:v1',
hormuzTracker: 'supply_chain:hormuz_tracker:v1',
simulationPackageLatest: 'forecast:simulation-package:latest',
simulationOutcomeLatest: 'forecast:simulation-outcome:latest',
newsThreatSummary: 'news:threat:summary:v1',
climateNews: 'climate:news-intelligence:v1',
pizzint: 'intelligence:pizzint:seed:v1',
resilienceStaticIndex: 'resilience:static:index:v1',
resilienceStaticFao: 'resilience:static:fao',
resilienceRanking: 'resilience:ranking:v9',
productCatalog: 'product-catalog:v2',
energySpineCountries: 'energy:spine:v1:_countries',
energyExposure: 'energy:exposure:v1:index',
energyMixAll: 'energy:mix:v1:_all',
regulatoryActions: 'regulatory:actions:v1',
energyIntelligence: 'energy:intelligence:feed:v1',
ieaOilStocks: 'energy:iea-oil-stocks:v1:index',
oilStocksAnalysis: 'energy:oil-stocks-analysis:v1',
jodiGas: 'energy:jodi-gas:v1:_countries',
lngVulnerability: 'energy:lng-vulnerability:v1',
jodiOil: 'energy:jodi-oil:v1:_countries',
chokepointBaselines: 'energy:chokepoint-baselines:v1',
portwatchChokepointsRef: 'portwatch:chokepoints:ref:v1',
chokepointFlows: 'energy:chokepoint-flows:v1',
emberElectricity: 'energy:ember:v1:_all',
resilienceIntervals: 'resilience:intervals:v1:US',
sprPolicies: 'energy:spr-policies:v1',
energyCrisisPolicies: 'energy:crisis-policies:v1',
regionalSnapshots: 'intelligence:regional-snapshots:summary:v1',
regionalBriefs: 'intelligence:regional-briefs:summary:v1',
recoveryFiscalSpace: 'resilience:recovery:fiscal-space:v1',
recoveryReserveAdequacy: 'resilience:recovery:reserve-adequacy:v1',
recoveryExternalDebt: 'resilience:recovery:external-debt:v1',
recoveryImportHhi: 'resilience:recovery:import-hhi:v1',
recoveryFuelStocks: 'resilience:recovery:fuel-stocks:v1',
goldExtended: 'market:gold-extended:v1',
goldEtfFlows: 'market:gold-etf-flows:v1',
goldCbReserves: 'market:gold-cb-reserves:v1',
};
const SEED_META = {
earthquakes: { key: 'seed-meta:seismology:earthquakes', maxStaleMin: 30 },
wildfires: { key: 'seed-meta:wildfire:fires', maxStaleMin: 360 }, // FIRMS NRT resets at midnight UTC; new-day data takes 3-6h to accumulate
outages: { key: 'seed-meta:infra:outages', maxStaleMin: 30 },
climateAnomalies: { key: 'seed-meta:climate:anomalies', maxStaleMin: 240 }, // runs as independent Railway cron (0 */2 * * *); 240 = 2x interval
climateDisasters: { key: 'seed-meta:climate:disasters', maxStaleMin: 720 }, // runs every 6h; 720min = 2x interval
climateAirQuality:{ key: 'seed-meta:health:air-quality', maxStaleMin: 180 }, // hourly cron; 180 = 3x interval — shares meta key with healthAirQuality (same seeder run)
climateZoneNormals: { key: 'seed-meta:climate:zone-normals', maxStaleMin: 89280 }, // monthly cron on the 1st; 62d = 2x 31-day cadence
co2Monitoring: { key: 'seed-meta:climate:co2-monitoring', maxStaleMin: 4320 }, // daily cron at 06:00 UTC; 72h tolerates two missed runs
oceanIce: { key: 'seed-meta:climate:ocean-ice', maxStaleMin: 2880 }, // daily cron at 08:00 UTC; 48h = 2× interval, tolerates one missed run
climateNews: { key: 'seed-meta:climate:news-intelligence', maxStaleMin: 90 }, // relay loop every 30min; 90 = 3× interval
unrestEvents: { key: 'seed-meta:unrest:events', maxStaleMin: 120 }, // 45min cron; 120 = 2h grace (was 75 = 30min buffer, too tight)
cyberThreats: { key: 'seed-meta:cyber:threats', maxStaleMin: 240 }, // 2h interval; 240min = 2x interval
cryptoQuotes: { key: 'seed-meta:market:crypto', maxStaleMin: 30 },
etfFlows: { key: 'seed-meta:market:etf-flows', maxStaleMin: 60 },
gulfQuotes: { key: 'seed-meta:market:gulf-quotes', maxStaleMin: 30 },
stablecoinMarkets:{ key: 'seed-meta:market:stablecoins', maxStaleMin: 60 },
naturalEvents: { key: 'seed-meta:natural:events', maxStaleMin: 360 }, // 2h cron; 3x interval; was 120 (TTL was 60min — panel went dark before health alarmed)
flightDelays: { key: 'seed-meta:aviation:faa', maxStaleMin: 90 }, // CACHE_TTL=7200s; matches notamClosures from same cron
notamClosures: { key: 'seed-meta:aviation:notam', maxStaleMin: 240 }, // 2h interval; 240min = 2x interval
predictionMarkets: { key: 'seed-meta:prediction:markets', maxStaleMin: 90 },
newsInsights: { key: 'seed-meta:news:insights', maxStaleMin: 30 },
marketQuotes: { key: 'seed-meta:market:stocks', maxStaleMin: 30 },
commodityQuotes: { key: 'seed-meta:market:commodities', maxStaleMin: 30 },
goldExtended: { key: 'seed-meta:market:gold-extended', maxStaleMin: 30 },
goldEtfFlows: { key: 'seed-meta:market:gold-etf-flows', maxStaleMin: 2880 }, // SPDR publishes daily; 2× = 48h tolerance
goldCbReserves: { key: 'seed-meta:market:gold-cb-reserves', maxStaleMin: 44640 }, // IMF IFS is monthly w/ ~2-3mo lag; 31d tolerance
// RPC/warm-ping keys — seed-meta written by relay loops or handlers
// serviceStatuses: moved to ON_DEMAND — RPC-populated, no dedicated seed, goes stale when no users visit
cableHealth: { key: 'seed-meta:cable-health', maxStaleMin: 90 }, // ais-relay warm-ping runs every 30min; 90min = 3× interval catches missed pings without false positives
macroSignals: { key: 'seed-meta:economic:macro-signals', maxStaleMin: 20 },
bisPolicy: { key: 'seed-meta:economic:bis', maxStaleMin: 10080 }, // runSeed('economic','bis',...) writes seed-meta:economic:bis
// seed-bis-extended.mjs writes per-dataset seed-meta keys ONLY when that
// specific dataset published fresh entries — so a single-dataset BIS outage
// (e.g. WS_DSR 500s) goes stale in health without falsely dragging down the
// healthy ones. 24h = 2× 12h cron.
bisDsr: { key: 'seed-meta:economic:bis-dsr', maxStaleMin: 1440 },
bisPropertyResidential:{ key: 'seed-meta:economic:bis-property-residential', maxStaleMin: 1440 },
bisPropertyCommercial: { key: 'seed-meta:economic:bis-property-commercial', maxStaleMin: 1440 },
imfMacro: { key: 'seed-meta:economic:imf-macro', maxStaleMin: 100800 }, // monthly seed; 100800min = 70 days = 2× interval (absorbs one missed run)
imfGrowth: { key: 'seed-meta:economic:imf-growth', maxStaleMin: 100800 }, // monthly seed; 70d threshold matches imfMacro (same WEO release cadence)
imfLabor: { key: 'seed-meta:economic:imf-labor', maxStaleMin: 100800 }, // monthly seed; 70d threshold matches imfMacro
imfExternal: { key: 'seed-meta:economic:imf-external', maxStaleMin: 100800 }, // monthly seed; 70d threshold matches imfMacro
shippingRates: { key: 'seed-meta:supply_chain:shipping', maxStaleMin: 420 },
chokepoints: { key: 'seed-meta:supply_chain:chokepoints', maxStaleMin: 60 },
// minerals + giving: on-demand cachedFetchJson only, no seed-meta writer — freshness checked via TTL
// bisExchange + bisCredit: extras written by same BIS script via writeExtraKey, no dedicated seed-meta
fxYoy: { key: 'seed-meta:economic:fx-yoy', maxStaleMin: 1500 }, // daily cron; 25h tolerance + 1h drift
gpsjam: { key: 'seed-meta:intelligence:gpsjam', maxStaleMin: 720 },
positiveGeoEvents:{ key: 'seed-meta:positive-events:geo', maxStaleMin: 60 },
riskScores: { key: 'seed-meta:intelligence:risk-scores', maxStaleMin: 30 }, // CII warm-ping every 8min; 30min = ~3.5x interval,
iranEvents: { key: 'seed-meta:conflict:iran-events', maxStaleMin: 20160 }, // manual seed from LiveUAMap; 20160 = 14d = 2× weekly cadence
ucdpEvents: { key: 'seed-meta:conflict:ucdp-events', maxStaleMin: 420 },
militaryFlights: { key: 'seed-meta:military:flights', maxStaleMin: 30 }, // cron ~10min (LIVE_TTL=600s); 30min = 3x interval,
satellites: { key: 'seed-meta:intelligence:satellites', maxStaleMin: 240 }, // CelesTrak every 120min; 240min = absorbs one missed cycle
weatherAlerts: { key: 'seed-meta:weather:alerts', maxStaleMin: 45 }, // relay loop every 15min; 45 = 3× interval (was 30 = 2×, too tight on relay hiccup)
spending: { key: 'seed-meta:economic:spending', maxStaleMin: 120 },
techEvents: { key: 'seed-meta:research:tech-events', maxStaleMin: 480 },
gdeltIntel: { key: 'seed-meta:intelligence:gdelt-intel', maxStaleMin: 420 }, // 6h cron + 1h grace; CACHE_TTL is 24h so per-topic merge always has a prior snapshot
forecasts: { key: 'seed-meta:forecast:predictions', maxStaleMin: 90 },
sectors: { key: 'seed-meta:market:sectors', maxStaleMin: 30 },
techReadiness: { key: 'seed-meta:economic:worldbank-techreadiness:v1', maxStaleMin: 10080 },
progressData: { key: 'seed-meta:economic:worldbank-progress:v1', maxStaleMin: 10080 },
renewableEnergy: { key: 'seed-meta:economic:worldbank-renewable:v1', maxStaleMin: 10080 },
intlDelays: { key: 'seed-meta:aviation:intl', maxStaleMin: 90 },
// faaDelays shares seed-meta key with flightDelays — no duplicate entry needed here
theaterPosture: { key: 'seed-meta:theater-posture', maxStaleMin: 60 },
correlationCards: { key: 'seed-meta:correlation:cards', maxStaleMin: 15 },
portwatch: { key: 'seed-meta:supply_chain:portwatch', maxStaleMin: 720 },
portwatchPortActivity: { key: 'seed-meta:supply_chain:portwatch-ports', maxStaleMin: 2160 }, // 12h cron; 2160min = 36h = 3x interval
corridorrisk: { key: 'seed-meta:supply_chain:corridorrisk', maxStaleMin: 120 },
chokepointTransits: { key: 'seed-meta:supply_chain:chokepoint_transits', maxStaleMin: 30 }, // relay every 10min; 30min = 3x interval,
transitSummaries: { key: 'seed-meta:supply_chain:transit-summaries', maxStaleMin: 30 }, // relay every 10min; 30min = 3x interval,
usniFleet: { key: 'seed-meta:military:usni-fleet', maxStaleMin: 720 }, // relay loop every 6h; 720 = 2× interval (was 480 = 1.3×, too tight)
securityAdvisories: { key: 'seed-meta:intelligence:advisories', maxStaleMin: 120 },
customsRevenue: { key: 'seed-meta:trade:customs-revenue', maxStaleMin: 1440 },
comtradeFlows: { key: 'seed-meta:trade:comtrade-flows', maxStaleMin: 2880 }, // 24h cron; 2880min = 48h = 2x interval
blsSeries: { key: 'seed-meta:economic:bls-series', maxStaleMin: 2880 }, // daily seed; 2880min = 48h = 2x interval
sanctionsPressure: { key: 'seed-meta:sanctions:pressure', maxStaleMin: 720 },
crossSourceSignals: { key: 'seed-meta:intelligence:cross-source-signals', maxStaleMin: 30 }, // 15min cron; 30min = 2x interval
regionalSnapshots: { key: 'seed-meta:intelligence:regional-snapshots', maxStaleMin: 720 }, // 6h cron via seed-bundle-derived-signals; 720min = 12h = 2x interval
regionalBriefs: { key: 'seed-meta:intelligence:regional-briefs', maxStaleMin: 20160 }, // weekly cron; 20160min = 14 days = 2x interval
sanctionsEntities: { key: 'seed-meta:sanctions:entities', maxStaleMin: 1440 }, // 12h cron; 1440min = 24h = 2x interval
radiationWatch: { key: 'seed-meta:radiation:observations', maxStaleMin: 30 },
groceryBasket: { key: 'seed-meta:economic:grocery-basket', maxStaleMin: 10080 }, // weekly seed; 10080 = 7 days
bigmac: { key: 'seed-meta:economic:bigmac', maxStaleMin: 10080 }, // weekly seed; 10080 = 7 days
fuelPrices: { key: 'seed-meta:economic:fuel-prices', maxStaleMin: 10080 }, // weekly seed; 10080 = 7 days
faoFoodPriceIndex: { key: 'seed-meta:economic:fao-ffpi', maxStaleMin: 86400 }, // monthly seed; 86400 = 60 days (2x interval)
thermalEscalation: { key: 'seed-meta:thermal:escalation', maxStaleMin: 360 }, // cron every 2h; 360 = 3x interval (was 240 = 2x)
nationalDebt: { key: 'seed-meta:economic:national-debt', maxStaleMin: 10080 }, // 7 days — monthly seed
tariffTrendsUs: { key: 'seed-meta:trade:tariffs:v1:840:all:10', maxStaleMin: 900 },
// publish.ts runs once daily (02:30 UTC); seed-meta TTL=52h — maxStaleMin must cover the full 24h cycle
consumerPricesOverview: { key: 'seed-meta:consumer-prices:overview:ae', maxStaleMin: 1500 }, // 25h = 24h cadence + 1h grace
consumerPricesCategories: { key: 'seed-meta:consumer-prices:categories:ae:30d', maxStaleMin: 1500 },
consumerPricesMovers: { key: 'seed-meta:consumer-prices:movers:ae:30d', maxStaleMin: 1500 },
consumerPricesSpread: { key: 'seed-meta:consumer-prices:retailer-spread:ae:essentials-ae', maxStaleMin: 1500 },
consumerPricesFreshness: { key: 'seed-meta:consumer-prices:freshness:ae', maxStaleMin: 1500 },
// defiTokens/aiTokens/otherTokens all share one seed run (seed-token-panels cron, every 30min)
defiTokens: { key: 'seed-meta:market:token-panels', maxStaleMin: 90 },
aiTokens: { key: 'seed-meta:market:token-panels', maxStaleMin: 90 },
otherTokens: { key: 'seed-meta:market:token-panels', maxStaleMin: 90 },
fredBatch: { key: 'seed-meta:economic:fred:v1:FEDFUNDS:0', maxStaleMin: 1500 }, // daily cron
ecbEstr: { key: 'seed-meta:economic:ecb-short-rates', maxStaleMin: 4320 }, // daily ECB publish; 4320min = 3d = TTL/interval
ecbEuribor3m: { key: 'seed-meta:economic:ecb-short-rates', maxStaleMin: 4320 }, // shared meta key with ecbEstr
ecbEuribor6m: { key: 'seed-meta:economic:ecb-short-rates', maxStaleMin: 4320 }, // shared meta key with ecbEstr
ecbEuribor1y: { key: 'seed-meta:economic:ecb-short-rates', maxStaleMin: 4320 }, // shared meta key with ecbEstr
gscpi: { key: 'seed-meta:economic:gscpi', maxStaleMin: 2880 }, // 24h interval; 2880min = 48h = 2x interval
fearGreedIndex: { key: 'seed-meta:market:fear-greed', maxStaleMin: 720 }, // 6h cron; 720min = 12h = 2x interval
breadthHistory: { key: 'seed-meta:market:breadth-history', maxStaleMin: 2880 }, // daily cron at 21:00 ET; 2880min = 48h = 2x interval
hormuzTracker: { key: 'seed-meta:supply_chain:hormuz_tracker', maxStaleMin: 2880 }, // daily cron; 2880min = 48h = 2x interval
earningsCalendar: { key: 'seed-meta:market:earnings-calendar', maxStaleMin: 1440 }, // 12h cron; 1440min = 24h = 2x interval
econCalendar: { key: 'seed-meta:economic:econ-calendar', maxStaleMin: 1440 }, // 12h cron; 1440min = 24h = 2x interval
cotPositioning: { key: 'seed-meta:market:cot', maxStaleMin: 14400 }, // weekly CFTC release; 14400min = 10d = 1.4x interval (weekend + delay buffer)
crudeInventories: { key: 'seed-meta:economic:crude-inventories', maxStaleMin: 20160 }, // weekly EIA data; 20160min = 14 days = 2x weekly cadence
natGasStorage: { key: 'seed-meta:economic:nat-gas-storage', maxStaleMin: 20160 }, // weekly EIA data; 20160min = 14 days = 2x weekly cadence
spr: { key: 'seed-meta:economic:spr', maxStaleMin: 20160 }, // weekly EIA data; 20160min = 14 days = 2x weekly cadence
refineryInputs: { key: 'seed-meta:economic:refinery-inputs', maxStaleMin: 20160 }, // weekly EIA data; 20160min = 14 days = 2x weekly cadence
ecbFxRates: { key: 'seed-meta:economic:ecb-fx-rates', maxStaleMin: 5760 }, // daily seed (weekdays + holidays); 5760min = 96h = covers Wed→Mon Easter gap
eurostatCountryData: { key: 'seed-meta:economic:eurostat-country-data', maxStaleMin: 4320 }, // daily seed; 4320min = 3 days = 3x interval
eurostatHousePrices: { key: 'seed-meta:economic:eurostat-house-prices', maxStaleMin: 60 * 24 * 50 }, // weekly cron, annual data; 50d threshold = 35d TTL + 15d buffer
eurostatGovDebtQ: { key: 'seed-meta:economic:eurostat-gov-debt-q', maxStaleMin: 60 * 24 * 14 }, // 2d cron, quarterly data; 14d threshold matches TTL + quarterly release drift
eurostatIndProd: { key: 'seed-meta:economic:eurostat-industrial-production', maxStaleMin: 60 * 24 * 5 }, // daily cron, monthly data; 5d threshold matches TTL
euGasStorage: { key: 'seed-meta:economic:eu-gas-storage', maxStaleMin: 2880 }, // daily seed (T+1); 2880min = 48h = 2x interval
euYieldCurve: { key: 'seed-meta:economic:yield-curve-eu', maxStaleMin: 4320 }, // daily seed (weekdays only); 4320min = 72h = covers Fri→Mon gap
euFsi: { key: 'seed-meta:economic:fsi-eu', maxStaleMin: 20160 }, // weekly seed (Saturday); 20160min = 14d = 2x interval
newsThreatSummary: { key: 'seed-meta:news:threat-summary', maxStaleMin: 60 }, // relay classify every ~20min; 60min = 3x interval
shippingStress: { key: 'seed-meta:supply_chain:shipping_stress', maxStaleMin: 45 }, // relay loop every 15min; 45 = 3x interval (was 30 = 2×, too tight on relay hiccup)
diseaseOutbreaks: { key: 'seed-meta:health:disease-outbreaks', maxStaleMin: 2880 }, // daily seed; 2880 = 48h = 2x interval
healthAirQuality: { key: 'seed-meta:health:air-quality', maxStaleMin: 180 }, // hourly cron; 180 = 3x interval for shared health/climate seed
socialVelocity: { key: 'seed-meta:intelligence:social-reddit', maxStaleMin: 30 }, // relay loop every 10min; 30 = 3x interval (was 20 = equals retry window, too tight)
wsbTickers: { key: 'seed-meta:intelligence:wsb-tickers', maxStaleMin: 30 }, // relay loop every 10min; 30 = 3x interval
pizzint: { key: 'seed-meta:intelligence:pizzint', maxStaleMin: 30 }, // relay loop every 10min; 30 = 3x interval
productCatalog: { key: 'seed-meta:product-catalog', maxStaleMin: 1080 }, // relay loop every 6h; 1080 = 18h = 3x interval
vpdTrackerRealtime: { key: 'seed-meta:health:vpd-tracker', maxStaleMin: 2880 }, // daily seed (0 2 * * *); 2880min = 48h = 2x interval
vpdTrackerHistorical: { key: 'seed-meta:health:vpd-tracker', maxStaleMin: 2880 }, // shares seed-meta key with vpdTrackerRealtime (same run)
resilienceStaticIndex: { key: 'seed-meta:resilience:static', maxStaleMin: 576000 }, // annual October snapshot; 400d threshold matches TTL and preserves prior-year data on source outages
resilienceStaticFao: { key: 'seed-meta:resilience:static', maxStaleMin: 576000 }, // same seeder + same heartbeat as resilienceStaticIndex; required so EMPTY_DATA_OK + missing data degrades to STALE_SEED instead of silent OK
resilienceRanking: { key: 'seed-meta:resilience:ranking', maxStaleMin: 720 }, // on-demand RPC cache (6h TTL); 12h threshold catches stale rankings without paging on cold start
resilienceIntervals: { key: 'seed-meta:resilience:intervals', maxStaleMin: 20160 }, // weekly cron; 20160min = 14d = 2x interval
energyExposure: { key: 'seed-meta:economic:owid-energy-mix', maxStaleMin: 50400 }, // monthly cron on 1st; 50400min = 35d = TTL matches cron cadence + 5d buffer
energyMixAll: { key: 'seed-meta:economic:owid-energy-mix', maxStaleMin: 50400 }, // same seed run as energyExposure; shares seed-meta key
regulatoryActions: { key: 'seed-meta:regulatory:actions', maxStaleMin: 360 }, // 2h cron; 360min = 3x interval
energySpineCountries: { key: 'seed-meta:energy:spine', maxStaleMin: 2880 }, // daily cron (06:00 UTC); 2880min = 48h = 2x interval
electricityPrices: { key: 'seed-meta:energy:electricity-prices', maxStaleMin: 2880 }, // daily cron (14:00 UTC); 2880min = 48h = 2x interval
gasStorageCountries: { key: 'seed-meta:energy:gas-storage-countries', maxStaleMin: 2880 }, // daily cron at 10:30 UTC; 2880min = 48h = 2x interval
energyIntelligence: { key: 'seed-meta:energy:intelligence', maxStaleMin: 720 }, // 6h cron; 720min = 2x interval
jodiOil: { key: 'seed-meta:energy:jodi-oil', maxStaleMin: 60 * 24 * 40 }, // monthly cron on 25th; 40d threshold matches 35d TTL + 5d buffer
ieaOilStocks: { key: 'seed-meta:energy:iea-oil-stocks', maxStaleMin: 60 * 24 * 40 }, // monthly cron on 15th; 40d threshold = TTL_SECONDS
oilStocksAnalysis: { key: 'seed-meta:energy:oil-stocks-analysis', maxStaleMin: 60 * 24 * 50 }, // afterPublish of ieaOilStocks; 50d = matches seed-meta TTL (exceeds 40d data TTL)
jodiGas: { key: 'seed-meta:energy:jodi-gas', maxStaleMin: 60 * 24 * 40 }, // monthly cron on 25th; 40d threshold matches 35d TTL + 5d buffer
lngVulnerability: { key: 'seed-meta:energy:jodi-gas', maxStaleMin: 60 * 24 * 40 }, // written by jodi-gas seeder afterPublish; shares seed-meta key
chokepointBaselines: { key: 'seed-meta:energy:chokepoint-baselines', maxStaleMin: 60 * 24 * 400 }, // 400 days
sprPolicies: { key: 'seed-meta:energy:spr-policies', maxStaleMin: 60 * 24 * 400 }, // 400 days; static registry, same cadence as chokepoint baselines
energyCrisisPolicies: { key: 'seed-meta:energy:crisis-policies', maxStaleMin: 60 * 24 * 400 }, // static data, ~400d TTL matches seeder
aaiiSentiment: { key: 'seed-meta:market:aaii-sentiment', maxStaleMin: 20160 }, // weekly cron; 20160min = 14 days = 2x weekly cadence
portwatchChokepointsRef: { key: 'seed-meta:portwatch:chokepoints-ref', maxStaleMin: 60 * 24 * 14 }, // seed-bundle-portwatch runs this at WEEK cadence; 14d = 2× interval
chokepointFlows: { key: 'seed-meta:energy:chokepoint-flows', maxStaleMin: 720 }, // 6h cron; 720min = 2x interval
emberElectricity: { key: 'seed-meta:energy:ember', maxStaleMin: 2880 }, // daily cron (08:00 UTC); 2880min = 48h = 2x interval
cryptoSectors: { key: 'seed-meta:market:crypto-sectors', maxStaleMin: 120 }, // relay loop every ~30min; 120min = 2h = 4x interval
ddosAttacks: { key: 'seed-meta:cf:radar:ddos', maxStaleMin: 60 }, // written by seed-internet-outages afterPublish; outages cron ~15min; 60 = 4x interval
economicStress: { key: 'seed-meta:economic:stress-index', maxStaleMin: 180 }, // computed in seed-economy afterPublish; cron ~1h; 180min = 3x interval
marketImplications: { key: 'seed-meta:intelligence:market-implications', maxStaleMin: 120 }, // LLM-generated in seed-forecasts; cron ~1h; 120min = 2x interval
trafficAnomalies: { key: 'seed-meta:cf:radar:traffic-anomalies', maxStaleMin: 60 }, // written by seed-internet-outages afterPublish; outages cron ~15min; 60 = 4x interval
chokepointExposure: { key: 'seed-meta:supply_chain:chokepoint-exposure', maxStaleMin: 2880 }, // daily cron; 2880min = 48h = 2x interval
recoveryFiscalSpace: { key: 'seed-meta:resilience:recovery:fiscal-space', maxStaleMin: 86400 }, // monthly cron; 86400min = 60d = 2x interval
recoveryReserveAdequacy: { key: 'seed-meta:resilience:recovery:reserve-adequacy', maxStaleMin: 86400 }, // monthly cron; 86400min = 60d = 2x interval
recoveryExternalDebt: { key: 'seed-meta:resilience:recovery:external-debt', maxStaleMin: 86400 }, // monthly cron; 86400min = 60d = 2x interval
recoveryImportHhi: { key: 'seed-meta:resilience:recovery:import-hhi', maxStaleMin: 86400 }, // monthly cron; 86400min = 60d = 2x interval
recoveryFuelStocks: { key: 'seed-meta:resilience:recovery:fuel-stocks', maxStaleMin: 86400 }, // monthly cron; 86400min = 60d = 2x interval
};
// Standalone keys that are populated on-demand by RPC handlers (not seeds).
// Empty = WARN not CRIT since they only exist after first request.
const ON_DEMAND_KEYS = new Set([
'riskScoresLive',
'usniFleetStale', 'positiveEventsLive',
'bisPolicy', 'bisExchange', 'bisCredit',
// bisDsr/bisPropertyResidential/bisPropertyCommercial have dedicated SEED_META
// entries (seed-bis-extended.mjs), so they are not on-demand.
'macroSignals', 'shippingRates', 'chokepoints', 'minerals', 'giving',
'cyberThreatsRpc', 'militaryBases', 'temporalAnomalies', 'displacement',
'corridorrisk', // intermediate key; data flows through transit-summaries:v1
'serviceStatuses', // RPC-populated; seed-meta written on fresh fetch only, goes stale between visits
'militaryForecastInputs', // intermediate seed-to-seed pipeline key; only populated after seed-military-flights runs
'marketImplications', // LLM-generated inside forecast cron; can fail silently on LLM errors — degrade to WARN not CRIT
'simulationPackageLatest', // written by writeSimulationPackage after deep forecast runs; only present after first successful deep run
'simulationOutcomeLatest', // written by writeSimulationOutcome after simulation runs; only present after first successful simulation
'newsThreatSummary', // relay classify loop — only written when mergedByCountry has entries; absent on quiet news periods
'resilienceRanking', // on-demand RPC cache populated after ranking requests; missing before first Pro use is expected
'recoveryFiscalSpace', 'recoveryReserveAdequacy', 'recoveryExternalDebt',
'recoveryImportHhi', 'recoveryFuelStocks', // recovery pillar: stub seeders not yet deployed, keys may be absent
'displacementPrev', // covered by cascade onto current-year displacement; empty most of the year
'fxYoy', // TRANSITIONAL (PR #3071): seed-fx-yoy Railway cron deployed manually after merge —
// gate as on-demand so a deploy-order race or first-cron-run failure doesn't
// fire a CRIT health alarm. Remove from this set after ~7 days of clean
// production cron runs (verify via `seed-meta:economic:fx-yoy.fetchedAt`).
]);
// Keys where 0 records is a valid healthy state (e.g. no airports closed,
// no earnings events this week, econ calendar quiet between seasons).
// The key must still exist in Redis; only the record count can be 0.
const EMPTY_DATA_OK_KEYS = new Set([
'notamClosures', 'faaDelays', 'gpsjam', 'positiveGeoEvents', 'weatherAlerts',
'earningsCalendar', 'econCalendar', 'cotPositioning',
'usniFleet', // usniFleetStale covers the fallback; relay outages → WARN not CRIT
'newsThreatSummary', // only written when classify produces country matches; quiet news periods = 0 countries, no write
'recoveryFiscalSpace',
'recoveryImportHhi', 'recoveryFuelStocks', // recovery pillar seeds: stub seeders write empty payloads until real sources are wired
'ddosAttacks', 'trafficAnomalies', // zero events during quiet periods is valid, not critical
'resilienceStaticFao', // empty aggregate = no IPC Phase 3+ countries this year (possible in theory); the key must exist but count=0 is fine
]);
// Cascade groups: if any key in the group has data, all empty siblings are OK.
// Theater posture uses live → stale → backup fallback chain.
const CASCADE_GROUPS = {
theaterPosture: ['theaterPosture', 'theaterPostureLive', 'theaterPostureBackup'],
theaterPostureLive: ['theaterPosture', 'theaterPostureLive', 'theaterPostureBackup'],
theaterPostureBackup: ['theaterPosture', 'theaterPostureLive', 'theaterPostureBackup'],
militaryFlights: ['militaryFlights', 'militaryFlightsStale'],
militaryFlightsStale: ['militaryFlights', 'militaryFlightsStale'],
// Displacement key embeds UTC year — on Jan 1 the new-year key may be empty
// for hours until the seed runs. Cascade onto the previous-year snapshot.
displacement: ['displacement', 'displacementPrev'],
displacementPrev: ['displacement', 'displacementPrev'],
};
const NEG_SENTINEL = '__WM_NEG__';
function parseRedisValue(raw) {
if (!raw || raw === NEG_SENTINEL) return null;
try { return JSON.parse(raw); } catch { return raw; }
}
// Real data is always >0 bytes. The negative-cache sentinel is exactly
// NEG_SENTINEL.length bytes (10), so any strlen > 0 that is NOT exactly that
// length counts as data. The previous `> 10` heuristic misclassified
// legitimately small payloads (`{}`, `[]`, `0`) as missing.
function strlenIsData(strlen) {
return strlen > 0 && strlen !== NEG_SENTINEL.length;
}
function readSeedMeta(seedCfg, keyMetaValues, keyMetaErrors, now) {
if (!seedCfg) {
return { seedAge: null, seedStale: null, seedError: false, metaReadFailed: false, metaCount: null };
}
// Per-command Redis errors on the GET seed-meta half of the pipeline must
// not silently fall through to STALE_SEED — promote to REDIS_PARTIAL.
if (keyMetaErrors.get(seedCfg.key)) {
return { seedAge: null, seedStale: null, seedError: false, metaReadFailed: true, metaCount: null };
}
const meta = parseRedisValue(keyMetaValues.get(seedCfg.key));
if (meta?.status === 'error') {
return { seedAge: null, seedStale: true, seedError: true, metaReadFailed: false, metaCount: null };
}
let seedAge = null;
let seedStale = true;
if (meta?.fetchedAt) {
seedAge = Math.round((now - meta.fetchedAt) / 60_000);
seedStale = seedAge > seedCfg.maxStaleMin;
}
const metaCount = meta?.count ?? meta?.recordCount ?? null;
return { seedAge, seedStale, seedError: false, metaReadFailed: false, metaCount };
}
function isCascadeCovered(name, hasData, keyStrens, keyErrors) {
const siblings = CASCADE_GROUPS[name];
if (!siblings || hasData) return false;
for (const sibling of siblings) {
if (sibling === name) continue;
const sibKey = STANDALONE_KEYS[sibling] ?? BOOTSTRAP_KEYS[sibling];
if (!sibKey) continue;
if (keyErrors.get(sibKey)) continue;
if (strlenIsData(keyStrens.get(sibKey) ?? 0)) return true;
}
return false;
}
function classifyKey(name, redisKey, opts, ctx) {
const { keyStrens, keyErrors, keyMetaValues, keyMetaErrors, now } = ctx;
const seedCfg = SEED_META[name];
const isOnDemand = !!opts.allowOnDemand && ON_DEMAND_KEYS.has(name);
const meta = readSeedMeta(seedCfg, keyMetaValues, keyMetaErrors, now);
// Per-command Redis errors (data STRLEN or seed-meta GET) propagate as their
// own bucket — don't conflate with "key missing", since ops needs to know if
// the read itself failed.
if (keyErrors.get(redisKey) || meta.metaReadFailed) {
const entry = { status: 'REDIS_PARTIAL', records: null };
if (seedCfg) entry.maxStaleMin = seedCfg.maxStaleMin;
return entry;
}
const strlen = keyStrens.get(redisKey) ?? 0;
const hasData = strlenIsData(strlen);
const { seedAge, seedStale, seedError, metaCount } = meta;
// When the data key is gone the meta count is meaningless; force records=0
// so we never display the contradictory "EMPTY records=N>0" pair (item 1).
const records = hasData ? (metaCount ?? 1) : 0;
const cascadeCovered = isCascadeCovered(name, hasData, keyStrens, keyErrors);
let status;
if (seedError) status = 'SEED_ERROR';
else if (!hasData) {
if (cascadeCovered) status = 'OK_CASCADE';
else if (EMPTY_DATA_OK_KEYS.has(name)) status = seedStale === true ? 'STALE_SEED' : 'OK';
else if (isOnDemand) status = 'EMPTY_ON_DEMAND';
else status = 'EMPTY';
} else if (records === 0) {
// hasData is true in this branch, so cascade can never apply (isCascadeCovered
// short-circuits when hasData=true). Cascade only shields wholly absent keys.
if (EMPTY_DATA_OK_KEYS.has(name)) status = seedStale === true ? 'STALE_SEED' : 'OK';
else if (isOnDemand) status = 'EMPTY_ON_DEMAND';
else status = 'EMPTY_DATA';
} else if (seedStale === true) status = 'STALE_SEED';
else status = 'OK';
const entry = { status, records };
if (seedAge !== null) entry.seedAgeMin = seedAge;
if (seedCfg) entry.maxStaleMin = seedCfg.maxStaleMin;
return entry;
}
const STATUS_COUNTS = {
OK: 'ok',
OK_CASCADE: 'ok',
STALE_SEED: 'warn',
SEED_ERROR: 'warn',
EMPTY_ON_DEMAND: 'warn',
REDIS_PARTIAL: 'warn',
EMPTY: 'crit',
EMPTY_DATA: 'crit',
};
export default async function handler(req, ctx) {
const headers = {
'Content-Type': 'application/json',
'Cache-Control': 'private, no-store, max-age=0',
'CDN-Cache-Control': 'no-store',
'CF-Cache-Status': 'BYPASS',
'Access-Control-Allow-Origin': '*',
};
if (req.method === 'OPTIONS') {
return new Response(null, { status: 204, headers });
}
const now = Date.now();
const allDataKeys = [
...Object.values(BOOTSTRAP_KEYS),
...Object.values(STANDALONE_KEYS),
];
const allMetaKeys = Object.values(SEED_META).map(s => s.key);
// STRLEN for data keys avoids loading large blobs into memory (OOM prevention).
// NEG_SENTINEL ('__WM_NEG__') is 10 bytes — strlenIsData() rejects exactly
// that length while accepting any other non-zero strlen as data.
let results;
try {
const commands = [
...allDataKeys.map(k => ['STRLEN', k]),
...allMetaKeys.map(k => ['GET', k]),
];
if (!getRedisCredentials()) throw new Error('Redis not configured');
results = await redisPipeline(commands, 8_000);
if (!results) throw new Error('Redis request failed');
} catch (err) {
return jsonResponse({
status: 'REDIS_DOWN',
error: err.message,
checkedAt: new Date(now).toISOString(),
}, 200, headers);
}
// keyStrens: byte length per data key (0 = missing/empty/sentinel)
// keyErrors: per-command Redis errors so we can surface REDIS_PARTIAL
const keyStrens = new Map();
const keyErrors = new Map();
for (let i = 0; i < allDataKeys.length; i++) {
const r = results[i];
if (r?.error) keyErrors.set(allDataKeys[i], r.error);
keyStrens.set(allDataKeys[i], r?.result ?? 0);
}
// keyMetaValues: parsed seed-meta objects (GET, small payloads)
// keyMetaErrors: per-command errors so a single GET failure surfaces as
// REDIS_PARTIAL instead of silently degrading to STALE_SEED.
const keyMetaValues = new Map();
const keyMetaErrors = new Map();
for (let i = 0; i < allMetaKeys.length; i++) {
const r = results[allDataKeys.length + i];
if (r?.error) keyMetaErrors.set(allMetaKeys[i], r.error);
keyMetaValues.set(allMetaKeys[i], r?.result ?? null);
}
const classifyCtx = { keyStrens, keyErrors, keyMetaValues, keyMetaErrors, now };
const checks = {};
const counts = { ok: 0, warn: 0, onDemandWarn: 0, crit: 0 };
let totalChecks = 0;
const sources = [
[BOOTSTRAP_KEYS, { allowOnDemand: false }],
[STANDALONE_KEYS, { allowOnDemand: true }],
];
for (const [registry, opts] of sources) {
for (const [name, redisKey] of Object.entries(registry)) {
totalChecks++;
const entry = classifyKey(name, redisKey, opts, classifyCtx);
checks[name] = entry;
const bucket = STATUS_COUNTS[entry.status] ?? 'warn';
counts[bucket]++;
if (entry.status === 'EMPTY_ON_DEMAND') counts.onDemandWarn++;
}
}
// On-demand keys that simply haven't been requested yet should not flip
// overall to WARNING — they're warn-level only for visibility.
const realWarnCount = counts.warn - counts.onDemandWarn;
const critCount = counts.crit;
let overall;
if (critCount === 0 && realWarnCount === 0) overall = 'HEALTHY';
else if (critCount === 0) overall = 'WARNING';
// Degraded threshold scales with registry size so adding keys doesn't
// silently raise the page-out bar. ~3% of total keys (was hardcoded 3).
else if (critCount / totalChecks <= 0.03) overall = 'DEGRADED';
else overall = 'UNHEALTHY';
const httpStatus = 200;
if (overall !== 'HEALTHY' && overall !== 'WARNING') {
const problemKeys = Object.entries(checks)
.filter(([, c]) => c.status === 'EMPTY' || c.status === 'EMPTY_DATA' || c.status === 'STALE_SEED' || c.status === 'SEED_ERROR' || c.status === 'REDIS_PARTIAL')
.map(([k, c]) => `${k}:${c.status}${c.seedAgeMin != null ? `(${c.seedAgeMin}min)` : ''}`);
console.log('[health] %s crits=[%s]', overall, problemKeys.join(', '));
// Persist last failure snapshot for post-mortem. Vercel edge isolates can
// terminate before a fire-and-forget Promise resolves; ctx.waitUntil keeps
// the runtime alive until the write completes.
const persist = redisPipeline([['SET', 'health:last-failure', JSON.stringify({
at: new Date(now).toISOString(),
status: overall,
critCount,
crits: problemKeys,
}), 'EX', 86400]]).catch(() => {});
if (ctx && typeof ctx.waitUntil === 'function') ctx.waitUntil(persist);
}
const url = new URL(req.url);
const compact = url.searchParams.get('compact') === '1';
const body = {
status: overall,
summary: {
total: totalChecks,
ok: counts.ok,
// `warn` excludes on-demand-empty (cosmetic warns); `onDemandWarn` is
// surfaced separately so readers can reconcile against `overall`.
warn: realWarnCount,
onDemandWarn: counts.onDemandWarn,
crit: critCount,
},
checkedAt: new Date(now).toISOString(),
};
if (!compact) {
body.checks = checks;
} else {
const problems = {};
for (const [name, check] of Object.entries(checks)) {
if (check.status !== 'OK' && check.status !== 'OK_CASCADE') problems[name] = check;
}
if (Object.keys(problems).length > 0) body.problems = problems;
}
return new Response(JSON.stringify(body, null, compact ? 0 : 2), {
status: httpStatus,
headers,
});
}
Reference in New Issue View Git Blame Copy Permalink