mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
52659ce192
* docs(resilience): PR 1 foundation — Option B framing + v2 energy construct spec
First commit in PR 1 of the resilience repair plan. Zero scoring-behaviour
change; sets up the construct contract that the code changes will implement.
Declares the framing decision required by plan section 3.2 before any
scorer code lands: Option B (power-system security) is adopted. Electricity
grids are the dominant short-horizon shock-transmission channel, and the
choice lets the v2 energy indicator set share one denominator (percent of
electricity generation) instead of mixing primary-energy and power-system
measures in a composite.
Methodology doc changes:
- Energy Domain section now documents both the legacy indicator set
(still the default) and the v2 indicator set (flag-gated), under a
single #### Energy H4 heading so the methodology-doc linter still
asserts dimension-id parity with the registry.
- v2 indicators: importedFossilDependence (EG.ELC.FOSL.ZS x
max(EG.IMP.CONS.ZS, 0)), lowCarbonGenerationShare (EG.ELC.NUCL.ZS +
EG.ELC.RNEW.ZS), powerLossesPct (EG.ELC.LOSS.ZS), reserveMarginPct
(IEA), euGasStorageStress (renamed + scoped to EU), energyPriceStress
(retained at 0.15 weight).
- Retired under v2: electricityConsumption, gasShare, coalShare,
dependency (all into importedFossilDependence), renewShare.
- electricityAccess moves from energy to infrastructure under v2.
- Added a v2.1 changelog section documenting the flag-gated rollout,
acceptance gates (per plan section 6), and snapshot filenames for
the post-flag-flip captures.
- Known-limitations items 1-3 updated to note PR 1 lands the v2
construct behind RESILIENCE_ENERGY_V2_ENABLED (default off).
Methodology-doc linter + mdx-lint + typecheck all clean. Indicator
registry, seeders, and scorer rewrite land in subsequent commits on
this same branch.
* feat(resilience): PR 1 — RESILIENCE_ENERGY_V2_ENABLED flag + scoreEnergy v2 + registry entries
Second commit in PR 1 of the resilience repair plan. Lands the flag,
the v2 scorer code path, and the registry entries the methodology
doc referenced. Default is flag off; published rankings are unchanged
until the flag flips in a later commit (after seeders land and the
acceptance-gate rerun produces a fresh post-flip snapshot).
Changes:
- _shared.ts: isEnergyV2Enabled() function reader on the canonical
RESILIENCE_ENERGY_V2_ENABLED env var. Dynamic read (like
isPillarCombineEnabled) so tests can flip per-case.
- _dimension-scorers.ts:
- New Redis key constants for the three v2 seed keys plus the
reserved reserveMargin key (seeder deferred per plan §3.1
open-question).
- EU_GAS_STORAGE_COUNTRIES set (EU + EFTA + UK) for the renamed
euGasStorageStress signal per plan §3.5 point 2.
- isEnergyV2EnabledLocal() — private duplicate of the flag reader
to avoid a circular import (_shared.ts already imports from
this module). Same env-var contract.
- scoreEnergy split into scoreEnergyLegacy() + scoreEnergyV2().
Public scoreEnergy() branches on the flag. Legacy path is
byte-identical to the pre-commit behaviour.
- scoreEnergyV2() reads four new bulk payloads, composes
importedFossilDependence = fossilElectricityShare × max(netImports, 0)/100
per plan §3.2, collapses net exporters to 0, and gates
euGasStorageStress on EU membership so non-EU countries
re-normalise rather than getting penalised for a regional
signal.
- _indicator-registry.ts: four new entries under `dimension: 'energy'`
with `tier: 'experimental'` — importedFossilDependence (0.35),
lowCarbonGenerationShare (0.20), powerLossesPct (0.10),
reserveMarginPct (0.10). Experimental tier keeps them out of the
Core coverage gate until seed coverage is confirmed.
- compare-resilience-current-vs-proposed.mjs: new
'bulk-v1-country-value' shape family in the extraction dispatcher.
EXTRACTION_RULES now covers the four v2 registry indicators so
the per-indicator influence harness tracks them from day one.
When the seeders are absent, pairedSampleSize = 0 and Pearson = 0
— the harness output surfaces the "no influence yet" state rather
than silently dropping the indicators.
- tests/resilience-energy-v2.test.mts: 11 new tests pinning:
- flag-off = legacy behaviour preserved (v2 seed keys have no
effect when flag is off — catches accidental cross-path reads)
- flag-on = v2 composite behaves correctly:
- lower fossilElectricityShare raises score
- net exporter with 90% fossil > net importer with 90% fossil
(max(·, 0) collapse verified)
- higher lowCarbonGenerationShare raises score (nuclear credit)
- higher powerLossesPct lowers score
- euGasStorageStress is invariant for non-EU, responds for DE
- all v2 inputs absent = graceful degradation, coverage < 1.0
106 resilience tests pass (existing + 11 new). Typecheck clean. Biome
clean. No production behaviour change with flag off (default).
Next commits on this branch: three World Bank seeders for the v2 keys,
health.js + SEED_META registration (gated ON_DEMAND_KEYS until Railway
cron provisions), acceptance-gate rerun at flag-flip time.
* feat(resilience): PR 1 — three WB seeders + health registration for v2 energy construct
Third commit in PR 1. Lands the seed scripts for the three v2 energy
indicator source keys, registered in api/health.js with ON_DEMAND_KEYS
gating until Railway cron provisions.
New seeders (weekly cron cadence, 8d maxStaleMin = 2x interval):
- scripts/seed-low-carbon-generation.mjs
Pulls EG.ELC.NUCL.ZS + EG.ELC.RNEW.ZS from World Bank, sums per
country into `resilience:low-carbon-generation:v1`. Partial
coverage (one series missing) still emits a value using the
observed half — the scorer's 0-80 saturating goalpost tolerates
it and the underlying construct is "firm low-carbon share".
- scripts/seed-fossil-electricity-share.mjs
Pulls EG.ELC.FOSL.ZS into `resilience:fossil-electricity-share:v1`.
Feeds the importedFossilDependence composite at score time
(composite = fossilShare × max(netImports, 0) / 100 per plan §3.2).
- scripts/seed-power-reliability.mjs
Pulls EG.ELC.LOSS.ZS into `resilience:power-losses:v1`. Direct
grid-integrity signal replacing the retired electricityConsumption
wealth proxy.
All three follow the existing seed-recovery-*.mjs template:
- Shape: { countries: { [ISO2]: { value, year } }, seededAt }
- runSeed() from _seed-utils.mjs with schemaVersion=1, ttl=35d
- validateFn floor of 150 countries (WB coverage is 150-180 for
the three indicators; below 150 = transient fetch failure)
- ISO3 → ISO2 mapping via scripts/shared/iso3-to-iso2.json
No reserveMargin seeder is shipped in this commit per plan §3.1 open
question: IEA electricity-balance coverage is sparse outside OECD+G20,
and the indicator will likely ship as 'unmonitored' with weight 0.05
if it lands at all. The Redis key (`resilience:reserve-margin:v1`) is
reserved in _dimension-scorers.ts so the v2 scorer shape is stable.
api/health.js:
- SEED_DOMAINS: add `lowCarbonGeneration`, `fossilElectricityShare`,
`powerLosses` → their Redis keys.
- SEED_META: same three, pointing at `seed-meta:resilience:*` meta
keys with maxStaleMin=11520 (8d, per the worldmonitor
health-maxstalemin-write-cadence pattern: 2x weekly cron).
- ON_DEMAND_KEYS: three new entries gated as TRANSITIONAL until
Railway cron provisions and the first clean run completes. Remove
from this set after ~7 days of green production runs.
Typecheck clean; existing 106 resilience tests pass (seeders have no
in-repo callers yet, so nothing depends on them executing). Real-API
integration tests land when Railway cron is provisioned.
Next commit: Railway cron configuration + bundle-runner wiring.
* feat(resilience): PR 1 — bundle-runner + acceptance-gate verdict + flag-flip runbook
Final commit in the PR 1 tranche. Lands the three remaining pieces so
the flag-flip is fully operable once Railway cron provisions.
- scripts/seed-bundle-resilience-energy-v2.mjs
Railway cron bundle wrapping the three v2 energy seeders
(low-carbon-generation, fossil-electricity-share, power-losses).
Weekly cadence (7-day intervalMs); the underlying data is annual
at source so polling more frequently just hammers the World Bank
API. 5-minute per-script timeout. Mirrors the existing
seed-bundle-resilience-recovery.mjs pattern.
- scripts/compare-resilience-current-vs-proposed.mjs: acceptanceGates
block. Programmatic evaluation of plan §6 gates using the inputs
the harness already computes:
gate-1-spearman Spearman vs baseline >= 0.85
gate-2-country-drift Max country drift vs baseline <= 15
gate-6-cohort-median Cohort median shift vs baseline <= 10
gate-7-matched-pair Every pair holds expected direction
gate-9-effective-influence >= 80% Core indicators measurable
gate-universe-integrity No cohort/pair endpoint missing from scorable
Thresholds are encoded in a const so they can't silently soften.
Output verdict is PASS / CONDITIONAL / BLOCK. Emitted in
summary.acceptanceVerdict for at-a-glance PR comment pasting, with
full per-gate detail in acceptanceGates.results.
- docs/methodology/energy-v2-flag-flip-runbook.md
Operator runbook for the flag flip. Pre-flip checklist (seeders
green, health endpoint green, ON_DEMAND_KEYS graduation, Spearman
verification), flip procedure (pre-flip snapshot, dry-run, cache
prefix bump, Vercel env flip, post-flip snapshot, methodology
doc reclassification), rollback procedure, and a reference table
for the three possible verdict states.
PR 1 is now code-complete pending:
1. Railway cron provisioning (ops, not code)
2. Flag flip + acceptance-gate rerun (follows runbook, not code)
3. Reserve-margin seeder (deferred per plan §3.1 open-question)
Zero scoring-behaviour change in this commit. 121 resilience tests
pass, typecheck clean.
* fix(resilience): PR 1 — drop unseeded reserveMargin from scorer + fix composite extractor
Addresses two P1 review findings on PR #3289.
Finding 1: scoreEnergyV2 read resilience:reserve-margin:v1 at weight
0.10 but no seeder ships in this PR (indicator deferred per plan
§3.1 open-question). On flag flip that slot would be permanently
null, silently renormalizing the remaining 90% of weight and
producing a construct different from what the methodology doc
describes. Fix: remove reserve-margin from the v2 reader +
blend entirely. Redistribute its 0.10 weight to powerLossesPct
(now 0.20); both are grid-integrity signals per plan §3.1, and
the original plan split electricityConsumption's 0.30 weight
across powerLossesPct + reserveMarginPct + importedFossilDependence
— without reserveMarginPct, powerLossesPct carries the shared
grid-integrity load until the IEA seeder ships.
v2 weights now: 0.35 + 0.20 + 0.20 + 0.10 + 0.15 = 1.00
(importedFossilDependence + lowCarbonGenerationShare +
powerLossesPct + euGasStorageStress + energyPriceStress)
Reserve-margin Redis key constant stays reserved so the v2
scorer shape is stable when a future commit lands the seeder;
split 0.10 back out of powerLossesPct at that point.
Methodology doc, _shared.ts flag comment, and v2 test suite all
updated to the 5-indicator shape. New regression test asserts
that changing reserve-margin Redis content has zero effect on
the v2 score — guards against a future commit accidentally
wiring the reader back in without its seeder.
Finding 2: scripts/compare-resilience-current-vs-proposed.mjs
measured importedFossilDependence by reading fossilElectricityShare
alone. The scorer defines it as fossilShare × max(netImports, 0)
/ 100, so the extractor zeroed out net exporters and
under-reported net importers — making gate-9 effective-influence
wrong for the centrepiece construct change of PR 1.
Fix: new 'imported-fossil-dependence-composite' extractor type
in applyExtractionRule that recomputes the same composite from
both inputs (fossilShare bulk payload + staticRecord.iea.
energyImportDependency.value). Stays in lockstep with the
scorer — drift between the two would break gate-9's
interpretation.
New unit tests pin:
- net importer: 80% × max(60, 0) / 100 = 48 ✓
- net exporter: 80% × max(-40, 0) / 100 = 0 ✓
- missing either input → null
64 resilience tests pass; typecheck clean. Flag-off path is
still byte-identical to pre-PR behaviour.
* docs(resilience): PR 1 — align methodology doc with actual shipped indicators and seeders
Addresses P1 review on docs/methodology/country-resilience-index.mdx
lines 29 and 574-575. The doc still described reserveMarginPct as a
shipped v2 indicator and listed seed-net-energy-imports.mjs in the
new-seeders list, neither of which the branch actually ships.
Doc changes to match the code in this branch:
Known-limitations item 1: restated to describe the actual v2
replacement footprint — powerLossesPct at 0.20 (temporarily
absorbing reserveMarginPct's 0.10) plus accessToElectricityPct
moved to infrastructure. reserveMarginPct is named as a deferred
companion with the split-out instructions for when its seeder
lands.
v2.1 changelog (Indicators added): split into "live in PR 1" and
"deferred in PR 1" so the reader can distinguish which entries
match real code. importedFossilDependence's composite formula
now written out and the net-imports source attributed to the
existing resilience:static.iea path (not a new seeder).
v2.1 changelog (New seeders): lists the three actual files that
ship in this branch (seed-low-carbon-generation, seed-fossil-
electricity-share, seed-power-reliability) and explicitly notes
seed-net-energy-imports.mjs is NOT a new seeder — the
EG.IMP.CONS.ZS series is already fetched by seed-resilience-
static.mjs. Adds the bundle-runner reference.
Methodology-doc linter + mdx-lint both pass (125/125). Typecheck
clean. Doc is now the source of truth for what PR 1 actually ships.
* fix(resilience): PR 1 — sync powerLossesPct registry weight with scorer (0.10 → 0.20)
Reviewer-caught mismatch between INDICATOR_REGISTRY and scoreEnergyV2.
The previous commit redistributed the deferred reserveMarginPct's 0.10
weight into powerLossesPct in the SCORER but left the REGISTRY entry
unchanged at 0.10. Two downstream effects:
1. scripts/compare-resilience-current-vs-proposed.mjs copies
`spec.weight` into `nominalWeight` for gate-9 reporting, so
powerLossesPct's nominal influence would be under-reported by
half in every post-flip acceptance run — exactly the harness PR 1
relies on for merge evidence.
2. Methodology doc vs registry vs scorer drift is the pattern the
methodology-doc linter is supposed to catch; it passes here
because the linter only checks dimension-id parity, not weights.
Registry is now the only remaining source of truth to keep in
lockstep with the scorer.
Change:
- `_indicator-registry.ts` powerLossesPct.weight: 0.1 → 0.2
- Inline comment names the deferral and instructs: "when the IEA
electricity-balance seeder lands, split 0.10 back out and restore
reserveMarginPct at 0.10. Keep this field in lockstep with
scoreEnergyV2 ... because the PR 0 compare harness copies
spec.weight into nominalWeight for gate-9 reporting."
Experimental weights per dimension invariant still holds (0.35 + 0.20
+ 0.20 = 0.75 for energy, well under the 1.0 ceiling). 64 resilience
tests pass, typecheck clean.
780 lines
53 KiB
JavaScript
780 lines
53 KiB
JavaScript
import { jsonResponse } from './_json-response.js';
|
||
// Seed-envelope helper. PR 1 imports it here so PR 2 can wire envelope-aware
|
||
// reads at specific call sites without further plumbing. It's a no-op on
|
||
// legacy-shape seed-meta values (they have no `_seed` wrapper and pass through
|
||
// as `.data`), so importing it is behavior-preserving.
|
||
import { unwrapEnvelope } from './_seed-envelope.js';
|
||
// @ts-expect-error — JS module, no declaration file
|
||
import { redisPipeline, getRedisCredentials } from './_upstash-json.js';
|
||
|
||
export const config = { runtime: 'edge' };
|
||
|
||
const BOOTSTRAP_KEYS = {
|
||
earthquakes: 'seismology:earthquakes:v1',
|
||
outages: 'infra:outages:v1',
|
||
sectors: 'market:sectors:v2',
|
||
etfFlows: 'market:etf-flows:v1',
|
||
climateAnomalies: 'climate:anomalies:v2',
|
||
climateDisasters: 'climate:disasters:v1',
|
||
climateAirQuality: 'climate:air-quality:v1',
|
||
co2Monitoring: 'climate:co2-monitoring:v1',
|
||
oceanIce: 'climate:ocean-ice:v1',
|
||
wildfires: 'wildfire:fires:v1',
|
||
marketQuotes: 'market:stocks-bootstrap:v1',
|
||
commodityQuotes: 'market:commodities-bootstrap:v1',
|
||
cyberThreats: 'cyber:threats-bootstrap:v2',
|
||
techReadiness: 'economic:worldbank-techreadiness:v1',
|
||
progressData: 'economic:worldbank-progress:v1',
|
||
renewableEnergy: 'economic:worldbank-renewable:v1',
|
||
positiveGeoEvents: 'positive_events:geo-bootstrap:v1',
|
||
riskScores: 'risk:scores:sebuf:stale:v1',
|
||
naturalEvents: 'natural:events:v1',
|
||
flightDelays: 'aviation:delays-bootstrap:v1',
|
||
newsInsights: 'news:insights:v1',
|
||
predictionMarkets: 'prediction:markets-bootstrap:v1',
|
||
cryptoQuotes: 'market:crypto:v1',
|
||
gulfQuotes: 'market:gulf-quotes:v1',
|
||
stablecoinMarkets: 'market:stablecoins:v1',
|
||
unrestEvents: 'unrest:events:v1',
|
||
iranEvents: 'conflict:iran-events:v1',
|
||
ucdpEvents: 'conflict:ucdp-events:v1',
|
||
weatherAlerts: 'weather:alerts:v1',
|
||
spending: 'economic:spending:v1',
|
||
techEvents: 'research:tech-events-bootstrap:v1',
|
||
gdeltIntel: 'intelligence:gdelt-intel:v1',
|
||
correlationCards: 'correlation:cards-bootstrap:v1',
|
||
forecasts: 'forecast:predictions:v2',
|
||
securityAdvisories: 'intelligence:advisories-bootstrap:v1',
|
||
customsRevenue: 'trade:customs-revenue:v1',
|
||
comtradeFlows: 'comtrade:flows:v1',
|
||
blsSeries: 'bls:series:v1',
|
||
sanctionsPressure: 'sanctions:pressure:v1',
|
||
crossSourceSignals: 'intelligence:cross-source-signals:v1',
|
||
sanctionsEntities: 'sanctions:entities:v1',
|
||
radiationWatch: 'radiation:observations:v1',
|
||
consumerPricesOverview: 'consumer-prices:overview:ae',
|
||
consumerPricesCategories: 'consumer-prices:categories:ae:30d',
|
||
consumerPricesMovers: 'consumer-prices:movers:ae:30d',
|
||
consumerPricesSpread: 'consumer-prices:retailer-spread:ae:essentials-ae',
|
||
consumerPricesFreshness: 'consumer-prices:freshness:ae',
|
||
groceryBasket: 'economic:grocery-basket:v1',
|
||
bigmac: 'economic:bigmac:v1',
|
||
fuelPrices: 'economic:fuel-prices:v1',
|
||
faoFoodPriceIndex: 'economic:fao-ffpi:v1',
|
||
nationalDebt: 'economic:national-debt:v1',
|
||
defiTokens: 'market:defi-tokens:v1',
|
||
aiTokens: 'market:ai-tokens:v1',
|
||
otherTokens: 'market:other-tokens:v1',
|
||
fredBatch: 'economic:fred:v1:FEDFUNDS:0',
|
||
ecbEstr: 'economic:fred:v1:ESTR:0',
|
||
ecbEuribor3m: 'economic:fred:v1:EURIBOR3M:0',
|
||
ecbEuribor6m: 'economic:fred:v1:EURIBOR6M:0',
|
||
ecbEuribor1y: 'economic:fred:v1:EURIBOR1Y:0',
|
||
fearGreedIndex: 'market:fear-greed:v1',
|
||
breadthHistory: 'market:breadth-history:v1',
|
||
euYieldCurve: 'economic:yield-curve-eu:v1',
|
||
earningsCalendar: 'market:earnings-calendar:v1',
|
||
econCalendar: 'economic:econ-calendar:v1',
|
||
cotPositioning: 'market:cot:v1',
|
||
hyperliquidFlow: 'market:hyperliquid:flow:v1',
|
||
crudeInventories: 'economic:crude-inventories:v1',
|
||
natGasStorage: 'economic:nat-gas-storage:v1',
|
||
spr: 'economic:spr:v1',
|
||
refineryInputs: 'economic:refinery-inputs:v1',
|
||
ecbFxRates: 'economic:ecb-fx-rates:v1',
|
||
eurostatCountryData: 'economic:eurostat-country-data:v1',
|
||
eurostatHousePrices: 'economic:eurostat:house-prices:v1',
|
||
eurostatGovDebtQ: 'economic:eurostat:gov-debt-q:v1',
|
||
eurostatIndProd: 'economic:eurostat:industrial-production:v1',
|
||
euGasStorage: 'economic:eu-gas-storage:v1',
|
||
euFsi: 'economic:fsi-eu:v1',
|
||
shippingStress: 'supply_chain:shipping_stress:v1',
|
||
diseaseOutbreaks: 'health:disease-outbreaks:v1',
|
||
healthAirQuality: 'health:air-quality:v1',
|
||
socialVelocity: 'intelligence:social:reddit:v1',
|
||
wsbTickers: 'intelligence:wsb-tickers:v1',
|
||
vpdTrackerRealtime: 'health:vpd-tracker:realtime:v1',
|
||
vpdTrackerHistorical: 'health:vpd-tracker:historical:v1',
|
||
electricityPrices: 'energy:electricity:v1:index',
|
||
gasStorageCountries: 'energy:gas-storage:v1:_countries',
|
||
aaiiSentiment: 'market:aaii-sentiment:v1',
|
||
cryptoSectors: 'market:crypto-sectors:v1',
|
||
ddosAttacks: 'cf:radar:ddos:v1',
|
||
economicStress: 'economic:stress-index:v1',
|
||
trafficAnomalies: 'cf:radar:traffic-anomalies:v1',
|
||
};
|
||
|
||
const STANDALONE_KEYS = {
|
||
serviceStatuses: 'infra:service-statuses:v1',
|
||
macroSignals: 'economic:macro-signals:v1',
|
||
bisPolicy: 'economic:bis:policy:v1',
|
||
bisExchange: 'economic:bis:eer:v1',
|
||
fxYoy: 'economic:fx:yoy:v1',
|
||
bisCredit: 'economic:bis:credit:v1',
|
||
bisDsr: 'economic:bis:dsr:v1',
|
||
bisPropertyResidential: 'economic:bis:property-residential:v1',
|
||
bisPropertyCommercial: 'economic:bis:property-commercial:v1',
|
||
imfMacro: 'economic:imf:macro:v2',
|
||
imfGrowth: 'economic:imf:growth:v1',
|
||
imfLabor: 'economic:imf:labor:v1',
|
||
imfExternal: 'economic:imf:external:v1',
|
||
climateZoneNormals: 'climate:zone-normals:v1',
|
||
shippingRates: 'supply_chain:shipping:v2',
|
||
chokepoints: 'supply_chain:chokepoints:v4',
|
||
minerals: 'supply_chain:minerals:v2',
|
||
giving: 'giving:summary:v1',
|
||
gpsjam: 'intelligence:gpsjam:v2',
|
||
theaterPosture: 'theater_posture:sebuf:stale:v1',
|
||
theaterPostureLive: 'theater-posture:sebuf:v1',
|
||
theaterPostureBackup: 'theater-posture:sebuf:backup:v1',
|
||
riskScoresLive: 'risk:scores:sebuf:v1',
|
||
usniFleet: 'usni-fleet:sebuf:v1',
|
||
usniFleetStale: 'usni-fleet:sebuf:stale:v1',
|
||
faaDelays: 'aviation:delays:faa:v1',
|
||
intlDelays: 'aviation:delays:intl:v3',
|
||
notamClosures: 'aviation:notam:closures:v2',
|
||
positiveEventsLive: 'positive-events:geo:v1',
|
||
cableHealth: 'cable-health-v1',
|
||
cyberThreatsRpc: 'cyber:threats:v2',
|
||
militaryBases: 'military:bases:active',
|
||
militaryFlights: 'military:flights:v1',
|
||
militaryFlightsStale: 'military:flights:stale:v1',
|
||
temporalAnomalies: 'temporal:anomalies:v1',
|
||
displacement: `displacement:summary:v1:${new Date().getUTCFullYear()}`,
|
||
displacementPrev: `displacement:summary:v1:${new Date().getUTCFullYear() - 1}`,
|
||
satellites: 'intelligence:satellites:tle:v1',
|
||
portwatch: 'supply_chain:portwatch:v1',
|
||
portwatchPortActivity: 'supply_chain:portwatch-ports:v1:_countries',
|
||
corridorrisk: 'supply_chain:corridorrisk:v1',
|
||
chokepointTransits: 'supply_chain:chokepoint_transits:v1',
|
||
transitSummaries: 'supply_chain:transit-summaries:v1',
|
||
thermalEscalation: 'thermal:escalation:v1',
|
||
tariffTrendsUs: 'trade:tariffs:v1:840:all:10',
|
||
militaryForecastInputs: 'military:forecast-inputs:stale:v1',
|
||
gscpi: 'economic:fred:v1:GSCPI:0',
|
||
marketImplications: 'intelligence:market-implications:v1',
|
||
hormuzTracker: 'supply_chain:hormuz_tracker:v1',
|
||
simulationPackageLatest: 'forecast:simulation-package:latest',
|
||
simulationOutcomeLatest: 'forecast:simulation-outcome:latest',
|
||
newsThreatSummary: 'news:threat:summary:v1',
|
||
climateNews: 'climate:news-intelligence:v1',
|
||
pizzint: 'intelligence:pizzint:seed:v1',
|
||
resilienceStaticIndex: 'resilience:static:index:v1',
|
||
resilienceStaticFao: 'resilience:static:fao',
|
||
resilienceRanking: 'resilience:ranking:v10',
|
||
productCatalog: 'product-catalog:v2',
|
||
energySpineCountries: 'energy:spine:v1:_countries',
|
||
energyExposure: 'energy:exposure:v1:index',
|
||
energyMixAll: 'energy:mix:v1:_all',
|
||
regulatoryActions: 'regulatory:actions:v1',
|
||
energyIntelligence: 'energy:intelligence:feed:v1',
|
||
ieaOilStocks: 'energy:iea-oil-stocks:v1:index',
|
||
oilStocksAnalysis: 'energy:oil-stocks-analysis:v1',
|
||
eiaPetroleum: 'energy:eia-petroleum:v1',
|
||
jodiGas: 'energy:jodi-gas:v1:_countries',
|
||
lngVulnerability: 'energy:lng-vulnerability:v1',
|
||
jodiOil: 'energy:jodi-oil:v1:_countries',
|
||
chokepointBaselines: 'energy:chokepoint-baselines:v1',
|
||
portwatchChokepointsRef: 'portwatch:chokepoints:ref:v1',
|
||
chokepointFlows: 'energy:chokepoint-flows:v1',
|
||
emberElectricity: 'energy:ember:v1:_all',
|
||
resilienceIntervals: 'resilience:intervals:v1:US',
|
||
sprPolicies: 'energy:spr-policies:v1',
|
||
energyCrisisPolicies: 'energy:crisis-policies:v1',
|
||
regionalSnapshots: 'intelligence:regional-snapshots:summary:v1',
|
||
regionalBriefs: 'intelligence:regional-briefs:summary:v1',
|
||
recoveryFiscalSpace: 'resilience:recovery:fiscal-space:v1',
|
||
recoveryReserveAdequacy: 'resilience:recovery:reserve-adequacy:v1',
|
||
recoveryExternalDebt: 'resilience:recovery:external-debt:v1',
|
||
recoveryImportHhi: 'resilience:recovery:import-hhi:v1',
|
||
recoveryFuelStocks: 'resilience:recovery:fuel-stocks:v1',
|
||
// PR 1 v2 energy-construct seeds. ON_DEMAND_KEYS until Railway cron
|
||
// provisions; see below.
|
||
lowCarbonGeneration: 'resilience:low-carbon-generation:v1',
|
||
fossilElectricityShare: 'resilience:fossil-electricity-share:v1',
|
||
powerLosses: 'resilience:power-losses:v1',
|
||
goldExtended: 'market:gold-extended:v1',
|
||
goldEtfFlows: 'market:gold-etf-flows:v1',
|
||
goldCbReserves: 'market:gold-cb-reserves:v1',
|
||
// Relay-side loop heartbeats. ais-relay.cjs writes these on successful child
|
||
// exit for the two execFile-spawned seeders (chokepoint-flows, climate-news).
|
||
// A stale heartbeat means the relay loop itself is broken (child dying at
|
||
// import, parent event-loop blocked, container in a restart loop, etc.)
|
||
// and alarms earlier than the underlying seed-meta staleness window.
|
||
chokepointFlowsRelayHeartbeat: 'relay:heartbeat:chokepoint-flows',
|
||
climateNewsRelayHeartbeat: 'relay:heartbeat:climate-news',
|
||
telegramFeed: 'intelligence:telegram-feed:v1',
|
||
};
|
||
|
||
const SEED_META = {
|
||
earthquakes: { key: 'seed-meta:seismology:earthquakes', maxStaleMin: 30 },
|
||
wildfires: { key: 'seed-meta:wildfire:fires', maxStaleMin: 360 }, // FIRMS NRT resets at midnight UTC; new-day data takes 3-6h to accumulate
|
||
outages: { key: 'seed-meta:infra:outages', maxStaleMin: 30 },
|
||
climateAnomalies: { key: 'seed-meta:climate:anomalies', maxStaleMin: 240 }, // runs as independent Railway cron (0 */2 * * *); 240 = 2x interval
|
||
climateDisasters: { key: 'seed-meta:climate:disasters', maxStaleMin: 720 }, // runs every 6h; 720min = 2x interval
|
||
climateAirQuality:{ key: 'seed-meta:health:air-quality', maxStaleMin: 180 }, // hourly cron; 180 = 3x interval — shares meta key with healthAirQuality (same seeder run)
|
||
climateZoneNormals: { key: 'seed-meta:climate:zone-normals', maxStaleMin: 89280 }, // monthly cron on the 1st; 62d = 2x 31-day cadence
|
||
co2Monitoring: { key: 'seed-meta:climate:co2-monitoring', maxStaleMin: 4320 }, // daily cron at 06:00 UTC; 72h tolerates two missed runs
|
||
oceanIce: { key: 'seed-meta:climate:ocean-ice', maxStaleMin: 2880 }, // daily cron at 08:00 UTC; 48h = 2× interval, tolerates one missed run
|
||
climateNews: { key: 'seed-meta:climate:news-intelligence', maxStaleMin: 90 }, // relay loop every 30min; 90 = 3× interval
|
||
unrestEvents: { key: 'seed-meta:unrest:events', maxStaleMin: 120 }, // 45min cron; 120 = 2h grace (was 75 = 30min buffer, too tight)
|
||
cyberThreats: { key: 'seed-meta:cyber:threats', maxStaleMin: 240 }, // 2h interval; 240min = 2x interval
|
||
cryptoQuotes: { key: 'seed-meta:market:crypto', maxStaleMin: 30 },
|
||
etfFlows: { key: 'seed-meta:market:etf-flows', maxStaleMin: 60 },
|
||
gulfQuotes: { key: 'seed-meta:market:gulf-quotes', maxStaleMin: 30 },
|
||
stablecoinMarkets:{ key: 'seed-meta:market:stablecoins', maxStaleMin: 60 },
|
||
naturalEvents: { key: 'seed-meta:natural:events', maxStaleMin: 360 }, // 2h cron; 3x interval; was 120 (TTL was 60min — panel went dark before health alarmed)
|
||
flightDelays: { key: 'seed-meta:aviation:faa', maxStaleMin: 90 }, // CACHE_TTL=7200s; matches notamClosures from same cron
|
||
notamClosures: { key: 'seed-meta:aviation:notam', maxStaleMin: 240 }, // 2h interval; 240min = 2x interval
|
||
predictionMarkets: { key: 'seed-meta:prediction:markets', maxStaleMin: 90 },
|
||
newsInsights: { key: 'seed-meta:news:insights', maxStaleMin: 30 },
|
||
marketQuotes: { key: 'seed-meta:market:stocks', maxStaleMin: 30 },
|
||
commodityQuotes: { key: 'seed-meta:market:commodities', maxStaleMin: 30 },
|
||
goldExtended: { key: 'seed-meta:market:gold-extended', maxStaleMin: 30 },
|
||
goldEtfFlows: { key: 'seed-meta:market:gold-etf-flows', maxStaleMin: 2880 }, // SPDR publishes daily; 2× = 48h tolerance
|
||
goldCbReserves: { key: 'seed-meta:market:gold-cb-reserves', maxStaleMin: 44640 }, // IMF IFS is monthly w/ ~2-3mo lag; 31d tolerance
|
||
// RPC/warm-ping keys — seed-meta written by relay loops or handlers
|
||
// serviceStatuses: moved to ON_DEMAND — RPC-populated, no dedicated seed, goes stale when no users visit
|
||
cableHealth: { key: 'seed-meta:cable-health', maxStaleMin: 90 }, // ais-relay warm-ping runs every 30min; 90min = 3× interval catches missed pings without false positives
|
||
macroSignals: { key: 'seed-meta:economic:macro-signals', maxStaleMin: 150 }, // seed-economy cron; primary key energy-prices has same 150min threshold
|
||
bisPolicy: { key: 'seed-meta:economic:bis', maxStaleMin: 10080 }, // runSeed('economic','bis',...) writes seed-meta:economic:bis
|
||
// seed-bis-extended.mjs writes per-dataset seed-meta keys ONLY when that
|
||
// specific dataset published fresh entries — so a single-dataset BIS outage
|
||
// (e.g. WS_DSR 500s) goes stale in health without falsely dragging down the
|
||
// healthy ones. 24h = 2× 12h cron.
|
||
bisDsr: { key: 'seed-meta:economic:bis-dsr', maxStaleMin: 1440 },
|
||
bisPropertyResidential:{ key: 'seed-meta:economic:bis-property-residential', maxStaleMin: 1440 },
|
||
bisPropertyCommercial: { key: 'seed-meta:economic:bis-property-commercial', maxStaleMin: 1440 },
|
||
imfMacro: { key: 'seed-meta:economic:imf-macro', maxStaleMin: 100800 }, // monthly seed; 100800min = 70 days = 2× interval (absorbs one missed run)
|
||
imfGrowth: { key: 'seed-meta:economic:imf-growth', maxStaleMin: 100800 }, // monthly seed; 70d threshold matches imfMacro (same WEO release cadence)
|
||
imfLabor: { key: 'seed-meta:economic:imf-labor', maxStaleMin: 100800 }, // monthly seed; 70d threshold matches imfMacro
|
||
imfExternal: { key: 'seed-meta:economic:imf-external', maxStaleMin: 100800 }, // monthly seed; 70d threshold matches imfMacro
|
||
shippingRates: { key: 'seed-meta:supply_chain:shipping', maxStaleMin: 420 },
|
||
chokepoints: { key: 'seed-meta:supply_chain:chokepoints', maxStaleMin: 60, minRecordCount: 13 }, // 13 canonical chokepoints; get-chokepoint-status writes covered-count → < 13 = upstream partial (portwatch/ArcGIS dropped some)
|
||
// minerals + giving: on-demand cachedFetchJson only, no seed-meta writer — freshness checked via TTL
|
||
// bisExchange + bisCredit: extras written by same BIS script via writeExtraKey, no dedicated seed-meta
|
||
fxYoy: { key: 'seed-meta:economic:fx-yoy', maxStaleMin: 1500 }, // daily cron; 25h tolerance + 1h drift
|
||
gpsjam: { key: 'seed-meta:intelligence:gpsjam', maxStaleMin: 720 },
|
||
positiveGeoEvents:{ key: 'seed-meta:positive-events:geo', maxStaleMin: 60 },
|
||
riskScores: { key: 'seed-meta:intelligence:risk-scores', maxStaleMin: 30 }, // CII warm-ping every 8min; 30min = ~3.5x interval,
|
||
iranEvents: { key: 'seed-meta:conflict:iran-events', maxStaleMin: 20160 }, // manual seed from LiveUAMap; 20160 = 14d = 2× weekly cadence
|
||
ucdpEvents: { key: 'seed-meta:conflict:ucdp-events', maxStaleMin: 420 },
|
||
militaryFlights: { key: 'seed-meta:military:flights', maxStaleMin: 30 }, // cron ~10min (LIVE_TTL=600s); 30min = 3x interval,
|
||
satellites: { key: 'seed-meta:intelligence:satellites', maxStaleMin: 240 }, // CelesTrak every 120min; 240min = absorbs one missed cycle
|
||
weatherAlerts: { key: 'seed-meta:weather:alerts', maxStaleMin: 45 }, // relay loop every 15min; 45 = 3× interval (was 30 = 2×, too tight on relay hiccup)
|
||
spending: { key: 'seed-meta:economic:spending', maxStaleMin: 120 },
|
||
techEvents: { key: 'seed-meta:research:tech-events', maxStaleMin: 480 },
|
||
gdeltIntel: { key: 'seed-meta:intelligence:gdelt-intel', maxStaleMin: 420 }, // 6h cron + 1h grace; CACHE_TTL is 24h so per-topic merge always has a prior snapshot
|
||
telegramFeed: { key: 'seed-meta:intelligence:telegram-feed:v1', maxStaleMin: 10 }, // 60s poll interval; 10min grace catches poll failures before they go stale in the panel
|
||
forecasts: { key: 'seed-meta:forecast:predictions', maxStaleMin: 90 },
|
||
sectors: { key: 'seed-meta:market:sectors', maxStaleMin: 30 },
|
||
techReadiness: { key: 'seed-meta:economic:worldbank-techreadiness:v1', maxStaleMin: 10080 },
|
||
progressData: { key: 'seed-meta:economic:worldbank-progress:v1', maxStaleMin: 10080 },
|
||
renewableEnergy: { key: 'seed-meta:economic:worldbank-renewable:v1', maxStaleMin: 10080 },
|
||
intlDelays: { key: 'seed-meta:aviation:intl', maxStaleMin: 90 },
|
||
// faaDelays shares seed-meta key with flightDelays — no duplicate entry needed here
|
||
theaterPosture: { key: 'seed-meta:theater-posture', maxStaleMin: 60 },
|
||
correlationCards: { key: 'seed-meta:correlation:cards', maxStaleMin: 15 },
|
||
portwatch: { key: 'seed-meta:supply_chain:portwatch', maxStaleMin: 720 },
|
||
portwatchPortActivity: { key: 'seed-meta:supply_chain:portwatch-ports', maxStaleMin: 2160 }, // 12h cron; 2160min = 36h = 3x interval
|
||
corridorrisk: { key: 'seed-meta:supply_chain:corridorrisk', maxStaleMin: 120 },
|
||
chokepointTransits: { key: 'seed-meta:supply_chain:chokepoint_transits', maxStaleMin: 30 }, // relay every 10min; 30min = 3x interval,
|
||
transitSummaries: { key: 'seed-meta:supply_chain:transit-summaries', maxStaleMin: 30 }, // relay every 10min; 30min = 3x interval,
|
||
usniFleet: { key: 'seed-meta:military:usni-fleet', maxStaleMin: 720 }, // relay loop every 6h; 720 = 2× interval (was 480 = 1.3×, too tight)
|
||
securityAdvisories: { key: 'seed-meta:intelligence:advisories', maxStaleMin: 120 },
|
||
customsRevenue: { key: 'seed-meta:trade:customs-revenue', maxStaleMin: 1440 },
|
||
comtradeFlows: { key: 'seed-meta:trade:comtrade-flows', maxStaleMin: 2880 }, // 24h cron; 2880min = 48h = 2x interval
|
||
blsSeries: { key: 'seed-meta:economic:bls-series', maxStaleMin: 2880 }, // daily seed; 2880min = 48h = 2x interval
|
||
sanctionsPressure: { key: 'seed-meta:sanctions:pressure', maxStaleMin: 720 },
|
||
crossSourceSignals: { key: 'seed-meta:intelligence:cross-source-signals', maxStaleMin: 30 }, // 15min cron; 30min = 2x interval
|
||
regionalSnapshots: { key: 'seed-meta:intelligence:regional-snapshots', maxStaleMin: 720 }, // 6h cron via seed-bundle-derived-signals; 720min = 12h = 2x interval
|
||
regionalBriefs: { key: 'seed-meta:intelligence:regional-briefs', maxStaleMin: 20160 }, // weekly cron; 20160min = 14 days = 2x interval
|
||
sanctionsEntities: { key: 'seed-meta:sanctions:entities', maxStaleMin: 1440 }, // 12h cron; 1440min = 24h = 2x interval
|
||
radiationWatch: { key: 'seed-meta:radiation:observations', maxStaleMin: 30 },
|
||
groceryBasket: { key: 'seed-meta:economic:grocery-basket', maxStaleMin: 10080 }, // weekly seed; 10080 = 7 days
|
||
bigmac: { key: 'seed-meta:economic:bigmac', maxStaleMin: 10080 }, // weekly seed; 10080 = 7 days
|
||
fuelPrices: { key: 'seed-meta:economic:fuel-prices', maxStaleMin: 10080 }, // weekly seed; 10080 = 7 days
|
||
faoFoodPriceIndex: { key: 'seed-meta:economic:fao-ffpi', maxStaleMin: 86400 }, // monthly seed; 86400 = 60 days (2x interval)
|
||
thermalEscalation: { key: 'seed-meta:thermal:escalation', maxStaleMin: 360 }, // cron every 2h; 360 = 3x interval (was 240 = 2x)
|
||
nationalDebt: { key: 'seed-meta:economic:national-debt', maxStaleMin: 86400 }, // monthly seed (seed-bundle-macro intervalMs: 30 * DAY); 60d = 2x interval absorbs one missed run. Prior 10080 (7d) was narrower than the cron interval so every cron past day 7 alarmed STALE_SEED.
|
||
tariffTrendsUs: { key: 'seed-meta:trade:tariffs:v1:840:all:10', maxStaleMin: 900 },
|
||
// publish.ts runs once daily (02:30 UTC); seed-meta TTL=52h — maxStaleMin must cover the full 24h cycle
|
||
consumerPricesOverview: { key: 'seed-meta:consumer-prices:overview:ae', maxStaleMin: 1500 }, // 25h = 24h cadence + 1h grace
|
||
consumerPricesCategories: { key: 'seed-meta:consumer-prices:categories:ae:30d', maxStaleMin: 1500 },
|
||
consumerPricesMovers: { key: 'seed-meta:consumer-prices:movers:ae:30d', maxStaleMin: 1500 },
|
||
consumerPricesSpread: { key: 'seed-meta:consumer-prices:retailer-spread:ae:essentials-ae', maxStaleMin: 1500 },
|
||
consumerPricesFreshness: { key: 'seed-meta:consumer-prices:freshness:ae', maxStaleMin: 1500 },
|
||
// defiTokens/aiTokens/otherTokens all share one seed run (seed-token-panels cron, every 30min)
|
||
defiTokens: { key: 'seed-meta:market:token-panels', maxStaleMin: 90 },
|
||
aiTokens: { key: 'seed-meta:market:token-panels', maxStaleMin: 90 },
|
||
otherTokens: { key: 'seed-meta:market:token-panels', maxStaleMin: 90 },
|
||
fredBatch: { key: 'seed-meta:economic:fred:v1:FEDFUNDS:0', maxStaleMin: 1500 }, // daily cron
|
||
ecbEstr: { key: 'seed-meta:economic:ecb-short-rates', maxStaleMin: 4320 }, // daily ECB publish; 4320min = 3d = TTL/interval
|
||
ecbEuribor3m: { key: 'seed-meta:economic:ecb-short-rates', maxStaleMin: 4320 }, // shared meta key with ecbEstr
|
||
ecbEuribor6m: { key: 'seed-meta:economic:ecb-short-rates', maxStaleMin: 4320 }, // shared meta key with ecbEstr
|
||
ecbEuribor1y: { key: 'seed-meta:economic:ecb-short-rates', maxStaleMin: 4320 }, // shared meta key with ecbEstr
|
||
gscpi: { key: 'seed-meta:economic:gscpi', maxStaleMin: 2880 }, // 24h interval; 2880min = 48h = 2x interval
|
||
fearGreedIndex: { key: 'seed-meta:market:fear-greed', maxStaleMin: 720 }, // 6h cron; 720min = 12h = 2x interval
|
||
breadthHistory: { key: 'seed-meta:market:breadth-history', maxStaleMin: 5760 }, // cron at 02:00 UTC, Tue-Sat (captures Mon-Fri market close); max gap Sat→Tue = 72h + 24h miss buffer = 96h = 5760min. 48h was wrong — alarmed every Monday morning when Sun+Mon are intentionally skipped.
|
||
hormuzTracker: { key: 'seed-meta:supply_chain:hormuz_tracker', maxStaleMin: 2880 }, // daily cron; 2880min = 48h = 2x interval
|
||
earningsCalendar: { key: 'seed-meta:market:earnings-calendar', maxStaleMin: 1440 }, // 12h cron; 1440min = 24h = 2x interval
|
||
econCalendar: { key: 'seed-meta:economic:econ-calendar', maxStaleMin: 1440 }, // 12h cron; 1440min = 24h = 2x interval
|
||
cotPositioning: { key: 'seed-meta:market:cot', maxStaleMin: 14400 }, // weekly CFTC release; 14400min = 10d = 1.4x interval (weekend + delay buffer)
|
||
hyperliquidFlow: { key: 'seed-meta:market:hyperliquid-flow', maxStaleMin: 15 }, // Railway cron 5min; 15min = 3x interval
|
||
crudeInventories: { key: 'seed-meta:economic:crude-inventories', maxStaleMin: 20160 }, // weekly EIA data; 20160min = 14 days = 2x weekly cadence
|
||
natGasStorage: { key: 'seed-meta:economic:nat-gas-storage', maxStaleMin: 20160 }, // weekly EIA data; 20160min = 14 days = 2x weekly cadence
|
||
spr: { key: 'seed-meta:economic:spr', maxStaleMin: 20160 }, // weekly EIA data; 20160min = 14 days = 2x weekly cadence
|
||
refineryInputs: { key: 'seed-meta:economic:refinery-inputs', maxStaleMin: 20160 }, // weekly EIA data; 20160min = 14 days = 2x weekly cadence
|
||
ecbFxRates: { key: 'seed-meta:economic:ecb-fx-rates', maxStaleMin: 5760 }, // daily seed (weekdays + holidays); 5760min = 96h = covers Wed→Mon Easter gap
|
||
eurostatCountryData: { key: 'seed-meta:economic:eurostat-country-data', maxStaleMin: 4320 }, // daily seed; 4320min = 3 days = 3x interval
|
||
eurostatHousePrices: { key: 'seed-meta:economic:eurostat-house-prices', maxStaleMin: 60 * 24 * 50 }, // weekly cron, annual data; 50d threshold = 35d TTL + 15d buffer
|
||
eurostatGovDebtQ: { key: 'seed-meta:economic:eurostat-gov-debt-q', maxStaleMin: 60 * 24 * 14 }, // 2d cron, quarterly data; 14d threshold matches TTL + quarterly release drift
|
||
eurostatIndProd: { key: 'seed-meta:economic:eurostat-industrial-production', maxStaleMin: 60 * 24 * 5 }, // daily cron, monthly data; 5d threshold matches TTL
|
||
euGasStorage: { key: 'seed-meta:economic:eu-gas-storage', maxStaleMin: 2880 }, // daily seed (T+1); 2880min = 48h = 2x interval
|
||
euYieldCurve: { key: 'seed-meta:economic:yield-curve-eu', maxStaleMin: 4320 }, // daily seed (weekdays only); 4320min = 72h = covers Fri→Mon gap
|
||
euFsi: { key: 'seed-meta:economic:fsi-eu', maxStaleMin: 20160 }, // weekly seed (Saturday); 20160min = 14d = 2x interval
|
||
newsThreatSummary: { key: 'seed-meta:news:threat-summary', maxStaleMin: 60 }, // relay classify every ~20min; 60min = 3x interval
|
||
shippingStress: { key: 'seed-meta:supply_chain:shipping_stress', maxStaleMin: 45 }, // relay loop every 15min; 45 = 3x interval (was 30 = 2×, too tight on relay hiccup)
|
||
diseaseOutbreaks: { key: 'seed-meta:health:disease-outbreaks', maxStaleMin: 2880 }, // daily seed; 2880 = 48h = 2x interval
|
||
healthAirQuality: { key: 'seed-meta:health:air-quality', maxStaleMin: 180 }, // hourly cron; 180 = 3x interval for shared health/climate seed
|
||
socialVelocity: { key: 'seed-meta:intelligence:social-reddit', maxStaleMin: 180 }, // relay loop every 60min (hourly, bumped from 10min to reduce Reddit IP blocking); 180 = 3x interval
|
||
wsbTickers: { key: 'seed-meta:intelligence:wsb-tickers', maxStaleMin: 180 }, // relay loop every 60min; 180 = 3x interval
|
||
pizzint: { key: 'seed-meta:intelligence:pizzint', maxStaleMin: 30 }, // relay loop every 10min; 30 = 3x interval
|
||
productCatalog: { key: 'seed-meta:product-catalog', maxStaleMin: 1080 }, // relay loop every 6h; 1080 = 18h = 3x interval
|
||
vpdTrackerRealtime: { key: 'seed-meta:health:vpd-tracker', maxStaleMin: 2880 }, // daily seed (0 2 * * *); 2880min = 48h = 2x interval
|
||
vpdTrackerHistorical: { key: 'seed-meta:health:vpd-tracker', maxStaleMin: 2880 }, // shares seed-meta key with vpdTrackerRealtime (same run)
|
||
resilienceStaticIndex: { key: 'seed-meta:resilience:static', maxStaleMin: 576000 }, // annual October snapshot; 400d threshold matches TTL and preserves prior-year data on source outages
|
||
resilienceStaticFao: { key: 'seed-meta:resilience:static', maxStaleMin: 576000 }, // same seeder + same heartbeat as resilienceStaticIndex; required so EMPTY_DATA_OK + missing data degrades to STALE_SEED instead of silent OK
|
||
resilienceRanking: { key: 'seed-meta:resilience:ranking', maxStaleMin: 720 }, // RPC cache (12h TTL, refreshed every 6h by seed-resilience-scores cron via refreshRankingAggregate); 12h staleness threshold = 2 missed cron ticks
|
||
resilienceIntervals: { key: 'seed-meta:resilience:intervals', maxStaleMin: 20160 }, // weekly cron; 20160min = 14d = 2x interval
|
||
energyExposure: { key: 'seed-meta:economic:owid-energy-mix', maxStaleMin: 50400 }, // monthly cron on 1st; 50400min = 35d = TTL matches cron cadence + 5d buffer
|
||
energyMixAll: { key: 'seed-meta:economic:owid-energy-mix', maxStaleMin: 50400 }, // same seed run as energyExposure; shares seed-meta key
|
||
regulatoryActions: { key: 'seed-meta:regulatory:actions', maxStaleMin: 360 }, // 2h cron; 360min = 3x interval
|
||
energySpineCountries: { key: 'seed-meta:energy:spine', maxStaleMin: 2880 }, // daily cron (06:00 UTC); 2880min = 48h = 2x interval
|
||
electricityPrices: { key: 'seed-meta:energy:electricity-prices', maxStaleMin: 2880 }, // daily cron (14:00 UTC); 2880min = 48h = 2x interval
|
||
gasStorageCountries: { key: 'seed-meta:energy:gas-storage-countries', maxStaleMin: 2880 }, // daily cron at 10:30 UTC; 2880min = 48h = 2x interval
|
||
energyIntelligence: { key: 'seed-meta:energy:intelligence', maxStaleMin: 720 }, // 6h cron; 720min = 2x interval
|
||
jodiOil: { key: 'seed-meta:energy:jodi-oil', maxStaleMin: 60 * 24 * 40 }, // monthly cron on 25th; 40d threshold matches 35d TTL + 5d buffer
|
||
ieaOilStocks: { key: 'seed-meta:energy:iea-oil-stocks', maxStaleMin: 60 * 24 * 40 }, // monthly cron on 15th; 40d threshold = TTL_SECONDS
|
||
oilStocksAnalysis: { key: 'seed-meta:energy:oil-stocks-analysis', maxStaleMin: 60 * 24 * 50 }, // afterPublish of ieaOilStocks; 50d = matches seed-meta TTL (exceeds 40d data TTL)
|
||
eiaPetroleum: { key: 'seed-meta:energy:eia-petroleum', maxStaleMin: 4320 }, // daily bundle cron (seed-bundle-energy-sources); 72h = 3× interval, well under 7d data TTL
|
||
jodiGas: { key: 'seed-meta:energy:jodi-gas', maxStaleMin: 60 * 24 * 40 }, // monthly cron on 25th; 40d threshold matches 35d TTL + 5d buffer
|
||
lngVulnerability: { key: 'seed-meta:energy:jodi-gas', maxStaleMin: 60 * 24 * 40 }, // written by jodi-gas seeder afterPublish; shares seed-meta key
|
||
chokepointBaselines: { key: 'seed-meta:energy:chokepoint-baselines', maxStaleMin: 60 * 24 * 400 }, // 400 days
|
||
sprPolicies: { key: 'seed-meta:energy:spr-policies', maxStaleMin: 60 * 24 * 400 }, // 400 days; static registry, same cadence as chokepoint baselines
|
||
energyCrisisPolicies: { key: 'seed-meta:energy:crisis-policies', maxStaleMin: 60 * 24 * 400 }, // static data, ~400d TTL matches seeder
|
||
aaiiSentiment: { key: 'seed-meta:market:aaii-sentiment', maxStaleMin: 20160 }, // weekly cron; 20160min = 14 days = 2x weekly cadence
|
||
portwatchChokepointsRef: { key: 'seed-meta:portwatch:chokepoints-ref', maxStaleMin: 60 * 24 * 14 }, // seed-bundle-portwatch runs this at WEEK cadence; 14d = 2× interval
|
||
chokepointFlows: { key: 'seed-meta:energy:chokepoint-flows', maxStaleMin: 720 }, // 6h cron; 720min = 2x interval
|
||
// Relay-side heartbeat written by ais-relay.cjs on successful child exit.
|
||
// Detects "relay loop fires but child dies at import/runtime" failures
|
||
// (e.g. ERR_MODULE_NOT_FOUND from a missing Dockerfile COPY) 4h earlier
|
||
// than the 720min seed-meta threshold above. TTL is 18h on the writer.
|
||
chokepointFlowsRelayHeartbeat: { key: 'relay:heartbeat:chokepoint-flows', maxStaleMin: 480 }, // 6h loop; 8h alarm
|
||
climateNewsRelayHeartbeat: { key: 'relay:heartbeat:climate-news', maxStaleMin: 60 }, // 30m loop; 60m alarm
|
||
emberElectricity: { key: 'seed-meta:energy:ember', maxStaleMin: 2880 }, // daily cron (08:00 UTC); 2880min = 48h = 2x interval
|
||
cryptoSectors: { key: 'seed-meta:market:crypto-sectors', maxStaleMin: 120 }, // relay loop every ~30min; 120min = 2h = 4x interval
|
||
ddosAttacks: { key: 'seed-meta:cf:radar:ddos', maxStaleMin: 60 }, // written by seed-internet-outages afterPublish; outages cron ~15min; 60 = 4x interval
|
||
economicStress: { key: 'seed-meta:economic:stress-index', maxStaleMin: 180 }, // computed in seed-economy afterPublish; cron ~1h; 180min = 3x interval
|
||
marketImplications: { key: 'seed-meta:intelligence:market-implications', maxStaleMin: 120 }, // LLM-generated in seed-forecasts; cron ~1h; 120min = 2x interval
|
||
trafficAnomalies: { key: 'seed-meta:cf:radar:traffic-anomalies', maxStaleMin: 60 }, // written by seed-internet-outages afterPublish; outages cron ~15min; 60 = 4x interval
|
||
chokepointExposure: { key: 'seed-meta:supply_chain:chokepoint-exposure', maxStaleMin: 2880 }, // daily cron; 2880min = 48h = 2x interval
|
||
recoveryFiscalSpace: { key: 'seed-meta:resilience:recovery:fiscal-space', maxStaleMin: 86400 }, // monthly cron; 86400min = 60d = 2x interval
|
||
recoveryReserveAdequacy: { key: 'seed-meta:resilience:recovery:reserve-adequacy', maxStaleMin: 86400 }, // monthly cron; 86400min = 60d = 2x interval
|
||
recoveryExternalDebt: { key: 'seed-meta:resilience:recovery:external-debt', maxStaleMin: 86400 }, // monthly cron; 86400min = 60d = 2x interval
|
||
recoveryImportHhi: { key: 'seed-meta:resilience:recovery:import-hhi', maxStaleMin: 86400 }, // monthly cron; 86400min = 60d = 2x interval
|
||
recoveryFuelStocks: { key: 'seed-meta:resilience:recovery:fuel-stocks', maxStaleMin: 86400 }, // monthly cron; 86400min = 60d = 2x interval
|
||
// PR 1 v2 energy seeds — weekly cron (8d * 1440 = 11520min = 2x interval).
|
||
// Listed in ON_DEMAND_KEYS below until Railway cron provisions and
|
||
// the first clean run lands; after that they graduate to the normal
|
||
// SEED_META staleness check like the recovery seeds above.
|
||
lowCarbonGeneration: { key: 'seed-meta:resilience:low-carbon-generation', maxStaleMin: 11520 },
|
||
fossilElectricityShare: { key: 'seed-meta:resilience:fossil-electricity-share', maxStaleMin: 11520 },
|
||
powerLosses: { key: 'seed-meta:resilience:power-losses', maxStaleMin: 11520 },
|
||
};
|
||
|
||
// Standalone keys that are populated on-demand by RPC handlers (not seeds).
|
||
// Empty = WARN not CRIT since they only exist after first request.
|
||
const ON_DEMAND_KEYS = new Set([
|
||
'riskScoresLive',
|
||
'usniFleetStale', 'positiveEventsLive',
|
||
'bisPolicy', 'bisExchange', 'bisCredit',
|
||
// bisDsr/bisPropertyResidential/bisPropertyCommercial have dedicated SEED_META
|
||
// entries (seed-bis-extended.mjs), so they are not on-demand.
|
||
'macroSignals', 'shippingRates', 'chokepoints', 'minerals', 'giving',
|
||
'cyberThreatsRpc', 'militaryBases', 'temporalAnomalies', 'displacement',
|
||
'corridorrisk', // intermediate key; data flows through transit-summaries:v1
|
||
'serviceStatuses', // RPC-populated; seed-meta written on fresh fetch only, goes stale between visits
|
||
'militaryForecastInputs', // intermediate seed-to-seed pipeline key; only populated after seed-military-flights runs
|
||
'marketImplications', // LLM-generated inside forecast cron; can fail silently on LLM errors — degrade to WARN not CRIT
|
||
'simulationPackageLatest', // written by writeSimulationPackage after deep forecast runs; only present after first successful deep run
|
||
'simulationOutcomeLatest', // written by writeSimulationOutcome after simulation runs; only present after first successful simulation
|
||
'newsThreatSummary', // relay classify loop — only written when mergedByCountry has entries; absent on quiet news periods
|
||
'resilienceRanking', // on-demand RPC cache populated after ranking requests; missing before first Pro use is expected
|
||
'recoveryFiscalSpace', 'recoveryReserveAdequacy', 'recoveryExternalDebt',
|
||
'recoveryImportHhi', 'recoveryFuelStocks', // recovery pillar: stub seeders not yet deployed, keys may be absent
|
||
// PR 1 v2 energy-construct seeds. TRANSITIONAL: the three seeders
|
||
// ship with their health registry rows in this PR but Railway cron
|
||
// is provisioned as a follow-up action. Gated as on-demand until
|
||
// the first clean run lands; graduate out of this set after ~7 days
|
||
// of successful production cron runs (verify via
|
||
// `seed-meta:resilience:{low-carbon-generation,fossil-electricity-share,power-losses}.fetchedAt`).
|
||
'lowCarbonGeneration', 'fossilElectricityShare', 'powerLosses',
|
||
'displacementPrev', // covered by cascade onto current-year displacement; empty most of the year
|
||
'fxYoy', // TRANSITIONAL (PR #3071): seed-fx-yoy Railway cron deployed manually after merge —
|
||
// gate as on-demand so a deploy-order race or first-cron-run failure doesn't
|
||
// fire a CRIT health alarm. Remove from this set after ~7 days of clean
|
||
// production cron runs (verify via `seed-meta:economic:fx-yoy.fetchedAt`).
|
||
'hyperliquidFlow', // TRANSITIONAL: seed-hyperliquid-flow runs inside seed-bundle-market-backup on
|
||
// Railway; gate as on-demand so initial deploy-order race or first cold-start
|
||
// snapshot doesn't CRIT. Remove after ~7 days of clean production cron runs.
|
||
'chokepointFlowsRelayHeartbeat', // TRANSITIONAL (PR #3133): ais-relay.cjs writes this on the
|
||
// first successful child exit after a deploy. Vercel deploys
|
||
// api/health.js instantly, but Railway rebuild + 6h initial
|
||
// loop interval means the key is absent for up to ~6h post-merge.
|
||
// Gate as on-demand so the deploy window doesn't CRIT. Remove
|
||
// after ~7 days of clean production runs (verify via
|
||
// `relay:heartbeat:chokepoint-flows.fetchedAt`).
|
||
'climateNewsRelayHeartbeat', // TRANSITIONAL (PR #3133): same deploy-order rationale.
|
||
// 30min initial loop, so window is shorter but still present.
|
||
// Remove after ~7 days alongside the chokepoint-flows entry.
|
||
'eiaPetroleum', // TRANSITIONAL: gold-standard migration of /api/eia/petroleum
|
||
// from live Vercel fetch to Redis-reader (seed-bundle-energy-sources
|
||
// daily cron). SEED_META entry above enforces 72h staleness — this
|
||
// ON_DEMAND gate only softens the absent-on-deploy case (Vercel
|
||
// deploys instantly; Railway EIA_API_KEY + first daily tick ~24h
|
||
// behind). STALE_SEED still fires if data goes stale after first seed.
|
||
// Remove from this set after ~7 days of clean cron runs so
|
||
// never-provisioned Railway promotes EMPTY_ON_DEMAND → EMPTY (CRIT).
|
||
]);
|
||
|
||
// Keys where 0 records is a valid healthy state (e.g. no airports closed,
|
||
// no earnings events this week, econ calendar quiet between seasons).
|
||
// The key must still exist in Redis; only the record count can be 0.
|
||
const EMPTY_DATA_OK_KEYS = new Set([
|
||
'notamClosures', 'faaDelays', 'gpsjam', 'positiveGeoEvents', 'weatherAlerts',
|
||
'earningsCalendar', 'econCalendar', 'cotPositioning',
|
||
'usniFleet', // usniFleetStale covers the fallback; relay outages → WARN not CRIT
|
||
'newsThreatSummary', // only written when classify produces country matches; quiet news periods = 0 countries, no write
|
||
'recoveryFiscalSpace',
|
||
'recoveryImportHhi', 'recoveryFuelStocks', // recovery pillar seeds: stub seeders write empty payloads until real sources are wired
|
||
'ddosAttacks', 'trafficAnomalies', // zero events during quiet periods is valid, not critical
|
||
'resilienceStaticFao', // empty aggregate = no IPC Phase 3+ countries this year (possible in theory); the key must exist but count=0 is fine
|
||
'cableHealth', // `cables: {}` = no active subsea cable disruptions per NGA NAVAREA warnings — all cables implicitly healthy. Also covers NGA-upstream-down windows where get-cable-health writes back the fallback response (empty cables); without this, those would alarm EMPTY_DATA.
|
||
]);
|
||
|
||
// Cascade groups: if any key in the group has data, all empty siblings are OK.
|
||
// Theater posture uses live → stale → backup fallback chain.
|
||
const CASCADE_GROUPS = {
|
||
theaterPosture: ['theaterPosture', 'theaterPostureLive', 'theaterPostureBackup'],
|
||
theaterPostureLive: ['theaterPosture', 'theaterPostureLive', 'theaterPostureBackup'],
|
||
theaterPostureBackup: ['theaterPosture', 'theaterPostureLive', 'theaterPostureBackup'],
|
||
militaryFlights: ['militaryFlights', 'militaryFlightsStale'],
|
||
militaryFlightsStale: ['militaryFlights', 'militaryFlightsStale'],
|
||
// Displacement key embeds UTC year — on Jan 1 the new-year key may be empty
|
||
// for hours until the seed runs. Cascade onto the previous-year snapshot.
|
||
displacement: ['displacement', 'displacementPrev'],
|
||
displacementPrev: ['displacement', 'displacementPrev'],
|
||
};
|
||
|
||
|
||
const NEG_SENTINEL = '__WM_NEG__';
|
||
|
||
|
||
function parseRedisValue(raw) {
|
||
if (!raw || raw === NEG_SENTINEL) return null;
|
||
try { return JSON.parse(raw); } catch { return raw; }
|
||
}
|
||
|
||
// Real data is always >0 bytes. The negative-cache sentinel is exactly
|
||
// NEG_SENTINEL.length bytes (10), so any strlen > 0 that is NOT exactly that
|
||
// length counts as data. The previous `> 10` heuristic misclassified
|
||
// legitimately small payloads (`{}`, `[]`, `0`) as missing.
|
||
function strlenIsData(strlen) {
|
||
return strlen > 0 && strlen !== NEG_SENTINEL.length;
|
||
}
|
||
|
||
function readSeedMeta(seedCfg, keyMetaValues, keyMetaErrors, now) {
|
||
if (!seedCfg) {
|
||
return { seedAge: null, seedStale: null, seedError: false, metaReadFailed: false, metaCount: null };
|
||
}
|
||
// Per-command Redis errors on the GET seed-meta half of the pipeline must
|
||
// not silently fall through to STALE_SEED — promote to REDIS_PARTIAL.
|
||
if (keyMetaErrors.get(seedCfg.key)) {
|
||
return { seedAge: null, seedStale: null, seedError: false, metaReadFailed: true, metaCount: null };
|
||
}
|
||
// Unwrap through the envelope helper. Legacy seed-meta is a bare
|
||
// `{ fetchedAt, recordCount, sourceVersion, status? }` object with no `_seed`
|
||
// wrapper, so `unwrapEnvelope` returns it as `.data` unchanged. PR 2 wires
|
||
// true envelope reads at the canonical-key layer; this import establishes
|
||
// the dependency so behavior stays byte-identical in PR 1.
|
||
const meta = unwrapEnvelope(parseRedisValue(keyMetaValues.get(seedCfg.key))).data;
|
||
if (meta?.status === 'error') {
|
||
return { seedAge: null, seedStale: true, seedError: true, metaReadFailed: false, metaCount: null };
|
||
}
|
||
let seedAge = null;
|
||
let seedStale = true;
|
||
if (meta?.fetchedAt) {
|
||
seedAge = Math.round((now - meta.fetchedAt) / 60_000);
|
||
seedStale = seedAge > seedCfg.maxStaleMin;
|
||
}
|
||
const metaCount = meta?.count ?? meta?.recordCount ?? null;
|
||
return { seedAge, seedStale, seedError: false, metaReadFailed: false, metaCount };
|
||
}
|
||
|
||
function isCascadeCovered(name, hasData, keyStrens, keyErrors) {
|
||
const siblings = CASCADE_GROUPS[name];
|
||
if (!siblings || hasData) return false;
|
||
for (const sibling of siblings) {
|
||
if (sibling === name) continue;
|
||
const sibKey = STANDALONE_KEYS[sibling] ?? BOOTSTRAP_KEYS[sibling];
|
||
if (!sibKey) continue;
|
||
if (keyErrors.get(sibKey)) continue;
|
||
if (strlenIsData(keyStrens.get(sibKey) ?? 0)) return true;
|
||
}
|
||
return false;
|
||
}
|
||
|
||
function classifyKey(name, redisKey, opts, ctx) {
|
||
const { keyStrens, keyErrors, keyMetaValues, keyMetaErrors, now } = ctx;
|
||
const seedCfg = SEED_META[name];
|
||
const isOnDemand = !!opts.allowOnDemand && ON_DEMAND_KEYS.has(name);
|
||
|
||
const meta = readSeedMeta(seedCfg, keyMetaValues, keyMetaErrors, now);
|
||
|
||
// Per-command Redis errors (data STRLEN or seed-meta GET) propagate as their
|
||
// own bucket — don't conflate with "key missing", since ops needs to know if
|
||
// the read itself failed.
|
||
if (keyErrors.get(redisKey) || meta.metaReadFailed) {
|
||
const entry = { status: 'REDIS_PARTIAL', records: null };
|
||
if (seedCfg) entry.maxStaleMin = seedCfg.maxStaleMin;
|
||
return entry;
|
||
}
|
||
|
||
const strlen = keyStrens.get(redisKey) ?? 0;
|
||
const hasData = strlenIsData(strlen);
|
||
const { seedAge, seedStale, seedError, metaCount } = meta;
|
||
|
||
// When the data key is gone the meta count is meaningless; force records=0
|
||
// so we never display the contradictory "EMPTY records=N>0" pair (item 1).
|
||
const records = hasData ? (metaCount ?? 1) : 0;
|
||
const cascadeCovered = isCascadeCovered(name, hasData, keyStrens, keyErrors);
|
||
|
||
let status;
|
||
if (seedError) status = 'SEED_ERROR';
|
||
else if (!hasData) {
|
||
if (cascadeCovered) status = 'OK_CASCADE';
|
||
else if (EMPTY_DATA_OK_KEYS.has(name)) status = seedStale === true ? 'STALE_SEED' : 'OK';
|
||
else if (isOnDemand) status = 'EMPTY_ON_DEMAND';
|
||
else status = 'EMPTY';
|
||
} else if (records === 0) {
|
||
// hasData is true in this branch, so cascade can never apply (isCascadeCovered
|
||
// short-circuits when hasData=true). Cascade only shields wholly absent keys.
|
||
if (EMPTY_DATA_OK_KEYS.has(name)) status = seedStale === true ? 'STALE_SEED' : 'OK';
|
||
else if (isOnDemand) status = 'EMPTY_ON_DEMAND';
|
||
else status = 'EMPTY_DATA';
|
||
} else if (seedStale === true) status = 'STALE_SEED';
|
||
// Coverage threshold: producers that know their canonical shape size can
|
||
// declare minRecordCount. When the writer reports a count below threshold
|
||
// (e.g., 10/13 chokepoints because portwatch dropped some), this degrades
|
||
// to COVERAGE_PARTIAL (warn) instead of reporting OK. Producer must write
|
||
// seed-meta.recordCount using the *covered* count, not the shape size.
|
||
else if (seedCfg?.minRecordCount != null && records < seedCfg.minRecordCount) status = 'COVERAGE_PARTIAL';
|
||
else status = 'OK';
|
||
|
||
const entry = { status, records };
|
||
if (seedAge !== null) entry.seedAgeMin = seedAge;
|
||
if (seedCfg) entry.maxStaleMin = seedCfg.maxStaleMin;
|
||
if (seedCfg?.minRecordCount != null) entry.minRecordCount = seedCfg.minRecordCount;
|
||
return entry;
|
||
}
|
||
|
||
const STATUS_COUNTS = {
|
||
OK: 'ok',
|
||
OK_CASCADE: 'ok',
|
||
STALE_SEED: 'warn',
|
||
SEED_ERROR: 'warn',
|
||
EMPTY_ON_DEMAND: 'warn',
|
||
REDIS_PARTIAL: 'warn',
|
||
COVERAGE_PARTIAL: 'warn',
|
||
EMPTY: 'crit',
|
||
EMPTY_DATA: 'crit',
|
||
};
|
||
|
||
export default async function handler(req, ctx) {
|
||
const headers = {
|
||
'Content-Type': 'application/json',
|
||
'Cache-Control': 'private, no-store, max-age=0',
|
||
'CDN-Cache-Control': 'no-store',
|
||
'CF-Cache-Status': 'BYPASS',
|
||
'Access-Control-Allow-Origin': '*',
|
||
};
|
||
|
||
if (req.method === 'OPTIONS') {
|
||
return new Response(null, { status: 204, headers });
|
||
}
|
||
|
||
const now = Date.now();
|
||
|
||
const allDataKeys = [
|
||
...Object.values(BOOTSTRAP_KEYS),
|
||
...Object.values(STANDALONE_KEYS),
|
||
];
|
||
const allMetaKeys = Object.values(SEED_META).map(s => s.key);
|
||
|
||
// STRLEN for data keys avoids loading large blobs into memory (OOM prevention).
|
||
// NEG_SENTINEL ('__WM_NEG__') is 10 bytes — strlenIsData() rejects exactly
|
||
// that length while accepting any other non-zero strlen as data.
|
||
let results;
|
||
try {
|
||
const commands = [
|
||
...allDataKeys.map(k => ['STRLEN', k]),
|
||
...allMetaKeys.map(k => ['GET', k]),
|
||
];
|
||
if (!getRedisCredentials()) throw new Error('Redis not configured');
|
||
results = await redisPipeline(commands, 8_000);
|
||
if (!results) throw new Error('Redis request failed');
|
||
} catch (err) {
|
||
return jsonResponse({
|
||
status: 'REDIS_DOWN',
|
||
error: err.message,
|
||
checkedAt: new Date(now).toISOString(),
|
||
}, 200, headers);
|
||
}
|
||
|
||
// keyStrens: byte length per data key (0 = missing/empty/sentinel)
|
||
// keyErrors: per-command Redis errors so we can surface REDIS_PARTIAL
|
||
const keyStrens = new Map();
|
||
const keyErrors = new Map();
|
||
for (let i = 0; i < allDataKeys.length; i++) {
|
||
const r = results[i];
|
||
if (r?.error) keyErrors.set(allDataKeys[i], r.error);
|
||
keyStrens.set(allDataKeys[i], r?.result ?? 0);
|
||
}
|
||
// keyMetaValues: parsed seed-meta objects (GET, small payloads)
|
||
// keyMetaErrors: per-command errors so a single GET failure surfaces as
|
||
// REDIS_PARTIAL instead of silently degrading to STALE_SEED.
|
||
const keyMetaValues = new Map();
|
||
const keyMetaErrors = new Map();
|
||
for (let i = 0; i < allMetaKeys.length; i++) {
|
||
const r = results[allDataKeys.length + i];
|
||
if (r?.error) keyMetaErrors.set(allMetaKeys[i], r.error);
|
||
keyMetaValues.set(allMetaKeys[i], r?.result ?? null);
|
||
}
|
||
|
||
const classifyCtx = { keyStrens, keyErrors, keyMetaValues, keyMetaErrors, now };
|
||
const checks = {};
|
||
const counts = { ok: 0, warn: 0, onDemandWarn: 0, crit: 0 };
|
||
let totalChecks = 0;
|
||
|
||
const sources = [
|
||
[BOOTSTRAP_KEYS, { allowOnDemand: false }],
|
||
[STANDALONE_KEYS, { allowOnDemand: true }],
|
||
];
|
||
for (const [registry, opts] of sources) {
|
||
for (const [name, redisKey] of Object.entries(registry)) {
|
||
totalChecks++;
|
||
const entry = classifyKey(name, redisKey, opts, classifyCtx);
|
||
checks[name] = entry;
|
||
const bucket = STATUS_COUNTS[entry.status] ?? 'warn';
|
||
counts[bucket]++;
|
||
if (entry.status === 'EMPTY_ON_DEMAND') counts.onDemandWarn++;
|
||
}
|
||
}
|
||
|
||
// On-demand keys that simply haven't been requested yet should not flip
|
||
// overall to WARNING — they're warn-level only for visibility.
|
||
const realWarnCount = counts.warn - counts.onDemandWarn;
|
||
const critCount = counts.crit;
|
||
|
||
let overall;
|
||
if (critCount === 0 && realWarnCount === 0) overall = 'HEALTHY';
|
||
else if (critCount === 0) overall = 'WARNING';
|
||
// Degraded threshold scales with registry size so adding keys doesn't
|
||
// silently raise the page-out bar. ~3% of total keys (was hardcoded 3).
|
||
else if (critCount / totalChecks <= 0.03) overall = 'DEGRADED';
|
||
else overall = 'UNHEALTHY';
|
||
|
||
const httpStatus = 200;
|
||
|
||
if (overall !== 'HEALTHY') {
|
||
// problemKeys includes seedAgeMin for the snapshot (useful for post-mortem),
|
||
// but the dedupe signature uses only key:status (no age) so a long STALE_SEED
|
||
// window doesn't produce a new log entry on every poll.
|
||
const problemKeys = Object.entries(checks)
|
||
.filter(([, c]) => c.status !== 'OK' && c.status !== 'OK_CASCADE' && c.status !== 'EMPTY_ON_DEMAND')
|
||
.map(([k, c]) => `${k}:${c.status}${c.seedAgeMin != null ? `(${c.seedAgeMin}min)` : ''}`);
|
||
const sigKeys = Object.entries(checks)
|
||
.filter(([, c]) => c.status !== 'OK' && c.status !== 'OK_CASCADE' && c.status !== 'EMPTY_ON_DEMAND')
|
||
.map(([k, c]) => `${k}:${c.status}`)
|
||
.sort();
|
||
console.log('[health] %s problems=[%s]', overall, problemKeys.join(', '));
|
||
const snapshot = {
|
||
at: new Date(now).toISOString(),
|
||
status: overall,
|
||
critCount,
|
||
warnCount: realWarnCount,
|
||
problems: problemKeys,
|
||
};
|
||
// Dedupe: only LPUSH when the incident signature (status + problem set,
|
||
// excluding seedAgeMin) changes. Read the previous sig first, then write
|
||
// everything (last-failure + sig + LPUSH) in one atomic pipeline so the
|
||
// sig only advances when the LPUSH succeeds. If the pipeline fails, the
|
||
// sig stays stale and the next poll retries the append.
|
||
const sig = `${overall}|${sigKeys.join(',')}`;
|
||
const prevSigResult = await redisPipeline([['GET', 'health:failure-log-sig']], 4_000).catch(() => null);
|
||
const prevSig = prevSigResult?.[0]?.result ?? '';
|
||
const persistCmds = [
|
||
['SET', 'health:last-failure', JSON.stringify(snapshot), 'EX', 86400],
|
||
];
|
||
if (sig !== prevSig) {
|
||
persistCmds.push(
|
||
['LPUSH', 'health:failure-log', JSON.stringify(snapshot)],
|
||
['LTRIM', 'health:failure-log', 0, 49],
|
||
['EXPIRE', 'health:failure-log', 86400 * 7],
|
||
['SET', 'health:failure-log-sig', sig, 'EX', 86400],
|
||
);
|
||
}
|
||
const persist = redisPipeline(persistCmds, 4_000).catch(() => {});
|
||
if (ctx && typeof ctx.waitUntil === 'function') ctx.waitUntil(persist);
|
||
} else {
|
||
// Clear the sig on recovery so a recurrence of the same problem set
|
||
// after a healthy gap is logged as a new incident, not deduped against
|
||
// the previous one.
|
||
const clear = redisPipeline([['DEL', 'health:failure-log-sig']], 4_000).catch(() => {});
|
||
if (ctx && typeof ctx.waitUntil === 'function') ctx.waitUntil(clear);
|
||
}
|
||
|
||
const url = new URL(req.url);
|
||
const compact = url.searchParams.get('compact') === '1';
|
||
|
||
const body = {
|
||
status: overall,
|
||
summary: {
|
||
total: totalChecks,
|
||
ok: counts.ok,
|
||
// `warn` excludes on-demand-empty (cosmetic warns); `onDemandWarn` is
|
||
// surfaced separately so readers can reconcile against `overall`.
|
||
warn: realWarnCount,
|
||
onDemandWarn: counts.onDemandWarn,
|
||
crit: critCount,
|
||
},
|
||
checkedAt: new Date(now).toISOString(),
|
||
};
|
||
|
||
if (!compact) {
|
||
body.checks = checks;
|
||
} else {
|
||
const problems = {};
|
||
for (const [name, check] of Object.entries(checks)) {
|
||
if (check.status !== 'OK' && check.status !== 'OK_CASCADE') problems[name] = check;
|
||
}
|
||
if (Object.keys(problems).length > 0) body.problems = problems;
|
||
}
|
||
|
||
return new Response(JSON.stringify(body, null, compact ? 0 : 2), {
|
||
status: httpStatus,
|
||
headers,
|
||
});
|
||
}
|