mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
main
22 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
84ee2beb3e |
feat(energy): Energy Atlas end-to-end — pipelines + storage + shortages + disruptions + country drill-down (#3294)
* feat(energy): pipeline registries (gas + oil) — evidence-based schema
Day 6 of the Energy Atlas Release 1 plan (Week 2). First curated asset
registry for the atlas — the real gap vs GEF.
## Curated data (critical assets only, not global completeness)
scripts/data/pipelines-gas.json — 12 critical gas lines:
Nord Stream 1/2 (offline; Swedish EEZ sabotage 2022; EU sanctions refs),
TurkStream, Yamal–Europe (offline; Polish counter-sanctions),
Brotherhood/Soyuz (offline; Ukraine transit expired 2024-12-31),
Power of Siberia, Dolphin, Medgaz, TAP, TANAP,
Central Asia–China, Langeled.
scripts/data/pipelines-oil.json — 12 critical oil lines:
Druzhba North/South (N offline per EU 2022/879; S under landlocked
derogation), CPC, ESPO (+ price-cap sanction ref), BTC, TAPS,
Habshan–Fujairah (Hormuz bypass), Keystone, Kirkuk–Ceyhan (offline
since 2023 ICC ruling), Baku–Supsa, Trans-Mountain (TMX expansion
May 2024), ESPO spur to Daqing.
Scope note: 75+ each is Week 2b work via GEM bulk import. Today's cut
is curated from first-hand operator disclosures + regulator filings so
I can stand behind every evidence field.
## Evidence-based schema (not conclusion labels)
Per docs/methodology/pipelines.mdx: no bare `sanctions_blocked` field.
Every pipeline carries an evidence bundle with `physicalState`,
`physicalStateSource`, `operatorStatement`, `commercialState`,
`sanctionRefs[]`, `lastEvidenceUpdate`, `classifierVersion`,
`classifierConfidence`. The public badge (`flowing|reduced|offline|
disputed`) is derived server-side from this bundle at read time.
## Seeder
scripts/seed-pipelines.mjs — single process publishes BOTH keys
(energy:pipelines:{gas,oil}:v1) via two runSeed() calls. Tiny datasets
(<20KB each) so co-location is cheap and guarantees classifierVersion
consistency.
Conventions followed (worldmonitor-bootstrap-registration skill):
- TTL 21d = 3× weekly cadence (gold-standard per
feedback_seeder_gold_standard.md)
- maxStaleMin 20_160 = 2× cadence (health-maxstalemin-write-cadence skill)
- sourceVersion + schemaVersion + recordCount + declareRecords wired
(seed-contract-foundation)
- Zero-case explicitly NOT allowed — MIN_PIPELINES_PER_REGISTRY=8 floor
## Health registration (dual, per feedback_two_health_endpoints_must_match)
- api/health.js: BOOTSTRAP_KEYS adds pipelinesGas + pipelinesOil;
SEED_META adds both with maxStaleMin=20_160.
- api/seed-health.js: mirror entries with intervalMin=10_080 (maxStaleMin/2).
## Bundle registration
scripts/seed-bundle-energy-sources.mjs adds a single Pipelines entry
(not two) because seed-pipelines.mjs publishes both keys in one run —
listing oil separately would double-execute. Monitoring of the oil key
staleness happens in api/health.js instead.
## Tests (tests/pipelines-registry.test.mts)
17 passing node:test assertions covering:
- Schema validation (both registries pass validateRegistry)
- Identity resolution (no id collisions, id matches object key)
- Country ISO2 normalization (from/to/transit all match /^[A-Z]{2}$/)
- Endpoint geometry within Earth bounds
- Evidence rigor: non-flowing badges require at least one supporting
evidence source (operator statement / sanctionRefs / ais-relay /
satellite / press)
- ClassifierConfidence in 0..1
- Commodity/capacity pairing (gas uses capacityBcmYr, oil uses
capacityMbd — mixing = test fail)
- validateRegistry rejects: empty object, null, no-evidence fixtures,
below-floor counts
Typecheck clean (both tsconfig.json and tsconfig.api.json).
Next: Day 7 will add list-pipelines / get-pipeline-detail RPCs in
supply-chain/v1. Day 8 ships PipelineStatusPanel with DeckGL PathLayer
consuming the registry.
* fix(energy): split seed-pipelines.mjs into two entry points — runSeed hard-exits
High finding from PR review. scripts/seed-pipelines.mjs called runSeed()
twice in one process and awaited Promise.all. But runSeed() in
scripts/_seed-utils.mjs hard-exits via process.exit on ~9 terminal paths
(lines 816, 820, 839, 888, 917, 989, plus fetch-retry 946, fatal 859,
skipped-lock 81). The first runSeed to reach any terminal path exits the
entire node process, so the second runSeed's resolve never fires — only
one of energy:pipelines:{gas,oil}:v1 would ever be written.
Since the bundle scheduled seed-pipelines.mjs exactly once, and both
api/health.js and api/seed-health.js expect both keys populated, the
other registry would stay permanently EMPTY/STALE after deploy.
Fix: split into two entry-point scripts around a shared utility.
- scripts/_pipeline-registry.mjs (NEW, was seed-pipelines.mjs) — shared
helpers ONLY. Exports GAS_CANONICAL_KEY, OIL_CANONICAL_KEY,
PIPELINES_TTL_SECONDS, MAX_STALE_MIN, buildGasPayload, buildOilPayload,
validateRegistry, recordCount, declareRecords. Underscore prefix marks
it as non-entry-point (matches _seed-utils.mjs / _seed-envelope-source.mjs
convention).
- scripts/seed-pipelines-gas.mjs (NEW) — imports from the shared module,
single runSeed('energy','pipelines-gas',…) call.
- scripts/seed-pipelines-oil.mjs (NEW) — same shape, oil.
- scripts/seed-bundle-energy-sources.mjs — register BOTH seeders (not one).
- scripts/seed-pipelines.mjs — deleted.
- tests/pipelines-registry.test.mts — update import path to the shared
module. All 17 tests still pass.
Typecheck clean (both configs). Tests pass. No other consumers import
from the deleted script.
* fix(energy): complete pipeline bootstrap registration per 4-file checklist
High finding from PR review. My earlier PR description claimed
worldmonitor-bootstrap-registration was complete, but I only touched two
of the four registries (api/health.js + api/seed-health.js). The bootstrap
hydration payload itself (api/bootstrap.js) and the shared cache-keys
registry (server/_shared/cache-keys.ts) still had no entry for either
pipeline key, so any consumer that reads bootstrap data would see
pipelinesGas/pipelinesOil as missing on first load.
Files updated this commit:
- api/bootstrap.js — KEYS map + SLOW_KEYS set both gain pipelinesGas +
pipelinesOil. Placed next to sprPolicies (same curated-registry cadence
and tier). Slow tier is correct: weekly cron, not needed on first paint.
- server/_shared/cache-keys.ts — PIPELINES_GAS_KEY + PIPELINES_OIL_KEY
exported constants (matches SPR_POLICIES_KEY pattern), BOOTSTRAP_KEYS map
entries, and BOOTSTRAP_TIERS entries (both 'slow').
Not touched (intentional):
- server/gateway.ts — pipeline data is free-tier per the Energy Atlas
plan; no PREMIUM_RPC_PATHS entry required. Energy Atlas monetization
hooks (scenario runner, MCP tools, subscriptions) are Release 2.
Full 4-file checklist now complete:
✅ server/_shared/cache-keys.ts (this commit)
✅ api/bootstrap.js (this commit)
✅ api/health.js (earlier in PR)
✅ api/seed-health.js (earlier in PR — dual-registry rule)
Typecheck clean (both configs).
* feat(energy): ListPipelines + GetPipelineDetail RPCs with evidence-derived badges
Day 7 of the Energy Atlas Release 1 plan (Week 2). Exposes the pipeline
registries (shipped in Day 6) via two supply-chain RPCs and ships the
evidence-to-badge derivation server-side.
## Proto
proto/worldmonitor/supply_chain/v1/list_pipelines.proto — new:
- ListPipelinesRequest { commodity_type?: 'gas' | 'oil' }
- ListPipelinesResponse { pipelines[], fetched_at, classifier_version, upstream_unavailable }
- GetPipelineDetailRequest { pipeline_id (required, query-param) }
- GetPipelineDetailResponse { pipeline?, revisions[], fetched_at, unavailable }
- PipelineEntry — wire shape mirroring scripts/data/pipelines-{gas,oil}.json
+ a server-derived public_badge field
- PipelineEvidence, OperatorStatement, SanctionRef, LatLon, PipelineRevisionEntry
service.proto adds both rpc methods with HTTP_METHOD_GET + path bindings:
/api/supply-chain/v1/list-pipelines
/api/supply-chain/v1/get-pipeline-detail
`make generate` regenerated src/generated/{client,server}/… + docs/api/
OpenAPI json/yaml.
## Evidence-derivation
server/worldmonitor/supply-chain/v1/_pipeline-evidence.ts — new.
derivePublicBadge(evidence) → 'flowing' | 'reduced' | 'offline' | 'disputed'
is deterministic + versioned (DERIVER_VERSION='badge-deriver-v1').
Rules (first match wins):
1. offline + sanctionRef OR expired/suspended commercial → offline
2. offline + operator statement → offline
3. offline + only press/ais/satellite → disputed (single-source negative claim)
4. reduced → reduced
5. flowing → flowing
6. unknown / malformed → disputed
Staleness guard: non-flowing badges on >14d-old evidence demote to
disputed. Flowing is the optimistic default — stale "still flowing" is
safer than stale "offline". Matches seed-pipelines-{gas,oil}.mjs maxStaleMin.
Tests (tests/pipeline-evidence-derivation.test.mts) — 15 passing cases
covering happy paths, disputed fallbacks, staleness guard, versioning.
## Handlers
server/worldmonitor/supply-chain/v1/list-pipelines.ts
- Reads energy:pipelines:{gas,oil}:v1 via getCachedJson.
- projectPipeline() narrows the Upstash `unknown` into PipelineEntry
shape + calls derivePublicBadge.
- Honors commodity_type filter (skip the opposite registry's Redis read
when the client pre-filters).
- Returns upstream_unavailable=true when BOTH registries miss.
server/worldmonitor/supply-chain/v1/get-pipeline-detail.ts
- Scans both registries by id (ids are globally unique per
tests/pipelines-registry.test.mts).
- Empty revisions[] for now; auto-revision log wires up in Week 3.
handler.ts registers both into supplyChainHandler.
## Gateway
server/gateway.ts adds 'static' cache-tier for both new RPC paths
(registry is slow-moving; 'static' matches the other read-mostly
supply-chain endpoints).
## Consumer wiring
Not in this commit — PipelineStatusPanel (Day 8) is what will call
listPipelines/getPipelineDetail via the generated client. pipelinesGas
+ pipelinesOil stay in PENDING_CONSUMERS until Day 8.
Typecheck clean (both configs). 15 new tests + 17 registry tests all pass.
* feat(energy): PipelineStatusPanel — evidence-backed status table + drawer
Day 8 of the Energy Atlas Release 1 plan. First consumer of the Day 6–7
registries + RPCs.
## What this PR adds
- src/components/PipelineStatusPanel.ts — new panel (id=pipeline-status).
* Bootstrap-hydrates from pipelinesGas + pipelinesOil for instant first
paint; falls through to listPipelines() RPC if bootstrap misses.
Background re-fetch runs on every render so a classifier-version bump
between bootstrap stamp and first view produces a visible update.
* Table rows sorted non-flowing-first (offline / reduced / disputed
before flowing) — what an atlas reader cares about.
* Click-to-expand drawer calls getPipelineDetail() lazily — operator
statements, sanction refs (with clickable source URLs), commercial
state, classifier version + confidence %, capacity + route metadata.
* publicBadge color-chip palette matches the methodology doc.
* Attribution footer with GEM (CC-BY 4.0) credit + classifier version.
- src/components/index.ts — barrel export.
- src/app/panel-layout.ts — import + createPanel('pipeline-status', …).
- src/config/panels.ts — ENERGY_PANELS adds 'pipeline-status' at priority 1.
## PENDING_CONSUMERS cleanup
tests/bootstrap.test.mjs — removes 'pipelinesGas' + 'pipelinesOil' from
the allowlist. The invariant "every bootstrap key has a getHydratedData
consumer" now enforces real wiring for these keys: the panel literally
calls getHydratedData('pipelinesGas') and getHydratedData('pipelinesOil').
Future regressions that remove the consumer will fail pre-push.
## Consumer contract verified
- 67 tests pass including bootstrap.test.mjs consumer coverage check.
- Typecheck clean.
- No DeckGL PathLayer in this commit — existing 'pipelines-layer' has a
separate data source, so modifying DeckGLMap.ts to overlay evidence-
derived badges on the map is a follow-up commit to avoid clobbering.
## Out of scope for Day 8 (next steps on same PR)
- DeckGL PathLayer integration (color pipelines on the main map by
publicBadge, click-to-open this drawer) — Day 8b commit.
- Storage facility registry + StorageFacilityMapPanel — Days 9-10.
* fix(energy): PipelineStatusPanel bootstrap path — client-side badge derivation
High finding from PR review. The Day-8 panel crashed on first paint
whenever bootstrap hydration succeeded, because:
- Bootstrap hydrates raw scripts/data/pipelines-{gas,oil}.json verbatim.
- That JSON does NOT include publicBadge — that field is only added by
the server handler's projectPipeline() in list-pipelines.ts.
- PipelineStatusPanel passed raw entries into badgeChip(), which called
badgeLabel(undefined).charAt(0) → TypeError.
The background RPC refresh that would have repaired the data never ran
because the panel threw before reaching it. So the exact bootstrap path
newly wired in commit
|
||
|
Elie Habib
|
cd5ed0d183 |
feat(seeds): BIS DSR + property prices (2 of 7) (#3048)
* feat(seeds): BIS DSR + property prices (2 of 7) Ships 2 of 7 BIS dataflows flagged as genuinely new signals in #3026 — the rest are redundant with IMF/WB or are low-fit global aggregates. New seeder: scripts/seed-bis-extended.mjs - WS_DSR household debt service ratio (% income, quarterly) - WS_SPP residential property prices (real index, quarterly) - WS_CPP commercial property prices (real index, quarterly) Gold-standard pattern: atomic publish + writeExtraKey for extras, retry on missing startPeriod, TTL = 3 days (3× 12h cron), runSeed drives seed-meta:economic:bis-extended. Series selection scores dimension matches (PP_VALUATION=R / UNIT_MEASURE=628 for property, DSR_BORROWERS=P / DSR_ADJUST=A for DSR), then falls back to observation count. Wired into: - bootstrap (slow tier) + cache-keys.ts - api/health.js (STANDALONE_KEYS + SEED_META, maxStaleMin = 24h) - api/mcp.ts get_economic_data tool (_cacheKeys + _freshnessChecks) - resilience macroFiscal: new householdDebtService sub-metric (weight 0.05, currentAccountPct rebalanced 0.3 → 0.25) - Housing Cycle tile on CountryDeepDivePanel (Economic Indicators card) with euro-area (XM) fallback for EU member states - seed-bundle-macro Railway cron (BIS-Extended, 12h interval) Tests: tests/bis-extended-seed.test.mjs covers CSV parsing, series selection, quarter math + YoY. Updated resilience golden-value tests for the macroFiscal weight rebalance. Closes #3026 https://claude.ai/code/session_01DDo39mPD9N2fNHtUntHDqN * fix(resilience): unblock PR #3048 on #3046 stack - rebase onto #3046; final macroFiscal weights: govRevenue 0.40, currentAccount 0.20, debtGrowth 0.20, unemployment 0.15, householdDebtService 0.05 (=1.00) - add updateHousingCycle? stub to CountryBriefPanel interface so country-intel dispatch typechecks - add HR to EURO_AREA fallback set (joined euro 2023-01-01) - seed-bis-extended: extend SPP/CPP TTLs when DSR fetch returns empty so the rejected publish does not silently expire the still-good property keys - update resilience goldens for the 5-sub-metric macroFiscal blend * fix(country-brief): housing tile renders em-dash for null change values The new Housing cycle tile used `?? 0` to default qoqChange/yoyChange/change when missing, fabricating a flat "0.0%" label (with positive-trend styling) for countries with no prior comparable period. Fetch path and builders correctly return null; the panel was coercing it. formatPctTrend now accepts null|undefined and returns an em-dash, matching how other cards surface unavailable metrics. Drop the `?? 0` fallbacks at the three housing call sites. * fix(seed-health): register economic:bis-extended seed-meta monitoring 12h Railway cron writes seed-meta:economic:bis-extended but it was missing from SEED_DOMAINS, so /api/seed-health never reported its freshness. intervalMin=720 matches maxStaleMin/2 (1440/2) from api/health.js. * fix(seed-bis-extended): decouple DSR/SPP/CPP so one fetch failure doesn't block the others Previously validate() required data.entries.length > 0 on the DSR slice after publishTransform pulled it out of the aggregate payload. If WS_DSR fetch failed but WS_SPP / WS_CPP succeeded, validate() rejected the publish → afterPublish() never ran → fresh SPP/CPP data was silently discarded and only the old snapshots got a TTL bump. This treats the three datasets as independent: - SPP and CPP are now published (or have their existing TTLs extended) as side-effects of fetchAll(), per-dataset. A failure in one never affects the others. - DSR continues to flow through runSeed's canonical-key path. When DSR is empty, publishTransform yields { entries: [] } so atomicPublish skips the canonical write (preserving the old DSR snapshot); runSeed's skipped branch extends its TTL and refreshes seed-meta. Shape B (one runSeed call, semantics changed) chosen over Shape A (three sequential runSeed calls) because runSeed owns the lock + process.exit lifecycle and can't be safely called three times in a row, and Shape B keeps the single aggregate seed-meta:economic:bis-extended key that health.js already monitors. Tests cover both failure modes: - DSR empty + SPP/CPP healthy → SPP/CPP written, DSR TTL extended - DSR healthy + SPP/CPP empty → DSR written, SPP/CPP TTLs extended * fix(health): per-dataset seed-meta for BIS DSR/SPP/CPP Health was pointing bisDsr / bisPropertyResidential / bisPropertyCommercial at the shared seed-meta:economic:bis-extended key, which runSeed refreshes on every run (including its validation-failed "skipped" branch). A DSR-only outage therefore left bisDsr reporting fresh in api/health.js while the resilience scorer consumed stale/missing economic:bis:dsr:v1 data. Write a dedicated seed-meta key per dataset ONLY when that dataset actually published fresh entries. The aggregate bis-extended key stays as a "seeder ran" signal in api/seed-health.js. * fix(seed-bis-extended): write DSR seed-meta only after atomicPublish succeeds Previously fetchAll() wrote seed-meta:economic:bis-dsr inline before runSeed/atomicPublish ran. If atomicPublish then failed (Redis hiccup, validate rejection, etc.), seed-meta was already bumped — health would report DSR fresh while the canonical key was stale. Move the DSR seed-meta write into a dsrAfterPublish callback passed to runSeed via the existing afterPublish hook, which fires only after a successful canonical publish. SPP/CPP paths already used this ordering inside publishDatasetIndependently; this brings DSR in line. Adds a regression test exercising dsrAfterPublish with mocked Upstash: populated DSR -> single SET on seed-meta key; null/empty DSR -> zero Redis calls. * fix(resilience): per-dataset BIS seed-meta keys in freshness overrides SOURCE_KEY_META_OVERRIDES previously collapsed economic:bis:dsr:v1 and both property-* sourceKeys onto the aggregate seed-meta:economic:bis-extended key. api/health.js (SEED_META) writes per-dataset keys (seed-meta:economic:bis-dsr / bis-property-residential / bis-property-commercial), so a DSR-only outage showed stale in /api/health but the resilience dimension freshness code still reported macroFiscal inputs as fresh. Map each BIS sourceKey to its dedicated seed-meta key to match health.js. The aggregate bis-extended key is still written by the seeder and read by api/seed-health.js as a "seeder ran" signal, so it is retained upstream. * fix(bis): prefer households in DSR + per-dataset freshness in MCP Greptile review catches on #3048: 1. buildDsr() was selecting DSR_BORROWERS='P' (private non-financial) while the UI labels it "Household DSR" and resilience scoring uses it as `householdDebtService`. Changed to 'H' (households). Countries without an H series now get dropped rather than silently mislabeled. 2. api/mcp.ts get_economic_data still read only the aggregate seed-meta:economic:bis-extended for freshness. If DSR goes stale while SPP/CPP keep publishing, MCP would report the BIS block as fresh even though one of its returned keys is stale. Swapped to the three per-dataset seed-meta keys (bis-dsr, bis-property-residential, bis-property-commercial), matching the fix already applied to /api/health and the resilience dimension-freshness pipeline. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Elie Habib
|
f5d8ff9458 |
feat(seeds): Eurostat house prices + quarterly debt + industrial production (#3047)
* feat(seeds): Eurostat house prices + quarterly debt + industrial production Adds three new Eurostat overlay seeders covering all 27 EU members plus EA20 and EU27_2020 aggregates (issue #3028): - prc_hpi_a (annual house price index, 10y sparkline, TTL 35d) key: economic:eurostat:house-prices:v1 complements BIS WS_SPP (#3026) for the Housing cycle tile - gov_10q_ggdebt (quarterly gov debt %GDP, 8q sparkline, TTL 14d) key: economic:eurostat:gov-debt-q:v1 upgrades National Debt card cadence from annual IMF to quarterly for EU - sts_inpr_m (monthly industrial production, 12m sparkline, TTL 5d) key: economic:eurostat:industrial-production:v1 feeds "Real economy pulse" sparkline on Economic Indicators card Shared JSON-stat parser in scripts/_eurostat-utils.mjs handles the EL/GR and EA20 geo quirks and returns full time series for sparklines. Wires each seeder into bootstrap (SLOW_KEYS), health registries (keys + seed-meta thresholds matched to cadence), macro seed bundle, cache-keys shared module, and the MCP tool registry (get_eu_housing_cycle, get_eu_quarterly_gov_debt, get_eu_industrial_production). MCP tool count updated to 31. Tests cover JSON-stat parsing, sparkline ordering, EU-only coverage gating (non-EU geos return null so panels never render blank tiles), validator thresholds, and registry wiring across all surfaces. https://claude.ai/code/session_01Tgm6gG5yUMRoc2LRAKvmza * fix(bootstrap): register new Eurostat keys in tiers, defer consumers Adds eurostatHousePrices/GovDebtQ/IndProd to BOOTSTRAP_TIERS ('slow') to match SLOW_KEYS in api/bootstrap.js, and lists them as PENDING_CONSUMERS in the hydration coverage test (panel wiring lands in follow-up). * fix(eurostat): raise seeder coverage thresholds to catch partial publishes The three Eurostat overlay seeders (house prices, quarterly gov debt, monthly industrial production) all validated with makeValidator(10) against a fixed 29-geo universe (EU27 + EA20 + EU27_2020). A bad run returning only 10-15 geos would pass validation and silently publish a snapshot missing most of the EU. Raise thresholds to near-complete coverage, with a small margin for geos with patchy reporting: - house prices (annual): 10 -> 24 - gov debt (quarterly): 10 -> 24 - industrial prod (monthly): 10 -> 22 (monthly is slightly patchier) Add a guard test that asserts every overlay seeder keeps its threshold >=22 so this regression can't reappear. * fix(seed-health): register 3 Eurostat seed-meta entries house-prices, gov-debt-q, industrial-production were wired in api/health.js SEED_META but missing from api/seed-health.js SEED_DOMAINS, so /api/seed-health would not surface their freshness. intervalMin = health.js maxStaleMin / 2 per convention. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Elie Habib
|
71a6309503 |
feat(seeds): expand IMF WEO coverage — growth, labor, external themes (#3027) (#3046)
* feat(seeds): expand IMF WEO coverage — growth, labor, external themes (#3027) Adds three new SDMX-3.0 seeders alongside the existing imf-macro seeder to surface 15+ additional WEO indicators across ~210 countries at zero incremental API cost. Bundled into seed-bundle-imf-extended.mjs on the same monthly Railway cron cadence. Seeders + Redis keys: - seed-imf-growth.mjs → economic:imf:growth:v1 NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP - seed-imf-labor.mjs → economic:imf:labor:v1 LUR (unemployment), LP (population) - seed-imf-external.mjs → economic:imf:external:v1 BX, BM, BCA, TM_RPCH, TX_RPCH (+ derived trade balance) - seed-imf-macro.mjs extended with PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP All four seeders share the 35-day TTL (monthly WEO release) and ~210 country coverage via the same imfSdmxFetchIndicator helper. Wiring: - api/bootstrap.js, api/health.js, server/_shared/cache-keys.ts — register new keys, mark them slow-tier, add SEED_META freshness thresholds matching the imfMacro entry (70d = 2× monthly cadence) - server/worldmonitor/resilience/v1/_dimension-freshness.ts — override entries for the dash-vs-colon seed-meta key shape - _indicator-registry.ts — add LUR as a 4th macroFiscal sub-metric (enrichment tier, weight 0.15); rebalance govRevenuePct (0.5→0.4) and currentAccountPct (0.3→0.25) so weights still sum to 1.0 - _dimension-scorers.ts — read economic:imf:labor:v1 in scoreMacroFiscal, normalize LUR with goalposts 3% (best) → 25% (worst); null-tolerant so weightedBlend redistributes when labor data is unavailable - api/mcp.ts — new get_country_macro tool bundling all four IMF keys with a single freshness check; describes per-country fields including growth/inflation/labor/BOP for LLM-driven country screening - src/services/imf-country-data.ts — bootstrap-cached client + pure buildImfEconomicIndicators helper - src/app/country-intel.ts — async-fetch the IMF bundle on country selection and merge real GDP growth, CPI inflation, unemployment, and GDP/capita rows into the Economic Indicators card; bumps card cap from 3 → 6 rows to fit live signals + IMF context Tests: - tests/seed-imf-extended.test.mjs — 13 unit tests across the three new seeders' pure helpers (canonical keys, ISO3→ISO2 mapping, aggregate filtering, derived savings-investment gap & trade balance, validate thresholds) - tests/imf-country-data.test.mts — 6 tests for the panel rendering helper, including stagflation flag and high-unemployment trend - tests/resilience-dimension-scorers.test.mts — new LUR sub-metric test (tight vs slack labor); existing scoreMacroFiscal coverage assertions updated for the new 4-metric weight split - tests/helpers/resilience-fixtures.mts — labor fixture for NO/US/YE so the existing macroFiscal ordering test still resolves the LUR weight - tests/bootstrap.test.mjs — register imfGrowth/imfLabor/imfExternal as pending consumers (matching imfMacro) - tests/mcp.test.mjs — bump tools/list count 28 → 29 https://claude.ai/code/session_018enRzZuRqaMudKsLD5RLZv * fix(resilience): update macroFiscal goldens for LUR weight rebalance Recompute pinned fixture values after adding labor-unemployment as 4th macroFiscal sub-metric (weight rebalance in _indicator-registry). Also align seed-imf-external tradeBalance to a single reference year to avoid mixing ex/im values from different WEO vintages. * fix(seeds): tighten IMF coverage gates to reject partial snapshots IMF WEO growth/labor/external indicators report ~210 countries for healthy runs. Previous thresholds (150/100/150) let a bad IMF run overwrite a good snapshot with dozens of missing countries and still pass validation. Raise all three to >=190, matching the pattern of sibling seeders and leaving a ~20-country margin for indicators with slightly narrower reporting. Labor validator unions LUR + population (LP), so healthy coverage tracks LP (~210), not LUR (~100) — the old 100 threshold was based on a misread of the union logic. * fix(seed-health): register imf-growth/labor/external seed-meta keys Missing SEED_DOMAINS entries meant the 3 new IMF WEO seeders (growth, labor, external) had no /api/seed-health visibility. intervalMin=50400 matches health.js maxStaleMin/2 (100800/2) — same monthly WEO cadence as imf-macro. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Elie Habib
|
3696aba2d1 |
fix(infra): sync health/bootstrap/cache-keys parity (4 keys + 6 DATA_KEYS + 5 SEED_META) (#3015)
* fix(infra): sync health/bootstrap/cache-keys parity (4 BOOTSTRAP_CACHE_KEYS + 6 DATA_KEYS + 5 SEED_META) Audit found 4 bootstrap.js keys (consumerPrices*) missing from cache-keys.ts BOOTSTRAP_CACHE_KEYS, 6 bootstrapped keys invisible to health DATA_KEYS monitoring (cryptoSectors, ddosAttacks, economicStress, insights, predictions, trafficAnomalies), and 5 bootstrapped keys with no SEED_META staleness detection (cryptoSectors, ddosAttacks, economicStress, marketImplications, trafficAnomalies). Keys without seed-meta writers (bisCredit, bisExchange, giving, minerals, serviceStatuses, temporalAnomalies) were verified as on-demand/derived and correctly skipped. * fix(health): write seed-meta on empty DDoS/anomalies data Prevents false STALE_SEED alerts when Cloudflare returns zero events. Extracts writeSeedMeta() helper from writeExtraKeyWithMeta(). * fix(health): remove duplicate insights/predictions aliases, fix test regex P1: insights/predictions duplicate newsInsights/predictionMarkets P2: keyRe now captures non-versioned consumer-prices keys * fix(health): add ddosAttacks/trafficAnomalies to EMPTY_DATA_OK_KEYS Zero DDoS events or traffic anomalies is a valid quiet-period state, not a critical failure. |
||
|
Elie Habib
|
2decda6508 |
feat(wsb): add Reddit/WSB ticker scanner seeder + bootstrap registration (#2916)
* feat(wsb): add Reddit/WSB ticker scanner seeder + bootstrap registration Seeder in ais-relay.cjs fetches from r/wallstreetbets, r/stocks, r/investing every 10min. Extracts ticker mentions, validates against known ticker set, aggregates by frequency and engagement, writes top 50 to intelligence:wsb-tickers:v1. 4-file bootstrap registration: cache-keys.ts, bootstrap.js, health.js with SEED_META maxStaleMin=30. * fix(wsb): remove duplicate CEO + fix avgUpvoteRatio divisor * fix(wsb): require ticker validation set + condition seed-meta on write + add seed-health 1. Skip seed when ticker validation set is empty (cold start/bootstrap miss) 2. Only write seed-meta after successful canonical write 3. Register in api/seed-health.js for dedicated monitoring * fix(wsb): case-insensitive $ticker matching + BRK.B dotted symbol support * fix(wsb): split $-prefixed vs bare ticker extraction + BRK.B→BRK-B normalization 1. $-prefixed tickers ($nvda, $BRK.B) skip whitelist validation (strong signal) — catches GME, AMC, PLTR etc. not in the narrow market watchlist 2. Bare uppercase tokens still validated against known set (high false-positive) 3. BRK.B normalized to BRK-B before validation (dot→dash) 4. Empty known set no longer skips seed — $-prefixed tickers still extracted * fix(wsb): skip bare-uppercase branch entirely when ticker set unavailable |
||
|
Elie Habib
|
1af73975b9 |
feat(energy): SPR policy classification layer (#2881)
* feat(energy): add SPR policy classification layer with 66-country registry Static JSON registry classifying strategic petroleum reserve regimes for 66 countries (all IEA members + major producers/consumers). Integrates into energy profile handler, shock model limitations, analyst context, spine seeder, and CDP UI. - scripts/data/spr-policies.json: 66-entry registry with regime, source, asOf - scripts/seed-spr-policies.mjs: seeder following chokepoint-baselines pattern - Proto fields 51-59 on GetCountryEnergyProfileResponse - Handler reads SPR registry from Redis, populates proto fields - Shock model adds fuel-mode-gated SPR limitations for non-IEA gov SPR - Analyst context refactored to accumulator pattern (IEA + SPR parts) - CDP UI: SPR badge for non-IEA government_spr, muted text for spare_capacity - Spine integration: SPR fields in shockInputs + hasSprPolicy coverage flag - Cache keys, health, bootstrap, seed-health registrations - Tests: registry shape, ISO2, regime enum, required entries, no estimatedFillPct * fix(energy): remove SPR from bootstrap (server-only); narrow SPR hasAny gate to renderable regimes * feat(energy): render "no known SPR" risk note for countries with regime=none * fix(energy): human-readable SPR regime labels; parallelize spine+registry reads in analyst |
||
|
Elie Habib
|
bc33fe955a |
fix(energy): orphaned cleanup + dataAvailable docs (V6-5) (#2851)
* fix(energy): remove orphaned chokepointTransits bootstrap; document dataAvailable semantics (V6-5) * test(energy): remove chokepointTransits assertions from supply-chain-v2 tests |
||
|
Elie Habib
|
aa794e1369 |
feat(portwatch): seed per-country port activity (Endpoints 3+4) (#2786)
* feat(portwatch): seed per-country port activity (Endpoints 3+4) * fix(portwatch): register portwatch-ports seed-meta in api/seed-health.js * fix(portwatch): correct Endpoint 3 field names and move Redis writes out of fetchAll() * fix(portwatch): add Endpoint 4 pagination loop and fix anomalySignal divisor symmetry * fix(portwatch): stable pagination order + add portwatchPortActivity to PENDING_CONSUMERS * fix(portwatch): degradation guard + hoist prevCountryKeys for correct catch-block TTL extension |
||
|
Elie Habib
|
cf27ffbfde |
feat(portwatch): seed chokepoints reference (Endpoint 2) (#2785)
* feat(portwatch): seed chokepoints reference data from Endpoint 2 * test(bootstrap): exempt portwatchChokepointsRef from consumer check (UI consumer in future PR) * fix(portwatch): register chokepoints-ref seed-meta in api/seed-health.js * fix(portwatch): add returnGeometry=false to reduce ArcGIS response size * fix(portwatch): raise validateFn threshold to 27 (guards against partial ArcGIS responses) * fix(portwatch): validateFn requires exactly 28 chokepoints (reject partial ArcGIS responses) |
||
|
Elie Habib
|
492a99eccd |
feat(resilience): IMF macro phase 2 — current account + inflation proxy (#2766)
* feat(resilience): IMF macro phase 2 — current account + inflation proxy
Replace BIS credit (40-country curated list) with IMF WEO current account
balance (~185 countries) in scoreMacroFiscal, and add IMF CPI inflation as
tier-2 fallback for non-BIS countries in scoreCurrencyExternal. New seeder:
scripts/seed-imf-macro.mjs (PCPIPCH + BCA_NGDPD, key: economic:imf:macro:v1,
TTL: 35 days). api/health.js registers imfMacro as STANDALONE + ON_DEMAND.
* fix(resilience): BIS outage uses IMF proxy; imfMacro not on-demand
Two reviewer issues addressed:
1. scoreCurrencyExternal was short-circuiting to {score:50, coverage:0}
on BIS outage (bisExchangeRaw==null) before checking IMF inflation.
Now tries IMF proxy first in both cases (BIS absent from curated list
and BIS seed outage), with coverage=0.35 for outage vs 0.45 for
curated-list-absent (primary source unavailable reduces confidence).
Adds pinning test for BIS null + IMF present → coverage=0.35 path.
2. imfMacro was misclassified as ON_DEMAND in health.js even though it
has a dedicated seeder and seed-meta entry. Removed from ON_DEMAND_KEYS
so a broken monthly cron surfaces as a seeded-data failure, not a
silent EMPTY_ON_DEMAND warning.
* fix(resilience): dynamic WEO year + 2x stale window for imfMacro
- seed-imf-macro.mjs: replace hard-coded 2024 periods/sourceVersion
with weoYears() computed at runtime (currentYear, currentYear-1,
currentYear-2) so the monthly cron always fetches the latest WEO
vintage without code changes (e.g. 2025,2024,2023 once April WEO
publishes)
- api/health.js: imfMacro.maxStaleMin 50400→100800 (1× interval →
2× interval = 70 days). Matches repo pattern for monthly seeds
(faoFoodPriceIndex uses 86400 = 2× its 30-day interval). Prevents
false STALE_SEED flaps from normal schedule slip or one missed run.
* fix(resilience): use loadSharedConfig for iso2-to-iso3 in seed-imf-macro
Direct readFileSync(../shared/...) fails in Railway where rootDirectory=scripts
isolates the build context. loadSharedConfig() from _seed-utils.mjs tries
../shared/ (local dev) then ./shared/ (Railway) — same pattern as all other
seeders that need shared data files.
* fix(bootstrap): register imfMacro in BOOTSTRAP_CACHE_KEYS and SLOW_KEYS
economic:imf:macro:v1 was seeded and used by scoreMacroFiscal /
scoreCurrencyExternal but absent from bootstrap hydration, causing
cold SPA loads to silently degrade to debt-only scoring for ~130
non-BIS countries. Add to SLOW_KEYS consistent with bisExchange/bisCredit.
* fix(cache-keys): add imfMacro to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS
Required by bootstrap.test.mjs: tier sets in bootstrap.js must match
BOOTSTRAP_TIERS in cache-keys.ts. imfMacro is slow-tier (monthly seed,
large payload) consistent with bisExchange/bisCredit.
* test(bootstrap): add imfMacro to PENDING_CONSUMERS
imfMacro is currently backend-only (resilience scorer reads it from Redis
directly). Frontend consumer is not yet wired in src/. Add to
PENDING_CONSUMERS to pass bootstrap consumer coverage check.
* test(resilience): update score assertions after sanctions:country-counts rebase
Rebasing on main picked up PR #2763 (sanctions:country-counts:v1 replacing
sanctions:pressure:v1). Combined with IMF macro fixtures, the economic
domain average shifts from 70 → 63.67 and overall from 68.81 → 67.41.
|
||
|
Elie Habib
|
f3843aaaf1 |
feat(energy): seed EIA chokepoint baseline volumes (#2735)
* feat(energy): seed EIA chokepoint baseline volumes - Add scripts/seed-chokepoint-baselines.mjs with 7 hardcoded EIA 2023 chokepoints (Hormuz through Panama), 400-day TTL, no network calls - Add tests/chokepoint-baselines-seed.test.mjs with 14 test cases covering payload shape, key constants, TTL, and validateFn - Register seed-chokepoint-baselines in railway-set-watch-paths.mjs with annual cron (0 0 1 1 *) * fix(energy): 3 review fixes for chokepoint-baselines PR P1 — IEA seed: move per-country Redis writes from fetch phase to afterPublish pipeline. fetchIeaOilStocks now returns pure data; publishTransform builds the canonical index; writeCountryKeys sends all 32 country keys atomically via pipeline in the publish phase. A mid-run Redis failure can no longer leave a partially-updated snapshot with a stale index. P2 — Wire chokepointBaselines into bootstrap: add to BOOTSTRAP_CACHE_KEYS + SLOW_KEYS in api/bootstrap.js and server/_shared/cache-keys.ts + BOOTSTRAP_TIERS. P3 — Wire IEA seed operationally: add seed-iea-oil-stocks service to railway-set-watch-paths.mjs (monthly cron 0 6 20 * *) and ieaOilStocks health entry (40-day maxStaleMin) to api/health.js. * fix(test): add chokepointBaselines to PENDING_CONSUMERS Frontend consumer not yet implemented; consistent with chokepointTransits, correlationCards, euGasStorage which are also wired to bootstrap ahead of their UI panels. * fix(energy): register country keys in extraKeys for TTL preservation afterPublish runs in the publish phase but is NOT included in runSeed's failure-path TTL extension. Replace afterPublish+writeCountryKeys with COUNTRY_EXTRA_KEYS (one entry per COUNTRY_MAP iso2) declared as extraKeys: - On fetch failure or validation skip: runSeed extends TTL for all 32 country keys alongside the canonical index - On successful publish: writeExtraKey writes each country key with a per-iso2 transform; no dangling index entries after failed refreshes Also removes now-unused getRedisCredentials import. * fix(energy): 3 follow-up review fixes High — seed-meta TTL: writeFreshnessMetadata now accepts a ttlSeconds param and uses max(7d, ttlSeconds). runSeed passes its data TTL so monthly/annual seeds (IEA: 40d, chokepoint: 400d) no longer lose their seed-meta key on day 8 before health maxStaleMin is reached. Medium — Turkey name: IEA API returns "Turkiye" (no umlaut) while COUNTRY_MAP keys "Türkiye". parseRecord now normalizes the alias before lookup; TR is no longer silently dropped. Test added to cover the normalized form. Medium — Bootstrap revert: remove chokepointBaselines from BOOTSTRAP_CACHE_KEYS, SLOW_KEYS (bootstrap.js), BOOTSTRAP_TIERS (cache-keys.ts), and PENDING_CONSUMERS (bootstrap test) until a src/ consumer exists. Static 7-entry payload should not load on every bootstrap request for a feature with no frontend. * fix(seed-utils): pass ttlSeconds to writeFreshnessMetadata on skip path The validation-skip branch at runSeed:657 was still calling writeFreshnessMetadata without ttlSeconds, reintroducing the 7-day meta TTL for any monthly/annual seed that hits an empty-data run. * fix(test): restore chokepointBaselines in PENDING_CONSUMERS Rebase conflict resolution kept chokepointBaselines in BOOTSTRAP_CACHE_KEYS but the follow-up fix commit's test change auto-merged and removed it from PENDING_CONSUMERS. Re-add it so the consumer-coverage test passes while the frontend consumer is still pending. * fix(iea): align COUNTRY_MAP to ASCII Turkiye key (matches main + test) main (PR #2733) uses 'Turkiye' (no umlaut) as the COUNTRY_MAP key directly. Our branch had 'Türkiye' + parseRecord normalization. Align with main's approach: single key, no normalization shim needed. |
||
|
Elie Habib
|
a277b0f363 |
feat(energy): ENTSO-E + EIA-930 electricity spot prices (#2712)
* feat(seeds): ENTSO-E and EIA-930 electricity spot prices
- New seed script: scripts/seed-electricity-prices.mjs
- Fetches EU day-ahead prices from ENTSO-E API (11 regions, XML parsing, no external deps)
- Fetches US balancing area demand from EIA-930 API (6 regions)
- Batches ENTSO-E requests 3 at a time with 300ms delay
- Writes per-region keys (energy:electricity:v1:{region}) + index key
- Falls back gracefully when ENTSO_E_TOKEN is absent (EIA-only path)
- Preserves previous snapshot via extendExistingTtl when <3 ENTSO-E regions return
- TTL: 3 days (259200s); isMain guard present
- Tests: tests/electricity-prices-seed.test.mjs (12 tests, all passing)
- api/health.js: added electricityPrices to BOOTSTRAP_KEYS and SEED_META
- server/_shared/cache-keys.ts: added ELECTRICITY_KEY_PREFIX + ELECTRICITY_INDEX_KEY
- scripts/railway-set-watch-paths.mjs: added seed-electricity-prices service config (0 14 * * *)
Task: energy-phase2-unit-d
* fix(electricity): negative prices, US data on EU failure, bootstrap key
- parseEntsoEPrice regex now matches negative prices (-?[\d.]+)
- On EU coverage below threshold, still write US EIA data instead of
discarding it. EU keys get TTL extended as before.
- Add electricityPrices to bootstrap.js BOOTSTRAP_CACHE_KEYS + SLOW_KEYS
- Add negative price tests
* fix(electricity): add to shared BOOTSTRAP_CACHE_KEYS for RPC path parity
|
||
|
Elie Habib
|
802e2ca66c |
feat(climate): add Climate News panel (#2722)
* feat(climate): add Climate News panel consuming ListClimateNews RPC - ClimateNewsPanel: scrollable news cards with source badge, time-ago, title, and summary snippet. Links to original articles. - Bootstrap-hydrated: instant render from climateNews key, background RPC refresh. - Registered in panels.ts, commands.ts (searchable), panel-layout.ts. - Removed climateNews from PENDING_CONSUMERS in bootstrap test. - Added i18n keys for en locale. Closes #2611 * fix(review): sanitize external URLs in climate news cards * fix(review): wire climate-news into App prime path and refresh scheduler Without prime wiring, the panel renders but stays empty until manual refresh. Added shouldPrime + refreshScheduler entries matching the FAO panel pattern. Refresh interval: 30min (matches seed cadence). |
||
|
Fayez Bast
|
b2bae30bd8 |
Add climate news seed and ListClimateNews RPC (#2532)
* Add climate news seed and ListClimateNews RPC * Wire climate news into bootstrap and fix generated climate stubs * fix(climate): align seed health interval and parse Atom entries per feed * fix(climate-news): TTL 90min, retry timer on failure, named cache key constant - CACHE_TTL: 1800 to 5400 (90min = 3x 30-min relay interval, gold standard) - ais-relay: add 20-min retry timer on subprocess failure; clear on success - cache-keys.ts: export CLIMATE_NEWS_KEY named constant - list-climate-news.ts: import CLIMATE_NEWS_KEY instead of hard-coding string --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
bb4f8dcb12 |
feat(climate): add WMO normals seeding and CO2 monitoring (#2531)
* feat(climate): add WMO normals seeding and CO2 monitoring * fix(climate): skip missing normals per-zone and align anomaly tooltip copy * fix(climate): remove normals from bootstrap and harden health/cache key wiring * feat(climate): version anomaly cache to v2, harden seed freshness, and align CO2/normal baselines |
||
|
Elie Habib
|
b6847e5214 |
feat(feeds): GIE AGSI+ EU gas storage seeder (#2281) (#2339)
* feat(feeds): GIE AGSI+ EU gas storage seeder — European energy security indicator (#2281) - New scripts/seed-gie-gas-storage.mjs: fetches EU aggregate gas storage fill % from GIE AGSI+ API, computes 1-day change, trend (injecting/withdrawing/stable), and days-of-consumption estimate; TTL=259200s (3x daily); isMain guard; CHROME_UA; validates fillPct in (0,100]; graceful degradation when GIE_API_KEY absent - New proto/worldmonitor/economic/v1/get_eu_gas_storage.proto + GetEuGasStorage RPC wired into EconomicService - New server/worldmonitor/economic/v1/get-eu-gas-storage.ts handler (reads seeded Redis key) - api/health.js: BOOTSTRAP_KEYS + SEED_META (maxStaleMin=2880, 2x daily cadence) - api/bootstrap.js: euGasStorage key in SLOW_KEYS bucket - Regenerated src/generated/ + docs/api/ via make generate * fix(feeds): wire euGasStorage into cache-keys, gateway tier, and test PENDING_CONSUMERS - server/_shared/cache-keys.ts: add euGasStorage → economic:eu-gas-storage:v1 (slow tier) - server/gateway.ts: add /api/economic/v1/get-eu-gas-storage → slow RPC_CACHE_TIER - tests/bootstrap.test.mjs: add euGasStorage to PENDING_CONSUMERS (no frontend panel yet) * fix(gie-gas-storage): normalize seededAt to string to match proto int64 contract Proto int64 seeded_at maps to string in JS; seed was writing Date.now() (number). Fix seed to write String(Date.now()) and add handler-side normalization for any stale Redis entries that may have the old numeric format. * fix(feeds): coerce nullable fillPctChange1d/gasDaysConsumption to 0 (#2281) Greptile P1: both fields could be null (single data-point run or missing volume) but the proto interface declares them as non-optional numbers. Seed script now returns 0 instead of null; handler defensively coerces nulls from older cached blobs via nullish coalescing. Dead null-guard on trend derivation also removed. |
||
|
Elie Habib
|
aab494fa52 |
fix(cors): prevent CF from caching per-origin ACAO responses (#1890)
Cloudflare (in front of api.worldmonitor.app) ignores Vary: Origin and caches the first response's Access-Control-Allow-Origin header, serving it to all subsequent origins. This broke CORS for Vercel preview deployments which got ACAO: worldmonitor.app instead of their own origin. Fix: remove public/s-maxage from Cache-Control in TIER_HEADERS and bootstrap TIER_CACHE so CF sees no shared-cache directive and does not cache. Vercel CDN continues to cache aggressively via CDN-Cache-Control (which it respects over Cache-Control) and correctly varies by origin. Update bootstrap test to reflect new cache header strategy. |
||
|
Elie Habib
|
bcccb3fb9c |
test: cover runtime env guardrails (#1650)
* fix(data): restore bootstrap and cache test coverage * test: cover runtime env guardrails * fix(test): align security header tests with current vercel.json Update catch-all source pattern, geolocation policy value, and picture-in-picture origins to match current production config. |
||
|
Elie Habib
|
fe67111dc9 |
feat: harness engineering P0 - linting, testing, architecture docs (#1587)
* feat: harness engineering P0 - linting, testing, architecture docs
Add foundational infrastructure for agent-first development:
- AGENTS.md: agent entry point with progressive disclosure to deeper docs
- ARCHITECTURE.md: 12-section system reference with source-file refs and ownership rule
- Biome 2.4.7 linter with project-tuned rules, CI workflow (lint-code.yml)
- Architectural boundary lint enforcing forward-only dependency direction (lint-boundaries.mjs)
- Unit test CI workflow (test.yml), all 1083 tests passing
- Fixed 9 pre-existing test failures (bootstrap sync, deploy-config headers, globe parity, redis mocks, geometry URL, import.meta.env null safety)
- Fixed 12 architectural boundary violations (types moved to proper layers)
- Added 3 missing cache tier entries in gateway.ts
- Synced cache-keys.ts with bootstrap.js
- Renamed docs/architecture.mdx to "Design Philosophy" with cross-references
- Deprecated legacy docs/Docs_To_Review/ARCHITECTURE.md
- Harness engineering roadmap tracking doc
* fix: address PR review feedback on harness-engineering-p0
- countries-geojson.test.mjs: skip gracefully when CDN unreachable
instead of failing CI on network issues
- country-geometry-overrides.test.mts: relax timing assertion
(250ms -> 2000ms) for constrained CI environments
- lint-boundaries.mjs: implement the documented api/ boundary check
(was documented but missing, causing false green)
* fix(lint): scan api/ .ts files in boundary check
The api/ boundary check only scanned .js/.mjs files, missing the 25
sebuf RPC .ts edge functions. Now scans .ts files with correct rules:
- Legacy .js: fully self-contained (no server/ or src/ imports)
- RPC .ts: may import server/ and src/generated/ (bundled at deploy),
but blocks imports from src/ application code
* fix(lint): detect import() type expressions in boundary lint
- Move AppContext back to app/app-context.ts (aggregate type that
references components/services/utils belongs at the top, not types/)
- Move HappyContentCategory and TechHQ to types/ (simple enums/interfaces)
- Boundary lint now catches import('@/layer') expressions, not just
from '@/layer' imports
- correlation-engine imports of AppContext marked boundary-ignore
(type-only imports of top-level aggregate)
|
||
|
Elie Habib
|
2e93e0e8ed |
perf(bootstrap): tier slow/fast data for ~46% CDN egress reduction (#838)
Split bootstrap endpoint into slow-changing (1h TTL: BIS rates, minerals, sectors, etc.) and fast-changing (10min TTL: earthquakes, outages, macro signals, etc.) tiers via ?tier=slow|fast query param. Frontend fetches both tiers in parallel with shared 800ms timeout. Partial failure is graceful — panels fall through to individual RPCs. Backward compatible: no ?tier= param returns all keys at s-maxage=600. Also removes orphaned ucdpEvents key (no getHydratedData consumer). |
||
|
Elie Habib
|
98d231595e |
perf: bootstrap endpoint + polling optimization (#495)
* perf: bootstrap endpoint + polling optimization (phases 3-4)
Replace 15+ individual RPC calls on startup with a single /api/bootstrap
batch call that fetches pre-cached data from Redis. Consolidate 6 panel
setInterval timers into the central RefreshScheduler for hidden-tab
awareness (10x multiplier) and adaptive backoff (up to 4x for unchanged
data). Convert IntelligenceGapBadge from 10s polling to event-driven
updates with 60s safety fallback.
* fix(bootstrap): inline Redis + cache keys in edge function
Vercel Edge Functions cannot resolve cross-directory TypeScript imports
from server/_shared/. Inline getCachedJsonBatch and BOOTSTRAP_CACHE_KEYS
directly in api/bootstrap.js. Add sync test to ensure inlined keys stay
in sync with the canonical server/_shared/cache-keys.ts registry.
* test: add Edge Function module isolation guard for all api/*.js files
Prevents any Edge Function from importing from ../server/ or ../src/
which breaks Vercel builds. Scans all 12 non-helper Edge Functions.
* fix(bootstrap): read unprefixed cache keys on all environments
Preview deploys set VERCEL_ENV=preview which caused getKeyPrefix() to
prefix Redis keys with preview:<sha>:, but handlers only write to
unprefixed keys on production. Bootstrap is a read-only consumer of
production cache — always read unprefixed keys.
* fix(bootstrap): wire sectors hydration + add coverage guard
- Wire getHydratedData('sectors') in data-loader to skip Yahoo Finance
fetch when bootstrap provides sector data
- Add test ensuring every bootstrap key has a getHydratedData consumer
— prevents adding keys without wiring them
* fix(server): resolve 25 TypeScript errors + add server typecheck to CI
- _shared.ts: remove unused `delay` variable
- list-etf-flows.ts: add missing `rateLimited` field to 3 return literals
- list-market-quotes.ts: add missing `rateLimited` field to 4 return literals
- get-cable-health.ts: add non-null assertions for regex groups and array access
- list-positive-geo-events.ts: add non-null assertion for array index
- get-chokepoint-status.ts: add required fields to request objects
- CI: run `typecheck:api` (tsconfig.api.json) alongside `typecheck` to catch
server/ TS errors before merge
|