mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
6e401ad02f49cc12a8b77e49218f2e2f24a934cf
70 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
6e401ad02f |
feat(supply-chain): Global Shipping Intelligence — Sprint 0 + Sprint 1 (#2870)
* feat(supply-chain): Sprint 0 — chokepoint registry, HS2 sectors, war_risk_tier
- src/config/chokepoint-registry.ts: single source of truth for all 13
canonical chokepoints with displayName, relayName, portwatchName,
corridorRiskName, baselineId, shockModelSupported, routeIds, lat/lon
- src/config/hs2-sectors.ts: static dictionary for all 99 HS2 chapters
with category, shockModelSupported (true only for HS27), cargoType
- server/worldmonitor/supply-chain/v1/_chokepoint-ids.ts: migrated to
derive CANONICAL_CHOKEPOINTS from chokepoint-registry; no data duplication
- src/config/geo.ts + src/types/index.ts: added chokepointId field to
StrategicWaterway interface and all 13 STRATEGIC_WATERWAYS entries
- src/components/MapPopup.ts: switched chokepoint matching from fragile
name.toLowerCase() to direct chokepointId === id comparison
- server/worldmonitor/intelligence/v1/_shock-compute.ts: migrated from old
IDs (hormuz/malacca/babelm) to canonical IDs (hormuz_strait/malacca_strait/
bab_el_mandeb); same for CHOKEPOINT_LNG_EXPOSURE
- proto/worldmonitor/supply_chain/v1/supply_chain_data.proto: added
WarRiskTier enum + war_risk_tier field (field 16) on ChokepointInfo
- get-chokepoint-status.ts: populates warRiskTier from ChokepointConfig.threatLevel
via new threatLevelToWarRiskTier() helper (FREE field, no PRO gate)
* feat(supply-chain): Sprint 1 — country chokepoint exposure index + sector ring
S1.1: scripts/shared/country-port-clusters.json
~130 country → {nearestRouteIds, coastSide} mappings derived from trade route
waypoints; covers all 6 seeded Comtrade reporters plus major trading nations.
S1.2: scripts/seed-hs2-chokepoint-exposure.mjs
Daily cron seeder. Pure computation — reads country-port-clusters.json,
scores each country against CHOKEPOINT_REGISTRY route overlap, writes
supply-chain:exposure:{iso2}:{hs2}:v1 keys + seed-meta (24h TTL).
S1.3: RPC get-country-chokepoint-index (PRO-gated, request-varying)
- proto: GetCountryChokepointIndexRequest/Response + ChokepointExposureEntry
- handler: isCallerPremium gate; cachedFetchJson 24h; on-demand for any iso2
- cache-keys.ts: CHOKEPOINT_EXPOSURE_KEY(iso2, hs2) constant
- health.js: chokepointExposure SEED_META entry (48h threshold)
- gateway.ts: slow-browser cache tier
- service client: fetchCountryChokepointIndex() exported
S1.4: Chokepoint popup HS2 sector ring chart (PRO-gated)
Static trade-sector breakdown (IEA/UNCTAD estimates) per 9 major chokepoints.
SVG donut ring + legend shown for PRO users; blurred lockout + gate-hit
analytics for free users. Wired into renderWaterwayPopup().
🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.com/claude-code) + Compound Engineering v2.49.0
Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
* fix(tests): update energy-shock-v2 tests to use canonical chokepoint IDs
CHOKEPOINT_EXPOSURE and CHOKEPOINT_LNG_EXPOSURE keys were migrated from
short IDs (hormuz, malacca, babelm) to canonical registry IDs
(hormuz_strait, malacca_strait, bab_el_mandeb) in Sprint 0.
Test fixtures were not updated at the time; fix them now.
* fix(tests): update energy-shock-seed chokepoint ID to canonical form
VALID_CHOKEPOINTS changed to canonical IDs in Sprint 0; the seed test
that checks valid IDs was not updated alongside it.
* fix(cache-keys): reword JSDoc comment to avoid confusing bootstrap test regex
The comment "NOT in BOOTSTRAP_CACHE_KEYS" caused the bootstrap.test.mjs
regex to match the comment rather than the actual export declaration,
resulting in 0 entries found. Rephrase to "excluded from bootstrap".
* fix(supply-chain): address P1 review findings for chokepoint exposure index
- Add get-country-chokepoint-index to PREMIUM_RPC_PATHS (CDN bypass)
- Validate iso2/hs2 params before Redis key construction (cache injection)
- Fix seeder TTL to 172800s (2× interval) and extend TTL on skipped lock
- Fix CHOKEPOINT_EXPOSURE_SEED_META_KEY to match seeder write key
- Render placeholder sectors behind blur gate (DOM data leakage)
- Document get-country-chokepoint-index in widget agent system prompts
* fix(lint): resolve Biome CI failures
- Add biome.json overrides to silence noVar in HTML inline scripts,
disable linting for public/ vendor/build artifacts and pro-test/
- Remove duplicate NG and MW keys from country-port-clusters.json
- Use import attributes (with) instead of deprecated assert syntax
* fix(build): drop JSON import attribute — esbuild rejects `with` syntax
---------
Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
|
||
|
Elie Habib
|
6a87cf9530 |
feat(energy): bootstrap oilStocksAnalysis hydration + LNG vulnerability UI (#2854)
* feat(energy): bootstrap oilStocksAnalysis hydration + LNG vulnerability UI (V6-1) Register oilStocksAnalysis and lngVulnerability in BOOTSTRAP_CACHE_KEYS (both tiers: slow). Wire getHydratedData fast-path for oil stocks analysis, add fetchLngVulnerability() with bootstrap fallback, and render a top-5 LNG-dependent countries table in EnergyComplexPanel. * fix(energy): multiply LNG share by 100 for display; remove bootstrap-refetch fallback * fix(energy): use String() instead of toLocaleString() for LNG imports display |
||
|
Elie Habib
|
bc33fe955a |
fix(energy): orphaned cleanup + dataAvailable docs (V6-5) (#2851)
* fix(energy): remove orphaned chokepointTransits bootstrap; document dataAvailable semantics (V6-5) * test(energy): remove chokepointTransits assertions from supply-chain-v2 tests |
||
|
Elie Habib
|
65b4655dc6 |
fix(digest): namespace accumulator by language, add per-severity caps (#2826)
* fix(digest): namespace accumulator by language, add per-severity caps
Root cause: digest:accumulator:v1:${variant} was shared across all
languages. A buildDigest("full", "de") request wrote German stories
to the same accumulator the English digest cron consumed, leaking
non-English headlines into English email digests.
Fix (3 layers):
1. Accumulator key is now language-aware:
digest:accumulator:v1:${variant}:${lang}
writeStoryTracking receives lang and writes to the correct key.
Cron reads from the lang-specific key (defaults to 'en').
2. Defense-in-depth: lang is stored on story:track:v1:* hash records.
Cron filters stories where track.lang !== target lang.
3. Per-severity display caps use named constants:
CRITICAL=Infinity, HIGH=15, MEDIUM=10 (was hardcoded 10 for all).
Both text and HTML formatters use the same constants.
* fix(digest): remove track.lang cron filter, rely solely on accumulator key
track.lang is written by writeStoryTracking via HSET which overwrites
on every call. If the same normalized title appears in multiple
languages within the 48h TTL, the last writer wins the lang field.
Using it as a cron-side filter creates a race where valid stories get
dropped. The accumulator key namespacing (variant:lang) is the sole
language isolation mechanism.
|
||
|
Elie Habib
|
b8924eb90f |
feat(energy): Ember monthly electricity seed (V5-6a) (#2815)
* feat(energy): Ember monthly electricity seed — V5-6a New seed-ember-electricity.mjs writes energy:ember:v1:<ISO2> and energy:ember:v1:_all from Ember Climate's monthly generation CSV (CC BY 4.0). Daily cron at 08:00 UTC, TTL 72h (3x interval), >=60 country coverage guard. Registers in api/health.js, api/seed-health.js, cache-keys.ts, and ais-relay.cjs. Dockerfile.relay COPY added. 🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/code) + Compound Engineering v2.49.0 Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> * fix(energy): add _country-resolver.mjs to Dockerfile.relay; correct Ember intervalMin (V5-6a) Two bugs in the Ember seed PR: 1. Dockerfile.relay was missing COPY for _country-resolver.mjs, which seed-ember-electricity.mjs imports. Would have crashed with ERR_MODULE_NOT_FOUND on first run in production. 2. api/seed-health.js had intervalMin:720 (12h) for a daily (24h) cron. With stale threshold = intervalMin*2, this gave only 24h grace -- the seed would flap stale during the CSV download window. Corrected to intervalMin:1440 so stale threshold = 48h (2x interval). * fix(energy): wire energy:ember:v1:_all into bootstrap hydration (V5-6a) Greptile P1: api/bootstrap.js was missing the emberElectricity slow-key entry, violating the AGENTS.md requirement that new data sources be registered for bootstrap hydration. energy:ember:v1:_all is a ~60-country bulk map (monthly cadence) - added to SLOW_KEYS consistent with faoFoodPriceIndex and other monthly-release bulk keys. Also updates server/_shared/cache-keys.ts BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS to keep the bootstrap test coverage green (bootstrap test validates that SLOW_KEYS and BOOTSTRAP_TIERS are in sync). * fix(energy): 3 review fixes for Ember seed (V5-6a) 1. Ember URL: updated to correct current download URL (old path returned HTTP 404, seeder could never run). 2. Count-drop guard after failure: failure path now preserves the previous recordCount in seed-meta instead of writing 0, so the 75% drop guard stays active after a failed run. 3. api/seed-health.js: status:error now marks seed as stale/error immediately instead of only checking age; prevents /api/seed-health showing ok for 48h while the seeder is failing. * fix(energy): correct Ember CSV column names + fix skipped-path meta (V5-6a) 1. CSV schema: parser was using country_code/series/unit/value/date but the real Ember CSV headers are "ISO 3 code"/"Variable"/"Unit"/ "Value"/"Date". Added COLS constants and updated all row field accesses. The schema sentinel (hasFossil check) was always firing because r.series was always undefined, causing every seeder run to abort. Updated test fixtures to use real column names. 2. Skipped-path meta: lock.skipped branch now reads existing meta and preserves recordCount and status while refreshing fetchedAt. Previously writing recordCount:0 disabled the count-drop guard after any skipped run and made health endpoints see false-ok with zero count. * fix(energy): remove skipped-path meta write + revert premature bootstrap (V5-6a) 1. lock.skipped: removed seed-meta write from the skipped path. The running instance writes correct meta on completion; refreshing fetchedAt on skip masked relay/lock failures from health endpoints. 2. Bootstrap: removed emberElectricity from BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS — no consumer exists in src/ yet. Per energyv5.md, bootstrap registration is deferred to PR7 when consumers land. * fix(energy): split ember pipeline writes; fix health.js recordCount lookup - api/health.js: add recordCount fallback in both seed-meta count reads so the Ember domain shows correct record count instead of always 1 - scripts/seed-ember-electricity.mjs: split single pipeline into Phase A (per-country + _all data) and Phase B (seed-meta only after Phase A succeeds) to prevent preservePreviousSnapshot reading a partial _all key * fix(energy): split ember pipeline writes; align SEED_ERROR in health.js; add tests * fix(energy): atomic rollback on partial pipeline failure; seedError priority in health cascade * fix(energy): DEL obsolete per-country keys on publish, rollback, and restore * fix(energy): MULTI/EXEC atomic pipeline; null recordCount on read-miss; dataWritten guard --------- Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> |
||
|
Elie Habib
|
d96259048d |
feat(energy): canonical energy spine — V5-1 (#2798)
* feat(energy): canonical energy spine seeder + handler read-through (V5-1) - Add scripts/seed-energy-spine.mjs: daily seeder that assembles one energy:spine:v1:<ISO2> key per country from 6 domain keys (OWID mix, JODI oil, JODI gas, IEA stocks, ENTSO-E electricity, GIE gas storage) - TTL 172800s (48h), count-drop guard at 80%, schema sentinel for OWID mix, lock pattern + Redis pipeline batch write mirroring seed-owid-energy-mix.mjs - Update get-country-energy-profile.ts to read from spine first; fall back to existing 6-key Promise.allSettled join on spine miss - Update chat-analyst-context.ts buildProductSupply/buildGasFlows/buildOilStocksCover to prefer spine key; fall through to direct domain key reads on miss - Update get-country-intel-brief.ts to read energy mix from spine.mix + sources.mixYear before falling back to energy:mix:v1: direct key - Add ENERGY_SPINE_KEY_PREFIX and ENERGY_SPINE_COUNTRIES_KEY to cache-keys.ts - Add energySpineCountries to api/health.js STANDALONE_KEYS and SEED_META - Add Railway cron comment (0 6 * * *) in ais-relay.cjs - Add tests/energy-spine-seed.test.mjs: 26 tests covering spine build logic, IEA anomaly guard, JODI oil fallback, schema sentinel, count-drop math * fix(energy): add cache-keys module replacement in redis-caching test The new ENERGY_SPINE_KEY_PREFIX import in get-country-intel-brief.ts was not patched in the importPatchedTsModule call used by redis-caching tests. Add cache-keys to the replacement map to resolve the module. * fix(energy): add missing fields to spine entry and read-through - buildOilFields: add product importsKbd (gasoline/diesel/jet/lpg) and belowObligation - buildMixFields: add windShare, solarShare, hydroShare - buildGasStorageFields: new helper storing fillPct, fillPctChange1d, trend - buildSpineEntry: add gasStorage section using new helper - EnergySpine interface: extend oil/mix/gasStorage to match seeder output - buildResponseFromSpine: read all new fields instead of hard-coding 0/false * fix(energy): exclude electricity/gas-storage from spine; add seed-health entry Spine-first path was returning stale gas-storage and electricity data for up to 8h after seeding (spine runs 06:00 UTC, gas storage updates 10:30 UTC, electricity updates 14:00 UTC). Fix: handler now reads gas-storage and electricity directly in parallel with the spine read (3-key allSettled). Fallback path drops from 6 to 4 keys since gas-storage and electricity are already fetched in the hot path. Also registers energy:spine in api/seed-health.js (daily cron, maxStaleMin inferred as 2×intervalMin = 2880 min). Seeder (seed-energy-spine.mjs) and its tests updated to reflect the narrowed spine schema — electricity and gasStorage fields removed from buildSpineEntry. * fix(energy): address Greptile P2 review findings - chat-analyst-context: return undefined when gas imports are 0 rather than falling back to totalDemandTj with an "imports" label — avoids mislabeling domestic gas demand as imports for net exporters (RU, QA, etc.) - seed-energy-spine: add status:'ok' to success-path seed-meta write so all seed-meta records have a consistent status field regardless of path |
||
|
Elie Habib
|
47af642d24 |
feat(energy): live chokepoint flow calibration from PortWatch DWT — V5-2 (#2797)
* feat(energy): chokepoint flow calibration seeder — V5-2 (Phase 4 PR A)
- Add CHOKEPOINT_FLOWS_KEY to server/_shared/cache-keys.ts
- Add energy:chokepoint-flows:v1 to health.js monitoring (maxStaleMin: 720)
- Add 6h chokepoint flow seed loop to ais-relay.cjs (seed-chokepoint-flows.mjs)
- Fix seeder to use degraded mode instead of throwing when PortWatch absent
- Add degraded-mode and ID-mapping tests to chokepoint-flows-seed.test.mjs
* fix(energy): restore throw for PortWatch absent + register seed-health
- seed-chokepoint-flows: revert degraded-path from warn+return{} back to
throw; PortWatch absent is an upstream-not-ready error, not a data-quality
issue — must throw so startChokepointFlowsSeedLoop schedules 20-min retry
- api/seed-health.js: add energy:chokepoint-flows to SEED_DOMAINS so
/api/seed-health surfaces missing/stale signal (intervalMin: 360 = 6h cron)
- tests: update degraded-mode assertions to match restored throw behavior
|
||
|
Elie Habib
|
aa794e1369 |
feat(portwatch): seed per-country port activity (Endpoints 3+4) (#2786)
* feat(portwatch): seed per-country port activity (Endpoints 3+4) * fix(portwatch): register portwatch-ports seed-meta in api/seed-health.js * fix(portwatch): correct Endpoint 3 field names and move Redis writes out of fetchAll() * fix(portwatch): add Endpoint 4 pagination loop and fix anomalySignal divisor symmetry * fix(portwatch): stable pagination order + add portwatchPortActivity to PENDING_CONSUMERS * fix(portwatch): degradation guard + hoist prevCountryKeys for correct catch-block TTL extension |
||
|
Elie Habib
|
cf27ffbfde |
feat(portwatch): seed chokepoints reference (Endpoint 2) (#2785)
* feat(portwatch): seed chokepoints reference data from Endpoint 2 * test(bootstrap): exempt portwatchChokepointsRef from consumer check (UI consumer in future PR) * fix(portwatch): register chokepoints-ref seed-meta in api/seed-health.js * fix(portwatch): add returnGeometry=false to reduce ArcGIS response size * fix(portwatch): raise validateFn threshold to 27 (guards against partial ArcGIS responses) * fix(portwatch): validateFn requires exactly 28 chokepoints (reject partial ArcGIS responses) |
||
|
Elie Habib
|
d64172e67e |
fix(resilience): calibrate scoring anchors — gov revenue, GPI, inflation cap (#2769)
* fix(resilience): calibrate scoring anchors — gov revenue, GPI, inflation cap Three evidence-based anchor fixes resolving score compression and fragile-state inflation (Haiti 56→expected ≤35, Somalia ~50→near-0 on next seed run): 1. Replace debt/GDP with gov revenue/GDP in scoreMacroFiscal (weight=0.5) Debt/GDP is gamed by HIPC relief: Somalia (5% debt post-cancellation) and Haiti (15%) scored near-100 on fiscal resilience despite being credit-excluded. Gov revenue as % GDP (IMF GGR_NGDP, anchor 5%=0 → 45%=100) directly measures fiscal capacity. seed-imf-macro.mjs now fetches GGR_NGDP alongside PCPIPCH + BCA_NGDPD. 2. Fix GPI anchor in scoreSocialCohesion: worst 4.0 → 3.6 Empirical GPI range is 1.1–3.4 (2024). Yemen (3.4) was scoring 20/100 instead of near-0. With anchor 3.6, Yemen scores 8, Somalia scores 19, Haiti scores 30. 3. Tighten inflation proxy cap in scoreCurrencyExternal: 100% → 50% 50%+ annual inflation is already catastrophic instability. Tighter cap better differentiates fragile states: Haiti 39% drops from score 61 → 22. Research basis: INFORM Risk Index, FSI methodology, ND-GAIN, World Bank silent debt crisis documentation, OECD States of Fragility 2022 normalization guidelines. * fix(seed): use GGR_G01_GDP_PT for gov revenue — GGR_NGDP returns empty payload GGR_NGDP is a WEO indicator not exposed in the DataMapper values API; GGR_G01_GDP_PT (Fiscal Monitor) returns 212 countries and covers the same concept: general government revenue as % of GDP. * fix(resilience): bump imf-macro key to v2 — atomise govRevenuePct rollout v1 was seeded 35-day TTL by PR #2766 without govRevenuePct; the seeder would skip the stale-check and never write the new field until expiry. v2 forces a clean seed on first Railway deploy, ensuring the scorer's 0.5-weight fiscal-capacity leg is always backed by real data. * fix(health): bump imfMacro key to v2 in health.js Missed in the key-bump commit; health endpoint was still checking the old v1 key and would report imfMacro as permanently stale after v2 is seeded. |
||
|
Elie Habib
|
492a99eccd |
feat(resilience): IMF macro phase 2 — current account + inflation proxy (#2766)
* feat(resilience): IMF macro phase 2 — current account + inflation proxy
Replace BIS credit (40-country curated list) with IMF WEO current account
balance (~185 countries) in scoreMacroFiscal, and add IMF CPI inflation as
tier-2 fallback for non-BIS countries in scoreCurrencyExternal. New seeder:
scripts/seed-imf-macro.mjs (PCPIPCH + BCA_NGDPD, key: economic:imf:macro:v1,
TTL: 35 days). api/health.js registers imfMacro as STANDALONE + ON_DEMAND.
* fix(resilience): BIS outage uses IMF proxy; imfMacro not on-demand
Two reviewer issues addressed:
1. scoreCurrencyExternal was short-circuiting to {score:50, coverage:0}
on BIS outage (bisExchangeRaw==null) before checking IMF inflation.
Now tries IMF proxy first in both cases (BIS absent from curated list
and BIS seed outage), with coverage=0.35 for outage vs 0.45 for
curated-list-absent (primary source unavailable reduces confidence).
Adds pinning test for BIS null + IMF present → coverage=0.35 path.
2. imfMacro was misclassified as ON_DEMAND in health.js even though it
has a dedicated seeder and seed-meta entry. Removed from ON_DEMAND_KEYS
so a broken monthly cron surfaces as a seeded-data failure, not a
silent EMPTY_ON_DEMAND warning.
* fix(resilience): dynamic WEO year + 2x stale window for imfMacro
- seed-imf-macro.mjs: replace hard-coded 2024 periods/sourceVersion
with weoYears() computed at runtime (currentYear, currentYear-1,
currentYear-2) so the monthly cron always fetches the latest WEO
vintage without code changes (e.g. 2025,2024,2023 once April WEO
publishes)
- api/health.js: imfMacro.maxStaleMin 50400→100800 (1× interval →
2× interval = 70 days). Matches repo pattern for monthly seeds
(faoFoodPriceIndex uses 86400 = 2× its 30-day interval). Prevents
false STALE_SEED flaps from normal schedule slip or one missed run.
* fix(resilience): use loadSharedConfig for iso2-to-iso3 in seed-imf-macro
Direct readFileSync(../shared/...) fails in Railway where rootDirectory=scripts
isolates the build context. loadSharedConfig() from _seed-utils.mjs tries
../shared/ (local dev) then ./shared/ (Railway) — same pattern as all other
seeders that need shared data files.
* fix(bootstrap): register imfMacro in BOOTSTRAP_CACHE_KEYS and SLOW_KEYS
economic:imf:macro:v1 was seeded and used by scoreMacroFiscal /
scoreCurrencyExternal but absent from bootstrap hydration, causing
cold SPA loads to silently degrade to debt-only scoring for ~130
non-BIS countries. Add to SLOW_KEYS consistent with bisExchange/bisCredit.
* fix(cache-keys): add imfMacro to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS
Required by bootstrap.test.mjs: tier sets in bootstrap.js must match
BOOTSTRAP_TIERS in cache-keys.ts. imfMacro is slow-tier (monthly seed,
large payload) consistent with bisExchange/bisCredit.
* test(bootstrap): add imfMacro to PENDING_CONSUMERS
imfMacro is currently backend-only (resilience scorer reads it from Redis
directly). Frontend consumer is not yet wired in src/. Add to
PENDING_CONSUMERS to pass bootstrap consumer coverage check.
* test(resilience): update score assertions after sanctions:country-counts rebase
Rebasing on main picked up PR #2763 (sanctions:country-counts:v1 replacing
sanctions:pressure:v1). Combined with IMF macro fixtures, the economic
domain average shifts from 70 → 63.67 and overall from 68.81 → 67.41.
|
||
|
Elie Habib
|
f3843aaaf1 |
feat(energy): seed EIA chokepoint baseline volumes (#2735)
* feat(energy): seed EIA chokepoint baseline volumes - Add scripts/seed-chokepoint-baselines.mjs with 7 hardcoded EIA 2023 chokepoints (Hormuz through Panama), 400-day TTL, no network calls - Add tests/chokepoint-baselines-seed.test.mjs with 14 test cases covering payload shape, key constants, TTL, and validateFn - Register seed-chokepoint-baselines in railway-set-watch-paths.mjs with annual cron (0 0 1 1 *) * fix(energy): 3 review fixes for chokepoint-baselines PR P1 — IEA seed: move per-country Redis writes from fetch phase to afterPublish pipeline. fetchIeaOilStocks now returns pure data; publishTransform builds the canonical index; writeCountryKeys sends all 32 country keys atomically via pipeline in the publish phase. A mid-run Redis failure can no longer leave a partially-updated snapshot with a stale index. P2 — Wire chokepointBaselines into bootstrap: add to BOOTSTRAP_CACHE_KEYS + SLOW_KEYS in api/bootstrap.js and server/_shared/cache-keys.ts + BOOTSTRAP_TIERS. P3 — Wire IEA seed operationally: add seed-iea-oil-stocks service to railway-set-watch-paths.mjs (monthly cron 0 6 20 * *) and ieaOilStocks health entry (40-day maxStaleMin) to api/health.js. * fix(test): add chokepointBaselines to PENDING_CONSUMERS Frontend consumer not yet implemented; consistent with chokepointTransits, correlationCards, euGasStorage which are also wired to bootstrap ahead of their UI panels. * fix(energy): register country keys in extraKeys for TTL preservation afterPublish runs in the publish phase but is NOT included in runSeed's failure-path TTL extension. Replace afterPublish+writeCountryKeys with COUNTRY_EXTRA_KEYS (one entry per COUNTRY_MAP iso2) declared as extraKeys: - On fetch failure or validation skip: runSeed extends TTL for all 32 country keys alongside the canonical index - On successful publish: writeExtraKey writes each country key with a per-iso2 transform; no dangling index entries after failed refreshes Also removes now-unused getRedisCredentials import. * fix(energy): 3 follow-up review fixes High — seed-meta TTL: writeFreshnessMetadata now accepts a ttlSeconds param and uses max(7d, ttlSeconds). runSeed passes its data TTL so monthly/annual seeds (IEA: 40d, chokepoint: 400d) no longer lose their seed-meta key on day 8 before health maxStaleMin is reached. Medium — Turkey name: IEA API returns "Turkiye" (no umlaut) while COUNTRY_MAP keys "Türkiye". parseRecord now normalizes the alias before lookup; TR is no longer silently dropped. Test added to cover the normalized form. Medium — Bootstrap revert: remove chokepointBaselines from BOOTSTRAP_CACHE_KEYS, SLOW_KEYS (bootstrap.js), BOOTSTRAP_TIERS (cache-keys.ts), and PENDING_CONSUMERS (bootstrap test) until a src/ consumer exists. Static 7-entry payload should not load on every bootstrap request for a feature with no frontend. * fix(seed-utils): pass ttlSeconds to writeFreshnessMetadata on skip path The validation-skip branch at runSeed:657 was still calling writeFreshnessMetadata without ttlSeconds, reintroducing the 7-day meta TTL for any monthly/annual seed that hits an empty-data run. * fix(test): restore chokepointBaselines in PENDING_CONSUMERS Rebase conflict resolution kept chokepointBaselines in BOOTSTRAP_CACHE_KEYS but the follow-up fix commit's test change auto-merged and removed it from PENDING_CONSUMERS. Re-add it so the consumer-coverage test passes while the frontend consumer is still pending. * fix(iea): align COUNTRY_MAP to ASCII Turkiye key (matches main + test) main (PR #2733) uses 'Turkiye' (no umlaut) as the COUNTRY_MAP key directly. Our branch had 'Türkiye' + parseRecord normalization. Align with main's approach: single key, no normalization shim needed. |
||
|
Elie Habib
|
52eef2d52b |
feat(energy): seed JODI oil product-level supply data (#2732)
* feat(energy): seed JODI oil product-level supply data
- Add scripts/seed-jodi-oil.mjs: fetches JODI World annual CSV files (primary + secondary, current + prior year) in parallel, parses KBD rows, builds per-country payloads with gasoline/diesel/jet/fuelOil/lpg/crude fields, applies anomaly cap (TOTDEMO > 10,000 KBD for non-US = null), assessment_code=3 = null, coverage gate requires 60+ countries
- Write energy:jodi-oil:v1:{ISO2} per-country keys + energy:jodi-oil:v1:_countries index via Redis pipeline; TTL 35 days
- Add tests/jodi-oil-seed.test.mjs: 33 unit tests covering CSV parsing, schema validation, null handling, anomaly cap, assessment code filtering, coverage gate, and key constant contracts
- Register seed-jodi-oil service in scripts/railway-set-watch-paths.mjs with monthly cron (0 6 25 * *)
- Add jodiOil health entry in api/health.js with 40-day maxStaleMin
Task: energy2.5.md PR 1a
* fix(energy): write seed-meta on coverage gate failure in jodi-oil seeder
- After extendExistingTtl on coverage gate failure, write META_KEY with
recordCount=0 so health checks have a seed-meta entry on fresh deploy
- Apply the same fix in the catch path to prevent STALE_SEED on first run
* fix(energy): address 5 review issues in jodi-oil seeder
- P1: mergeSourceRows() throws when both secondary files fail instead of
silently publishing a crude-only snapshot with null product fields
- P1: preservation paths no longer write fetchedAt to seed-meta; existing
meta ages naturally so health reflects actual data freshness, not the
failed attempt (matches ais-relay.cjs gold standard)
- P1: coverage-gate and catch paths now read CANONICAL_KEY from Redis to
extend TTL on all previously-stored country keys, not just the
currently-parsed subset, preventing dangling _countries index entries
- P2: wire jodiOil into api/bootstrap.js BOOTSTRAP_CACHE_KEYS
- P2: fix logSeedResult arg order (count was 'jodi-oil' string, now
countries.length; recordCount in structured log is now numeric)
- Tests: add 5 mergeSourceRows tests covering partial-failure modes
* fix(energy): add jodiOil to SLOW_KEYS and BOOTSTRAP_TIERS
Bootstrap test enforces SLOW_KEYS + FAST_KEYS == BOOTSTRAP_CACHE_KEYS
and BOOTSTRAP_TIERS parity. Add jodiOil: slow to all three tier maps.
* fix(energy): require secondary-product row in dataMonth; fix CSV parser
- extractCountryData now skips months that contain only crude rows;
if secondary/currentYear.csv fails, the algorithm falls back to a
prior-year month that has actual product data rather than publishing
a snapshot with null gasoline/diesel/jet/fuelOil/lpg fields
- parseCsv replaced naive split(',') with a proper quoted-field state
machine that handles commas inside quoted fields and escaped double-
quotes (RFC 4180 subset)
- Add 4 tests: crude-only fallback, null-when-no-secondary, quoted-comma
field, and escaped-double-quote field
|
||
|
Elie Habib
|
83ff5772b2 | feat(energy): energy Phase 2 analyst context integration (#2721) | ||
|
Elie Habib
|
066712e859 |
feat(energy): IEA/OPEC energy intelligence RSS feed (#2713)
* feat(seeds): IEA and OPEC energy intelligence RSS feed - Add scripts/seed-energy-intelligence.mjs: parses IEA news, IEA reports, and OPEC press RSS feeds; filters by 20 energy keywords; deduplicates by URL (keeps most recent); excludes items older than 30 days; limits to 30 most recent items; TTL 86400s (24h); validates >= 3 items - Add tests/energy-intelligence-seed.test.mjs: 10 tests covering parseRssItems, filterEnergyRelevant, deduplicateByUrl, age filter, and key constants - Add ENERGY_INTELLIGENCE_KEY to server/_shared/cache-keys.ts - Add energyIntelligence to BOOTSTRAP_KEYS and SEED_META in api/health.js - Add seed-energy-intelligence service override to railway-set-watch-paths.mjs with 6h cron schedule * fix(seeds): replace dead IEA RSS (404) with OilPrice.com; keep OPEC best-effort * fix(seeds): fix energyIntelligence health placement, key format, validate export, entity decoding P1: energyIntelligence was in health.js BOOTSTRAP_KEYS but absent from api/bootstrap.js and BOOTSTRAP_CACHE_KEYS. The feed has no SPA consumer (server-side read only via chat-analyst-context), so it belongs in STANDALONE_KEYS in health.js, not BOOTSTRAP_KEYS. Moved accordingly: health monitoring is preserved via SEED_META, and the bootstrap test invariants (every bootstrap key must have a getHydratedData consumer) are satisfied. Key format: CANONICAL_KEY renamed energy:intelligence:v1:feed → energy:intelligence:feed:v1 to comply with the :v\d+$ convention enforced by bootstrap.test.mjs. Updated in health.js, cache-keys.ts (standalone export), and seed-energy-intelligence.mjs. P2: export validate() from seed-energy-intelligence.mjs and add tests covering the skip path (< 3 items → false, exactly 3 → true, > 3 → true). OPEC is best-effort and OilPrice is primary, so sub-threshold runs are a real production scenario. Quality: expand decodeHtmlEntities to handle numeric decimal/hex character references (’ ’) and common named entities (' … — – ‘ ’ “ ”). & decoded last to handle double-encoded sequences correctly. Five new tests added. * fix(seeds): remove unused extendExistingTtl import from seed-energy-intelligence |
||
|
Elie Habib
|
a277b0f363 |
feat(energy): ENTSO-E + EIA-930 electricity spot prices (#2712)
* feat(seeds): ENTSO-E and EIA-930 electricity spot prices
- New seed script: scripts/seed-electricity-prices.mjs
- Fetches EU day-ahead prices from ENTSO-E API (11 regions, XML parsing, no external deps)
- Fetches US balancing area demand from EIA-930 API (6 regions)
- Batches ENTSO-E requests 3 at a time with 300ms delay
- Writes per-region keys (energy:electricity:v1:{region}) + index key
- Falls back gracefully when ENTSO_E_TOKEN is absent (EIA-only path)
- Preserves previous snapshot via extendExistingTtl when <3 ENTSO-E regions return
- TTL: 3 days (259200s); isMain guard present
- Tests: tests/electricity-prices-seed.test.mjs (12 tests, all passing)
- api/health.js: added electricityPrices to BOOTSTRAP_KEYS and SEED_META
- server/_shared/cache-keys.ts: added ELECTRICITY_KEY_PREFIX + ELECTRICITY_INDEX_KEY
- scripts/railway-set-watch-paths.mjs: added seed-electricity-prices service config (0 14 * * *)
Task: energy-phase2-unit-d
* fix(electricity): negative prices, US data on EU failure, bootstrap key
- parseEntsoEPrice regex now matches negative prices (-?[\d.]+)
- On EU coverage below threshold, still write US EIA data instead of
discarding it. EU keys get TTL extended as before.
- Add electricityPrices to bootstrap.js BOOTSTRAP_CACHE_KEYS + SLOW_KEYS
- Add negative price tests
* fix(electricity): add to shared BOOTSTRAP_CACHE_KEYS for RPC path parity
|
||
|
Elie Habib
|
5494577a4d |
feat(energy): EIA SPR levels and refinery utilization (#2710)
* feat(seeds): EIA SPR levels and refinery utilization rates - Add fetchSprLevels() fetching WCSSTUS1 (Strategic Petroleum Reserve) from EIA /v2/petroleum/stoc/wstk/data/ - Add fetchRefineryUtilization() fetching WCRFPUS2 (refinery utilization %) from EIA /v2/petroleum/pnp/wiup/data/ - Export parseEiaSprRow and parseEiaRefineryRow helpers for testability - Both integrated into fetchAll() via Promise.allSettled; written with writeExtraKeyWithMeta at SPR_TTL/REFINERY_TTL (21 days, 3x weekly cadence) - Add SPR_KEY and REFINERY_UTIL_KEY exports to server/_shared/cache-keys.ts - Register both keys in api/health.js BOOTSTRAP_KEYS and SEED_META (maxStaleMin: 20160) - 25 passing unit tests in tests/economy-eia-spr-seed.test.mjs * fix(seeds): switch refinery series WCRFPUS2→WCRRIUS2 (EIA v2 exposes inputs not %) * fix(seeds): add isMain guard to seed-economy.mjs (fixes CI test import side-effect) * fix(seeds): rename refineryUtil→refineryInputs, export TTL constants, fix tautological tests - Rename REFINERY_UTIL_KEY → REFINERY_INPUTS_KEY and 'economic:refinery-util:v1' → 'economic:refinery-inputs:v1' in cache-keys.ts, health.js, and seed-economy.mjs. The seeded data is crude oil input volume (WCRRIUS2, MBBL/D), not a utilization rate (%). Keeping 'util' in the key would cause future consumers to mislabel the metric as a percentage. - Export SPR_TTL and REFINERY_INPUTS_TTL from seed-economy.mjs so tests can import them directly instead of copying local literals. - Replace the four tautological constant/TTL tests in economy-eia-spr-seed.test.mjs with tests that import the real exported values and assert consumer payload shape. The old tests compared hardcoded locals to themselves and would pass even after a key rename or TTL change. - Add a comment to the SPR payload shape test warning consumers not to divide barrels again (values are already in M bbl as returned by EIA WCSSTUS1). * fix(seeds): rename fetchRefineryUtilization→fetchRefineryInputs |
||
|
Elie Habib
|
e86de6ffec |
feat(energy): EU country-level gas storage via GIE AGSI+ (#2709)
* feat(seeds): EU country-level gas storage via GIE AGSI+ * fix(seeds): use correct GIE AGSI+ country param (country=ISO2 not country_code=) * fix(seeds): preserve old fetchedAt on failure, raise MIN_VALID_COUNTRIES to 24 - preservePreviousSnapshot now reads the existing seed-meta and reuses its fetchedAt instead of stamping Date.now(). A failed run that keeps extending the snapshot no longer resets the clock, so health staleness detection fires correctly after maxStaleMin elapses. - health.js (both BOOTSTRAP_KEYS and STANDALONE_KEYS blocks): treat meta.status === 'error' as immediately stale, independent of fetchedAt. - Raise MIN_VALID_COUNTRIES from 15 to 24 (85% of 28) so a partial upstream outage dropping several countries doesn't silently publish a truncated dataset. Tests updated to match. * fix(seeds): empty-string fallthrough in parseFillEntry, drop unused watch path - parseFillEntry: switch numeric field selectors from ?? to || so that empty-string API responses (full: "", gasInStorage: "") fall through to the next candidate instead of producing NaN and silently dropping the country. Date field keeps ?? since an empty string is the intended sentinel for missing dates. - Add two tests covering the empty-string fallthrough for fill and gwh. - preservePreviousSnapshot: remove GAS_STORAGE_META_KEY from extendExistingTtl — the SET that follows handles its TTL, making the EXPIRE redundant. - railway-set-watch-paths: drop _country-resolver.mjs from the seed-gas-storage-countries watch list; the seed does not import it. |
||
|
Elie Habib
|
f210c5511a |
feat(regulatory): add tier classification and Redis publish (#2691)
* feat(regulatory): add tier classification and Redis publish Builds on the fetch/parse layer from #2564. Adds keyword-based tier classification (high/medium/low/unknown) and publishes to Redis via runSeed with 6h TTL. - HIGH: enforcement, fraud, penalty, injunction, etc. - MEDIUM: rulemaking, guidance, investigation, etc. - LOW: routine notices matching title patterns - Register REGULATORY_ACTIONS_KEY in cache-keys.ts Closes #2493 Co-authored-by: Lucas Passos <lspassos1@users.noreply.github.com> * fix(regulatory): reject empty payloads, add health monitoring - validateFn now requires actions.length > 0 to prevent overwriting a healthy snapshot with an empty one on parser regression - Register regulatory:actions:v1 in STANDALONE_KEYS (api/health.js) - Add seed-meta:regulatory:actions to SEED_META (maxStaleMin: 360, 3x the 2h cron interval) - Add seed-health.js monitoring (intervalMin: 120) --------- Co-authored-by: Lucas Passos <lspassos1@users.noreply.github.com> |
||
|
Fayez Bast
|
8609ad1384 |
feat: " climate disasters alerts seeders " (#2550)
* Revert "Revert "feat(climate): add climate disasters seed + ListClimateDisast…"
This reverts commit
|
||
|
Elie Habib
|
e7508a6e8d |
feat(energy/phase-1): OWID energy mix + per-country exposure index (#2684)
* feat(energy/phase-1): ingest OWID energy mix + per-country exposure index
Phase 1 of energy data expansion. Grounds WM Analyst and resilience scores
in real per-country generation mix data rather than a single import-dependency
metric. Subsequent phases will add live LNG/electricity/coal price feeds.
Changes:
- scripts/seed-owid-energy-mix.mjs: new Railway monthly cron that fetches
OWID CSV (~200 countries), parses latest-year generation shares
(coal/gas/oil/nuclear/renewables), and writes energy:mix:v1:{ISO2} +
energy:exposure:v1:index (top-20 per fuel type). Coverage gates: min 150
countries, max 15% regression vs previous run. Failure path extends TTL.
- server/worldmonitor/intelligence/v1/chat-analyst-context.ts: new
energyExposure field fetched from energy:exposure:v1:index; included for
economic + geo domain focus so analyst can cite specific exposed countries
- server/worldmonitor/intelligence/v1/chat-analyst-prompt.ts: Energy Exposure
section injected after macroSignals; added to economic + geo DOMAIN_SECTIONS
- server/worldmonitor/resilience/v1/_dimension-scorers.ts: scoreEnergy()
upgraded from 2 metrics to 5 (importDep 35%, gasShare 20%, coalShare 15%,
renewShare 20% inverted, priceStress 10%); reads energy:mix:v1:{ISO2}
- server/worldmonitor/intelligence/v1/get-country-intel-brief.ts: energy mix
injected into LLM context when generating country briefs
- api/health.js + server/_shared/cache-keys.ts: health monitoring for new keys
- tests: fixtures and assertions updated for all affected subsystems
* fix(energy/phase-1): extend TTL on per-country keys in failure preservation path
preservePreviousSnapshot() was only extending TTL on the exposure index
and meta keys. On repeated failures near the 35-day TTL boundary, all
~185 energy:mix:v1:{ISO2} keys could expire while the index survived,
causing scoreEnergy() to silently degrade to 2-metric blend for every
country without any health alert.
Fix: read the existing exposure index on failure, extract known ISO2
codes from all fuel arrays, and include all per-country keys in the
extendExistingTtl call.
* fix(energy/phase-1): three correctness issues from review
1. Meta TTL too short: OWID_META_KEY was expiring after 7 days on a monthly
cron, disabling the MAX_DROP_PCT regression gate and causing health.js to
report energyExposure as stale for most of the month. Changed to
OWID_TTL_SECONDS (35 days) on both success and error paths.
2. Failure preservation incomplete: preservePreviousSnapshot() was recovering
ISO2 codes from the exposure index top-20 buckets, leaving countries not
in any top-20 without TTL extension. Fix: write energy:mix:v1:_countries
(full ISO2 list) on every successful run; failure path reads this key to
extend TTL on all per-country keys unconditionally.
3. Country brief cache not invalidated on energy data updates: the cache key
was hashed from context+framework only, so updated OWID annual data was
silently ignored in cached briefs. Fix: fetch energy mix before key
computation and include the data year as :eYYYY suffix in the cache key;
also eliminates the duplicate getCachedJson call.
* fix(energy/phase-1): buildExposureIndex filters per-metric + add unit tests
Bug: buildExposureIndex pre-filtered the full country list to only those
with gasShare|coalShare non-null, then used that restricted list for oil,
imported, and renewable rankings too. Countries with valid oil/import/
renewables data but no gas/coal value (e.g. oil-only producers) were
silently excluded from those buckets.
Fix: each bucket now filters only on its own metric from the full country
set. Also: year derived from all countries, not the pre-filtered subset.
Tests (tests/owid-energy-mix-seed.test.mjs):
- oil-only country appears in oil/imported buckets, not gas/coal
- MT (null gas/coal, valid import) correctly appears in imported bucket
- each bucket sorted descending by share
- top entry per bucket matches expected country from fixture data
- cap at 20 entries enforced
- all-null year values return null without throwing
- exported key constants match expected naming contract
- OWID_TTL_SECONDS covers the monthly cron cadence
* fix(energy/phase-1): skip energyExposure fetch for market/military domains
assembleAnalystContext() was fetching energy:exposure:v1:index on every
call regardless of domainFocus, even for market and military where
DOMAIN_SECTIONS intentionally excludes energyExposure. The data was
fetched, parsed, and silently discarded — a wasted Redis round-trip on
every market/military analyst query.
Fix: gate the fetch behind ENERGY_EXPOSURE_DOMAINS (geo, economic, all).
Also exclude energyExposureResult from the degraded failCount when it was
not fetched, so market/military degraded detection is unaffected.
|
||
|
Elie Habib
|
4e9f25631c |
feat(economic): add FAO Food Price Index panel (#2682)
* feat(economic): add FAO Food Price Index panel Adds a new panel tracking the FAO Global Food Price Index (FFPI) for the past 12 months, complementing existing consumer prices, fuel prices, and Big Mac Index trackers. - proto: GetFaoFoodPriceIndex RPC with 6-series response (Food, Cereals, Meat, Dairy, Oils, Sugar + MoM/YoY pct) - seeder: seed-fao-food-price-index.mjs with 90-day TTL (3× monthly), isMain guard, parseVal NaN safety, correct 13-point slice - handler/gateway: static tier RPC wired into economicHandler - bootstrap/health: bootstrapped as SLOW_KEY; maxStaleMin=86400 (60 days) - panel: SVG multi-line chart with 6 series, auto-scaled Y axis, headline with MoM/YoY indicators, info tooltip, bootstrap hydration - CMD+K: panel:fao-food-price-index with fao/ffpi/food keywords - Railway: fao-ffpi cron seeder service (0.5 vCPU, 0.5 GB, daily 08:45) - locales: full en.json keys for panel UI strings - ais-relay: faoFoodPriceIndex added to economic bootstrap context * fix(economic): add faoFoodPriceIndex to cache-keys.ts BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS * fix(economic): correct cron comment in fao seeder to reflect daily schedule |
||
|
Fayez Bast
|
02f55dc584 |
feat(climate): add ocean ice indicators seed and RPC (#2652)
* feat(climate): add ocean ice indicators seed and RPC * fix(review): restore MCP maxStaleMin, widen health threshold, harden sea level parser, type globe.gl shim - Restore get_climate_data _maxStaleMin to 2880 (was accidentally lowered to 1440) - Bump oceanIce SEED_META maxStaleMin from 1440 to 2880 (2× daily interval, tolerates one missed run) - Add fallback regex patterns for NASA sea level overlay HTML parsing - Replace globe.gl GlobeInstance `any` with typed interface (index sig stays `any` for Three.js compat) * fix(review): merge prior cache on partial failures, fix fallback regex, omit trend without baseline - P1: fetchOceanIceData() now reads prior cache and merges last-known-good indicators when any upstream source fails, preventing partial overwrites from erasing previously healthy data - P1: sea level fallback regex now requires "current" context to avoid matching the historical 1993 baseline rate instead of the current rate - P2: classifyArcticTrend() returns null (omitted from payload) when no climatology baseline exists, instead of misleadingly labeling as "average" - Added tests for all three fixes * fix(review): merge prior cache by source field group, not whole object Prior-cache merge was too coarse: Object.assign(payload, priorCache) reintroduced stale arctic_extent_anomaly_mkm2 and arctic_trend from prior cache when sea-ice succeeded but intentionally omitted those fields (no climatology baseline), and an unrelated source like OHC or sea level failed in the same run. Fix: define per-source field groups (seaIce, seaLevel, ohc, sst). Only fall back to prior cache fields for groups whose source failed entirely. When a source succeeds, only its returned fields appear in the payload, even if it omits fields it previously provided. Added test covering the exact combined case: sea-ice climatology unavailable + unrelated source failure + prior-cache merge enabled. --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
9d94ad36aa |
feat(climate+health):add shared air quality seed and mirrored health (#2634)
* feat(climate+health):add shared air quality seed and mirrored health/climate RPCs * feat(climate+health):add shared air quality seed and mirrored health/climate RPCs * fix(air-quality): address review findings — TTL, seed-health, FAST_KEYS, shared meta - Raise CACHE_TTL from 3600 to 10800 (3× the 1h cron cadence; gold standard) - Add health:air-quality to api/seed-health.js SEED_DOMAINS so monitoring dashboard tracks freshness - Remove climateAirQuality and healthAirQuality from FAST_KEYS (large station payloads; load in slow batch) - Point climateAirQuality SEED_META to same meta key as healthAirQuality (same seeder run, one source of truth) * fix(bootstrap): move air quality keys to SLOW tier — large station payloads avoid critical-path batch * fix(air-quality): fix malformed OpenAQ URL and remove from bootstrap until panel exists - Drop deprecated first URL attempt (parameters=pm25, order_by=lastUpdated, sort=desc); use correct v3 params (parameters_id=2, sort_order=desc) directly — eliminates guaranteed 4xx retry cycle per page on 20-page crawl - Remove climateAirQuality and healthAirQuality from BOOTSTRAP_CACHE_KEYS, SLOW_KEYS, and BOOTSTRAP_TIERS — no panel consumes these yet; adding thousands of station records to every startup bootstrap is pure payload bloat - Remove normalizeAirQualityPayload helpers from bootstrap.js (no longer called) - Update service wrappers to fetch via RPC directly; re-add bootstrap hydration when a panel actually needs it * fix(air-quality): raise lock TTL to 3600s to cover 20-page crawl worst case 2 OpenAQ calls × 20 pages × (30s timeout × 3 attempts) = 3600s max runtime. Previous 600s TTL allowed concurrent cron runs on any degraded upstream. --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
d04852bf02 |
fix(relay): proxy fallback for Yahoo/Crypto, isolate OREF proxy (#2627)
* fix(relay): proxy fallback for Yahoo/Crypto, isolate OREF proxy, fix Dockerfile Yahoo Finance and CoinPaprika fail from Railway datacenter IPs (rate limiting). Added PROXY_URL fallback to fetchYahooChartDirect (used by 5 seeders) and relay chart proxy endpoint. Added shared _fetchCoinPaprikaTickers with proxy fallback + 5min cache (3 crypto seeders share one fetch). Added CoinPaprika fallback to CryptoSectors (previously had none). Isolated OREF_PROXY_AUTH exclusively for OREF alerts. OpenSky, seed-military-flights, and _proxy-utils now fall back to PROXY_URL instead of the expensive IL-exit proxy. Added seed-climate-news.mjs + _seed-utils.mjs COPY to Dockerfile.relay (missing since PR #2532). Added pizzint bootstrap hydration to cache-keys.ts, bootstrap.js, and src/services/pizzint.ts. * fix(relay): address review — remove unused reverseMap, guard double proxy - Remove dead reverseMap identity map in CryptoSectors Paprika fallback - Add _proxied flag to handleYahooChartRequest._tryProxy to prevent double proxy call on timeout→destroy→error sequence |
||
|
Elie Habib
|
c51717e76a |
feat(digest): daily digest notification mode (#2614)
* feat(digest): add daily digest notification mode (Enhancement 2) - convex/schema.ts: add digestMode/digestHour/digestTimezone to alertRules - convex/alertRules.ts: setDigestSettings mutation, setDigestSettingsForUser internal mutation, getDigestRules internal query - convex/http.ts: GET /relay/digest-rules for Railway cron; set-digest-settings action in /relay/notification-channels - cache-keys.ts: DIGEST_LAST_SENT_KEY + DIGEST_ACCUMULATOR_TTL (48h); fix accumulator EXPIRE to use 48h instead of 7-day STORY_TTL - notification-relay.cjs: skip digest-mode rules in processEvent — prevents daily/weekly users from receiving both real-time and digest messages - seed-digest-notifications.mjs: new Railway cron (every 30 min) — queries due rules, ZRANGEBYSCORE accumulator, batch HGETALL story tracks, derives phase, formats digest per channel, updates digest:last-sent - notification-channels.ts: DigestMode type, digest fields on AlertRule, setDigestSettings() client function - api/notification-channels.ts: set-digest-settings action * fix(digest): correct twice_daily scheduling and only advance lastSent on confirmed delivery isDue() only checked a single hour slot, so twice_daily users got one digest per day instead of two. Now checks both primaryHour and (primaryHour+12)%24 for twice_daily. All four send functions returned void and errors were swallowed, causing dispatched=true to be set unconditionally. Replaced with boolean returns and anyDelivered guard so lastSentKey is only written when at least one channel confirms a 2xx delivery. * fix(digest): add discord to deactivate allowlist, bounds-check digestHour, minor cleanup /relay/deactivate was rejecting channelType="discord" with 400, so stale Discord webhooks were never auto-deactivated. Added "discord" to the validation guard. Added 0-23 integer bounds check for digestHour in both setDigestSettings mutations to reject bad values at the DB layer rather than silently storing them. Removed unused createHash import and added AbortSignal.timeout(10000) to upstashRest to match upstashPipeline and prevent cron hangs. * fix(daily-digest): add DIGEST_CRON_ENABLED guard, IANA timezone validation, and Digest Mode UI - seed-digest-notifications.mjs: exit 0 when DIGEST_CRON_ENABLED=0 so Railway cron does not error on intentionally disabled runs - convex/alertRules.ts: validate digestTimezone via Intl.DateTimeFormat; throw ConvexError with descriptive message for invalid IANA strings - preferences-content.ts: add Digest Mode section with mode select (realtime/ daily/twice_daily/weekly), delivery hour select, and timezone input; details panel hidden in realtime mode; wired to setDigestSettings with 800ms debounce Fixes gaps F, G, I from docs/plans/2026-04-02-003-fix-news-alerts-pr-gaps-plan.md * fix(digest): close digest blackhole and wire timezone validation through internal mutation - convex/alertRules.ts: add IANA timezone validation to setDigestSettingsForUser (internalMutation called by http.ts); the public mutation already validated but the edge/relay path bypassed it - preferences-content.ts: add VITE_DIGEST_CRON_ENABLED browser flag; when =0, disable the digest mode select and show only Real-time with a note so users cannot enter a blackhole state where the relay skips their rule and the cron never runs Addresses P1 and P2 review findings on #2614 * fix(digest): restore missing > closing the usDigestDetails div opening tag * feat(digest): redesign email to match WorldMonitor design system Dark theme (#0a0a0a bg, #111 cards), #4ade80 green accent, 4px top bar, table-based logo header, severity-bucketed story cards with colored left borders, stats row (total/critical/high), green CTA button. Plain text fallback preserved for Telegram/Slack/Discord channels. * test(digest): add rollout-flag and timezone-validation regression tests Covers three paths flagged as untested by reviewers: - VITE_DIGEST_CRON_ENABLED gates digest-mode options and usDigestDetails visibility - setDigestSettings (public) validates digestTimezone via Intl.DateTimeFormat - setDigestSettingsForUser (internalMutation) also validates digestTimezone to prevent silent bypass through the edge-to-Convex path |
||
|
Elie Habib
|
6c017998d3 |
feat(e3): story persistence tracking (#2620)
* feat(e3): story persistence tracking
Adds cross-cycle story tracking layer to the RSS digest pipeline:
- Proto: StoryMeta message + StoryPhase enum on NewsItem (fields 9-11).
importanceScore and corroborationCount stubs added for E1.
- list-feed-digest.ts: builds corroboration map across ALL items before
truncation; batch-reads existing story:track hashes from Redis; writes
HINCRBY/HSET/HSETNX/SADD/EXPIRE per story in 80-story pipeline chunks;
attaches StoryMeta (firstSeen, mentionCount, sourceCount, phase) to
each proto item using read-back data.
- cache-keys.ts: STORY_TRACK_KEY_PREFIX, STORY_SOURCES_KEY_PREFIX,
DIGEST_ACCUMULATOR_KEY_PREFIX, STORY_TRACKING_TTL_S.
- src/types/index.ts: StoryMeta, StoryPhase, NewsItem extended.
- data-loader.ts: protoItemToNewsItem maps STORY_PHASE_* → client phase.
- NewsPanel.ts: BREAKING/DEVELOPING/ONGOING phase badges in item rows.
New story first appearance: phase=BREAKING. After 2 mentions within 2h:
DEVELOPING. After 6+ mentions or >2h: SUSTAINED. If score drops below
50% of peak: FADING (used by E1; defaults to SUSTAINED for now).
Redis keys per story (48h TTL):
story:track:v1:<hash16> → hash (firstSeen,lastSeen,mentionCount,...)
story:sources:v1:<hash16> → set (feed names, for cross-source count)
* fix(e3): correct storyMeta staleness and mentionCount semantics
P1 — storyMeta was always one cycle behind because storyTracks was read
before writeStoryTracking ran. Fix: keep read-before-write but compute
storyMeta from merged in-memory state (stale.mentionCount + 1, fresh
sourceCount from corroborationMap). New stories get mentionCount=1 and
phase=BREAKING in the same cycle they first appear — no extra Redis
round-trip needed.
P2 — mentionCount incremented once per item occurrence, so a story seen
in 3 sources in its first cycle was immediately stored as mentionCount=3.
Fix: deduplicate by titleHash in writeStoryTracking so each unique story
gets exactly one HINCRBY per digest cycle regardless of source count.
SADD still collects all sources for the set key.
* fix(e3): Unicode hash collision, ALERT badge regression, FADING comment
P1 — normalizeTitle used [^\w\s] without the u flag; \w is ASCII-only
so every Arabic/CJK/Cyrillic title stripped to "" and shared one Redis
hash. Fixed: use /[^\p{L}\p{N}\s]/gu (Unicode property escapes require
the u flag).
P1 — ALERT badge was gated on !item.storyMeta, suppressing the indicator
for any tracked story regardless of isAlert. Phase and alert are
orthogonal signals; ALERT now renders unconditionally when isAlert=true.
P2 — FADING branch is intentionally inactive until E1 ships real scores
(currentScore/peakScore placeholder 0 via HSETNX). Added comment to
document the intentional ordering.
* fix(news-alerts): skip sustained/fading stories in breaking alert selectBest
Sustained and fading story phases are already well-covered by the feed;
only breaking and developing phases warrant a banner interrupt. Items
without storyMeta (phase unspecified) pass through unchanged.
Fixes gap C from docs/plans/2026-04-02-003-fix-news-alerts-pr-gaps-plan.md
* fix(e3): remove rebase artifacts from list-feed-digest
Removes a stray closing brace, duplicate ASCII normalizeTitle
(Unicode-aware version from the fix commit is correct), and
a leftover storyPhase assignment that references a removed field.
All typecheck and typecheck:api pass clean.
|
||
|
Elie Habib
|
8d8cf56ce2 |
feat(scoring): composite importance score + story tracking infrastructure (#2604)
* feat(scoring): add composite importance score + story tracking infrastructure
- Extract SOURCE_TIERS/getSourceTier to server/_shared/source-tiers.ts so
server handlers can import it without pulling in client-only modules;
src/config/feeds.ts re-exports for backward compatibility
- Add story tracking Redis key helpers to cache-keys.ts
(story:track:v1, story:sources:v1, story:peak:v1, digest:accumulator:v1)
- Export SEVERITY_SCORES from _classifier.ts for server-side score math
- Add upstashPipeline() to redis.ts for arbitrary batched Redis writes
- Add importanceScore/corroborationCount/storyPhase fields to proto,
generated TS, and src/types NewsItem
- Add StoryMeta message and StoryPhase enum to proto
- In list-feed-digest.ts:
- Build corroboration map across full corpus BEFORE per-category truncation
- Compute importanceScore (severity 40% + tier 20% + corroboration 30%
+ recency 10%) per item
- Sort by importanceScore desc before truncating at MAX_ITEMS_PER_CATEGORY
- Write story:track / story:sources / story:peak / digest:accumulator
to Redis in 80-story pipeline batches after each digest build
Score gate in notification-relay.cjs follows in the next PR (shadow mode,
behind IMPORTANCE_SCORE_LIVE flag). RELAY_GATES_READY removal of
/api/notify comes after 48h shadow comparison confirms parity.
* fix(scoring): add storyPhase field + regenerate proto types
- Add storyPhase to ParsedItem and toProtoItem (defaults UNSPECIFIED)
- Regenerate service_server.ts: required fields, StoryPhase type relocated
- Regenerate service_client.ts and OpenAPI docs from buf generate
- Fix typecheck:api error on missing required storyPhase in NewsItem
* fix(scoring): address all code review findings from PR #2604
P1:
- await writeStoryTracking instead of fire-and-forget to prevent
silent data loss on edge isolate teardown
- remove duplicate upstashPipeline; use existing runRedisPipeline
- strip non-https links before Redis write (XSS prevention)
- implement storyPhase read path: HGETALL batch + computePhase()
so BREAKING/DEVELOPING/SUSTAINED/FADING badges are now live
P2/P3:
- extend STORY_TTL 48h → 7 days (sustained stories no longer reset)
- extract SCORE_WEIGHTS named constants with rationale comment
- move SEVERITY_SCORES out of _classifier.ts into list-feed-digest.ts
- add normalizeTitle comment referencing todo #102
- pre-compute title hashes once, share between phase read + write
* fix(scoring): correct enrichment-before-scoring and write-before-read ordering
Two sequencing bugs:
1. enrichWithAiCache ran after truncation (post-slice), so items whose
threat level was upgraded by the LLM cache could have already been
cut from the top-20, and downgraded items kept inflated scores.
Fix: enrich ALL items from the full corpus before scoring, so
importanceScore always uses the final post-LLM classification level.
2. Phase HGETALL read happened before writeStoryTracking, meaning
first-time stories had no Redis entry and always returned UNSPECIFIED
instead of BREAKING, and all existing stories lagged one cycle behind.
Fix: write tracking first, then read back for phase assignment.
|
||
|
Fayez Bast
|
b2bae30bd8 |
Add climate news seed and ListClimateNews RPC (#2532)
* Add climate news seed and ListClimateNews RPC * Wire climate news into bootstrap and fix generated climate stubs * fix(climate): align seed health interval and parse Atom entries per feed * fix(climate-news): TTL 90min, retry timer on failure, named cache key constant - CACHE_TTL: 1800 to 5400 (90min = 3x 30-min relay interval, gold standard) - ais-relay: add 20-min retry timer on subprocess failure; clear on success - cache-keys.ts: export CLIMATE_NEWS_KEY named constant - list-climate-news.ts: import CLIMATE_NEWS_KEY instead of hard-coding string --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
bb4f8dcb12 |
feat(climate): add WMO normals seeding and CO2 monitoring (#2531)
* feat(climate): add WMO normals seeding and CO2 monitoring * fix(climate): skip missing normals per-zone and align anomaly tooltip copy * fix(climate): remove normals from bootstrap and harden health/cache key wiring * feat(climate): version anomaly cache to v2, harden seed freshness, and align CO2/normal baselines |
||
|
Elie Habib
|
ae4010a795 |
Revert "feat(climate): add climate disasters seed + ListClimateDisasters RPC with snake_case Redis payload (#2535)" (#2544)
This reverts commit
|
||
|
Fayez Bast
|
e2dea9440d | feat(climate): add climate disasters seed + ListClimateDisasters RPC with snake_case Redis payload (#2535) | ||
|
Elie Habib
|
aa3e84f0ab |
feat(economic): Economic Stress Composite Index panel (FRED 6-series, 0-100 score) (#2461)
* feat(economic): Economic Stress Composite Index panel (FRED 6-series, 0-100 score) - Add T10Y3M and STLFSI4 to FRED_SERIES in seed-economy.mjs and ALLOWED_SERIES - Create proto message GetEconomicStressResponse with EconomicStressComponent - Register GetEconomicStress RPC in EconomicService (GET /api/economic/v1/get-economic-stress) - Add seed-economic-stress.mjs: reads 6 pre-seeded FRED keys via Redis pipeline, computes weighted composite score (0-100) with labels Low/Moderate/Elevated/Severe/Critical - Create server handler get-economic-stress.ts reading from economic:stress-index:v1 - Register economicStress in BOOTSTRAP_CACHE_KEYS (both cache-keys.ts and api/bootstrap.js) as slow tier - Add gateway.ts cache tier entry (slow) for new RPC route - Create EconomicStressPanel.ts: composite score header, gradient needle bar, 2x3 component grid with score bars, desktop notification on threshold cross (>=70, >=85) - Wire economic-stress panel in panels.ts (all 4 variants), panel-layout.ts, and data-loader.ts - Regenerate OpenAPI docs and TypeScript client/server types * fix(economic-stress): null for missing FRED data + tech variant panel - Add 'economic-stress' panel to TECH_PANELS defaults (was missing, only appeared in full/finance/commodity variants) - Seed: write rawValue: null + missing: true when no valid FRED observation found, preventing zero-valued yield curve/bank spread readings from being conflated with missing data - Proto: add missing bool field to EconomicStressComponent message; regenerate client/server types + OpenAPI docs - Server handler: propagate missing flag from Redis; pass rawValue: 0 on wire when missing to satisfy proto double type - Panel: guard on c.missing (not rawValue === 0) to show grey N/A card with no score bar for unavailable components * fix(economic-stress): add purple Critical zone to gradient bar Update gradient stops to match the 5 equal tier boundaries (0-20-40-60-80-100), adding the #8e44ad purple stop at 80% so scores 80-100 render as Critical purple instead of plain red. |
||
|
Elie Habib
|
1e1f377078 |
feat(panels): Disease Outbreaks, Shipping Stress, Social Velocity, nuclear test site enrichment (#2375)
* feat(panels): Disease Outbreaks, Shipping Stress, Social Velocity, nuclear test site monitoring - Add HealthService proto with ListDiseaseOutbreaks RPC (WHO + ProMED RSS) - Add GetShippingStress RPC to SupplyChainService (Yahoo Finance carrier ETFs) - Add GetSocialVelocity RPC to IntelligenceService (Reddit r/worldnews + r/geopolitics) - Enrich earthquake seed with Haversine nuclear test-site proximity scoring - Add 5 nuclear test sites to NUCLEAR_FACILITIES (Punggye-ri, Lop Nur, Novaya Zemlya, Nevada NTS, Semipalatinsk) - Add shipping stress + social velocity seed loops to ais-relay.cjs - Add seed-disease-outbreaks.mjs Railway cron script - Wire all new RPCs: edge functions, handlers, gateway cache tiers, health.js STANDALONE_KEYS/SEED_META * fix(relay): apply gold standard retry/TTL-extend pattern to shipping-stress and social-velocity seeders * fix(review): address all PR #2375 review findings - health.js: shippingStress maxStaleMin 30→45 (3x interval), socialVelocity 20→30 (3x interval) - health.js: remove shippingStress/diseaseOutbreaks/socialVelocity from ON_DEMAND_KEYS (relay/cron seeds, not on-demand) - cache-keys.ts: add shippingStress, diseaseOutbreaks, socialVelocity to BOOTSTRAP_CACHE_KEYS - ais-relay.cjs: stressScore formula 50→40 (neutral market = moderate, not elevated) - ais-relay.cjs: fetchedAt Date.now() (consistent with other seeders) - ais-relay.cjs: deduplicate cross-subreddit article URLs in social velocity loop - seed-disease-outbreaks.mjs: WHO URL → specific DON RSS endpoint (not dead general news feed) - seed-disease-outbreaks.mjs: validate() requires outbreaks.length >= 1 (reject empty array) - seed-disease-outbreaks.mjs: stable id using hash(link) not array index - seed-disease-outbreaks.mjs: RSS regexes use [\s\S]*? for CDATA multiline content - seed-earthquakes.mjs: Lop Nur coordinates corrected (41.39,89.03 not 41.75,88.35) - seed-earthquakes.mjs: sourceVersion bumped to usgs-4.5-day-nuclear-v1 - earthquake.proto: fields 8-11 marked optional (distinguish not-enriched from enriched=false/0) - buf generate: regenerate seismology service stubs * revert(cache-keys): don't add new keys to bootstrap without frontend consumers * fix(panels): address all P1/P2/P3 review findings for PR #2375 - proto: add INT64_ENCODING_NUMBER annotation + sebuf import to get_shipping_stress.proto (run make generate) - bootstrap: register shippingStress (fast), socialVelocity (fast), diseaseOutbreaks (slow) in api/bootstrap.js + cache-keys.ts - relay: update WIDGET_SYSTEM_PROMPT with new bootstrap keys and live RPCs for health/supply-chain/intelligence - seeder: remove broken ProMED feed URL (promedmail.org/feed/ returns HTML 404); add 500K size guard to fetchRssItems; replace private COUNTRY_CODE_MAP with shared geo-extract.mjs; remove permanently-empty location field; bump sourceVersion to who-don-rss-v2 - handlers: remove dead .catch from all 3 new RPC handlers; fix stressLevel fallback to low; fix fetchedAt fallback to 0 - services: add fetchShippingStress, disease-outbreaks.ts, social-velocity.ts with getHydratedData consumers |
||
|
Elie Habib
|
e3b863d30f |
feat(panels): EU data tabs for Energy, Macro, Yield, Commodities panels (#2355)
* feat(panels): add EU data tabs to EnergyComplex, MacroTiles, YieldCurve, Commodities - EnergyComplexPanel: US Nat Gas Storage (EIA) + EU Gas Storage (GIE AGSI+) sections below crude - MacroTilesPanel: US/EU tab toggle; EU shows avg HICP/unemployment/GDP for DE/FR/IT/ES + ECB ESTR - YieldCurvePanel: Curve/ECB Rates tab; Rates shows ESTR + EURIBOR 3M/6M/1Y sparklines via FRED - CommoditiesPanel: Commodities/EUR FX tab; FX shows ECB EUR pairs (USD/GBP/JPY/CHF/CAD/CNY/AUD) - Add getEuGasStorageData() + getEurostatCountryData() service functions with circuit breakers - Register eurostatCountryData in bootstrap.js + cache-keys.ts (SLOW tier) - Wire fetchNatGasStorageRpc, getEuGasStorageData, getEcbFxRatesData in data-loader.loadOilAnalytics * fix(commodities-panel): always re-render on data update so FX tab bar appears |
||
|
Elie Habib
|
9480b547d5 |
feat(feeds): US Natural Gas Storage weekly seeder (EIA NW2_EPG0_SWO_R48_BCF) (#2353)
* feat(feeds): US Natural Gas Storage seeder via EIA (NW2_EPG0_SWO_R48_BCF)
Adds weekly EIA natural gas working gas storage for the Lower-48 states
(series NW2_EPG0_SWO_R48_BCF, in Bcf), mirroring the crude inventories
pattern exactly. Companion dataset to EU gas storage (GIE AGSI+).
- proto: GetNatGasStorage RPC + NatGasStorageWeek message
- seed-economy.mjs: fetchNatGasStorage() in Promise.allSettled, writes
economic:nat-gas-storage:v1 with 21-day TTL (3x weekly cadence)
- server handler: getNatGasStorage reads seeded key from Redis
- gateway: /api/economic/v1/get-nat-gas-storage → static tier
- health.js: BOOTSTRAP_KEYS + SEED_META (14-day maxStaleMin)
- bootstrap.js: KEYS + SLOW_KEYS
- cache-keys.ts: BOOTSTRAP_CACHE_KEYS + BOOTSTRAP_TIERS (slow)
* feat(feeds): add fetchNatGasStorageRpc consumer in economic service
Adds getHydratedData('natGasStorage') consumer required by bootstrap
key registry test, plus circuit breaker and RPC wrapper mirroring
fetchCrudeInventoriesRpc pattern.
|
||
|
Elie Habib
|
4438ef587f |
feat(feeds): ECB CISS European financial stress index (#2278) (#2334)
* feat(feeds): ECB CISS European financial stress seeder + GetEuFsi RPC (#2278) - New seed-fsi-eu.mjs fetches ECB CISS (0-1 systemic stress index for Euro area) via SDMX-JSON REST API (free, no auth); TTL=604800s (7d, weekly data cadence) - New GetEuFsi RPC in EconomicService with proto + handler; cache tier: slow - FSIPanel now shows EU CISS gauge below US FSI with label thresholds: Low<0.2, Moderate<0.4, Elevated<0.6, High>=0.6 - Registered economic:fsi-eu:v1 in health.js BOOTSTRAP_KEYS + SEED_META, bootstrap.js, cache-keys.ts BOOTSTRAP_TIERS; hydrated via getHydratedData('euFsi') - All 2348 test:data tests pass; typecheck + typecheck:api clean * fix(ecb-ciss): address code review findings on PR #2334 - Raise FSI_EU_TTL from 604800s (7d) to 864000s (10d) to match other weekly seeds (bigmac, groceryBasket, fuelPrices) and provide a 3-day buffer against cron-drift or missed Saturday runs - Format latestDate via toLocaleDateString() in FSIPanel CISS section instead of displaying the raw ISO string (e.g. "2025-04-04") * fix(ecb-ciss): address Greptile review comments on PR #2334 - Fix misleading "Daily frequency" comment in seed-fsi-eu.mjs (SDMX uses 'D' series key but only Friday/weekly observations are present) - Replace latestValue > 0 guards with Number.isFinite() in FSIPanel.ts so a valid CISS reading of exactly 0 is not incorrectly excluded * chore: regenerate proto outputs after rebase |
||
|
Elie Habib
|
b6847e5214 |
feat(feeds): GIE AGSI+ EU gas storage seeder (#2281) (#2339)
* feat(feeds): GIE AGSI+ EU gas storage seeder — European energy security indicator (#2281) - New scripts/seed-gie-gas-storage.mjs: fetches EU aggregate gas storage fill % from GIE AGSI+ API, computes 1-day change, trend (injecting/withdrawing/stable), and days-of-consumption estimate; TTL=259200s (3x daily); isMain guard; CHROME_UA; validates fillPct in (0,100]; graceful degradation when GIE_API_KEY absent - New proto/worldmonitor/economic/v1/get_eu_gas_storage.proto + GetEuGasStorage RPC wired into EconomicService - New server/worldmonitor/economic/v1/get-eu-gas-storage.ts handler (reads seeded Redis key) - api/health.js: BOOTSTRAP_KEYS + SEED_META (maxStaleMin=2880, 2x daily cadence) - api/bootstrap.js: euGasStorage key in SLOW_KEYS bucket - Regenerated src/generated/ + docs/api/ via make generate * fix(feeds): wire euGasStorage into cache-keys, gateway tier, and test PENDING_CONSUMERS - server/_shared/cache-keys.ts: add euGasStorage → economic:eu-gas-storage:v1 (slow tier) - server/gateway.ts: add /api/economic/v1/get-eu-gas-storage → slow RPC_CACHE_TIER - tests/bootstrap.test.mjs: add euGasStorage to PENDING_CONSUMERS (no frontend panel yet) * fix(gie-gas-storage): normalize seededAt to string to match proto int64 contract Proto int64 seeded_at maps to string in JS; seed was writing Date.now() (number). Fix seed to write String(Date.now()) and add handler-side normalization for any stale Redis entries that may have the old numeric format. * fix(feeds): coerce nullable fillPctChange1d/gasDaysConsumption to 0 (#2281) Greptile P1: both fields could be null (single data-point run or missing volume) but the proto interface declares them as non-optional numbers. Seed script now returns 0 instead of null; handler defensively coerces nulls from older cached blobs via nullish coalescing. Dead null-guard on trend derivation also removed. |
||
|
Elie Habib
|
b5faffb341 |
feat(feeds): ECB reference FX rates seeder (#2280) (#2337)
* feat(feeds): ECB daily reference FX rates seeder -- EUR/USD/GBP/JPY/CHF (#2280) - New scripts/seed-ecb-fx-rates.mjs: fetches EUR/USD/GBP/JPY/CHF/CAD/AUD/CNY daily from ECB Data Portal (no API key required) - New getEcbFxRates RPC in economic service (proto + handler + gateway slow tier) - api/health.js: ecbFxRates in BOOTSTRAP_KEYS + SEED_META (2880min maxStale) - api/bootstrap.js: ecbFxRates registered as SLOW_KEYS bootstrap entry - TTL: 259200s (3x daily interval), isMain guard, CHROME_UA, validate>=3 pairs * fix(feeds): register ecbFxRates in cache-keys.ts + add frontend hydration consumer * fix(ecb-fx-rates): use dynamic CURRENCY dim position in series key parsing CURRENCY is at index 1 in EXR series keys (FREQ:CURRENCY:CURRENCY_DENOM:EXR_TYPE:EXR_SUFFIX). Using hardcoded keyParts[0] always read the FREQ field (always "0"), so all series resolved to currencyCodes[0] (AUD) and only 1 pair was written to Redis instead of all 7. Fix: find CURRENCY position via findIndex() and use keyParts[currencyDimPos] to extract the correct per-series currency index. * fix(feeds): hoist obs-dimension lookup + fix ECB UTC publication time - Hoist obsPeriods/timeDim/timeValues above the series loop (loop-invariant, P2) - Fix updatedAt timestamp: ECB publishes at 16:00 CET = 14:00 UTC (not 16:00 UTC), use T14:00:00Z (P2) * chore(generated): fix EcbFxRate schema ordering in EconomicService openapi.json buf generate places EcbFxRate alphabetically before EconomicEvent; align committed file with code-generator output so proto freshness hook passes. |
||
|
Elie Habib
|
f3b0280227 |
feat(economic): EIA weekly crude oil inventory seeder (#2142) (#2168)
* feat(economic): EIA weekly crude oil inventory seeder (#2142) - scripts/seed-economy.mjs: add fetchCrudeInventories() fetching WCRSTUS1, compute weeklyChangeMb, write economic:crude-inventories:v1 (10-day TTL) - proto/worldmonitor/economic/v1/get_crude_inventories.proto: new proto with CrudeInventoryWeek and GetCrudeInventories RPC - server/worldmonitor/economic/v1/get-crude-inventories.ts: RPC handler reading seeded key with getCachedJson(..., true) - server/worldmonitor/economic/v1/handler.ts: wire in getCrudeInventories - server/gateway.ts: add static cache tier for /api/economic/v1/get-crude-inventories - api/health.js: crudeInventories in BOOTSTRAP_KEYS + SEED_META (maxStaleMin: 20160, 2x weekly cadence) - src/services/economic/index.ts: add fetchCrudeInventoriesRpc() with circuit breaker - src/components/EnergyComplexPanel.ts: surface 8-week sparkline and WoW change in energy panel - src/app/data-loader.ts: call fetchCrudeInventoriesRpc() in loadOilAnalytics() * fix: remove stray market-implications gateway entry from crude-inventories branch * fix(crude-inventories): address ce-review P1/P2 findings before merge - api/bootstrap.js: register crudeInventories in BOOTSTRAP_CACHE_KEYS + SLOW_KEYS (P1-001) - server/_shared/cache-keys.ts: add crudeInventories key + tier to match bootstrap.js - api/health.js: remove bundled marketImplications (belongs in separate PR) (P1-002) - src/services/economic/index.ts: add isFeatureAvailable('energyEia') gate (P2-003) - src/services/economic/index.ts: use getHydratedData('crudeInventories') on first load - proto/get_crude_inventories.proto: weekly_change_mb → optional double (P2-004) - scripts/seed-economy.mjs: CRUDE_INVENTORIES_TTL 10d → 21d (3× cadence) (P2-005) - scripts/seed-economy.mjs: period format validation with YYYY-MM-DD regex (P3-007) - src/app/data-loader.ts: warn on crude fetch rejection (P2-006) * fix(crude-inventories): schema validation, MIN_ITEMS gate, handler logging, raw=true docs - Handler: document raw=true param, log errors instead of silent catch - Seeder: CRUDE_MIN_WEEKS=4 guard prevents quota-hit empty writes - Seeder: isValidWeek() schema validation before Redis write * chore: regenerate openapi docs after rebase (adds getCrudeInventories + getEconomicCalendar) * fix(gateway): add list-market-implications to RPC_CACHE_TIER * chore: exclude todos/ from markdownlint |
||
|
Elie Habib
|
01f6057389 |
feat(simulation): MiroFish Phase 2 — theater-limited simulation runner (#2220)
* feat(simulation): MiroFish Phase 2 — theater-limited simulation runner Adds the simulation execution layer that consumes simulation-package.json and produces simulation-outcome.json for maritime chokepoint + energy/logistics theaters, closing the WorldMonitor → MiroFish handoff loop. Changes: - scripts/seed-forecasts.mjs: 2-round LLM simulation runner (prompt builders, JSON extractor, runTheaterSimulation, writeSimulationOutcome, task queue with NX dedup lock, runSimulationWorker poll loop) - scripts/process-simulation-tasks.mjs: standalone worker entry point - proto: GetSimulationOutcome RPC + make generate - server/worldmonitor/forecast/v1/get-simulation-outcome.ts: RPC handler - server/gateway.ts: slow tier for get-simulation-outcome - api/health.js: simulationOutcomeLatest in STANDALONE + ON_DEMAND keys - tests: 14 new tests for simulation runner functions * fix(simulation): address P1/P2 code review findings from PR #2220 Security (P1 #018): - sanitizeForPrompt() applied to all entity/seed fields interpolated into Round 1 prompt (entityId, class, stance, seedId, type, timing) - sanitizeForPrompt() applied to actorId and entityIds in Round 2 prompt - sanitizeForPrompt() + length caps applied to all LLM array fields written to R2 (dominantReactions, stabilizers, invalidators, keyActors, timingMarkers) Validation (P1 #019): - Added validateRunId() regex guard - Applied in enqueueSimulationTask() and processNextSimulationTask() loop Type safety (P1 #020): - Added isOutcomePointer() and isPackagePointer() type guards in TS handlers - Replaced unsafe as-casts with runtime-validated guards in both handlers Correctness (P2 #022): - Log warning when pkgPointer.runId does not match task runId Architecture (P2 #024): - isMaritimeChokeEnergyCandidate() accepts both flat and nested topBucketId - Call site simplified to pass theater directly Performance (P2 #025): - SIMULATION_ROUND1_MAX_TOKENS raised 1800 to 2200 - Added max 3 initialReactions instruction to Round 1 prompt Maintainability (P2 #026): - Simulation pointer keys exported from server/_shared/cache-keys.ts - Both TS handlers import from shared location Documentation (P2 #027): - Strengthened runId no-op description in proto and OpenAPI spec * fix(todos): add blank lines around lists in markdown todo files * style(api): reformat openapi yaml to match linter output * test(simulation): add flat-shape filter test + getSimulationOutcome handler coverage Two tests identified as missing during PR #2220 review: 1. isMaritimeChokeEnergyCandidate flat-shape tests — covers the || candidate.topBucketId normalization added in the P1/P2 review pass. The existing tests only used the nested marketContext.topBucketId shape; this adds the flat root-field shape that arrives from the simulation-package.json JSON (selectedTheaters entries have topBucketId at root). 2. getSimulationOutcome handler structural tests — verifies the isOutcomePointer guard, found:false NOT_FOUND return, found:true success path, note population on runId mismatch, and redis_unavailable error string. Follows the readSrc static-analysis pattern used elsewhere in server-handlers.test.mjs (handler imports Redis so full integration test would require a test Redis instance). |
||
|
Elie Habib
|
e548e6cca5 |
feat(intelligence): cross-source signal aggregator with composite escalation (#2143) (#2164)
* feat(intelligence): cross-source signal aggregator with composite escalation (#2143) Adds a threshold-based signal aggregator seeder that reads 15+ already-seeded Redis keys every 15 minutes, ranks cross-domain signals by severity, and detects composite escalation when >=3 signal categories co-fire in the same theater. * fix(cross-source-signals): wire panel data loading, inline styles, seeder cleanup - New src/services/cross-source-signals.ts: fetch via IntelligenceServiceClient with circuit breaker - data-loader.ts: add loadCrossSourceSignals() + startup batch entry (SITE_VARIANT !== 'happy' guard) - App.ts: add primeVisiblePanelData entry + scheduleRefresh at 15min interval - base.ts: add crossSourceSignals: 15 * 60 * 1000 to REFRESH_INTERVALS - CrossSourceSignalsPanel.ts: replace all CSS class usage with inline styles (MarketImplicationsPanel pattern) - seed-cross-source-signals.mjs: remove dead isMain var, fix afterPublish double-write, deterministic signal IDs, GDELT per-topic tone keys (military/nuclear/maritime) with 3-point declining trend + < -1.5 threshold per spec, bundled topics fallback * fix(cross-source-signals): complete bootstrap wiring, seeder fixes, cmd-k entry - cache-keys.ts: add crossSourceSignals to BOOTSTRAP_CACHE_KEYS + BOOTSTRAP_TIERS (slow) - bootstrap.js: add crossSourceSignals key + SLOW_KEYS entry - cross-source-signals.ts: add getHydratedData('crossSourceSignals') bootstrap hydration - seed script: fix isDeclinig typo, maritime theater ternary (Global->Indo-Pacific), displacement year dynamic - commands.ts: add panel:cross-source-signals to cmd-k * feat(cross-source-signals): redesign panel — severity bars, filled badges, icons, theater pills - 4px severity accent bar on all signal rows (scannable without reading badges) - Filled severity badges: CRITICAL=solid red/white, HIGH=faint red bg, MED=faint yellow bg - Type badge emoji prefix: ⚡ composite, 🔴 geo-physical, 📡 EW, ✈️ military, 📊 market, ⚠️ geopolitical - Composite card: full glow (box-shadow) instead of 3px left border only - Theater pill with inline age: "Middle East · 8m ago" - Contributor pills: individual chips instead of dot-separated string - Pulsing dot on composite escalation banner * fix(cross-source-signals): code review fixes — module-level Sets, signal cap, keyframe scoping, OREF expansion - Replace per-call Array literals in list-cross-source-signals.ts with module-level Set constants for O(1) lookups - Add index-based fallback ID in normalizeSignal to avoid undefined ids - Remove unused military:flights:stale:v1 from SOURCE_KEYS - Add MAX_SIGNALS=30 cap before writing to Redis - Expand extractOrefAlertCluster to any "do not travel" advisory (not just Israel) - Add BASE_WEIGHT inline documentation explaining scoring scale - Fix animation keyframe: move from setContent() <style> block to constructor (injected once), rename to cross-source-pulse-dot - Fix GDELT extractor to read per-topic gdelt:intel:tone:{topic} keys with correct decline logic - Fix isDeclinig typo, maritime dead ternary, and displacement year reference |
||
|
Elie Habib
|
7013b2f9f1 |
feat(market): Fear & Greed Index 2.0 — 10-category composite sentiment panel (#2181)
* Add Fear & Greed Index 2.0 reverse engineering brief Analyzes the 10-category weighted composite (Sentiment, Volatility, Positioning, Trend, Breadth, Momentum, Liquidity, Credit, Macro, Cross-Asset) with scoring formulas, data source audit, and implementation plan for building it as a worldmonitor panel. https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Add seed script implementation plan to F&G brief Details exact endpoints, Yahoo symbols (17 calls), Redis key schema, computed metrics, FRED series to add (BAMLC0A0CM, SOFR), CNN/AAII sources, output JSON schema, and estimated runtime (~8s per seed run). https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Update brief: all sources are free, zero paid APIs needed - CBOE CDN CSVs for put/call ratios (totalpc.csv, equitypc.csv) - CNN dataviz API for Fear & Greed (production.dataviz.cnn.io) - Yahoo Finance for VIX9D/VIX3M/SKEW/RSP/NYA (standard symbols) - FRED for IG spread (BAMLC0A0CM) and SOFR (add to existing array) - AAII scrape for bull/bear survey (only medium-effort source) - Breadth via RSP/SPY divergence + NYSE composite (no scraping) https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Add verified Yahoo symbols for breadth + finalized source list New discoveries: - ^MMTH = % stocks above 200 DMA (direct Yahoo symbol!) - C:ISSU = NYSE advance/decline data - CNN endpoint accepts date param for historical data - CBOE CSVs have data back to 2003 - 33 total calls per seed run, ~6s runtime All 10 categories now have confirmed free sources. https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Rewrite F&G brief as forward-looking design doc Remove all reverse-engineering language, screenshot references, and discovery notes. Clean structure: goal, scoring model, data sources, formulas, seed script plan, implementation phases, MVP path. https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * docs: apply gold standard corrections to fear-greed-index-2.0 brief * feat(market): add Fear & Greed Index 2.0 — 10-category composite sentiment panel Composite 0-100 index from 10 weighted categories: sentiment (CNN F&G, AAII, crypto F&G), volatility (VIX, term structure), positioning (P/C ratio, SKEW), trend (SPX vs MAs), breadth (% >200d, RSP/SPY divergence), momentum (sector RSI, ROC), liquidity (M2, Fed BS, SOFR), credit (HY/IG spreads), macro (Fed rate, yield curve, unemployment), cross-asset (gold/bonds/DXY vs equities). Data layer: - seed-fear-greed.mjs: 19 Yahoo symbols (150ms gaps), CBOE P/C CSVs, CNN F&G API, AAII scrape (degraded-safe), FRED Redis reads. TTL 64800s. - seed-economy.mjs: add BAMLC0A0CM (IG spread) and SOFR to FRED_SERIES. - Bootstrap 4-file checklist: cache-keys, bootstrap.js, health.js, handler. Proto + RPC: - get_fear_greed_index.proto with FearGreedCategory message. - get-fear-greed-index.ts handler reads seeded Redis data. Frontend: - FearGreedPanel with gauge, 9-metric header grid, 10-category breakdown. - Self-loading via bootstrap hydration + RPC fallback. - Registered in panel-layout, App.ts (prime + refresh), panel config, Cmd-K commands, finance variant, i18n (en/ar/zh/es). * fix(market): add RPC_CACHE_TIER entry for get-fear-greed-index * fix(docs): escape bare angle bracket in fear-greed brief for MDX * fix(docs): fix markdown lint errors in fear-greed brief (blank lines around headings/lists) * fix(market): fix seed-fear-greed bugs from code review - fredLatest/fredNMonthsAgo: guard parseFloat with Number.isFinite to handle FRED's "." missing-data sentinel (was returning NaN which propagated through scoring as a truthy non-null value) - Remove 3 unused Yahoo symbols (^NYA, HYG, LQD) that were fetched but not referenced in any scoring category (saves ~450ms per run) - fedRateStr: display effective rate directly instead of deriving target range via (fedRate - 0.25) which was incorrect * fix(market): address P2/P3 review findings in Fear & Greed - FearGreedPanel: add mapSeedPayload() to correctly map raw seed JSON to proto-shaped FearGreedData; bootstrap hydration was always falling through to RPC because seed shape (composite.score) differs from proto shape (compositeScore) - FearGreedPanel: fix fmt() — remove === 0 guard and add explicit > 0 checks on VIX and P/C Ratio display to handle proto default zeros without masking genuine zero values (e.g. pctAbove200d) - seed-fear-greed: remove broken history write — each run overwrote the key with a single-entry array (no read-then-append), making the 90-day TTL meaningless; no consumer exists yet so defer to later - seed-fear-greed: extract hySpreadVal const to avoid double fredLatest call - seed-fear-greed: fix stale comment (19 symbols → 16 after prior cleanup) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Elie Habib
|
a1c3c1d684 |
feat(panels): AI Market Implications — LLM trade signals from live world state (#2146) (#2165)
* fix(intelligence): use camelCase field names for ListMarketImplicationsResponse
* fix(bootstrap): register marketImplications in cache-keys.ts and add hydration consumer
* chore: stage all market-implications feature files for proto freshness check
* feat(market-implications): add LLM routing env vars for market implications stage
* fix(market-implications): move types to services layer to fix boundary violation
* fix: add list-market-implications gateway tier entry
* fix(market-implications): add health.js entries + i18n tooltip key
- api/health.js: add marketImplications to BOOTSTRAP_KEYS
('intelligence:market-implications:v1') and SEED_META
(seed-meta:intelligence:market-implications, maxStaleMin=150 = 2x
the 75min TTL, matching gold standard)
- en.json: add components.marketImplications.infoTooltip which was
referenced in MarketImplicationsPanel but missing from locales
* fix(market-implications): wire CMD+K entry and panels.marketImplications i18n key
- commands.ts: add panel:market-implications command with trade/signal
keywords so the panel appears in CMD+K search
- en.json: add panels.marketImplications used by UnifiedSettings panel
toggle display and SearchModal label resolution
|
||
|
Elie Habib
|
e08457aadf |
feat(fuel-prices): add retail gasoline and diesel prices panel (#2150)
* feat(fuel-prices): add retail fuel prices panel and seeder - Add ListFuelPrices proto RPC with FuelPrice/FuelCountryPrice messages - Create seed-fuel-prices.mjs seeder with 5 sources: Malaysia (data.gov.my), Spain (minetur.gob.es), Mexico (datos.gob.mx), US EIA, EU oil bulletin CSV - Add list-fuel-prices.ts RPC handler reading from Redis seed cache - Wire handler into EconomicService handler.ts - Register fuelPrices in cache-keys.ts, bootstrap.js, health.js, gateway.ts - Add FuelPricesPanel frontend component with gasoline/diesel/source columns - Wire panel into panel-layout.ts, App.ts (prime + refresh scheduler) - Add panel config, command entry with fuel/gas/diesel/petrol keywords - Add fuelPrices refresh interval (6h) in base.ts - Add i18n keys in en.json (panels.fuelPrices + components.fuelPrices) Task: fuel-prices * fix(fuel-prices): add BR/NZ/UK sources, fix EU non-euro currency, review fixes - Add fetchBrazil() — ANP CSVs (GASOLINA + DIESEL), Promise.allSettled for independent partial results, BRL→USD via FX - Add fetchNewZealand() — MBIE weekly-table.csv, Board price national avg, NZD→USD - Add fetchUK_ModeA() — CMA retailer JSON feeds (Asda/BP/JET/MFG/Sainsbury's/ Morrisons), E10+B7 pence→GBP, max-date observedAt across retailers - Fix EU non-euro members (BG/CZ/DK/HU/PL/RO/SE) using local currency FX on EUR-denominated prices — all EU entries now use currency:'EUR' - Fix fetchBrazil Promise.all → Promise.allSettled (partial CSV failure no longer discards both fuels) - Fix UK observedAt: keep latest date across retailers (not last-processed) - Fix WoW anomaly: omit wowPct instead of setting to 0 - Lift parseEUPrice out of inner loop to module scope - Pre-compute parseBRDate per row to avoid double-conversion - Update infoTooltip: describe methodology without exposing source URLs - Add BRL, NZD, GBP to FX symbols list * fix(fuel-prices): fix 4 live data bugs found in external review EU CSV: replace hardcoded 2024 URLs (both returning 404) with dynamic discovery — scrape EC energy page for current CSV link, fall back to generated YYYY-MM patterns for last 4 months. NZ: live MBIE header has no Region column (Week,Date,Fuel,Variable,Value, Unit,Status) — remove regionIdx guard that was forcing return []. Values are in NZD c/L not NZD/L — divide by 100 before storing. UK: last_updated is DD/MM/YYYY HH:mm:ss not ISO — parse to YYYY-MM-DD before lexicographic max-date comparison; previous code stored the seed-run date instead of the latest retailer timestamp. Panel: source column fell back to — for diesel-only countries because it only read gas?.source. Use (gas ?? dsl)?.source so diesel-only rows display their source correctly. |
||
|
Elie Habib
|
ddc6603cce |
feat(infra): Cloudflare Radar DDoS attacks + traffic anomaly endpoints (#2067)
* feat(infra): add Cloudflare Radar DDoS attacks + traffic anomaly endpoints Extends the existing Cloudflare Radar integration (internet outages) with two new data streams, both confirmed accessible with the current token: - L3/L4 DDoS attack summaries (protocol + vector breakdowns, 7d window) - Traffic anomaly events (DNS/BGP/ICMP anomalies with country + ASN context) Changes: - proto: add DdosAttackSummaryEntry + TrafficAnomaly messages; new list_internet_ddos_attacks.proto and list_internet_traffic_anomalies.proto; wire two new RPCs into InfrastructureService - buf generate: regenerated server/client TypeScript from updated protos - seed-internet-outages.mjs: add fetchDdosData() + fetchTrafficAnomalies() called inside fetchAll() before runSeed() (process.exit-safe pattern); writes cf:radar:ddos:v1 and cf:radar:traffic-anomalies:v1 - list-ddos-attacks.ts + list-traffic-anomalies.ts: read-from-seed handlers - handler.ts: wire new handlers - cache-keys.ts + api/bootstrap.js: add ddosAttacks + trafficAnomalies bootstrap keys (fast tier); kept in sync to pass bootstrap parity tests - gateway.ts: add RPC_CACHE_TIER entries (slow) for new routes - services/infrastructure: add fetchDdosAttacks() + fetchTrafficAnomalies() with circuit breakers + hydration support UI surface (cards alongside outage map) deferred to follow-up. Closes #2043 * fix(i18n): rename Internet Outages → Internet Disruptions Broader term covers outages, DDoS events, and traffic anomalies now seeded from Cloudflare Radar. Updated in en.json (layer label, tooltip, country brief count strings), map-layer-definitions.ts fallback label, and commands.ts search keywords. Other locales retain their translated strings (not degraded — they already use broader equivalents like "internet disruption" in many langs). * feat(map): render traffic anomalies + DDoS target locations on disruptions layer Adds geo-coordinates to both data types so they appear as map markers under the Internet Disruptions toggle alongside existing outage circles. - Proto: add latitude/longitude to TrafficAnomaly (fields 10/11), add new DdosLocationHit message, add top_target_locations to DdosAttacksResponse - Seeder: resolve lat/lon from COUNTRY_COORDS for traffic anomalies; fetch CF Radar top/locations/target endpoint for DDoS top-target locations - Server handler: pass topTargetLocations through from Redis seed cache - DeckGLMap: amber trafficAnomaly layer + purple ddosHit layer with tooltips - GlobeMap: TrafficAnomalyMarker + DdosHitMarker with emoji indicators - MapContainer: expose setTrafficAnomalies() + setDdosLocations() setters - data-loader: fire-and-forget anomaly/DDoS fetches after outages load * fix(review): address code review findings + add Internet Disruptions panel - fix: totalCount returns filtered count when country param is set - fix: countryName uses clientCountryName fallback (was always empty) - fix: remove duplicate toEpochMsFromIso (consolidate into toEpochMs) - fix: anomalies guard >= 0 → > 0 (don't write empty array to Redis) - fix: GlobeMap uses named top-level imports instead of inline imports - feat: InternetDisruptionsPanel with 3 tabs (Outages / DDoS / Anomalies) |
||
|
Elie Habib
|
7711e9de03 |
feat(consumer-prices): add basket price monitoring domain (#1901)
* feat(consumer-prices): add basket price monitoring domain
Adds end-to-end consumer price tracking to enable inflation monitoring
across key markets, starting with UAE essentials basket.
- consumer-prices-core/: companion scraping service with pluggable
acquisition providers (Playwright, Exa, Firecrawl, Parallel P0),
config-driven retailer YAML, Postgres schema, Redis snapshots
- proto/worldmonitor/consumer_prices/v1/: 6-RPC service definition
- api/consumer-prices/v1/[rpc].ts: Vercel edge route
- server/worldmonitor/consumer-prices/v1/: Redis-backed RPC handlers
- src/services/consumer-prices/: circuit breakers + bootstrap hydration
- src/components/ConsumerPricesPanel.ts: 5-tab panel (overview /
categories / movers / spread / health)
- scripts/seed-consumer-prices.mjs: Railway cron seed script
- Wire into bootstrap, health, panels, gateway, cache-keys, locale
* fix(consumer-prices): resolve all code review findings
P0: populate topCategories — categoryResult was fetched but never used.
Added buildTopCategories() helper with grouped CTE query that extracts
current_index and week-over-week pct per category.
P1 (4 fixes):
- aggregate: replace N+1 getBaselinePrice loop with single batch query
getBaselinePrices(ids[], date) via ANY($1) — eliminates 119 DB roundtrips
per basket run
- aggregate/computeValueIndex: was dividing all category floors by the same
arbitrary first baseline; now uses per-item floor price with per-item
baseline (same methodology as fixed index but with cheapest price)
- basket-series endpoint now seeded: added buildBasketSeriesSnapshot() to
worldmonitor.ts, /basket-series route in companion API, publish.ts writes
7d/30d/90d series per basket, seed script fetches and writes all three ranges
- scrape: call teardownAll() after each retailer run to close Playwright
browser; without this the Chromium process leaked on Railway
P2 (4 fixes):
- db/client: remove rejectUnauthorized: false — was bypassing TLS cert
validation on all non-localhost connections
- publish: seed-meta now writes { fetchedAt, recordCount } matching the format
expected by _seed-utils.mjs writeExtraKeyWithMeta (was writing { fetchedAt, key })
- products: remove unused getMatchedProductsForBasket — exact duplicate of
getBasketRows in aggregate.ts; never imported by anything
Snapshot type overhaul:
- Flatten WMOverviewSnapshot to match proto GetConsumerPriceOverviewResponse
(was nested under overview:{}; handlers read flat)
- All asOf fields changed from number to string (int64 → string per proto JSON)
- freshnessMin/parseSuccessRate null -> 0 defaults
- lastRunAt changed from epoch number to ISO string
- Mover items now include currentPrice and currencyCode
- emptyOverview/Movers/Spread/Freshness in seed script use String(Date.now())
* feat(consumer-prices): wire Exa search engine as acquisition backend for UAE retailers
Ports the proven Exa+summary price extraction from PR #1904 (seed-grocery-basket.mjs)
into consumer-prices-core as ExaSearchAdapter, replacing unvalidated Playwright CSS
scraping for all three UAE retailers (Carrefour, Lulu, Noon).
- New ExaSearchAdapter: discovers targets from basket YAML config (one per item),
calls Exa API with contents.summary to get AI-extracted prices, uses matchPrice()
regex (ISO codes + symbol fallback + CURRENCY_MIN guards) to extract AED amounts
- New db/queries/matches.ts: upsertProductMatch() + getBasketItemId() for auto-linking
scraped Exa results to basket items without a separate matching step
- scrape.ts: selects ExaSearchAdapter when config.adapter === 'exa-search'; after
insertObservation(), auto-creates canonical product and product_match (status: 'auto')
so aggregate.ts can compute indices immediately without manual review
- All three UAE retailer YAMLs switched to adapter: exa-search and enabled: true;
CSS extraction blocks removed (not used by search adapter)
- config/types.ts: adds 'exa-search' to adapter enum
* fix(consumer-prices): use EXA_API_KEYS (with fallback to EXA_API_KEY) matching PR #1904 pattern
* fix(consumer-prices): wire ConsumerPricesPanel in layout + fix movers limit:0 bug
Addresses Codex P1 findings on PR #1901:
- panel-layout.ts: import and createPanel('consumer-prices') so the panel
actually renders in finance/commodity variants where it is enabled in config
- consumer-prices/index.ts: limit was hardcoded 0 causing slice(0,0) to always
return empty risers/fallers after bootstrap is consumed; fixed to 10
* fix(consumer-prices): add categories snapshot to close P2 gap
consumer-prices:categories:ae:* was in BOOTSTRAP_KEYS but had no producer,
so the Categories tab always showed upstreamUnavailable.
- buildCategoriesSnapshot() in worldmonitor.ts — wraps buildTopCategories()
and returns WMCategoriesSnapshot matching ListConsumerPriceCategoriesResponse
- /categories route in consumer-prices-core API
- publish.ts writes consumer-prices:categories:{market}:{range} for 7d/30d/90d
- seed-consumer-prices.mjs fetches all three ranges from consumer-prices-core
and writes them to Redis alongside the other snapshots
P1 issues (snapshot structure mismatch + limit:0 movers) were already fixed
in earlier commits on this branch.
* fix(types): add variants? to PANEL_CATEGORY_MAP type
|
||
|
Elie Habib
|
a8f8c0aa61 |
feat(economic): Middle East grocery basket price index (#1904)
* feat(economic): add ME grocery basket price index Adds a grocery basket price comparison panel for 9 Middle East countries (UAE, KSA, Qatar, Kuwait, Bahrain, Oman, Egypt, Jordan, Lebanon) using EXA AI to discover prices from regional e-commerce sites (Carrefour, Lulu, Noon, Amazon) and Yahoo Finance for FX rates. - proto: ListGroceryBasketPrices RPC with CountryBasket/GroceryItemPrice messages - seed: seed-grocery-basket.mjs, 90 EXA calls/run, 150ms delay, hardcoded FX fallbacks for pegged GCC currencies, 6h TTL - handler: seed-only RPC reading economic:grocery-basket:v1 - gateway: static cache tier for the new route - bootstrap/health: groceryBasket key in SLOW tier, 720min stale threshold - frontend: GroceryBasketPanel with scrollable table, cheapest/priciest column highlighting, styles moved to panels.css - panel disabled by default until seed is run on Railway * fix(generated): restore @ts-nocheck in economic service codegen * fix(grocery-basket): tighten seed health staleness and seed script robustness - Set maxStaleMin to 360 (6h) matching CACHE_TTL so health alerts on first missed run - Use ?? over || for FX fallback to handle 0-value rates correctly - Add labeled regex patterns with bare-number warning in extractPrice - Replace conditional delay logic with unconditional per-item sleep * fix(grocery-basket): fix EXA API format and price extraction after live validation - Use contents.summary format (not top-level summary) — previous format returned no data - Support EXA_API_KEYS (comma-separated) in addition to EXA_API_KEY - Extract price from plain-text summary string (EXA returns text, not JSON schema) - Remove bare-number fallback — too noisy (matched "500" from "pasta 500g" as SAR 500) - Fix LBP FX rate zero-guard: use fallback when Yahoo returns 0 for ultra-low-value currencies Validated locally: 9 countries seeded, Redis write confirmed, ~111s runtime * fix(grocery-basket): validate extracted currency against expected country currency - matchPrice now returns the currency code alongside the price - extractPrice rejects results where currency != expected country currency (prevents AED prices from being treated as JOD prices on gcc.luluhypermarket.com) - Tighten item queries (white granulated sugar, spaghetti pasta, etc.) to reduce irrelevant product matches like Stevia on sugar queries - Replace Jordan's gcc.luluhypermarket.com (GCC-only) with carrefour.jo + ounasdelivery.com - Sync scripts/shared/grocery-basket.json * feat(bigmac): add Big Mac Index seed + drop grocery basket includeDomains Grocery basket: - Remove includeDomains restriction — EXA neural search finds better sources than hardcoded domain lists; currency validation prevents contamination - Tighten query strings (supermarket retail price suffix) Big Mac seed (scripts/seed-bigmac.mjs): - Two-tier search: specialist sites (theburgerindex.com, eatmyindex.com) first, fall back to open search for countries without per-country indexed pages - Handle thousands-separator prices (480,000 LBP) - Accept USD prices from cost-of-living index sites as fallback - Exclude ranking/average pages (Numbeo country_price_rankings, Expatistan) - Validated live: 7/9 countries with confirmed prices UAE=19AED, KSA=19SAR, QAR=22QAR, KWD=1.4KWD, EGP=135EGP, JOD=3JOD, LBP=480kLBP * feat(economic): expand grocery basket to 24 global countries, drop Big Mac tier-2 search Grocery basket: extend coverage from 9 MENA to 24 countries across all regions (US, UK, DE, FR, JP, CN, IN, AU, CA, BR, MX, ZA, TR, NG, KR, SG, PK, AE, SA, EG, KE, AR, ID, PH). Add FX fallbacks and fxSymbols for all 23 new currencies. CCY regex in seed script updated to match all supported currency codes. Big Mac: remove tier-2 open search (too noisy, non-specialist pages report combo prices or global averages). Specialist sites only (theburgerindex.com, eatmyindex.com) for clean per-country data. * feat(bigmac): wire Big Mac Index RPC, proto, bootstrap, health Add ListBigMacPrices RPC end-to-end: - proto/list_bigmac_prices.proto: BigMacCountryPrice + request/response - service.proto: register ListBigMacPrices endpoint (GET /list-bigmac-prices) - buf generate: regenerate service_server.ts + all client stubs - server/list-bigmac-prices.ts: seed-only handler reads economic:bigmac:v1 - handler.ts: wire listBigMacPrices into EconomicServiceHandler - api/bootstrap.js: bigmac key in BOOTSTRAP_CACHE_KEYS + SLOW_KEYS - api/health.js: bigmac key in BOOTSTRAP_KEYS + SEED_META (maxStaleMin: 1440) - _bootstrap-cache-key-refs.ts: groceryBasket + bigmac refs * feat(bigmac): add BigMacPanel + register in panel layout BigMacPanel renders a country-by-country Big Mac price table sorted by USD price (cheapest/most expensive highlighted). Wired into bootstrap hydration, refresh scheduler, and panel registry. Registered in panels.ts (enabled: false, to be flipped once seed data is verified). Also updates grocery basket i18n from ME-specific to global wording. * fix(bigmac): register bigmac in cache-keys and RPC_CACHE_TIER Add bigmac to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in server/_shared/cache-keys.ts, and to RPC_CACHE_TIER (static tier) in gateway.ts. Both were caught by bootstrap and RPC tier parity tests. * fix(generated): restore @ts-nocheck in all generated service files after buf regenerate buf generate does not emit @ts-nocheck. Previous convention restores it manually post-generate to suppress strict type errors in generated code. * fix(grocery-basket): restore includeDomains per country, add userLocation, fix currency symbol parsing Root cause of 0-item countries (UK, JP, IN, NG): removing includeDomains caused EXA neural search to return USD-priced global comparison pages (Numbeo, Tridge, Expatistan) which currency validation correctly rejected. Fixes: - Add per-country sites[] in grocery-basket.json (researched local supermarket/retailer domains: tesco.com, kaufland.de, bigbasket.com, etc.) - Pass includeDomains: country.sites to restrict EXA to local retailers - Pass userLocation: country.code (ISO) to bias results to target country - Add currency symbol fallback regex (£→GBP, €→EUR, ¥→JPY, ₹→INR, ₩→KRW, ₦→NGN, R$→BRL) — sites like BigBasket use ₹ not INR - Summary query now explicitly requests ISO currency code - Simplify item queries (drop country name — context from domains) Smoke test results: UK sugar → GBP 1.09 (tesco.com) ✓ IN rice → ₹66 (bigbasket.com) ✓ JP rice → JPY 500 (kakaku.com) ✓ * fix(grocery-basket): add Firecrawl fallback, parallel items, bulk caps, currency floors - Add Firecrawl as JS-SPA fallback after EXA (handles noon.com, coupang, daraz, tokopedia, lazada) - Parallelize item fetching per country with 200ms stagger: runtime 38min to ~4.5min - Add CURRENCY_MIN floors (NGN:50, IDR:500, KRW:1000, etc.) to reject product codes as prices - Add ITEM_USD_MAX bulk caps (sugar:8USD, salt:5USD, rice:6USD, etc.) applied to both EXA and Firecrawl - Fix SA: use noon.com/saudi-en + carrefour.com.sa (removes luluhypermarket cross-country pollution) - Fix EG: use carrefouregypt.com + spinneys.com.eg + seoudi.com (removes GCC luluhypermarket) - Expand sites for DE, MX, ZA, TR, NG, KR, IN, PK, AR, ID, PH to improve coverage - Sync scripts/shared/grocery-basket.json with shared/grocery-basket.json * fix(grocery-basket): address PR review comments P1+P2 P1 - fix ranking with incomplete data: only include countries with >=70% item coverage (>=7/10) in cheapest/mostExpensive ranking — prevents a country with 4/10 items appearing cheapest due to missing data P1 - fix regex false-match on pack sizes / weights: try currency-first pattern (SAR 8.99) before number-first to avoid matching pack counts; use matchAll and take last match P2 - mark seed-miss responses as non-cacheable: add upstream_unavailable to proto + return true on empty seed so gateway sets Cache-Control: no-store on cold deploy * fix(generated): update EconomicService OpenAPI docs for upstream_unavailable field |
||
|
Elie Habib
|
c658b8eb94 |
feat(economic): National Debt Clock — live ticking debt estimates for 180+ countries (#1923)
* feat(economic): add National Debt Clock panel with IMF + Treasury data - Proto: GetNationalDebt RPC in EconomicService with NationalDebtEntry message - Seed: seed-national-debt.mjs fetches IMF WEO (debt%, GDP, deficit%) + US Treasury FiscalData in parallel; filters aggregates/territories; sorts by total debt; 35-day TTL for monthly Railway cron - Handler: get-national-debt.ts reads seeded Redis cache key economic:national-debt:v1 - Registry: nationalDebt key added to cache-keys.ts, bootstrap.js (SLOW tier), health.js (maxStaleMin=10080), gateway.ts (daily cache tier) - Service: getNationalDebtData() in economic/index.ts with bootstrap hydration + RPC fallback - Panel: NationalDebtPanel.ts with sort tabs (Total/Debt-GDP/1Y Growth), search, live ticking via direct DOM manipulation (avoids setContent debounce) - Tests: 10 seed formula tests + 8 ticker math tests; all 2064 suite tests green * fix(economic): address code review findings for national debt clock * fix(economic): guard runSeed() call to prevent process.exit in test imports seed-national-debt.mjs called runSeed() at module top-level. When imported by tests (to access computeEntries), the seed ran, hit missing Redis creds in CI, and called process.exit(1), failing the entire test suite. Guard with isMain check so runSeed() only fires on direct execution. |
||
|
Elie Habib
|
c0bf784d21 |
feat(finance): crypto sectors heatmap + DeFi/AI/Alt token panels + expanded crypto news (#1900)
* feat(market): add crypto sectors heatmap and token panels (DeFi, AI, Other) backend - Add shared/crypto-sectors.json, defi-tokens.json, ai-tokens.json, other-tokens.json configs - Add scripts/seed-crypto-sectors.mjs and seed-token-panels.mjs seed scripts - Add proto messages for ListCryptoSectors, ListDefiTokens, ListAiTokens, ListOtherTokens - Add change7d field (field 6) to CryptoQuote proto message - Run buf generate to produce updated TypeScript bindings - Add server handlers for all 4 new RPCs reading from seeded Redis cache - Wire handlers into marketHandler and register cache keys with BOOTSTRAP_TIERS=slow - Wire seedCryptoSectors and seedTokenPanels into ais-relay.cjs seedAllMarketData loop * feat(panels): add crypto sectors heatmap and token panels (DeFi, AI, Other) - Add TokenData interface to src/types/index.ts - Wire ListCryptoSectorsResponse/ListDefiTokensResponse/ListAiTokensResponse/ListOtherTokensResponse into market service with circuit breakers and hydration fallbacks - Add CryptoHeatmapPanel, TokenListPanel, DefiTokensPanel, AiTokensPanel, OtherTokensPanel to MarketPanel.ts - Register 4 new panels in panels.ts FINANCE_PANELS and cryptoDigital category - Instantiate new panels in panel-layout.ts - Load data in data-loader.ts loadMarkets() alongside existing crypto fetch * fix(crypto-panels): resolve test failures and type errors post-review - Add @ts-nocheck to regenerated market service_server/client (matches repo convention) - Add 4 new RPC routes to RPC_CACHE_TIER in gateway.ts (route-cache-tier test) - Sync scripts/shared/ with shared/ for new token/sector JSON configs - Restore non-market generated files to origin/main state (avoid buf version diff) * fix(crypto-panels): address code review findings (P1-P3) - ais-relay seedTokenPanels: add empty-guard before Redis write to prevent overwriting cached data when all IDs are unresolvable - server _feeds.ts: sync 4 missing crypto feeds (Wu Blockchain, Messari, NFT News, Stablecoin Policy) with client-side feeds.ts - data-loader: expose panel refs outside try block so catch can call showRetrying(); log error instead of swallowing silently - MarketPanel: replace hardcoded English error strings with t() calls (failedSectorData / failedCryptoData) to honour user locale - seed-token-panels.mjs: remove unused getRedisCredentials import - cache-keys.ts: one BOOTSTRAP_TIERS entry per line for consistency * fix(crypto-panels): three correctness fixes for RSS proxy, refresh, and Redis write visibility - api/_rss-allowed-domains.js: add 7 new crypto domains (decrypt.co, blockworks.co, thedefiant.io, bitcoinmagazine.com, www.dlnews.com, cryptoslate.com, unchainedcrypto.com) so rss-proxy.js accepts the new finance feeds instead of rejecting them as disallowed hosts - src/App.ts: add crypto-heatmap/defi-tokens/ai-tokens/other-tokens to the periodic markets refresh viewport condition so panels on screen continue receiving live updates, not just the initial load - ais-relay seedTokenPanels: capture upstashSet return values and log PARTIAL if any Redis write fails, matching seedCryptoSectors pattern |