Commit Graph

209 Commits

Author SHA1 Message Date
Elie Habib
29925204fa feat(oref): Tzeva Adom as primary alert source + Hebrew translation dictionaries (#2863)
* feat(oref): add Tzeva Adom as primary alert source with Hebrew translations

Tzeva Adom API (api.tzevaadom.co.il/notifications) is now the primary
source for Israeli siren alerts. Free, no proxy needed, works from any
IP. OREF direct (via residential proxy) remains as fallback.

Includes Hebrew→English translation dictionaries:
- 1,305 Israeli localities from CBS data
- 28 threat type terms (missiles, rockets, drones, infiltration, etc.)

Alert titles and location names are now served in English.

Fallback chain: Tzeva Adom (primary) → OREF direct (proxy, fallback)

* fix(oref): add threat categorization + misplaced-city guard to Tzeva Adom

- categorizeOrefThreat() classifies Hebrew/English alerts into
  MISSILE/ROCKET/DRONE/MORTAR/INFILTRATION/EARTHQUAKE/TSUNAMI/HAZMAT
- Detects when API puts a city name in the threat field and moves it
  to locations (known API quirk)

* fix(oref): decouple siren poll loop from OREF proxy availability

SIREN_ALERTS_ENABLED is always true (Tzeva Adom needs no proxy).
OREF_PROXY_AVAILABLE gates only the OREF fallback path.
The poll loop now starts regardless of proxy config, using Tzeva Adom
as primary and OREF as fallback only when OREF_PROXY_AUTH is set.

Response payloads report configured: true so the panel activates.

* fix(oref): preserve error state when both siren sources fail

When Tzeva Adom returns null and OREF proxy is unavailable, return
early with lastError set instead of falling through and clearing
the error. Prevents a false green state in the panel when both
sources are down.

* fix(oref): rebuild response cache on source outage

Without calling orefPreSerializeResponses() in the failure branch,
the /oref/alerts handler keeps serving stale _alertsCache from the
last successful poll, masking the outage.
2026-04-09 12:39:34 +04:00
Elie Habib
e200cfdc60 fix(energy): remove MULTI/EXEC from Ember pipeline; drop ais-relay Ember loop (#2836)
* fix(energy): remove MULTI/EXEC from Ember pipeline (unsupported by Upstash REST); remove redundant ais-relay Ember loop

* test(energy): remove duplicate pipeline failure detection describe block

Lines 289-301 were a strict subset of the block at lines 235-253,
causing duplicate test entries in reports.
2026-04-08 22:28:30 +04:00
Elie Habib
b8924eb90f feat(energy): Ember monthly electricity seed (V5-6a) (#2815)
* feat(energy): Ember monthly electricity seed — V5-6a

New seed-ember-electricity.mjs writes energy:ember:v1:<ISO2> and
energy:ember:v1:_all from Ember Climate's monthly generation CSV (CC BY 4.0).
Daily cron at 08:00 UTC, TTL 72h (3x interval), >=60 country coverage guard.

Registers in api/health.js, api/seed-health.js, cache-keys.ts, and
ais-relay.cjs. Dockerfile.relay COPY added.

🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/code) + Compound Engineering v2.49.0

Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>

* fix(energy): add _country-resolver.mjs to Dockerfile.relay; correct Ember intervalMin (V5-6a)

Two bugs in the Ember seed PR:
1. Dockerfile.relay was missing COPY for _country-resolver.mjs, which
   seed-ember-electricity.mjs imports. Would have crashed with
   ERR_MODULE_NOT_FOUND on first run in production.
2. api/seed-health.js had intervalMin:720 (12h) for a daily (24h) cron.
   With stale threshold = intervalMin*2, this gave only 24h grace --
   the seed would flap stale during the CSV download window.
   Corrected to intervalMin:1440 so stale threshold = 48h (2x interval).

* fix(energy): wire energy:ember:v1:_all into bootstrap hydration (V5-6a)

Greptile P1: api/bootstrap.js was missing the emberElectricity slow-key
entry, violating the AGENTS.md requirement that new data sources be
registered for bootstrap hydration.

energy:ember:v1:_all is a ~60-country bulk map (monthly cadence) -
added to SLOW_KEYS consistent with faoFoodPriceIndex and other
monthly-release bulk keys.

Also updates server/_shared/cache-keys.ts BOOTSTRAP_CACHE_KEYS and
BOOTSTRAP_TIERS to keep the bootstrap test coverage green (bootstrap
test validates that SLOW_KEYS and BOOTSTRAP_TIERS are in sync).

* fix(energy): 3 review fixes for Ember seed (V5-6a)

1. Ember URL: updated to correct current download URL (old path
   returned HTTP 404, seeder could never run).
2. Count-drop guard after failure: failure path now preserves the
   previous recordCount in seed-meta instead of writing 0, so the
   75% drop guard stays active after a failed run.
3. api/seed-health.js: status:error now marks seed as stale/error
   immediately instead of only checking age; prevents /api/seed-health
   showing ok for 48h while the seeder is failing.

* fix(energy): correct Ember CSV column names + fix skipped-path meta (V5-6a)

1. CSV schema: parser was using country_code/series/unit/value/date
   but the real Ember CSV headers are "ISO 3 code"/"Variable"/"Unit"/
   "Value"/"Date". Added COLS constants and updated all row field
   accesses. The schema sentinel (hasFossil check) was always firing
   because r.series was always undefined, causing every seeder run to
   abort. Updated test fixtures to use real column names.
2. Skipped-path meta: lock.skipped branch now reads existing meta and
   preserves recordCount and status while refreshing fetchedAt.
   Previously writing recordCount:0 disabled the count-drop guard after
   any skipped run and made health endpoints see false-ok with zero count.

* fix(energy): remove skipped-path meta write + revert premature bootstrap (V5-6a)

1. lock.skipped: removed seed-meta write from the skipped path. The
   running instance writes correct meta on completion; refreshing
   fetchedAt on skip masked relay/lock failures from health endpoints.
2. Bootstrap: removed emberElectricity from BOOTSTRAP_CACHE_KEYS and
   BOOTSTRAP_TIERS — no consumer exists in src/ yet. Per energyv5.md,
   bootstrap registration is deferred to PR7 when consumers land.

* fix(energy): split ember pipeline writes; fix health.js recordCount lookup

- api/health.js: add recordCount fallback in both seed-meta count reads so
  the Ember domain shows correct record count instead of always 1
- scripts/seed-ember-electricity.mjs: split single pipeline into Phase A
  (per-country + _all data) and Phase B (seed-meta only after Phase A
  succeeds) to prevent preservePreviousSnapshot reading a partial _all key

* fix(energy): split ember pipeline writes; align SEED_ERROR in health.js; add tests

* fix(energy): atomic rollback on partial pipeline failure; seedError priority in health cascade

* fix(energy): DEL obsolete per-country keys on publish, rollback, and restore

* fix(energy): MULTI/EXEC atomic pipeline; null recordCount on read-miss; dataWritten guard

---------

Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
2026-04-08 12:25:54 +04:00
Elie Habib
d96259048d feat(energy): canonical energy spine — V5-1 (#2798)
* feat(energy): canonical energy spine seeder + handler read-through (V5-1)

- Add scripts/seed-energy-spine.mjs: daily seeder that assembles one
  energy:spine:v1:<ISO2> key per country from 6 domain keys (OWID mix,
  JODI oil, JODI gas, IEA stocks, ENTSO-E electricity, GIE gas storage)
- TTL 172800s (48h), count-drop guard at 80%, schema sentinel for OWID mix,
  lock pattern + Redis pipeline batch write mirroring seed-owid-energy-mix.mjs
- Update get-country-energy-profile.ts to read from spine first; fall back
  to existing 6-key Promise.allSettled join on spine miss
- Update chat-analyst-context.ts buildProductSupply/buildGasFlows/buildOilStocksCover
  to prefer spine key; fall through to direct domain key reads on miss
- Update get-country-intel-brief.ts to read energy mix from spine.mix + sources.mixYear
  before falling back to energy:mix:v1: direct key
- Add ENERGY_SPINE_KEY_PREFIX and ENERGY_SPINE_COUNTRIES_KEY to cache-keys.ts
- Add energySpineCountries to api/health.js STANDALONE_KEYS and SEED_META
- Add Railway cron comment (0 6 * * *) in ais-relay.cjs
- Add tests/energy-spine-seed.test.mjs: 26 tests covering spine build logic,
  IEA anomaly guard, JODI oil fallback, schema sentinel, count-drop math

* fix(energy): add cache-keys module replacement in redis-caching test

The new ENERGY_SPINE_KEY_PREFIX import in get-country-intel-brief.ts
was not patched in the importPatchedTsModule call used by redis-caching
tests. Add cache-keys to the replacement map to resolve the module.

* fix(energy): add missing fields to spine entry and read-through

- buildOilFields: add product importsKbd (gasoline/diesel/jet/lpg) and belowObligation
- buildMixFields: add windShare, solarShare, hydroShare
- buildGasStorageFields: new helper storing fillPct, fillPctChange1d, trend
- buildSpineEntry: add gasStorage section using new helper
- EnergySpine interface: extend oil/mix/gasStorage to match seeder output
- buildResponseFromSpine: read all new fields instead of hard-coding 0/false

* fix(energy): exclude electricity/gas-storage from spine; add seed-health entry

Spine-first path was returning stale gas-storage and electricity data for
up to 8h after seeding (spine runs 06:00 UTC, gas storage updates 10:30 UTC,
electricity updates 14:00 UTC).

Fix: handler now reads gas-storage and electricity directly in parallel with
the spine read (3-key allSettled). Fallback path drops from 6 to 4 keys since
gas-storage and electricity are already fetched in the hot path.

Also registers energy:spine in api/seed-health.js (daily cron, maxStaleMin
inferred as 2×intervalMin = 2880 min).

Seeder (seed-energy-spine.mjs) and its tests updated to reflect the narrowed
spine schema — electricity and gasStorage fields removed from buildSpineEntry.

* fix(energy): address Greptile P2 review findings

- chat-analyst-context: return undefined when gas imports are 0 rather
  than falling back to totalDemandTj with an "imports" label — avoids
  mislabeling domestic gas demand as imports for net exporters (RU, QA, etc.)
- seed-energy-spine: add status:'ok' to success-path seed-meta write so
  all seed-meta records have a consistent status field regardless of path
2026-04-07 23:40:25 +04:00
Elie Habib
47af642d24 feat(energy): live chokepoint flow calibration from PortWatch DWT — V5-2 (#2797)
* feat(energy): chokepoint flow calibration seeder — V5-2 (Phase 4 PR A)

- Add CHOKEPOINT_FLOWS_KEY to server/_shared/cache-keys.ts
- Add energy:chokepoint-flows:v1 to health.js monitoring (maxStaleMin: 720)
- Add 6h chokepoint flow seed loop to ais-relay.cjs (seed-chokepoint-flows.mjs)
- Fix seeder to use degraded mode instead of throwing when PortWatch absent
- Add degraded-mode and ID-mapping tests to chokepoint-flows-seed.test.mjs

* fix(energy): restore throw for PortWatch absent + register seed-health

- seed-chokepoint-flows: revert degraded-path from warn+return{} back to
  throw; PortWatch absent is an upstream-not-ready error, not a data-quality
  issue — must throw so startChokepointFlowsSeedLoop schedules 20-min retry
- api/seed-health.js: add energy:chokepoint-flows to SEED_DOMAINS so
  /api/seed-health surfaces missing/stale signal (intervalMin: 360 = 6h cron)
- tests: update degraded-mode assertions to match restored throw behavior
2026-04-07 22:51:16 +04:00
Elie Habib
d2f39f606a feat(supply-chain): move PortWatch seeding to standalone seed-portwatch.mjs (#2770)
* feat(supply-chain): move PortWatch seeding to standalone seed-portwatch.mjs

Move PortWatch ArcGIS data fetching out of ais-relay.cjs into a new
standalone Railway cron seeder. Expands vessel type coverage from the
old n_tanker/n_cargo/n_total aggregate to all five individual types
(container, dry bulk, general cargo, RoRo, tanker) plus five DWT
capacity fields per chokepoint.

The relay no longer fetches from ArcGIS — it reads supply_chain:portwatch:v1
from Redis as before. seedTransitSummaries() hydration path is unchanged.

* fix(supply-chain): always read PortWatch from Redis in seedTransitSummaries

Remove the latestPortwatchData in-memory cache guard. With the seed loop
moved to standalone seed-portwatch.mjs, a warm relay would never re-read
from Redis after cold-start hydration, causing stale history and
wowChangePct to be republished into transit-summaries:v1 indefinitely.

seedTransitSummaries now reads supply_chain:portwatch:v1 fresh from
Redis on every 10-minute cycle, picking up the standalone seeder's 6h
refreshes immediately.
2026-04-06 19:58:53 +04:00
Elie Habib
8bcc051006 fix(usni): add missing region coords for Laem Chabang, Split, Pacific (#2711)
USNI fleet tracker reported unknown regions for vessels at
Laem Chabang (Thailand naval base), Split (Croatia, Adriatic),
and generic "Pacific" heading.
2026-04-05 08:39:52 +04:00
Elie Habib
0f0145770f fix(relay): add proxy fallback for Spending/GSCPI, fix OpenSky TLS (#2708)
* fix(relay): add proxy fallback for Spending/GSCPI, fix OpenSky TLS

- Spending (USASpending.gov): add proxy fallback for POST requests
- GSCPI (NY Fed CSV): add proxy fallback
- OpenSky: remove tls=false override that broke Decodo TLS tunnel
  (EPROTO: packet length too long). Decodo always requires TLS.
- Same TLS fix in seed-military-flights.mjs

* fix(relay): force tls=true on Spending/GSCPI proxy calls

parseProxyConfig defaults tls=true for bare strings, but if PROXY_URL
uses http:// scheme it would be false. Force tls=true to match all
other proxy calls in the relay (Decodo always requires TLS).
2026-04-05 08:33:03 +04:00
Elie Habib
15bd6f31ca refactor: consolidate proxy tunnel into shared _proxy-utils.cjs (#2702)
* refactor: consolidate 5 proxy tunnel implementations into _proxy-utils.cjs

5 near-identical HTTP CONNECT proxy tunnel implementations (3 in
ais-relay.cjs, 1 in _seed-utils.mjs, 1 in seed-military-flights.mjs)
consolidated into two shared functions in _proxy-utils.cjs:

- proxyConnectTunnel(): low-level CONNECT + TLS wrapping, returns socket
- proxyFetch(): high-level fetch with decompression, custom headers,
  POST support, timeout

All consumers now call the shared implementation:
- _seed-utils.mjs httpsProxyFetchRaw: 75 lines -> 6 lines
- ais-relay.cjs ytFetchViaProxy: 40 lines -> 5 lines
- ais-relay.cjs _openskyProxyConnect: 35 lines -> 8 lines
- ais-relay.cjs inline Dodo CONNECT: 25 lines -> 10 lines
- seed-military-flights.mjs proxyFetchJson: 70 lines -> 14 lines

Also wires weather alerts proxy fallback (fixes STALE_SEED health crit).

Net: -104 lines. Resolves the TODO at _seed-utils.mjs:311.

* fix(proxy): default tls=true for bare proxy strings

parseProxyConfig returned no tls field for bare-format proxies
(user:pass@host:port and host:port:user:pass). proxyConnectTunnel
checked proxyConfig.tls and used plain TCP when it was undefined,
breaking connections to Decodo which requires TLS. Only http:// URLs
should use plain TCP.

* fix(proxy): timeout covers full response, pass targetPort through

- Move clearTimeout from header arrival to stream end, so a server
  that stalls after 200 OK headers still hits the timeout
- Make targetPort configurable in proxyConnectTunnel (was hardcoded
  443), pass through from _openskyProxyConnect
2026-04-05 08:01:27 +04:00
Elie Habib
f82b41a9bd fix(market): replace dead GOGL.OL ticker with EGLE in shipping stress (#2705)
Golden Ocean Group (GOGL.OL) returns 404 on Yahoo Finance — ticker
delisted or dropped from index. Replace with Eagle Bulk Shipping
(EGLE), same dry bulk carrier sector.
2026-04-05 07:58:10 +04:00
Elie Habib
f68b9f6da0 fix(catalog): restore tls:true for Decodo proxy + Railway needs correct API key (#2700)
Decodo port 10001 requires TLS. Previous fix removed tls:true override,
causing "Parse Error: Expected HTTP/" on every proxy attempt.

Direct fetch 401 = DODO_API_KEY on Railway is the wrong key. Must use
the REST API key (rVYXvMEjbpQ...), not the Convex component key.
2026-04-05 07:27:28 +04:00
Elie Habib
36be5667d5 feat(catalog): seed Dodo prices from Railway relay with proxy fallback (#2680)
* feat(catalog): seed Dodo prices from Railway relay, Vercel reads only

Dodo API rejects Vercel Edge datacenter IPs (401). Moved price
fetching to ais-relay seed loop on Railway (1h interval, 2h TTL).
Direct fetch first, PROXY_URL fallback if blocked.

Vercel /api/product-catalog now reads from Redis only (gold standard
pattern). Falls back to static prices if Redis empty.

* fix(catalog): change Dodo price seed interval to 12h (TTL 24h)

* fix(catalog): add fixed_price support, restore edge Dodo fallback on purge

P1: Seeder now reads product.price?.price ?? product.price?.fixed_price
(matches previous edge endpoint behavior).

P1: After cache purge, edge endpoint tries Dodo directly as backup
before falling back to static prices. If Dodo succeeds, re-caches.
Prevents 12h gap of fallback-only after manual purge.

* fix(catalog): don't overwrite seeded Redis key from edge, add seed-meta

P1: Edge fallback no longer writes to Redis (avoids overwriting the
Railway-seeded entry with short TTL). Returns result directly with
60s cache.

P2: Seeder now writes seed-meta:product-catalog with fetchedAt,
recordCount, and priceSource for health monitoring.

* fix(catalog): auth proxy, only write dodo-sourced prices, 6h interval

P1: Proxy path now sends Authorization header (ytFetchViaProxy doesn't
support custom headers, so manual CONNECT tunnel with auth).

P1: Only writes to Redis when ALL prices come from Dodo (priceSource=dodo).
Partial/fallback results extend existing TTL but don't overwrite.
Prevents transient outages from pinning stale prices for hours.

Interval: 6h seed, 12h TTL.

* fix(catalog): add health.js monitoring, consistent cachedUntil

P2: Added productCatalog to STANDALONE_KEYS + SEED_META in health.js
(maxStaleMin: 1080 = 3x 6h interval).

P2: Fallback response now includes cachedUntil (consistent contract).

P1: Proxy CONNECT pattern matches existing ytFetchViaProxy (USNI)
which works for Decodo TLS proxies.

* fix(catalog): respect parsed proxy TLS flag instead of forcing tls:true
2026-04-04 18:33:48 +04:00
Elie Habib
4e9f25631c feat(economic): add FAO Food Price Index panel (#2682)
* feat(economic): add FAO Food Price Index panel

Adds a new panel tracking the FAO Global Food Price Index (FFPI) for the
past 12 months, complementing existing consumer prices, fuel prices, and
Big Mac Index trackers.

- proto: GetFaoFoodPriceIndex RPC with 6-series response (Food, Cereals,
  Meat, Dairy, Oils, Sugar + MoM/YoY pct)
- seeder: seed-fao-food-price-index.mjs with 90-day TTL (3× monthly),
  isMain guard, parseVal NaN safety, correct 13-point slice
- handler/gateway: static tier RPC wired into economicHandler
- bootstrap/health: bootstrapped as SLOW_KEY; maxStaleMin=86400 (60 days)
- panel: SVG multi-line chart with 6 series, auto-scaled Y axis, headline
  with MoM/YoY indicators, info tooltip, bootstrap hydration
- CMD+K: panel:fao-food-price-index with fao/ffpi/food keywords
- Railway: fao-ffpi cron seeder service (0.5 vCPU, 0.5 GB, daily 08:45)
- locales: full en.json keys for panel UI strings
- ais-relay: faoFoodPriceIndex added to economic bootstrap context

* fix(economic): add faoFoodPriceIndex to cache-keys.ts BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS

* fix(economic): correct cron comment in fao seeder to reflect daily schedule
2026-04-04 17:33:54 +04:00
Elie Habib
d04852bf02 fix(relay): proxy fallback for Yahoo/Crypto, isolate OREF proxy (#2627)
* fix(relay): proxy fallback for Yahoo/Crypto, isolate OREF proxy, fix Dockerfile

Yahoo Finance and CoinPaprika fail from Railway datacenter IPs (rate
limiting). Added PROXY_URL fallback to fetchYahooChartDirect (used by
5 seeders) and relay chart proxy endpoint. Added shared
_fetchCoinPaprikaTickers with proxy fallback + 5min cache (3 crypto
seeders share one fetch). Added CoinPaprika fallback to CryptoSectors
(previously had none).

Isolated OREF_PROXY_AUTH exclusively for OREF alerts. OpenSky,
seed-military-flights, and _proxy-utils now fall back to PROXY_URL
instead of the expensive IL-exit proxy.

Added seed-climate-news.mjs + _seed-utils.mjs COPY to Dockerfile.relay
(missing since PR #2532). Added pizzint bootstrap hydration to
cache-keys.ts, bootstrap.js, and src/services/pizzint.ts.

* fix(relay): address review — remove unused reverseMap, guard double proxy

- Remove dead reverseMap identity map in CryptoSectors Paprika fallback
- Add _proxied flag to handleYahooChartRequest._tryProxy to prevent
  double proxy call on timeout→destroy→error sequence
2026-04-03 00:08:37 +04:00
Elie Habib
7b6aae4670 fix(pizzint): move PizzINT fetch to Railway seed, fix stale data (#2626)
PizzINT API blocked Vercel datacenter IPs (403), causing the panel to
serve 5h-stale data showing all locations as CLOSED. Moved upstream
fetch to ais-relay seed loop on Railway (10min interval, 30min TTL)
following gold standard pattern. Vercel handler now reads from Redis
seed key only. Also fixed spike_magnitude string bug ("HIGH" instead
of number) from upstream API change.
2026-04-02 22:56:23 +04:00
Elie Habib
6b21566338 feat(news): composite importance score (E1) (#2621)
* feat(news): composite importance score (E1)

Adds a 0-100 importance score to every digest NewsItem, computed from four
weighted signals:

  severity    x 0.40  (critical=100, high=75, medium=50, low=25, info=0)
  source tier x 0.20  (tier 1 wire services score highest)
  corroboration x 0.25  (unique sources covering the same story in this cycle)
  recency     x 0.15  (linear decay to 0 over 30 min)

Key changes:
- proto: add importance_score (9) + corroboration_count (10) to NewsItem
- server/_shared/source-tiers.ts: new shared source tier lookup for server code
- list-feed-digest.ts: corroboration map before truncation, sort by importanceScore
- _classifier.ts: export SEVERITY_VALUES
- notification-relay.cjs: score gate behind IMPORTANCE_SCORE_LIVE env flag;
  shadow log to shadow:score-log:v1 (7-day window, always runs)
- ais-relay.cjs: include pubDate + importanceScore in rss_alert payloads

IMPORTANT: IMPORTANCE_SCORE_LIVE=1 must NOT be set until scores are validated
in shadow:score-log:v1 over several days.

* fix(news-alerts): add relay gates, importance score threshold, and RELAY_GATES_READY guard

- ais-relay.cjs: add RELAY_GATES_READY gate that skips tier-4 sources and
  stale items (>15min) when relay takes over external notifications
- breaking-news-alerts.ts: add RELAY_GATES_READY constant (VITE_ prefix for
  browser) and IMPORTANCE_SCORE_MIN=30 threshold; suppress /api/notify when
  relay is active; skip items below importance threshold in selectBest;
  propagate importanceScore into BreakingAlert for downstream logging

Fixes gap A (relay tier/recency gate) and gap B (RELAY_GATES_READY guard)
from docs/plans/2026-04-02-003-fix-news-alerts-pr-gaps-plan.md

* fix(relay): remove duplicate shadowLogScore from rebase artifact

Biome noRedeclare: two shadowLogScore definitions landed in
notification-relay.cjs after rebase. Removed the older version
(threshold=65, JSON blob) in favour of the newer E1 version
(SHADOW_SCORE_LOG_KEY constant, timestamped member encoding).
2026-04-02 21:35:20 +04:00
Elie Habib
c8beaf1fa2 feat(relay-gates): shadow score log + importance gate + 6h recency filter (#2607)
* feat(relay-gates): shadow score log + importance gate + recency filter

* fix(relay-gates): scope score gate to rss_alert only, fix shadow log ordering

P1: score gate fired on all event types; oref_siren, conflict_escalation,
notam_closure etc. have no importanceScore and read as 0, so they were
silently dropped when IMPORTANCE_SCORE_LIVE=1. Gate now only applies to
rss_alert events, which are the only type that carry importanceScore.

P2: shadow log used importanceScore as the sorted-set score, making it
neither time-sortable nor a true rolling window (EXPIRE refresh just
extended lifetime on each write). Now uses timestamp_ms as score for
time ordering, and prunes entries older than 7 days via ZREMRANGEBYSCORE
on each write instead of resetting a TTL. importanceScore is preserved
in the member string for analysis.

* feat(relay-gates): gate browser /api/notify dispatch behind VITE_RELAY_GATES_READY

When set to 'true', the client skips the XHR/fetch call to /api/notify so the
Railway relay becomes the sole notification path, eliminating double-dispatch.
Default (flag absent/false) preserves the existing browser-dispatch path.
2026-04-02 20:46:28 +04:00
Fayez Bast
b2bae30bd8 Add climate news seed and ListClimateNews RPC (#2532)
* Add climate news seed and ListClimateNews RPC

* Wire climate news into bootstrap and fix generated climate stubs

* fix(climate): align seed health interval and parse Atom entries per feed

* fix(climate-news): TTL 90min, retry timer on failure, named cache key constant

- CACHE_TTL: 1800 to 5400 (90min = 3x 30-min relay interval, gold standard)
- ais-relay: add 20-min retry timer on subprocess failure; clear on success
- cache-keys.ts: export CLIMATE_NEWS_KEY named constant
- list-climate-news.ts: import CLIMATE_NEWS_KEY instead of hard-coding string

---------

Co-authored-by: Elie Habib <elie.habib@gmail.com>
2026-04-02 08:55:22 +04:00
Elie Habib
5b0cf24ba1 feat(notifications): move alert detection fully server-side (#2590)
* feat(notifications): move event detection server-side in ais-relay

The notification system previously relied on the browser being open to
publish events to wm:events:queue. The relay queue was always empty
unless a user had worldmonitor loaded in a tab.

This moves all alert detection to ais-relay.cjs, which runs 24/7 on
Railway. Five detection points now publish directly to the relay queue:

- OREF sirens: fires on changed && alerts.length > 0 (every 5min)
- News classify: fires when LLM classifies a new title as critical/high (every 15min)
- NWS weather: fires for Extreme/Severe active alerts (every 15min)
- Aviation: fires for closure/severe/major airport disruptions (every 30min)
- NOTAM: fires for newly appeared ICAO airport closures (every 2h)

Infrastructure added: upstashLpush, upstashSetNx, publishNotificationEvent
with 30-min server-side scan-dedup (wm:notif:scan-dedup) to prevent the
same alert firing on every poll cycle.

The browser path (/api/notify) is unchanged and remains as a fallback for
when a tab is open. The relay already handles broadcast events correctly.

* feat(notifications): add market, finance, and threat alert detectors

Extends the server-side notification pipeline with 6 additional detectors:

Market & finance:
- Equity moves: stocks with |change%| >= 5 (high) or >= 10 (critical)
- Commodity moves: oil/gold/etc with |change%| >= 5 (high) or >= 10 (critical)
- Crypto moves: |24h change%| >= 10 (high) or >= 20 (critical)
  eventType: market_alert — 1h dedup per symbol/direction

Threat intelligence:
- Cyber threats: new critical/high-severity indicators from Feodo, URLhaus,
  OTX, AbuseIPDB — eventType: cyber_threat — 12h dedup per indicator
- Conflict escalation: UCDP events with >= 10 deaths (high) or >= 50 (critical)
  eventType: conflict_escalation — 24h dedup per event ID

Logistics/supply chain:
- Corridor risk: shipping corridors with risk score >= 50 (Hormuz, Red Sea, etc.)
  eventType: corridor_risk — 1h dedup per corridor
- Shipping stress index: global stress score >= 75 triggers high/critical
  eventType: shipping_stress — 2h dedup

Also: publishNotificationEvent gains optional dedupTtl param (default 1800s)
so callers can tune dedup window per event type without touching scan-dedup logic.

All thresholds chosen to avoid alert fatigue while catching genuinely notable
events. Module-level Sets (cyberPrevAlertedIds, ucdpPrevAlertedIds) track
notified items across cycles, auto-cleared at 500 entries.

* fix(notifications): rollback dedup key on LPUSH failure; scope rss_alert dedup per-variant

Two P1 fixes in publishNotificationEvent:

1. If LPUSH fails transiently the scan-dedup key was already set, silently
   suppressing retries for the full dedupTtl (30min-24h). Now calls
   upstashDel on LPUSH failure so the next poll cycle can retry.
   Added upstashDel helper following the same https.request pattern.

2. rss_alert scan-dedup key was global (eventType:title), so the first
   variant to classify a headline suppressed all subsequent variants.
   A finance or tech user missed alerts if the same headline appeared
   earlier in another variant pass. Key now includes variant suffix
   so each variant publishes independently; relay delivery dedup handles
   duplicates at the consumer side.

* fix(notifications): move rss_alert publish outside seenTitles guard

The publishNotificationEvent call was still inside the seenTitles.has()
check even after scoping the dedup key per-variant. seenTitles is shared
across all classify variants, so once variant A sees a title, variant B
never called publishNotificationEvent at all, bypassing the Redis dedup
entirely. Finance/tech users could still miss alerts for headlines first
encountered in another variant pass.

Fix: separate concerns at the call site. seenTitles guards country
attribution stats only (prevents double-counting). Notification publish
is now unconditional within the classified entry loop, relying solely on
the variant-scoped scan-dedup key in Redis for idempotency.

* fix(notifications): aviation change-detection, NOTAM restart-state, weather title fallback

Aviation P1: Added aviationPrevAlertedSet for in-process change detection so
only airports newly entering severe/major state trigger a notification. The
previous code had no such state, meaning every 30-min poll re-notified for
persistent disruptions since the default dedupTtl matched the poll interval.
dedupTtl raised to 4h to guard against restarts.

NOTAM P1: notamPrevClosed is now persisted to Redis (notam:prev-closed-state:v1)
after each seed and loaded before the first change-detection diff. Previously,
any Railway restart caused the first post-restart poll to treat all currently-
closed airports as new and fire up to 3 false-positive notifications. dedupTtl
raised to 6h (3x the 2h poll interval).

Weather P2: a.headline || a.event can be undefined when neither field is
populated, causing all title-less alerts to collide on a single dedup key.
Falls back to 'Weather alert' so each alert gets an accurate dedup hash.
2026-04-01 12:45:06 +04:00
Elie Habib
0b947ab897 fix(macro-signals): direct-first fetch for Yahoo, proxy as fallback only (#2508)
* fix(macro-signals): direct-first fetch for Yahoo, proxy as fallback only

fetchJsonSafe was proxy-only when proxyAuth was set — it called curlFetch
directly with no direct attempt. This caused Yahoo (and other APIs) to
silently fail whenever the proxy misbehaved, even when the API itself was
healthy. Mirrors the same direct-first pattern already in fredFetchJson.

* fix(shipping): use GOGL.OL (Oslo) ticker for Golden Ocean Group — NASDAQ GOGL returns 404 on Yahoo Finance
2026-03-29 16:51:20 +04:00
Elie Habib
ba54dc12d7 feat(commodity): gold layer enhancements (#2464)
* feat(commodity): add gold layer enhancements from fork review

Enrich the commodity variant with learnings from Yazan-Abuawwad/gold-monitor fork:

- Add 10 missing gold mines to MINING_SITES: Muruntau (world's largest
  open-pit gold mine), Kibali (DRC), Sukhoi Log (Russia, development),
  Ahafo (Ghana), Loulo-Gounkoto (Mali), South Deep (SA), Kumtor
  (Kyrgyzstan), Yanacocha (Peru), Cerro Negro (Argentina), Tropicana
  (Australia). Covers ~40% of top-20 global mines previously absent.

- Add XAUUSD=X spot gold and 9 FX pairs (EUR, GBP, JPY, CNY, INR, AUD,
  CHF, CAD, TRY) to shared/commodities.json. All =X symbols auto-seeded
  via existing seedCommodityQuotes() — no new seeder needed. Registered
  in YAHOO_ONLY_SYMBOLS in both _shared.ts and ais-relay.cjs.

- Add XAU/FX tab to CommoditiesPanel showing gold priced in 10 currencies.
  Computed live from GC=F * FX rates. Commodity variant only.

- Fix InsightsPanel brief title: commodity variant now shows
  "⛏️ COMMODITY BRIEF" instead of "🌍 WORLD BRIEF".

- Route commodity variant daily market brief to commodity feed categories
  (commodity-news, gold-silver, mining-news, energy, critical-minerals)
  via new newsCategories option on BuildDailyMarketBriefOptions.

- Add Gold Silver Worlds + FX Empire Gold direct RSS feeds to gold-silver
  panel (9 sources total, up from 7).

* fix(commodity): address review findings from PR #2464

- Fix USDCHF=X multiply direction: was true (wrong), now false (USD/CHF is USD-per-CHF convention)
- Fix newsCategories augments BRIEF_NEWS_CATEGORIES instead of replacing (preserves macro/Fed context in commodity brief)
- Add goldsilverworlds.com + www.fxempire.com to RSS allowlist (api + shared + scripts/shared)
- Rename "Metals" tab label conditionally: commodity variant gets "Metals", others keep "Commodities"
- Reset _tab to "commodities" when hasXau becomes false (prevent stale XAU tab re-activation)
- Add Number.isFinite() guard in _renderXau() before computing xauPrice
- Narrow fxMap filter to =X symbols only
- Collapse redundant two-branch number formatter to Math.round().toLocaleString()
- Remove XAUUSD=X from shared/commodities.json: seeded but never displayed (saves 150ms/cycle)

* feat(mcp): add get_commodity_geo tool and update get_market_data description

* fix(commodity): correct USDCHF direction, replace headline categories, restore dep overrides

* fix(commodity): empty XAU grid fallback and restore FRED timeout to 20s

* fix(commodity): remove XAU/USD from MCP description, revert Metals tab label

* fix(commodity): remove dead XAUUSD=X from YAHOO_ONLY_SYMBOLS

XAU widget uses GC=F as base price, not XAUUSD=X. Symbol was never
seeded (not in commodities.json) and never referenced in the UI.
2026-03-29 11:13:40 +04:00
Elie Habib
c8a66d2629 fix(wingbits): clamp bbox coords to valid geographic ranges before API call (#2453)
Map projections can send lamin/lamax outside [-90,90] (e.g. a zoomed-out
viewport). The isFinite validation passes these through, but Wingbits rejects
la < -90 with a Zod too_small error (HTTP 400). Clamp all four bbox values
to valid lat/lon ranges before computing centerLat/centerLon so the API
payload is always valid regardless of what the client sends.
2026-03-29 00:38:54 +04:00
Elie Habib
d01469ba9c feat(aviation): add SearchGoogleFlights and SearchGoogleDates RPCs (#2446)
* feat(aviation): add SearchGoogleFlights and SearchGoogleDates RPCs

Port Google Flights internal API from fli Python library as two new
aviation service RPCs routed through the Railway relay.

- Proto: SearchGoogleFlights and SearchGoogleDates messages and RPCs
- Relay: handleGoogleFlightsSearch and handleGoogleFlightsDates handlers
  with JSONP parsing, 61-day chunking for date ranges, cabin/stops/sort mappers
- Server handlers: forward params to relay google-flights endpoints
- gateway.ts: no-store for flights, medium cache for dates

* feat(mcp): expose search_flights and search_flight_prices_by_date tools

* test(mcp): update tool count to 24 after adding search_flights and search_flight_prices_by_date

* fix(aviation): address PR review issues in Google Flights RPCs

P1: airline filtering — use gfParseAirlines() in relay (handles comma-joined
  string from codegen) and parseStringArray() in server handlers

P1: partial chunk failure now sets degraded: true instead of silently
  returning incomplete data as success; relay includes partial: true flag

P2: round-trip date search validates trip_duration > 0 before proceeding;
  returns 400 when is_round_trip=true and duration is absent/zero

P2: relay mappers accept user-friendly aliases ('0'/'1' for max_stops,
  'price'/'departure' for sort_by) alongside symbolic enum values;
  MCP tool docs updated to match

* fix(aviation): use cachedFetchJson in searchGoogleDates for stampede protection

Medium cache tier (10 min) requires Redis-level coalescing to prevent
concurrent requests from all hitting the relay before cache warms.
Cache key includes all request params (sorted airlines for stable keys).

* fix(aviation): always use getAll() for airlines in relay; add multi-airline tests

The OR short-circuit (get() || getAll()) meant get() returned the first
airline value (truthy), so getAll() never ran and only one airline was
forwarded to Google. Fix: unconditionally use getAll().

Tests cover: multi-airline repeated params, single airline, empty array,
comma-joined string from codegen, partial degraded flag propagation.
2026-03-28 23:07:18 +04:00
Elie Habib
f5a1512d65 fix(usni): use https.request for proxy CONNECT (Decodo port 10001 is HTTPS) (#2408) 2026-03-28 11:32:53 +04:00
Elie Habib
966a8af9cb fix(relay): clear proxy CONNECT timeout before downloading response body (#2405) 2026-03-28 10:32:52 +04:00
Elie Habib
f56e7c24ad refactor(proxy): extract shared _proxy-utils.cjs, support Decodo host:port:user:pass format (#2399)
Previously each seeder (ais-relay.cjs, _seed-utils.mjs, seed-fear-greed.mjs,
seed-disease-outbreaks.mjs) had its own inline resolveProxy() with slightly
different implementations. This caused USNI seeding to fail because
parseProxyUrl() only handled URL format while PROXY_URL uses Decodo
host:port:user:pass format.

- Add scripts/_proxy-utils.cjs with parseProxyConfig(), resolveProxyConfig(),
  resolveProxyString() handling both http://user:pass@host:port and
  host:port:user:pass formats
- ais-relay.cjs: require _proxy-utils.cjs, alias parseProxyUrl = parseProxyConfig
- _seed-utils.mjs: import resolveProxyString via createRequire, delegate resolveProxy()
- seed-fear-greed.mjs, seed-disease-outbreaks.mjs: remove inline resolveProxy(),
  import from _seed-utils.mjs instead
2026-03-28 08:35:19 +04:00
Elie Habib
1e1f377078 feat(panels): Disease Outbreaks, Shipping Stress, Social Velocity, nuclear test site enrichment (#2375)
* feat(panels): Disease Outbreaks, Shipping Stress, Social Velocity, nuclear test site monitoring

- Add HealthService proto with ListDiseaseOutbreaks RPC (WHO + ProMED RSS)
- Add GetShippingStress RPC to SupplyChainService (Yahoo Finance carrier ETFs)
- Add GetSocialVelocity RPC to IntelligenceService (Reddit r/worldnews + r/geopolitics)
- Enrich earthquake seed with Haversine nuclear test-site proximity scoring
- Add 5 nuclear test sites to NUCLEAR_FACILITIES (Punggye-ri, Lop Nur, Novaya Zemlya, Nevada NTS, Semipalatinsk)
- Add shipping stress + social velocity seed loops to ais-relay.cjs
- Add seed-disease-outbreaks.mjs Railway cron script
- Wire all new RPCs: edge functions, handlers, gateway cache tiers, health.js STANDALONE_KEYS/SEED_META

* fix(relay): apply gold standard retry/TTL-extend pattern to shipping-stress and social-velocity seeders

* fix(review): address all PR #2375 review findings

- health.js: shippingStress maxStaleMin 30→45 (3x interval), socialVelocity 20→30 (3x interval)
- health.js: remove shippingStress/diseaseOutbreaks/socialVelocity from ON_DEMAND_KEYS (relay/cron seeds, not on-demand)
- cache-keys.ts: add shippingStress, diseaseOutbreaks, socialVelocity to BOOTSTRAP_CACHE_KEYS
- ais-relay.cjs: stressScore formula 50→40 (neutral market = moderate, not elevated)
- ais-relay.cjs: fetchedAt Date.now() (consistent with other seeders)
- ais-relay.cjs: deduplicate cross-subreddit article URLs in social velocity loop
- seed-disease-outbreaks.mjs: WHO URL → specific DON RSS endpoint (not dead general news feed)
- seed-disease-outbreaks.mjs: validate() requires outbreaks.length >= 1 (reject empty array)
- seed-disease-outbreaks.mjs: stable id using hash(link) not array index
- seed-disease-outbreaks.mjs: RSS regexes use [\s\S]*? for CDATA multiline content
- seed-earthquakes.mjs: Lop Nur coordinates corrected (41.39,89.03 not 41.75,88.35)
- seed-earthquakes.mjs: sourceVersion bumped to usgs-4.5-day-nuclear-v1
- earthquake.proto: fields 8-11 marked optional (distinguish not-enriched from enriched=false/0)
- buf generate: regenerate seismology service stubs

* revert(cache-keys): don't add new keys to bootstrap without frontend consumers

* fix(panels): address all P1/P2/P3 review findings for PR #2375

- proto: add INT64_ENCODING_NUMBER annotation + sebuf import to get_shipping_stress.proto (run make generate)
- bootstrap: register shippingStress (fast), socialVelocity (fast), diseaseOutbreaks (slow) in api/bootstrap.js + cache-keys.ts
- relay: update WIDGET_SYSTEM_PROMPT with new bootstrap keys and live RPCs for health/supply-chain/intelligence
- seeder: remove broken ProMED feed URL (promedmail.org/feed/ returns HTML 404); add 500K size guard to fetchRssItems; replace private COUNTRY_CODE_MAP with shared geo-extract.mjs; remove permanently-empty location field; bump sourceVersion to who-don-rss-v2
- handlers: remove dead .catch from all 3 new RPC handlers; fix stressLevel fallback to low; fix fetchedAt fallback to 0
- services: add fetchShippingStress, disease-outbreaks.ts, social-velocity.ts with getHydratedData consumers
2026-03-27 22:33:45 +04:00
Elie Habib
47f0dd133d fix(widgets): restore iframe content after drag, remove color-cycle button (#2368)
* fix(widgets): restore iframe content after drag, remove color-cycle button

- Fix drag-induced blank content: use WeakMap keyed by iframe element to persist
  HTML across DOM moves; persistent load listener (no {once}) re-posts on every
  browser re-navigation triggered by drag/drop repositioning
- Remove cycleAccentColor, ACCENT_COLORS, and colorBtn from CustomWidgetPanel
  header; chatBtn (sparkle) and PRO badge remain; applyAccentColor kept for
  saved specs
- Update tests: remove ACCENT_COLORS count test, saveWidget persistence test,
  and changeAccent i18n assertion (all for deleted feature)

* fix(widgets): use correct postMessage key 'html' not 'storedHtml'

* fix(widgets): remove duplicate panel title header, fix sandbox CSP beacon error

- System prompt: NEVER add .panel-header or title to widget body; outer panel
  frame already shows the title; updated both basic and PRO prompts
- widget-sanitizer: strip leading .panel-header from generated HTML as safety
  net in both wrapWidgetHtml and wrapProWidgetHtml
- vercel.json: add https://static.cloudflareinsights.com to sandbox script-src
  so Cloudflare beacon injection no longer triggers CSP console errors

* fix(widgets): correct iframe font by anchoring html+body font-family with !important
2026-03-27 16:52:56 +04:00
Elie Habib
7f594a31c9 feat(widget-agent): prompt injection guardrails + scope enforcement (#2367)
Prevent users from hijacking the widget agent for off-topic AI tasks
or prompt injection attacks that would waste Anthropic API budget.

Layer 1 -- Hard reject before any API call (isWidgetInjectionAttempt):
- Detects instruction override patterns (ignore/disregard/forget previous
  instructions), role hijacking (act as, pretend to be, your new persona),
  prompt exfiltration (show/reveal system prompt), structural markers
  ([SYSTEM], <system>, ###system), and jailbreak vocabulary (DAN, developer
  mode). Returns 400 with no tokens spent.

Layer 2 -- System prompt scope enforcement (both basic and PRO):
- Added NON-NEGOTIABLE refusal block at top of both system prompts.
  Model is instructed to refuse all off-topic requests silently with a
  fixed widget-html stub response instead of engaging or explaining.

Layer 3 -- Tool result sanitization (sanitizeToolContent):
- Applied to all web search results and WM API responses before they
  enter the conversation context. Strips obvious injection patterns from
  untrusted third-party content (snippets, titles, API fields).
2026-03-27 16:06:40 +04:00
Elie Habib
62b65b648f feat(widget-agent): bootstrap catalog + tool budget + emergency emit (#2366)
* chore: redeploy to pick up WORLDMONITOR_VALID_KEYS fix

* feat(widget-agent): bootstrap catalog + tool budget + emergency emit

Fix design drift and tool loop exhaustion in the widget agent.

Bootstrap data source:
- Add /api/bootstrap?keys=<key> as the PREFERRED data source in both
  system prompts. Bootstrap is pre-seeded, instant, and returns the
  exact same data the dashboard panels display (techReadiness, riskScores,
  marketQuotes, earthquakes, wildfires, etc. -- 60+ keys).
- Reorganize prompt: Option 1 = bootstrap (grouped by category), Option 2
  = live RPCs (only for parameterized/custom queries). search_web is last resort.

Tool budget (Option E):
- Add "max 3 tool calls" rule to both system prompts. After 2 failed calls,
  agent must emit HTML immediately -- no more probing. Prevents the agent
  burning all turns on sequential endpoint guesses.

Emergency emit (Option D):
- On turn maxTurns-2, inject a FINAL TURN directive and pass tools:[]
  to force end_turn instead of another tool_use. Agent must emit HTML
  now with whatever data it collected.

Partial recovery:
- On loop exhaustion, scan all assistant messages for any partial
  widget-html markers and emit the best one found, instead of always
  returning an error to the user.
2026-03-27 15:58:06 +04:00
Elie Habib
f9e127471f fix(widget): sandbox connect-src cdn.jsdelivr.net + Sentry CSP/5xx tracking (#2365)
* fix(widget): allow cdn.jsdelivr.net in sandbox CSP + Sentry error tracking

- Fix Chart.js source map noise: relax sandbox connect-src from 'none' to
  https://cdn.jsdelivr.net (both vercel.json header and meta CSP in buildWidgetDoc)
- Add Sentry API 5xx capture in premiumFetch via reportServerError() -- fires on
  any status >= 500 before response is returned, tags kind: api_5xx
- Add securitypolicyviolation listener in main.ts for parent-page CSP violations,
  filters browser-extension and blob origins, tags kind: csp_violation

* feat(widget): inject panel design system into PRO widget sandbox

Problem: PRO widgets used a disconnected design (large bold titles,
custom tab buttons, hardcoded hex colors) because the sandbox iframe
had no panel CSS classes and the agent had no examples to follow.

Fix:
- buildWidgetDoc: add .panel-header, .panel-title, .panel-tabs,
  .panel-tab, .panel-tab.active, .disp-stats-grid, .disp-stat-box,
  .disp-stat-value, .disp-stat-label, and --accent CSS variable to
  the iframe's <style> block so they work without a custom <style>
- WIDGET_PRO_SYSTEM_PROMPT: add concrete HTML examples for panel
  header+tabs, stat boxes, and Chart.js color setup using CSS vars;
  prohibit h1/h2/h3 large titles; document the switchTab() pattern
- Test: assert all panel classes and --accent are present in document

Agent now has classes to USE instead of inventing its own styling.

* feat(widget-agent): open API allowlist to all /api/ paths with compact taxonomy

Problem: widget agent only knew 14 hardcoded endpoints and prioritized
search_web even when a WorldMonitor data source was available.

- Replace WIDGET_ALLOWED_ENDPOINTS Set with isWidgetEndpointAllowed()
  function: permits any /api/ path, blocks inference/write endpoints
  (analyze-stock, backtest-stock, summarize-article, classify-event, etc.)
- Replace per-URL endpoint lists in both WIDGET_SYSTEM_PROMPT and
  WIDGET_PRO_SYSTEM_PROMPT with a compact service-grouped taxonomy:
  service + method names only, no full URL repeated 60 times (~400
  tokens vs ~1200 for 4x more endpoint coverage)
- Strengthen prioritization: "ALWAYS use first, ONLY fall back to
  search_web if no matching service exists" (was "preferred for topics")
- Add 30+ new endpoints: earthquakes, wildfires, cyber threats, sanctions,
  consumer prices, FRED series, BLS, Big Mac, fuel, grocery, ETF flows,
  shipping rates, chokepoints, critical minerals, GPS interference, etc.

* fix(csp): add safari-web-extension: scheme to CSP violation filter
2026-03-27 15:52:02 +04:00
Elie Habib
6dbd2be2f1 fix(usni): try US proxy fallback when IL proxy fails for USNI fleet scrape (#2363)
* fix(usni): try US proxy fallback when IL proxy fails for USNI fleet scrape

IL-targeted OREF proxy was getting Cloudflare-blocked on USNI (US news site).
Now tries IL proxy first, then YOUTUBE_PROXY_URL (US Decodo proxy), then direct.
Prevents STALE_SEED health warnings when IL exit nodes are blocked.

* fix(relay): use PROXY_URL for USNI, not OREF_PROXY_AUTH (IL-only)

* fix(usni): hoist PROXY_URL constant, separate JSON parse errors from proxy errors
2026-03-27 15:46:21 +04:00
Elie Habib
130a8c04fc fix(widget-agent): correct Sonnet 4.6 model ID (no date suffix) (#2357) 2026-03-27 12:39:51 +04:00
Elie Habib
5d68f0ae6b fix(intelligence): land news:threat:summary:v1 CII work missed from PR #2096 (#2356)
* feat(intelligence): emit news:threat:summary:v1 from relay classify loop for CII

During seedClassifyForVariant(), attribute each title to ISO2 countries
while both title and classification result are in scope. At the end of
seedClassify(), merge per-country threat counts across all variants and
write news:threat:summary:v1 (20min TTL) with { byCountry: { [iso2]: {
critical, high, medium, low, info } }, generatedAt }.

get-risk-scores.ts reads the new key via fetchAuxiliarySources() and
applies weighted scores (critical→4, high→2, medium→1, low→0.5, info→0,
capped at 20) per country into the information component of CII eventScore.

Closes #2053

* fix(intelligence): register news:threat-summary in health.js and expand tests

- Add newsThreatSummary to BOOTSTRAP_KEYS (seed-meta:news:threat-summary,
  maxStaleMin: 60) so relay classify outages surface in health dashboard
- Add 4 tests: boost verification, cap-at-20, unknown-country safety,
  null-threatSummary zero baseline

* fix(classify): de-dup cross-variant titles and attribute to last-mentioned country

P1-A: seedClassify() was summing byCountry across all 5 variants (full/tech/
finance/happy/commodity) without de-duplicating. Shared feeds (CNBC, Yahoo
Finance, FT, HN, Ars) let a single headline count up to 4x before reaching
CII, saturating threatSummaryScore on one story.
Fix: pass seenTitles Set into seedClassifyForVariant; skip attribution for
titles already counted by an earlier variant.

P1-B: matchCountryNamesInText() was attributing every country mentioned in a
headline equally. "UK and US launch strikes on Yemen" raised GB, US, and YE
with identical weight, inflating actor-country CII.
Fix: return only the last country in document order — the grammatical object
of the headline, which is the primary affected country in SVO structure.

* fix(classify): replace last-position heuristic with preposition-pattern attribution

The previous "last-mentioned country" fix still failed for:
- "Yemen says UK and US strikes hit Hodeidah" → returned US (wrong)
- "US strikes on Yemen condemned by Iran" → returned IR (wrong)

Both failures stem from position not conveying grammatical role. Switch to a
preposition/verb-pattern approach: only attribute to a country that immediately
follows a locative preposition (in/on/against/at/into/targeting/toward) or an
attack verb (invades/attacks/bombs/hits/strikes). No pattern match → return []
(skip attribution rather than attribute to the wrong country).

* fix(classify): fix regex hitting, gaza/hamas geo mapping, seed-meta always written

- hitt?(?:ing|s)? instead of hit(?:s|ting)? so "hitting" is matched
- gaza → PS (Palestinian Territories), hamas → PS (was IL)
- seed-meta:news:threat-summary written unconditionally so health check
  does not fire false alerts during no-attribution runs
2026-03-27 12:21:23 +04:00
Elie Habib
eb253b24c5 fix(widget-agent): support Clerk JWT auth + fix P1/P2 relay issues (#2306)
Clerk PRO users had no path to authenticate to the widget agent.
Their PRO status comes from getAuthState().user?.role === 'pro',
not from wm-widget-key/wm-pro-key (tester keys), so getWidgetAgentKey()
returned '' and every request got 403.

Fix:
- Add api/widget-agent.ts (Vercel edge): validates Clerk Bearer JWT
  (plan === 'pro') OR tester keys, then proxies SSE to relay with
  real server-side WIDGET_AGENT_KEY/PRO_WIDGET_KEY. Clerk users never
  see the relay secrets.
- Update widgetAgentUrl() to route prod browser traffic through
  /api/widget-agent. runtime.ts interceptor injects Authorization:
  Bearer <clerk_jwt> automatically for same-origin requests.
- Desktop still bypasses to relay directly (SSE streaming compat).

Also fix two relay issues from code review of #2304:
- P2: requireWidgetAgentAccess returned 503 when WIDGET_AGENT_KEY was
  empty even if PRO_WIDGET_KEY was set — blocks PRO-only deployments.
  Fix: guard is now !widgetKeyConfigured && !proKeyConfigured.
- P1: request admitted via pro key with no body.tier defaulted to
  basic rate-limit bucket. requireWidgetAgentAccess now returns
  admittedAs:'pro'|'basic'; handleWidgetAgentRequest uses it to
  default tier to 'pro' and skip the redundant pro-key re-check.
2026-03-26 20:25:18 +04:00
Elie Habib
0f288afbde fix(widget-agent): allow PRO key alone to pass requireWidgetAgentAccess (#2304)
PRO users who have wm-pro-key but no wm-widget-key were getting 403
because requireWidgetAgentAccess rejected the empty X-Widget-Key before
reaching the PRO key check. Allow a valid X-Pro-Key to satisfy the gate
on its own since PRO tier is a superset of basic.
2026-03-26 20:07:58 +04:00
Elie Habib
1a2f1abf27 fix(wingbits): cap bbox to 2000 nm — stop persistent 400 errors (#2293)
* fix(wingbits): cap bbox dimensions to 2000 nm to stop persistent 400 errors

The v1 API rejects requests with overly large bounding boxes. Three call
sites were sending sizes that exceeded the API limit:
- Global callsign fallback: replaced w:21600 h:10800 full-globe box with
  three 2000 nm regional areas (Europe/Middle East, Asia-Pacific, Americas)
- Viewport handler: added Math.min(..., WINGBITS_MAX_BOX_NM) cap so
  zoomed-out views don't produce multi-thousand-nm requests
- Theater posture handler: same cap on per-theater w/h computation

Also added response body logging to both error paths so the actual API
error message surfaces in logs rather than just the status code.

* fix(wingbits): log error body on regional callsign fallback non-OK response
2026-03-26 17:50:26 +04:00
Elie Habib
d851921fe1 fix(relay): refresh seed-meta on empty NWS response to prevent false STALE_SEED (#2256)
Root cause: seedWeatherAlerts() had two early-return paths that skipped the
seed-meta upstash write — alerts.length===0 (quiet weather) and !resp.ok.
After a transient NWS fetch failure, subsequent successful-but-empty runs
never bumped fetchedAt, causing health.js to see the old timestamp grow stale.

- Update seed-meta on alerts.length===0 (NWS OK, just no active alerts)
- Keep !resp.ok path as-is (prolonged NWS outage should still alert)
- Add weatherAlerts to EMPTY_DATA_OK_KEYS (0 alerts = valid quiet state)
2026-03-25 23:21:46 +04:00
Elie Habib
7a98b35988 fix(notam): reduce ICAO polling to 2h, detect quota exhaustion with 24h backoff (#2145)
ICAO free tier allows ~1000 calls/month. 30min interval = 1440/month, burning
quota in ~3 weeks. Reduce to 2h (360 calls/month).

On "Reach call limit" response: resolve null (sentinel), extend data key TTL,
write seed-meta with quotaExhausted:true so health.js stays green, back off 24h
instead of retrying in 20min (which would waste remaining quota).

Update health.js notamClosures maxStaleMin 90 -> 240 (2x 2h interval).
2026-03-23 15:55:30 +04:00
Elie Habib
eaf4c771ad feat(market+economy): add Russell 2000 index and GSCPI seeder (#2140)
Russell 2000 (^RUT):
- Add to MARKET_SYMBOLS and YAHOO_ONLY in ais-relay.cjs
- Covers the 4th major US index missing from the panel

GSCPI (NY Fed Global Supply Chain Pressure Index):
- New seedGscpi() loop in ais-relay.cjs — fetches monthly CSV from
  newyorkfed.org (no API key required), parses wide-format vintage CSV
- Stored as economic:fred:v1:GSCPI:0 in FRED-compatible format so the
  existing GetFredSeriesBatch RPC serves it without any proto changes
- TTL 72h (3x 24h interval), retry 20min on failure — gold standard pattern
- Add GSCPI to ALLOWED_SERIES in get-fred-series-batch.ts
- Add gscpi to STANDALONE_KEYS + SEED_META in health.js (maxStaleMin 2880)
2026-03-23 15:30:20 +04:00
Elie Habib
1d28c352da feat(commodities): expand tracking to 23 symbols — agriculture and coal (#2135)
* feat(commodities): expand tracking to cover agricultural and coal futures

Adds 9 new commodity symbols to cover the price rally visible in our
intelligence feeds: Newcastle Coal (MTF=F), Wheat (ZW=F), Corn (ZC=F),
Soybeans (ZS=F), Rough Rice (ZR=F), Coffee (KC=F), Sugar (SB=F),
Cocoa (CC=F), and Cotton (CT=F).

Also fixes ais-relay seeder to use display names from commodities.json
instead of raw symbols, so seeded data is self-consistent.

* fix(commodities): gold standard cache, 3-col grid, cleanup

- Add upstashExpire on zero-quotes failure path so bootstrap key TTL
  extends during Yahoo outages (gold standard pattern)
- Remove unreachable fallback in retry loop (COMMODITY_META always has
  the symbol since it mirrors COMMODITY_SYMBOLS)
- Switch commodities panel to 3-column grid (19 items → ~7 rows vs 10)
2026-03-23 14:19:20 +04:00
Elie Habib
68b38b29ea fix(seeders): apply gold standard TTL-extend+retry pattern to Aviation, NOTAM, Cyber, PositiveEvents (#2127) 2026-03-23 10:56:00 +04:00
Elie Habib
b33d30578c fix(satellites): resilient seed — extend TTL on failure, retry in 20min, 6h TTL, health 240min (#2125) 2026-03-23 10:54:22 +04:00
Elie Habib
df29d59ff7 fix(health): enforce 1h+ TTL buffer across all seed jobs (#2072)
Full audit of seed TTL vs cron cadence. Rule: TTL >= cron_interval + 1h.

CRITICAL (TTL = cron, 0 buffer):
- seed-supply-chain-trade: tariffTrendsUs TRADE_TTL(6h) → TARIFF_TTL(8h)
- seed-supply-chain-trade: customsRevenue TRADE_TTL(6h) → CUSTOMS_TTL(24h)
- seed-sanctions-pressure: CACHE_TTL 12h → 15h (12h cron, 3h buffer)
- seed-usa-spending: CACHE_TTL 1h → 2h (1h cron, 1h buffer)

WARN (<1h buffer):
- seed-security-advisories: TTL 2h → 3h (1h cron, now 2h buffer)
- seed-token-panels: TTL 1h → 90min (30min cron, now 1h buffer)
- seed-etf-flows: TTL 1h → 90min (15min cron, now 75min buffer)
- seed-stablecoin-markets: TTL 1h → 90min (10min cron, now 80min buffer)
- seed-gulf-quotes: TTL 1h → 90min (10min cron, now 80min buffer)
- seed-crypto-quotes: TTL 1h → 2h (5min cron, now 115min buffer)
- ais-relay CRYPTO_SEED_TTL: 1h → 2h
- ais-relay STABLECOIN_SEED_TTL: 1h → 2h
- ais-relay SECTORS_SEED_TTL: 1h → 2h
2026-03-22 22:55:06 +04:00
Elie Habib
ab26f3c62f fix(relay): add global Wingbits fallback for callsign-only index misses (#2030)
* fix(relay): add global Wingbits fallback for callsign-only index misses

When a callsign search hits the relay but the flight isn't in any
recent viewport (index miss), the relay now attempts a worldwide
Wingbits API call filtered by callsign server-side.

Previously: index miss → immediate empty response
Now: index miss → global bbox call → filter by callsign → return matches

Fallback is graceful: timeouts/errors return empty rather than 5xx.
Matched flights are also written into the index for subsequent queries.

* fix(aviation): try Wingbits before OpenSky for callsign-only searches

Commercial flights like UAE20, THY6260 are Wingbits-exclusive and
invisible to OpenSky receivers. The old order (OpenSky first) wasted
10s+ on OpenSky global states/all (rate-limited, no callsign filter),
then returned early if any OpenSky positions existed — never reaching
Wingbits. Result: source:"none" even when the flight was visible on map.

New order for callsign-only queries:
  1. Wingbits relay (index + global fallback on miss) — fast and exact
  2. OpenSky skipped (no callsign filter, rate-limited, wrong source)

OpenSky and Wingbits bbox fallback paths unchanged for bbox/icao24 queries.
2026-03-22 08:48:25 +04:00
Elie Habib
90fabfdddb fix(aviation): fix callsign search by using in-memory position index in relay (#2026)
The global Wingbits bbox call (-80/-180/80/180) was unreliable — Wingbits
returns 401/empty for worldwide queries. Replace it with an in-memory index
populated from every successful bbox response. Callsign-only queries check the
index (fresh within 5 min) without hitting the Wingbits API.

Also fix cache key bug: callsign-only searches previously fell through to
'aviation:track:all:v1' (shared across all searches). Now keyed per callsign.
Use shorter TTLs: 60s positive, 10s negative for callsign searches.

Result: if CTN465 or SWR785B is visible in any user's viewport, the relay
index has it and CMD+K 'flight CTN465' returns it immediately.
2026-03-21 23:25:53 +04:00
Elie Habib
7008cd5959 fix(aviation): route callsign search through Wingbits relay, suppress simulated fallback (#2019)
- Extend /wingbits/track relay endpoint to accept ?callsign=EK36 without bbox,
  using global bounding box (-80/-180/80/180) for worldwide aircraft lookup
- Filter by callsign in relay response loop to return only matching flights
- Server: try Wingbits for callsign-only searches (previously only tried for bbox)
- Server: return empty positions (not simulated) for explicit callsign/icao24 lookups
- Client: filter out simulated-source positions before displaying search results
2026-03-21 22:57:03 +04:00
Elie Habib
e0800f1a31 fix(resilience): extend TTLs and add Finnhub/FRED fallbacks for macro signals (#1982)
* fix(panels): hide finance-only panels on non-finance variants

stock-analysis, stock-backtest, and daily-market-brief are FINANCE_PANELS-only
but persisted panelSettings from a previous finance/full session caused them
to appear on tech variant. Guard creation with variantPanelKeys set derived
from VARIANT_DEFAULTS[SITE_VARIANT].

* fix(pro): remove duplicate Create with AI button, gate MCP connect as pro

With unified isProUser() both widget creation buttons showed simultaneously.
Remove the basic-tier 'Create with AI' block (duplicate of 'Create Interactive
Widget'). Gate 'Connect with MCP' with isProUser() — was previously ungated.

* feat(pro): gate export (⬇) and playback () toolbar buttons for pro users

Both setupExportPanel() and setupPlaybackControl() now return early
unless isProUser() — either wm-widget-key or wm-pro-key grants access.

* revert(panels): remove incorrect variant guard on stock/daily-market panels

The variantPanelKeys.has() guards were wrong — shouldCreatePanel() already
handles variant defaults via panelSettings. Users who manually enable these
panels on any variant should see them (locked if not pro, unlocked if pro).
The guard broke cross-variant user customization.

* fix(resilience): extend TTLs and add Finnhub/FRED fallbacks for macro signals

- MACRO_TTL 1800s → 21600s (6h) so stale data serves during Yahoo outages
- MARKET_SEED_TTL 1800s → 7200s (2h) in ais-relay for sectors/quotes
- Finnhub stock/candle fallback for QQQ and XLP when Yahoo returns null
- Finnhub crypto/candle fallback for BTC (BINANCE:BTCUSDT) when Yahoo fails
- FRED DEXJPUS fallback for JPY/USD historical prices (no new API key needed)

Prevents Macro Stress and Sector Heatmap panels from showing "unavailable"
during Yahoo Finance 429 rate-limit windows from Railway IPs.
2026-03-21 12:28:44 +04:00
Elie Habib
87fe620c97 fix(tokens): add CoinPaprika fallback and runSeed() to token panels (#1977)
CoinGecko was silently failing on the relay (no API key, rate-limited by
the main crypto seed loop), leaving defi/ai/other-tokens keys empty.

- seed-token-panels.mjs: rewrite with CoinGecko → CoinPaprika fallback,
  use runSeed() wrapper so seed-meta:market:token-panels is written
- ais-relay.cjs: add fetchTokenPanelsCoinPaprika() fallback; relay no
  longer silently drops token panels on CoinGecko failure
- health.js: add defiTokens, aiTokens, otherTokens to BOOTSTRAP_KEYS;
  add tokenPanels to SEED_META (maxStaleMin: 90, 30-min cron × 3)
- Railway: provision seed-token-panels cron service (*/30 * * * *,
  1vCPU/1GB, watchPatterns scoped to token panel files)
2026-03-21 11:27:02 +04:00
Elie Habib
549084fbca fix(relay): add missing USNI regions and remove dead NZ safetravel feed (#1969)
* fix(relay): add missing USNI regions and remove dead NZ safetravel feed

USNI fleet tracker warns on unknown regions "Tasman Sea" and "Eastern
Atlantic" — add coords to USNI_REGION_COORDS map.

safetravel.govt.nz/news/feed redirects to /404 on every request; remove
from advisory feeds and RSS allowed domains.

* fix: remove safetravel.govt.nz from edge function allowed domains copy

* fix: remove safetravel.govt.nz from shared allowed domains copy
2026-03-21 08:58:21 +04:00