Commit Graph

296 Commits

Author SHA1 Message Date
Elie Habib
8cdca53bd8 feat(forecast): persist run-level world state (#1773)
* feat(forecast): persist run-level world state

* fix(forecast): align world state artifacts
2026-03-17 18:21:56 +04:00
Elie Habib
6d8109a85b feat(widgets): PRO interactive widgets via iframe srcdoc (#1771)
* feat(widgets): add PRO interactive widgets via iframe srcdoc

Introduces a PRO tier for AI-generated widgets that supports full JS
execution (Chart.js, sortable tables, animated counters) via sandboxed
iframes — no Docker, no build step required.

Key design decisions:
- Server returns <body> + inline <script> only; client builds the full
  <!DOCTYPE html> skeleton with CSP guaranteed as the first <head> child
  so the AI can never inject or bypass the security policy
- sandbox="allow-scripts" only — no allow-same-origin, no allow-forms
- PRO HTML stored in separate wm-pro-html-{id} localStorage key to
  isolate 80KB quota pressure from the main widget metadata array
- Raw localStorage.setItem() for PRO writes with HTML-first write order
  and metadata rollback on failure (bypasses saveToStorage which swallows
  QuotaExceededError)
- Separate PRO_WIDGET_KEY env var + x-pro-key header gate on Railway
- Separate rate limit bucket (20/hr PRO vs 10/hr basic)
- Claude Sonnet 4.6 (8192 tokens, 10 turns, 120s) for PRO vs Haiku for
  basic; health endpoint exposes proKeyConfigured for modal preflight

* feat(pro): gate finance panels and widget buttons behind wm-pro-key

The PRO localStorage key now unlocks the three previously desktop-only
finance panels (stock-analysis, stock-backtest, daily-market-brief) on
the web variant, giving PRO users access without needing WORLDMONITOR_API_KEY.

Button visibility is now cleanly separated by key:
- wm-widget-key only → basic "Create with AI" button
- wm-pro-key only    → PRO "Create Interactive" button only
- both keys          → both buttons
- no key             → neither button

Widget boot loader also accepts either key so PRO-only users see their
saved interactive widgets on page load.

* fix(widgets): inject Chart.js CDN into PRO iframe shell so new Chart() is defined
2026-03-17 18:10:10 +04:00
Elie Habib
76fe050b01 feat(health): restore seed-meta tracking for 4 degraded keys (#1769)
* feat(health): restore seed-meta tracking for riskScores, serviceStatuses, cableHealth, chokepoints

These 4 keys were reporting STALE_SEED / going untracked because their
warm-ping loops never wrote seed-meta. PR #1649 removed seed-meta from
cachedFetchJson but no replacement tracking was added, so health.js
lost visibility into their freshness.

Changes:
- ais-relay.cjs: seedCiiWarmPing() writes seed-meta:intelligence:risk-scores after success
- ais-relay.cjs: seedServiceStatuses() writes seed-meta:infra:service-statuses after success
- ais-relay.cjs: new startChokepointWarmPingLoop() — 30 min warm-ping for supply_chain:chokepoints:v4
- ais-relay.cjs: new startCableHealthWarmPingLoop() — 30 min warm-ping + seed-meta:cable-health write
- get-cable-health.ts: switch to cachedFetchJsonWithMeta, write seed-meta:cable-health on fresh fetch
- api/health.js: re-add SEED_META entries for serviceStatuses (30 min), cableHealth (60 min), riskScores (15 min)
- api/health.js: remove riskScores, serviceStatuses, cableHealth from ON_DEMAND_KEYS — they now have proper freshness tracking

* fix(health): only write seed-meta on genuinely fresh data (P1 review fixes)

Fixes two P1 issues from PR review:

1. seedCableHealthWarmPing() was writing seed-meta:cable-health after
   any 200 response, defeating the source==='fresh' guard already present
   in getCableHealth(). Removed the relay write — the handler owns it.

2. seedServiceStatuses() was writing seed-meta:infra:service-statuses
   after any 200, but listServiceStatuses() can return in-memory fallback
   statuses on upstream scrape failures with a 200. The relay write was
   advancing fetchedAt even when stale fallback data was returned.
   Fix: switch handler to cachedFetchJsonWithMeta and only write seed-meta
   when source==='fresh' (i.e. upstream status pages were actually scraped).
   Removed the relay write entirely.

* fix(health): only write risk-scores seed-meta when data is present

The CII warm-ping wrote seed-meta after any 200 response, but the RPC
can return cached/stale fallback data with 0 scores during upstream
outages. This masked staleness in health checks. Now only writes
seed-meta when count > 0 (meaningful data received).
2026-03-17 18:03:59 +04:00
Elie Habib
894a3a1108 fix(seeds): extend WB key TTLs in relay drop guard (#1770)
* fix(seeds): extend WB key TTLs in relay drop guard

The ais-relay seedWorldBank() percentage-drop guard returned early
without extending TTLs. During persistent partial WB outages, this
would let all 6 keys (3 data + 3 seed-meta) expire after 7 days.
Now extends TTLs on all WB keys before returning, matching the
standalone seed-wb-indicators.mjs behavior.

* fix(seeds): check upstashExpire return values in WB drop guard

upstashExpire returns false on failure (never throws), so the prior
code always logged success. Now checks all 6 return values and logs
partial failure count if any EXPIRE calls fail.
2026-03-17 17:55:59 +04:00
Elie Habib
6a87e56dfc fix(ucdp): never overwrite existing data with empty results (#1766)
* fix(ucdp): never overwrite existing data with empty results

The standalone UCDP seed was writing 0 events to Redis when the API
returned empty, overwriting the last known good data. Health then
reported EMPTY_DATA CRIT even though valid data existed before.

Now extends TTL on both the data key and seed-meta when 0 events
are produced, preserving the last good payload until the next
successful fetch.

* fix(ucdp): verify EXPIRE responses before logging success

Check HTTP status of both EXPIRE calls. Log warnings on failure
instead of always claiming TTL was extended.
2026-03-17 17:45:13 +04:00
Elie Habib
83fe44afa3 fix(seeds): add empty-data guards and fix health semantics (#1767)
Health semantics:
- Add faaDelays + gpsjam to EMPTY_DATA_OK_KEYS (0 records = calm, not error)
- Fix EMPTY_DATA_OK_KEYS branch to still check seed-meta freshness
  (prevents stale empty caches from staying green indefinitely)

Seed guards:
- seed-airport-delays: fix meta key in fetch-failure path
  (seed-meta:aviation:delays -> seed-meta:aviation:faa + seed-meta:aviation:notam)
- seed-military-flights: add full TTL extension on zero-flights branch
  (was exiting without preserving any derived data TTLs)
- seed-wb-indicators: add percentage-drop guard (new count < 50% of cached
  = likely partial API failure, extend TTL instead of overwriting)
- ais-relay.cjs: same percentage-drop guard for WB dual writer

Codex-reviewed plan (5 rounds, approved).
2026-03-17 16:12:05 +04:00
Elie Habib
503598fe7b fix(forecast): address P2/P3 code review findings from PR #1761 (#1765)
- Strip enrichmentMeta from bootstrap.js forecasts payload (seed-internal, not for clients)
- Rename quietDomainBonus -> priorityDomainBonus (it applies to priority domains, not quiet ones)
- Extract cyber score formula magic numbers into named constants (CYBER_SCORE_TYPE_MULTIPLIER, etc.)
- Pre-compute analysisPriority in rankForecastsForAnalysis to avoid double-call per comparison
- Log when filterPublishedForecasts weak-fallback gate suppresses forecasts
- Log how many fallback narratives populateFallbackNarratives applies
- Add // penalties comment header in computeAnalysisPriority
2026-03-17 14:32:57 +04:00
Elie Habib
3702463321 Add thermal escalation seeded service (#1747)
* feat(thermal): add thermal escalation seeded service

Cherry-picked from codex/thermal-escalation-phase1 and retargeted
to main. Includes thermal escalation seed script, RPC handler,
proto definitions, bootstrap/health/seed-health wiring, gateway
cache tier, client service, and tests.

* fix(thermal): wire data-loader, fix typing, recalculate summary

Wire fetchThermalEscalations into data-loader.ts with panel forwarding,
freshness tracking, and variant gating. Fix seed-health intervalMin from
90 to 180 to match 3h TTL. Replace 8 as-any casts with typed interface.
Recalculate summary counts after maxItems slice.

* fix(thermal): enforce maxItems on hydrated data + fix bootstrap keys

Codex P2: hydration branch now slices clusters to maxItems before
mapping, matching the RPC fallback behavior.

Also add thermalEscalation to bootstrap.js BOOTSTRAP_CACHE_KEYS and
SLOW_KEYS (was lost during conflict resolution).

* fix(thermal): recalculate summary on sliced hydrated clusters

When maxItems truncates the cluster array from bootstrap hydration,
the summary was still using the original full-set counts. Now
recalculates clusterCount, elevatedCount, spikeCount, etc. on the
sliced array, matching the handler's behavior.
2026-03-17 14:24:26 +04:00
Elie Habib
e2f0811330 fix(forecast): tighten quality and enrichment balance (#1761) 2026-03-17 14:03:13 +04:00
Elie Habib
e486f077c7 fix(sanctions): add progress logging to seed (fetch size, entry count, new count) (#1758) 2026-03-17 13:26:22 +04:00
Elie Habib
43b6b04415 chore(forecast): log llm stage routing (#1754) 2026-03-17 13:23:10 +04:00
Elie Habib
80134e3306 fix(sanctions): add fast-xml-parser to Railway scripts deps (#1755)
seed-sanctions-pressure.mjs imports fast-xml-parser to parse OFAC SDN
XML feeds, but the package was never added to scripts/package.json.
Railway deploys crash with ERR_MODULE_NOT_FOUND on startup.
2026-03-17 13:01:12 +04:00
Elie Habib
1a59114d05 refactor(sanctions): simplify handler to Redis-read-only, fix seed OOM risk (#1753)
* refactor(sanctions): simplify handler to Redis-read-only, fix seed OOM risk

Handler (424→56 lines):
- Remove live OFAC fetch fallback from Vercel Edge handler: XMLParser,
  OFAC_SOURCES, fetchSource, collectPressure, cachedFetchJson fallback.
  Vercel reads Redis only; Railway makes all external API calls.
- On seed miss/empty, return emptyResponse() matching the radiation pattern.

Seed:
- Fetch SDN then Consolidated sequentially instead of Promise.all.
  Combined parallel parse peaks at ~150MB, tight against 512MB heap limit.

Tests:
- Add gold standard compliance assertions (no XMLParser, no OFAC_SOURCES).
- Add memory safety assertion (no Promise.all on OFAC sources).
- Replace handler XML-function tests (removed code) with Redis-read assertions.

* chore: exclude DMCA-TAKEDOWN-NOTICE.md from markdownlint
2026-03-17 12:20:10 +04:00
Elie Habib
1c0e292260 feat(llm): support forecast model overrides (#1751) 2026-03-17 12:17:25 +04:00
Elie Habib
babb9b6836 feat(sanctions): add OFAC sanctions pressure intelligence (#1739)
* feat(sanctions): add OFAC sanctions pressure intelligence

* fix(sanctions): strip _state from API response, fix code/name alignment, cap seed limit

- trimResponse now destructures _state before spreading to prevent seed
  internals leaking to API clients during the atomicPublish→afterPublish window
- buildLocationMap and extractPartyCountries now sort (code, name) as aligned
  pairs instead of calling uniqueSorted independently on each array; fixes
  code↔name mispairing for OFAC-specific codes like XC (Crimea) where
  alphabetic order of codes and names diverges
- DEFAULT_RECENT_LIMIT reduced from 120 to 60 to match MAX_ITEMS_LIMIT so
  seeded entries beyond the handler cap are not written unnecessarily
- Add tests/sanctions-pressure.test.mjs covering all three invariants

* fix(sanctions): register sanctions:pressure:v1 in health.js BOOTSTRAP_KEYS and SEED_META

Adds sanctionsPressure to health.js so the health endpoint monitors the
seeded key for emptiness (CRIT) and freshness via seed-meta:sanctions:pressure
(maxStaleMin: 720 matches 12h seed TTL). Without this, health was blind to
stale or missing sanctions data.
2026-03-17 11:52:32 +04:00
Elie Habib
4353c20637 feat(widgets): AI widget builder with live WorldMonitor data (#1732) 2026-03-17 09:23:04 +04:00
Elie Habib
3897f8263d feat: add Radiation Watch with seeded anomaly intelligence, map layers, and country exposure (#1735) 2026-03-17 09:18:06 +04:00
Elie Habib
ce0f529204 feat(forecast): add trace quality summary (#1746)
* feat(forecast): add trace quality summary

* fix(forecast): split traced and full-run quality metrics
2026-03-17 08:22:20 +04:00
Elie Habib
47eb306195 fix(forecast): improve ranking and enrichment coverage (#1745) 2026-03-17 07:54:01 +04:00
Elie Habib
9f1fc9c236 fix(military): prefer direct opensky over proxy (#1742) 2026-03-17 07:26:14 +04:00
Elie Habib
9e58365587 fix(seeds): extend seed-meta TTL alongside data keys on fetch failure (#1724)
When upstream APIs fail and seeds extend existing data key TTLs, the
seed-meta key was left untouched. Health checks use seed-meta fetchedAt
to determine staleness, so preserved data still triggered STALE_SEED
warnings even though the data was valid.

Now all TTL extension paths include the corresponding seed-meta key:
- _seed-utils.mjs runSeed() (fetch failure + validation skip)
- fetch-gpsjam.mjs (Wingbits 500 fallback)
- seed-airport-delays.mjs (FAA fetch failure)
- seed-military-flights.mjs (OpenSky fetch failure)
- seed-service-statuses.mjs (RPC fetch failure)
2026-03-17 06:35:12 +04:00
Elie Habib
10f619326e fix(military): use opensky oauth in seed (#1738)
* fix(military): use opensky oauth in seed

* feat(military): log opensky source usage
2026-03-17 06:31:46 +04:00
Elie Habib
22dafc9774 feat(military): add enrichment audit waterfall (#1730)
* fix(military): improve source-backed flight inference

* fix(military): tighten operator metadata matching

* fix(military): tighten source hint inference

* feat(military): add enrichment audit waterfall

* feat(military): capture live source shape gaps
2026-03-17 01:41:54 +04:00
Elie Habib
5add5b3558 fix(military): improve source-backed flight inference (#1716)
* fix(military): improve source-backed flight inference

* fix(military): tighten operator metadata matching

* fix(military): tighten source hint inference
2026-03-16 19:08:51 +04:00
Elie Habib
179b1d7047 fix(ucdp): page error logging, page-0 fallback, TTL extension (#1717)
* fix(ucdp): add page error logging, page-0 fallback, and TTL extension on empty

Three resilience improvements for UCDP seed loop:

1. Log actual error messages on page fetch failures instead of silently
   swallowing them. Enables diagnosing API outages vs rate limits.

2. Fall back to page 0 data when all newest-page fetches fail. Page 0
   is already fetched during version discovery, so this is free. Provides
   partial (older) data instead of writing 0 events.

3. When 0 events remain after processing, extend existing Redis key TTL
   instead of overwriting with empty payload. Preserves stale-but-valid
   data for the next cycle rather than causing EMPTY_DATA CRIT in health.

* fix(ucdp): remove page-0 fallback, stop seed-meta on failed fetches

P1 fixes from review:
- Remove page-0 fallback that overwrote last known good cache with
  stale historical data. Extend existing key TTL instead.
- Stop writing fresh seed-meta timestamps when no new payload is
  written (both all-pages-failed and empty-after-filtering branches).
  Health checks should reflect actual data freshness, not failed attempts.

Add 6 targeted source-analysis tests verifying:
- Error logging on page failures
- No page-0 data injection
- TTL extension on failure branches
- seed-meta only written on successful publish
2026-03-16 17:25:15 +04:00
Elie Habib
fbb8f15943 fix(seeds): skip transient redis lock timeouts (#1714)
* fix(seeds): skip transient redis lock timeouts

* docs(seeds): clarify transient redis error matching

* test: expand transient redis error coverage

Add tests for ECONNRESET, DNS failure (EAI_AGAIN), ETIMEDOUT, and
negative cases (HTTP 403, payload size) to confirm isTransientRedisError
only matches network-level failures, not app-level Redis errors.
2026-03-16 11:57:52 +04:00
Elie Habib
be93a940a3 fix(military): tighten flight classification (#1713)
* fix(forecast): bundle military surge inputs

* fix(military): tighten flight classification
2026-03-16 09:32:13 +04:00
Elie Habib
467608c2d7 chore: clear baseline lint debt (173 warnings → 49) (#1712)
Mechanical fixes across 13 files:
- isNaN() → Number.isNaN() (all values already numeric from parseFloat/parseInt)
- let → const where never reassigned
- Math.pow() → ** operator
- Unnecessary continue in for loop
- Useless string escape in test description
- Missing parseInt radix parameter
- Remove unused private class member (write-only counter)
- Prefix unused function parameter with _

Config: suppress noImportantStyles (CSS !important is intentional) and
useLiteralKeys (bracket notation used for computed/dynamic keys) in
biome.json. Remaining 49 warnings are all noExcessiveCognitiveComplexity
(already configured as warn, safe to address incrementally).
2026-03-16 08:48:00 +04:00
Elie Habib
a4914607bb fix(forecast): bundle military surge inputs (#1706) 2026-03-16 08:40:14 +04:00
Elie Habib
63fe04d78f fix(seeds): extend existing cache TTL on validation failure (#1705)
When a seed fetches data but validation rejects it (e.g. FIRMS API
returns 0 fires due to timeout), extend the existing key's TTL
instead of letting it expire. Old data survives until the next
successful fetch. Applies to all seeds using runSeed().
2026-03-16 08:10:42 +04:00
Elie Habib
4636fadd33 fix(forecast): seed military surge signals (#1696)
* fix(forecast): filter low-signal panel forecasts

* fix(forecast): seed military surge signals
2026-03-16 01:08:25 +04:00
Elie Habib
94d4e7a99f fix(forecast): filter low-signal panel forecasts (#1687) 2026-03-15 23:01:32 +04:00
Elie Habib
14f2e60ca2 fix(forecast): log and honor full trace count (#1683) 2026-03-15 21:24:08 +04:00
Elie Habib
6ef6b38b7c fix(trade): align flows cache key with seed (#1677)
* fix(trade): align flows cache key with seed (US vs World, not China)

The seed writes trade:flows:v1:840:000:10 (US vs World) but the
data-loader requested trade:flows:v1:840:156:10 (US vs China),
causing perpetual cache misses and a hidden Flows tab.

* feat(seed): add bilateral trade flow pairs (US-China, US-Germany, etc.)

Seed now writes both reporter-vs-World AND key bilateral pairs
so switching between global and bilateral views hits warm cache.

* feat(seed): add World-China and World-US bilateral flow pairs

* fix(trade): revert flows to US-China (840/156), seed now covers this key

The bilateral seed entries now write trade:flows:v1:840:156:10,
so the original US-China request hits cache. Keeps the panel
showing bilateral data consistent with the tariffs partner.
2026-03-15 20:59:13 +04:00
Elie Habib
e07bc8ede5 fix(aviation): stop Vercel from calling AviationStack directly (#1674)
* fix(aviation): stop Vercel from calling AviationStack directly

- get-airport-ops-summary: read from relay seed cache (aviation:delays:intl:v3)
  instead of calling fetchAviationStackDelays() on every cache miss
- list-airport-flights + get-flight-status: proxy through Railway relay
  /aviationstack endpoint instead of calling AviationStack from Vercel edge
- Add /aviationstack proxy endpoint to ais-relay with 2min in-memory cache

Vercel should NEVER call external paid APIs directly. Railway relay is
the sole egress point for AviationStack (gold standard).

* fix(config): update aviationStack feature to require WS_RELAY_URL

Aviation handlers now proxy through Railway relay instead of calling
AviationStack directly. Update runtime-config to reflect the actual
dependency.
2026-03-15 20:18:48 +04:00
Elie Habib
8619cd16aa fix(forecast): store full traces and tighten matching (#1673)
* fix(forecast): restore missing domains and split cyber

* fix(forecast): store full traces and tighten matching
2026-03-15 19:51:09 +04:00
Elie Habib
3526ad559c fix(seed): bump BDI body size limit to 1MB, add gzip encoding (#1672)
HandyBulk page is ~662KB, exceeding the 500KB guard. Also adds
Accept-Encoding header to reduce transfer size.
2026-03-15 19:41:14 +04:00
Elie Habib
9196731bb9 fix(forecast): bump TTL from 3600s to 4800s to cover cron gap (#1668)
Hourly cron + 3600s TTL = data expires right as the next seed starts,
causing a ~30s EMPTY window. Bumped to 4800s (80min) so old data
persists while the new seed runs.
2026-03-15 19:17:44 +04:00
Elie Habib
72b6f9e832 feat(supply-chain): add SCFI, CCFI, and BDI freight indices (#1666)
* feat(supply-chain): add SCFI, CCFI, and BDI freight indices to shipping tab

Transform the Shipping Rates tab from 2 lagging monthly FRED indices into
a real-time freight cost dashboard with container and bulk shipping rates.

Seed script: add fetchSCFI/fetchCCFI (SSE JSON API) and fetchBDI (HandyBulk
HTML scrape) with inline history accumulation using source observation dates.
Handler: make cache-only (seed is sole aggregator, no FRED fallback on miss).
Panel: group indices into Container Rates, Bulk Shipping, Economic Indicators.
Tests: 26 functional tests with fixture data for parsers, history, and handler.

* fix(supply-chain): use raw Redis read and correct SCFI composite unit

- Handler: switch from cachedFetchJson (env-prefixed) to getCachedJson(key, true)
  so preview deployments read the unprefixed seed key correctly
- Seed: SCFI composite is a dimensionless index, not USD/TEU (route-level unit)
- Tests: update assertions to match both fixes
2026-03-15 19:14:11 +04:00
Elie Habib
4c11b46be3 feat(trade): add US Treasury customs revenue to Trade Policy panel (#1663)
* feat(trade): add US Treasury customs revenue to Trade Policy panel

US customs duties revenue spiked 4-5x under Trump tariffs (from
$7B/month to $27-31B/month) but the WTO tariff data only goes to
2024. Adds Treasury MTS data showing monthly customs revenue.

- Add GetCustomsRevenue RPC (proto, handler, cache tier)
- Add Treasury fetch to seed-supply-chain-trade.mjs (free API, no key)
- Add Revenue tab to TradePolicyPanel with FYTD YoY comparison
- Fix WTO gate: per-tab gating so Revenue works without WTO key
- Wire bootstrap hydration, health, seed-health tracking

* test(trade): add customs revenue feature tests

22 structural tests covering:
- Handler: raw key mode, empty-cache behavior, correct Redis key
- Seed: Treasury API URL, classification filter, timeout, row
  validation, amount conversion, sort order, seed-meta naming
- Panel: WTO gate fix (per-tab not panel-wide), revenue tab
  defaults when WTO key missing, dynamic FYTD comparison
- Client: no WTO feature gate, bootstrap hydration, type exports

* fix(trade): align FYTD comparison by fiscal month count

Prior FY comparison was filtering by calendar month, which compared
5 months of FY2026 (Oct-Feb) against only 2 months of FY2025
(Jan-Feb), inflating the YoY percentage. Now takes the first N
months of the prior FY matching the current FY month count.

* fix(trade): register treasury_revenue DataSourceId and localize revenue tab

- Add treasury_revenue to DataSourceId union type so freshness
  tracking actually works (was silently ignored)
- Register in data-freshness.ts source config + gap messages
- Add i18n keys: revenue tab label, empty state, unavailable banner
- Update infoTooltip to include Revenue tab description

* fix(trade): complete revenue tab localization

Use t() for all remaining hardcoded strings: footer source labels,
FYTD summary headline, prior-year comparison, and table column
headers. Wire the fytdLabel/vsPriorFy keys that were added but
not used.

* fix(test): update revenue source assertion for localized string
2026-03-15 19:04:23 +04:00
Elie Habib
4022d129fd fix(forecast): restore missing domains and split cyber (#1660) 2026-03-15 17:46:26 +04:00
Elie Habib
8d6bf37240 fix(predictions): prepend event title to short Kalshi outcome labels (#1659)
Kalshi multi-outcome events return market titles like "Before 2035",
"Rhode Island", "Johnny Depp" which are meaningless without the parent
event context. Now combines event title with market title when the
market title is short and doesn't contain a question mark.

Before: "Before 2035" (KALSHI)
After:  "Will AGI be achieved?: Before 2035" (KALSHI)
2026-03-15 17:44:31 +04:00
Elie Habib
c4fd49a284 fix(health): close corridorrisk health monitoring gap (#1658)
The corridorrisk raw key (2h TTL) expires between hourly seed cycles,
causing health to report EMPTY even though data flows correctly through
transit-summaries:v1.

- Increase CORRIDOR_RISK_TTL from 2h to 4h (3 retries before expiry)
- Add corridorrisk to ON_DEMAND_KEYS (WARN instead of CRIT when empty)
2026-03-15 17:14:26 +04:00
Elie Habib
bebf2918f2 debug(seeds): add R2 trace logging to seed-forecasts (#1655)
Adds [Trace] log lines before/after R2 export and [R2] config
resolution debug output to diagnose missing trace artifacts.
2026-03-15 16:58:24 +04:00
Elie Habib
d6f7df9746 fix(seeds): lazy-load @aws-sdk/client-s3 in R2 storage helper (#1654)
The top-level import crashes seed-forecasts on Railway when the
package isn't installed. Dynamic import defers the load to when
S3 mode is actually used, allowing the seed to run without the
SDK when R2 is not configured.
2026-03-15 16:24:38 +04:00
Elie Habib
39931456a1 feat(forecast): add structured scenario pipeline and trace export (#1646)
* feat(forecast): add AI Forecasts prediction module (Pro-tier)

MiroFish-inspired prediction engine that generates structured forecasts
across 6 domains (conflict, market, supply chain, political, military,
infrastructure) using existing WorldMonitor data streams.

- Proto definitions for ForecastService with GetForecasts RPC
- Dedicated seed script (seed-forecasts.mjs) with 6 domain detectors,
  cross-domain cascade resolver, prediction market calibration, and
  trend detection via prior snapshot comparison
- Premium-gated RPC handler (PREMIUM_RPC_PATHS enforcement)
- Lazy-loaded ForecastPanel with domain filters, probability bars,
  trend arrows, signal evidence, and cascade links
- Health monitoring integration (seed-meta freshness tracking)
- Refresh scheduler with API key guard

* test(forecast): add 47 unit tests for forecast detectors and utilities

Covers forecastId, normalize, resolveCascades, calibrateWithMarkets,
computeTrends, and smoke tests for all 6 domain detectors. Exports
testable functions from seed script with direct-run guard.

* fix(forecast): domain mismatch 'infra' vs 'infrastructure', add panel category

- Seed script used 'infra' but ForecastPanel filtered on 'infrastructure',
  causing Infra tab to show zero results
- Added 'forecast' to intelligence category in PANEL_CATEGORY_MAP

* fix(forecast): move CSS to one-time injection, improve type safety

- P2: Move style block from setContent to one-time document.head injection
  to prevent CSS accumulation on repeated renders
- P3: Replace +toFixed(3) with Math.round for readability in seed script
- P3: Use Forecast type instead of any[] in RPC handler filter

* fix(forecast): handle sebuf proto data shapes from Redis

Detectors now normalize CII scores from server-side proto format
(combinedScore, TREND_DIRECTION_RISING, region) to uniform shape.
Outage severity handles proto enum format (SEVERITY_LEVEL_HIGH).
Added confidence floor of 0.3 for single-source predictions.

Verified against live Redis: 2 predictions generated (Iran infra
shutdown, IL political instability).

* feat(forecast): unlock AI Forecasts on web, lock desktop only (trial)

- Remove forecast RPC from PREMIUM_RPC_PATHS (web access is free)
- Panel locked on desktop only (same as oref-sirens/telegram-intel)
- Remove API key guards from data-loader and refresh scheduler
- Web users get full access during trial period

* chore: regenerate proto types with make generate

Re-ran make generate after rebasing on main. Plugin v0.7.0 dropped
@ts-nocheck from output, added it back to all 50 generated files.
Fixed 4 type errors from proto codegen changes:
- MarketSource enum -> string union type
- TemporalAnomalyProto -> TemporalAnomaly rename
- webcam lastUpdated number -> string

* chore: add proto freshness check to pre-push hook

Runs make generate before push and compares checksums of generated files.
If proto types are stale, blocks push with instructions to regenerate.
Skips gracefully if buf CLI is not installed.

* fix(forecast): use chokepoints v4 key, include ciiContribution in unrest

- P1: Switch chokepoints input from stale v2 to active v4 Redis key,
  matching bootstrap.js and cache-keys.ts
- P2: Add ciiContribution to unrest component fallback chain in
  normalizeCiiEntry so political detector reads the correct sebuf field

* feat(forecast): Phase 2 LLM scenario enrichment + confidence model

MiroFish-inspired enhancements:
- LLM scenario narratives via Groq/OpenRouter (narrative-only, no numeric
  adjustment). Evidence-grounded prompts with mandatory signal citation
  and few-shot examples from MiroFish's SECTION_SYSTEM_PROMPT_TEMPLATE.
- Top-4 predictions batched into single LLM call for cost efficiency.
- News context from newsInsights attached to all predictions for LLM
  prompt grounding (NOT in signals, cannot affect confidence).
- Deterministic confidence model: source diversity via SIGNAL_TO_SOURCE
  mapping (deduplicates cii+cii_delta, theater+indicators) + calibration
  agreement from prediction market drift. Floor 0.2, ceiling 1.0.
- Output validation: rejects scenarios without signal references.
- Truncated JSON repair for small model output.
- Structured JSON logging for LLM calls.
- Redis cache for LLM scenarios (1h TTL).
- 23 new tests (70 total), all passing.
- Live-tested: OpenRouter gemini-2.5-flash produces evidence-grounded
  scenario narratives from real WorldMonitor data.

* feat(forecast): Phase 3 multi-perspective scenarios, projections, data-driven cascades

MiroFish-inspired enhancements:
- Multi-perspective LLM analysis: top-2 predictions get strategic,
  regional, and contrarian viewpoints via combined LLM call
- Probability projections: domain-specific decay curves (h24/d7/d30)
  anchored to timeHorizon so probability equals projections[timeHorizon]
- Data-driven cascade rules: moved from hardcoded array to JSON config
  (scripts/data/cascade-rules.json) with schema validation, named
  predicate evaluators, unknown key rejection, and fallback to defaults
- 4 new cascade paths: infrastructure->supply_chain, infrastructure->market
  (both requiresSeverity:total), conflict->political, political->market
- Proto: added Perspectives and Projections messages to Forecast
- ForecastPanel: renders projections row and conditional perspectives toggle
- 89 tests (19 new), all passing
- Live-tested: OpenRouter produces perspectives from real data

* feat(forecast): Phase 4 data utilization + entity graph

Fixes data gaps that prevented 4 of 6 detectors from firing:
- Input normalizers: chokepoint v4 shape + GPS hexes-to-zones mapping
- Chokepoint warm-ping (production-only, requires WM_API_BASE_URL)
- Lowered CII conflict threshold from 70 to 60, gated on level=high|critical

4 new standalone detectors:
- UCDP conflict zones (10+ events per country)
- Cyber threat concentration (5+ threats per country)
- GPS jamming in maritime shipping zones (5 regions)
- Prediction markets as signals (60-90% probability markets)

Entity-relationship graph (file-based, 38 nodes):
- Countries, theaters, commodities, chokepoints, alliances
- Alias table resolves both ISO codes and display names
- Graph cascade discovery links predictions across entities

Result: 51 predictions (up from 1-2), spanning conflict, infrastructure,
and supply chain domains. 112 tests, all passing.

* fix(forecast): redis cache format, signal source mapping, type safety

Fresh-eyes audit fixes:
- BUG: redisSet used wrong Upstash API format (POST body with {value,ex}
  instead of command array ['SET',key,value,'EX',ttl]). LLM cache writes
  were silently failing, causing fresh LLM calls every run.
- BUG: prediction_market signal type missing from SIGNAL_TO_SOURCE,
  inflating confidence for market-derived predictions.
- CLEANUP: Remove unnecessary (f as any) casts in ForecastPanel since
  generated Forecast type already has projections/perspectives fields.
- CLEANUP: Bump health maxStaleMin from 60 to 90 to avoid false STALE
  alerts when LLM calls add latency to seed runs.

* feat(forecast): headline-entity matching with news corroboration signals

Uses entity graph aliases to match headlines to predictions by
country/theater (excludes commodity/infrastructure nodes to prevent
false positives). Predictions with matching headlines get a
news_corroboration signal visible in the panel.

Also fixes buildUserPrompt to merge unique headlines from ALL
predictions in the LLM batch (was only reading preds[0].newsContext).

Live-tested: 13 of 51 predictions now have corroborating headlines
(Iran, Israel, Syria, Ukraine, etc). 116 tests, all passing.

* feat(forecast): add country-codes.json for headline-entity matching

56 countries with ISO codes, full names, and scoring keywords (extracted
from src/config/countries.ts + UCDP-relevant additions). Used by
attachNewsContext for richer headline matching via getSearchTermsForRegion
which combines country-codes + entity graph + keyword aliases.

14/57 predictions now have news corroboration (limited by headline
coverage, not matching quality: only 8 headlines currently available).

* feat(forecast): read 300 headlines from news digest instead of 8

Read news:digest:v1:full:en (300 headlines across 16 categories) instead
of just news:insights:v1 topStories (8 headlines). Fallback to topStories
if digest is unavailable.

Result: news corroboration jumped from 25% to 64% (38/59 predictions).

* fix(forecast): handle parenthetical country names in headline matching

Strip suffixes like '(Zaire)', '(Burma)', '(Soviet Union)' from UCDP
region names before matching against country-codes.json. Also use
includes() for reverse name lookup to catch partial matches.

Corroboration: 64% -> 69% (41/59). Remaining 18 unmatched are countries
with no current English-language news coverage.

* fix(forecast): cache validated LLM output, add digest test, log cache errors

Fresh-eyes audit fixes:
- Combined LLM cache now stores only validated items (was caching raw
  unvalidated output, serving potentially invalid scenarios on cache hit)
- redisSet logs warnings on failure (was silently swallowing all errors)
- Added digest-based test for attachNewsContext (primary path was untested)
- Fixed test arity: attachNewsContext(preds, news, digest) with 3 params

* fix(forecast): remove dead confidenceFromSources, reduce warm-ping timeout

- P2: Remove confidenceFromSources (dead code, computeConfidence overwrites
  all initial confidence values). Inline the formula in original detectors.
- P3: Reduce warm-ping timeout from 30s to 15s (non-critical step)
- P3: Add trial status comment on forecast panel config

* fix(forecast): resolve ISO codes to country names, fix market detector, safe pre-push

P1 fixes from code review:
- CII ISO codes (IL, IR) now resolved to full country names (Israel, Iran)
  via country-codes.json. Prevents substring false positives (IL matching
  Chile) in event correlation. Uses word-boundary regex for matching.
- Market detector CII-to-theater mapping now uses entity graph traversal
  instead of broken theater-name substring matching. Iran correctly maps
  to Middle East theater via graph links.
- Pre-push hook no longer runs destructive git checkout on proto freshness
  failure. Reports mismatch and exits without modifying worktree.

* feat(forecast): add structured scenario pipeline and trace export

* fix(forecast): hydrate bootstrap and trim generated drift

* fix(forecast): keep required supply-chain contract updates

* fix(ci): add forecasts to cache-keys registry and regenerate proto

Add forecasts entry to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in
cache-keys.ts to match api/bootstrap.js. Regenerate SupplyChain proto
to fix duplicate TransitDayCount and add riskSummary/riskReportAction.
2026-03-15 15:57:22 +04:00
Elie Habib
222dd99f75 fix(data): restore bootstrap and cache test coverage (#1649)
* fix(data): restore bootstrap and cache test coverage

* fix: resolve linting and test failures

- Remove dead writeSeedMeta/estimateRecordCount functions from redis.ts
  (intentionally removed from cachedFetchJson; seed-meta now written
  only by explicit seed flows, not generic cache reads)
- Fix globe dayNight test to match actual code (forces dayNight: false
  + hideLayerToggle, not catalog-based exclusion)
- Fix country-geometry test mock URL from CDN to /data/countries.geojson
  (source changed to use local bundled file)

* fix(lint): remove duplicate llm-health key in redis-caching test

Duplicate object key '../../../_shared/llm-health' caused the stub
to be overwritten by the real module. Removed the second entry so
the test correctly uses the stub.
2026-03-15 15:42:27 +04:00
Elie Habib
4fea7ce102 fix(seeds): allow conflict-intel seed to succeed without ACLED keys (#1651)
Validation now accepts empty ACLED events array when humanitarian or
pizzint data was fetched. Previously the seed wrote extra keys
(humanitarian, pizzint) but skipped the canonical key because
validateFn required non-empty events.
2026-03-15 15:37:40 +04:00
Elie Habib
4130572445 fix(usni): move USNI fleet seed back to AIS relay (fixed IP for Froxy proxy) (#1648)
The standalone seed-usni-fleet.mjs cannot reach USNI because:
1. USNI Cloudflare blocks Node.js TLS fingerprint (JA3)
2. curl is not installed on Railway cron containers
3. Froxy residential proxy is IP-whitelisted to the relay fixed IP

Move the USNI seed loop back into ais-relay.cjs where it has access to
curl + the whitelisted proxy. Uses orefCurlFetch for the fetch, same
pattern as the OREF alerts loop. Writes to the same Redis keys
(usni-fleet:sebuf:v1, stale:v1, seed-meta:military:usni-fleet).

6h seed interval, 7h TTL, 7d stale TTL (unchanged from standalone).
2026-03-15 14:27:19 +04:00
Steven J. Miklovic
6e32a346c3 Add Greek news channels & feed (#1602)
* Add Greek news channels

* Add ERT and SKAI hlsUrl to LiveNewsPanel.ts

---------

Co-authored-by: Elie Habib <elie.habib@gmail.com>
2026-03-15 13:40:20 +04:00