mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
58e42aadf951d8ea35403cbb60a6cb1792713d61
18 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sebastien Melki
|
58e42aadf9 |
chore(api): enforce sebuf contract + migrate drifting endpoints (#3207) (#3242)
* chore(api): enforce sebuf contract via exceptions manifest (#3207) Adds api/api-route-exceptions.json as the single source of truth for non-proto /api/ endpoints, with scripts/enforce-sebuf-api-contract.mjs gating every PR via npm run lint:api-contract. Fixes the root-only blind spot in the prior allowlist (tests/edge-functions.test.mjs), which only scanned top-level *.js files and missed nested paths and .ts endpoints — the gap that let api/supply-chain/v1/country-products.ts and friends drift under proto domain URL prefixes unchallenged. Checks both directions: every api/<domain>/v<N>/[rpc].ts must pair with a generated service_server.ts (so a deleted proto fails CI), and every generated service must have an HTTP gateway (no orphaned generated code). Manifest entries require category + reason + owner, with removal_issue mandatory for temporary categories (deferred, migration-pending) and forbidden for permanent ones. .github/CODEOWNERS pins the manifest to @SebastienMelki so new exceptions don't slip through review. The manifest only shrinks: migration-pending entries (19 today) will be removed as subsequent commits in this PR land each migration. * refactor(maritime): migrate /api/ais-snapshot → maritime/v1.GetVesselSnapshot (#3207) The proto VesselSnapshot was carrying density + disruptions but the frontend also needed sequence, relay status, and candidate_reports to drive the position-callback system. Those only lived on the raw relay passthrough, so the client had to keep hitting /api/ais-snapshot whenever callbacks were registered and fall back to the proto RPC only when the relay URL was gone. This commit pushes all three missing fields through the proto contract and collapses the dual-fetch-path into one proto client call. Proto changes (proto/worldmonitor/maritime/v1/): - VesselSnapshot gains sequence, status, candidate_reports. - GetVesselSnapshotRequest gains include_candidates (query: include_candidates). Handler (server/worldmonitor/maritime/v1/get-vessel-snapshot.ts): - Forwards include_candidates to ?candidates=... on the relay. - Separate 5-min in-memory caches for the candidates=on and candidates=off variants; they have very different payload sizes and should not share a slot. - Per-request in-flight dedup preserved per-variant. Frontend (src/services/maritime/index.ts): - fetchSnapshotPayload now calls MaritimeServiceClient.getVesselSnapshot directly with includeCandidates threaded through. The raw-relay path, SNAPSHOT_PROXY_URL, DIRECT_RAILWAY_SNAPSHOT_URL and LOCAL_SNAPSHOT_FALLBACK are gone — production already routed via Vercel, the "direct" branch only ever fired on localhost, and the proto gateway covers both. - New toLegacyCandidateReport helper mirrors toDensityZone/toDisruptionEvent. api/ais-snapshot.js deleted; manifest entry removed. Only reduced the codegen scope to worldmonitor.maritime.v1 (buf generate --path) — regenerating the full tree drops // @ts-nocheck from every client/server file and surfaces pre-existing type errors across 30+ unrelated services, which is not in scope for this PR. Shape-diff vs legacy payload: - disruptions / density: proto carries the same fields, just with the GeoCoordinates wrapper and enum strings (remapped client-side via existing toDisruptionEvent / toDensityZone helpers). - sequence, status.{connected,vessels,messages}: now populated from the proto response — was hardcoded to 0/false in the prior proto fallback. - candidateReports: same shape; optional numeric fields come through as 0 instead of undefined, which the legacy consumer already handled. * refactor(sanctions): migrate /api/sanctions-entity-search → LookupSanctionEntity (#3207) The proto docstring already claimed "OFAC + OpenSanctions" coverage but the handler only fuzzy-matched a local OFAC Redis index — narrower than the legacy /api/sanctions-entity-search, which proxied OpenSanctions live (the source advertised in docs/api-proxies.mdx). Deleting the legacy without expanding the handler would have been a silent coverage regression for external consumers. Handler changes (server/worldmonitor/sanctions/v1/lookup-entity.ts): - Primary path: live search against api.opensanctions.org/search/default with an 8s timeout and the same User-Agent the legacy edge fn used. - Fallback path: the existing OFAC local fuzzy match, kept intact for when OpenSanctions is unreachable / rate-limiting. - Response source field flips between 'opensanctions' (happy path) and 'ofac' (fallback) so clients can tell which index answered. - Query validation tightened: rejects q > 200 chars (matches legacy cap). Rate limiting: - Added /api/sanctions/v1/lookup-entity to ENDPOINT_RATE_POLICIES at 30/min per IP — matches the legacy createIpRateLimiter budget. The gateway already enforces per-endpoint policies via checkEndpointRateLimit. Docs: - docs/api-proxies.mdx — dropped the /api/sanctions-entity-search row (plus the orphaned /api/ais-snapshot row left over from the previous commit in this PR). - docs/panels/sanctions-pressure.mdx — points at the new RPC URL and describes the OpenSanctions-primary / OFAC-fallback semantics. api/sanctions-entity-search.js deleted; manifest entry removed. * refactor(military): migrate /api/military-flights → ListMilitaryFlights (#3207) Legacy /api/military-flights read a pre-baked Redis blob written by the seed-military-flights cron and returned flights in a flat app-friendly shape (lat/lon, lowercase enums, lastSeenMs). The proto RPC takes a bbox, fetches OpenSky live, classifies server-side, and returns nested GeoCoordinates + MILITARY_*_TYPE_* enum strings + lastSeenAt — same data, different contract. fetchFromRedis in src/services/military-flights.ts was doing nothing sebuf-aware. Renamed it to fetchViaProto and rewrote to: - Instantiate MilitaryServiceClient against getRpcBaseUrl(). - Iterate MILITARY_QUERY_REGIONS (PACIFIC + WESTERN) in parallel — same regions the desktop OpenSky path and the seed cron already use, so dashboard coverage tracks the analytic pipeline. - Dedup by hexCode across regions. - Map proto → app shape via new mapProtoFlight helper plus three reverse enum maps (AIRCRAFT_TYPE_REVERSE, OPERATOR_REVERSE, CONFIDENCE_REVERSE). The seed cron (scripts/seed-military-flights.mjs) stays put: it feeds regional-snapshot mobility, cross-source signals, correlation, and the health freshness check (api/health.js: 'military:flights:v1'). None of those read the legacy HTTP endpoint; they read the Redis key directly. The proto handler uses its own per-bbox cache keys under the same prefix, so dashboard traffic no longer races the seed cron's blob — the two paths diverge by a small refresh lag, which is acceptable. Docs: dropped the /api/military-flights row from docs/api-proxies.mdx. api/military-flights.js deleted; manifest entry removed. Shape-diff vs legacy: - f.location.{latitude,longitude} → f.lat, f.lon - f.aircraftType: MILITARY_AIRCRAFT_TYPE_TANKER → 'tanker' via reverse map - f.operator: MILITARY_OPERATOR_USAF → 'usaf' via reverse map - f.confidence: MILITARY_CONFIDENCE_LOW → 'low' via reverse map - f.lastSeenAt (number) → f.lastSeen (Date) - f.enrichment → f.enriched (with field renames) - Extra fields registration / aircraftModel / origin / destination / firstSeenAt now flow through where proto populates them. * fix(supply-chain): thread includeCandidates through chokepoint status (#3207) Caught by tsconfig.api.json typecheck in the pre-push hook (not covered by the plain tsc --noEmit run that ran before I pushed the ais-snapshot commit). The chokepoint status handler calls getVesselSnapshot internally with a static no-auth request — now required to include the new includeCandidates bool from the proto extension. Passing false: server-internal callers don't need per-vessel reports. * test(maritime): update getVesselSnapshot cache assertions (#3207) The ais-snapshot migration replaced the single cachedSnapshot/cacheTimestamp pair with a per-variant cache so candidates-on and candidates-off payloads don't evict each other. Pre-push hook surfaced that tests/server-handlers still asserted the old variable names. Rewriting the assertions to match the new shape while preserving the invariants they actually guard: - Freshness check against slot TTL. - Cache read before relay call. - Per-slot in-flight dedup. - Stale-serve on relay failure (result ?? slot.snapshot). * chore(proto): restore // @ts-nocheck on regenerated maritime files (#3207) I ran 'buf generate --path worldmonitor/maritime/v1' to scope the proto regen to the one service I was changing (to avoid the toolchain drift that drops @ts-nocheck from 60+ unrelated files — separate issue). But the repo convention is the 'make generate' target, which runs buf and then sed-prepends '// @ts-nocheck' to every generated .ts file. My scoped command skipped the sed step. The proto-check CI enforces the sed output, so the two maritime files need the directive restored. * refactor(enrichment): decomm /api/enrichment/{company,signals} legacy edge fns (#3207) Both endpoints were already ported to IntelligenceService: - getCompanyEnrichment (/api/intelligence/v1/get-company-enrichment) - listCompanySignals (/api/intelligence/v1/list-company-signals) No frontend callers of the legacy /api/enrichment/* paths exist. Removes: - api/enrichment/company.js, signals.js, _domain.js - api-route-exceptions.json migration-pending entries (58 remain) - docs/api-proxies.mdx rows for /api/enrichment/{company,signals} - docs/architecture.mdx reference updated to the IntelligenceService RPCs Verified: typecheck, typecheck:api, lint:api-contract (89 files / 58 entries), lint:boundaries, tests/edge-functions.test.mjs (136 pass), tests/enrichment-caching.test.mjs (14 pass — still guards the intelligence/v1 handlers), make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(leads): migrate /api/{contact,register-interest} → LeadsService (#3207) New leads/v1 sebuf service with two POST RPCs: - SubmitContact → /api/leads/v1/submit-contact - RegisterInterest → /api/leads/v1/register-interest Handler logic ported 1:1 from api/contact.js + api/register-interest.js: - Turnstile verification (desktop sources bypass, preserved) - Honeypot (website field) silently accepts without upstream calls - Free-email-domain gate on SubmitContact (422 ApiError) - validateEmail (disposable/offensive/typo-TLD/MX) on RegisterInterest - Convex writes via ConvexHttpClient (contactMessages:submit, registerInterest:register) - Resend notification + confirmation emails (HTML templates unchanged) Shared helpers moved to server/_shared/: - turnstile.ts (getClientIp + verifyTurnstile) - email-validation.ts (disposable/offensive/MX checks) Rate limits preserved via ENDPOINT_RATE_POLICIES: - submit-contact: 3/hour per IP (was in-memory 3/hr) - register-interest: 5/hour per IP (was in-memory 5/hr; desktop sources previously capped at 2/hr via shared in-memory map — now 5/hr like everyone else, accepting the small regression in exchange for Upstash-backed global limiting) Callers updated: - pro-test/src/App.tsx contact form → new submit-contact path - src-tauri/sidecar/local-api-server.mjs cloud-fallback rewrites /api/register-interest → /api/leads/v1/register-interest when proxying; keeps local path for older desktop builds - src/services/runtime.ts isKeyFreeApiTarget allows both old and new paths through the WORLDMONITOR_API_KEY-optional gate Tests: - tests/contact-handler.test.mjs rewritten to call submitContact handler directly; asserts on ValidationError / ApiError - tests/email-validation.test.mjs + tests/turnstile.test.mjs point at the new server/_shared/ modules Deleted: api/contact.js, api/register-interest.js, api/_ip-rate-limit.js, api/_turnstile.js, api/_email-validation.js, api/_turnstile.test.mjs. Manifest entries removed (58 → 56). Docs updated (api-platform, api-commerce, usage-rate-limits). Verified: npm run typecheck + typecheck:api + lint:api-contract (88 files / 56 entries) + lint:boundaries pass; full test:data (5852 tests) passes; make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(pro-test): rebuild bundle for leads/v1 contact form (#3207) Updates the enterprise contact form to POST to /api/leads/v1/submit-contact (old path /api/contact removed in the previous commit). Bundle is rebuilt from pro-test/src/App.tsx source change in |
||
|
Elie Habib
|
96fca1dc2b |
fix(supply-chain): popup-keyed history re-query + dataAvailable flag (#3187)
* fix(supply-chain): popup-keyed history re-query + dataAvailable flag for partial coverage Two P1 findings on #3185 post-merge review: 1. MapPopup cross-chokepoint history contamination Popup's async history resolve re-queried [data-transit-chart] without a cpId key. User opens popup A → fetch starts for cpA; user opens popup B before it resolves → cpA's history mounts into cpB's chart container. Fix: add data-transit-chart-id keyed by cpId; re-query by it on resolve. Mirrors SupplyChainPanel's existing data-chart-cp-id pattern. 2. Partial portwatch coverage still looked healthy Previous fix emits all 13 canonical summaries (zero-state fill for missing IDs) and records pwCovered in seed-meta, but: - get-chokepoint-status still zero-filled missing chokepoints and cached the response as healthy — panel rendered silent empty rows. - api/health.js only degrades on recordCount=0, so 10/13 partial read as OK despite the UI hiding entire chokepoints. Fix: - proto: TransitSummary.data_available (field 12). Writer tags with Boolean(cpData). Status RPC passes through; defaults true for pre-fix payloads (absence = covered). - Status RPC writes seed-meta recordCount as covered count (not shape size), and flips response-level upstreamUnavailable on partial. - api/health.js: new minRecordCount field on SEED_META entries + new COVERAGE_PARTIAL status (warn rollup). chokepoints entry declares minRecordCount: 13. recordCount < 13 → COVERAGE_PARTIAL. - Client (panel + popup): skip stats/chart rendering when !dataAvailable; show "Transit data unavailable (upstream partial)" microcopy so users understand the gap. 5759/5759 data tests pass. Typecheck + typecheck:api clean. * fix(supply-chain): guarantee Simulate Closure button exits Computing state User reports "Simulate Closure does nothing beyond write Computing…" — the button sticks at Computing forever. Two causes: 1. Scenario worker appears down (0 scenario-result:* keys in Redis in the last 24h of 24h-TTL). Railway-side — separate intervention needed to redeploy scripts/scenario-worker.mjs. 2. Client leaked the "Computing…" state on multiple exit paths: - signal.aborted early-return inside the poll loop never reset the button. Second click fired abort on first → first returned without resetting → button stayed "Computing…" until next render. - !this.content.isConnected early-return also skipped reset (less user-visible but same class of bug). - catch block swallowed AbortError without resetting. - POST /run had no hard timeout — a hanging edge function left the button in Computing indefinitely. Fix: - resetButton(text) helper touches the btn only if still connected; applied in every exit path (abort, timeout, post-success, catch). - AbortSignal.any([caller, AbortSignal.timeout(20_000)]) on POST /run. - console.error on failure so Simulate Closure errors surface in ops. - Error message includes "scenario worker may be down" on loop timeout so operators see the right suspect. Backend observations (for follow-up): - Hormuz backend is healthy (/api/health chokepoints OK, 13 records, 1 min old; live RPC has hormuz_strait.riskLevel=critical, wow=-22, flowEstimate present; GetChokepointHistory returns 174 entries). User-reported "Hormuz empty" is likely browser/CDN stale cache from before PR #3185; hard refresh should resolve. - scenario-worker.mjs has zero result keys in 24h. Railway service needs verification/redeployment. * fix(scenario): wrong Upstash RPUSH format silently broke every Simulate Closure Railway scenario-worker log shows every job failing field validation since at least 03:06Z today: [scenario-worker] Job failed field validation, discarding: ["{\"jobId\":\"scenario:1776535792087:cynxx5v4\",... The leading [" in the payload is the smoking gun. api/scenario/v1/run.ts was POSTing to /rpush/{key} with body `[payload]`, expecting Upstash to unpack the array and push one string value. Upstash does NOT parse that form — it stored the literal `["{...}"]` string as a single list value. Worker BLMOVEs the literal string → JSON.parse → array → destructure `{jobId, scenarioId, iso2}` on an array returns undefined for all three → every job discarded without writing a result. Client poll returns `pending` for the full 60s timeout, then (on the prior client code path) leaked the stuck "Computing…" button state indefinitely. Fix: use the standard Upstash REST command format — POST to the base URL with body `["RPUSH", key, value]`. Matches scripts/ais-relay.cjs upstashLpush. After this, the scenario-queue:pending list stores the raw payload string, BLMOVE returns the payload, JSON.parse gives the object, validation passes, computeScenario runs, result key gets written, client poll sees `done`. Zero result keys existed in prod Redis in the last 24h (24h TTL on scenario-result:*) — confirms the fix addresses the production outage. |
||
|
Elie Habib
|
3c47c1b222 |
fix(supply-chain): split chokepoint transit data + close silent zero-state cache (#3185)
* fix(supply-chain): split chokepoint transit data + close silent zero-state cache
Production supply-chain panel was rendering 13 empty chokepoints because
the getChokepointStatus RPC silently cached zero-state for 5 minutes:
1. supply_chain:transit-summaries:v1 grew to ~500 KB (180d × 13 × 14 fields
of history per chokepoint).
2. REDIS_OP_TIMEOUT_MS is 1.5 s. Vercel Sydney edge → Upstash for a 500 KB
GET consistently exceeded the budget; getCachedJson caught the AbortError
and returned null.
3. The 500 KB portwatch fallback read hit the same timeout.
4. summaries = {} → every summaries[cp.id] was undefined → 13 chokepoints
got the zero-state default → cached as a non-null success response for
REDIS_CACHE_TTL (5 min) instead of NEG_SENTINEL (120 s).
Fix (one PR, per docs/plans/chokepoint-rpc-payload-split.md):
- ais-relay.cjs: split seedTransitSummaries output.
- supply_chain:transit-summaries:v1 — compact (~30 KB, no history).
- supply_chain:transit-summaries:history:v1:{id} — per chokepoint
(~35 KB each, 13 keys). Both under the 1.5 s Redis read budget.
- New RPC GetChokepointHistory: lazy-loaded on card expand.
- get-chokepoint-status.ts: drop the 500 KB portwatch/corridorrisk/
chokepoint_transits fallback reads. Treat a null transit-summaries
read as upstreamUnavailable=true so cachedFetchJson writes NEG_SENTINEL
(2 min) instead of a 5-min zero-state pin. Omit history from the
response (proto field stays declared; empty array).
- server/_shared/redis.ts: tag AbortError timeouts with [REDIS-TIMEOUT]
key=… timeoutMs=… so log drains / Sentry-Vercel integration pick up
large-payload timeouts instead of them being silently swallowed.
- SupplyChainPanel.ts + MapPopup.ts: lazy-fetch history on card expand
via fetchChokepointHistory; session-scoped cache; graceful "History
unavailable" on empty/error. PRO gating on the map popup unchanged.
- Gateway: cache-tier entry for /get-chokepoint-history (slow).
- Tests: regression guards for upstreamUnavailable gate + per-id key
shape + handler wiring + proto query annotations.
Audit included in plan: no other RPC consumer read stacks >200 KB
besides displacement:summary:v1:2026 (724 KB, same risk, flagged for
follow-up PR). wildfire:fires:v1 at 1.7 MB loads via bootstrap (3 s
timeout, different path) — monitor but out of scope.
Expected impact:
- supply_chain:chokepoints:v4 payload drops from ~508 KB to <100 KB.
- supply_chain:transit-summaries:v1 drops from ~502 KB to <50 KB.
- RPC Redis reads stay well under 1.5 s in the hot path.
- Silent zero-state pinning is now impossible: null reads → 2-min neg
cache → self-heal on next relay tick.
* fix(supply-chain): address PR #3185 review — stop caching empty/error + fix partial coverage
Two P1 regressions caught in review:
1. Client cache poisoning on empty/error (MapPopup.ts, SupplyChainPanel.ts)
Empty-array is truthy in JS, so MapPopup's `!cached && !inflight` branch
never fired once we cached []. Neither `cached && cached.length` fired
either — popup stuck on "Loading transit history..." for the session.
SupplyChainPanel had the explicit `cached && !cached.length` branch but
still never retried, so the same transient became session-sticky there too.
Fix: cache ONLY non-empty successful responses. Empty/error show the
"History unavailable" placeholder but leave the cache untouched, so the
next re-expand retries. The /get-chokepoint-history gateway tier is
"slow" (5-min CF edge cache) → retries stay cheap.
2. Partial portwatch coverage treated as healthy (ais-relay.cjs)
seedTransitSummaries iterated Object.entries(pw), so if seed-portwatch
dropped N of 13 chokepoints (ArcGIS reject/empty), summaries had <13 keys.
get-chokepoint-status upstreamUnavailable fires only on fully-empty
summaries, so the N missing chokepoints fell through to zero-state rows
that got pinned in cache for 5 minutes.
Fix: iterate CANONICAL_IDS (Object.keys(CHOKEPOINT_THREAT_LEVELS)) and
fill zero-state for any ID missing from pw. Shape is consistently 13
keys. Track pwCovered → envelope + seed-meta recordCount reflect real
upstream coverage (not shape size), so health.js can distinguish 13/13
healthy from 10/13 partial. Warn-log on shortfall.
Tests: new regression guards
- panel must NOT cache empty arrays (historyCache.set with []).
- writer must iterate CANONICAL_IDS, not Object.entries(pw).
- seed-meta recordCount binds to pwCovered.
5718/5718 data tests pass. typecheck + typecheck:api clean.
|
||
|
Elie Habib
|
c72251178c |
feat(route-explorer): Sprint 4 — strategic-product impact tab + get-route-impact RPC (#2996)
* feat(route-explorer): Sprint 4 — strategic-product impact tab
Adds the Impact tab to the Route Explorer, powered by a new
get-route-impact RPC that returns strategic-product trade data for
any country pair.
Backend:
- New proto get_route_impact.proto with GetRouteImpact{Request,Response}
+ StrategicProduct message
- New handler server/worldmonitor/supply-chain/v1/get-route-impact.ts:
reads comtrade:bilateral-hs4:{iso2}:v1 store, computes lane value for
selected HS2, top 5 strategic products by value with chokepoint
exposure, resilience score (server-side from Redis), dependency flags
- Cache key ROUTE_IMPACT_KEY in cache-keys.ts (NOT in BOOTSTRAP_KEYS)
- Gateway + premium-paths registered as slow-browser premium RPC
- Client wrapper fetchRouteImpact in supply-chain/index.ts
Impact tab UI:
- CountryImpactTab.ts: strategic products table (top 5 by value),
lane value card for selected HS2, hs2InSeededUniverse banner when
HS2 is not in the 14 seeded sectors, comtradeSource states
(missing/empty/bilateral-hs4), drill-sideways on product row click
- LeftRail.updateDependencyFlags: renders flags from Impact response
with color-coded badges (compound_risk/single_source/diversifiable)
Data flow:
- fetchImpact fires in parallel with fetchResilience after lane data
loads, generation-scoped
- Impact response updates left-rail flags + resilience score
- Drill-sideways: clicking a product row switches the explorer's HS2
and re-queries all tabs
Server-side resilience:
- get-route-impact reads resilience:score:v8:{iso2} from Redis directly
so the data is available for future email briefs without client calls
Plan: docs/plans/2026-04-11-001-feat-worldwide-route-explorer-plan.md
* fix(route-explorer): real exposure score for flags + tabstrip sync on drill
P1: computeDependencyFlags hardcoded primaryExposure=80 whenever any
chokepoint existed, fabricating SINGLE_CORRIDOR_CRITICAL without using
real exposure data. Replaced with computeRealExposureScore that uses the
same route-cluster overlap logic as get-sector-dependency, computing the
actual exposure percentage before comparing against the >80 threshold.
P2: handleDrillSideways set state.tab=1 directly without going through
setTab(), leaving the tabstrip visually and semantically on Impact while
content showed Current. Now calls setTab(1) which updates both the
tabstrip active state and aria-selected.
* fix(route-explorer): guard resilience overwrite + normalize HS2 filter
P1: fetchImpact could zero the left-rail resilience score when
get-route-impact returned resilienceScore=0 (Redis miss fallback),
overwriting a valid score set by the concurrent fetchResilience call.
Now only applies the server-side score when it is actually > 0.
P2: HS4-to-HS2 matching used a redundant dual-condition filter
(hs4ToHs2 + startsWith) that masked a potential normalization bug.
Simplified to normalize hs2 once via parseInt then use a single
hs4ToHs2 comparison.
|
||
|
Elie Habib
|
822eef0fa6 |
feat(supply-chain): Sprint 1 — Route Explorer wrapper RPC (#2980)
* feat(supply-chain): Sprint 1 — Route Explorer wrapper RPC
Adds an internal wrapper around the vendor-only route-intelligence
compute so the upcoming Route Explorer UI can call it from a browser
PRO session instead of forcing an X-WorldMonitor-Key API gate.
Backend:
- New proto get-route-explorer-lane.proto with GetRouteExplorerLane{Request,Response}
- New handler server/worldmonitor/supply-chain/v1/get-route-explorer-lane.ts
- New static lookup tables _route-explorer-static-tables.ts:
TRANSIT_DAYS_BY_ROUTE_ID, FREIGHT_USD_BY_CARGO_TYPE,
BYPASS_CORRIDOR_GEOMETRY_BY_ID — covers all 5 land-bridge corridors
plus every sea-alternative corridor with hand-curated coordinates
- Wired into supply-chain handler.ts service dispatcher
- Cache key ROUTE_EXPLORER_LANE_KEY in cache-keys.ts (NOT in BOOTSTRAP_KEYS)
- Gateway entry: PREMIUM_RPC_PATHS + RPC_CACHE_TIER 'slow-browser'
- Premium path entry in src/shared/premium-paths.ts so browser PRO auth attaches
Response contract enriches route-intelligence with:
- primaryRouteGeometry polyline from TRADE_ROUTES (lon/lat pairs)
- fromPort/toPort coords on every bypass option so the client can call
MapContainer.setBypassRoutes directly without geometry lookups
- status: 'active' | 'proposed' | 'unavailable' derived from corridor notes
to honestly label kra_canal_future and black_sea_western_ports
- estTransitDaysRange + estFreightUsdPerTeuRange from static tables
- noModeledLane: true when origin/destination clusters share no routes
Client wrapper fetchRouteExplorerLane added to src/services/supply-chain/index.ts.
Tests: tests/route-explorer-lane.test.mts — 30-query smoke matrix
(10 country pairs × 3 HS2 codes), structural assertions only, no
hard-coded transit/cost values. Test exposes a pure computeLane()
function with an injectable status map so it does not need Redis.
Gap report (from smoke run): 12 of 30 queries fall back to a synthetic
primaryRouteId because the destination's port cluster has no shared route
with the origin (US-JP, ZA-IN, CL-CN, TR-DE × 3 HS2 each). These pairs
return noModeledLane:true; Sprint 3 will render an empty-state for them.
Plan: docs/plans/2026-04-11-001-feat-worldwide-route-explorer-plan.md
* fix(route-explorer): address PR #2980 review findings
P1: bypass warRiskTier was hard-coded to WAR_RISK_TIER_NORMAL, dropping
the live risk signal from chokepoint status. Now derived from the
statusMap via the corridor's primaryChokepointId.
P2: freight fallback in emptyResponse and client-side empty payload used
a cargo-agnostic container range for all cargo types. Removed the ranges
entirely from fallback/noModeledLane responses; they are only present
when the lane is actually modeled.
Suggestion: when noModeledLane is true, the response now returns empty
primaryRouteId, empty geometry, empty exposures, empty bypasses, and
omits transit/freight ranges. Previously it returned plausible-looking
synthetic data from the origin's first route which could mislead the UI.
Tests updated to assert the noModeledLane contract: empty fields when
the flag is set, non-empty ranges only when the lane is modeled.
* fix(route-explorer): cargo-aware route ranking + bypass waypoint risk
P1: primary route selection was order-dependent, picking whichever
shared route the origin cluster listed first. Mixed clusters like
CN/JP could return an energy lane for a container request. Now ranks
shared routes by cargo-category compatibility (container→container,
tanker→energy, bulk→bulk, roro→container) before selecting.
P1: bypass warRiskTier was copied from the primary chokepoint instead
of derived from the corridor's own waypointChokepointIds. This
overstated risk for alternatives like Cape of Good Hope whose waypoints
may have a lower risk tier. Now uses max-tier across waypoint
chokepoints, matching get-bypass-options.ts logic.
Suggestion: placeholder corridors with addedTransitDays=0 (like
gibraltar_no_bypass, cape_of_good_hope_is_bypass) are now filtered out.
Previously they could surface as active alternatives.
Regression tests added:
- CN→JP tanker: asserts energy route is selected over container route
- CN→DE with faked Suez=CRITICAL / Cape=NORMAL: asserts Cape bypass
shows NORMAL, not CRITICAL
- ES→EG: asserts zero-transit-day placeholders are excluded
* fix(route-explorer): scope exposures to primary route + narrow placeholder filter
P1: chokepointExposures and bypassOptions were computed from the full
sharedRoutes set, mixing data from energy/container corridors into a
single response. Now scoped to the cargo-ranked primaryRouteId only,
matching the proto contract that exposures are "on the primary route."
P2: the addedTransitDays === 0 filter was too broad and removed
kra_canal_future (a proposed bypass with real modeling). Narrowed to an
explicit PLACEHOLDER_CORRIDOR_IDS set (gibraltar_no_bypass,
cape_of_good_hope_is_bypass) so proposed zero-day corridors survive and
are surfaced with CORRIDOR_STATUS_PROPOSED.
Regression tests:
- chokepointExposures follow primaryRouteId (CN->JP container)
- kra_canal_future appears as CORRIDOR_STATUS_PROPOSED for Malacca routes
- placeholder filter still excludes explicit placeholders
* fix(route-explorer): address PR #2980 review comments
1. Unavailable corridors without waypoints (e.g. black_sea_western_ports)
now derive WAR_RISK_TIER_WAR_ZONE from their CORRIDOR_STATUS_UNAVAILABLE
status, instead of returning WAR_RISK_TIER_UNSPECIFIED. Corridors with
waypointChokepointIds still use max-tier across those waypoints.
2. Added fixture test with non-empty status map (suez=75/HIGH,
malacca=30/ELEVATED) so disruptionScore and warRiskTier assertions are
not trivially satisfied by the empty-map default path.
3. Documented the single-chokepoint bypass design gap in the test gap report:
bypassOptions only cover the primary chokepoint; multi-chokepoint routes
show exposure for all but bypass guidance for only the top one. Sprint 3
will decide whether to expand to top-N or add a UI hint.
|
||
|
Elie Habib
|
a742537ae5 |
feat(supply-chain): Sprint D — GetSectorDependency RPC + vendor route-intelligence API + webhooks (#2905)
* feat(supply-chain): Sprint D — GetSectorDependency RPC + vendor route-intelligence API + webhooks * fix(supply-chain): move bypass-corridors + chokepoint-registry to server/_shared to fix api/ boundary violations * fix(supply-chain): webhooks — persist secret, fix sub-resource routing, add ownership check * fix(supply-chain): address PR #2905 review findings - Use SHA-256(apiKey) for ownerTag instead of last-12-chars (unambiguous ownership) - Implement GET /api/v2/shipping/webhooks list route via per-owner Redis Set index - Tighten SSRF: https-only, expanded metadata hostname blocklist, document DNS rebinding edge-runtime limitation - Fix get-sector-dependency.ts stale src/config/ imports → server/_shared/ (Greptile P1) * fix(supply-chain): getSectorDependency returns blank primaryChokepointId for landlocked countries computeExposures() previously mapped over all of CHOKEPOINT_REGISTRY even when nearestRouteIds was empty, producing a full array of score-0 entries in registry insertion order. The caller's exposures[0] then picked the first registry entry (Suez) as the "primary" chokepoint despite primaryChokepointExposure = 0. LI, AD, SM, BT and other landlocked countries were all silently assigned a fake chokepoint. Fix: guard at the top of computeExposures() -- return [] when input is empty so primaryChokepointId stays '' and primaryChokepointExposure stays 0. |
||
|
Elie Habib
|
23ed4eba44 |
fix(supply-chain): address all code review findings from PR #2873 (#2878)
* fix(supply-chain): address all code review findings from PR #2873 - Rename costIncreasePct → supplyDeficitPct (semantic correction) - Add primaryChokepointWarRiskTier to GetBypassOptionsResponse - Consolidate ThreatLevel/threatLevelToWarRiskTier into _insurance-tier.ts - Replace inline CpEntry/ChokepointStatusCacheEntry with ChokepointInfo - Add outer cachedFetchJson wrapper (3 serial Redis reads → 1 on warm path) - Add hs2 validation guard matching sibling handler pattern - Extract CHOKEPOINT_STATUS_KEY constant; eliminate string literal duplication - Add SCORE_RISK_WEIGHT/SCORE_COST_WEIGHT named constants; clamp liveScore ≥ 0 - Add Math.max(0,...) to liveScore for sub-1.0 cost multiplier corridors - Fix closurePct: req.closurePct ?? 100 (was || which falsy-coalesced zero) - Type fetchBypassOptions cargoType as CargoType (was implicit string) - Add exhaustiveness check to threatLevelToInsurancePremiumBps switch - Move TIER_RANK to module level in _insurance-tier.ts - Update WIDGET_PRO_SYSTEM_PROMPT with both new PRO RPCs * fix(supply-chain): fix supplyDeficitPct averaging and coverageDays sentinel - Remove .filter(d > 0) from productDeficits: zero-deficit products have demand and must stay in the denominator to avoid overstating the average - Clamp coverageDays = Math.max(0, effectiveCoverDays): prevents -1 net-exporter sentinel from leaking into the public API response - Update proto comment: document 0 for net exporters - Add test assertions for both contracts * chore(api-docs): regenerate OpenAPI docs for coverage_days comment update * refactor(supply-chain): use CHOKEPOINT_STATUS_KEY in chokepoint-status writer The key was extracted to cache-keys.ts in the previous commit but the primary writer (getChokepointStatus) and BOOTSTRAP_CACHE_KEYS still embedded the raw string literal. Import the constant at both sites to complete the refactor. * test: update supply-chain-v2 assertions for CHOKEPOINT_STATUS_KEY refactor Handler now imports CHOKEPOINT_STATUS_KEY as REDIS_CACHE_KEY from cache-keys.ts rather than defining a local constant. BOOTSTRAP_CACHE_KEYS also references the constant. Update source-string assertions to match the new patterns. * fix: keep BOOTSTRAP_CACHE_KEYS.chokepoints as string literal bootstrap.test.mjs enforces string-literal values in BOOTSTRAP_CACHE_KEYS via regex. CHOKEPOINT_STATUS_KEY is used in handler imports and is the primary dedup win; the static registry entry stays as-is per test contract. |
||
|
Elie Habib
|
bd07829518 |
feat(supply-chain): Sprint 2 — bypass corridor intelligence + cost shock engine (#2873)
* feat(supply-chain): Sprint 2 — bypass corridor intelligence + cost shock engine - src/config/bypass-corridors.ts: ~40 bypass corridors for all 13 chokepoints - server/supply-chain/v1/get-bypass-options.ts: PRO-gated RPC, live bypass scoring from chokepoint status cache - server/supply-chain/v1/get-country-cost-shock.ts: PRO-gated RPC, war risk premium BPS + energy coverage days (HS 27) - server/supply-chain/v1/_insurance-tier.ts: pure function, Lloyd's JWC threat → premium BPS - gateway.ts + premium-paths.ts: registered both RPCs as slow-browser + PRO-gated - src/services/supply-chain/index.ts: fetchBypassOptions + fetchCountryCostShock client methods - proto: GetBypassOptions + GetCountryCostShock messages + service registrations - tests/supply-chain-sprint2.test.mjs: 61 tests covering all new components Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> * fix(cost-shock): call computeEnergyShockScenario directly instead of reading wrong cache key The old code read from `energy:shock:${iso2}:${chokepointId}:v1` which never matches the actual v2 cache key written by compute-energy-shock.ts. Fix by calling computeEnergyShockScenario() directly (it handles v2 caching internally) and mapping effectiveCoverDays + crude product deficitPct to the response fields. * fix(cost-shock): average refined product deficitPct instead of looking for non-existent 'crude' product --------- Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> |
||
|
Elie Habib
|
6e401ad02f |
feat(supply-chain): Global Shipping Intelligence — Sprint 0 + Sprint 1 (#2870)
* feat(supply-chain): Sprint 0 — chokepoint registry, HS2 sectors, war_risk_tier
- src/config/chokepoint-registry.ts: single source of truth for all 13
canonical chokepoints with displayName, relayName, portwatchName,
corridorRiskName, baselineId, shockModelSupported, routeIds, lat/lon
- src/config/hs2-sectors.ts: static dictionary for all 99 HS2 chapters
with category, shockModelSupported (true only for HS27), cargoType
- server/worldmonitor/supply-chain/v1/_chokepoint-ids.ts: migrated to
derive CANONICAL_CHOKEPOINTS from chokepoint-registry; no data duplication
- src/config/geo.ts + src/types/index.ts: added chokepointId field to
StrategicWaterway interface and all 13 STRATEGIC_WATERWAYS entries
- src/components/MapPopup.ts: switched chokepoint matching from fragile
name.toLowerCase() to direct chokepointId === id comparison
- server/worldmonitor/intelligence/v1/_shock-compute.ts: migrated from old
IDs (hormuz/malacca/babelm) to canonical IDs (hormuz_strait/malacca_strait/
bab_el_mandeb); same for CHOKEPOINT_LNG_EXPOSURE
- proto/worldmonitor/supply_chain/v1/supply_chain_data.proto: added
WarRiskTier enum + war_risk_tier field (field 16) on ChokepointInfo
- get-chokepoint-status.ts: populates warRiskTier from ChokepointConfig.threatLevel
via new threatLevelToWarRiskTier() helper (FREE field, no PRO gate)
* feat(supply-chain): Sprint 1 — country chokepoint exposure index + sector ring
S1.1: scripts/shared/country-port-clusters.json
~130 country → {nearestRouteIds, coastSide} mappings derived from trade route
waypoints; covers all 6 seeded Comtrade reporters plus major trading nations.
S1.2: scripts/seed-hs2-chokepoint-exposure.mjs
Daily cron seeder. Pure computation — reads country-port-clusters.json,
scores each country against CHOKEPOINT_REGISTRY route overlap, writes
supply-chain:exposure:{iso2}:{hs2}:v1 keys + seed-meta (24h TTL).
S1.3: RPC get-country-chokepoint-index (PRO-gated, request-varying)
- proto: GetCountryChokepointIndexRequest/Response + ChokepointExposureEntry
- handler: isCallerPremium gate; cachedFetchJson 24h; on-demand for any iso2
- cache-keys.ts: CHOKEPOINT_EXPOSURE_KEY(iso2, hs2) constant
- health.js: chokepointExposure SEED_META entry (48h threshold)
- gateway.ts: slow-browser cache tier
- service client: fetchCountryChokepointIndex() exported
S1.4: Chokepoint popup HS2 sector ring chart (PRO-gated)
Static trade-sector breakdown (IEA/UNCTAD estimates) per 9 major chokepoints.
SVG donut ring + legend shown for PRO users; blurred lockout + gate-hit
analytics for free users. Wired into renderWaterwayPopup().
🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.com/claude-code) + Compound Engineering v2.49.0
Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
* fix(tests): update energy-shock-v2 tests to use canonical chokepoint IDs
CHOKEPOINT_EXPOSURE and CHOKEPOINT_LNG_EXPOSURE keys were migrated from
short IDs (hormuz, malacca, babelm) to canonical registry IDs
(hormuz_strait, malacca_strait, bab_el_mandeb) in Sprint 0.
Test fixtures were not updated at the time; fix them now.
* fix(tests): update energy-shock-seed chokepoint ID to canonical form
VALID_CHOKEPOINTS changed to canonical IDs in Sprint 0; the seed test
that checks valid IDs was not updated alongside it.
* fix(cache-keys): reword JSDoc comment to avoid confusing bootstrap test regex
The comment "NOT in BOOTSTRAP_CACHE_KEYS" caused the bootstrap.test.mjs
regex to match the comment rather than the actual export declaration,
resulting in 0 entries found. Rephrase to "excluded from bootstrap".
* fix(supply-chain): address P1 review findings for chokepoint exposure index
- Add get-country-chokepoint-index to PREMIUM_RPC_PATHS (CDN bypass)
- Validate iso2/hs2 params before Redis key construction (cache injection)
- Fix seeder TTL to 172800s (2× interval) and extend TTL on skipped lock
- Fix CHOKEPOINT_EXPOSURE_SEED_META_KEY to match seeder write key
- Render placeholder sectors behind blur gate (DOM data leakage)
- Document get-country-chokepoint-index in widget agent system prompts
* fix(lint): resolve Biome CI failures
- Add biome.json overrides to silence noVar in HTML inline scripts,
disable linting for public/ vendor/build artifacts and pro-test/
- Remove duplicate NG and MW keys from country-port-clusters.json
- Use import attributes (with) instead of deprecated assert syntax
* fix(build): drop JSON import attribute — esbuild rejects `with` syntax
---------
Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
|
||
|
Elie Habib
|
a09f49ff9c |
feat(supply-chain): energy flow estimates per chokepoint (mb/d card row) (#2780)
* feat(supply-chain): energy flow estimates per chokepoint (mb/d card row) - Add FlowEstimate proto message + ChokepointInfo field 15; regenerate stubs - Add baselineId mapping to _chokepoint-ids.ts (7 of 13 chokepoints) - Add relayId to seed-chokepoint-baselines.mjs CHOKEPOINTS entries - New seed-chokepoint-flows.mjs: reads portwatch + baselines, computes 7d tanker avg vs 90d baseline, outputs flow_ratio and current_mbd; prefers DWT (capTanker) when available; flags disruption if last 3 days each below 0.85 threshold; writes energy:chokepoint-flows:v1 (TTL 3d) - get-chokepoint-status.ts: parallel-reads flows key, attaches flowEstimate - SupplyChainPanel: compact card gains mb/d row (red <85%, amber <95%) - 19 new unit tests for flow computation and seeder contract * fix(chokepoint-flows): base useDwt on 90d baseline window, not recent 7 days Zero recent capTanker is the disruption signal, not a reason to fall back to vessel counts. Switching metrics during peak disruption caused the seeder to report a higher (less accurate) flow estimate exactly when oil-flow collapse is most acute. useDwt is now locked to whether the baseline window has DWT data -- stable across disruption events. Adds regression test covering DWT-collapse scenario. * fix(chokepoint-flows): require majority DWT coverage in baseline before activating DWT mode capBaselineSum > 0 would activate DWT on a single non-zero day during partial data roll-out, pulling down the baseline average via zero-filled gaps. Now requires >= ceil(prev90.length / 2) days with DWT data. ArcGIS data is all-or-nothing per chokepoint in practice, so this guard catches edge cases without affecting normal operation. |
||
|
Elie Habib
|
190095ca89 |
feat(supply-chain): stacked vessel-type transit chart with 7d MA, DWT tab, zoom (#2777)
* feat(supply-chain): stacked vessel-type transit chart with 7d MA, DWT tab, zoom - Update TransitDayCount proto (fields 6-14): container, dry_bulk, general_cargo, roro, cap_* DWT capacity fields; regenerate TS types - Rewrite transit-chart.ts: 5-type stacked bar (container/dryBulk/ generalCargo/roro/tanker), 7d MA dashed overlay, Transit Calls / Trade Volume tab toggle, 1m/3m/6m zoom buttons, richer tooltip - SupplyChainPanel: enlarge chart placeholder min-height 120->200px * fix(transit-chart): stop control clicks bubbling + track source div in destroy - stopPropagation on controls container prevents tab/zoom button clicks from collapsing the chokepoint card - source div now tracked as this.source and cleaned up in destroy(), preventing duplicate attribution lines on repeated remounts * fix(transit-chart): import from generated client, reuse data in onMouseMove - Import TransitDayCount from generated client stub instead of server layer; keeps src/ imports within src/ - onMouseMove: reuse already-bound data array for MA computation instead of calling visibleData() again on every mouse event |
||
|
Elie Habib
|
1f56afeb82 |
feat(panels): disease outbreaks panel/layer, social velocity panel, shipping stress tab (#2383)
* feat(panels): disease outbreaks panel/layer, social velocity panel, shipping stress tab - DiseaseOutbreaksPanel: feed-style panel with alert/warning/watch filter pills, source links, relative timestamps (WHO/ProMED/HealthMap) - SocialVelocityPanel: ranked Reddit trending posts by velocity score with subreddit badge, vote/comment counts, velocity bar - SupplyChainPanel: Stress tab with composite stress gauge and carrier table with sparklines (GetShippingStressResponse) - diseaseOutbreaks map layer: ScatterplotLayer via country centroids, color/radius by alert level, tooltip - MapContainer.setDiseaseOutbreaks(): cached setter with DeckGLMap delegation - data-loader: loadDiseaseOutbreaks/loadSocialVelocity/loadSupplyChain with stress wired into tasks - MapLayers.diseaseOutbreaks added to types, layer registry (globe icon), full variant order, all default objects 🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.com/claude-code) + Compound Engineering v2.49.0 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(supply-chain): add upstreamUnavailable to ShippingStressResponse, restore test-compatible banner guard * fix(panels): filter pills use alertLevel equality, sanitizeUrl on hrefs, globe TODO, E2E layer enabled --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Elie Habib
|
1e1f377078 |
feat(panels): Disease Outbreaks, Shipping Stress, Social Velocity, nuclear test site enrichment (#2375)
* feat(panels): Disease Outbreaks, Shipping Stress, Social Velocity, nuclear test site monitoring - Add HealthService proto with ListDiseaseOutbreaks RPC (WHO + ProMED RSS) - Add GetShippingStress RPC to SupplyChainService (Yahoo Finance carrier ETFs) - Add GetSocialVelocity RPC to IntelligenceService (Reddit r/worldnews + r/geopolitics) - Enrich earthquake seed with Haversine nuclear test-site proximity scoring - Add 5 nuclear test sites to NUCLEAR_FACILITIES (Punggye-ri, Lop Nur, Novaya Zemlya, Nevada NTS, Semipalatinsk) - Add shipping stress + social velocity seed loops to ais-relay.cjs - Add seed-disease-outbreaks.mjs Railway cron script - Wire all new RPCs: edge functions, handlers, gateway cache tiers, health.js STANDALONE_KEYS/SEED_META * fix(relay): apply gold standard retry/TTL-extend pattern to shipping-stress and social-velocity seeders * fix(review): address all PR #2375 review findings - health.js: shippingStress maxStaleMin 30→45 (3x interval), socialVelocity 20→30 (3x interval) - health.js: remove shippingStress/diseaseOutbreaks/socialVelocity from ON_DEMAND_KEYS (relay/cron seeds, not on-demand) - cache-keys.ts: add shippingStress, diseaseOutbreaks, socialVelocity to BOOTSTRAP_CACHE_KEYS - ais-relay.cjs: stressScore formula 50→40 (neutral market = moderate, not elevated) - ais-relay.cjs: fetchedAt Date.now() (consistent with other seeders) - ais-relay.cjs: deduplicate cross-subreddit article URLs in social velocity loop - seed-disease-outbreaks.mjs: WHO URL → specific DON RSS endpoint (not dead general news feed) - seed-disease-outbreaks.mjs: validate() requires outbreaks.length >= 1 (reject empty array) - seed-disease-outbreaks.mjs: stable id using hash(link) not array index - seed-disease-outbreaks.mjs: RSS regexes use [\s\S]*? for CDATA multiline content - seed-earthquakes.mjs: Lop Nur coordinates corrected (41.39,89.03 not 41.75,88.35) - seed-earthquakes.mjs: sourceVersion bumped to usgs-4.5-day-nuclear-v1 - earthquake.proto: fields 8-11 marked optional (distinguish not-enriched from enriched=false/0) - buf generate: regenerate seismology service stubs * revert(cache-keys): don't add new keys to bootstrap without frontend consumers * fix(panels): address all P1/P2/P3 review findings for PR #2375 - proto: add INT64_ENCODING_NUMBER annotation + sebuf import to get_shipping_stress.proto (run make generate) - bootstrap: register shippingStress (fast), socialVelocity (fast), diseaseOutbreaks (slow) in api/bootstrap.js + cache-keys.ts - relay: update WIDGET_SYSTEM_PROMPT with new bootstrap keys and live RPCs for health/supply-chain/intelligence - seeder: remove broken ProMED feed URL (promedmail.org/feed/ returns HTML 404); add 500K size guard to fetchRssItems; replace private COUNTRY_CODE_MAP with shared geo-extract.mjs; remove permanently-empty location field; bump sourceVersion to who-don-rss-v2 - handlers: remove dead .catch from all 3 new RPC handlers; fix stressLevel fallback to low; fix fetchedAt fallback to 0 - services: add fetchShippingStress, disease-outbreaks.ts, social-velocity.ts with getHydratedData consumers |
||
|
Elie Habib
|
f336418c17 |
feat(advisories): gold standard migration for security advisories (#1637)
* feat(advisories): gold standard migration for security advisories Move security advisories from client-side RSS fetching (24 feeds per page load) to Railway cron seed with Redis-read-only Vercel handler. - Add seed script fetching via relay RSS proxy with domain allowlist - Add ListSecurityAdvisories proto, handler, and RPC cache tier - Add bootstrap hydration key for instant page load - Rewrite client service: bootstrap -> RPC fallback, no browser RSS - Wire health.js, seed-health.js, and dataSize tracking * fix(advisories): empty RPC returns ok:true, use full country map P1 fixes from Codex review: - Return ok:true for empty-but-successful RPC responses so the panel clears to empty instead of stuck loading on cold environments - Replace 50-entry hardcoded country map with 251-entry shared config generated from the project GeoJSON + aliases, matching coverage of the old client-side nameToCountryCode matcher * fix(advisories): add Cote d'Ivoire and other missing country aliases Adds 14 missing aliases including "cote d ivoire" (US State Dept title format), common article-prefixed names (the Bahamas, the Gambia), and alternative official names (Czechia, Eswatini, Cabo Verde, Timor-Leste). * fix(proto): inject @ts-nocheck via Makefile generate target buf generate does not emit @ts-nocheck, but tsc strict mode rejects the generated code. Adding a post-generation sed step in the Makefile ensures both CI proto-freshness (make generate + diff) and CI typecheck (tsc --noEmit) pass consistently. |
||
|
Elie Habib
|
45f5e5a457 |
feat(forecast): AI Forecasts prediction module (#1579)
* feat(forecast): add AI Forecasts prediction module (Pro-tier)
MiroFish-inspired prediction engine that generates structured forecasts
across 6 domains (conflict, market, supply chain, political, military,
infrastructure) using existing WorldMonitor data streams.
- Proto definitions for ForecastService with GetForecasts RPC
- Dedicated seed script (seed-forecasts.mjs) with 6 domain detectors,
cross-domain cascade resolver, prediction market calibration, and
trend detection via prior snapshot comparison
- Premium-gated RPC handler (PREMIUM_RPC_PATHS enforcement)
- Lazy-loaded ForecastPanel with domain filters, probability bars,
trend arrows, signal evidence, and cascade links
- Health monitoring integration (seed-meta freshness tracking)
- Refresh scheduler with API key guard
* test(forecast): add 47 unit tests for forecast detectors and utilities
Covers forecastId, normalize, resolveCascades, calibrateWithMarkets,
computeTrends, and smoke tests for all 6 domain detectors. Exports
testable functions from seed script with direct-run guard.
* fix(forecast): domain mismatch 'infra' vs 'infrastructure', add panel category
- Seed script used 'infra' but ForecastPanel filtered on 'infrastructure',
causing Infra tab to show zero results
- Added 'forecast' to intelligence category in PANEL_CATEGORY_MAP
* fix(forecast): move CSS to one-time injection, improve type safety
- P2: Move style block from setContent to one-time document.head injection
to prevent CSS accumulation on repeated renders
- P3: Replace +toFixed(3) with Math.round for readability in seed script
- P3: Use Forecast type instead of any[] in RPC handler filter
* fix(forecast): handle sebuf proto data shapes from Redis
Detectors now normalize CII scores from server-side proto format
(combinedScore, TREND_DIRECTION_RISING, region) to uniform shape.
Outage severity handles proto enum format (SEVERITY_LEVEL_HIGH).
Added confidence floor of 0.3 for single-source predictions.
Verified against live Redis: 2 predictions generated (Iran infra
shutdown, IL political instability).
* feat(forecast): unlock AI Forecasts on web, lock desktop only (trial)
- Remove forecast RPC from PREMIUM_RPC_PATHS (web access is free)
- Panel locked on desktop only (same as oref-sirens/telegram-intel)
- Remove API key guards from data-loader and refresh scheduler
- Web users get full access during trial period
* chore: regenerate proto types with make generate
Re-ran make generate after rebasing on main. Plugin v0.7.0 dropped
@ts-nocheck from output, added it back to all 50 generated files.
Fixed 4 type errors from proto codegen changes:
- MarketSource enum -> string union type
- TemporalAnomalyProto -> TemporalAnomaly rename
- webcam lastUpdated number -> string
* fix(forecast): use chokepoints v4 key, include ciiContribution in unrest
- P1: Switch chokepoints input from stale v2 to active v4 Redis key,
matching bootstrap.js and cache-keys.ts
- P2: Add ciiContribution to unrest component fallback chain in
normalizeCiiEntry so political detector reads the correct sebuf field
* feat(forecast): Phase 2 LLM scenario enrichment + confidence model
MiroFish-inspired enhancements:
- LLM scenario narratives via Groq/OpenRouter (narrative-only, no numeric
adjustment). Evidence-grounded prompts with mandatory signal citation
and few-shot examples from MiroFish's SECTION_SYSTEM_PROMPT_TEMPLATE.
- Top-4 predictions batched into single LLM call for cost efficiency.
- News context from newsInsights attached to all predictions for LLM
prompt grounding (NOT in signals, cannot affect confidence).
- Deterministic confidence model: source diversity via SIGNAL_TO_SOURCE
mapping (deduplicates cii+cii_delta, theater+indicators) + calibration
agreement from prediction market drift. Floor 0.2, ceiling 1.0.
- Output validation: rejects scenarios without signal references.
- Truncated JSON repair for small model output.
- Structured JSON logging for LLM calls.
- Redis cache for LLM scenarios (1h TTL).
- 23 new tests (70 total), all passing.
- Live-tested: OpenRouter gemini-2.5-flash produces evidence-grounded
scenario narratives from real WorldMonitor data.
* feat(forecast): Phase 3 multi-perspective scenarios, projections, data-driven cascades
MiroFish-inspired enhancements:
- Multi-perspective LLM analysis: top-2 predictions get strategic,
regional, and contrarian viewpoints via combined LLM call
- Probability projections: domain-specific decay curves (h24/d7/d30)
anchored to timeHorizon so probability equals projections[timeHorizon]
- Data-driven cascade rules: moved from hardcoded array to JSON config
(scripts/data/cascade-rules.json) with schema validation, named
predicate evaluators, unknown key rejection, and fallback to defaults
- 4 new cascade paths: infrastructure->supply_chain, infrastructure->market
(both requiresSeverity:total), conflict->political, political->market
- Proto: added Perspectives and Projections messages to Forecast
- ForecastPanel: renders projections row and conditional perspectives toggle
- 89 tests (19 new), all passing
- Live-tested: OpenRouter produces perspectives from real data
* feat(forecast): Phase 4 data utilization + entity graph
Fixes data gaps that prevented 4 of 6 detectors from firing:
- Input normalizers: chokepoint v4 shape + GPS hexes-to-zones mapping
- Chokepoint warm-ping (production-only, requires WM_API_BASE_URL)
- Lowered CII conflict threshold from 70 to 60, gated on level=high|critical
4 new standalone detectors:
- UCDP conflict zones (10+ events per country)
- Cyber threat concentration (5+ threats per country)
- GPS jamming in maritime shipping zones (5 regions)
- Prediction markets as signals (60-90% probability markets)
Entity-relationship graph (file-based, 38 nodes):
- Countries, theaters, commodities, chokepoints, alliances
- Alias table resolves both ISO codes and display names
- Graph cascade discovery links predictions across entities
Result: 51 predictions (up from 1-2), spanning conflict, infrastructure,
and supply chain domains. 112 tests, all passing.
* fix(forecast): redis cache format, signal source mapping, type safety
Fresh-eyes audit fixes:
- BUG: redisSet used wrong Upstash API format (POST body with {value,ex}
instead of command array ['SET',key,value,'EX',ttl]). LLM cache writes
were silently failing, causing fresh LLM calls every run.
- BUG: prediction_market signal type missing from SIGNAL_TO_SOURCE,
inflating confidence for market-derived predictions.
- CLEANUP: Remove unnecessary (f as any) casts in ForecastPanel since
generated Forecast type already has projections/perspectives fields.
- CLEANUP: Bump health maxStaleMin from 60 to 90 to avoid false STALE
alerts when LLM calls add latency to seed runs.
* feat(forecast): headline-entity matching with news corroboration signals
Uses entity graph aliases to match headlines to predictions by
country/theater (excludes commodity/infrastructure nodes to prevent
false positives). Predictions with matching headlines get a
news_corroboration signal visible in the panel.
Also fixes buildUserPrompt to merge unique headlines from ALL
predictions in the LLM batch (was only reading preds[0].newsContext).
Live-tested: 13 of 51 predictions now have corroborating headlines
(Iran, Israel, Syria, Ukraine, etc). 116 tests, all passing.
* feat(forecast): add country-codes.json for headline-entity matching
56 countries with ISO codes, full names, and scoring keywords (extracted
from src/config/countries.ts + UCDP-relevant additions). Used by
attachNewsContext for richer headline matching via getSearchTermsForRegion
which combines country-codes + entity graph + keyword aliases.
14/57 predictions now have news corroboration (limited by headline
coverage, not matching quality: only 8 headlines currently available).
* feat(forecast): read 300 headlines from news digest instead of 8
Read news:digest:v1:full:en (300 headlines across 16 categories) instead
of just news:insights:v1 topStories (8 headlines). Fallback to topStories
if digest is unavailable.
Result: news corroboration jumped from 25% to 64% (38/59 predictions).
* fix(forecast): handle parenthetical country names in headline matching
Strip suffixes like '(Zaire)', '(Burma)', '(Soviet Union)' from UCDP
region names before matching against country-codes.json. Also use
includes() for reverse name lookup to catch partial matches.
Corroboration: 64% -> 69% (41/59). Remaining 18 unmatched are countries
with no current English-language news coverage.
* fix(forecast): cache validated LLM output, add digest test, log cache errors
Fresh-eyes audit fixes:
- Combined LLM cache now stores only validated items (was caching raw
unvalidated output, serving potentially invalid scenarios on cache hit)
- redisSet logs warnings on failure (was silently swallowing all errors)
- Added digest-based test for attachNewsContext (primary path was untested)
- Fixed test arity: attachNewsContext(preds, news, digest) with 3 params
* fix(forecast): remove dead confidenceFromSources, reduce warm-ping timeout
- P2: Remove confidenceFromSources (dead code, computeConfidence overwrites
all initial confidence values). Inline the formula in original detectors.
- P3: Reduce warm-ping timeout from 30s to 15s (non-critical step)
- P3: Add trial status comment on forecast panel config
* fix(forecast): resolve ISO codes to country names, fix market detector, safe pre-push
P1 fixes from code review:
- CII ISO codes (IL, IR) now resolved to full country names (Israel, Iran)
via country-codes.json. Prevents substring false positives (IL matching
Chile) in event correlation. Uses word-boundary regex for matching.
- Market detector CII-to-theater mapping now uses entity graph traversal
instead of broken theater-name substring matching. Iran correctly maps
to Middle East theater via graph links.
- Pre-push hook no longer runs destructive git checkout on proto freshness
failure. Reports mismatch and exits without modifying worktree.
|
||
|
Elie Habib
|
0383253a59 |
feat(supply-chain): chokepoint transit intelligence with 3 data sources (#1560)
* feat(supply-chain): replace S&P Global with 3 free maritime data sources Replace expensive S&P Global Maritime API with IMF PortWatch (vessel transit counts), CorridorRisk (risk intelligence), and AISStream chokepoint crossing counter. All external API calls run on Railway relay, Vercel reads Redis only. - Add 4 new chokepoints (10 total): Cape of Good Hope, Gibraltar, Bosphorus, Dardanelles - Add TransitSummary proto (field 14) with today counts, WoW%, 180d history, risk context - Add D3 multi-line chart (tanker vs cargo) with expandable chokepoint cards - Add crossing detection with enter+dwell+exit semantics, 30min cooldown, 5min min dwell - Add PortWatch seed loop (6h), CorridorRisk seed loop (1h), transit seed loop (10min) - Add canonical chokepoint ID map for cross-source name resolution - 177 tests passing across 6 test files * fix(supply-chain): address P2 review findings - Discard partial PortWatch pagination results on mid-page failure (prevents truncated history with wrong WoW numbers cached for 6h) - Rename "Transit today" to "24h" label (rolling 24h window, not calendar day) - Fix chart label from "30d" to "180d" (matches actual PortWatch query range) - Add 30s initial seed for chokepoint transits on relay cold start (prevents 10min gap of zero transit data) * feat(supply-chain): swap D3 chart for TradingView lightweight-charts Replace hand-rolled D3 SVG transit chart with lightweight-charts v5 canvas rendering for Bloomberg-quality time-series visualization. - Add TransitChart helper class with mount/destroy lifecycle, theme listener, and autoSize support - Use MutationObserver (not rAF) to mount chart after setContent debounce - Clean up chart on tab switch, collapse, and re-render (no orphaned canvases) - Respond to theme-changed events via chart.applyOptions() - D3 stays for other 5 components (ProgressCharts, RenewableEnergy, etc.) * feat(supply-chain): add geo coords and trade routes for 4 new chokepoints Cherry-pick from PR #1511: Cape of Good Hope, Gibraltar, Bosphorus, and Dardanelles map-layer coordinates and trade route definitions. * fix(supply-chain): health.js v2->v4 key + double cache TTLs for missed seeds - health.js chokepoints key was still v2, now v4 (matches handler + bootstrap) - PortWatch TTL: 21600s (6h) -> 43200s (12h), seed interval stays 6h - CorridorRisk TTL: 3600s (1h) -> 7200s (2h), seed interval stays 1h - Ensures one missed seed run doesn't expire the key and cause empty data |
||
|
Elie Habib
|
c2f17dec45 |
fix(supply-chain): resolve P1 threat zeroing and P2 geo-first misclassification (#964)
* enhance supply chain panel * fix(supply-chain): resolve P1 threat zeroing and P2 geo-first misclassification P1: threat baseline is now always applied regardless of config staleness — stale config only adds a review-recommended note, never zeros the score. P2: resolveChokepointId now checks text evidence first and only falls back to proximity when text has no confident match. Adds regression test: text "Bab el-Mandeb" with location near Suez correctly resolves to bab_el_mandeb. --------- Co-authored-by: fayez bast <fayezbast15@gmail.com> |
||
|
Sebastien Melki
|
6669d373cf |
feat: convert 52 API endpoints from POST to GET for edge caching (#468)
* feat: convert 52 API endpoints from POST to GET for edge caching Convert all cacheable sebuf RPC endpoints to HTTP GET with query/path parameters, enabling CDN edge caching to reduce costs. Flatten nested request types (TimeRange, PaginationRequest, BoundingBox) into scalar query params. Add path params for resource lookups (GetFredSeries, GetHumanitarianSummary, GetCountryStockIndex, GetCountryIntelBrief, GetAircraftDetails). Rewrite router with hybrid static/dynamic matching for path param support. Kept as POST: SummarizeArticle, ClassifyEvent, RecordBaselineSnapshot, GetAircraftDetailsBatch, RegisterInterest. Generated with sebuf v0.9.0 (protoc-gen-ts-client, protoc-gen-ts-server). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add rate_limited field to market response protos The rateLimited field was hand-patched into generated files on main but never declared in the proto definitions. Regenerating wiped it out, breaking the build. Now properly defined in both ListEtfFlowsResponse and ListMarketQuotesResponse protos. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove accidentally committed .planning files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |