mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
main
100 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sebastien Melki
|
1db05e6caa |
feat(usage): per-request Axiom telemetry pipeline (gateway + upstream attribution) (#3403)
* feat(gateway): thread Vercel Edge ctx through createDomainGateway (#3381) PR-0 of the Axiom usage-telemetry stack. Pure infra change: no telemetry emission yet, only the signature plumbing required for ctx.waitUntil to exist on the hot path. - createDomainGateway returns (req, ctx) instead of (req) - rewriteToSebuf propagates ctx to its target gateway - 5 alias callsites updated to pass ctx through - ~30 [rpc].ts callsites unchanged (export default createDomainGateway(...)) Pattern reference: api/notification-channels.ts:166. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(usage): pure UsageIdentity resolver + Axiom emit primitives (#3381) server/_shared/usage-identity.ts - buildUsageIdentity: pure function, consumes already-resolved gateway state. - Static ENTERPRISE_KEY_TO_CUSTOMER map (explicit, reviewable in code). - Does not re-verify JWTs or re-validate API keys. server/_shared/usage.ts - buildRequestEvent / buildUpstreamEvent: allowlisted-primitive builders only. Never accept Request/Response — additive field leaks become structurally impossible. - emitUsageEvents → ctx.waitUntil(sendToAxiom). Direct fetch, 1.5s timeout, no retry, gated by USAGE_TELEMETRY=1 and AXIOM_API_TOKEN. - Sliding-window circuit breaker (5% over 5min, min 20 samples). Trips with one structured console.error; subsequent drops are 1%-sampled console.warn. - Header derivers reuse Vercel/CF headers for request_id, region, country, reqBytes; ua_hash null unless USAGE_UA_PEPPER is set (no stable fingerprinting). - Dev-only x-usage-telemetry response header for 2-second debugging. server/_shared/auth-session.ts - New resolveClerkSession returning { userId, orgId } in one JWT verify so customer_id can be Clerk org id without a second pass. resolveSessionUserId kept as back-compat wrapper. No emission wiring yet — that lands in the next commit (gateway request event + 403 + 429). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(gateway): emit Axiom request events on every return path (#3381) Wires the request-event side of the Axiom usage-telemetry stack. Behind USAGE_TELEMETRY=1 — no-op when the env var is unset. Emit points (each builds identity from accumulated gateway state): - origin_403 disallowed origin → reason=origin_403 - API access subscription required (403) - legacy bearer 401 / 403 / 401-without-bearer - entitlement check fail-through - endpoint rate-limit 429 → reason=rate_limit_429 - global rate-limit 429 → reason=rate_limit_429 - 405 method not allowed - 404 not found - 304 etag match (resolved cache tier) - 200 GET with body (resolved cache tier, real res_bytes) - streaming / non-GET-200 final return (res_bytes best-effort) Identity inputs (UsageIdentityInput): - sessionUserId / clerkOrgId from new resolveClerkSession (one JWT verify) - isUserApiKey + userApiKeyCustomerRef from validateUserApiKey result - enterpriseApiKey when keyCheck.valid + non-wm_ wmKey present - widgetKey from x-widget-key header (best-effort) - tier captured opportunistically from existing getEntitlements calls Header derivers reuse Vercel/CF metadata (x-vercel-id, x-vercel-ip-country, cf-ipcountry, content-length, sentry-trace) — no new geo lookup, no new crypto on the hot path. ua_hash null unless USAGE_UA_PEPPER is set. Dev-only x-usage-telemetry response header (ok | degraded | off) attached on the response paths for 2-second debugging in non-production. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(usage): upstream events via implicit request scope (#3381) Closes the upstream-attribution side of the Axiom usage-telemetry stack without requiring leaf-handler changes (per koala's review). server/_shared/usage.ts - AsyncLocalStorage-backed UsageScope: gateway sets it once per request, fetch helpers read from it lazily. Defensive import — if the runtime rejects node:async_hooks, scope helpers degrade to no-ops and the request event is unaffected. - runWithUsageScope(scope, fn) / getUsageScope() exports. server/gateway.ts - Wraps matchedHandler in runWithUsageScope({ ctx, requestId, customerId, route, tier }) so deep fetchers can attribute upstream calls without threading state through every handler signature. server/_shared/redis.ts - cachedFetchJsonWithMeta accepts opts.usage = { provider, operation? }. Only the provider label is required to opt in — request_id / customer_id / route / tier flow implicitly from UsageScope. - Emits on the fresh path only (cache hits don't emit; the inbound request event already records cache_status). - cache_status correctly distinguishes 'miss' vs 'neg-sentinel' by construction, matching NEG_SENTINEL handling. - Telemetry never throws — failures are swallowed in the lazy-import catch, sink itself short-circuits on USAGE_TELEMETRY=0. server/_shared/fetch-json.ts - New optional { provider, operation } in FetchJsonOptions. Same opt-in-by-provider model as cachedFetchJsonWithMeta. Auto-derives host from URL. Reads body via .text() so response_bytes is recorded (best-effort; chunked responses still report 0). Net result: any handler that uses fetchJson or cachedFetchJsonWithMeta gets full per-customer upstream attribution by adding two fields to the options bag. No signature changes anywhere else. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(gateway): address round-1 codex feedback on usage telemetry - ctx is now optional on the createDomainGateway handler signature so direct callers (tests, non-Vercel paths) no longer crash on emit - legacy premium bearer-token routes (resilience, shipping-v2) propagate session.userId into the usage accumulator so successful requests are attributed instead of emitting as anon - after checkEntitlement allows a tier-gated route, re-read entitlements (Redis-cached + in-flight coalesced) to populate usage.tier so analyze-stock & co. emit the correct tier rather than 0 - domain extraction now skips a leading vN segment, so /api/v2/shipping/* records domain="shipping" instead of "v2" Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(usage): assert telemetry payload + identity resolver + operator guide - tests/usage-telemetry-emission.test.mts stubs globalThis.fetch to capture the Axiom ingest POST body and asserts the four review-flagged fields end-to-end through the gateway: domain on /api/v2/<svc>/* (was "v2"), customer_id on legacy premium bearer success (was null/anon), tier on entitlement-gated success via the Convex fallback path (was 0), plus a ctx-optional regression guard - server/__tests__/usage-identity.test.ts unit-tests the pure buildUsageIdentity() resolver across every auth_kind branch, tier coercion, and the secret-handling invariant (raw enterprise key never lands in any output field) - docs/architecture/usage-telemetry.md is the operator + dev guide: field reference, architecture, configuration, failure modes, local workflow, eight Axiom APL recipes, and runbooks for adding fields / new gateway return paths Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(usage): make recorder.settled robust to nested waitUntil Promise.all(pending) snapshotted the array at call time, missing the inner ctx.waitUntil(sendToAxiom(...)) that emitUsageEvents pushes after the outer drain begins. Tests passed only because the fetch spy resolved in an earlier microtask tick. Replace with a quiescence loop so the helper survives any future async in the emit path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: trigger preview * fix(usage): address koala #3403 review — collapse nested waitUntil, widget-key validation, neg-sentinel status, auth_* reasons P1 - Collapse nested ctx.waitUntil at all 3 emit sites (gateway.ts emitRequest, fetch-json.ts, redis.ts emitUpstreamFromHook). Export sendToAxiom and call it directly inside the outer waitUntil so Edge runtimes don't drop the delivery promise after the response phase. - Validate X-Widget-Key against WIDGET_AGENT_KEY before populating usage.widgetKey so unauthenticated callers can't spoof per-customer attribution. P2 - Emit on OPTIONS preflight (new 'preflight' RequestReason). - Gate cachedFetchJsonWithMeta upstreamStatus=200 on result != null so the neg-sentinel branch no longer reports as a successful upstream call. - Extend RequestReason with auth_401/auth_403/tier_403 and replace reason:'ok' on every auth/tier-rejection emit path. - Replace 32-bit FNV-1a with a two-round XOR-folded 64-bit variant in hashKeySync (collision space matters once widget-key adoption grows). Verification - tests/usage-telemetry-emission.test.mts — 6/6 - tests/premium-stock-gateway.test.mts + tests/gateway-cdn-origin-policy.test.mts — 15/15 - npx vitest run server/__tests__/usage-identity.test.ts — 13/13 - npx tsc --noEmit clean Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: trigger preview rebuild for AXIOM_API_TOKEN * chore(usage): note Axiom region in ingest URL comment Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * debug(usage): unconditional logs in sendToAxiom for preview troubleshooting Temporary — to be reverted once Axiom delivery is confirmed working in preview. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(usage): add 'live' cache tier + revert preview debug logs - Sync UsageCacheTier with the local CacheTier in gateway.ts (main added 'live' in PR #3402 — synthetic merge with main was failing typecheck:api). - Revert temporary unconditional debug logs in sendToAxiom now that Axiom delivery is verified end-to-end on preview (event landed with all fields populated, including the new auth_401 reason from the koala #3403 fix). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Elie Habib
|
5c955691a9 |
feat(energy-atlas): live tanker map layer + contract (parity PR 3, plan U7-U8) (#3402)
* feat(energy-atlas): live tanker map layer + contract (PR 3, plan U7-U8)
Lands the third and final parity-push surface — per-vessel tanker positions
inside chokepoint bounding boxes, refreshed every 60s. Closes the visual
gap with peer reference energy-intel sites for the live AIS tanker view.
Per docs/plans/2026-04-25-003-feat-energy-parity-pushup-plan.md PR 3.
Codex-approved through 8 review rounds against origin/main @
|
||
|
Elie Habib
|
84ee2beb3e |
feat(energy): Energy Atlas end-to-end — pipelines + storage + shortages + disruptions + country drill-down (#3294)
* feat(energy): pipeline registries (gas + oil) — evidence-based schema
Day 6 of the Energy Atlas Release 1 plan (Week 2). First curated asset
registry for the atlas — the real gap vs GEF.
## Curated data (critical assets only, not global completeness)
scripts/data/pipelines-gas.json — 12 critical gas lines:
Nord Stream 1/2 (offline; Swedish EEZ sabotage 2022; EU sanctions refs),
TurkStream, Yamal–Europe (offline; Polish counter-sanctions),
Brotherhood/Soyuz (offline; Ukraine transit expired 2024-12-31),
Power of Siberia, Dolphin, Medgaz, TAP, TANAP,
Central Asia–China, Langeled.
scripts/data/pipelines-oil.json — 12 critical oil lines:
Druzhba North/South (N offline per EU 2022/879; S under landlocked
derogation), CPC, ESPO (+ price-cap sanction ref), BTC, TAPS,
Habshan–Fujairah (Hormuz bypass), Keystone, Kirkuk–Ceyhan (offline
since 2023 ICC ruling), Baku–Supsa, Trans-Mountain (TMX expansion
May 2024), ESPO spur to Daqing.
Scope note: 75+ each is Week 2b work via GEM bulk import. Today's cut
is curated from first-hand operator disclosures + regulator filings so
I can stand behind every evidence field.
## Evidence-based schema (not conclusion labels)
Per docs/methodology/pipelines.mdx: no bare `sanctions_blocked` field.
Every pipeline carries an evidence bundle with `physicalState`,
`physicalStateSource`, `operatorStatement`, `commercialState`,
`sanctionRefs[]`, `lastEvidenceUpdate`, `classifierVersion`,
`classifierConfidence`. The public badge (`flowing|reduced|offline|
disputed`) is derived server-side from this bundle at read time.
## Seeder
scripts/seed-pipelines.mjs — single process publishes BOTH keys
(energy:pipelines:{gas,oil}:v1) via two runSeed() calls. Tiny datasets
(<20KB each) so co-location is cheap and guarantees classifierVersion
consistency.
Conventions followed (worldmonitor-bootstrap-registration skill):
- TTL 21d = 3× weekly cadence (gold-standard per
feedback_seeder_gold_standard.md)
- maxStaleMin 20_160 = 2× cadence (health-maxstalemin-write-cadence skill)
- sourceVersion + schemaVersion + recordCount + declareRecords wired
(seed-contract-foundation)
- Zero-case explicitly NOT allowed — MIN_PIPELINES_PER_REGISTRY=8 floor
## Health registration (dual, per feedback_two_health_endpoints_must_match)
- api/health.js: BOOTSTRAP_KEYS adds pipelinesGas + pipelinesOil;
SEED_META adds both with maxStaleMin=20_160.
- api/seed-health.js: mirror entries with intervalMin=10_080 (maxStaleMin/2).
## Bundle registration
scripts/seed-bundle-energy-sources.mjs adds a single Pipelines entry
(not two) because seed-pipelines.mjs publishes both keys in one run —
listing oil separately would double-execute. Monitoring of the oil key
staleness happens in api/health.js instead.
## Tests (tests/pipelines-registry.test.mts)
17 passing node:test assertions covering:
- Schema validation (both registries pass validateRegistry)
- Identity resolution (no id collisions, id matches object key)
- Country ISO2 normalization (from/to/transit all match /^[A-Z]{2}$/)
- Endpoint geometry within Earth bounds
- Evidence rigor: non-flowing badges require at least one supporting
evidence source (operator statement / sanctionRefs / ais-relay /
satellite / press)
- ClassifierConfidence in 0..1
- Commodity/capacity pairing (gas uses capacityBcmYr, oil uses
capacityMbd — mixing = test fail)
- validateRegistry rejects: empty object, null, no-evidence fixtures,
below-floor counts
Typecheck clean (both tsconfig.json and tsconfig.api.json).
Next: Day 7 will add list-pipelines / get-pipeline-detail RPCs in
supply-chain/v1. Day 8 ships PipelineStatusPanel with DeckGL PathLayer
consuming the registry.
* fix(energy): split seed-pipelines.mjs into two entry points — runSeed hard-exits
High finding from PR review. scripts/seed-pipelines.mjs called runSeed()
twice in one process and awaited Promise.all. But runSeed() in
scripts/_seed-utils.mjs hard-exits via process.exit on ~9 terminal paths
(lines 816, 820, 839, 888, 917, 989, plus fetch-retry 946, fatal 859,
skipped-lock 81). The first runSeed to reach any terminal path exits the
entire node process, so the second runSeed's resolve never fires — only
one of energy:pipelines:{gas,oil}:v1 would ever be written.
Since the bundle scheduled seed-pipelines.mjs exactly once, and both
api/health.js and api/seed-health.js expect both keys populated, the
other registry would stay permanently EMPTY/STALE after deploy.
Fix: split into two entry-point scripts around a shared utility.
- scripts/_pipeline-registry.mjs (NEW, was seed-pipelines.mjs) — shared
helpers ONLY. Exports GAS_CANONICAL_KEY, OIL_CANONICAL_KEY,
PIPELINES_TTL_SECONDS, MAX_STALE_MIN, buildGasPayload, buildOilPayload,
validateRegistry, recordCount, declareRecords. Underscore prefix marks
it as non-entry-point (matches _seed-utils.mjs / _seed-envelope-source.mjs
convention).
- scripts/seed-pipelines-gas.mjs (NEW) — imports from the shared module,
single runSeed('energy','pipelines-gas',…) call.
- scripts/seed-pipelines-oil.mjs (NEW) — same shape, oil.
- scripts/seed-bundle-energy-sources.mjs — register BOTH seeders (not one).
- scripts/seed-pipelines.mjs — deleted.
- tests/pipelines-registry.test.mts — update import path to the shared
module. All 17 tests still pass.
Typecheck clean (both configs). Tests pass. No other consumers import
from the deleted script.
* fix(energy): complete pipeline bootstrap registration per 4-file checklist
High finding from PR review. My earlier PR description claimed
worldmonitor-bootstrap-registration was complete, but I only touched two
of the four registries (api/health.js + api/seed-health.js). The bootstrap
hydration payload itself (api/bootstrap.js) and the shared cache-keys
registry (server/_shared/cache-keys.ts) still had no entry for either
pipeline key, so any consumer that reads bootstrap data would see
pipelinesGas/pipelinesOil as missing on first load.
Files updated this commit:
- api/bootstrap.js — KEYS map + SLOW_KEYS set both gain pipelinesGas +
pipelinesOil. Placed next to sprPolicies (same curated-registry cadence
and tier). Slow tier is correct: weekly cron, not needed on first paint.
- server/_shared/cache-keys.ts — PIPELINES_GAS_KEY + PIPELINES_OIL_KEY
exported constants (matches SPR_POLICIES_KEY pattern), BOOTSTRAP_KEYS map
entries, and BOOTSTRAP_TIERS entries (both 'slow').
Not touched (intentional):
- server/gateway.ts — pipeline data is free-tier per the Energy Atlas
plan; no PREMIUM_RPC_PATHS entry required. Energy Atlas monetization
hooks (scenario runner, MCP tools, subscriptions) are Release 2.
Full 4-file checklist now complete:
✅ server/_shared/cache-keys.ts (this commit)
✅ api/bootstrap.js (this commit)
✅ api/health.js (earlier in PR)
✅ api/seed-health.js (earlier in PR — dual-registry rule)
Typecheck clean (both configs).
* feat(energy): ListPipelines + GetPipelineDetail RPCs with evidence-derived badges
Day 7 of the Energy Atlas Release 1 plan (Week 2). Exposes the pipeline
registries (shipped in Day 6) via two supply-chain RPCs and ships the
evidence-to-badge derivation server-side.
## Proto
proto/worldmonitor/supply_chain/v1/list_pipelines.proto — new:
- ListPipelinesRequest { commodity_type?: 'gas' | 'oil' }
- ListPipelinesResponse { pipelines[], fetched_at, classifier_version, upstream_unavailable }
- GetPipelineDetailRequest { pipeline_id (required, query-param) }
- GetPipelineDetailResponse { pipeline?, revisions[], fetched_at, unavailable }
- PipelineEntry — wire shape mirroring scripts/data/pipelines-{gas,oil}.json
+ a server-derived public_badge field
- PipelineEvidence, OperatorStatement, SanctionRef, LatLon, PipelineRevisionEntry
service.proto adds both rpc methods with HTTP_METHOD_GET + path bindings:
/api/supply-chain/v1/list-pipelines
/api/supply-chain/v1/get-pipeline-detail
`make generate` regenerated src/generated/{client,server}/… + docs/api/
OpenAPI json/yaml.
## Evidence-derivation
server/worldmonitor/supply-chain/v1/_pipeline-evidence.ts — new.
derivePublicBadge(evidence) → 'flowing' | 'reduced' | 'offline' | 'disputed'
is deterministic + versioned (DERIVER_VERSION='badge-deriver-v1').
Rules (first match wins):
1. offline + sanctionRef OR expired/suspended commercial → offline
2. offline + operator statement → offline
3. offline + only press/ais/satellite → disputed (single-source negative claim)
4. reduced → reduced
5. flowing → flowing
6. unknown / malformed → disputed
Staleness guard: non-flowing badges on >14d-old evidence demote to
disputed. Flowing is the optimistic default — stale "still flowing" is
safer than stale "offline". Matches seed-pipelines-{gas,oil}.mjs maxStaleMin.
Tests (tests/pipeline-evidence-derivation.test.mts) — 15 passing cases
covering happy paths, disputed fallbacks, staleness guard, versioning.
## Handlers
server/worldmonitor/supply-chain/v1/list-pipelines.ts
- Reads energy:pipelines:{gas,oil}:v1 via getCachedJson.
- projectPipeline() narrows the Upstash `unknown` into PipelineEntry
shape + calls derivePublicBadge.
- Honors commodity_type filter (skip the opposite registry's Redis read
when the client pre-filters).
- Returns upstream_unavailable=true when BOTH registries miss.
server/worldmonitor/supply-chain/v1/get-pipeline-detail.ts
- Scans both registries by id (ids are globally unique per
tests/pipelines-registry.test.mts).
- Empty revisions[] for now; auto-revision log wires up in Week 3.
handler.ts registers both into supplyChainHandler.
## Gateway
server/gateway.ts adds 'static' cache-tier for both new RPC paths
(registry is slow-moving; 'static' matches the other read-mostly
supply-chain endpoints).
## Consumer wiring
Not in this commit — PipelineStatusPanel (Day 8) is what will call
listPipelines/getPipelineDetail via the generated client. pipelinesGas
+ pipelinesOil stay in PENDING_CONSUMERS until Day 8.
Typecheck clean (both configs). 15 new tests + 17 registry tests all pass.
* feat(energy): PipelineStatusPanel — evidence-backed status table + drawer
Day 8 of the Energy Atlas Release 1 plan. First consumer of the Day 6–7
registries + RPCs.
## What this PR adds
- src/components/PipelineStatusPanel.ts — new panel (id=pipeline-status).
* Bootstrap-hydrates from pipelinesGas + pipelinesOil for instant first
paint; falls through to listPipelines() RPC if bootstrap misses.
Background re-fetch runs on every render so a classifier-version bump
between bootstrap stamp and first view produces a visible update.
* Table rows sorted non-flowing-first (offline / reduced / disputed
before flowing) — what an atlas reader cares about.
* Click-to-expand drawer calls getPipelineDetail() lazily — operator
statements, sanction refs (with clickable source URLs), commercial
state, classifier version + confidence %, capacity + route metadata.
* publicBadge color-chip palette matches the methodology doc.
* Attribution footer with GEM (CC-BY 4.0) credit + classifier version.
- src/components/index.ts — barrel export.
- src/app/panel-layout.ts — import + createPanel('pipeline-status', …).
- src/config/panels.ts — ENERGY_PANELS adds 'pipeline-status' at priority 1.
## PENDING_CONSUMERS cleanup
tests/bootstrap.test.mjs — removes 'pipelinesGas' + 'pipelinesOil' from
the allowlist. The invariant "every bootstrap key has a getHydratedData
consumer" now enforces real wiring for these keys: the panel literally
calls getHydratedData('pipelinesGas') and getHydratedData('pipelinesOil').
Future regressions that remove the consumer will fail pre-push.
## Consumer contract verified
- 67 tests pass including bootstrap.test.mjs consumer coverage check.
- Typecheck clean.
- No DeckGL PathLayer in this commit — existing 'pipelines-layer' has a
separate data source, so modifying DeckGLMap.ts to overlay evidence-
derived badges on the map is a follow-up commit to avoid clobbering.
## Out of scope for Day 8 (next steps on same PR)
- DeckGL PathLayer integration (color pipelines on the main map by
publicBadge, click-to-open this drawer) — Day 8b commit.
- Storage facility registry + StorageFacilityMapPanel — Days 9-10.
* fix(energy): PipelineStatusPanel bootstrap path — client-side badge derivation
High finding from PR review. The Day-8 panel crashed on first paint
whenever bootstrap hydration succeeded, because:
- Bootstrap hydrates raw scripts/data/pipelines-{gas,oil}.json verbatim.
- That JSON does NOT include publicBadge — that field is only added by
the server handler's projectPipeline() in list-pipelines.ts.
- PipelineStatusPanel passed raw entries into badgeChip(), which called
badgeLabel(undefined).charAt(0) → TypeError.
The background RPC refresh that would have repaired the data never ran
because the panel threw before reaching it. So the exact bootstrap path
newly wired in commit
|
||
|
Sebastien Melki
|
58e42aadf9 |
chore(api): enforce sebuf contract + migrate drifting endpoints (#3207) (#3242)
* chore(api): enforce sebuf contract via exceptions manifest (#3207) Adds api/api-route-exceptions.json as the single source of truth for non-proto /api/ endpoints, with scripts/enforce-sebuf-api-contract.mjs gating every PR via npm run lint:api-contract. Fixes the root-only blind spot in the prior allowlist (tests/edge-functions.test.mjs), which only scanned top-level *.js files and missed nested paths and .ts endpoints — the gap that let api/supply-chain/v1/country-products.ts and friends drift under proto domain URL prefixes unchallenged. Checks both directions: every api/<domain>/v<N>/[rpc].ts must pair with a generated service_server.ts (so a deleted proto fails CI), and every generated service must have an HTTP gateway (no orphaned generated code). Manifest entries require category + reason + owner, with removal_issue mandatory for temporary categories (deferred, migration-pending) and forbidden for permanent ones. .github/CODEOWNERS pins the manifest to @SebastienMelki so new exceptions don't slip through review. The manifest only shrinks: migration-pending entries (19 today) will be removed as subsequent commits in this PR land each migration. * refactor(maritime): migrate /api/ais-snapshot → maritime/v1.GetVesselSnapshot (#3207) The proto VesselSnapshot was carrying density + disruptions but the frontend also needed sequence, relay status, and candidate_reports to drive the position-callback system. Those only lived on the raw relay passthrough, so the client had to keep hitting /api/ais-snapshot whenever callbacks were registered and fall back to the proto RPC only when the relay URL was gone. This commit pushes all three missing fields through the proto contract and collapses the dual-fetch-path into one proto client call. Proto changes (proto/worldmonitor/maritime/v1/): - VesselSnapshot gains sequence, status, candidate_reports. - GetVesselSnapshotRequest gains include_candidates (query: include_candidates). Handler (server/worldmonitor/maritime/v1/get-vessel-snapshot.ts): - Forwards include_candidates to ?candidates=... on the relay. - Separate 5-min in-memory caches for the candidates=on and candidates=off variants; they have very different payload sizes and should not share a slot. - Per-request in-flight dedup preserved per-variant. Frontend (src/services/maritime/index.ts): - fetchSnapshotPayload now calls MaritimeServiceClient.getVesselSnapshot directly with includeCandidates threaded through. The raw-relay path, SNAPSHOT_PROXY_URL, DIRECT_RAILWAY_SNAPSHOT_URL and LOCAL_SNAPSHOT_FALLBACK are gone — production already routed via Vercel, the "direct" branch only ever fired on localhost, and the proto gateway covers both. - New toLegacyCandidateReport helper mirrors toDensityZone/toDisruptionEvent. api/ais-snapshot.js deleted; manifest entry removed. Only reduced the codegen scope to worldmonitor.maritime.v1 (buf generate --path) — regenerating the full tree drops // @ts-nocheck from every client/server file and surfaces pre-existing type errors across 30+ unrelated services, which is not in scope for this PR. Shape-diff vs legacy payload: - disruptions / density: proto carries the same fields, just with the GeoCoordinates wrapper and enum strings (remapped client-side via existing toDisruptionEvent / toDensityZone helpers). - sequence, status.{connected,vessels,messages}: now populated from the proto response — was hardcoded to 0/false in the prior proto fallback. - candidateReports: same shape; optional numeric fields come through as 0 instead of undefined, which the legacy consumer already handled. * refactor(sanctions): migrate /api/sanctions-entity-search → LookupSanctionEntity (#3207) The proto docstring already claimed "OFAC + OpenSanctions" coverage but the handler only fuzzy-matched a local OFAC Redis index — narrower than the legacy /api/sanctions-entity-search, which proxied OpenSanctions live (the source advertised in docs/api-proxies.mdx). Deleting the legacy without expanding the handler would have been a silent coverage regression for external consumers. Handler changes (server/worldmonitor/sanctions/v1/lookup-entity.ts): - Primary path: live search against api.opensanctions.org/search/default with an 8s timeout and the same User-Agent the legacy edge fn used. - Fallback path: the existing OFAC local fuzzy match, kept intact for when OpenSanctions is unreachable / rate-limiting. - Response source field flips between 'opensanctions' (happy path) and 'ofac' (fallback) so clients can tell which index answered. - Query validation tightened: rejects q > 200 chars (matches legacy cap). Rate limiting: - Added /api/sanctions/v1/lookup-entity to ENDPOINT_RATE_POLICIES at 30/min per IP — matches the legacy createIpRateLimiter budget. The gateway already enforces per-endpoint policies via checkEndpointRateLimit. Docs: - docs/api-proxies.mdx — dropped the /api/sanctions-entity-search row (plus the orphaned /api/ais-snapshot row left over from the previous commit in this PR). - docs/panels/sanctions-pressure.mdx — points at the new RPC URL and describes the OpenSanctions-primary / OFAC-fallback semantics. api/sanctions-entity-search.js deleted; manifest entry removed. * refactor(military): migrate /api/military-flights → ListMilitaryFlights (#3207) Legacy /api/military-flights read a pre-baked Redis blob written by the seed-military-flights cron and returned flights in a flat app-friendly shape (lat/lon, lowercase enums, lastSeenMs). The proto RPC takes a bbox, fetches OpenSky live, classifies server-side, and returns nested GeoCoordinates + MILITARY_*_TYPE_* enum strings + lastSeenAt — same data, different contract. fetchFromRedis in src/services/military-flights.ts was doing nothing sebuf-aware. Renamed it to fetchViaProto and rewrote to: - Instantiate MilitaryServiceClient against getRpcBaseUrl(). - Iterate MILITARY_QUERY_REGIONS (PACIFIC + WESTERN) in parallel — same regions the desktop OpenSky path and the seed cron already use, so dashboard coverage tracks the analytic pipeline. - Dedup by hexCode across regions. - Map proto → app shape via new mapProtoFlight helper plus three reverse enum maps (AIRCRAFT_TYPE_REVERSE, OPERATOR_REVERSE, CONFIDENCE_REVERSE). The seed cron (scripts/seed-military-flights.mjs) stays put: it feeds regional-snapshot mobility, cross-source signals, correlation, and the health freshness check (api/health.js: 'military:flights:v1'). None of those read the legacy HTTP endpoint; they read the Redis key directly. The proto handler uses its own per-bbox cache keys under the same prefix, so dashboard traffic no longer races the seed cron's blob — the two paths diverge by a small refresh lag, which is acceptable. Docs: dropped the /api/military-flights row from docs/api-proxies.mdx. api/military-flights.js deleted; manifest entry removed. Shape-diff vs legacy: - f.location.{latitude,longitude} → f.lat, f.lon - f.aircraftType: MILITARY_AIRCRAFT_TYPE_TANKER → 'tanker' via reverse map - f.operator: MILITARY_OPERATOR_USAF → 'usaf' via reverse map - f.confidence: MILITARY_CONFIDENCE_LOW → 'low' via reverse map - f.lastSeenAt (number) → f.lastSeen (Date) - f.enrichment → f.enriched (with field renames) - Extra fields registration / aircraftModel / origin / destination / firstSeenAt now flow through where proto populates them. * fix(supply-chain): thread includeCandidates through chokepoint status (#3207) Caught by tsconfig.api.json typecheck in the pre-push hook (not covered by the plain tsc --noEmit run that ran before I pushed the ais-snapshot commit). The chokepoint status handler calls getVesselSnapshot internally with a static no-auth request — now required to include the new includeCandidates bool from the proto extension. Passing false: server-internal callers don't need per-vessel reports. * test(maritime): update getVesselSnapshot cache assertions (#3207) The ais-snapshot migration replaced the single cachedSnapshot/cacheTimestamp pair with a per-variant cache so candidates-on and candidates-off payloads don't evict each other. Pre-push hook surfaced that tests/server-handlers still asserted the old variable names. Rewriting the assertions to match the new shape while preserving the invariants they actually guard: - Freshness check against slot TTL. - Cache read before relay call. - Per-slot in-flight dedup. - Stale-serve on relay failure (result ?? slot.snapshot). * chore(proto): restore // @ts-nocheck on regenerated maritime files (#3207) I ran 'buf generate --path worldmonitor/maritime/v1' to scope the proto regen to the one service I was changing (to avoid the toolchain drift that drops @ts-nocheck from 60+ unrelated files — separate issue). But the repo convention is the 'make generate' target, which runs buf and then sed-prepends '// @ts-nocheck' to every generated .ts file. My scoped command skipped the sed step. The proto-check CI enforces the sed output, so the two maritime files need the directive restored. * refactor(enrichment): decomm /api/enrichment/{company,signals} legacy edge fns (#3207) Both endpoints were already ported to IntelligenceService: - getCompanyEnrichment (/api/intelligence/v1/get-company-enrichment) - listCompanySignals (/api/intelligence/v1/list-company-signals) No frontend callers of the legacy /api/enrichment/* paths exist. Removes: - api/enrichment/company.js, signals.js, _domain.js - api-route-exceptions.json migration-pending entries (58 remain) - docs/api-proxies.mdx rows for /api/enrichment/{company,signals} - docs/architecture.mdx reference updated to the IntelligenceService RPCs Verified: typecheck, typecheck:api, lint:api-contract (89 files / 58 entries), lint:boundaries, tests/edge-functions.test.mjs (136 pass), tests/enrichment-caching.test.mjs (14 pass — still guards the intelligence/v1 handlers), make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(leads): migrate /api/{contact,register-interest} → LeadsService (#3207) New leads/v1 sebuf service with two POST RPCs: - SubmitContact → /api/leads/v1/submit-contact - RegisterInterest → /api/leads/v1/register-interest Handler logic ported 1:1 from api/contact.js + api/register-interest.js: - Turnstile verification (desktop sources bypass, preserved) - Honeypot (website field) silently accepts without upstream calls - Free-email-domain gate on SubmitContact (422 ApiError) - validateEmail (disposable/offensive/typo-TLD/MX) on RegisterInterest - Convex writes via ConvexHttpClient (contactMessages:submit, registerInterest:register) - Resend notification + confirmation emails (HTML templates unchanged) Shared helpers moved to server/_shared/: - turnstile.ts (getClientIp + verifyTurnstile) - email-validation.ts (disposable/offensive/MX checks) Rate limits preserved via ENDPOINT_RATE_POLICIES: - submit-contact: 3/hour per IP (was in-memory 3/hr) - register-interest: 5/hour per IP (was in-memory 5/hr; desktop sources previously capped at 2/hr via shared in-memory map — now 5/hr like everyone else, accepting the small regression in exchange for Upstash-backed global limiting) Callers updated: - pro-test/src/App.tsx contact form → new submit-contact path - src-tauri/sidecar/local-api-server.mjs cloud-fallback rewrites /api/register-interest → /api/leads/v1/register-interest when proxying; keeps local path for older desktop builds - src/services/runtime.ts isKeyFreeApiTarget allows both old and new paths through the WORLDMONITOR_API_KEY-optional gate Tests: - tests/contact-handler.test.mjs rewritten to call submitContact handler directly; asserts on ValidationError / ApiError - tests/email-validation.test.mjs + tests/turnstile.test.mjs point at the new server/_shared/ modules Deleted: api/contact.js, api/register-interest.js, api/_ip-rate-limit.js, api/_turnstile.js, api/_email-validation.js, api/_turnstile.test.mjs. Manifest entries removed (58 → 56). Docs updated (api-platform, api-commerce, usage-rate-limits). Verified: npm run typecheck + typecheck:api + lint:api-contract (88 files / 56 entries) + lint:boundaries pass; full test:data (5852 tests) passes; make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(pro-test): rebuild bundle for leads/v1 contact form (#3207) Updates the enterprise contact form to POST to /api/leads/v1/submit-contact (old path /api/contact removed in the previous commit). Bundle is rebuilt from pro-test/src/App.tsx source change in |
||
|
Elie Habib
|
6977e9d0fe |
fix(gateway): accept Dodo entitlement as pro, not just Clerk role — unblocks paying users (#3249)
* fix(gateway): accept Dodo entitlement as pro, not just Clerk role
The gateway's legacy premium-paths gate (lines 388-401) was rejecting
authenticated Bearer users with 403 "Pro subscription required"
whenever session.role !== 'pro' — which is EVERY paying Dodo
subscriber, because the Dodo webhook pipeline writes Convex
entitlements and does NOT sync Clerk publicMetadata.role.
So the flow was:
- User pays, Dodo webhook fires, Convex entitlement tier=1 written
- User loads the dashboard, Clerk token includes Bearer but role='free'
- Gateway sees role!=='pro' → 403 on every intelligence/trade/
economic/sanctions premium endpoint
- User sees a blank dashboard despite having paid
This is the exact split-brain documented at the frontend layer
(src/services/panel-gating.ts:11-27): "The Convex entitlement check
is the authoritative signal for paying customers — Clerk
`publicMetadata.plan` is NOT written by our webhook pipeline". The
frontend was fixed by having hasPremiumAccess() fall through to
isEntitled() from Convex. The backend gateway still had the
Clerk-role-only gate, so paying users got rejected even though
their Convex entitlement was active.
Align the gateway gate with the logic already in
server/_shared/premium-check.ts::isCallerPremium (line 44-49):
1. If Clerk role === 'pro' → allow (fast path, no Redis/Convex I/O)
2. Else if session.userId → look up Convex entitlement; allow if
tier >= 1 AND validUntil >= Date.now() (covers lapsed subs)
3. Else → 403
Same two-signal semantics as the per-handler isCallerPremium, so
the gateway and handlers can't disagree on who is premium. Uses
the already-imported getEntitlements function (line 345 already
imports it dynamically; promoting to top-level import since the new
site is in a hotter path).
Impact: unblocks all Dodo subscribers whose Clerk role is still
'free' — the common case after any fresh Pro purchase and for
every user since webhook-based role sync was never wired up.
Reported 2026-04-21 post-purchase flow: user completed Dodo payment,
landed back on dashboard, saw 403s on get-regional-snapshot,
get-tariff-trends, list-comtrade-flows, get-national-debt,
deduct-situation — all 5 are in PREMIUM_RPC_PATHS but not in
ENDPOINT_ENTITLEMENTS, so they hit this legacy gate.
* fix(gateway): move entitlement fallback to the gate that actually fires
Reviewer caught that the previous iteration of this fix put the
entitlement fallback at line ~400, inside an `if (sessionUserId &&
!keyCheck.valid && needsLegacyProBearerGate)` branch that's
unreachable for the case the PR was supposed to fix:
- sessionUserId is only resolved when isTierGated is true (line 292)
— JWKS lookup is intentionally skipped for non-tier-gated paths.
- needsLegacyProBearerGate IS the non-tier-gated set
(PREMIUM_RPC_PATHS && !isTierGated).
- So sessionUserId is null, the branch never enters, and the actual
legacy-Bearer rejection still happens earlier at line 367 inside
the `keyCheck.required && !keyCheck.valid` branch.
Move the entitlement fallback INTO the line-367 check, where the
Bearer is already being validated and `session.userId` is already
exposed on the validateBearerToken() result. No extra JWKS round-trip
needed (validateBearerToken already verified the JWT). The previously-
added line-400 block is removed since it never ran.
Now for a paying Dodo subscriber whose Clerk role is still 'free':
- Bearer validates → role !== 'pro'
- Fall through: getEntitlements(session.userId) → tier=1, validUntil future
- allowed = true, request proceeds to handler
Same fail-closed semantics as before for the negative cases:
- Anonymous → no Bearer → 401
- Bearer with invalid JWT → 401
- Free user with no Dodo entitlement → 403
- Pro user whose Dodo subscription lapsed (validUntil < now) → 403
* chore(gateway): drop redundant dynamic getEntitlements import
Greptile spotted that the previous commit promoted getEntitlements to
a top-level import for the new line-385 fallback site, but the older
dynamic import at line 345 (in the user-API-key entitlement check
branch) was left in place. Same module, same symbol, so the dynamic
import is now dead weight that just adds a microtask boundary to the
hot path.
Drop it; line 345's `getEntitlements(sessionUserId)` call now resolves
through the top-level import like the line-385 site already does.
|
||
|
Elie Habib
|
3c47c1b222 |
fix(supply-chain): split chokepoint transit data + close silent zero-state cache (#3185)
* fix(supply-chain): split chokepoint transit data + close silent zero-state cache
Production supply-chain panel was rendering 13 empty chokepoints because
the getChokepointStatus RPC silently cached zero-state for 5 minutes:
1. supply_chain:transit-summaries:v1 grew to ~500 KB (180d × 13 × 14 fields
of history per chokepoint).
2. REDIS_OP_TIMEOUT_MS is 1.5 s. Vercel Sydney edge → Upstash for a 500 KB
GET consistently exceeded the budget; getCachedJson caught the AbortError
and returned null.
3. The 500 KB portwatch fallback read hit the same timeout.
4. summaries = {} → every summaries[cp.id] was undefined → 13 chokepoints
got the zero-state default → cached as a non-null success response for
REDIS_CACHE_TTL (5 min) instead of NEG_SENTINEL (120 s).
Fix (one PR, per docs/plans/chokepoint-rpc-payload-split.md):
- ais-relay.cjs: split seedTransitSummaries output.
- supply_chain:transit-summaries:v1 — compact (~30 KB, no history).
- supply_chain:transit-summaries:history:v1:{id} — per chokepoint
(~35 KB each, 13 keys). Both under the 1.5 s Redis read budget.
- New RPC GetChokepointHistory: lazy-loaded on card expand.
- get-chokepoint-status.ts: drop the 500 KB portwatch/corridorrisk/
chokepoint_transits fallback reads. Treat a null transit-summaries
read as upstreamUnavailable=true so cachedFetchJson writes NEG_SENTINEL
(2 min) instead of a 5-min zero-state pin. Omit history from the
response (proto field stays declared; empty array).
- server/_shared/redis.ts: tag AbortError timeouts with [REDIS-TIMEOUT]
key=… timeoutMs=… so log drains / Sentry-Vercel integration pick up
large-payload timeouts instead of them being silently swallowed.
- SupplyChainPanel.ts + MapPopup.ts: lazy-fetch history on card expand
via fetchChokepointHistory; session-scoped cache; graceful "History
unavailable" on empty/error. PRO gating on the map popup unchanged.
- Gateway: cache-tier entry for /get-chokepoint-history (slow).
- Tests: regression guards for upstreamUnavailable gate + per-id key
shape + handler wiring + proto query annotations.
Audit included in plan: no other RPC consumer read stacks >200 KB
besides displacement:summary:v1:2026 (724 KB, same risk, flagged for
follow-up PR). wildfire:fires:v1 at 1.7 MB loads via bootstrap (3 s
timeout, different path) — monitor but out of scope.
Expected impact:
- supply_chain:chokepoints:v4 payload drops from ~508 KB to <100 KB.
- supply_chain:transit-summaries:v1 drops from ~502 KB to <50 KB.
- RPC Redis reads stay well under 1.5 s in the hot path.
- Silent zero-state pinning is now impossible: null reads → 2-min neg
cache → self-heal on next relay tick.
* fix(supply-chain): address PR #3185 review — stop caching empty/error + fix partial coverage
Two P1 regressions caught in review:
1. Client cache poisoning on empty/error (MapPopup.ts, SupplyChainPanel.ts)
Empty-array is truthy in JS, so MapPopup's `!cached && !inflight` branch
never fired once we cached []. Neither `cached && cached.length` fired
either — popup stuck on "Loading transit history..." for the session.
SupplyChainPanel had the explicit `cached && !cached.length` branch but
still never retried, so the same transient became session-sticky there too.
Fix: cache ONLY non-empty successful responses. Empty/error show the
"History unavailable" placeholder but leave the cache untouched, so the
next re-expand retries. The /get-chokepoint-history gateway tier is
"slow" (5-min CF edge cache) → retries stay cheap.
2. Partial portwatch coverage treated as healthy (ais-relay.cjs)
seedTransitSummaries iterated Object.entries(pw), so if seed-portwatch
dropped N of 13 chokepoints (ArcGIS reject/empty), summaries had <13 keys.
get-chokepoint-status upstreamUnavailable fires only on fully-empty
summaries, so the N missing chokepoints fell through to zero-state rows
that got pinned in cache for 5 minutes.
Fix: iterate CANONICAL_IDS (Object.keys(CHOKEPOINT_THREAT_LEVELS)) and
fill zero-state for any ID missing from pw. Shape is consistently 13
keys. Track pwCovered → envelope + seed-meta recordCount reflect real
upstream coverage (not shape size), so health.js can distinguish 13/13
healthy from 10/13 partial. Warn-log on shortfall.
Tests: new regression guards
- panel must NOT cache empty arrays (historyCache.set with []).
- writer must iterate CANONICAL_IDS, not Object.entries(pw).
- seed-meta recordCount binds to pwCovered.
5718/5718 data tests pass. typecheck + typecheck:api clean.
|
||
|
Elie Habib
|
de769ce8e1 |
fix(api): unblock Pro API clients at edge + accept x-api-key alias (#3155)
* fix(api): unblock Pro API clients at edge + accept x-api-key alias Fixes #3146: Pro API subscriber getting 403 when calling from Railway. Two independent layers were blocking server-side callers: 1. Vercel Edge Middleware (middleware.ts) blocks any UA matching /bot|curl\/|python-requests|go-http|java\//, which killed every legitimate server-to-server API client before the gateway even saw the request. Add bypass: requests carrying an `x-worldmonitor-key` or `x-api-key` header that starts with `wm_` skip the UA gate. The prefix is a cheap client-side signal, not auth — downstream server/gateway.ts still hashes the key and validates against the Convex `userApiKeys` table + entitlement check. 2. Header name mismatch. Docs/gateway only accepted `X-WorldMonitor-Key`, but most API clients default to `x-api-key`. Accept both header names in: - api/_api-key.js (legacy static-key allowlist) - server/gateway.ts (user-issued Convex-backed keys) - server/_shared/premium-check.ts (isCallerPremium) Add `X-Api-Key` to CORS Allow-Headers in server/cors.ts and api/_cors.js so browser preflights succeed. Follow-up outside this PR (Cloudflare dashboard, not in repo): - Extend the "Allow api access with WM" custom WAF rule to also match `starts_with(http.request.headers["x-api-key"][0], "wm_")`, so CF Managed Rules don't block requests using the x-api-key header name. - Update the api-cors-preflight CF Worker's corsHeaders to include `X-Api-Key` (memory: cors-cloudflare-worker.md — Worker overrides repo CORS on api.worldmonitor.app). * fix(api): tighten middleware bypass shape + finish x-api-key alias coverage Addresses review findings on #3155: 1. middleware.ts bypass was too loose. "Starts with wm_" let any caller send X-Api-Key: wm_fake and skip the UA gate, shifting unauthenticated scraper load onto the gateway's Convex lookup. Tighten to the exact key format emitted by src/services/api-keys.ts:generateKey — `^wm_[a-f0-9]{40}$` (wm_ + 20 random bytes as hex). Still a cheap edge heuristic (no hash lookup in middleware), but raises spoofing from trivial prefix match to a specific 43-char shape. 2. Alias was incomplete on bespoke endpoints outside the shared gateway: - api/v2/shipping/route-intelligence.ts: async wm_ user-key fallback now reads X-Api-Key as well - api/v2/shipping/webhooks.ts: webhook ownership fingerprint now reads X-Api-Key as well (same key value → same SHA-256 → same ownerTag, so a user registering with either header can manage their webhook from the other) - api/widget-agent.ts: accept X-Api-Key in the auth read AND in the OPTIONS Allow-Headers list - api/chat-analyst.ts: add X-Api-Key to the OPTIONS Allow-Headers list (auth path goes through shared helpers already aliased) |
||
|
Sebastien Melki
|
a4d9b0a5fa |
feat(auth): user-facing API key management (create / list / revoke) (#3125)
* feat(auth): user-facing API key management (create / list / revoke) Adds full-stack API key management so authenticated users can create, list, and revoke their own API keys from the Settings UI. Backend: - Convex `userApiKeys` table with SHA-256 key hash storage - Mutations: createApiKey, listApiKeys, revokeApiKey - Internal query validateKeyByHash + touchKeyLastUsed for gateway - HTTP endpoints: /api/api-keys (CRUD) + /api/internal-validate-api-key - Gateway middleware validates user-owned keys via Convex + Redis cache Frontend: - New "API Keys" tab in UnifiedSettings (visible when signed in) - Create form with copy-on-creation banner (key shown once) - List with prefix display, timestamps, and revoke action - Client-side key generation + hashing (plaintext never sent to DB) Closes #3116 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(api-keys): address PR review — cache invalidation, prefix validation, revoked-key guard - Invalidate Redis cache on key revocation so gateway rejects revoked keys immediately instead of waiting for 5-min TTL expiry (P1) - Enforce `wm_` prefix format with regex instead of loose length check (P2) - Skip `touchKeyLastUsed` for revoked keys to preserve clean audit trail (P2) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(api-keys): address consolidated PR review (P0–P3) P0: gate createApiKey on pro entitlement (tier >= 1); isCallerPremium now verifies key-owner tier instead of treating existence as premium. P1: wire wm_ user keys into the domain gateway auth path with async Convex-backed validation; user keys go through entitlement checks (only admin keys bypass). Lower cache TTL 300s → 60s and await revocation cache-bust instead of fire-and-forget. P2: remove dead HTTP create/list/revoke path from convex/http.ts; switch to cachedFetchJson (stampede protection, env-prefixed keys, standard NEG_SENTINEL); add tenancy check on cache-invalidation endpoint via new /api/internal-get-key-owner route; add 22 Convex tests covering tier gate, per-user limit, duplicate hash, ownership revoke guard, getKeyOwner, and touchKeyLastUsed debounce. P3: tighten keyPrefix regex to exactly 5 hex chars; debounce touchKeyLastUsed (5 min); surface PRO_REQUIRED in UI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(api-keys): gate on apiAccess (not tier), wire wm_ keys through edge routes, harden error paths - Gate API key creation/validation on features.apiAccess instead of tier >= 1. Pro (tier 1, apiAccess=false) can no longer mint keys — only API_STARTER+. - Wire wm_ user keys through standalone edge routes (shipping/route-intelligence, shipping/webhooks) that were short-circuiting on validateApiKey before async Convex validation could run. - Restore fail-soft behavior in validateUserApiKey: transient Convex/network errors degrade to unauthorized instead of bubbling a 500. - Fail-closed on cache invalidation endpoint: ownership check errors now return 503 instead of silently proceeding (surfaces Convex outages in logs). - Tests updated: positive paths use api_starter (apiAccess=true), new test locks Pro-without-API-access rejection. 23 tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(webhooks): remove wm_ user key fallback from shipping webhooks Webhook ownership is keyed to SHA-256(apiKey) via callerFingerprint(), not to the user. With user-owned keys (up to 5 per user), this causes cross-key blindness (webhooks invisible when calling with a different key) and revoke-orphaning (revoking the creating key makes the webhook permanently unmanageable). User keys remain supported on the read-only route-intelligence endpoint. Webhook ownership migration to userId will follow in a separate PR. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
e32d9b631c |
feat(market): Hyperliquid perp positioning flow as leading indicator (#3074)
* feat(market): Hyperliquid perp positioning flow as leading indicator Adds a 4-component composite (funding × volume × OI × basis) "positioning stress" score for ~14 perps spanning crypto (BTC/ETH/SOL), tokenized gold (PAXG), commodity perps (WTI, Brent, Gold, Silver, Pt, Pd, Cu, NatGas), and FX perps (EUR, JPY). Polls Hyperliquid /info every 5min via Railway cron; publishes a single self-contained snapshot with embedded sparkline arrays (60 samples = 5h history). Surfaces as a new "Perp Flow" tab in CommoditiesPanel with separate Commodities / FX sections. Why: existing CFTC COT is weekly + US-centric; market quotes are price-only. Hyperliquid xyz: perps give 24/7 global positioning data that has been shown to lead spot moves on commodities and FX by minutes-to-hours. Implementation: - scripts/seed-hyperliquid-flow.mjs — pure scoring math, symbol whitelist, content-type + schema validation, prior-state read via readSeedSnapshot(), warmup contract (first run / post-outage zeroes vol/OI deltas), missing-symbol carry-forward, $500k/24h min-notional guard to suppress thin xyz: noise. TTL 2700s (9× cadence). - proto/worldmonitor/market/v1/get_hyperliquid_flow.proto + service.proto registration; make generate regenerated client/server bindings. - server/worldmonitor/market/v1/get-hyperliquid-flow.ts — getCachedJson reader matching get-cot-positioning.ts seeded-handler pattern. - server/gateway.ts cache-tier entry (medium). - api/health.js: hyperliquidFlow registered with maxStaleMin:15 (3× cadence) + transitional ON_DEMAND_KEYS gate for the first ~7 days of bake-in. - api/seed-health.js mirror with intervalMin:5. - scripts/seed-bundle-market-backup.mjs entry (NIXPACKS auto-redeploy on scripts/** watch). - src/components/MarketPanel.ts: CommoditiesPanel grows a Perp Flow tab + fetchHyperliquidFlow() RPC method; OI Δ1h derived from sparkOi tail. - src/App.ts: prime via primeVisiblePanelData() + recurring refresh via refreshScheduler.scheduleRefresh() at 5min cadence (panel does NOT own setInterval; matches the App.ts:1251 lifecycle convention). - 28 unit tests covering scoring parity, warmup flag, min-notional guard, schema rejection, missing-symbol carry-forward, post-outage cold start, sparkline cap, alert threshold. Tests: test:data 5169/5169, hyperliquid-flow-seed 28/28, route-cache-tier 5/5, typecheck + typecheck:api green. One pre-existing test:sidecar failure (cloud-fallback origin headers) is unrelated and reproduces on origin/main. * fix(hyperliquid-flow): address review feedback — volume baseline window, warmup lifecycle, error logging Two real correctness bugs and four review nits from PR #3074 review pass. Correctness fixes: 1. Volume baseline was anchored to the OLDEST 12 samples, not the newest. sparkVol is newest-at-tail (shiftAndAppend), so slice(0, 12) pinned the rolling mean to the first hour of data forever once len >= 12. Volume scoring would drift further from current conditions each poll. Switched to slice(-VOLUME_BASELINE_MIN_SAMPLES) so the baseline tracks the most recent window. Regression test added. 2. Warmup flag flipped to false on the second successful poll while volume scoring still needed 12+ samples to activate. UI told users warmup lasted ~1h but the badge disappeared after 5 min. Tied per-asset warmup to real baseline readiness (coldStart OR vol samples < 12 OR prior OI missing). Snapshot-level warmup = any asset still warming. Three new tests cover the persist-through-baseline-build, clear-once-ready, and missing-OI paths. Review nits: - Handler: bare catch swallowed Redis/parse errors; now logs err.message. - Panel: bare catch in fetchHyperliquidFlow hid RPC 500s; now logs. - MarketPanel.ts: deleted hand-rolled RawHyperliquidAsset; mapHyperliquidFlowResponse now takes GetHyperliquidFlowResponse from the generated client so proto drift fails compilation instead of silently. - Seeder: added @ts-check + JSDoc on computeAsset per type-safety rule. - validateUpstream: MAX_UPSTREAM_UNIVERSE=2000 cap bounds memory. - buildSnapshot: logs unknown xyz: perps upstream (once per run) so ops sees when Hyperliquid adds markets we could whitelist. Tests: 37/37 green; typecheck + typecheck:api clean. * fix(hyperliquid-flow): wire bootstrap hydration per AGENTS.md mandate Greptile review caught that AGENTS.md:187 mandates new data sources be wired into bootstrap hydration. Plan had deferred this on "lazy deep-dive signal" grounds, but the project convention is binding. - server/_shared/cache-keys.ts: add hyperliquidFlow to BOOTSTRAP_CACHE_KEYS + BOOTSTRAP_TIERS ('slow' — non-blocking, page-load-parallel). - api/bootstrap.js: add to inlined BOOTSTRAP_CACHE_KEYS + SLOW_KEYS so bootstrap.test.mjs canonical-mirror assertions pass. - src/components/MarketPanel.ts: - Import getHydratedData from @/services/bootstrap. - New mapHyperliquidFlowSeed() normalizes the raw seed-JSON shape (numeric fields) into HyperliquidFlowView. The RPC mapper handles the proto shape (string-encoded numbers); bootstrap emits the raw blob. - fetchHyperliquidFlow now reads hydrated data first, renders immediately, then refreshes from RPC — mirrors FearGreedPanel pattern. Tests: 72/72 green (bootstrap + cache-tier + hyperliquid-flow-seed). |
||
|
Elie Habib
|
281a7c0728 | chore: regenerate MarketService OpenAPI specs for GetGoldIntelligence (#3011) | ||
|
Elie Habib
|
793d7df9dc | feat(energy-crisis): add IEA 2026 Energy Crisis Policy Response Tracker panel and seeder (#3008) | ||
|
Elie Habib
|
c26ae6b827 | feat(energy): add Oil Inventories panel with SVG charts (#3003) | ||
|
Elie Habib
|
c72251178c |
feat(route-explorer): Sprint 4 — strategic-product impact tab + get-route-impact RPC (#2996)
* feat(route-explorer): Sprint 4 — strategic-product impact tab
Adds the Impact tab to the Route Explorer, powered by a new
get-route-impact RPC that returns strategic-product trade data for
any country pair.
Backend:
- New proto get_route_impact.proto with GetRouteImpact{Request,Response}
+ StrategicProduct message
- New handler server/worldmonitor/supply-chain/v1/get-route-impact.ts:
reads comtrade:bilateral-hs4:{iso2}:v1 store, computes lane value for
selected HS2, top 5 strategic products by value with chokepoint
exposure, resilience score (server-side from Redis), dependency flags
- Cache key ROUTE_IMPACT_KEY in cache-keys.ts (NOT in BOOTSTRAP_KEYS)
- Gateway + premium-paths registered as slow-browser premium RPC
- Client wrapper fetchRouteImpact in supply-chain/index.ts
Impact tab UI:
- CountryImpactTab.ts: strategic products table (top 5 by value),
lane value card for selected HS2, hs2InSeededUniverse banner when
HS2 is not in the 14 seeded sectors, comtradeSource states
(missing/empty/bilateral-hs4), drill-sideways on product row click
- LeftRail.updateDependencyFlags: renders flags from Impact response
with color-coded badges (compound_risk/single_source/diversifiable)
Data flow:
- fetchImpact fires in parallel with fetchResilience after lane data
loads, generation-scoped
- Impact response updates left-rail flags + resilience score
- Drill-sideways: clicking a product row switches the explorer's HS2
and re-queries all tabs
Server-side resilience:
- get-route-impact reads resilience:score:v8:{iso2} from Redis directly
so the data is available for future email briefs without client calls
Plan: docs/plans/2026-04-11-001-feat-worldwide-route-explorer-plan.md
* fix(route-explorer): real exposure score for flags + tabstrip sync on drill
P1: computeDependencyFlags hardcoded primaryExposure=80 whenever any
chokepoint existed, fabricating SINGLE_CORRIDOR_CRITICAL without using
real exposure data. Replaced with computeRealExposureScore that uses the
same route-cluster overlap logic as get-sector-dependency, computing the
actual exposure percentage before comparing against the >80 threshold.
P2: handleDrillSideways set state.tab=1 directly without going through
setTab(), leaving the tabstrip visually and semantically on Impact while
content showed Current. Now calls setTab(1) which updates both the
tabstrip active state and aria-selected.
* fix(route-explorer): guard resilience overwrite + normalize HS2 filter
P1: fetchImpact could zero the left-rail resilience score when
get-route-impact returned resilienceScore=0 (Redis miss fallback),
overwriting a valid score set by the concurrent fetchResilience call.
Now only applies the server-side score when it is actually > 0.
P2: HS4-to-HS2 matching used a redundant dual-condition filter
(hs4ToHs2 + startsWith) that masked a potential normalization bug.
Simplified to normalize hs2 once via parseInt then use a single
hs4ToHs2 comparison.
|
||
|
Elie Habib
|
e070a97c3d |
Phase 3 PR2: Weekly regional briefs (LLM seeder + RPC) (#2989)
* feat(intelligence): weekly regional briefs (Phase 3 PR2) Phase 3 PR2 of the Regional Intelligence Model. Adds LLM-powered weekly intelligence briefs per region, completing the core feature set. ## New seeder: scripts/seed-regional-briefs.mjs Standalone weekly cron script (not part of the 6h derived-signals bundle). For each non-global region: 1. Read the latest snapshot via two-hop Redis read 2. Read recent regime transitions from the history log (#2981) 3. Call the LLM once per region with regime trajectory + balance + triggers + narrative context 4. Write structured brief to intelligence:regional-briefs:v1:weekly:{region} with 8-day TTL (survives one missed weekly run) Reuses the same injectable-callLlm + parse-validation + provider-chain pattern from narrative.mjs and weekly-brief.mjs. ## New module: scripts/regional-snapshot/weekly-brief.mjs generateWeeklyBrief(region, snapshot, transitions, opts?) -> { region_id, generated_at, period_start, period_end, situation_recap, regime_trajectory, key_developments[], risk_outlook, provider, model } buildBriefPrompt() — pure prompt builder parseBriefJson() — JSON parser with prose-extraction fallback emptyBrief() — canonical empty shape Global region is skipped. Provider chain: Groq -> OpenRouter. Validate callback ensures only parseable responses pass (narrative.mjs PR #2960 review fix pattern). ## Proto + RPC: GetRegionalBrief proto/worldmonitor/intelligence/v1/get_regional_brief.proto - GetRegionalBriefRequest { region_id } - GetRegionalBriefResponse { brief: RegionalBrief } - RegionalBrief { region_id, generated_at, period_start, period_end, situation_recap, regime_trajectory, key_developments[], risk_outlook, provider, model } ## Server handler server/worldmonitor/intelligence/v1/get-regional-brief.ts Simple getCachedJson read + adaptBrief snake->camel adapter. Returns upstreamUnavailable: true on Redis failure so the gateway skips caching (matching the get-regime-history pattern from #2981). ## Premium gating + cache tier src/shared/premium-paths.ts + server/gateway.ts RPC_CACHE_TIER ## Tests — 27 new unit tests buildBriefPrompt (5): region/balance/transitions/narrative rendered, empty transitions handled, missing fields tolerated parseBriefJson (5): valid JSON, garbage, all-empty, cap at 5, prose extraction generateWeeklyBrief (6): success, global skip, LLM fail, garbage, exception, period_start/end delta emptyBrief (2): region_id + empty fields handler (4): key prefix, adapter export, upstreamUnavailable, registration security (2): premium path + cache tier proto (3): RPC declared, import wired, RegionalBrief fields ## Verification - npm run test:data: 4651/4651 pass - npm run typecheck + typecheck:api: clean - biome lint: clean * fix(intelligence): address 3 review findings on #2989 P2 #1 — no consumer surface for GetRegionalBrief Acknowledged. The consumer is the RegionalIntelligenceBoard panel, which will call GetRegionalBrief and render a weekly brief block. This wiring is Phase 3 PR3 (UI) scope — the RPC + Redis key are the delivery mechanism, not the end surface. No code change in this commit; the RPC is ready for the panel to consume. P2 #2 — readRecentTransitions collapses failure to [] readRecentTransitions returned [] on Redis/network failure, which is indistinguishable from a genuinely quiet week. The LLM then generates a brief claiming "no regime transitions" when in reality the upstream is down — fabricating false input. Fix: return null on failure. The seeder skips the region with a clear log message when transitions is null, so the brief is never written with unreliable input. Empty array [] now only means genuinely no transitions in the 7-day window. P2 #3 — parseBriefJson accepts briefs the seeder rejects parseBriefJson treated non-empty key_developments as valid even if situation_recap was empty. The seeder gate only writes when brief.situation_recap is truthy. That mismatch means the validator pass + provider-fallback logic could accept a response that the seeder then silently drops. Fix: require situation_recap in parseBriefJson for valid=true, matching the seeder gate. Now both checks agree on what constitutes a usable brief, and the provider-fallback chain correctly falls through when a provider returns a brief with developments but no recap. * fix(intelligence): TTL path-segment fix + seed-meta always-write (Greptile P1+P2 on #2989) P1 — TTL silently not applied (briefs never expire) Upstash REST ignores query-string SET options (?EX=N). The correct form is path-segment: /set/{key}/{value}/EX/{seconds}. Without this fix every brief persists indefinitely and Redis storage grows unboundedly across weekly runs. P2 — seed-meta not written when all regions skipped writeExtraKeyWithMeta was gated on generated > 0. If every region was skipped (no snapshot yet, or LLM failed), seed-meta was never written, making the seeder indistinguishable from "never ran" in health tooling. Now writes seed-meta whenever failed === 0, carrying regionsSkipped count. P2 #3 (validate gate) — already fixed in previous commit (parseBriefJson now requires situation_recap for valid=true). * fix(intelligence): register regional-briefs in health.js SEED_META + STANDALONE_KEYS (review P2 on #2989) * fix(intelligence): register regional-briefs in api/seed-health.js (review P2 on #2989) * fix(intelligence): raise brief TTL to 15 days to cover missed weekly cycle (review P2 on #2989) * fix(intelligence): distinguish missing-key from Redis-error + coverage-gated health (review P2s on #2989) P2 #1 — false upstreamUnavailable before first seed getCachedJson returns null for both "key missing" and "Redis failed", so the handler was advertising an outage for every region before the first weekly seed ran. Switched to getRawJson (throws on Redis errors) so null = genuinely missing key → clean empty 200, and thrown error = upstream failure → upstreamUnavailable: true for gateway no-store. P2 #2 — partial run hides coverage loss in health The seed-meta was written with generated count even if only 1 of 7 regions produced a brief. /api/health treats any positive recordCount as healthy, so broad regional failure was invisible to operators. Fix: recordCount is set to 0 when generated < ceil(expectedRegions/2). This makes /api/health report EMPTY_DATA for severely partial runs while still writing seed-meta (so the seeder is confirmed to have run). coverageOk flag in the summary payload lets operators drill into the exact coverage state. * fix(intelligence): tighten coverage gate to expectedRegions-1 (review P2 on #2989) |
||
|
Elie Habib
|
822eef0fa6 |
feat(supply-chain): Sprint 1 — Route Explorer wrapper RPC (#2980)
* feat(supply-chain): Sprint 1 — Route Explorer wrapper RPC
Adds an internal wrapper around the vendor-only route-intelligence
compute so the upcoming Route Explorer UI can call it from a browser
PRO session instead of forcing an X-WorldMonitor-Key API gate.
Backend:
- New proto get-route-explorer-lane.proto with GetRouteExplorerLane{Request,Response}
- New handler server/worldmonitor/supply-chain/v1/get-route-explorer-lane.ts
- New static lookup tables _route-explorer-static-tables.ts:
TRANSIT_DAYS_BY_ROUTE_ID, FREIGHT_USD_BY_CARGO_TYPE,
BYPASS_CORRIDOR_GEOMETRY_BY_ID — covers all 5 land-bridge corridors
plus every sea-alternative corridor with hand-curated coordinates
- Wired into supply-chain handler.ts service dispatcher
- Cache key ROUTE_EXPLORER_LANE_KEY in cache-keys.ts (NOT in BOOTSTRAP_KEYS)
- Gateway entry: PREMIUM_RPC_PATHS + RPC_CACHE_TIER 'slow-browser'
- Premium path entry in src/shared/premium-paths.ts so browser PRO auth attaches
Response contract enriches route-intelligence with:
- primaryRouteGeometry polyline from TRADE_ROUTES (lon/lat pairs)
- fromPort/toPort coords on every bypass option so the client can call
MapContainer.setBypassRoutes directly without geometry lookups
- status: 'active' | 'proposed' | 'unavailable' derived from corridor notes
to honestly label kra_canal_future and black_sea_western_ports
- estTransitDaysRange + estFreightUsdPerTeuRange from static tables
- noModeledLane: true when origin/destination clusters share no routes
Client wrapper fetchRouteExplorerLane added to src/services/supply-chain/index.ts.
Tests: tests/route-explorer-lane.test.mts — 30-query smoke matrix
(10 country pairs × 3 HS2 codes), structural assertions only, no
hard-coded transit/cost values. Test exposes a pure computeLane()
function with an injectable status map so it does not need Redis.
Gap report (from smoke run): 12 of 30 queries fall back to a synthetic
primaryRouteId because the destination's port cluster has no shared route
with the origin (US-JP, ZA-IN, CL-CN, TR-DE × 3 HS2 each). These pairs
return noModeledLane:true; Sprint 3 will render an empty-state for them.
Plan: docs/plans/2026-04-11-001-feat-worldwide-route-explorer-plan.md
* fix(route-explorer): address PR #2980 review findings
P1: bypass warRiskTier was hard-coded to WAR_RISK_TIER_NORMAL, dropping
the live risk signal from chokepoint status. Now derived from the
statusMap via the corridor's primaryChokepointId.
P2: freight fallback in emptyResponse and client-side empty payload used
a cargo-agnostic container range for all cargo types. Removed the ranges
entirely from fallback/noModeledLane responses; they are only present
when the lane is actually modeled.
Suggestion: when noModeledLane is true, the response now returns empty
primaryRouteId, empty geometry, empty exposures, empty bypasses, and
omits transit/freight ranges. Previously it returned plausible-looking
synthetic data from the origin's first route which could mislead the UI.
Tests updated to assert the noModeledLane contract: empty fields when
the flag is set, non-empty ranges only when the lane is modeled.
* fix(route-explorer): cargo-aware route ranking + bypass waypoint risk
P1: primary route selection was order-dependent, picking whichever
shared route the origin cluster listed first. Mixed clusters like
CN/JP could return an energy lane for a container request. Now ranks
shared routes by cargo-category compatibility (container→container,
tanker→energy, bulk→bulk, roro→container) before selecting.
P1: bypass warRiskTier was copied from the primary chokepoint instead
of derived from the corridor's own waypointChokepointIds. This
overstated risk for alternatives like Cape of Good Hope whose waypoints
may have a lower risk tier. Now uses max-tier across waypoint
chokepoints, matching get-bypass-options.ts logic.
Suggestion: placeholder corridors with addedTransitDays=0 (like
gibraltar_no_bypass, cape_of_good_hope_is_bypass) are now filtered out.
Previously they could surface as active alternatives.
Regression tests added:
- CN→JP tanker: asserts energy route is selected over container route
- CN→DE with faked Suez=CRITICAL / Cape=NORMAL: asserts Cape bypass
shows NORMAL, not CRITICAL
- ES→EG: asserts zero-transit-day placeholders are excluded
* fix(route-explorer): scope exposures to primary route + narrow placeholder filter
P1: chokepointExposures and bypassOptions were computed from the full
sharedRoutes set, mixing data from energy/container corridors into a
single response. Now scoped to the cargo-ranked primaryRouteId only,
matching the proto contract that exposures are "on the primary route."
P2: the addedTransitDays === 0 filter was too broad and removed
kra_canal_future (a proposed bypass with real modeling). Narrowed to an
explicit PLACEHOLDER_CORRIDOR_IDS set (gibraltar_no_bypass,
cape_of_good_hope_is_bypass) so proposed zero-day corridors survive and
are surfaced with CORRIDOR_STATUS_PROPOSED.
Regression tests:
- chokepointExposures follow primaryRouteId (CN->JP container)
- kra_canal_future appears as CORRIDOR_STATUS_PROPOSED for Malacca routes
- placeholder filter still excludes explicit placeholders
* fix(route-explorer): address PR #2980 review comments
1. Unavailable corridors without waypoints (e.g. black_sea_western_ports)
now derive WAR_RISK_TIER_WAR_ZONE from their CORRIDOR_STATUS_UNAVAILABLE
status, instead of returning WAR_RISK_TIER_UNSPECIFIED. Corridors with
waypointChokepointIds still use max-tier across those waypoints.
2. Added fixture test with non-empty status map (suez=75/HIGH,
malacca=30/ELEVATED) so disruptionScore and warRiskTier assertions are
not trivially satisfied by the empty-map default path.
3. Documented the single-chokepoint bypass design gap in the test gap report:
bypassOptions only cover the primary chokepoint; multi-chokepoint routes
show exposure for all but bypass guidance for only the top one. Sprint 3
will decide whether to expand to top-N or add a UI hint.
|
||
|
Elie Habib
|
19d67cea94 |
Phase 3 PR1: Regime drift history (writer + RPC) (#2981)
* feat(intelligence): regime drift history (Phase 3 PR1)
Phase 3 PR1 of the Regional Intelligence Model. Adds an append-only
regime transition log per region plus a premium-gated RPC to read it.
## What landed
### New writer module: scripts/regional-snapshot/regime-history.mjs
Single public entry point:
recordRegimeTransition(region, snapshot, diff, opts?)
-> { recorded, entry, pushed, trimmed }
Pure builder + Redis-ops orchestrator + dependency-injected publisher.
Flow:
1. buildTransitionEntry() returns null when diff has no regime_changed
(steady-state snapshots produce no entry — pure transition stream)
2. publishTransitionWithOps() LPUSHes onto
intelligence:regime-history:v1:{region}, then LTRIMs to keep the
most recent REGIME_HISTORY_MAX (100) entries
3. defaultPublisher binds real Upstash REST calls; tests inject an
in-memory ops object for offline coverage
LTRIM failure is non-fatal — entry already landed, next cycle will
re-trim. LPUSH failure short-circuits and reports pushed=false. The
recorder NEVER throws and is wrapped in its own try/catch in the seed
loop so snapshot persist is never blocked.
### seed-regional-snapshots.mjs hook
Added a regime-history call alongside the existing alert-emitter call,
right after persistSnapshot success. Same best-effort contract:
unconditional try/catch, log warn on throw, continue main loop.
### Proto + RPC: GetRegimeHistory
proto/worldmonitor/intelligence/v1/get_regime_history.proto
- GetRegimeHistoryRequest { region_id, limit (0..100) }
- GetRegimeHistoryResponse { transitions: RegimeTransition[] }
- RegimeTransition { region_id, label, previous_label,
transitioned_at, transition_driver, snapshot_id }
region_id validated as strict kebab-case (same regex as
get-regional-snapshot). limit capped server-side at MAX_LIMIT=100,
defaulting to 50 when omitted.
Added to IntelligenceService in service.proto. Generated openapi
JSON/YAML committed via `make generate`.
### Server handler: server/worldmonitor/intelligence/v1/get-regime-history.ts
LRANGE-based read (newest-first because the writer LPUSHes). adapter
is a dedicated exported function adaptTransition(raw) for testability.
LRANGE helper is inlined here because server/_shared/redis.ts has no
list helpers yet — this is the first list-reading handler in the
intelligence service. If a second list reader lands, the helper can
be promoted to a shared util.
Empty list / Redis miss / failed JSON parse all return
{ transitions: [] } so the client can distinguish "never changed" from
"upstream broken" via the HTTP status code, not the body.
Registered in handler.ts.
### Premium gating + cache tier
src/shared/premium-paths.ts: added /api/intelligence/v1/get-regime-history
server/gateway.ts RPC_CACHE_TIER: same path with 'slow' tier (matches
route-parity contract enforced by
tests/route-cache-tier.test.mjs)
## Tests — 44 new unit tests
tests/regional-snapshot-regime-history.test.mjs (22 tests):
buildTransitionEntry (7):
- null on missing diff/region/snapshot
- returns entry on regime change
- first-ever transition (empty previous_label)
- falls back to generated_at when transitioned_at is missing
- preserves snapshot_id
publishTransitionWithOps (8):
- happy path (LPUSH + LTRIM both succeed)
- canonical key prefix
- LTRIM uses REGIME_HISTORY_MAX-1 stop
- LPUSH failure → not pushed, LTRIM not called
- LTRIM failure → pushed=true, trimmed=false (non-fatal)
- LPUSH/LTRIM throwing caught and reported
- null/empty entry → no-op
recordRegimeTransition (5):
- no-op on no regime change
- records on regime change
- publisher returning false → recorded=false
- publisher exceptions swallowed
- critical escalation labels preserved
module constants (2): key prefix + max are valid
tests/get-regime-history.test.mts (22 tests):
adaptTransition (4):
- all fields snake → camel
- missing fields → empty/zero defaults
- first-ever transition shape preserved
- non-numeric transitioned_at → 0
handler structural checks (7): canonical key prefix, LRANGE usage,
adapter export, handler export signature, MAX_LIMIT cap matches
writer, missing-region short-circuit, malformed-entry filter
intelligence handler registration (2): import + registration
security wiring (2): premium path + cache-tier entry
proto definition (7): RPC method declared, import wired, request
shape, kebab regex, limit bounds, RegimeTransition fields,
response shape
## Verification
- node --test tests/regional-snapshot-regime-history.test.mjs: 22/22 pass
- npx tsx --test tests/get-regime-history.test.mts: 22/22 pass
- npm run test:data: 4621/4621 pass
- npm run typecheck: clean
- npm run typecheck:api: clean
- biome lint on touched files: clean
## Deferred to future iterations
- Phase 3 PR2: weekly regional briefs LLM seeder (consumes regime history
to highlight drift events in the weekly summary)
- Phase 3 PR3: UI block in RegionalIntelligenceBoard for regime drift
timeline (can ride alongside or after PR2)
- Drift analytics: % of last N days spent in each regime, transition
frequency rolling window, regime cycle detection
- Alert triggers on drift cycles (e.g., "thrashed between regimes 3 times
in 7 days")
* fix(intelligence): address 2 review findings on #2981
P2 #1 — transition_driver always empty in the live path
buildRegimeState(balance, previousLabel, '') at Step 11 passed an empty
driver because the diff hasn't been computed yet. The regime-history
recorder reads snapshot.regime.transition_driver which was therefore
always '' in production, despite tests exercising synthetic fixtures
with a populated driver.
Fix: after Step 15 derives triggerReason via inferTriggerReason(diff),
backfill regime.transition_driver = triggerReason when a genuine regime
change occurred. This ensures both the persisted snapshot's regime block
AND the regime-history entry carry the real driver (e.g., 'regime_shift',
'trigger_activation', 'corridor_break').
Added 2 regression tests: populated driver flows through, and pre-fix
empty-driver snapshots remain back-compatible.
P2 #2 — Redis failure returns cached false-empty history
get-regime-history.ts returned 200 {transitions:[]} on LRANGE failure.
The gateway caches 200 GET responses at the slow tier, so a transient
Upstash outage would be pinned as a false-empty history until the cache
TTL expired.
Fix: when redisLrange returns null (Redis unavailable or credentials
missing), the response now includes upstreamUnavailable: true in the
body. The gateway already checks for this flag in the response body
(line 434) and sets Cache-Control: no-store, so transient failures are
not cached.
Added 1 structural test asserting the upstreamUnavailable flag is set.
Verification:
- 24/24 writer tests, 23/23 handler tests, 4624/4624 full suite pass
- npm run typecheck: clean
- biome lint on touched files: clean
* fix(intelligence): correct misleading 'log once per region' comment (Greptile P2)
|
||
|
Elie Habib
|
7da202c25d |
Phase 1 PR1: RegionalSnapshot proto + RPC handler (#2951)
* feat(intelligence): add RegionalSnapshot proto definition
Defines the canonical RegionalSnapshot wire format for Phase 1 of the
Regional Intelligence Model. Mirrors the TypeScript contract in
shared/regions.types.d.ts that Phase 0 landed with.
New proto file: proto/worldmonitor/intelligence/v1/get_regional_snapshot.proto
Messages:
- RegionalSnapshot (13 top-level fields matching the spec)
- SnapshotMeta (11 fields including snapshot_id, narrative_provider,
narrative_model, trigger_reason, snapshot_confidence, missing_inputs,
stale_inputs, valid_until, versions)
- RegimeState (label + transition history)
- BalanceVector (7 axes: 4 pressures + 3 buffers + net_balance + decomposed
drivers)
- BalanceDriver (axis, magnitude, evidence_ids, orientation)
- ActorState (leverage_score, role, domains, delta, evidence_ids)
- LeverageEdge (actor-to-actor directed influence)
- ScenarioSet + ScenarioLane (per-horizon distribution normalizing to 1.0)
- TransmissionPath (typed fields: severity, confidence, latency_hours,
magnitude range, asset class, template provenance)
- TriggerLadder + Trigger + TriggerThreshold (structured operator/value/
window/baseline)
- MobilityState + AirspaceStatus + FlightCorridorStress + AirportNodeStatus
- EvidenceItem (typed origin for the trust trail)
- RegionalNarrative + NarrativeSection (LLM-synthesized text with
evidence_ids on every section)
RPC: GetRegionalSnapshot(GetRegionalSnapshotRequest) -> GetRegionalSnapshotResponse
- GET /api/intelligence/v1/get-regional-snapshot
- region_id validated as lowercase kebab via buf.validate regex
- No other parameters; the handler reads canonical state
Generated code committed alongside:
- src/generated/client/worldmonitor/intelligence/v1/service_client.ts
- src/generated/server/worldmonitor/intelligence/v1/service_server.ts
- docs/api/IntelligenceService.openapi.{json,yaml}
The generated TypeScript types use camelCase per standard buf codegen, while
Phase 0 persists snapshots in Redis using the snake_case shape from
shared/regions.types.d.ts. The handler lands in a follow-up commit with a
localized snake_case -> camelCase adapter so Phase 0 code stays frozen.
Spec: docs/internal/pro-regional-intelligence-upgrade.md
* feat(intelligence): get-regional-snapshot RPC handler
Reads canonical persisted RegionalSnapshot for a region via the two-hop
lookup pattern established by the Phase 0 persist layer:
1. GET intelligence:snapshot:v1:{region}:latest -> snapshot_id
2. GET intelligence:snapshot-by-id:v1:{snapshot_id} -> full snapshot JSON
Returns empty response (snapshot omitted) when:
- No latest pointer exists (seed has never run or unknown region)
- Latest pointer references a pruned or TTL-expired snapshot
- Snapshot JSON is malformed
The handler does NOT recompute on miss. One writer (the seed bundle),
canonical reads. Matches the architecture commitment in the spec.
Includes a full snake_case -> camelCase adapter so the persisted Phase 0
shape (shared/regions.types.d.ts) maps cleanly onto the camelCase proto
wire format generated by buf. The adapter is the single bridge between
the two shapes; Phase 0 code stays frozen. Adapter handles every nested
message: SnapshotMeta, RegimeState, BalanceVector (+pressures/buffers
drivers), ActorState, LeverageEdge, ScenarioSet (+lanes +transmissions),
TransmissionPath, TriggerLadder (+triggers +thresholds), MobilityState
(+airspace +flight corridors +airports), EvidenceItem, RegionalNarrative
(+5 sections +watch items).
Wiring:
- Registered on intelligenceHandler in handler.ts
- Added to PREMIUM_RPC_PATHS (src/shared/premium-paths.ts) so the
gateway enforces Pro subscription or API key
- Added to RPC_CACHE_TIER with 'slow' tier (300s browser, 1800s edge)
matching similar premium intelligence RPCs
Not in this PR:
- LLM narrative generator (follow-up PR2, wires into snapshot writer)
- RegionalIntelligenceBoard panel UI (follow-up PR3)
- ENDPOINT_ENTITLEMENTS tier-specific enforcement (PREMIUM_RPC_PATHS
alone is the Pro gate; only stock-analysis endpoints currently use
tier-specific enforcement)
* test(intelligence): unit tests for get-regional-snapshot adapter + structural checks
29 tests across 5 suites covering:
adaptSnapshot (18 tests): real unit tests of the snake_case -> camelCase
adapter with synthetic persisted snapshots. Covers every nested message
(SnapshotMeta, RegimeState, BalanceVector with 7 axes + decomposed drivers,
ActorState, LeverageEdge, ScenarioSet with nested lanes and transmissions,
TriggerLadder with all 3 buckets + TriggerThreshold, MobilityState with
airspace/flights/airports, EvidenceItem, RegionalNarrative with all 5
sections + watch_items). Also asserts empty-default behavior when
nested fields are missing.
Handler structural checks (8 tests): validates import of getCachedJson,
canonical key prefixes, two-hop lookup ordering, empty-response fallbacks
on missing pointer or malformed snapshot, and export signature matching
the service interface.
Registration (2 tests): confirms getRegionalSnapshot is imported and
registered on the intelligenceHandler object.
Security wiring (2 tests): confirms the endpoint is in PREMIUM_RPC_PATHS
and RPC_CACHE_TIER with 'slow' tier.
Proto definition (3 tests): confirms the RPC method declaration, region_id
validation regex, RegionalSnapshot top-level field layout, and
BalanceVector 7-axis declaration.
* fix(intelligence): address Greptile P2 review findings on #2951
Two P2 findings from Greptile on the RegionalSnapshot proto+RPC PR.
1) region_id regex permitted trailing and consecutive hyphens
Old: ^[a-z][a-z0-9-]*$ — accepted "mena-", "east-asia-", "foo--bar"
New: ^[a-z][a-z0-9]*(-[a-z0-9]+)*$ — strict kebab-case, every hyphen must be
followed by at least one alphanumeric character. Regenerated openapi JSON/YAML
via `make generate`. Test assertion updated to match.
2) RPC_CACHE_TIER entry looked like dead code for premium paths
Greptile flagged that `isPremium` short-circuits the tier lookup to
'slow-browser' before RPC_CACHE_TIER is consulted, so the entry is never read
at runtime. Kept the entry because `tests/route-cache-tier.test.mjs` enforces
a parity contract requiring every generated GET route to have an explicit
tier. Added a NOTE comment in gateway.ts explaining the policy, and updated
the security-wiring test with a rationale comment so future maintainers know
the entry is intentional documentation, not a stale wire.
|
||
|
Elie Habib
|
46c35e6073 | feat(breadth): add market breadth history chart (#2932) | ||
|
Elie Habib
|
55c9c36de2 |
feat(stocks): add insider transaction tracking to stock analysis panel (#2928)
* feat(stocks): add insider transaction tracking to stock analysis panel Shows 6-month insider buy/sell activity from Finnhub: total buys, sells, net value, and recent named-exec transactions. Gracefully skips when FINNHUB_API_KEY is unavailable. * fix: add cache tier entry for get-insider-transactions route * fix(stocks): add insider RPC to premium paths + fix empty/stale states * fix(stocks): add insider RPC to premium paths + fix empty/stale states - Add /api/market/v1/get-insider-transactions to PREMIUM_RPC_PATHS - Return unavailable:false with empty transactions when Finnhub has no data (panel shows "No insider transactions" instead of "unavailable") - Mark stale insider data on refresh failures to avoid showing outdated info - Update test to match new empty-data behavior * fix(stocks): unblock stock-analysis render and surface exercise-only insider activity - loadStockAnalysis no longer awaits loadInsiderDataForPanel before panel.renderAnalyses. The insider fetch now fires in parallel after the primary render at both the cached-snapshot and live-fetch call sites. When insider data arrives, loadInsiderDataForPanel re-renders the panel so the section fills in asynchronously without holding up the analyst report on a secondary Finnhub RPC. - Add transaction code 'M' (exercise / conversion of derivative) to the allowed set in get-insider-transactions so symbols whose only recent Form 4 activity is option/RSU exercises no longer appear as "No insider transactions in the last 6 months". Exercises do not contribute to buys/sells dollar totals because transactionPrice is the strike price, not a market transaction. - Panel table now uses a neutral (dim) color for non-buy/non-sell rows (M rows) instead of the buy/sell green/red binary. - Tests cover: exercise-only activity producing non-empty transactions with zero buys/sells, and blended P/S/M activity preserving all three rows. * fix(stocks): prevent cached insider fetch from clobbering live render - Cached-path insider enrichment only runs when no live fetch is coming - Added generation counter to guard against concurrent loadStockAnalysis calls - Stale insider fetches now no-op instead of reverting panel state * fix(stocks): hide transient insider-unavailable flash and zero out strike-derived values - renderInsiderSection returns empty string when insider data is not yet fetched, so the transient "Insider data unavailable" card no longer flashes on initial render before the RPC completes - Exercise rows (code M) now carry value: 0 on the server and render a dash placeholder in the Value cell, matching how the buy/sell totals already exclude strike-derived dollar amounts * fix(stocks): exclude non-market Form 4 codes (A/D/F) from insider buy/sell totals Form 4 codes A (grant/award), D (disposition to issuer), and F (tax/exercise payment) are not open-market trades and should not drive insider conviction totals. Only P (open-market purchase) and S (open-market sale) now feed the buy/sell aggregates. A/D/F rows are still surfaced in the transaction list alongside M (exercise) with value zeroed out so the panel does not look empty. |
||
|
Elie Habib
|
a742537ae5 |
feat(supply-chain): Sprint D — GetSectorDependency RPC + vendor route-intelligence API + webhooks (#2905)
* feat(supply-chain): Sprint D — GetSectorDependency RPC + vendor route-intelligence API + webhooks * fix(supply-chain): move bypass-corridors + chokepoint-registry to server/_shared to fix api/ boundary violations * fix(supply-chain): webhooks — persist secret, fix sub-resource routing, add ownership check * fix(supply-chain): address PR #2905 review findings - Use SHA-256(apiKey) for ownerTag instead of last-12-chars (unambiguous ownership) - Implement GET /api/v2/shipping/webhooks list route via per-owner Redis Set index - Tighten SSRF: https-only, expanded metadata hostname blocklist, document DNS rebinding edge-runtime limitation - Fix get-sector-dependency.ts stale src/config/ imports → server/_shared/ (Greptile P1) * fix(supply-chain): getSectorDependency returns blank primaryChokepointId for landlocked countries computeExposures() previously mapped over all of CHOKEPOINT_REGISTRY even when nearestRouteIds was empty, producing a full array of score-0 entries in registry insertion order. The caller's exposures[0] then picked the first registry entry (Suez) as the "primary" chokepoint despite primaryChokepointExposure = 0. LI, AD, SM, BT and other landlocked countries were all silently assigned a fake chokepoint. Fix: guard at the top of computeExposures() -- return [] when input is empty so primaryChokepointId stays '' and primaryChokepointExposure stays 0. |
||
|
Elie Habib
|
bd07829518 |
feat(supply-chain): Sprint 2 — bypass corridor intelligence + cost shock engine (#2873)
* feat(supply-chain): Sprint 2 — bypass corridor intelligence + cost shock engine - src/config/bypass-corridors.ts: ~40 bypass corridors for all 13 chokepoints - server/supply-chain/v1/get-bypass-options.ts: PRO-gated RPC, live bypass scoring from chokepoint status cache - server/supply-chain/v1/get-country-cost-shock.ts: PRO-gated RPC, war risk premium BPS + energy coverage days (HS 27) - server/supply-chain/v1/_insurance-tier.ts: pure function, Lloyd's JWC threat → premium BPS - gateway.ts + premium-paths.ts: registered both RPCs as slow-browser + PRO-gated - src/services/supply-chain/index.ts: fetchBypassOptions + fetchCountryCostShock client methods - proto: GetBypassOptions + GetCountryCostShock messages + service registrations - tests/supply-chain-sprint2.test.mjs: 61 tests covering all new components Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> * fix(cost-shock): call computeEnergyShockScenario directly instead of reading wrong cache key The old code read from `energy:shock:${iso2}:${chokepointId}:v1` which never matches the actual v2 cache key written by compute-energy-shock.ts. Fix by calling computeEnergyShockScenario() directly (it handles v2 caching internally) and mapping effectiveCoverDays + crude product deficitPct to the response fields. * fix(cost-shock): average refined product deficitPct instead of looking for non-existent 'crude' product --------- Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> |
||
|
Elie Habib
|
6e401ad02f |
feat(supply-chain): Global Shipping Intelligence — Sprint 0 + Sprint 1 (#2870)
* feat(supply-chain): Sprint 0 — chokepoint registry, HS2 sectors, war_risk_tier
- src/config/chokepoint-registry.ts: single source of truth for all 13
canonical chokepoints with displayName, relayName, portwatchName,
corridorRiskName, baselineId, shockModelSupported, routeIds, lat/lon
- src/config/hs2-sectors.ts: static dictionary for all 99 HS2 chapters
with category, shockModelSupported (true only for HS27), cargoType
- server/worldmonitor/supply-chain/v1/_chokepoint-ids.ts: migrated to
derive CANONICAL_CHOKEPOINTS from chokepoint-registry; no data duplication
- src/config/geo.ts + src/types/index.ts: added chokepointId field to
StrategicWaterway interface and all 13 STRATEGIC_WATERWAYS entries
- src/components/MapPopup.ts: switched chokepoint matching from fragile
name.toLowerCase() to direct chokepointId === id comparison
- server/worldmonitor/intelligence/v1/_shock-compute.ts: migrated from old
IDs (hormuz/malacca/babelm) to canonical IDs (hormuz_strait/malacca_strait/
bab_el_mandeb); same for CHOKEPOINT_LNG_EXPOSURE
- proto/worldmonitor/supply_chain/v1/supply_chain_data.proto: added
WarRiskTier enum + war_risk_tier field (field 16) on ChokepointInfo
- get-chokepoint-status.ts: populates warRiskTier from ChokepointConfig.threatLevel
via new threatLevelToWarRiskTier() helper (FREE field, no PRO gate)
* feat(supply-chain): Sprint 1 — country chokepoint exposure index + sector ring
S1.1: scripts/shared/country-port-clusters.json
~130 country → {nearestRouteIds, coastSide} mappings derived from trade route
waypoints; covers all 6 seeded Comtrade reporters plus major trading nations.
S1.2: scripts/seed-hs2-chokepoint-exposure.mjs
Daily cron seeder. Pure computation — reads country-port-clusters.json,
scores each country against CHOKEPOINT_REGISTRY route overlap, writes
supply-chain:exposure:{iso2}:{hs2}:v1 keys + seed-meta (24h TTL).
S1.3: RPC get-country-chokepoint-index (PRO-gated, request-varying)
- proto: GetCountryChokepointIndexRequest/Response + ChokepointExposureEntry
- handler: isCallerPremium gate; cachedFetchJson 24h; on-demand for any iso2
- cache-keys.ts: CHOKEPOINT_EXPOSURE_KEY(iso2, hs2) constant
- health.js: chokepointExposure SEED_META entry (48h threshold)
- gateway.ts: slow-browser cache tier
- service client: fetchCountryChokepointIndex() exported
S1.4: Chokepoint popup HS2 sector ring chart (PRO-gated)
Static trade-sector breakdown (IEA/UNCTAD estimates) per 9 major chokepoints.
SVG donut ring + legend shown for PRO users; blurred lockout + gate-hit
analytics for free users. Wired into renderWaterwayPopup().
🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.com/claude-code) + Compound Engineering v2.49.0
Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
* fix(tests): update energy-shock-v2 tests to use canonical chokepoint IDs
CHOKEPOINT_EXPOSURE and CHOKEPOINT_LNG_EXPOSURE keys were migrated from
short IDs (hormuz, malacca, babelm) to canonical registry IDs
(hormuz_strait, malacca_strait, bab_el_mandeb) in Sprint 0.
Test fixtures were not updated at the time; fix them now.
* fix(tests): update energy-shock-seed chokepoint ID to canonical form
VALID_CHOKEPOINTS changed to canonical IDs in Sprint 0; the seed test
that checks valid IDs was not updated alongside it.
* fix(cache-keys): reword JSDoc comment to avoid confusing bootstrap test regex
The comment "NOT in BOOTSTRAP_CACHE_KEYS" caused the bootstrap.test.mjs
regex to match the comment rather than the actual export declaration,
resulting in 0 entries found. Rephrase to "excluded from bootstrap".
* fix(supply-chain): address P1 review findings for chokepoint exposure index
- Add get-country-chokepoint-index to PREMIUM_RPC_PATHS (CDN bypass)
- Validate iso2/hs2 params before Redis key construction (cache injection)
- Fix seeder TTL to 172800s (2× interval) and extend TTL on skipped lock
- Fix CHOKEPOINT_EXPOSURE_SEED_META_KEY to match seeder write key
- Render placeholder sectors behind blur gate (DOM data leakage)
- Document get-country-chokepoint-index in widget agent system prompts
* fix(lint): resolve Biome CI failures
- Add biome.json overrides to silence noVar in HTML inline scripts,
disable linting for public/ vendor/build artifacts and pro-test/
- Remove duplicate NG and MW keys from country-port-clusters.json
- Use import attributes (with) instead of deprecated assert syntax
* fix(build): drop JSON import attribute — esbuild rejects `with` syntax
---------
Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
|
||
|
Elie Habib
|
f179e03127 |
perf(cache): CF edge caching + eliminate request storms (~25M/week saved) (#2829)
* perf(cache): add CF edge caching, eliminate request storms Three changes to reduce origin request volume: 1. Gateway cache tiers now include `public, s-maxage` so Cloudflare can cache API responses at edge (previously browser-only). Bumped 27 slow-seeded endpoints to appropriate tiers (static->daily for 6h+ seeds, slow->static for 2h seeds). 2. Population exposure: moved computation client-side. The server handler is pure math on 20 hardcoded countries, no reason for network calls. Eliminates ~17.7M requests/week (20 calls per page load -> 0). 3. Consumer prices: wrapped fetchAllMarketsOverview in a circuit breaker so the combined 8-market result is cached as a unit. Returning visitors within 30min hit localStorage instead of firing 8 separate API calls. * test: update shipping-rates tier assertion (static -> daily) * test: update cache tier assertions for three-tier caching design * fix(security): force slow-browser tier for premium endpoints Premium endpoints (PREMIUM_RPC_PATHS + ENDPOINT_ENTITLEMENTS) must not get public s-maxage headers. CF would cache authenticated responses and serve them without re-running auth/entitlement checks. Force these to slow-browser tier (browser-only max-age, no public/s-maxage). * fix(security): add list-market-implications to PREMIUM_RPC_PATHS PRO-only panel endpoint was missing from premium paths, allowing CF edge caching to serve authenticated responses to unauthenticated users. * chore: disable deduct-situation panel and endpoint Panel set to enabled:false in panels.ts, server handler returns early with provider:'disabled'. Code preserved for re-enabling later. * fix(security): suppress CDN-Cache-Control for premium endpoints too P1: slow-browser tier still had CDN-Cache-Control with public s-maxage, letting Vercel CDN cache premium responses for same-origin requests. Now CDN caching is fully disabled for premium endpoints. P2: revert server-side deduct-situation disable. Keep backend intact so the published API and correlation engine enrichment still work. Only the panel is disabled (enabled:false in panels.ts). |
||
|
Elie Habib
|
80b24d8686 |
feat(energy): shock model v2 — live flow ratios, coverage, limitations (V5-3) (#2810)
* feat(energy): shock model v2 — live flow ratios, coverage, limitations (V5-3) Replace static CHOKEPOINT_EXPOSURE multipliers with live PortWatch flow ratios from energy:chokepoint-flows:v1. Add proto fields 11-19: per-source coverage flags (jodi_oil_coverage, comtrade_coverage, iea_stocks_coverage, portwatch_coverage), coverage_level, limitations[], degraded, chokepoint_confidence, live_flow_ratio. Expand ISO2_TO_COMTRADE from 6 to 150+ countries via _comtrade-reporters.ts. Partial-coverage path proxies Gulf share at 40%. Unsupported countries return structured metadata instead of opaque string. data_available (field 9) preserved for backward compat. * fix(energy): correct chokepoint-flows key shape in shock handler (V5-3) energy:chokepoint-flows:v1 is a flat object keyed by canonicalId (hormuz_strait, bab_el_mandeb, suez, malacca_strait), not an object with a chokepoints[] array. The wrong shape caused degraded=true and liveFlowRatio=null for every request, silently falling back to static CHOKEPOINT_EXPOSURE multipliers even when live PortWatch data was available. Fixes: ChokepointEntry interface, typecast, cpEntry lookup, degraded check. * fix(energy): 4 review fixes for shock v2 handler (V5-3) 1. Cache key v1→v2: response shape changed (fields 11-19 added), old v1 cache entries would be served without new coverage fields. 2. IEA unknown vs zero: when ieaStocksCoverage=false, assessment now shows "IEA cover: unknown" instead of "0 days" to avoid conflating missing data with real zero stock. 3. liveFlowRatio 0.0 truthiness: changed `if (liveFlowRatio)` to `if (liveFlowRatio != null)` — a blocked chokepoint (ratio=0.0) is now shown, not hidden as "no live data". 4. Badge cleared on request start: coverageBadge is now reset before each shock compute request so a failed request doesn't leave the previous result's badge visible. * fix(energy): 3 remaining review fixes for shock v2 (V5-3) 1. Flow column: gate on portwatchCoverage (bool) not liveFlowRatio!=null — proto double defaults to 0, so null-check was always true and degraded responses showed a misleading "Flow: 0%" column. 2. Degraded cache TTL: 5min when degraded=true, 1h when live data present — limits how long stale degraded state persists after PortWatch recovers. 3. IEA anomaly cover days: zero daysOfCover when ieaStocksCoverage=false so anomalous IEA data no longer contributes to effectiveCoverDays; panel IEA cover row also gated on ieaStocksCoverage. * fix(energy): netExporter from anomalous IEA + zero-days panel display (V5-3) 1. netExporter: gated on ieaStocksCoverage — anomalous IEA rows no longer drive the net-exporter assessment branch when coverage=false. 2. Panel IEA cover: removed effectiveCoverDays>0 guard so a real zero (reserves exhausted under scenario) renders as "0 days" instead of being silently hidden as if there were no IEA data. * fix(energy): handle net-exporter sentinel (-1) in IEA cover panel row * fix(energy): NaN guard on flowRatio; optional live_flow_ratio; coverageLevel includes IEA+degraded * fix(energy): narrow Gulf-share proxy to Comtrade-only; NaN guard computeGulfShare; EMPTY liveFlowRatio undefined * fix(energy): tighten ieaStocksCoverage null guard; cache key varies by degraded state * fix(energy): harden IEA/PortWatch input validation; reduce shock cache TTL * fix(energy): add null narrowing for daysOfCover to satisfy strict TS |
||
|
Elie Habib
|
1937dbf844 |
feat(portwatch): Maritime Activity section in CountryDeepDivePanel (PR C) (#2805)
* feat(portwatch): Maritime Activity section in CountryDeepDivePanel (PR C) - Add get_country_port_activity.proto with PortActivityEntry + CountryPortActivityResponse messages - Register GetCountryPortActivity RPC in service.proto with HTTP GET /get-country-port-activity path - Run make generate to produce updated service_client.ts and service_server.ts - Implement get-country-port-activity.ts handler: countries guard, top-5 slice, trendDelta→tankerCallsPrev mapping - Register handler in intelligence handler.ts and gateway.ts slow cache tier - Add CountryPortActivityData interface and updateMaritimeActivity? method to CountryBriefPanel - Implement updateMaritimeActivity in CountryDeepDivePanel: table with 5 ports, anomaly badge, trend color, IMF PortWatch footer - Add getCountryPortActivity call in country-intel.ts with stale guard - Add maritime-activity CMD+K entry in commands.ts - 29 source-string assertions in tests/country-port-activity.test.mjs (all pass) Task: PR C * fix(portwatch): pass trendDelta directly, add runtime trend tests |
||
|
Elie Habib
|
b9b552cfcd |
feat(energy): product supply shock scenario RPC (Phase 4 PR C) (#2768)
* feat(energy): ComputeEnergyShockScenario RPC + country brief shock UI (Phase 4 PR C) Adds on-demand product supply shock scenario computation from JODI Oil, Comtrade and IEA stocks data. * fix(tests): add runtime + intelligence-client stubs to resilience harness ResilienceWidget imports @/services/runtime which dynamically imports @/services/widget-store. Without stubbing runtime, esbuild bundled the chain and failed on loadFromStorage not exported by the utils stub. * fix(energy): dataAvailable requires Comtrade data; gate shock widget on jodiOilAvailable - dataAvailable now requires both jodiOil and comtradeHasData, eliminating the contradiction of returning true with an "insufficient data" assessment - Collapsed redundant !hasComtradeData branch into the unified !dataAvailable guard - Gate renderShockScenarioWidget() behind data.jodiOilAvailable in CountryDeepDivePanel to avoid rendering a widget that will always return dataAvailable: false * fix(energy): zero-import hasData=false; extract pure shock-compute for real unit coverage - `totalImports === 0` in `computeGulfShare` now returns `hasData: false` so the handler correctly falls through to the "insufficient data" branch instead of treating empty Comtrade rows as usable Gulf-share data - Extract `clamp`, `computeGulfShare`, `computeEffectiveCoverDays`, `buildAssessment`, `GULF_PARTNER_CODES`, and `CHOKEPOINT_EXPOSURE` into `_shock-compute.ts` - Handler delegates pure computation to imported functions; `getGulfCrudeShare` still owns Redis I/O and calls `computeGulfShare(flows)` for the math - Tests now import the real functions via `.js` ESM extension; all 24 test cases exercise actual production logic (was previously reimplemented inline) * fix(energy): handle net-exporter in shock assessment; fix tautological chokepoint tests * fix(energy): move net-exporter branch before low-Gulf-share check in buildAssessment A net exporter with gulfCrudeShare < 0.1 (e.g. Norway) incorrectly received "low Gulf crude dependence" instead of "net oil exporter". Adds regression test to cover the ordering case. |
||
|
Elie Habib
|
e0dc630ed5 |
feat(energy): days of cover global view (Phase 4 PR B) (#2767)
* feat(energy): days of cover analysis key + EnergyComplexPanel oil stocks section - seed-iea-oil-stocks.mjs exports buildOilStocksAnalysis and writes energy:oil-stocks-analysis:v1 via afterPublish hook after main index - Rankings sorted by daysOfCover desc (net-exporters last), vsObligation, obligationMet, regional summaries (Europe/Asia-Pacific/North America) - EnergyComplexPanel.setOilStocksAnalysis() renders IEA member table with below-obligation badges, rank, days vs 90d obligation, regional summary rows - Health monitoring: seed-meta:energy:oil-stocks-analysis (42d maxStaleMin) - Gateway cache tier: static (monthly seed data) - 13 new tests covering sorting, exclusions, regional rollups, obligation logic * feat(energy): add proto + regenerate service for oil stocks analysis RPC - Add get_oil_stocks_analysis.proto with OilStocksAnalysisMember, OilStocksRegionalSummary sub-messages, and GetOilStocksAnalysisResponse - Use proto3 optional fields for nullable int32 (daysOfCover, vsObligation, avgDays, minDays) avoiding google.protobuf.wrappers complexity - Regenerate service_client.ts + service_server.ts via make generate - Update handler fallback and panel null-safety guards for optional fields - Regenerated OpenAPI docs include getOilStocksAnalysis endpoint * fix(energy): preserve oil-stocks-analysis TTL via extraKeys; fix seed-meta TTL to exceed health threshold - Move ANALYSIS_KEY into ANALYSIS_EXTRA_KEY in extraKeys so runSeed() extends its TTL on fetch failure or validation skip (was only written in afterPublish, leaving the key unprotected on the sad path) - afterPublish now writes only the seed-meta for ANALYSIS_KEY with a 50-day TTL (Math.max(86400*50, TTL_SECONDS)) — exceeds the health maxStaleMin threshold - Add optional metaTtlSeconds param to writeExtraKeyWithMeta() (backward-compat, defaults to existing 7-day value for all other callers) - Update health.js oilStocksAnalysis maxStaleMin from 42d to 50d to stay below the new seed-meta TTL and avoid false stale/missing reports * fix(energy): preserve seed-meta:oil-stocks-analysis TTL via extraKeys on seeder failure |
||
|
Elie Habib
|
5dbc72d7c6 |
feat(energy): GetCountryEnergyProfile RPC — aggregate Phase 1/2/2.5 data per country (#2747)
* feat(energy): GetCountryEnergyProfile RPC — aggregate Phase 1/2/2.5 data per country Add new intelligence RPC that reads 6 Redis keys in parallel (OWID mix, EU gas storage, electricity prices, JODI oil, JODI gas, IEA oil stocks) and returns a unified energy profile per ISO2 country code. All signals are optional with graceful omission on missing/null keys. Also register the new route in gateway.ts with slow cache tier. * fix(energy): convert gasLngShare from 0-1 fraction to 0-100 percentage to match all other share fields * fix(energy): add lpgImportsKbd, fix electricityAvailable source/date consistency, drop false US electricity claim * chore(proto): regenerate OpenAPI specs after energy profile field description update |
||
|
Fayez Bast
|
8609ad1384 |
feat: " climate disasters alerts seeders " (#2550)
* Revert "Revert "feat(climate): add climate disasters seed + ListClimateDisast…"
This reverts commit
|
||
|
Elie Habib
|
4e9f25631c |
feat(economic): add FAO Food Price Index panel (#2682)
* feat(economic): add FAO Food Price Index panel Adds a new panel tracking the FAO Global Food Price Index (FFPI) for the past 12 months, complementing existing consumer prices, fuel prices, and Big Mac Index trackers. - proto: GetFaoFoodPriceIndex RPC with 6-series response (Food, Cereals, Meat, Dairy, Oils, Sugar + MoM/YoY pct) - seeder: seed-fao-food-price-index.mjs with 90-day TTL (3× monthly), isMain guard, parseVal NaN safety, correct 13-point slice - handler/gateway: static tier RPC wired into economicHandler - bootstrap/health: bootstrapped as SLOW_KEY; maxStaleMin=86400 (60 days) - panel: SVG multi-line chart with 6 series, auto-scaled Y axis, headline with MoM/YoY indicators, info tooltip, bootstrap hydration - CMD+K: panel:fao-food-price-index with fao/ffpi/food keywords - Railway: fao-ffpi cron seeder service (0.5 vCPU, 0.5 GB, daily 08:45) - locales: full en.json keys for panel UI strings - ais-relay: faoFoodPriceIndex added to economic bootstrap context * fix(economic): add faoFoodPriceIndex to cache-keys.ts BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS * fix(economic): correct cron comment in fao seeder to reflect daily schedule |
||
|
Lucas Passos
|
b539dfd9c3 |
fix(resilience): enforce premium gating for resilience RPCs (#2661)
* fix(resilience): gate premium RPCs through the current gateway path Root cause: resilience RPCs added only to PREMIUM_RPC_PATHS would still be reachable from trusted browser origins because the gateway only forced API-key enforcement for tier-gated endpoints.\n\nAdd the resilience score/ranking routes to the shared premium path set, force legacy premium paths through the API-key-or-bearer gate, and extend gateway tests to cover both resilience endpoints for API-key and bearer flows. * test(resilience): cover free-plan and invalid bearer on resilience endpoints --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
02f55dc584 |
feat(climate): add ocean ice indicators seed and RPC (#2652)
* feat(climate): add ocean ice indicators seed and RPC * fix(review): restore MCP maxStaleMin, widen health threshold, harden sea level parser, type globe.gl shim - Restore get_climate_data _maxStaleMin to 2880 (was accidentally lowered to 1440) - Bump oceanIce SEED_META maxStaleMin from 1440 to 2880 (2× daily interval, tolerates one missed run) - Add fallback regex patterns for NASA sea level overlay HTML parsing - Replace globe.gl GlobeInstance `any` with typed interface (index sig stays `any` for Three.js compat) * fix(review): merge prior cache on partial failures, fix fallback regex, omit trend without baseline - P1: fetchOceanIceData() now reads prior cache and merges last-known-good indicators when any upstream source fails, preventing partial overwrites from erasing previously healthy data - P1: sea level fallback regex now requires "current" context to avoid matching the historical 1993 baseline rate instead of the current rate - P2: classifyArcticTrend() returns null (omitted from payload) when no climatology baseline exists, instead of misleadingly labeling as "average" - Added tests for all three fixes * fix(review): merge prior cache by source field group, not whole object Prior-cache merge was too coarse: Object.assign(payload, priorCache) reintroduced stale arctic_extent_anomaly_mkm2 and arctic_trend from prior cache when sea-ice succeeded but intentionally omitted those fields (no climatology baseline), and an unrelated source like OHC or sea level failed in the same run. Fix: define per-source field groups (seaIce, seaLevel, ohc, sst). Only fall back to prior cache fields for groups whose source failed entirely. When a source succeeds, only its returned fields appear in the payload, even if it omits fields it previously provided. Added test covering the exact combined case: sea-ice climatology unavailable + unrelated source failure + prior-cache merge enabled. --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Lucas Passos
|
4b67012260 |
feat(resilience): add service proto and stub handlers (#2657)
* feat(resilience): add service proto and stub handlers Add the worldmonitor.resilience.v1 proto package, generated client/server artifacts, edge routing, and zero-state handler stubs so the domain is deployable before the seed and scoring layers land. Validation: - PATH="/Users/lucaspassos/go/bin:/Users/lucaspassos/.codex/tmp/arg0/codex-arg06nbVvG:/Users/lucaspassos/.antigravity/antigravity/bin:/Users/lucaspassos/.local/bin:/Users/lucaspassos/.codeium/windsurf/bin:/Users/lucaspassos/Library/Python/3.12/bin:/Library/Frameworks/Python.framework/Versions/3.12/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/pkg/env/active/bin:/opt/pmk/env/global/bin:/Library/Apple/usr/bin:/opt/homebrew/bin:/Applications/Codex.app/Contents/Resources" make generate - PATH="/Users/lucaspassos/go/bin:/Users/lucaspassos/.codex/tmp/arg0/codex-arg06nbVvG:/Users/lucaspassos/.antigravity/antigravity/bin:/Users/lucaspassos/.local/bin:/Users/lucaspassos/.codeium/windsurf/bin:/Users/lucaspassos/Library/Python/3.12/bin:/Library/Frameworks/Python.framework/Versions/3.12/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/pkg/env/active/bin:/opt/pmk/env/global/bin:/Library/Apple/usr/bin:/opt/homebrew/bin:/Applications/Codex.app/Contents/Resources" npx tsx --test tests/route-cache-tier.test.mjs tests/edge-functions.test.mjs - npm run typecheck (fails on upstream Dodo/Clerk baseline) - npm run typecheck:api (fails on upstream vitest baseline) - npm run test:data (fails on upstream dodopayments-checkout baseline via tests/runtime-config-panel-visibility.test.mjs) * fix(resilience): add countryCode validation to get-resilience-score Throw ValidationError when countryCode is missing instead of silently returning a zero-state response with an empty string country code. * fix(resilience): validate countryCode format and mark required in spec - Trim whitespace and reject non-ISO-3166-1 alpha-2 codes to prevent cache pollution from malformed aliases (e.g. 'USA', ' us ', 'foobar') - Add required: true to proto QueryConfig so generated OpenAPI spec matches runtime validation behavior - Regenerated OpenAPI artifacts via make generate --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
9d94ad36aa |
feat(climate+health):add shared air quality seed and mirrored health (#2634)
* feat(climate+health):add shared air quality seed and mirrored health/climate RPCs * feat(climate+health):add shared air quality seed and mirrored health/climate RPCs * fix(air-quality): address review findings — TTL, seed-health, FAST_KEYS, shared meta - Raise CACHE_TTL from 3600 to 10800 (3× the 1h cron cadence; gold standard) - Add health:air-quality to api/seed-health.js SEED_DOMAINS so monitoring dashboard tracks freshness - Remove climateAirQuality and healthAirQuality from FAST_KEYS (large station payloads; load in slow batch) - Point climateAirQuality SEED_META to same meta key as healthAirQuality (same seeder run, one source of truth) * fix(bootstrap): move air quality keys to SLOW tier — large station payloads avoid critical-path batch * fix(air-quality): fix malformed OpenAQ URL and remove from bootstrap until panel exists - Drop deprecated first URL attempt (parameters=pm25, order_by=lastUpdated, sort=desc); use correct v3 params (parameters_id=2, sort_order=desc) directly — eliminates guaranteed 4xx retry cycle per page on 20-page crawl - Remove climateAirQuality and healthAirQuality from BOOTSTRAP_CACHE_KEYS, SLOW_KEYS, and BOOTSTRAP_TIERS — no panel consumes these yet; adding thousands of station records to every startup bootstrap is pure payload bloat - Remove normalizeAirQualityPayload helpers from bootstrap.js (no longer called) - Update service wrappers to fetch via RPC directly; re-add bootstrap hydration when a panel actually needs it * fix(air-quality): raise lock TTL to 3600s to cover 20-page crawl worst case 2 OpenAQ calls × 20 pages × (30s timeout × 3 attempts) = 3600s max runtime. Previous 600s TTL allowed concurrent cron runs on any degraded upstream. --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Sebastien Melki
|
9893bb1cf2 |
feat: Dodo Payments integration + entitlement engine & webhook pipeline (#2024)
* feat(14-01): install @dodopayments/convex and register component - Install @dodopayments/convex@0.2.8 with peer deps satisfied - Create convex/convex.config.ts with defineApp() and dodopayments component - Add TODO for betterAuth registration when PR #1812 merges Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(14-01): extend schema with 6 payment tables for Dodo integration - Add subscriptions table with status enum, indexes, and raw payload - Add entitlements table (one record per user) with features blob - Add customers table keyed by userId with optional dodoCustomerId - Add webhookEvents table for full audit trail (retained forever) - Add paymentEvents table for billing history (charge/refund) - Add productPlans table for product-to-plan mapping in DB - All existing tables (registrations, contactMessages, counters) unchanged Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(14-01): add auth stub, env helper, and Dodo env var docs - Create convex/lib/auth.ts with resolveUserId (returns test-user-001 in dev) and requireUserId (throws on unauthenticated) as sole auth entry points - Create convex/lib/env.ts with requireEnv for runtime env var validation - Append DODO_API_KEY, DODO_WEBHOOK_SECRET, DODO_PAYMENTS_WEBHOOK_SECRET, and DODO_BUSINESS_ID to .env.example with setup instructions - Document dual webhook secret naming (library vs app convention) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(14-02): add seed mutation for product-to-plan mappings - Idempotent upsert mutation for 5 Dodo product-to-plan mappings - Placeholder product IDs to be replaced after Dodo dashboard setup - listProductPlans query for verification and downstream use Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(14-02): populate seed mutation with real Dodo product IDs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(15-02): add plan-to-features entitlements config map - Define PlanFeatures type with 5 feature dimensions - Add PLAN_FEATURES config for 6 tiers (free through enterprise) - Export getFeaturesForPlan helper with free-tier fallback - Export FREE_FEATURES constant for default entitlements Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(15-01): add webhook HTTP endpoint with signature verification - Custom httpAction verifying Dodo webhook signatures via @dodopayments/core - Returns 400 for missing headers, 401 for invalid signature, 500 for processing errors - HTTP router at /dodopayments-webhook dispatches POST to webhook handler - Synchronous processing before 200 response (within Dodo 15s timeout) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(15-02): add subscription lifecycle handlers and entitlement upsert - Add upsertEntitlements helper (creates/updates per userId, no duplicates) - Add isNewerEvent guard for out-of-order webhook rejection - Add handleSubscriptionActive (creates subscription + entitlements) - Add handleSubscriptionRenewed (extends period + entitlements) - Add handleSubscriptionOnHold (pauses without revoking entitlements) - Add handleSubscriptionCancelled (preserves entitlements until period end) - Add handleSubscriptionPlanChanged (updates plan + recomputes entitlements) - Add handlePaymentEvent (records charge events for succeeded/failed) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(15-01): add idempotent webhook event processor with dispatch skeleton - processWebhookEvent internalMutation with idempotency via by_webhookId index - Switch dispatch for 7 event types: 5 subscription + 2 payment events - Stub handlers log TODO for each event type (to be implemented in Plan 03) - Error handling marks failed events and re-throws for HTTP 500 + Dodo retry Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(15-01): complete webhook endpoint plan - Update auto-generated api.d.ts with new payment module types - SUMMARY, STATE, and ROADMAP updated (.planning/ gitignored) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(15-03): wire subscription handlers into webhook dispatch - Replace 6 stub handler functions with imports from subscriptionHelpers - All 7 event types (5 subscription + 2 payment) dispatch to real handlers - Error handling preserves failed event status in webhookEvents table - Complete end-to-end pipeline: HTTP action -> mutation -> handler functions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore(15-04): install convex-test, vitest, and edge-runtime; configure vitest - Add convex-test, vitest, @edge-runtime/vm as dev dependencies - Create vitest.config.mts scoped to convex/__tests__/ with edge-runtime environment - Add test:convex and test:convex:watch npm scripts * test(15-04): add 10 contract tests for webhook event processing pipeline - Test all 5 subscription lifecycle events (active, renewed, on_hold, cancelled, plan_changed) - Test both payment events (succeeded, failed) - Test deduplication by webhook-id (same id processed only once) - Test out-of-order event rejection (older timestamp skipped) - Test subscription reactivation (cancelled -> active on same subscription_id) - Verify entitlements created/updated with correct plan features * fix(15-04): exclude __tests__ from convex typecheck convex-test uses Vite-specific import.meta.glob and has generic type mismatches with tsc. Tests run correctly via vitest; excluding from convex typecheck avoids false positives. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(16-01): add tier levels to PLAN_FEATURES and create entitlement query - Add tier: number to PlanFeatures type (0=free, 1=pro, 2=api, 3=enterprise) - Add tier values to all plan entries in PLAN_FEATURES config - Create convex/entitlements.ts with getEntitlementsForUser public query - Free-tier fallback for missing or expired entitlements Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(16-01): create Redis cache sync action and wire upsertEntitlements - Create convex/payments/cacheActions.ts with syncEntitlementCache internal action - Wire upsertEntitlements to schedule cache sync via ctx.scheduler.runAfter(0, ...) - Add deleteRedisKey() to server/_shared/redis.ts for explicit cache invalidation - Redis keys use raw format (entitlements:{userId}) with 1-hour TTL - Cache write failures logged but do not break webhook pipeline Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(16-02): add entitlement enforcement to API gateway - Create entitlement-check middleware with Redis cache + Convex fallback - Replace PREMIUM_RPC_PATHS boolean Set with ENDPOINT_ENTITLEMENTS tier map - Wire checkEntitlement into gateway between API key and rate limiting - Add raw parameter to setCachedJson for user-scoped entitlement keys - Fail-open on missing auth/cache failures for graceful degradation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(16-03): create frontend entitlement service with reactive ConvexClient subscription - Add VITE_CONVEX_URL to .env.example for frontend Convex access - Create src/services/entitlements.ts with lazy-loaded ConvexClient - Export initEntitlementSubscription, onEntitlementChange, getEntitlementState, hasFeature, hasTier, isEntitled - ConvexClient only loaded when userId available and VITE_CONVEX_URL configured - Graceful degradation: log warning and skip when Convex unavailable Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(16-04): add 6 contract tests for Convex entitlement query - Free-tier defaults for unknown userId - Active entitlements for subscribed user - Free-tier fallback for expired entitlements - Correct tier mapping for api_starter and enterprise plans - getFeaturesForPlan fallback for unknown plan keys Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(16-04): add 6 unit tests for gateway entitlement enforcement - getRequiredTier: gated vs ungated endpoint tier lookup - checkEntitlement: ungated pass-through, missing userId graceful degradation - checkEntitlement: 403 for insufficient tier, null for sufficient tier - Dependency injection pattern (_testCheckEntitlement) for clean testability - vitest.config.mts include expanded to server/__tests__/ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(17-01): create Convex checkout session action - DodoPayments component wraps checkout with server-side API key - Accepts productId, returnUrl, discountCode, referralCode args - Always enables discount code input (PROMO-01) - Forwards affiliate referral as checkout metadata (PROMO-02) - Dark theme customization for checkout overlay Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(17-03): create PricingSection component with tier cards and billing toggle - 4 tiers: Free, Pro, API, Enterprise with feature comparison - Monthly/annual toggle with "Save 17%" badge for Pro - Checkout buttons using Dodo static payment links - Pro tier visually highlighted with green border and "Most Popular" badge - Staggered entrance animations via motion - Referral code forwarding via refCode prop Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(17-01): extract shared ConvexClient singleton, refactor entitlements - Create src/services/convex-client.ts with getConvexClient() and getConvexApi() - Lazy-load ConvexClient via dynamic import to preserve bundle size - Refactor entitlements.ts to use shared client instead of inline creation - Both checkout and entitlement services will share one WebSocket connection Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(17-03): integrate PricingSection into App.tsx with referral code forwarding - Import and render PricingSection between EnterpriseShowcase and PricingTable - Pass refCode from getRefCode() URL param to PricingSection for checkout link forwarding - Update navbar CTA and TwoPathSplit Pro CTA to anchor to #pricing section - Keep existing waitlist form in Footer for users not ready to buy - Build succeeds with no new errors Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update generated files after main merge and pro-test build Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: prefix unused ctx param in auth stub to pass typecheck Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(17-04): add 4 E2E contract tests for checkout-to-entitlement flow - Test product plan seeding and querying (5 plans verified) - Test pro_monthly checkout -> webhook -> entitlements (tier=1, no API) - Test api_starter checkout -> webhook -> entitlements (tier=2, apiAccess) - Test expired entitlements fall back to free tier (tier=0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(17-02): install dodopayments-checkout SDK and create checkout overlay service - Install dodopayments-checkout@1.8.0 overlay SDK - Create src/services/checkout.ts with initCheckoutOverlay, openCheckout, startCheckout, showCheckoutSuccess - Dark theme config matching dashboard aesthetic (green accent, dark bg) - Lazy SDK initialization on first use - Fallback to /pro page when Convex is unavailable Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(17-02): wire locked panel CTAs and post-checkout return handling - Create src/services/checkout-return.ts for URL param detection and cleanup - Update Panel.ts showLocked() CTA to trigger Dodo overlay checkout (web path) - Keep Tauri desktop path opening URL externally - Add handleCheckoutReturn() call in PanelLayoutManager constructor - Initialize checkout overlay with success banner callback - Dynamic import of checkout module to avoid loading until user clicks Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(17-02): add Upgrade to Pro section in UnifiedSettings modal - Add upgrade section at bottom of settings tab with value proposition - Wire CTA button to open Dodo checkout overlay via dynamic import - Close settings modal before opening checkout overlay - Tauri desktop fallback to external URL - Conditionally show "You're on Pro" when user has active entitlement Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove unused imports in entitlement-check test to pass typecheck:api Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(17-01): guard missing DODO_PAYMENTS_API_KEY with warning instead of silent undefined Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(17-01): use DODO_API_KEY env var name matching Convex dashboard config Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(17-02): add Dodo checkout domains to CSP frame-src directive Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(17-03): use test checkout domain for test-mode Dodo products Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(18-01): shared DodoPayments config and customer upsert in webhook - Create convex/lib/dodo.ts centralizing DodoPayments instance and API exports - Refactor checkout.ts to import from shared config (remove inline instantiation) - Add customer record upsert in handleSubscriptionActive for portal session support Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(18-01): billing queries and actions for subscription management - Add getSubscriptionForUser query (plan status, display name, renewal date) - Add getCustomerByUserId and getActiveSubscription internal queries - Add getCustomerPortalUrl action (creates Dodo portal session via SDK) - Add changePlan action (upgrade/downgrade with proration via Dodo SDK) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(18-02): add frontend billing service with reactive subscription watch - SubscriptionInfo interface for plan status display - initSubscriptionWatch() with ConvexClient onUpdate subscription - onSubscriptionChange() listener pattern with immediate fire for late subscribers - openBillingPortal() with Dodo Customer Portal fallback - changePlan() with prorated_immediately proration mode Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(18-02): add subscription status display and Manage Billing button to settings - Settings modal shows plan name, status badge, and renewal date for entitled users - Status-aware colors: green (active), yellow (on_hold), red (cancelled/expired) - Manage Billing button opens Dodo Customer Portal via billing service - initSubscriptionWatch called at dashboard boot alongside entitlements - Import initPaymentFailureBanner for Task 3 wiring Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(18-02): add persistent payment failure banner for on_hold subscriptions - Red fixed-position banner at top of dashboard when subscription is on_hold - Update Payment button opens Dodo billing portal - Dismiss button with sessionStorage persistence (avoids nagging in same session) - Auto-removes when subscription returns to active (reactive via Convex) - Event listeners attached directly to DOM (not via debounced setContent) - Wired into panel-layout constructor alongside subscription watch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address code review — identity bridge, entitlement gating, fail-closed, env hygiene P0: Checkout-to-user identity bridge - Pass userId as metadata.wm_user_id in checkout sessions - Webhook resolveUserId: try metadata first, then customer table, then dev-only fallback - Fail closed in production when no user identity can be resolved P0: Unify premium gating to read Dodo entitlements - data-loader.ts: hasPremiumAccess() checks isEntitled() || API key - panels.ts: isPanelEntitled checks isEntitled() before API key fallback - panel-layout.ts: reload on entitlement change to unlock panels P1: Fail closed on unknown product IDs - resolvePlanKey throws on unmapped product (webhook retries) - getFeaturesForPlan throws on unknown planKey P1: Env var hygiene - Canonical DODO_API_KEY (no dual-name fallback in dodo.ts) - console.error on missing key instead of silent empty string P1: Fix test suite scheduled function errors - Guard scheduler.runAfter with UPSTASH_REDIS_REST_URL check - Tests skip Redis cache sync, eliminating convex-test write errors P2: Webhook rollback durability - webhookMutations: return error instead of rethrow (preserves audit row) - webhookHandlers: check mutation return for error indicator P2: Product ID consolidation - New src/config/products.ts as single source of truth - Panel.ts and UnifiedSettings.ts import from shared config P2: ConvexHttpClient singleton in entitlement-check.ts P2: Concrete features validator in schema (replaces v.any()) P2: Tests seed real customer mapping (not fallback user) P3: Narrow eslint-disable to typed interfaces in subscriptionHelpers P3: Real ConvexClient type in convex-client.ts P3: Better dev detection in auth.ts (CONVEX_IS_DEV) P3: Add VITE_DODO_ENVIRONMENT to .env.example Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(payments): security audit hardening — auth gates, webhook retry, transaction safety - Gate all public billing/checkout endpoints with resolveUserId(ctx) auth check - Fix webhook retry: record events after processing, not before; delete failed events on retry - Fix transaction atomicity: let errors propagate so Convex rolls back partial writes - Fix isDevDeployment to use CONVEX_IS_DEV (same as lib/auth.ts) - Add missing handlers: subscription.expired, refund.*, dispute.* - Fix toEpochMs silent fallback — now warns on missing billing dates - Use validated payload directly instead of double-parsing webhook body - Fix multi-sub query to prioritize active > on_hold > cancelled > expired - Change .unique() to .first() on customer lookups (defensive against duplicates) - Update handleSubscriptionActive to patch planKey/dodoProductId on existing subs - Frontend: portal URL validation, getProWidgetKey(), subscription cleanup on destroy - Make seedProductPlans internalMutation, console.log → console.warn for ops signals Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address code review — identity bridge, entitlement gating, fail-safe dev detection P0: convex/lib/auth.ts + subscriptionHelpers.ts — remove CONVEX_CLOUD_URL heuristic that could treat production as dev. Now uses ctx.auth.getUserIdentity() as primary auth with CONVEX_IS_DEV-only dev fallback. P0: server/gateway.ts + auth-session.ts — add bearer token (Clerk JWT) support for tier-gated endpoints. Authenticated users bypass API key requirement; userId flows into x-user-id header for entitlement check. Activated by setting CLERK_JWT_ISSUER_DOMAIN env var. P1: src/services/user-identity.ts — centralized getUserId() replacing scattered getProWidgetKey() calls in checkout.ts, billing.ts, panel-layout.ts. P2: src/App.ts — premium panel prime/refresh now checks isEntitled() alongside WORLDMONITOR_API_KEY so Dodo-entitled web users get data loading. P2: convex/lib/dodo.ts + billing.ts — move Dodo SDK config from module scope into lazy/action-scoped init. Missing DODO_API_KEY now throws at action boundary instead of silently capturing empty string. Tests: webhook test payloads now include wm_user_id metadata (production path). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address P0 access control + P1 identity bridge + P1 entitlement reload loop P0: Remove userId from public billing function args (getSubscriptionForUser, getCustomerPortalUrl, changePlan) — use requireUserId(ctx) with no fallback to prevent unauthenticated callers from accessing arbitrary user data. P1: Add stable anonymous ID (wm-anon-id) in user-identity.ts so createCheckout always passes wm_user_id in metadata. Breaks the infinite webhook retry loop for brand-new purchasers with no auth/localStorage identity. P1: Skip initial entitlement snapshot in onEntitlementChange to prevent reload loop for existing premium users whose shouldUnlockPremium() is already true from legacy signals (API key / wm-pro-key). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address P1 billing flow + P1 anon ID claim path + P2 daily-market-brief P1 — Billing functions wired end-to-end for browser sessions: - getSubscriptionForUser, getCustomerPortalUrl, changePlan now accept userId from args (matching entitlements.ts pattern) with auth-first fallback. Once Clerk JWT is wired into ConvexClient.setAuth(), the auth path will take precedence automatically. - Frontend billing.ts passes userId from getUserId() on all calls. - Subscription watch, portal URL, and plan change all work for browser users with anon IDs. P1 — Anonymous ID → account claim path: - Added claimSubscription(anonId) mutation to billing.ts — reassigns subscriptions, entitlements, customers, and payment events from an anonymous browser ID to the authenticated user. - Documented the anon ID limitation in user-identity.ts with the migration plan (call claimSubscription on first Clerk session). - Created follow-up issue #2078 for the full claim/migration flow. P2 — daily-market-brief added to hasPremiumAccess() block in data-loader.ts loadAllData() so it loads on general data refresh paths (was only in primeVisiblePanelData startup path). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: P0 lock down billing write actions + P2 fix claimSubscription logic P0 — Billing access control locked down: - getCustomerPortalUrl and changePlan converted to internalAction — not callable from the browser, closing the IDOR hole on write paths. - getSubscriptionForUser stays as a public query with userId arg (read-only, matching the entitlements.ts pattern — low risk). - Frontend billing.ts: portal opens generic Dodo URL, changePlan returns "not available" stub. Both will be promoted once Clerk auth is wired into ConvexClient.setAuth(). P2 — claimSubscription merge logic fixed: - Entitlement comparison now uses features.tier first, breaks ties with validUntil (was comparing only validUntil which could downgrade tiers). - Added Redis cache invalidation after claim: schedules deleteEntitlementCache for the stale anon ID and syncEntitlementCache for the real user ID. - Added deleteEntitlementCache internal action to cacheActions.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(billing): strip Dodo vendor IDs from public query response Remove dodoSubscriptionId and dodoProductId from getSubscriptionForUser return — these vendor-level identifiers aren't used client-side and shouldn't be exposed over an unauthenticated fallback path. Addresses koala73's Round 5 P1 review on #2024. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(billing): address koala review cleanup items (P2/P3) - Remove stale dodoSubscriptionId/dodoProductId from SubscriptionInfo interface (server no longer returns them) - Remove dead `?? crypto.randomUUID()` fallback in checkout.ts (getUserId() always returns a string via getOrCreateAnonId()) - Remove unused "failed" status variant and errorMessage from webhookEvents schema (no code path writes them) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add missing isProUser import and allow trusted origins for tier-gated endpoints The typecheck failed because isProUser was used in App.ts but never imported. The unit test failed because the gateway forced API key validation for tier-gated endpoints even from trusted browser origins (worldmonitor.app), where the client-side isProUser() gate controls access instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(gateway): require credentials for premium endpoints regardless of origin Origin header is spoofable — it cannot be a security boundary. Premium endpoints now always require either an API key or a valid bearer token (via Clerk session). Authenticated users (sessionUserId present) bypass the API key check; unauthenticated requests to tier-gated endpoints get 401. Updated test to assert browserNoKey → 401 instead of 200. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tests): add catch-all route to legacy endpoint allowlist The [[...path]].js Vercel catch-all route (domain gateway entry point) was missing from ALLOWED_LEGACY_ENDPOINTS in the edge function tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * revert: remove [[...path]].js from legacy endpoint allowlist This file is Vercel-generated and gitignored — it only exists locally, not in the repo. Adding it to the allowlist caused CI to fail with "stale entry" since the file doesn't exist in CI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(payment): apply design system to payment UI (light/dark mode) - checkout.ts: add light themeConfig for Dodo overlay + pass theme flag based on current document.dataset.theme; previously only dark was configured so light-mode users got Dodo's default white UI - UnifiedSettings: replace hardcoded dark hex values (#1a1a1a, #323232, #fff, #909090) in upgrade section with CSS var-driven classes so the panel respects both light and dark themes - main.css: add .upgrade-pro-section / .upgrade-pro-cta / .manage-billing-btn classes using var(--green), var(--bg), var(--surface), var(--border), etc. * fix(checkout): remove invalid theme prop from CheckoutOptions * fix: regenerate package-lock.json with npm 10 (matches CI Node 22) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(gateway): enforce pro role check for authenticated free users on premium paths Free bearer token holders with a valid session bypassed PREMIUM_RPC_PATHS because sessionUserId being set caused forceKey=false, skipping the role check entirely. Now explicitly checks bearer role after API key gate. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove unused getSecretState import from data-loader Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: gate Dodo Payments init behind isProUser() (same as Clerk) Entitlement subscription, subscription watch, checkout overlay, and payment banners now only initialize for isProUser() — matching the Clerk auth gate so only wm-pro-key / wm-widget-key holders see it. Also consolidates inline isEntitled()||getSecretState()||role checks in App.ts to use the centralized hasPremiumAccess() from panel-gating. Both gates (Clerk + Dodo) to be removed when ready for all users. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: retrigger workflows * chore: retrigger CI * fix(redis): use POST method in deleteRedisKey for consistency All other write helpers use POST; DEL was implicitly using GET via fetch default. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(quick-5): resolve all P1 security issues from koala73 review P1-1: Fail-closed entitlement gate (403 when no userId or lookup fails) P1-2: Checkout requires auth (removed client-supplied userId fallback) P1-3: Removed dual auth (PREMIUM_RPC_PATHS) from gateway, single entitlement path P1-4: Typed features validator in cacheActions (v.object instead of v.any) P1-5: Typed ConvexClient API ref (typeof api instead of Record<string,any>) P1-6: Cache stampede mitigation via request coalescing (_inFlight map) Round8-A: getUserId() returns Clerk user.id, hasUserIdentity() checks real identity Round8-B: JWT verification pinned to algorithms: ['RS256'] - Updated entitlement tests for fail-closed behavior (7 tests pass) - Removed userId arg from checkout client call - Added env-aware Redis key prefix (live/test) - Reduced cache TTL from 3600 to 900 seconds - Added 5s timeout to Redis fetch calls Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(quick-5): resolve all P2 review issues from koala73 review P2-1: dispute.lost now revokes entitlements (downgrades to free tier) P2-2: rawPayload v.any() documented with JSDoc (intentional: external schema) P2-3: Redis keys prefixed with live/test env (already in P1 commit) P2-4: 5s timeout on Redis fetch calls (already in P1 commit) P2-5: Cache TTL reduced from 3600 to 900 seconds (already in P1 commit) P2-6: CONVEX_IS_DEV warning logged at module load time (once, not per-call) P2-7: claimSubscription uses .first() instead of .unique() (race safety) P2-8: toEpochMs fallback accepts eventTimestamp (all callers updated) P2-9: hasUserIdentity() checks real identity (already in P1 commit) P2-10: Duplicate JWT verification removed (already in P1 commit) P2-11: Subscription queries bounded with .take(10) P2-12: Entitlement subscription exposes destroyEntitlementSubscription() Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(convex): resolve TS errors in http.ts after merge Non-null assertions for anyApi dynamic module references and safe array element access in timing-safe comparison. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tests): update gateway tests for fail-closed entitlement system Tests now reflect the new behavior where: - API key + no auth session → 403 (entitlement check requires userId) - Valid bearer + no entitlement data → 403 (fail-closed) - Free bearer → 403 (entitlements unavailable) - Invalid bearer → 401 (no session, forceKey kicks in) - Public routes → 200 (unchanged) The old tests asserted PREMIUM_RPC_PATHS + JWT role behavior which was removed per koala73's P1-3 review (dual auth elimination). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(review): resolve 4 P1 + 2 P2 issues from koala73 round-9 review P1 fixes: - API-key holders bypass entitlement check (were getting 403) - Browser checkout passes userId for identity bridge (ConvexClient has no setAuth yet, so createCheckout accepts optional userId arg) - /pro pricing page embeds wm_user_id in Dodo checkout URL metadata so webhook can resolve identity for first-time purchasers - Remove isProUser() gate from entitlement/billing init — all users now subscribe to entitlement changes so upgrades take effect immediately without manual page reload P2 fixes: - destroyEntitlementSubscription() called on teardown to clear stale premium state across SPA sessions (sign-out / identity change) - Convex queries prefer resolveUserId(ctx) over client-supplied userId; documented as temporary until Clerk JWT wired into ConvexClient.setAuth() Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(review): resolve 4 P1 + 1 P2 from koala73 round-10 review P1 — Checkout identity no longer client-controlled: - createCheckout HMAC-signs userId with DODO_PAYMENTS_WEBHOOK_SECRET - Webhook resolveUserId only trusts metadata when HMAC signature is valid; unsigned/tampered metadata is rejected - /pro raw URLs no longer embed wm_user_id (eliminated URL tampering) - Purchases without signed metadata get synthetic "dodo:{customerId}" userId, claimable later via claimSubscription() P1 — IDOR on Convex queries addressed: - Both getEntitlementsForUser and getSubscriptionForUser now reject mismatched userId when the caller IS authenticated (authedUserId != args.userId → return defaults/null) - Created internal getEntitlementsByUserId for future gateway use - Pre-auth fallback to args.userId documented with TODO(clerk-auth) P1 — Clerk identity bridge fixed: - user-identity.ts now uses getCurrentClerkUser() from clerk.ts instead of reading window.Clerk?.user (which was never assigned) - Signed-in Clerk users now correctly resolve to their Clerk user ID P1 — Auth modal available for anonymous users: - Removed isProUser() gate from setupAuthWidget() in App.ts - Anonymous users can now click premium CTAs → sign-in modal opens P2 — .take(10) subscription cap: - Bumped to .take(50) in both getSubscriptionForUser and getActiveSubscription to avoid missing active subscriptions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(review): resolve remaining P2/P3 feedback from koala73 + greptile on PR #2024 JWKS: consolidate duplicate singletons — server/_shared/auth-session.ts now imports the shared JWKS from server/auth-session.ts (eliminates redundant cold-start fetch). Webhook: remove unreachable retry branch — webhookEvents.status is always "processed" (inserted only after success, rolled back on throw). Dead else removed. YAGNI: remove changePlan action + frontend stub (no callers — plan changes use Customer Portal). Remove unused by_status index on subscriptions table. DRY: consolidate identical pro_monthly/pro_annual into shared PRO_FEATURES constant. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(test): read CLERK_JWT_ISSUER_DOMAIN lazily in getJWKS() — fixes CI bearer token tests The shared getJWKS() was reading the env var from a module-scope const, which freezes at import time. Tests set the env var in before() hooks after import, so getJWKS() returned null and bearer tokens were never verified — causing 401 instead of 403 on entitlement checks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(payments): resolve all blocking issues from review rounds 1-2 Data integrity: - Use .first() instead of .unique() on entitlements.by_userId to survive concurrent webhook retries without permanently crashing reads - Add typed paymentEventStatus union to schema; replace dynamic dispute status string construction with explicit lookup map - Document accepted 15-min Redis cache sync staleness bound - Document webhook endpoint URL in .env.example Frontend lifecycle: - Add listeners.clear() to destroyEntitlementSubscription (prevents stale closure reload loop on destroy/re-init cycles) - Add destroyCheckoutOverlay() to reset initialized flag so new layouts can register their success callbacks - Complete PanelLayoutManager.destroy() — add teardown for checkout overlay, payment failure banner, and entitlement change listener - Preserve currentState across destroy; add resetEntitlementState() for explicit reset (e.g. logout) without clearing on every cycle Code quality: - Export DEV_USER_ID and isDev from lib/auth.ts; remove duplicates from subscriptionHelpers.ts (single source of truth) - Remove DODO_PAYMENTS_API_KEY fallback from billing.ts - Document why two Dodo SDK packages coexist (component vs REST) * fix(payments): address round-3 review findings — HMAC key separation, auth wiring, shape guards - Introduce DODO_IDENTITY_SIGNING_SECRET separate from webhook secret (todo 087) Rotating the webhook secret no longer silently breaks userId identity signing - Wire claimSubscription to Clerk sign-in in App.ts (todo 088) Paying anonymous users now have their entitlements auto-migrated on first sign-in - Promote getCustomerPortalUrl to public action + wire openBillingPortal (todo 089) Manage Billing button now opens personalized portal instead of generic URL - Add rate limit on claimSubscription (todo 090) - Add webhook rawPayload shape guard before handler dispatch (todo 096) Malformed payloads return 200 with log instead of crashing the handler - Remove dead exports: resetEntitlementState, customerPortal wrapper (todo 091) - Fix let payload; implicit any in webhookHandlers.ts (todo 092) - Fix test: use internal.* for internalMutation seedProductPlans (todo 095) Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> * fix(payments): address round-4 P1 findings — input validation, anon-id cleanup - claimSubscription: replace broken rate-limit (bypassed for new users, false positives on renewals) with UUID v4 format guard + self-claim guard; prevents cross-user subscription theft via localStorage injection - App.ts: always remove wm-anon-id after non-throwing claim completion (not only when subscriptions > 0); adds optional chaining on result.claimed; prevents cold Convex init on every sign-in for non-purchasers Resolves todos 097, 098, 099, 100 * fix(payments): address round-5 review findings — regex hoist, optional chaining - Hoist ANON_ID_REGEX to module scope (was re-allocated on every call) - Remove /i flag — crypto.randomUUID() always produces lowercase - result.claimed accessed directly (non-optional) — mutation return is typed - Revert removeItem from !client || !api branch — preserve anon-id on infrastructure failure; .catch path handles transient errors * fix(billing): wire setAuth, rebind watches, revoke on dispute.lost, defer JWT, bound collect - convex-client.ts: wire client.setAuth(getClerkToken) so claimSubscription and getCustomerPortalUrl no longer throw 'Authentication required' in production - clerk.ts: expose clearClerkTokenCache() for force-refresh handling - App.ts: rebind entitlement + subscription watches to real Clerk userId on every sign-in (destroy + reinit), fixing stale anon-UUID watches post-claim - subscriptionHelpers.ts: revoke entitlement to 'free' on dispute.lost + sync Redis cache; previously only logged a warning leaving pro access intact after chargeback - gateway.ts: compute isTierGated before resolveSessionUserId; defer JWKS+RS256 verification to inside if (isTierGated) — eliminates JWT work on ~136 non-gated endpoints - billing.ts: .take(1000) on paymentEvents collect — safety bound preventing runaway memory on pathological anonymous sessions before sign-in Closes P1: setAuth never wired (claimSubscription always throws in prod) Closes P2: watch rebind, dispute.lost revocation, gateway perf, unbounded collect * fix(security): address P1 review findings from round-6 audit - Remove _debug block from validateApiKey that contained all valid API keys in envVarRaw/parsedKeys fields (latent full-key disclosure risk) - Replace {db: any} with QueryCtx in getEntitlementsHandler (Convex type safety) - Add pre-insert re-check in upsertEntitlements with OCC race documentation - Fix dispute.lost handler to use eventTimestamp instead of Date.now() for validUntil/updatedAt (preserves isNewerEvent out-of-order replay protection) - Extract getFeaturesForPlan("free") to const in dispute.lost (3x → 1x call) Closes todos #103, #106, #107, #108 * fix(payments): address round-6 open items — throw on shape guard, sign-out cleanup, stale TODOs P2-4: Webhook shape guards now throw instead of returning silently, so Dodo retries on malformed payloads instead of losing events. P3-2: Sign-out branch now destroys entitlement and subscription watches for the previous userId. P3: Removed stale TODO(clerk-auth) comments — setAuth() is wired. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(payments): address codex review — preserve listeners, honest TODOs, retry comment - destroyEntitlementSubscription/destroySubscriptionWatch no longer clear listeners — PanelLayout registers them once and they must survive auth transitions (sign-out → sign-in would lose the premium-unlock reload) - Restore TODO(auth) on entitlements/billing public queries — the userId fallback is a real trust gap, not just a cold-start race - Add comment on webhook shape guards acknowledging Dodo retry budget tradeoff vs silent event loss Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(types): restore non-null assertions in convex/http.ts after merge Main removed ! and as-any casts, but our branch's generated types make anyApi properties possibly undefined. Re-added to fix CI typecheck. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(types): resolve TS errors from rebase — cast internal refs, remove unused imports Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: regenerate package-lock.json — add missing uqr@0.1.2 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(payments): resolve P2/P3 review findings for PR #2024 - Bound and parallelize claimSubscription reads with Promise.all (4x queries -> single round trip; .collect() -> .take() to cap memory) - Add returnUrl allowlist validation in createCheckout to prevent open redirect - Make openBillingPortal return Promise<string | null> for agent-native callers - Extend isCallerPremium with Dodo entitlement tier check (tier >= 1 is premium, unifying Clerk role:pro and Dodo subscriber as two signals for the same gate) - Call resetEntitlementState() on sign-out to prevent entitlement state leakage across sessions (destroyEntitlementSubscription preserves state for reconnects; resetEntitlementState is the explicit sign-out nullifier) - Merge handlePaymentEvent + handleRefundEvent -> handlePaymentOrRefundEvent (type inferred from event prefix; eliminates duplicate resolveUserId call) - Remove _testCheckEntitlement DI export from entitlement-check.ts; inline _checkEntitlementCore into checkEntitlement; tests now mock getCachedJson - Collapse 4 duplicate dispute status tests into test.each - Fix stale entitlement variable name in claimSubscription return value * fix(payments): harden auth and checkout ownership * fix(gateway): tighten auth env handling * fix(gateway): use convex site url fallback * fix(app): avoid redundant checkout resume * fix(convex): cast alertRules internal refs for PR-branch generated types --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
b2bae30bd8 |
Add climate news seed and ListClimateNews RPC (#2532)
* Add climate news seed and ListClimateNews RPC * Wire climate news into bootstrap and fix generated climate stubs * fix(climate): align seed health interval and parse Atom entries per feed * fix(climate-news): TTL 90min, retry timer on failure, named cache key constant - CACHE_TTL: 1800 to 5400 (90min = 3x 30-min relay interval, gold standard) - ais-relay: add 20-min retry timer on subprocess failure; clear on success - cache-keys.ts: export CLIMATE_NEWS_KEY named constant - list-climate-news.ts: import CLIMATE_NEWS_KEY instead of hard-coding string --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
bb4f8dcb12 |
feat(climate): add WMO normals seeding and CO2 monitoring (#2531)
* feat(climate): add WMO normals seeding and CO2 monitoring * fix(climate): skip missing normals per-zone and align anomaly tooltip copy * fix(climate): remove normals from bootstrap and harden health/cache key wiring * feat(climate): version anomaly cache to v2, harden seed freshness, and align CO2/normal baselines |
||
|
Elie Habib
|
ae4010a795 |
Revert "feat(climate): add climate disasters seed + ListClimateDisasters RPC with snake_case Redis payload (#2535)" (#2544)
This reverts commit
|
||
|
Fayez Bast
|
e2dea9440d | feat(climate): add climate disasters seed + ListClimateDisasters RPC with snake_case Redis payload (#2535) | ||
|
Elie Habib
|
8aee4d340e |
feat(intelligence): GetCountryRisk RPC + MCP tool for per-country risk scores (#2502)
* feat(intelligence): GetCountryRisk RPC for per-country risk intelligence Adds a new fast Redis-read RPC that consolidates CII score, travel advisory level, and OFAC sanctions exposure into a single per-country response. Replaces the need to call GetRiskScores (all-countries) and filter client-side. Wired to MCP as get_country_risk tool (no LLM, ~200ms, good for agent screening). - proto/intelligence/v1/get_country_risk.proto (new) - server/intelligence/v1/get-country-risk.ts (reads 3 pre-seeded Redis keys) - gateway.ts: slow cache tier - api/mcp.ts: RpcToolDef with 8s timeout - tests/mcp.test.mjs: update tool count 27→28 * fix(intelligence): upstream-unavailable signal, fetchedAt from CII, drop redundant catch P1: return upstreamUnavailable:true when all Redis reads are null — prevents CDN from caching false-negative sanctions/risk responses during Redis outages. P2: fetchedAt now uses cii.computedAt (actual data age) instead of request time. P2: removed redundant .catch(() => null) — getCachedJson already swallows errors. * fix(intelligence): accurate OFAC counts and country names for GetCountryRisk P1: sanctions:pressure:v1.countries is a top-12 slice — switch to a new sanctions:country-counts:v1 key (ISO2→count across ALL 40K+ OFAC entries). Written by seed-sanctions-pressure.mjs in afterPublish alongside entity index. P1: trigger upstreamUnavailable:true when sanctions key alone is missing, preventing false-negative sanctionsActive:false from being cached by CDN. P2: advisory seeder now writes byCountryName (ISO2→display name) derived from country-names.json reverse map. Handler uses it as fallback so countries outside TIER1_COUNTRIES (TH, CO, BD, IT...) get proper names. |
||
|
Elie Habib
|
f482c13d6b |
fix(gateway): gate CDN-Cache-Control on trusted origin (#2496)
* fix(gateway): gate CDN-Cache-Control on trusted origin to close cache bypass No-origin server-side requests (external scrapers) no longer receive a CDN-Cache-Control header, so Vercel CDN never caches their responses. Every request without a trusted origin must reach the edge function, ensuring validateApiKey() always runs and returns 401 for unauthenticated callers. Trusted origins (worldmonitor.app, Vercel previews, Tauri) are unchanged — CDN caching behaviour and hit rates for the app itself are unaffected. * docs: external API usage learnings from aeris-AgentR and HOPEFX * docs: fix markdown lint (blanks around lists) * docs: move external-learnings to separate PR |
||
|
Elie Habib
|
aa3e84f0ab |
feat(economic): Economic Stress Composite Index panel (FRED 6-series, 0-100 score) (#2461)
* feat(economic): Economic Stress Composite Index panel (FRED 6-series, 0-100 score) - Add T10Y3M and STLFSI4 to FRED_SERIES in seed-economy.mjs and ALLOWED_SERIES - Create proto message GetEconomicStressResponse with EconomicStressComponent - Register GetEconomicStress RPC in EconomicService (GET /api/economic/v1/get-economic-stress) - Add seed-economic-stress.mjs: reads 6 pre-seeded FRED keys via Redis pipeline, computes weighted composite score (0-100) with labels Low/Moderate/Elevated/Severe/Critical - Create server handler get-economic-stress.ts reading from economic:stress-index:v1 - Register economicStress in BOOTSTRAP_CACHE_KEYS (both cache-keys.ts and api/bootstrap.js) as slow tier - Add gateway.ts cache tier entry (slow) for new RPC route - Create EconomicStressPanel.ts: composite score header, gradient needle bar, 2x3 component grid with score bars, desktop notification on threshold cross (>=70, >=85) - Wire economic-stress panel in panels.ts (all 4 variants), panel-layout.ts, and data-loader.ts - Regenerate OpenAPI docs and TypeScript client/server types * fix(economic-stress): null for missing FRED data + tech variant panel - Add 'economic-stress' panel to TECH_PANELS defaults (was missing, only appeared in full/finance/commodity variants) - Seed: write rawValue: null + missing: true when no valid FRED observation found, preventing zero-valued yield curve/bank spread readings from being conflated with missing data - Proto: add missing bool field to EconomicStressComponent message; regenerate client/server types + OpenAPI docs - Server handler: propagate missing flag from Redis; pass rawValue: 0 on wire when missing to satisfy proto double type - Panel: guard on c.missing (not rawValue === 0) to show grey N/A card with no score bar for unavailable components * fix(economic-stress): add purple Critical zone to gradient bar Update gradient stops to match the 5 equal tier boundaries (0-20-40-60-80-100), adding the #8e44ad purple stop at 80% so scores 80-100 render as Critical purple instead of plain red. |
||
|
Elie Habib
|
d01469ba9c |
feat(aviation): add SearchGoogleFlights and SearchGoogleDates RPCs (#2446)
* feat(aviation): add SearchGoogleFlights and SearchGoogleDates RPCs
Port Google Flights internal API from fli Python library as two new
aviation service RPCs routed through the Railway relay.
- Proto: SearchGoogleFlights and SearchGoogleDates messages and RPCs
- Relay: handleGoogleFlightsSearch and handleGoogleFlightsDates handlers
with JSONP parsing, 61-day chunking for date ranges, cabin/stops/sort mappers
- Server handlers: forward params to relay google-flights endpoints
- gateway.ts: no-store for flights, medium cache for dates
* feat(mcp): expose search_flights and search_flight_prices_by_date tools
* test(mcp): update tool count to 24 after adding search_flights and search_flight_prices_by_date
* fix(aviation): address PR review issues in Google Flights RPCs
P1: airline filtering — use gfParseAirlines() in relay (handles comma-joined
string from codegen) and parseStringArray() in server handlers
P1: partial chunk failure now sets degraded: true instead of silently
returning incomplete data as success; relay includes partial: true flag
P2: round-trip date search validates trip_duration > 0 before proceeding;
returns 400 when is_round_trip=true and duration is absent/zero
P2: relay mappers accept user-friendly aliases ('0'/'1' for max_stops,
'price'/'departure' for sort_by) alongside symbolic enum values;
MCP tool docs updated to match
* fix(aviation): use cachedFetchJson in searchGoogleDates for stampede protection
Medium cache tier (10 min) requires Redis-level coalescing to prevent
concurrent requests from all hitting the relay before cache warms.
Cache key includes all request params (sorted airlines for stable keys).
* fix(aviation): always use getAll() for airlines in relay; add multi-airline tests
The OR short-circuit (get() || getAll()) meant get() returned the first
airline value (truthy), so getAll() never ran and only one airline was
forwarded to Google. Fix: unconditionally use getAll().
Tests cover: multi-airline repeated params, single airline, empty array,
comma-joined string from codegen, partial degraded flag propagation.
|
||
|
Elie Habib
|
1e1f377078 |
feat(panels): Disease Outbreaks, Shipping Stress, Social Velocity, nuclear test site enrichment (#2375)
* feat(panels): Disease Outbreaks, Shipping Stress, Social Velocity, nuclear test site monitoring - Add HealthService proto with ListDiseaseOutbreaks RPC (WHO + ProMED RSS) - Add GetShippingStress RPC to SupplyChainService (Yahoo Finance carrier ETFs) - Add GetSocialVelocity RPC to IntelligenceService (Reddit r/worldnews + r/geopolitics) - Enrich earthquake seed with Haversine nuclear test-site proximity scoring - Add 5 nuclear test sites to NUCLEAR_FACILITIES (Punggye-ri, Lop Nur, Novaya Zemlya, Nevada NTS, Semipalatinsk) - Add shipping stress + social velocity seed loops to ais-relay.cjs - Add seed-disease-outbreaks.mjs Railway cron script - Wire all new RPCs: edge functions, handlers, gateway cache tiers, health.js STANDALONE_KEYS/SEED_META * fix(relay): apply gold standard retry/TTL-extend pattern to shipping-stress and social-velocity seeders * fix(review): address all PR #2375 review findings - health.js: shippingStress maxStaleMin 30→45 (3x interval), socialVelocity 20→30 (3x interval) - health.js: remove shippingStress/diseaseOutbreaks/socialVelocity from ON_DEMAND_KEYS (relay/cron seeds, not on-demand) - cache-keys.ts: add shippingStress, diseaseOutbreaks, socialVelocity to BOOTSTRAP_CACHE_KEYS - ais-relay.cjs: stressScore formula 50→40 (neutral market = moderate, not elevated) - ais-relay.cjs: fetchedAt Date.now() (consistent with other seeders) - ais-relay.cjs: deduplicate cross-subreddit article URLs in social velocity loop - seed-disease-outbreaks.mjs: WHO URL → specific DON RSS endpoint (not dead general news feed) - seed-disease-outbreaks.mjs: validate() requires outbreaks.length >= 1 (reject empty array) - seed-disease-outbreaks.mjs: stable id using hash(link) not array index - seed-disease-outbreaks.mjs: RSS regexes use [\s\S]*? for CDATA multiline content - seed-earthquakes.mjs: Lop Nur coordinates corrected (41.39,89.03 not 41.75,88.35) - seed-earthquakes.mjs: sourceVersion bumped to usgs-4.5-day-nuclear-v1 - earthquake.proto: fields 8-11 marked optional (distinguish not-enriched from enriched=false/0) - buf generate: regenerate seismology service stubs * revert(cache-keys): don't add new keys to bootstrap without frontend consumers * fix(panels): address all P1/P2/P3 review findings for PR #2375 - proto: add INT64_ENCODING_NUMBER annotation + sebuf import to get_shipping_stress.proto (run make generate) - bootstrap: register shippingStress (fast), socialVelocity (fast), diseaseOutbreaks (slow) in api/bootstrap.js + cache-keys.ts - relay: update WIDGET_SYSTEM_PROMPT with new bootstrap keys and live RPCs for health/supply-chain/intelligence - seeder: remove broken ProMED feed URL (promedmail.org/feed/ returns HTML 404); add 500K size guard to fetchRssItems; replace private COUNTRY_CODE_MAP with shared geo-extract.mjs; remove permanently-empty location field; bump sourceVersion to who-don-rss-v2 - handlers: remove dead .catch from all 3 new RPC handlers; fix stressLevel fallback to low; fix fetchedAt fallback to 0 - services: add fetchShippingStress, disease-outbreaks.ts, social-velocity.ts with getHydratedData consumers |
||
|
Elie Habib
|
9480b547d5 |
feat(feeds): US Natural Gas Storage weekly seeder (EIA NW2_EPG0_SWO_R48_BCF) (#2353)
* feat(feeds): US Natural Gas Storage seeder via EIA (NW2_EPG0_SWO_R48_BCF)
Adds weekly EIA natural gas working gas storage for the Lower-48 states
(series NW2_EPG0_SWO_R48_BCF, in Bcf), mirroring the crude inventories
pattern exactly. Companion dataset to EU gas storage (GIE AGSI+).
- proto: GetNatGasStorage RPC + NatGasStorageWeek message
- seed-economy.mjs: fetchNatGasStorage() in Promise.allSettled, writes
economic:nat-gas-storage:v1 with 21-day TTL (3x weekly cadence)
- server handler: getNatGasStorage reads seeded key from Redis
- gateway: /api/economic/v1/get-nat-gas-storage → static tier
- health.js: BOOTSTRAP_KEYS + SEED_META (14-day maxStaleMin)
- bootstrap.js: KEYS + SLOW_KEYS
- cache-keys.ts: BOOTSTRAP_CACHE_KEYS + BOOTSTRAP_TIERS (slow)
* feat(feeds): add fetchNatGasStorageRpc consumer in economic service
Adds getHydratedData('natGasStorage') consumer required by bootstrap
key registry test, plus circuit breaker and RPC wrapper mirroring
fetchCrudeInventoriesRpc pattern.
|
||
|
Elie Habib
|
4438ef587f |
feat(feeds): ECB CISS European financial stress index (#2278) (#2334)
* feat(feeds): ECB CISS European financial stress seeder + GetEuFsi RPC (#2278) - New seed-fsi-eu.mjs fetches ECB CISS (0-1 systemic stress index for Euro area) via SDMX-JSON REST API (free, no auth); TTL=604800s (7d, weekly data cadence) - New GetEuFsi RPC in EconomicService with proto + handler; cache tier: slow - FSIPanel now shows EU CISS gauge below US FSI with label thresholds: Low<0.2, Moderate<0.4, Elevated<0.6, High>=0.6 - Registered economic:fsi-eu:v1 in health.js BOOTSTRAP_KEYS + SEED_META, bootstrap.js, cache-keys.ts BOOTSTRAP_TIERS; hydrated via getHydratedData('euFsi') - All 2348 test:data tests pass; typecheck + typecheck:api clean * fix(ecb-ciss): address code review findings on PR #2334 - Raise FSI_EU_TTL from 604800s (7d) to 864000s (10d) to match other weekly seeds (bigmac, groceryBasket, fuelPrices) and provide a 3-day buffer against cron-drift or missed Saturday runs - Format latestDate via toLocaleDateString() in FSIPanel CISS section instead of displaying the raw ISO string (e.g. "2025-04-04") * fix(ecb-ciss): address Greptile review comments on PR #2334 - Fix misleading "Daily frequency" comment in seed-fsi-eu.mjs (SDMX uses 'D' series key but only Friday/weekly observations are present) - Replace latestValue > 0 guards with Number.isFinite() in FSIPanel.ts so a valid CISS reading of exactly 0 is not incorrectly excluded * chore: regenerate proto outputs after rebase |
||
|
Elie Habib
|
1d0846fa98 |
feat(feeds): ECB Euro Area yield curve seeder (#2276) (#2330)
* feat(feeds): ECB Euro Area yield curve seeder — EU complement to US Treasury curve (#2276) - Add scripts/seed-yield-curve-eu.mjs: fetches ECB AAA sovereign spot rates (1Y-30Y) via ECB Data Portal SDMX-JSON, writes to economic:yield-curve-eu:v1, TTL=259200s (3x daily interval) - Add proto/worldmonitor/economic/v1/get_eu_yield_curve.proto + GetEuYieldCurve RPC - Add server/worldmonitor/economic/v1/get-eu-yield-curve.ts handler (reads from Redis cache) - Wire GetEuYieldCurve into handler.ts and gateway.ts (daily cache tier) - Update YieldCurvePanel.ts: fetches EU curve in parallel with US FRED data, overlays green dashed line on chart with ECB AAA legend entry - Update api/health.js: add euYieldCurve to BOOTSTRAP_KEYS + SEED_META (maxStaleMin=2880, daily seed) - Re-run buf generate to update generated client/server TypeScript types * fix(generated): restore @ts-nocheck in economic generated files after buf regeneration * fix(ecb-yield-curve): address Greptile review comments on PR #2330 Include priorValues in allValues for yMin/yMax scale computation so prior US curve polyline is never clipped outside the chart area. * fix(generated): regenerate OpenAPI docs after rebase conflict resolution Re-run make generate to fix schema ordering in EconomicService.openapi.yaml and .json after manual conflict resolution introduced ordering inconsistencies. * chore: regenerate proto outputs after rebase |
||
|
Elie Habib
|
b6847e5214 |
feat(feeds): GIE AGSI+ EU gas storage seeder (#2281) (#2339)
* feat(feeds): GIE AGSI+ EU gas storage seeder — European energy security indicator (#2281) - New scripts/seed-gie-gas-storage.mjs: fetches EU aggregate gas storage fill % from GIE AGSI+ API, computes 1-day change, trend (injecting/withdrawing/stable), and days-of-consumption estimate; TTL=259200s (3x daily); isMain guard; CHROME_UA; validates fillPct in (0,100]; graceful degradation when GIE_API_KEY absent - New proto/worldmonitor/economic/v1/get_eu_gas_storage.proto + GetEuGasStorage RPC wired into EconomicService - New server/worldmonitor/economic/v1/get-eu-gas-storage.ts handler (reads seeded Redis key) - api/health.js: BOOTSTRAP_KEYS + SEED_META (maxStaleMin=2880, 2x daily cadence) - api/bootstrap.js: euGasStorage key in SLOW_KEYS bucket - Regenerated src/generated/ + docs/api/ via make generate * fix(feeds): wire euGasStorage into cache-keys, gateway tier, and test PENDING_CONSUMERS - server/_shared/cache-keys.ts: add euGasStorage → economic:eu-gas-storage:v1 (slow tier) - server/gateway.ts: add /api/economic/v1/get-eu-gas-storage → slow RPC_CACHE_TIER - tests/bootstrap.test.mjs: add euGasStorage to PENDING_CONSUMERS (no frontend panel yet) * fix(gie-gas-storage): normalize seededAt to string to match proto int64 contract Proto int64 seeded_at maps to string in JS; seed was writing Date.now() (number). Fix seed to write String(Date.now()) and add handler-side normalization for any stale Redis entries that may have the old numeric format. * fix(feeds): coerce nullable fillPctChange1d/gasDaysConsumption to 0 (#2281) Greptile P1: both fields could be null (single data-point run or missing volume) but the proto interface declares them as non-optional numbers. Seed script now returns 0 instead of null; handler defensively coerces nulls from older cached blobs via nullish coalescing. Dead null-guard on trend derivation also removed. |
||
|
Elie Habib
|
c69a13a1a0 |
feat(feeds): Eurostat per-country economic data seeder (#2282) (#2340)
* feat(feeds): Eurostat per-country CPI/unemployment/GDP seeder for 10 EU member states (#2282) * fix(feeds): address Greptile P1/P2 issues in eurostat seed parser - P1: fix sparse-index iteration bug in parseEurostatResponse; loop now iterates over Object.keys(values) directly instead of using key count as sequential bound, correctly handling non-zero-based sparse indexes - P2: remove unused dimIdx, geoStride, timeStride, totalSize variables - P2: batch country fetches (3 at a time) to reduce peak Eurostat concurrency from 30 to 9 simultaneous requests * chore: regenerate EconomicService openapi JSON after rebase |
||
|
Elie Habib
|
b5faffb341 |
feat(feeds): ECB reference FX rates seeder (#2280) (#2337)
* feat(feeds): ECB daily reference FX rates seeder -- EUR/USD/GBP/JPY/CHF (#2280) - New scripts/seed-ecb-fx-rates.mjs: fetches EUR/USD/GBP/JPY/CHF/CAD/AUD/CNY daily from ECB Data Portal (no API key required) - New getEcbFxRates RPC in economic service (proto + handler + gateway slow tier) - api/health.js: ecbFxRates in BOOTSTRAP_KEYS + SEED_META (2880min maxStale) - api/bootstrap.js: ecbFxRates registered as SLOW_KEYS bootstrap entry - TTL: 259200s (3x daily interval), isMain guard, CHROME_UA, validate>=3 pairs * fix(feeds): register ecbFxRates in cache-keys.ts + add frontend hydration consumer * fix(ecb-fx-rates): use dynamic CURRENCY dim position in series key parsing CURRENCY is at index 1 in EXR series keys (FREQ:CURRENCY:CURRENCY_DENOM:EXR_TYPE:EXR_SUFFIX). Using hardcoded keyParts[0] always read the FREQ field (always "0"), so all series resolved to currencyCodes[0] (AUD) and only 1 pair was written to Redis instead of all 7. Fix: find CURRENCY position via findIndex() and use keyParts[currencyDimPos] to extract the correct per-series currency index. * fix(feeds): hoist obs-dimension lookup + fix ECB UTC publication time - Hoist obsPeriods/timeDim/timeValues above the series loop (loop-invariant, P2) - Fix updatedAt timestamp: ECB publishes at 16:00 CET = 14:00 UTC (not 16:00 UTC), use T14:00:00Z (P2) * chore(generated): fix EcbFxRate schema ordering in EconomicService openapi.json buf generate places EcbFxRate alphabetically before EconomicEvent; align committed file with code-generator output so proto freshness hook passes. |