mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
main
25 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
a04c53fe26 |
fix(build): pin sebuf plugin via PATH in make generate (#3371)
* fix(build): pin sebuf plugin via PATH in `make generate`
Without this, developers who have an older sebuf protoc-gen-openapiv3
binary installed via a package manager (Homebrew ships v0.7.0 at
/opt/homebrew/bin) hit this failure on a fresh `make generate`:
Failure: file ".../AviationService.openapi.yaml" was generated
multiple times: once by plugin "protoc-gen-openapiv3" and again
by plugin "protoc-gen-openapiv3"
Root cause: `buf.gen.yaml` declares three `protoc-gen-openapiv3`
invocations — the default yaml, a `format=json` variant, and a
`bundle_only=true, strategy: all` unified bundle. Sebuf v0.11.x
honors both `format=json` (emits .json extension) and `bundle_only=true`
(skips per-service emission), so the three invocations write distinct
files. Sebuf v0.7.x does NOT honor either option — it silently emits
the same per-service .yaml filenames from all three plugins and buf
rejects the collision.
`Makefile: install-plugins` installs v0.11.1 (SEBUF_VERSION) to
$HOME/go/bin. But the `generate` target doesn't prepend that to PATH,
so `which protoc-gen-openapiv3` resolves to the stale Homebrew binary
for anyone with both installed.
Verified by `go version -m`:
/opt/homebrew/bin/protoc-gen-openapiv3 — mod sebuf v0.7.0
/Users/eliehabib/go/bin/protoc-gen-openapiv3 — mod sebuf v0.11.1
Fix: prepend $$HOME/go/bin to PATH in the `generate` recipe. Matches
what .husky/pre-push:151-153 already does before invoking this target,
so CI and local behavior converge. No sebuf upstream bug.
* fix(build): follow GOBIN-then-GOPATH/bin for plugin PATH prefix
Reviewer (Codex) on PR #3371: the previous patch hardcoded
\$HOME/go/bin, which is only the default fallback when GOBIN is unset
AND GOPATH defaults to \$HOME/go. On machines with a custom GOBIN or
a non-default GOPATH, `go install` targets a different directory — so
hardcoding \$HOME/go/bin can force a stale binary from there to win
over the freshly-installed SEBUF_VERSION sitting at the actual install
location.
Fix: resolve the install dir the same way `go install` does:
GOBIN first, then GOPATH/bin.
Shell expression: `go env GOBIN` returns an empty string (exit 0) when
unset, so `||` alone doesn't cascade. Using explicit `[ -n "$gobin" ]`
instead.
Also dropped the misleading comment that claimed the pre-push hook
used the same rule — it still hardcodes \$HOME/go/bin. Called that out
in a note, but left the hook alone because its PATH prepend is
belt-and-suspenders (only matters for locating `buf` itself; the
Makefile's own recipe-level prepend decides plugin resolution).
Verified on a machine with empty GOBIN:
resolved → /Users/eliehabib/go/bin
And \`make generate\` succeeds without manual PATH overrides.
* fix(build): use first GOPATH entry for plugin PATH prefix
Reviewer (Codex) on commit
|
||
|
Sebastien Melki
|
e68a7147dd |
chore(api): sebuf migration follow-ups (post-#3242) (#3287)
* chore(api-manifest): rewrite brief-why-matters reason as proper internal-helper justification Carried in from #3248 merge as a band-aid (called out in #3242 review followup checklist item 7). The endpoint genuinely belongs in internal-helper — RELAY_SHARED_SECRET-bearer auth, cron-only caller, never reached by dashboards or partners. Same shape constraint as api/notify.ts. Replaces the apologetic "filed here to keep the lint green" framing with a proper structural justification: modeling it as a generated service would publish internal cron plumbing as user-facing API surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lint): premium-fetch parity check for ServiceClients (closes #3279) Adds scripts/enforce-premium-fetch.mjs — AST-walks src/, finds every `new <ServiceClient>(...)` (variable decl OR `this.foo =` assignment), tracks which methods each instance actually calls, and fails if any called method targets a path in src/shared/premium-paths.ts PREMIUM_RPC_PATHS without `{ fetch: premiumFetch }` on the constructor. Per-call-site analysis (not class-level) keeps the trade/index.ts pattern clean — publicClient with globalThis.fetch + premiumClient with premiumFetch on the same TradeServiceClient class — since publicClient never calls a premium method. Wired into: - npm run lint:premium-fetch - .husky/pre-push (right after lint:rate-limit-policies) - .github/workflows/lint-code.yml (right after lint:api-contract) Found and fixed three latent instances of the HIGH(new) #1 class from #3242 review (silent 401 → empty fallback for signed-in browser pros): - src/services/correlation-engine/engine.ts — IntelligenceServiceClient built with no fetch option called deductSituation. LLM-assessment overlay on convergence cards never landed for browser pros without a WM key. - src/services/economic/index.ts — EconomicServiceClient with globalThis.fetch called getNationalDebt. National-debt panel rendered empty for browser pros. - src/services/sanctions-pressure.ts — SanctionsServiceClient with globalThis.fetch called listSanctionsPressure. Sanctions-pressure panel rendered empty for browser pros. All three swap to premiumFetch (single shared client, mirrors the supply-chain/index.ts justification — premiumFetch no-ops safely on public methods, so the public methods on those clients keep working). Verification: - lint:premium-fetch clean (34 ServiceClient classes, 28 premium paths, 466 src/ files analyzed) - Negative test: revert any of the three to globalThis.fetch → exit 1 with file:line and called-premium-method names - typecheck + typecheck:api clean - lint:api-contract / lint:rate-limit-policies / lint:boundaries clean - tests/sanctions-pressure.test.mjs + premium-fetch.test.mts: 16/16 pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(military): fetchStaleFallback NEG_TTL=30s parity (closes #3277) The legacy /api/military-flights handler had NEG_TTL = 30_000ms — a short suppression window after a failed live + stale read so we don't Redis-hammer the stale key during sustained relay+seed outages. Carried into the sebuf list-military-flights handler: - Module-scoped `staleNegUntil` timestamp (per-isolate on Vercel Edge, which is fine — each warm isolate gets its own 30s suppression window). - Set whenever fetchStaleFallback returns null (key missing, parse fail, empty array after staleToProto filter, or thrown error). - Checked at the entry of fetchStaleFallback before doing the Redis read. - Test seam `_resetStaleNegativeCacheForTests()` exposed for unit tests. Test pinned in tests/redis-caching.test.mjs: drives a stale-empty cycle three times — first read hits Redis, second within window doesn't, after test-only reset it does again. Verified: 18/18 redis-caching tests pass, typecheck:api clean, lint:premium-fetch clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lint): rate-limit-policies regex → import() (closes #3278) The previous lint regex-parsed ENDPOINT_RATE_POLICIES from the source file. That worked because the literal happens to fit a single line per key today, but a future reformat (multi-line key wrap, formatter swap, etc.) would silently break the lint without breaking the build — exactly the failure mode that's worse than no lint at all. Fix: - Export ENDPOINT_RATE_POLICIES from server/_shared/rate-limit.ts. - Convert scripts/enforce-rate-limit-policies.mjs to async + dynamic import() of the policy object directly. Same TS module that the gateway uses at runtime → no source-of-truth drift possible. - Run via tsx (already a dev dep, used by test:data) so the .mjs shebang can resolve a .ts import. - npm script swapped to `tsx scripts/...`. .husky/pre-push uses `npm run lint:rate-limit-policies` so no hook change needed. Verified: - Clean: 6 policies / 182 gateway routes. - Negative test (rename a key to the original sanctions typo /api/sanctions/v1/lookup-entity): exit 1 with the same incident- attributed remedy message as before. - Reformat test (split a single-line entry across multiple lines): still passes — the property is what's read, not the source layout. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(shipping/v2): alertThreshold: 0 preserved; drop dead validation branch (#3242 followup) Before: alert_threshold was a plain int32. proto3 scalar default is 0, so the handler couldn't distinguish "partner explicitly sent 0 (deliver every disruption)" from "partner omitted the field (apply legacy default 50)" — both arrived as 0 and got coerced to 50 by `> 0 ? : 50`. Silent intent-drop for any partner who wanted every alert. The subsequent `alertThreshold < 0` branch was also unreachable after that coercion. After: - Proto field is `optional int32 alert_threshold` — TS type becomes `alertThreshold?: number`, so omitted = undefined and explicit 0 stays 0. - Handler uses `req.alertThreshold ?? 50` — undefined → 50, any number passes through unchanged. - Dead `< 0 || > 100` runtime check removed; buf.validate `int32.gte = 0, int32.lte = 100` already enforces the range at the wire layer. Partner wire contract: identical for the omit-field and 1..100 cases. Only behavioural change is explicit 0 — previously impossible to request, now honored per proto3 optional semantics. Scoped `buf generate --path worldmonitor/shipping/v2` to avoid the full- regen `@ts-nocheck` drift Seb documented in the #3242 PR comments. Re-applied `@ts-nocheck` on the two regenerated files manually. Tests: - `alertThreshold 0 coerces to 50` flipped to `alertThreshold 0 preserved`. - New test: `alertThreshold omitted (undefined) applies legacy default 50`. - `rejects > 100` test removed — proto/wire validation handles it; direct handler calls intentionally bypass wire and the handler no longer carries a redundant runtime range check. Verified: 18/18 shipping-v2-handler tests pass, typecheck + typecheck:api clean, all 4 custom lints clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(shipping/v2): document missing webhook delivery worker + DNS-rebinding contract (#3242 followup) #3242 followup checklist item 6 from @koala73 — sanity-check that the delivery worker honors the re-resolve-and-re-check contract that isBlockedCallbackUrl explicitly delegates to it. Audit finding: no delivery worker for shipping/v2 webhooks exists in this repo. Grep across the entire tree (excluding generated/dist) shows the only readers of webhook:sub:* records are the registration / inspection / rotate-secret handlers themselves. No code reads them and POSTs to the stored callbackUrl. The delivery worker is presumed to live in Railway (separate repo) or hasn't been built yet — neither is auditable from this repo. Refreshes the comment block at the top of webhook-shared.ts to: - explicitly state DNS rebinding is NOT mitigated at registration - spell out the four-step contract the delivery worker MUST follow (re-validate URL, dns.lookup, re-check resolved IP against patterns, fetch with resolved IP + Host header preserved) - flag the in-repo gap so anyone landing delivery code can't miss it Tracking the gap as #3288 — acceptance there is "delivery worker imports the patterns + helpers from webhook-shared.ts and applies the four steps before each send." Action moves to wherever the delivery worker actually lives (Railway likely). No code change. Tests + lints unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci(lint): add rate-limit-policies step (greptile P1 #3287) Pre-push hook ran lint:rate-limit-policies but the CI workflow did not, so fork PRs and --no-verify pushes bypassed the exact drift check the lint was added to enforce (closes #3278). Adding it right after lint:api-contract so it runs in the same context the lint was designed for. * refactor(lint): premium-fetch regex → import() + loop classRe (greptile P2 #3287) Two fragilities greptile flagged on enforce-premium-fetch.mjs: 1. loadPremiumPaths regex-parsed src/shared/premium-paths.ts with /'(\/api\/[^']+)'/g — same class of silent drift we just removed from enforce-rate-limit-policies in #3278. Reformatting the source Set (double quotes, spread, helper-computed entries) would drop paths from the lint while leaving the runtime untouched. Fix: flip the shebang to `#!/usr/bin/env -S npx tsx` and dynamic-import PREMIUM_RPC_PATHS directly, mirroring the rate-limit pattern. package.json lint:premium-fetch now invokes via tsx too so the npm-script path matches direct execution. 2. loadClientClassMap ran classRe.exec once, silently dropping every ServiceClient after the first if a file ever contained more than one. Current codegen emits one class per file so this was latent, but a template change would ship un-linted classes. Fix: collect every class-open match with matchAll, slice each class body with the next class's start as the boundary, and scan methods per-body so method-to-class binding stays correct even with multiple classes per file. Verification: - lint:premium-fetch clean (34 classes / 28 premium paths / 466 files — identical counts to pre-refactor, so no coverage regression). - Negative test: revert src/services/economic/index.ts to globalThis.fetch → exit 1 with file:line, bound var name, and premium method list (getNationalDebt). Restore → clean. - lint:rate-limit-policies still clean. * fix(shipping/v2): re-add alertThreshold handler range guard (greptile nit 1 #3287) Wire-layer buf.validate enforces 0..100, but direct handler invocation (internal jobs, test harnesses, future transports) bypasses it. Cheap invariant-at-the-boundary — rejects < 0 or > 100 with ValidationError before the record is stored. Tests: restored the rejects-out-of-range cases that were dropped when the branch was (correctly) deleted as dead code on the previous commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lint): premium-fetch method-regex → TS AST (greptile nits 2+5 #3287) loadClientClassMap: The method regex `async (\w+)\s*\([^)]*\)\s*:\s*Promise<[^>]+>\s*\{\s*let path = "..."` assumed (a) no nested `)` in arg types, (b) no nested `>` in the return type, (c) `let path = "..."` as the literal first statement. Any codegen template shift would silently drop methods with the lint still passing clean — the same silent-drift class #3287 just closed on the premium-paths side. Now walks the service_client.ts AST, matches `export class *ServiceClient`, iterates `MethodDeclaration` members, and reads the first `let path: string = '...'` variable statement as a StringLiteral. Tolerant to any reformatting of arg/return types or method shape. findCalls scope-blindness: Added limitation comment — the walker matches `<varName>.<method>()` anywhere in the file without respecting scope. Two constructions in different function scopes sharing a var name merge their called-method sets. No current src/ file hits this; the lint errs cautiously (flags both instances). Keeping the walker simple until scope-aware binding is needed. webhook-shared.ts: Inlined issue reference (#3288) so the breadcrumb resolves without bouncing through an MDX that isn't in the diff. Verification: - lint:premium-fetch clean — 34 classes / 28 premium paths / 489 files. Pre-refactor: 34 / 28 / 466. Class + path counts identical; file bump is from the main-branch rebase, not the refactor. - Negative test: revert src/services/economic/index.ts premiumFetch → globalThis.fetch. Lint exits 1 at `src/services/economic/index.ts:64:7` with `premium method(s) called: getNationalDebt`. Restore → clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lint): rate-limit OpenAPI regex → yaml parser (greptile nit 3 #3287) Input side (ENDPOINT_RATE_POLICIES) was flipped to live `import()` in |
||
|
Sebastien Melki
|
58e42aadf9 |
chore(api): enforce sebuf contract + migrate drifting endpoints (#3207) (#3242)
* chore(api): enforce sebuf contract via exceptions manifest (#3207) Adds api/api-route-exceptions.json as the single source of truth for non-proto /api/ endpoints, with scripts/enforce-sebuf-api-contract.mjs gating every PR via npm run lint:api-contract. Fixes the root-only blind spot in the prior allowlist (tests/edge-functions.test.mjs), which only scanned top-level *.js files and missed nested paths and .ts endpoints — the gap that let api/supply-chain/v1/country-products.ts and friends drift under proto domain URL prefixes unchallenged. Checks both directions: every api/<domain>/v<N>/[rpc].ts must pair with a generated service_server.ts (so a deleted proto fails CI), and every generated service must have an HTTP gateway (no orphaned generated code). Manifest entries require category + reason + owner, with removal_issue mandatory for temporary categories (deferred, migration-pending) and forbidden for permanent ones. .github/CODEOWNERS pins the manifest to @SebastienMelki so new exceptions don't slip through review. The manifest only shrinks: migration-pending entries (19 today) will be removed as subsequent commits in this PR land each migration. * refactor(maritime): migrate /api/ais-snapshot → maritime/v1.GetVesselSnapshot (#3207) The proto VesselSnapshot was carrying density + disruptions but the frontend also needed sequence, relay status, and candidate_reports to drive the position-callback system. Those only lived on the raw relay passthrough, so the client had to keep hitting /api/ais-snapshot whenever callbacks were registered and fall back to the proto RPC only when the relay URL was gone. This commit pushes all three missing fields through the proto contract and collapses the dual-fetch-path into one proto client call. Proto changes (proto/worldmonitor/maritime/v1/): - VesselSnapshot gains sequence, status, candidate_reports. - GetVesselSnapshotRequest gains include_candidates (query: include_candidates). Handler (server/worldmonitor/maritime/v1/get-vessel-snapshot.ts): - Forwards include_candidates to ?candidates=... on the relay. - Separate 5-min in-memory caches for the candidates=on and candidates=off variants; they have very different payload sizes and should not share a slot. - Per-request in-flight dedup preserved per-variant. Frontend (src/services/maritime/index.ts): - fetchSnapshotPayload now calls MaritimeServiceClient.getVesselSnapshot directly with includeCandidates threaded through. The raw-relay path, SNAPSHOT_PROXY_URL, DIRECT_RAILWAY_SNAPSHOT_URL and LOCAL_SNAPSHOT_FALLBACK are gone — production already routed via Vercel, the "direct" branch only ever fired on localhost, and the proto gateway covers both. - New toLegacyCandidateReport helper mirrors toDensityZone/toDisruptionEvent. api/ais-snapshot.js deleted; manifest entry removed. Only reduced the codegen scope to worldmonitor.maritime.v1 (buf generate --path) — regenerating the full tree drops // @ts-nocheck from every client/server file and surfaces pre-existing type errors across 30+ unrelated services, which is not in scope for this PR. Shape-diff vs legacy payload: - disruptions / density: proto carries the same fields, just with the GeoCoordinates wrapper and enum strings (remapped client-side via existing toDisruptionEvent / toDensityZone helpers). - sequence, status.{connected,vessels,messages}: now populated from the proto response — was hardcoded to 0/false in the prior proto fallback. - candidateReports: same shape; optional numeric fields come through as 0 instead of undefined, which the legacy consumer already handled. * refactor(sanctions): migrate /api/sanctions-entity-search → LookupSanctionEntity (#3207) The proto docstring already claimed "OFAC + OpenSanctions" coverage but the handler only fuzzy-matched a local OFAC Redis index — narrower than the legacy /api/sanctions-entity-search, which proxied OpenSanctions live (the source advertised in docs/api-proxies.mdx). Deleting the legacy without expanding the handler would have been a silent coverage regression for external consumers. Handler changes (server/worldmonitor/sanctions/v1/lookup-entity.ts): - Primary path: live search against api.opensanctions.org/search/default with an 8s timeout and the same User-Agent the legacy edge fn used. - Fallback path: the existing OFAC local fuzzy match, kept intact for when OpenSanctions is unreachable / rate-limiting. - Response source field flips between 'opensanctions' (happy path) and 'ofac' (fallback) so clients can tell which index answered. - Query validation tightened: rejects q > 200 chars (matches legacy cap). Rate limiting: - Added /api/sanctions/v1/lookup-entity to ENDPOINT_RATE_POLICIES at 30/min per IP — matches the legacy createIpRateLimiter budget. The gateway already enforces per-endpoint policies via checkEndpointRateLimit. Docs: - docs/api-proxies.mdx — dropped the /api/sanctions-entity-search row (plus the orphaned /api/ais-snapshot row left over from the previous commit in this PR). - docs/panels/sanctions-pressure.mdx — points at the new RPC URL and describes the OpenSanctions-primary / OFAC-fallback semantics. api/sanctions-entity-search.js deleted; manifest entry removed. * refactor(military): migrate /api/military-flights → ListMilitaryFlights (#3207) Legacy /api/military-flights read a pre-baked Redis blob written by the seed-military-flights cron and returned flights in a flat app-friendly shape (lat/lon, lowercase enums, lastSeenMs). The proto RPC takes a bbox, fetches OpenSky live, classifies server-side, and returns nested GeoCoordinates + MILITARY_*_TYPE_* enum strings + lastSeenAt — same data, different contract. fetchFromRedis in src/services/military-flights.ts was doing nothing sebuf-aware. Renamed it to fetchViaProto and rewrote to: - Instantiate MilitaryServiceClient against getRpcBaseUrl(). - Iterate MILITARY_QUERY_REGIONS (PACIFIC + WESTERN) in parallel — same regions the desktop OpenSky path and the seed cron already use, so dashboard coverage tracks the analytic pipeline. - Dedup by hexCode across regions. - Map proto → app shape via new mapProtoFlight helper plus three reverse enum maps (AIRCRAFT_TYPE_REVERSE, OPERATOR_REVERSE, CONFIDENCE_REVERSE). The seed cron (scripts/seed-military-flights.mjs) stays put: it feeds regional-snapshot mobility, cross-source signals, correlation, and the health freshness check (api/health.js: 'military:flights:v1'). None of those read the legacy HTTP endpoint; they read the Redis key directly. The proto handler uses its own per-bbox cache keys under the same prefix, so dashboard traffic no longer races the seed cron's blob — the two paths diverge by a small refresh lag, which is acceptable. Docs: dropped the /api/military-flights row from docs/api-proxies.mdx. api/military-flights.js deleted; manifest entry removed. Shape-diff vs legacy: - f.location.{latitude,longitude} → f.lat, f.lon - f.aircraftType: MILITARY_AIRCRAFT_TYPE_TANKER → 'tanker' via reverse map - f.operator: MILITARY_OPERATOR_USAF → 'usaf' via reverse map - f.confidence: MILITARY_CONFIDENCE_LOW → 'low' via reverse map - f.lastSeenAt (number) → f.lastSeen (Date) - f.enrichment → f.enriched (with field renames) - Extra fields registration / aircraftModel / origin / destination / firstSeenAt now flow through where proto populates them. * fix(supply-chain): thread includeCandidates through chokepoint status (#3207) Caught by tsconfig.api.json typecheck in the pre-push hook (not covered by the plain tsc --noEmit run that ran before I pushed the ais-snapshot commit). The chokepoint status handler calls getVesselSnapshot internally with a static no-auth request — now required to include the new includeCandidates bool from the proto extension. Passing false: server-internal callers don't need per-vessel reports. * test(maritime): update getVesselSnapshot cache assertions (#3207) The ais-snapshot migration replaced the single cachedSnapshot/cacheTimestamp pair with a per-variant cache so candidates-on and candidates-off payloads don't evict each other. Pre-push hook surfaced that tests/server-handlers still asserted the old variable names. Rewriting the assertions to match the new shape while preserving the invariants they actually guard: - Freshness check against slot TTL. - Cache read before relay call. - Per-slot in-flight dedup. - Stale-serve on relay failure (result ?? slot.snapshot). * chore(proto): restore // @ts-nocheck on regenerated maritime files (#3207) I ran 'buf generate --path worldmonitor/maritime/v1' to scope the proto regen to the one service I was changing (to avoid the toolchain drift that drops @ts-nocheck from 60+ unrelated files — separate issue). But the repo convention is the 'make generate' target, which runs buf and then sed-prepends '// @ts-nocheck' to every generated .ts file. My scoped command skipped the sed step. The proto-check CI enforces the sed output, so the two maritime files need the directive restored. * refactor(enrichment): decomm /api/enrichment/{company,signals} legacy edge fns (#3207) Both endpoints were already ported to IntelligenceService: - getCompanyEnrichment (/api/intelligence/v1/get-company-enrichment) - listCompanySignals (/api/intelligence/v1/list-company-signals) No frontend callers of the legacy /api/enrichment/* paths exist. Removes: - api/enrichment/company.js, signals.js, _domain.js - api-route-exceptions.json migration-pending entries (58 remain) - docs/api-proxies.mdx rows for /api/enrichment/{company,signals} - docs/architecture.mdx reference updated to the IntelligenceService RPCs Verified: typecheck, typecheck:api, lint:api-contract (89 files / 58 entries), lint:boundaries, tests/edge-functions.test.mjs (136 pass), tests/enrichment-caching.test.mjs (14 pass — still guards the intelligence/v1 handlers), make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(leads): migrate /api/{contact,register-interest} → LeadsService (#3207) New leads/v1 sebuf service with two POST RPCs: - SubmitContact → /api/leads/v1/submit-contact - RegisterInterest → /api/leads/v1/register-interest Handler logic ported 1:1 from api/contact.js + api/register-interest.js: - Turnstile verification (desktop sources bypass, preserved) - Honeypot (website field) silently accepts without upstream calls - Free-email-domain gate on SubmitContact (422 ApiError) - validateEmail (disposable/offensive/typo-TLD/MX) on RegisterInterest - Convex writes via ConvexHttpClient (contactMessages:submit, registerInterest:register) - Resend notification + confirmation emails (HTML templates unchanged) Shared helpers moved to server/_shared/: - turnstile.ts (getClientIp + verifyTurnstile) - email-validation.ts (disposable/offensive/MX checks) Rate limits preserved via ENDPOINT_RATE_POLICIES: - submit-contact: 3/hour per IP (was in-memory 3/hr) - register-interest: 5/hour per IP (was in-memory 5/hr; desktop sources previously capped at 2/hr via shared in-memory map — now 5/hr like everyone else, accepting the small regression in exchange for Upstash-backed global limiting) Callers updated: - pro-test/src/App.tsx contact form → new submit-contact path - src-tauri/sidecar/local-api-server.mjs cloud-fallback rewrites /api/register-interest → /api/leads/v1/register-interest when proxying; keeps local path for older desktop builds - src/services/runtime.ts isKeyFreeApiTarget allows both old and new paths through the WORLDMONITOR_API_KEY-optional gate Tests: - tests/contact-handler.test.mjs rewritten to call submitContact handler directly; asserts on ValidationError / ApiError - tests/email-validation.test.mjs + tests/turnstile.test.mjs point at the new server/_shared/ modules Deleted: api/contact.js, api/register-interest.js, api/_ip-rate-limit.js, api/_turnstile.js, api/_email-validation.js, api/_turnstile.test.mjs. Manifest entries removed (58 → 56). Docs updated (api-platform, api-commerce, usage-rate-limits). Verified: npm run typecheck + typecheck:api + lint:api-contract (88 files / 56 entries) + lint:boundaries pass; full test:data (5852 tests) passes; make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(pro-test): rebuild bundle for leads/v1 contact form (#3207) Updates the enterprise contact form to POST to /api/leads/v1/submit-contact (old path /api/contact removed in the previous commit). Bundle is rebuilt from pro-test/src/App.tsx source change in |
||
|
Elie Habib
|
234ec9bf45 |
chore(ci): enforce pro-test bundle freshness — local hook + CI backstop (#3229)
* chore(ci): enforce pro-test bundle freshness, prevent silent deploy staleness public/pro/ is committed to the repo and served verbatim by Vercel. The root build script only runs the main app's vite build — it does NOT run pro-test's build. So any PR that changes pro-test/src/** without manually running `cd pro-test && npm run build` and committing the regenerated chunks ships to production with a stale bundle. This footgun just cost us: PR #3227 fixed the Clerk "not loaded with Ui components" sign-in bug in source, merged, deployed — and the live site still threw the error because the committed chunk under public/pro/assets/ was the pre-fix build. PR #3228 fix-forwarded by rebuilding. Two-layer enforcement so it doesn't happen again: 1. .husky/pre-push — mirrors the existing proto freshness block. If pro-test/ changed vs origin/main, rebuild and `git diff --exit-code public/pro/`. Blocks the push with a clear message if the bundle is stale or untracked files appear. 2. .github/workflows/pro-bundle-freshness.yml — CI backstop on any PR touching pro-test/** or public/pro/**. Runs `npm ci + npm run build` in pro-test and fails the check if the working tree shows any diff or untracked files under public/pro/. Required before merge, so bypassing the local hook still can't land a stale bundle. Note: the hook's diff-against-origin/main check means it skips the build when pushing a branch that already matches main on pro-test/ (e.g. fix-forward branches that only touch public/pro/). CI covers that case via its public/pro/** path filter. * fix(hooks): scope pro-test freshness check to branch delta, not worktree The first version of this hook used `git diff --name-only origin/main -- pro-test/`, which compares the WORKING TREE to origin/main. That fires on unstaged local pro-test/ scratch edits and blocks pushing unrelated branches purely because of dirty checkout state. Switch to `$CHANGED_FILES` (computed earlier at line 77 from `git diff origin/main...HEAD`), which scopes the check to commits on the branch being pushed. This matches the convention the test-runner gates already use (lines 93-97). Also honor `$RUN_ALL` as the safety fallback when the branch delta can't be computed. * fix(hooks): trigger pro freshness check on public/pro/ too, match CI The first scoping fix used `^pro-test/` only, but the CI workflow keys off both `pro-test/**` AND `public/pro/**`. That left a gap: a bundle-only PR (e.g. a fix-forward rebuild like #3228, or a hand-edit to a committed asset) skipped the local check entirely while CI would still validate it. The hook and CI are now consistent. Trigger condition: `^(pro-test|public/pro)/` — the rebuild + diff check now fires whenever the branch delta touches either side of the source/artifact pair, matching the CI workflow's path filter. |
||
|
Elie Habib
|
12ea629a63 |
feat(supply-chain): Sprint C — scenario engine (templates, job API, Railway worker, map activation) (#2890)
* feat(supply-chain): Sprint C — scenario engine templates, job API, Railway worker, map activation
Adds the async scenario engine for supply chain disruption modelling:
- src/config/scenario-templates.ts: 6 pre-built ScenarioTemplate definitions
(Taiwan Strait closure, Suez+BaB simultaneous, Panama drought, Hormuz blockade,
Russia Baltic grain suspension, US electronics tariff shock) with costShockMultiplier
and optional HS2 sector scoping. Exports ScenarioVisualState + ScenarioResult
types (no UI imports, avoids MapContainer <-> DeckGLMap circular dep).
- api/scenario/v1/run.ts: PRO-gated edge function — validates scenarioId against
template registry and iso2 format, enqueues job to Redis scenario-queue:pending
via RPUSH. Returns {jobId, status:'pending'} HTTP 202.
- api/scenario/v1/status.ts: Edge function — validates jobId via regex to prevent
Redis key injection, reads scenario-result:{jobId}. Returns {status:'pending'}
when unprocessed, or full worker result when done.
- scripts/scenario-worker.mjs: Always-on Railway worker using BLMOVE LEFT RIGHT for
atomic FIFO dequeue+claim. Idempotency check before compute. Writes result with
24h TTL; writes {status:'failed'} on error; always cleans processing list in finally.
- DeckGLMap.ts: scenarioState field + setScenarioState(). createTradeRoutesLayer()
overrides arc color to orange for segments whose route waypoints intersect scenario
disruptedChokepointIds. Null state restores normal colors.
- MapContainer.ts: activateScenario(id, result) and deactivateScenario() broadcast
ScenarioVisualState to DeckGLMap. Globe/SVG deferred to Sprint D (best-effort).
🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.com/claude-code) + Compound Engineering v2.49.0
Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
* fix(supply-chain): move scenario-templates to server/ to satisfy arch boundary
api/ edge functions may not import from src/ app code. Move the authoritative
scenario-templates.ts to server/worldmonitor/supply-chain/v1/ and replace
src/config/scenario-templates.ts with a type-only re-export for src/ consumers.
* fix(supply-chain): guard scenario-worker runWorker() behind isMain check
Without the isMain guard, the pre-push hook picks up scenario-worker.mjs
as a seed test candidate (non-matching lines pass through sed unchanged)
and starts the long-running worker process, causing push failures.
* fix(pre-push): filter non-matching lines from seed test selector
The sed transform passes non-matching lines (e.g. scenario-worker.mjs)
through unchanged. Adding grep "^tests/" ensures only successfully
transformed test paths are passed to the test runner.
* fix(supply-chain): address PR #2890 review findings — worker data shapes + status PRO gate
Three bugs found in PR #2890 code review:
1. [High] scenario-worker.mjs read wrong cache shape for exposure data.
supply-chain:exposure:{iso2}:{hs2}:v1 caches GetCountryChokepointIndexResponse
({ iso2, hs2, exposures: [{chokepointId, exposureScore}], ... }), not a
chokepointId-keyed object. Worker now iterates data.exposures[], filters by
template.affectedChokepointIds, and ranks by exposureScore (importValue does
not exist on ChokepointExposureEntry). adjustedImpact = exposureScore x
(disruptionPct/100) x costShockMultiplier.
2. [Medium] api/scenario/v1/status.ts was not PRO-gated, allowing anyone with
a valid jobId to retrieve full premium scenario results. Added isCallerPremium()
check; returns HTTP 403 for non-PRO callers, matching run.ts behavior.
3. [Low] Worker parsed chokepoint status cache as Array but actual shape is
{ chokepoints: [], fetchedAt, upstreamUnavailable }. Fixed to access
cpData.chokepoints array.
* fix(scenario): per-country impactPct + O(1) route lookup in arc layer
- impactPct now reflects each country's relative share of the worst-hit
country (0-100) instead of the flat template.disruptionPct for all
- Pre-build routeId→waypoints Map in createTradeRoutesLayer() so
getColor() is O(1) per segment instead of O(n) per frame
* fix(scenario): rate limit, pipeline GETs, error sanitization, processing state, orphan drain
- Add per-user rate limit (10 jobs/min) + queue depth cap to run.ts
- Replace 594 sequential Redis GETs with single Upstash pipeline call
- Sanitize worker err.message to 'computation_error' in failed results
- Remove dead validateApiKey() calls (isCallerPremium covers this)
- Write processing state before computeScenario() starts
- Add SIGTERM handler + startup orphan drain to worker loop
- Validate dequeued job payload fields before use as Redis key fragments
- Fix maxImpact divide-by-zero with Math.max(..., 1)
- Hoist routeWaypoints Map to module level in DeckGLMap
- Add GET /api/scenario/v1/templates discovery endpoint
- Fix template sync comment to reference correct authoritative file
* docs(plan): mark Sprint C complete, record deferrals to Sprint D
- Sprint status table added: Sprints 0-2 merged, C ready to merge (#2890), A/B/D not started
- Sprint C checklist: 4 ACs checked off, panel UI + tariff-shock visual deferred
- Sprint D section updated to carry over Sprint C visual deferrals
- PR #2890 added to Related PRs
---------
Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
|
||
|
Elie Habib
|
feae07411c |
perf(hooks): smart pre-push skips irrelevant tests based on changed files (#2834)
* perf(hooks): smart pre-push skips irrelevant tests based on changed files Detects which files changed vs origin/main and only runs matching tests: - Frontend-only (src/, api/) changes skip the full 152-file test suite - Resilience changes run only resilience-*.test.* files - Seed script changes run only matching seed tests - Config/test file changes still run the full suite Adds timeout guards (5min full, 2min targeted) to prevent hangs. Skips markdown/MDX lint when no .md/.mdx files changed. Reduces push time for frontend changes from ~13min to ~30sec. * fix(hooks): address Greptile review on smart pre-push P1: Fixed || true on server tests to || exit 1 so failures block push. P1: Replaced elif chain with independent if blocks so multiple changed categories (e.g. server/ + scripts/) all run their tests instead of only the first match. P1: Added safety fallback when CHANGED_FILES is empty (git diff fails in fresh worktrees). Falls back to full test suite instead of silently skipping all tests. |
||
|
Elie Habib
|
380b495be8 |
feat(mcp): live airspace + maritime tools; fix OAuth consent UI (#2442)
* fix(oauth): fix CSS arrow bullets + add MCP branding to consent page
- CSS content:'\2192' (not HTML entity which doesn't work in CSS)
- Rename logo/title to "WorldMonitor MCP" on both consent and error pages
- Inject real news headlines into get_country_brief to prevent hallucination
Fetches list-feed-digest (4s budget), passes top-15 headlines as ?context=
to get-country-intel-brief; brief timeout reduced to 24s to stay under Edge ceiling
* feat(mcp): add get_airspace + get_maritime_activity live query tools
New tools answer real-time positional questions via existing bbox RPCs:
- get_airspace: civilian ADS-B (OpenSky) + military flights over any country
parallel-fetches track-aircraft + list-military-flights, capped at 100 each
- get_maritime_activity: AIS density zones + disruptions for a country's waters
calls get-vessel-snapshot with country bbox
Country → bounding box resolved via shared/country-bboxes.json (167 entries,
generated from public/data/countries.geojson by scripts/generate-country-bboxes.cjs).
Both API calls use 8s AbortSignal.timeout; get_airspace uses Promise.allSettled
so one failure doesn't block the other.
* docs: fix markdown lint in airspace/maritime plan (blank lines around lists)
* fix(oauth): use literal → in CSS content (\2192 is invalid JS octal in ESM)
* fix(hooks): extend bundle check to api/oauth/ subdirectory (was api/*.js, now uses find)
* fix(mcp): address P1 review findings from PR 2442
- JSON import: add 'with { type: json }' so node --test works without tsx loader
- get_airspace: surface upstream failures; partial outage => partial:true+warnings,
total outage => throw (prevents misleading zero-aircraft response)
- pre-push hook: add #!/usr/bin/env bash shebang (was no shebang, ran as /bin/sh
on Linux CI/contributors; process substitution + [[ ]] require bash)
* fix(mcp): replace JSON import attribute with TS module for Vercel compat
Vercel's esbuild bundler does not support `with { type: 'json' }` import
attributes, causing builds to fail with "Expected ';' but found 'with'".
Fix: generate shared/country-bboxes.ts (typed TS module) alongside the
existing JSON file. The TS import has no attributes and bundles cleanly
with all esbuild versions.
Also extend the pre-push bundle check to include api/*.ts root-level files
so this class of error is caught locally before push.
* fix(mcp): reduce get_country_brief timing budget to 24 s (6 s Edge margin)
Digest pre-fetch: 4 s → 2 s (cached endpoint, silent fallback on miss)
Brief call: 24 s → 22 s
Total worst-case: 24 s vs Vercel Edge 30 s hard kill — was 28 s (2 s margin)
* test(mcp): add coverage for get_airspace and get_maritime_activity
9 new tests:
- get_airspace: happy path, unknown code, partial failure (mil down),
total failure (-32603), type=civilian skips military fetch
- get_maritime_activity: happy path, unknown code, API failure (-32603),
empty snapshot handled gracefully
Also fixes import to use .ts extension so Node --test resolver finds the
country-bboxes module (tsx resolves .ts directly; .js alias only works
under moduleResolution:bundler at typecheck time)
* fix(mcp): use .js + .d.ts for country-bboxes — Vercel rejects .ts imports
Vercel edge bundler refuses .ts extension imports even from .ts edge
functions. Plain .js is the only safe runtime import for edge functions.
Pattern: generate shared/country-bboxes.js (pure ESM, no TS syntax) +
shared/country-bboxes.d.ts (type declaration). TypeScript uses the .d.ts
for tuple types at check time; Vercel and Node --test load the .js at
runtime. The previous .ts module is removed.
* test(mcp): update tool count to 26 (main added search_flights + search_flight_prices_by_date)
|
||
|
Elie Habib
|
68212e25a8 | fix(dev): add lint:boundaries to pre-push hook to match CI (#2167) | ||
|
Elie Habib
|
dc74ed69e7 |
fix(fuel-prices): add xlsx to scripts/package-lock.json (#2153)
* fix(fuel-prices): add xlsx to scripts/package-lock.json so npm ci succeeds * chore(pre-push): detect scripts/package-lock.json out-of-sync with package.json |
||
|
Elie Habib
|
19f3b0381a |
fix(health): tighten cyberThreats maxStaleMin 480->240 to match 2x interval rule (#2133)
Cyber seeder interval is 2h (CYBER_SEED_INTERVAL_MS). Gold standard requires maxStaleMin = 2x interval = 240min. Previous 480min let a dead cyber seeder go unnoticed for 8h before health raised WARN. |
||
|
Elie Habib
|
90a626dd26 |
fix(husky): guard against commits/pushes to merged or closed PR branches (#1979)
Pre-commit now checks gh pr state before allowing any commit — exits 1 with a clear message if the branch PR is MERGED or CLOSED, preventing orphaned commits (as happened on PRs #1893, #1898, and #1974). Pre-push: changed 2>/dev/null to 2>&1 so gh errors surface visibly, and added a state echo line so the current PR state is always logged. |
||
|
Elie Habib
|
c3f43376b7 |
fix(git): block push to merged/closed PR branches in pre-push hook (#1924)
Checks gh pr view <branch> --json state before running any tests. If MERGED or CLOSED, exits 1 with a clear message and instructions to branch from origin/main. Prevents orphaned commits like those on PRs #1893, #1898, #1911, #1921. |
||
|
DrDavidL
|
7fdfea854b |
security: add unicode safety guard to hooks and CI (#1710)
* security: add unicode safety guard to hooks and CI * fix(unicode-safety): drop FE0F, PUA; fix col tracking; scan .husky/ - Remove FE0F (emoji presentation selector) from suspicious set — it false-positives on ASCII keycap sequences (#️⃣ etc.) in source strings - Remove Private Use Area (E000–F8FF) check — not a parser attack vector and legitimately used by icon font string literals - Fix column tracking for astral-plane characters (cp > 0xFFFF): increment by 2 to match UTF-16 editor column positions - Remove now-unused prevCp variable - Add .husky/ to SCAN_ROOTS and '' to INCLUDED_EXTENSIONS so extensionless hook scripts (pre-commit, pre-push) are included in full-repo scans --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
bb92815fe0 |
refactor(sanctions): rename panel, restructure layout, fix stale empty cache (#1799)
* refactor(sanctions): rename panel, restructure layout, fix stale empty cache - Rename panel from "Sanctions Pressure" to "Sanctions & Designations" - Make countries the hero section; demote programs to bottom - Remove redundant Top Country/Program headline rows and Source Mix card - Summary now shows New, Total, Vessels, Aircraft (4 cards instead of 6) - Fix stale empty-state cache: circuit breaker now skips caching results with totalCount=0, so a failed seed no longer persists zeros in IndexedDB across reloads - Single "Sanctions data unavailable." empty state instead of three cascading "No X available" messages - Remove dead CSS for .sanctions-headlines / .sanctions-headline / -label / -value * fix(circuit-breaker): evict stale cache when shouldCache rejects SWR result When a background SWR refresh returns a result that shouldCache() rejects, evict the stale cache entry rather than silently discarding the result. Without this, a service returning empty after having real data (e.g. OFAC seed missing from Redis) continues serving the old payload indefinitely via stale-while-revalidate, up to the 24h persistent-cache ceiling. After this fix the stale window is bounded to one SWR cycle: the first foreground load after the feed goes down still serves cached data, the background refresh evicts it, and the next load surfaces the unavailable state. * fix(sanctions): move stale-cache eviction to service, revert circuit-breaker change The circuit-breaker SWR eviction on shouldCache rejection broke the existing market-quote test (which correctly expects stale prices to survive a transient empty response). The concerns are legitimately different: - Market quotes: shouldCache rejection must preserve stale cache (transient blips) - Sanctions: feed down must surface as unavailable, not serve old designations Fix: revert circuit-breaker.ts to original SWR behavior, and instead call breaker.clearCache() explicitly inside the sanctions fn() callback when totalCount === 0. shouldCache still prevents writing the empty result, and the explicit clearCache() evicts any stale entry so the next SWR cycle returns the unavailable state. * chore(hooks): add unit test suite to pre-push gate npm run test:data was missing from pre-push — only edge-functions.test.mjs and mdx-lint.test.mjs ran locally. CI catches all tests/*.test.mjs but the hook did not, causing a broken circuit-breaker change to reach GitHub before the regression was detected. |
||
|
Elie Habib
|
39931456a1 |
feat(forecast): add structured scenario pipeline and trace export (#1646)
* feat(forecast): add AI Forecasts prediction module (Pro-tier)
MiroFish-inspired prediction engine that generates structured forecasts
across 6 domains (conflict, market, supply chain, political, military,
infrastructure) using existing WorldMonitor data streams.
- Proto definitions for ForecastService with GetForecasts RPC
- Dedicated seed script (seed-forecasts.mjs) with 6 domain detectors,
cross-domain cascade resolver, prediction market calibration, and
trend detection via prior snapshot comparison
- Premium-gated RPC handler (PREMIUM_RPC_PATHS enforcement)
- Lazy-loaded ForecastPanel with domain filters, probability bars,
trend arrows, signal evidence, and cascade links
- Health monitoring integration (seed-meta freshness tracking)
- Refresh scheduler with API key guard
* test(forecast): add 47 unit tests for forecast detectors and utilities
Covers forecastId, normalize, resolveCascades, calibrateWithMarkets,
computeTrends, and smoke tests for all 6 domain detectors. Exports
testable functions from seed script with direct-run guard.
* fix(forecast): domain mismatch 'infra' vs 'infrastructure', add panel category
- Seed script used 'infra' but ForecastPanel filtered on 'infrastructure',
causing Infra tab to show zero results
- Added 'forecast' to intelligence category in PANEL_CATEGORY_MAP
* fix(forecast): move CSS to one-time injection, improve type safety
- P2: Move style block from setContent to one-time document.head injection
to prevent CSS accumulation on repeated renders
- P3: Replace +toFixed(3) with Math.round for readability in seed script
- P3: Use Forecast type instead of any[] in RPC handler filter
* fix(forecast): handle sebuf proto data shapes from Redis
Detectors now normalize CII scores from server-side proto format
(combinedScore, TREND_DIRECTION_RISING, region) to uniform shape.
Outage severity handles proto enum format (SEVERITY_LEVEL_HIGH).
Added confidence floor of 0.3 for single-source predictions.
Verified against live Redis: 2 predictions generated (Iran infra
shutdown, IL political instability).
* feat(forecast): unlock AI Forecasts on web, lock desktop only (trial)
- Remove forecast RPC from PREMIUM_RPC_PATHS (web access is free)
- Panel locked on desktop only (same as oref-sirens/telegram-intel)
- Remove API key guards from data-loader and refresh scheduler
- Web users get full access during trial period
* chore: regenerate proto types with make generate
Re-ran make generate after rebasing on main. Plugin v0.7.0 dropped
@ts-nocheck from output, added it back to all 50 generated files.
Fixed 4 type errors from proto codegen changes:
- MarketSource enum -> string union type
- TemporalAnomalyProto -> TemporalAnomaly rename
- webcam lastUpdated number -> string
* chore: add proto freshness check to pre-push hook
Runs make generate before push and compares checksums of generated files.
If proto types are stale, blocks push with instructions to regenerate.
Skips gracefully if buf CLI is not installed.
* fix(forecast): use chokepoints v4 key, include ciiContribution in unrest
- P1: Switch chokepoints input from stale v2 to active v4 Redis key,
matching bootstrap.js and cache-keys.ts
- P2: Add ciiContribution to unrest component fallback chain in
normalizeCiiEntry so political detector reads the correct sebuf field
* feat(forecast): Phase 2 LLM scenario enrichment + confidence model
MiroFish-inspired enhancements:
- LLM scenario narratives via Groq/OpenRouter (narrative-only, no numeric
adjustment). Evidence-grounded prompts with mandatory signal citation
and few-shot examples from MiroFish's SECTION_SYSTEM_PROMPT_TEMPLATE.
- Top-4 predictions batched into single LLM call for cost efficiency.
- News context from newsInsights attached to all predictions for LLM
prompt grounding (NOT in signals, cannot affect confidence).
- Deterministic confidence model: source diversity via SIGNAL_TO_SOURCE
mapping (deduplicates cii+cii_delta, theater+indicators) + calibration
agreement from prediction market drift. Floor 0.2, ceiling 1.0.
- Output validation: rejects scenarios without signal references.
- Truncated JSON repair for small model output.
- Structured JSON logging for LLM calls.
- Redis cache for LLM scenarios (1h TTL).
- 23 new tests (70 total), all passing.
- Live-tested: OpenRouter gemini-2.5-flash produces evidence-grounded
scenario narratives from real WorldMonitor data.
* feat(forecast): Phase 3 multi-perspective scenarios, projections, data-driven cascades
MiroFish-inspired enhancements:
- Multi-perspective LLM analysis: top-2 predictions get strategic,
regional, and contrarian viewpoints via combined LLM call
- Probability projections: domain-specific decay curves (h24/d7/d30)
anchored to timeHorizon so probability equals projections[timeHorizon]
- Data-driven cascade rules: moved from hardcoded array to JSON config
(scripts/data/cascade-rules.json) with schema validation, named
predicate evaluators, unknown key rejection, and fallback to defaults
- 4 new cascade paths: infrastructure->supply_chain, infrastructure->market
(both requiresSeverity:total), conflict->political, political->market
- Proto: added Perspectives and Projections messages to Forecast
- ForecastPanel: renders projections row and conditional perspectives toggle
- 89 tests (19 new), all passing
- Live-tested: OpenRouter produces perspectives from real data
* feat(forecast): Phase 4 data utilization + entity graph
Fixes data gaps that prevented 4 of 6 detectors from firing:
- Input normalizers: chokepoint v4 shape + GPS hexes-to-zones mapping
- Chokepoint warm-ping (production-only, requires WM_API_BASE_URL)
- Lowered CII conflict threshold from 70 to 60, gated on level=high|critical
4 new standalone detectors:
- UCDP conflict zones (10+ events per country)
- Cyber threat concentration (5+ threats per country)
- GPS jamming in maritime shipping zones (5 regions)
- Prediction markets as signals (60-90% probability markets)
Entity-relationship graph (file-based, 38 nodes):
- Countries, theaters, commodities, chokepoints, alliances
- Alias table resolves both ISO codes and display names
- Graph cascade discovery links predictions across entities
Result: 51 predictions (up from 1-2), spanning conflict, infrastructure,
and supply chain domains. 112 tests, all passing.
* fix(forecast): redis cache format, signal source mapping, type safety
Fresh-eyes audit fixes:
- BUG: redisSet used wrong Upstash API format (POST body with {value,ex}
instead of command array ['SET',key,value,'EX',ttl]). LLM cache writes
were silently failing, causing fresh LLM calls every run.
- BUG: prediction_market signal type missing from SIGNAL_TO_SOURCE,
inflating confidence for market-derived predictions.
- CLEANUP: Remove unnecessary (f as any) casts in ForecastPanel since
generated Forecast type already has projections/perspectives fields.
- CLEANUP: Bump health maxStaleMin from 60 to 90 to avoid false STALE
alerts when LLM calls add latency to seed runs.
* feat(forecast): headline-entity matching with news corroboration signals
Uses entity graph aliases to match headlines to predictions by
country/theater (excludes commodity/infrastructure nodes to prevent
false positives). Predictions with matching headlines get a
news_corroboration signal visible in the panel.
Also fixes buildUserPrompt to merge unique headlines from ALL
predictions in the LLM batch (was only reading preds[0].newsContext).
Live-tested: 13 of 51 predictions now have corroborating headlines
(Iran, Israel, Syria, Ukraine, etc). 116 tests, all passing.
* feat(forecast): add country-codes.json for headline-entity matching
56 countries with ISO codes, full names, and scoring keywords (extracted
from src/config/countries.ts + UCDP-relevant additions). Used by
attachNewsContext for richer headline matching via getSearchTermsForRegion
which combines country-codes + entity graph + keyword aliases.
14/57 predictions now have news corroboration (limited by headline
coverage, not matching quality: only 8 headlines currently available).
* feat(forecast): read 300 headlines from news digest instead of 8
Read news:digest:v1:full:en (300 headlines across 16 categories) instead
of just news:insights:v1 topStories (8 headlines). Fallback to topStories
if digest is unavailable.
Result: news corroboration jumped from 25% to 64% (38/59 predictions).
* fix(forecast): handle parenthetical country names in headline matching
Strip suffixes like '(Zaire)', '(Burma)', '(Soviet Union)' from UCDP
region names before matching against country-codes.json. Also use
includes() for reverse name lookup to catch partial matches.
Corroboration: 64% -> 69% (41/59). Remaining 18 unmatched are countries
with no current English-language news coverage.
* fix(forecast): cache validated LLM output, add digest test, log cache errors
Fresh-eyes audit fixes:
- Combined LLM cache now stores only validated items (was caching raw
unvalidated output, serving potentially invalid scenarios on cache hit)
- redisSet logs warnings on failure (was silently swallowing all errors)
- Added digest-based test for attachNewsContext (primary path was untested)
- Fixed test arity: attachNewsContext(preds, news, digest) with 3 params
* fix(forecast): remove dead confidenceFromSources, reduce warm-ping timeout
- P2: Remove confidenceFromSources (dead code, computeConfidence overwrites
all initial confidence values). Inline the formula in original detectors.
- P3: Reduce warm-ping timeout from 30s to 15s (non-critical step)
- P3: Add trial status comment on forecast panel config
* fix(forecast): resolve ISO codes to country names, fix market detector, safe pre-push
P1 fixes from code review:
- CII ISO codes (IL, IR) now resolved to full country names (Israel, Iran)
via country-codes.json. Prevents substring false positives (IL matching
Chile) in event correlation. Uses word-boundary regex for matching.
- Market detector CII-to-theater mapping now uses entity graph traversal
instead of broken theater-name substring matching. Iran correctly maps
to Middle East theater via graph links.
- Pre-push hook no longer runs destructive git checkout on proto freshness
failure. Reports mismatch and exits without modifying worktree.
* feat(forecast): add structured scenario pipeline and trace export
* fix(forecast): hydrate bootstrap and trim generated drift
* fix(forecast): keep required supply-chain contract updates
* fix(ci): add forecasts to cache-keys registry and regenerate proto
Add forecasts entry to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in
cache-keys.ts to match api/bootstrap.js. Regenerate SupplyChain proto
to fix duplicate TransitDayCount and add riskSummary/riskReportAction.
|
||
|
Elie Habib
|
047ab0dfa1 |
ci: add proto freshness check to pre-push hook (#1594)
Mirror the CI proto-check workflow locally so stale generated code is caught before push, not after. Only triggers when proto-related files changed. Gracefully skips if buf/plugins are not installed. |
||
|
Elie Habib
|
8766b45a57 |
fix(docs): comprehensive MDX angle bracket escaping (#1453)
* fix(docs): comprehensive MDX angle bracket escaping Escape all bare < patterns that MDX interprets as JSX across documentation.mdx, algorithms.mdx, ai-intelligence.mdx, data-sources.mdx, finance-data.mdx, relay-parameters.mdx, and maps-and-geocoding.mdx. * feat(lint): add MDX bare angle bracket lint to pre-push Adds tests/mdx-lint.test.mjs that scans all docs/*.mdx files for bare <digit and <hyphen patterns outside code fences. These break Mintlify's MDX parser. Wired into .husky/pre-push so issues are caught before reaching Mintlify. |
||
|
Elie Habib
|
2df7d23edb |
fix(dx): add node_modules guard to pre-push hook and pin Node 22 (#1368)
Worktrees start with no node_modules, causing tsc to fail with misleading TS2307 errors. The guard auto-runs npm install on first push. .nvmrc pins Node 22 to match CI and avoid Node 24 compat issues. |
||
|
Elie Habib
|
ef4920a058 |
ci(lint): add markdown lint to pre-push hook (#1166)
Add `npm run lint:md` to `.husky/pre-push` so markdown lint errors are caught locally before push. Fix existing violation in pro-test/README.md (MD012: multiple consecutive blank lines). |
||
|
Elie Habib
|
898ac7b1c4 |
perf(rss): route RSS direct to Railway, skip Vercel middleman (#961)
* perf(rss): route RSS direct to Railway, skip Vercel middleman
Vercel /api/rss-proxy has 65% error rate (207K failed invocations/12h).
Route browser RSS requests directly to Railway (proxy.worldmonitor.app)
via Cloudflare CDN, eliminating Vercel as middleman.
- Add VITE_RSS_DIRECT_TO_RELAY feature flag (default off) for staged rollout
- Centralize RSS proxy URL in rssProxyUrl() with desktop/dev/prod routing
- Make Railway /rss public (skip auth, keep rate limiting with CF-Connecting-IP)
- Add wildcard *.worldmonitor.app CORS + always emit Vary: Origin on /rss
- Extract ~290 RSS domains to shared/rss-allowed-domains.cjs (single source of truth)
- Convert Railway domain check to Set for O(1) lookups
- Remove rss-proxy from KEYED_CLOUD_API_PATTERN (no longer needs API key header)
- Add edge function test for shared domain list import
* fix(edge): replace node:module with JSON import for edge-compatible RSS domains
api/_rss-allowed-domains.js used createRequire from node:module which is
unsupported in Vercel Edge Runtime, breaking all edge functions (including
api/gpsjam). Replaced with JSON import attribute syntax that works in both
esbuild (Vercel build) and Node.js 22+ (tests).
Also fixed middleware.ts TS18048 error where VARIANT_OG[variant] could be
undefined.
* test(edge): add guard against node: built-in imports in api/ files
Scans ALL api/*.js files (including _ helpers) for node: module imports
which are unsupported in Vercel Edge Runtime. This would have caught the
createRequire(node:module) bug before it reached Vercel.
* fix(edge): inline domain array and remove NextResponse reference
- Replace `import ... with { type: 'json' }` in _rss-allowed-domains.js
with inline array — Vercel esbuild doesn't support import attributes
- Replace `NextResponse.next()` with bare `return` in middleware.ts —
NextResponse was never imported
* ci(pre-push): add esbuild bundle check and edge function tests
The pre-push hook now catches Vercel build failures locally:
- esbuild bundles each api/*.js entrypoint (catches import attribute
syntax, missing modules, and other bundler errors)
- runs edge function test suite (node: imports, module isolation)
|
||
|
Elie Habib
|
f75d420053 |
ci(hooks): add CJS syntax check to pre-push hook (#769)
tsc --noEmit skips .cjs files, so duplicate const declarations (like the UCDP_PAGE_SIZE crash in #766) pass undetected. This adds node -c validation for all scripts/*.cjs files before push. |
||
|
Elie Habib
|
3d2c638a72 |
feat(military): server-side military bases 125K + rate limiting (#496)
* feat(military): server-side military bases with 125K entries + rate limiting (#485) Migrate military bases from 224 static client-side entries to 125,380 server-side entries stored in Redis GEO sorted sets, served via bbox-filtered GEOSEARCH endpoint with server-side clustering. Data pipeline: - Pizzint/Polyglobe: 79,156 entries (Supabase extraction) - OpenStreetMap: 45,185 entries - MIRTA: 821 entries - Curated strategic: 218 entries - 277 proximity duplicates removed Server: - ListMilitaryBases RPC with GEOSEARCH + HMGET + tier/filter/clustering - Antimeridian handling (split bbox queries) - Blue-green Redis deployment with atomic version pointer switch - geoSearchByBox() + getHashFieldsBatch() helpers in redis.ts Security: - @upstash/ratelimit: 60 req/min sliding window per IP - IP spoofing fix: prioritize x-real-ip (Vercel-injected) over x-forwarded-for - Require API key for non-browser requests (blocks unauthenticated curl/scripts) - Input validation: allowlisted types/kinds, regex country, clamped bbox/zoom Frontend: - Viewport-driven loading with bbox quantization + debounce - Server-side grid clustering at low zoom levels - Enriched popup with kind, category badges (airforce/naval/nuclear/space) - Static 224 bases kept as search fallback + initial render * fix(military): fallback to production Redis keys in preview deployments Preview deployments prefix Redis keys with `preview:{sha}:` but military bases data is seeded to unprefixed (production) keys. When the prefixed `military:bases:active` key is missing, fall back to the unprefixed key and use raw (unprefixed) keys for geo/meta lookups. * fix: remove unused 'remaining' destructure in rate-limit (TS6133) * ci: add typecheck:api to pre-push hook to catch server-side TS errors * debug(military): add X-Bases-Debug response header for preview diagnostics * fix(bases): trigger initial server fetch on map load fetchServerBases() was only called on moveend — if the user never panned/zoomed, the API was never called and only the 224 static fallback bases showed. |
||
|
Elie Habib
|
f22ccac2eb |
fix(tech): use rss() for CISA feed, drop build from pre-push hook (#475)
- CISA Advisories used dead rss.worldmonitor.app domain (404), switch to rss() helper - Remove Vite build from pre-push hook (tsc already catches errors) |
||
|
Elie Habib
|
a560efff49 |
chore: bump v2.5.9 and update README for recent features (#398)
* fix: sort tariff datapoints newest-first in trade policy panel * fix: update tests broken by cachedFetchJson migration - Restore "Strip unterminated" comment in summarize-article.ts that tests use to locate the unterminated tag stripping section - Update ACLED tests to check for cachedFetchJson instead of removed getCachedJson/setCachedJson patterns * chore: bump version to 2.5.9 and make pre-push hook executable * docs: update README with supply chain intel, universal CII, Happy Monitor, security hardening, and recent features |
||
|
Elie Habib
|
07d0803014 |
Add WTO trade policy intelligence service with tariffs, flows, and barriers (#364)
* feat: add WTO trade policy service with 4 RPC endpoints and TradePolicyPanel Adds a new `trade` RPC domain backed by the WTO API (apiportal.wto.org) for trade policy intelligence: quantitative restrictions, tariff timeseries, bilateral trade flows, and SPS/TBT barrier notifications. New files: 6 protos, generated server/client, 4 server handlers + shared WTO fetch utility, client service with circuit breakers, TradePolicyPanel (4 tabs), and full API key infrastructure (Rust keychain, sidecar, runtime config). Panel registered for FULL and FINANCE variants with data loader integration, command palette entry, status panel tracking, data freshness monitoring, and i18n across all 17 locale files. https://claude.ai/code/session_01HZXyoQp6xK3TX8obDzv6Ye * chore: update package-lock.json https://claude.ai/code/session_01HZXyoQp6xK3TX8obDzv6Ye * fix: move tab click listener to constructor to prevent leak The delegated click handler was added inside render(), which runs on every data update (4× per load cycle). Since the listener targets this.content (a persistent container), each call stacked a duplicate handler. Moving it to the constructor binds it exactly once. https://claude.ai/code/session_01HZXyoQp6xK3TX8obDzv6Ye --------- Co-authored-by: Claude <noreply@anthropic.com> |