From e68a7147dd017fbbf175767645ebeb2c73942929 Mon Sep 17 00:00:00 2001 From: Sebastien Melki Date: Fri, 24 Apr 2026 18:00:41 +0300 Subject: [PATCH] chore(api): sebuf migration follow-ups (post-#3242) (#3287) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(api-manifest): rewrite brief-why-matters reason as proper internal-helper justification Carried in from #3248 merge as a band-aid (called out in #3242 review followup checklist item 7). The endpoint genuinely belongs in internal-helper — RELAY_SHARED_SECRET-bearer auth, cron-only caller, never reached by dashboards or partners. Same shape constraint as api/notify.ts. Replaces the apologetic "filed here to keep the lint green" framing with a proper structural justification: modeling it as a generated service would publish internal cron plumbing as user-facing API surface. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(lint): premium-fetch parity check for ServiceClients (closes #3279) Adds scripts/enforce-premium-fetch.mjs — AST-walks src/, finds every `new (...)` (variable decl OR `this.foo =` assignment), tracks which methods each instance actually calls, and fails if any called method targets a path in src/shared/premium-paths.ts PREMIUM_RPC_PATHS without `{ fetch: premiumFetch }` on the constructor. Per-call-site analysis (not class-level) keeps the trade/index.ts pattern clean — publicClient with globalThis.fetch + premiumClient with premiumFetch on the same TradeServiceClient class — since publicClient never calls a premium method. Wired into: - npm run lint:premium-fetch - .husky/pre-push (right after lint:rate-limit-policies) - .github/workflows/lint-code.yml (right after lint:api-contract) Found and fixed three latent instances of the HIGH(new) #1 class from #3242 review (silent 401 → empty fallback for signed-in browser pros): - src/services/correlation-engine/engine.ts — IntelligenceServiceClient built with no fetch option called deductSituation. LLM-assessment overlay on convergence cards never landed for browser pros without a WM key. - src/services/economic/index.ts — EconomicServiceClient with globalThis.fetch called getNationalDebt. National-debt panel rendered empty for browser pros. - src/services/sanctions-pressure.ts — SanctionsServiceClient with globalThis.fetch called listSanctionsPressure. Sanctions-pressure panel rendered empty for browser pros. All three swap to premiumFetch (single shared client, mirrors the supply-chain/index.ts justification — premiumFetch no-ops safely on public methods, so the public methods on those clients keep working). Verification: - lint:premium-fetch clean (34 ServiceClient classes, 28 premium paths, 466 src/ files analyzed) - Negative test: revert any of the three to globalThis.fetch → exit 1 with file:line and called-premium-method names - typecheck + typecheck:api clean - lint:api-contract / lint:rate-limit-policies / lint:boundaries clean - tests/sanctions-pressure.test.mjs + premium-fetch.test.mts: 16/16 pass Co-Authored-By: Claude Opus 4.7 (1M context) * fix(military): fetchStaleFallback NEG_TTL=30s parity (closes #3277) The legacy /api/military-flights handler had NEG_TTL = 30_000ms — a short suppression window after a failed live + stale read so we don't Redis-hammer the stale key during sustained relay+seed outages. Carried into the sebuf list-military-flights handler: - Module-scoped `staleNegUntil` timestamp (per-isolate on Vercel Edge, which is fine — each warm isolate gets its own 30s suppression window). - Set whenever fetchStaleFallback returns null (key missing, parse fail, empty array after staleToProto filter, or thrown error). - Checked at the entry of fetchStaleFallback before doing the Redis read. - Test seam `_resetStaleNegativeCacheForTests()` exposed for unit tests. Test pinned in tests/redis-caching.test.mjs: drives a stale-empty cycle three times — first read hits Redis, second within window doesn't, after test-only reset it does again. Verified: 18/18 redis-caching tests pass, typecheck:api clean, lint:premium-fetch clean. Co-Authored-By: Claude Opus 4.7 (1M context) * refactor(lint): rate-limit-policies regex → import() (closes #3278) The previous lint regex-parsed ENDPOINT_RATE_POLICIES from the source file. That worked because the literal happens to fit a single line per key today, but a future reformat (multi-line key wrap, formatter swap, etc.) would silently break the lint without breaking the build — exactly the failure mode that's worse than no lint at all. Fix: - Export ENDPOINT_RATE_POLICIES from server/_shared/rate-limit.ts. - Convert scripts/enforce-rate-limit-policies.mjs to async + dynamic import() of the policy object directly. Same TS module that the gateway uses at runtime → no source-of-truth drift possible. - Run via tsx (already a dev dep, used by test:data) so the .mjs shebang can resolve a .ts import. - npm script swapped to `tsx scripts/...`. .husky/pre-push uses `npm run lint:rate-limit-policies` so no hook change needed. Verified: - Clean: 6 policies / 182 gateway routes. - Negative test (rename a key to the original sanctions typo /api/sanctions/v1/lookup-entity): exit 1 with the same incident- attributed remedy message as before. - Reformat test (split a single-line entry across multiple lines): still passes — the property is what's read, not the source layout. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(shipping/v2): alertThreshold: 0 preserved; drop dead validation branch (#3242 followup) Before: alert_threshold was a plain int32. proto3 scalar default is 0, so the handler couldn't distinguish "partner explicitly sent 0 (deliver every disruption)" from "partner omitted the field (apply legacy default 50)" — both arrived as 0 and got coerced to 50 by `> 0 ? : 50`. Silent intent-drop for any partner who wanted every alert. The subsequent `alertThreshold < 0` branch was also unreachable after that coercion. After: - Proto field is `optional int32 alert_threshold` — TS type becomes `alertThreshold?: number`, so omitted = undefined and explicit 0 stays 0. - Handler uses `req.alertThreshold ?? 50` — undefined → 50, any number passes through unchanged. - Dead `< 0 || > 100` runtime check removed; buf.validate `int32.gte = 0, int32.lte = 100` already enforces the range at the wire layer. Partner wire contract: identical for the omit-field and 1..100 cases. Only behavioural change is explicit 0 — previously impossible to request, now honored per proto3 optional semantics. Scoped `buf generate --path worldmonitor/shipping/v2` to avoid the full- regen `@ts-nocheck` drift Seb documented in the #3242 PR comments. Re-applied `@ts-nocheck` on the two regenerated files manually. Tests: - `alertThreshold 0 coerces to 50` flipped to `alertThreshold 0 preserved`. - New test: `alertThreshold omitted (undefined) applies legacy default 50`. - `rejects > 100` test removed — proto/wire validation handles it; direct handler calls intentionally bypass wire and the handler no longer carries a redundant runtime range check. Verified: 18/18 shipping-v2-handler tests pass, typecheck + typecheck:api clean, all 4 custom lints clean. Co-Authored-By: Claude Opus 4.7 (1M context) * docs(shipping/v2): document missing webhook delivery worker + DNS-rebinding contract (#3242 followup) #3242 followup checklist item 6 from @koala73 — sanity-check that the delivery worker honors the re-resolve-and-re-check contract that isBlockedCallbackUrl explicitly delegates to it. Audit finding: no delivery worker for shipping/v2 webhooks exists in this repo. Grep across the entire tree (excluding generated/dist) shows the only readers of webhook:sub:* records are the registration / inspection / rotate-secret handlers themselves. No code reads them and POSTs to the stored callbackUrl. The delivery worker is presumed to live in Railway (separate repo) or hasn't been built yet — neither is auditable from this repo. Refreshes the comment block at the top of webhook-shared.ts to: - explicitly state DNS rebinding is NOT mitigated at registration - spell out the four-step contract the delivery worker MUST follow (re-validate URL, dns.lookup, re-check resolved IP against patterns, fetch with resolved IP + Host header preserved) - flag the in-repo gap so anyone landing delivery code can't miss it Tracking the gap as #3288 — acceptance there is "delivery worker imports the patterns + helpers from webhook-shared.ts and applies the four steps before each send." Action moves to wherever the delivery worker actually lives (Railway likely). No code change. Tests + lints unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) * ci(lint): add rate-limit-policies step (greptile P1 #3287) Pre-push hook ran lint:rate-limit-policies but the CI workflow did not, so fork PRs and --no-verify pushes bypassed the exact drift check the lint was added to enforce (closes #3278). Adding it right after lint:api-contract so it runs in the same context the lint was designed for. * refactor(lint): premium-fetch regex → import() + loop classRe (greptile P2 #3287) Two fragilities greptile flagged on enforce-premium-fetch.mjs: 1. loadPremiumPaths regex-parsed src/shared/premium-paths.ts with /'(\/api\/[^']+)'/g — same class of silent drift we just removed from enforce-rate-limit-policies in #3278. Reformatting the source Set (double quotes, spread, helper-computed entries) would drop paths from the lint while leaving the runtime untouched. Fix: flip the shebang to `#!/usr/bin/env -S npx tsx` and dynamic-import PREMIUM_RPC_PATHS directly, mirroring the rate-limit pattern. package.json lint:premium-fetch now invokes via tsx too so the npm-script path matches direct execution. 2. loadClientClassMap ran classRe.exec once, silently dropping every ServiceClient after the first if a file ever contained more than one. Current codegen emits one class per file so this was latent, but a template change would ship un-linted classes. Fix: collect every class-open match with matchAll, slice each class body with the next class's start as the boundary, and scan methods per-body so method-to-class binding stays correct even with multiple classes per file. Verification: - lint:premium-fetch clean (34 classes / 28 premium paths / 466 files — identical counts to pre-refactor, so no coverage regression). - Negative test: revert src/services/economic/index.ts to globalThis.fetch → exit 1 with file:line, bound var name, and premium method list (getNationalDebt). Restore → clean. - lint:rate-limit-policies still clean. * fix(shipping/v2): re-add alertThreshold handler range guard (greptile nit 1 #3287) Wire-layer buf.validate enforces 0..100, but direct handler invocation (internal jobs, test harnesses, future transports) bypasses it. Cheap invariant-at-the-boundary — rejects < 0 or > 100 with ValidationError before the record is stored. Tests: restored the rejects-out-of-range cases that were dropped when the branch was (correctly) deleted as dead code on the previous commit. Co-Authored-By: Claude Opus 4.7 (1M context) * refactor(lint): premium-fetch method-regex → TS AST (greptile nits 2+5 #3287) loadClientClassMap: The method regex `async (\w+)\s*\([^)]*\)\s*:\s*Promise<[^>]+>\s*\{\s*let path = "..."` assumed (a) no nested `)` in arg types, (b) no nested `>` in the return type, (c) `let path = "..."` as the literal first statement. Any codegen template shift would silently drop methods with the lint still passing clean — the same silent-drift class #3287 just closed on the premium-paths side. Now walks the service_client.ts AST, matches `export class *ServiceClient`, iterates `MethodDeclaration` members, and reads the first `let path: string = '...'` variable statement as a StringLiteral. Tolerant to any reformatting of arg/return types or method shape. findCalls scope-blindness: Added limitation comment — the walker matches `.()` anywhere in the file without respecting scope. Two constructions in different function scopes sharing a var name merge their called-method sets. No current src/ file hits this; the lint errs cautiously (flags both instances). Keeping the walker simple until scope-aware binding is needed. webhook-shared.ts: Inlined issue reference (#3288) so the breadcrumb resolves without bouncing through an MDX that isn't in the diff. Verification: - lint:premium-fetch clean — 34 classes / 28 premium paths / 489 files. Pre-refactor: 34 / 28 / 466. Class + path counts identical; file bump is from the main-branch rebase, not the refactor. - Negative test: revert src/services/economic/index.ts premiumFetch → globalThis.fetch. Lint exits 1 at `src/services/economic/index.ts:64:7` with `premium method(s) called: getNationalDebt`. Restore → clean. Co-Authored-By: Claude Opus 4.7 (1M context) * refactor(lint): rate-limit OpenAPI regex → yaml parser (greptile nit 3 #3287) Input side (ENDPOINT_RATE_POLICIES) was flipped to live `import()` in 4e79d029. Output side (OpenAPI routes) still regex-scraped top-level `paths:` keys with `/^\s{4}(\/api\/[^\s:]+):/gm` — hard-coded 4-space indent. Any YAML formatter change (2-space indent, flow style, line folding) would silently drop routes and let policy-drift slip through — same silent-drift class the input-side fix closed. Now uses the `yaml` package (already a dep) to parse each .openapi.yaml and reads `doc.paths` directly. Verification: - Clean: 6 policies / 189 routes (was 182 — yaml parser picks up a handful the regex missed, closing a silent coverage gap). - Negative test: rename policy key back to /api/sanctions/v1/lookup-entity → exits 1 with the same incident-attributed remedy. Restore → clean. Co-Authored-By: Claude Opus 4.7 (1M context) * chore(codegen): regenerate unified OpenAPI bundle for alert_threshold proto change The shipping/v2 webhook alert_threshold field was flipped from `int32` to `optional int32` with an expanded doc comment in f3339464. That comment now surfaces in the unified docs/api/worldmonitor.openapi.yaml bundle (introduced by #3341). Regenerated with sebuf v0.11.1 to pick it up. No behaviour change — bundle-only documentation drift. Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: Claude Opus 4.7 (1M context) --- .github/workflows/lint-code.yml | 2 + .husky/pre-push | 3 + api/api-route-exceptions.json | 2 +- docs/api/ShippingV2Service.openapi.json | 2 +- docs/api/ShippingV2Service.openapi.yaml | 7 +- docs/api/worldmonitor.openapi.yaml | 7 +- package.json | 3 +- .../shipping/v2/register_webhook.proto | 6 +- scripts/enforce-premium-fetch.mjs | 282 ++++++++++++++++++ scripts/enforce-rate-limit-policies.mjs | 59 ++-- server/_shared/rate-limit.ts | 6 +- .../military/v1/list-military-flights.ts | 29 +- .../shipping/v2/register-webhook.ts | 12 +- .../shipping/v2/webhook-shared.ts | 20 +- .../shipping/v2/service_client.ts | 2 +- .../shipping/v2/service_server.ts | 2 +- src/services/correlation-engine/engine.ts | 8 +- src/services/economic/index.ts | 10 +- src/services/sanctions-pressure.ts | 6 +- tests/redis-caching.test.mjs | 70 +++++ tests/shipping-v2-handler.test.mjs | 36 ++- 21 files changed, 524 insertions(+), 50 deletions(-) create mode 100644 scripts/enforce-premium-fetch.mjs diff --git a/.github/workflows/lint-code.yml b/.github/workflows/lint-code.yml index 4e274bafe..858ade3bd 100644 --- a/.github/workflows/lint-code.yml +++ b/.github/workflows/lint-code.yml @@ -44,6 +44,8 @@ jobs: - run: npm run lint - run: npm run lint:boundaries - run: npm run lint:api-contract + - run: npm run lint:rate-limit-policies + - run: npm run lint:premium-fetch - name: Markdown lint run: npm run lint:md - name: Version sync check diff --git a/.husky/pre-push b/.husky/pre-push index 21dabaca7..1e94fd0f2 100755 --- a/.husky/pre-push +++ b/.husky/pre-push @@ -68,6 +68,9 @@ npm run lint:boundaries || exit 1 echo "Running rate-limit policy coverage check..." npm run lint:rate-limit-policies || exit 1 +echo "Running premium-fetch parity check..." +npm run lint:premium-fetch || exit 1 + echo "Running edge function bundle check..." while IFS= read -r f; do npx esbuild "$f" --bundle --format=esm --platform=browser --outfile=/dev/null 2>/dev/null || { diff --git a/api/api-route-exceptions.json b/api/api-route-exceptions.json index 71283c803..446d6ae85 100644 --- a/api/api-route-exceptions.json +++ b/api/api-route-exceptions.json @@ -247,7 +247,7 @@ { "path": "api/internal/brief-why-matters.ts", "category": "internal-helper", - "reason": "Internal brief-pipeline helper — auth'd by RELAY_SHARED_SECRET (Railway cron only), not a user-facing API. Generated on merge of #3248 from main without a manifest entry; filed here to keep the lint green.", + "reason": "LLM-enrichment helper for the brief pipeline. Cron-triggered from Railway under RELAY_SHARED_SECRET bearer auth — never reached by dashboards, desktop, or partners. Request/response shape is an implementation detail of the brief renderer; modeling it as a generated service would publish internal cron plumbing as user-facing API surface. Same shape constraint as api/notify.ts (also cron-only).", "owner": "@SebastienMelki", "removal_issue": null }, diff --git a/docs/api/ShippingV2Service.openapi.json b/docs/api/ShippingV2Service.openapi.json index ef7238608..45d531e6b 100644 --- a/docs/api/ShippingV2Service.openapi.json +++ b/docs/api/ShippingV2Service.openapi.json @@ -1 +1 @@ -{"components":{"schemas":{"BypassOption":{"description":"Single bypass-corridor option around a disrupted chokepoint.","properties":{"activationThreshold":{"description":"Enum-like string, e.g., \"DISRUPTION_SCORE_60\".","type":"string"},"addedCostMultiplier":{"format":"double","type":"number"},"addedTransitDays":{"format":"int32","type":"integer"},"id":{"type":"string"},"name":{"type":"string"},"type":{"description":"Type of bypass (e.g., \"maritime_detour\", \"land_corridor\").","type":"string"}},"type":"object"},"ChokepointExposure":{"description":"Single chokepoint exposure for a route.","properties":{"chokepointId":{"type":"string"},"chokepointName":{"type":"string"},"exposurePct":{"format":"int32","type":"integer"}},"type":"object"},"Error":{"description":"Error is returned when a handler encounters an error. It contains a simple error message that the developer can customize.","properties":{"message":{"description":"Error message (e.g., 'user not found', 'database connection failed')","type":"string"}},"type":"object"},"FieldViolation":{"description":"FieldViolation describes a single validation error for a specific field.","properties":{"description":{"description":"Human-readable description of the validation violation (e.g., 'must be a valid email address', 'required field missing')","type":"string"},"field":{"description":"The field path that failed validation (e.g., 'user.email' for nested fields). For header validation, this will be the header name (e.g., 'X-API-Key')","type":"string"}},"required":["field","description"],"type":"object"},"ListWebhooksRequest":{"description":"ListWebhooksRequest has no fields — the owner is derived from the caller's\n API-key fingerprint (SHA-256 of X-WorldMonitor-Key).","type":"object"},"ListWebhooksResponse":{"description":"ListWebhooksResponse wire shape preserved exactly: the `webhooks` field\n name and the omission of `secret` are part of the partner contract.","properties":{"webhooks":{"items":{"$ref":"#/components/schemas/WebhookSummary"},"type":"array"}},"type":"object"},"RegisterWebhookRequest":{"description":"RegisterWebhookRequest creates a new chokepoint-disruption webhook\n subscription. Wire shape is byte-compatible with the pre-migration\n legacy POST body.","properties":{"alertThreshold":{"description":"Disruption-score threshold for delivery, 0-100. Default 50.","format":"int32","maximum":100,"minimum":0,"type":"integer"},"callbackUrl":{"description":"HTTPS callback URL. Must not resolve to a private/loopback address at\n registration time (SSRF guard). The delivery worker re-validates the\n resolved IP before each send to mitigate DNS rebinding.","maxLength":2048,"minLength":8,"type":"string"},"chokepointIds":{"items":{"description":"Zero or more chokepoint IDs to subscribe to. Empty list subscribes to\n the entire CHOKEPOINT_REGISTRY. Unknown IDs fail with 400.","type":"string"},"type":"array"}},"required":["callbackUrl"],"type":"object"},"RegisterWebhookResponse":{"description":"RegisterWebhookResponse wire shape preserved exactly — partners persist the\n `secret` because the server never returns it again except via rotate-secret.","properties":{"secret":{"description":"Raw 64-char lowercase hex secret (32 random bytes). No `whsec_` prefix.","type":"string"},"subscriberId":{"description":"`wh_` prefix + 24 lowercase hex chars (12 random bytes).","type":"string"}},"type":"object"},"RouteIntelligenceRequest":{"description":"RouteIntelligenceRequest scopes a route-intelligence query by origin and\n destination country. Query-parameter names are preserved verbatim from the\n legacy partner contract (fromIso2/toIso2/cargoType/hs2 — camelCase).","properties":{"cargoType":{"description":"Cargo type — one of: container (default), tanker, bulk, roro.\n Empty string defers to the server default. Unknown values are coerced to\n \"container\" to preserve legacy behavior.","type":"string"},"fromIso2":{"description":"Origin country, ISO-3166-1 alpha-2 uppercase.","pattern":"^[A-Z]{2}$","type":"string"},"hs2":{"description":"2-digit HS commodity code (default \"27\" — mineral fuels). Non-digit\n characters are stripped server-side to match legacy behavior.","type":"string"},"toIso2":{"description":"Destination country, ISO-3166-1 alpha-2 uppercase.","pattern":"^[A-Z]{2}$","type":"string"}},"required":["fromIso2","toIso2"],"type":"object"},"RouteIntelligenceResponse":{"description":"RouteIntelligenceResponse wire shape preserved byte-for-byte from the\n pre-migration JSON at docs/api-shipping-v2.mdx. `fetched_at` is intentionally\n a string (ISO-8601) rather than int64 epoch ms because partners depend on\n the ISO-8601 shape.","properties":{"bypassOptions":{"items":{"$ref":"#/components/schemas/BypassOption"},"type":"array"},"cargoType":{"type":"string"},"chokepointExposures":{"items":{"$ref":"#/components/schemas/ChokepointExposure"},"type":"array"},"disruptionScore":{"description":"Disruption score of the primary chokepoint, 0-100.","format":"int32","type":"integer"},"fetchedAt":{"description":"ISO-8601 timestamp of when the response was assembled.","type":"string"},"fromIso2":{"type":"string"},"hs2":{"type":"string"},"primaryRouteId":{"type":"string"},"toIso2":{"type":"string"},"warRiskTier":{"description":"War-risk tier enum string, e.g., \"WAR_RISK_TIER_NORMAL\" or \"WAR_RISK_TIER_ELEVATED\".","type":"string"}},"type":"object"},"ValidationError":{"description":"ValidationError is returned when request validation fails. It contains a list of field violations describing what went wrong.","properties":{"violations":{"description":"List of validation violations","items":{"$ref":"#/components/schemas/FieldViolation"},"type":"array"}},"required":["violations"],"type":"object"},"WebhookSummary":{"description":"Single webhook record in the list response. `secret` is intentionally\n omitted; use rotate-secret to obtain a new one.","properties":{"active":{"type":"boolean"},"alertThreshold":{"format":"int32","type":"integer"},"callbackUrl":{"type":"string"},"chokepointIds":{"items":{"type":"string"},"type":"array"},"createdAt":{"description":"ISO-8601 timestamp of registration.","type":"string"},"subscriberId":{"type":"string"}},"type":"object"}}},"info":{"title":"ShippingV2Service API","version":"1.0.0"},"openapi":"3.1.0","paths":{"/api/v2/shipping/route-intelligence":{"get":{"description":"RouteIntelligence scores a country-pair trade route for chokepoint exposure\n and current disruption risk. Partner-facing; wire shape is byte-compatible\n with the pre-migration JSON response documented at docs/api-shipping-v2.mdx.","operationId":"RouteIntelligence","parameters":[{"description":"Origin country, ISO-3166-1 alpha-2 uppercase.","in":"query","name":"fromIso2","required":false,"schema":{"type":"string"}},{"description":"Destination country, ISO-3166-1 alpha-2 uppercase.","in":"query","name":"toIso2","required":false,"schema":{"type":"string"}},{"description":"Cargo type — one of: container (default), tanker, bulk, roro.\n Empty string defers to the server default. Unknown values are coerced to\n \"container\" to preserve legacy behavior.","in":"query","name":"cargoType","required":false,"schema":{"type":"string"}},{"description":"2-digit HS commodity code (default \"27\" — mineral fuels). Non-digit\n characters are stripped server-side to match legacy behavior.","in":"query","name":"hs2","required":false,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/RouteIntelligenceResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"RouteIntelligence","tags":["ShippingV2Service"]}},"/api/v2/shipping/webhooks":{"get":{"description":"ListWebhooks returns the caller's registered webhooks filtered by the\n SHA-256 owner tag of the calling API key. The `secret` is intentionally\n omitted from the response; use rotate-secret to obtain a new one.","operationId":"ListWebhooks","responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ListWebhooksResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"ListWebhooks","tags":["ShippingV2Service"]},"post":{"description":"RegisterWebhook subscribes a callback URL to chokepoint disruption alerts.\n Returns the subscriberId and the raw HMAC secret — the secret is never\n returned again except via rotate-secret.","operationId":"RegisterWebhook","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/RegisterWebhookRequest"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/RegisterWebhookResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"RegisterWebhook","tags":["ShippingV2Service"]}}}} \ No newline at end of file +{"components":{"schemas":{"BypassOption":{"description":"Single bypass-corridor option around a disrupted chokepoint.","properties":{"activationThreshold":{"description":"Enum-like string, e.g., \"DISRUPTION_SCORE_60\".","type":"string"},"addedCostMultiplier":{"format":"double","type":"number"},"addedTransitDays":{"format":"int32","type":"integer"},"id":{"type":"string"},"name":{"type":"string"},"type":{"description":"Type of bypass (e.g., \"maritime_detour\", \"land_corridor\").","type":"string"}},"type":"object"},"ChokepointExposure":{"description":"Single chokepoint exposure for a route.","properties":{"chokepointId":{"type":"string"},"chokepointName":{"type":"string"},"exposurePct":{"format":"int32","type":"integer"}},"type":"object"},"Error":{"description":"Error is returned when a handler encounters an error. It contains a simple error message that the developer can customize.","properties":{"message":{"description":"Error message (e.g., 'user not found', 'database connection failed')","type":"string"}},"type":"object"},"FieldViolation":{"description":"FieldViolation describes a single validation error for a specific field.","properties":{"description":{"description":"Human-readable description of the validation violation (e.g., 'must be a valid email address', 'required field missing')","type":"string"},"field":{"description":"The field path that failed validation (e.g., 'user.email' for nested fields). For header validation, this will be the header name (e.g., 'X-API-Key')","type":"string"}},"required":["field","description"],"type":"object"},"ListWebhooksRequest":{"description":"ListWebhooksRequest has no fields — the owner is derived from the caller's\n API-key fingerprint (SHA-256 of X-WorldMonitor-Key).","type":"object"},"ListWebhooksResponse":{"description":"ListWebhooksResponse wire shape preserved exactly: the `webhooks` field\n name and the omission of `secret` are part of the partner contract.","properties":{"webhooks":{"items":{"$ref":"#/components/schemas/WebhookSummary"},"type":"array"}},"type":"object"},"RegisterWebhookRequest":{"description":"RegisterWebhookRequest creates a new chokepoint-disruption webhook\n subscription. Wire shape is byte-compatible with the pre-migration\n legacy POST body.","properties":{"alertThreshold":{"description":"Disruption-score threshold for delivery, 0-100. Default 50.\n proto3 `optional` so the handler can distinguish \"partner explicitly sent\n 0 (deliver every alert)\" from \"partner omitted the field (apply default\n 50)\". Without `optional`, both serialise to the proto3 scalar default of\n 0 and the handler can't tell them apart — flagged in #3242 review.","format":"int32","maximum":100,"minimum":0,"type":"integer"},"callbackUrl":{"description":"HTTPS callback URL. Must not resolve to a private/loopback address at\n registration time (SSRF guard). The delivery worker re-validates the\n resolved IP before each send to mitigate DNS rebinding.","maxLength":2048,"minLength":8,"type":"string"},"chokepointIds":{"items":{"description":"Zero or more chokepoint IDs to subscribe to. Empty list subscribes to\n the entire CHOKEPOINT_REGISTRY. Unknown IDs fail with 400.","type":"string"},"type":"array"}},"required":["callbackUrl"],"type":"object"},"RegisterWebhookResponse":{"description":"RegisterWebhookResponse wire shape preserved exactly — partners persist the\n `secret` because the server never returns it again except via rotate-secret.","properties":{"secret":{"description":"Raw 64-char lowercase hex secret (32 random bytes). No `whsec_` prefix.","type":"string"},"subscriberId":{"description":"`wh_` prefix + 24 lowercase hex chars (12 random bytes).","type":"string"}},"type":"object"},"RouteIntelligenceRequest":{"description":"RouteIntelligenceRequest scopes a route-intelligence query by origin and\n destination country. Query-parameter names are preserved verbatim from the\n legacy partner contract (fromIso2/toIso2/cargoType/hs2 — camelCase).","properties":{"cargoType":{"description":"Cargo type — one of: container (default), tanker, bulk, roro.\n Empty string defers to the server default. Unknown values are coerced to\n \"container\" to preserve legacy behavior.","type":"string"},"fromIso2":{"description":"Origin country, ISO-3166-1 alpha-2 uppercase.","pattern":"^[A-Z]{2}$","type":"string"},"hs2":{"description":"2-digit HS commodity code (default \"27\" — mineral fuels). Non-digit\n characters are stripped server-side to match legacy behavior.","type":"string"},"toIso2":{"description":"Destination country, ISO-3166-1 alpha-2 uppercase.","pattern":"^[A-Z]{2}$","type":"string"}},"required":["fromIso2","toIso2"],"type":"object"},"RouteIntelligenceResponse":{"description":"RouteIntelligenceResponse wire shape preserved byte-for-byte from the\n pre-migration JSON at docs/api-shipping-v2.mdx. `fetched_at` is intentionally\n a string (ISO-8601) rather than int64 epoch ms because partners depend on\n the ISO-8601 shape.","properties":{"bypassOptions":{"items":{"$ref":"#/components/schemas/BypassOption"},"type":"array"},"cargoType":{"type":"string"},"chokepointExposures":{"items":{"$ref":"#/components/schemas/ChokepointExposure"},"type":"array"},"disruptionScore":{"description":"Disruption score of the primary chokepoint, 0-100.","format":"int32","type":"integer"},"fetchedAt":{"description":"ISO-8601 timestamp of when the response was assembled.","type":"string"},"fromIso2":{"type":"string"},"hs2":{"type":"string"},"primaryRouteId":{"type":"string"},"toIso2":{"type":"string"},"warRiskTier":{"description":"War-risk tier enum string, e.g., \"WAR_RISK_TIER_NORMAL\" or \"WAR_RISK_TIER_ELEVATED\".","type":"string"}},"type":"object"},"ValidationError":{"description":"ValidationError is returned when request validation fails. It contains a list of field violations describing what went wrong.","properties":{"violations":{"description":"List of validation violations","items":{"$ref":"#/components/schemas/FieldViolation"},"type":"array"}},"required":["violations"],"type":"object"},"WebhookSummary":{"description":"Single webhook record in the list response. `secret` is intentionally\n omitted; use rotate-secret to obtain a new one.","properties":{"active":{"type":"boolean"},"alertThreshold":{"format":"int32","type":"integer"},"callbackUrl":{"type":"string"},"chokepointIds":{"items":{"type":"string"},"type":"array"},"createdAt":{"description":"ISO-8601 timestamp of registration.","type":"string"},"subscriberId":{"type":"string"}},"type":"object"}}},"info":{"title":"ShippingV2Service API","version":"1.0.0"},"openapi":"3.1.0","paths":{"/api/v2/shipping/route-intelligence":{"get":{"description":"RouteIntelligence scores a country-pair trade route for chokepoint exposure\n and current disruption risk. Partner-facing; wire shape is byte-compatible\n with the pre-migration JSON response documented at docs/api-shipping-v2.mdx.","operationId":"RouteIntelligence","parameters":[{"description":"Origin country, ISO-3166-1 alpha-2 uppercase.","in":"query","name":"fromIso2","required":false,"schema":{"type":"string"}},{"description":"Destination country, ISO-3166-1 alpha-2 uppercase.","in":"query","name":"toIso2","required":false,"schema":{"type":"string"}},{"description":"Cargo type — one of: container (default), tanker, bulk, roro.\n Empty string defers to the server default. Unknown values are coerced to\n \"container\" to preserve legacy behavior.","in":"query","name":"cargoType","required":false,"schema":{"type":"string"}},{"description":"2-digit HS commodity code (default \"27\" — mineral fuels). Non-digit\n characters are stripped server-side to match legacy behavior.","in":"query","name":"hs2","required":false,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/RouteIntelligenceResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"RouteIntelligence","tags":["ShippingV2Service"]}},"/api/v2/shipping/webhooks":{"get":{"description":"ListWebhooks returns the caller's registered webhooks filtered by the\n SHA-256 owner tag of the calling API key. The `secret` is intentionally\n omitted from the response; use rotate-secret to obtain a new one.","operationId":"ListWebhooks","responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ListWebhooksResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"ListWebhooks","tags":["ShippingV2Service"]},"post":{"description":"RegisterWebhook subscribes a callback URL to chokepoint disruption alerts.\n Returns the subscriberId and the raw HMAC secret — the secret is never\n returned again except via rotate-secret.","operationId":"RegisterWebhook","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/RegisterWebhookRequest"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/RegisterWebhookResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"RegisterWebhook","tags":["ShippingV2Service"]}}}} \ No newline at end of file diff --git a/docs/api/ShippingV2Service.openapi.yaml b/docs/api/ShippingV2Service.openapi.yaml index 79fa11136..f42cefe27 100644 --- a/docs/api/ShippingV2Service.openapi.yaml +++ b/docs/api/ShippingV2Service.openapi.yaml @@ -277,7 +277,12 @@ components: maximum: 100 minimum: 0 format: int32 - description: Disruption-score threshold for delivery, 0-100. Default 50. + description: |- + Disruption-score threshold for delivery, 0-100. Default 50. + proto3 `optional` so the handler can distinguish "partner explicitly sent + 0 (deliver every alert)" from "partner omitted the field (apply default + 50)". Without `optional`, both serialise to the proto3 scalar default of + 0 and the handler can't tell them apart — flagged in #3242 review. required: - callbackUrl description: |- diff --git a/docs/api/worldmonitor.openapi.yaml b/docs/api/worldmonitor.openapi.yaml index d3ed7a856..a99dd4bec 100644 --- a/docs/api/worldmonitor.openapi.yaml +++ b/docs/api/worldmonitor.openapi.yaml @@ -18929,7 +18929,12 @@ components: maximum: 100 minimum: 0 format: int32 - description: Disruption-score threshold for delivery, 0-100. Default 50. + description: |- + Disruption-score threshold for delivery, 0-100. Default 50. + proto3 `optional` so the handler can distinguish "partner explicitly sent + 0 (deliver every alert)" from "partner omitted the field (apply default + 50)". Without `optional`, both serialise to the proto3 scalar default of + 0 and the handler can't tell them apart — flagged in #3242 review. required: - callbackUrl description: |- diff --git a/package.json b/package.json index 9a6b12c35..c2436d094 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,8 @@ "lint:fix": "biome check ./src ./server ./api ./tests ./e2e ./scripts ./middleware.ts --fix", "lint:boundaries": "node scripts/lint-boundaries.mjs", "lint:api-contract": "node scripts/enforce-sebuf-api-contract.mjs", - "lint:rate-limit-policies": "node scripts/enforce-rate-limit-policies.mjs", + "lint:rate-limit-policies": "tsx scripts/enforce-rate-limit-policies.mjs", + "lint:premium-fetch": "tsx scripts/enforce-premium-fetch.mjs", "lint:unicode": "node scripts/check-unicode-safety.mjs", "lint:unicode:staged": "node scripts/check-unicode-safety.mjs --staged", "lint:md": "markdownlint-cli2 '**/*.md' '!**/node_modules/**' '!.agent/**' '!.agents/**' '!.claude/**' '!.factory/**' '!.windsurf/**' '!skills/**' '!docs/internal/**' '!docs/Docs_To_Review/**' '!todos/**' '!docs/plans/**' '!docs/brainstorms/**' '!docs/ideation/**'", diff --git a/proto/worldmonitor/shipping/v2/register_webhook.proto b/proto/worldmonitor/shipping/v2/register_webhook.proto index 0e86bd6f8..6fb678ff4 100644 --- a/proto/worldmonitor/shipping/v2/register_webhook.proto +++ b/proto/worldmonitor/shipping/v2/register_webhook.proto @@ -20,7 +20,11 @@ message RegisterWebhookRequest { // the entire CHOKEPOINT_REGISTRY. Unknown IDs fail with 400. repeated string chokepoint_ids = 2; // Disruption-score threshold for delivery, 0-100. Default 50. - int32 alert_threshold = 3 [ + // proto3 `optional` so the handler can distinguish "partner explicitly sent + // 0 (deliver every alert)" from "partner omitted the field (apply default + // 50)". Without `optional`, both serialise to the proto3 scalar default of + // 0 and the handler can't tell them apart — flagged in #3242 review. + optional int32 alert_threshold = 3 [ (buf.validate.field).int32.gte = 0, (buf.validate.field).int32.lte = 100 ]; diff --git a/scripts/enforce-premium-fetch.mjs b/scripts/enforce-premium-fetch.mjs new file mode 100644 index 000000000..8ff23ad36 --- /dev/null +++ b/scripts/enforce-premium-fetch.mjs @@ -0,0 +1,282 @@ +#!/usr/bin/env -S npx tsx +/** + * Validates that every `new (...)` instantiation in src/ which + * calls a method whose generated path is in PREMIUM_RPC_PATHS is constructed + * with `{ fetch: premiumFetch }`. + * + * Catches the HIGH(new) #1 class from #3242 review — SupplyChainServiceClient + * was constructed with globalThis.fetch (the generated default) and pro users + * silently got 401s the generated client swallowed into empty-fallback panels. + * Same class as #3233 (RegionalIntelligenceBoard / DeductionPanel / trade / + * country-intel) which was fixed manually because there was no enforcement. + * + * How it works: + * 1. Dynamic `import()` of PREMIUM_RPC_PATHS from src/shared/premium-paths.ts + * (via tsx, same pattern as enforce-rate-limit-policies.mjs) → set of + * premium HTTP paths. Live import means reformatting the source literal + * can never desync the lint from the runtime (#3287 follow-up). + * 2. Walk src/generated/client/ → map each ServiceClient class to its + * method-name → path table (the `let path = "/api/..."` line each + * generated method opens with). + * 3. Walk src/ (excluding generated) with the TypeScript AST. For each + * `new (...)` (variable decl OR `this.foo =` assignment): + * a. Capture the bound variable / member name. + * b. Find every `.(...)` call in the same file. + * c. If any called method has a premium path, the construction MUST + * use { fetch: premiumFetch }. Anything else fails the lint. + * + * Per-call-site analysis lets the trade/index.ts pattern (publicClient with + * globalThis.fetch + premiumClient with premiumFetch on the same class) + * stay clean, since publicClient never calls a premium method. + */ +import { readFileSync, readdirSync, statSync } from 'node:fs'; +import { join, relative, basename } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import ts from 'typescript'; + +const ROOT = new URL('..', import.meta.url).pathname; +const PREMIUM_PATHS_SRC = join(ROOT, 'src/shared/premium-paths.ts'); +const GEN_CLIENT_DIR = join(ROOT, 'src/generated/client'); +const SRC_DIR = join(ROOT, 'src'); + +function walk(dir, fn) { + for (const name of readdirSync(dir)) { + const full = join(dir, name); + const s = statSync(full); + if (s.isDirectory()) walk(full, fn); + else if (s.isFile()) fn(full); + } +} + +async function loadPremiumPaths() { + // Dynamic import via file URL — runs under tsx (the shebang) which + // transparently transpiles TS. Importing the live Set means any reformat of + // the source literal (single→double quotes, spread, helper-computed entries) + // can never desync the lint from the runtime. + const mod = await import(pathToFileURL(PREMIUM_PATHS_SRC).href); + if (!(mod.PREMIUM_RPC_PATHS instanceof Set) || mod.PREMIUM_RPC_PATHS.size === 0) { + throw new Error( + `${PREMIUM_PATHS_SRC} must export PREMIUM_RPC_PATHS as a non-empty Set — the lint relies on it.`, + ); + } + return mod.PREMIUM_RPC_PATHS; +} + +function loadClientClassMap() { + // AST walk rather than regex — the earlier regex + // /async (\w+)\s*\([^)]*\)\s*:\s*Promise<[^>]+>\s*\{\s*let path = "([^"]+)"/ + // assumed (a) no nested `)` in arg types, (b) no nested `>` in the return + // type, (c) `let path = "..."` as the literal first statement. Any shift in + // the codegen template would silently drop methods and the lint would pass + // clean with missing coverage — the same silent-drift class this PR closed + // on the premium-paths side (#3287 greptile nit 2). + const map = new Map(); + walk(GEN_CLIENT_DIR, (file) => { + if (basename(file) !== 'service_client.ts') return; + const src = readFileSync(file, 'utf8'); + const ast = ts.createSourceFile(file, src, ts.ScriptTarget.Latest, true); + + function visit(node) { + if ( + ts.isClassDeclaration(node) && + node.name && + /ServiceClient$/.test(node.name.text) && + node.modifiers?.some((m) => m.kind === ts.SyntaxKind.ExportKeyword) + ) { + const methods = new Map(); + for (const member of node.members) { + if (!ts.isMethodDeclaration(member)) continue; + if (!member.name || !ts.isIdentifier(member.name)) continue; + const methodName = member.name.text; + const body = member.body; + if (!body) continue; + // Look for the first `let path = "/api/..."` variable statement in + // the method body. Generated clients open each RPC method with it. + for (const stmt of body.statements) { + if (!ts.isVariableStatement(stmt)) continue; + const decl = stmt.declarationList.declarations[0]; + if ( + decl && + ts.isIdentifier(decl.name) && + decl.name.text === 'path' && + decl.initializer && + ts.isStringLiteral(decl.initializer) + ) { + methods.set(methodName, decl.initializer.text); + break; + } + } + } + map.set(node.name.text, methods); + } + ts.forEachChild(node, visit); + } + visit(ast); + }); + if (map.size === 0) { + throw new Error(`No ServiceClient classes parsed from ${GEN_CLIENT_DIR}`); + } + return map; +} + +function collectSourceFiles() { + const out = []; + walk(SRC_DIR, (file) => { + if (file.startsWith(GEN_CLIENT_DIR)) return; + if (!/\.(ts|tsx)$/.test(file)) return; + if (file.endsWith('.d.ts')) return; + out.push(file); + }); + return out; +} + +function getFetchOptionText(optionsArg) { + if (!optionsArg) return null; + if (!ts.isObjectLiteralExpression(optionsArg)) return optionsArg.getText(); + for (const prop of optionsArg.properties) { + if (!ts.isPropertyAssignment(prop)) continue; + const name = prop.name && ts.isIdentifier(prop.name) ? prop.name.text : null; + if (name === 'fetch') return prop.initializer.getText(); + } + return null; +} + +function checkFile(filePath, clientClassMap, premiumPaths) { + const src = readFileSync(filePath, 'utf8'); + const ast = ts.createSourceFile(filePath, src, ts.ScriptTarget.Latest, true); + + const instances = []; + + function recordInstance(varName, newExpr, posNode) { + const className = newExpr.expression.getText(); + if (!clientClassMap.has(className)) return; + const optionsArg = newExpr.arguments?.[1] ?? null; + const lc = ast.getLineAndCharacterOfPosition(posNode.getStart()); + instances.push({ + varName, + className, + optionsArg, + line: lc.line + 1, + column: lc.character + 1, + }); + } + + function visit(node) { + if ( + ts.isVariableDeclaration(node) && + node.initializer && + ts.isNewExpression(node.initializer) && + ts.isIdentifier(node.name) + ) { + recordInstance(node.name.text, node.initializer, node); + } else if ( + ts.isBinaryExpression(node) && + node.operatorToken.kind === ts.SyntaxKind.EqualsToken && + ts.isNewExpression(node.right) + ) { + const lhs = node.left.getText(); + recordInstance(lhs, node.right, node); + } + ts.forEachChild(node, visit); + } + visit(ast); + + if (instances.length === 0) return []; + + const violations = []; + for (const inst of instances) { + const methods = clientClassMap.get(inst.className); + const calledMethods = new Set(); + + // Scope-blind walk — matches any `.()` anywhere in the + // file. If two constructions in different function scopes share the same + // variable name (e.g. both declare `const client = new XServiceClient()` + // in unrelated functions), their called-method sets merge and the lint + // errs on the side of caution (flags premium calls against both + // instances). No current src/ file hits this — keeping the walker + // simple until scope-aware binding is actually needed (#3287 nit 5). + function findCalls(node) { + if ( + ts.isCallExpression(node) && + ts.isPropertyAccessExpression(node.expression) + ) { + const objText = node.expression.expression.getText(); + const methodName = node.expression.name.text; + if (objText === inst.varName) calledMethods.add(methodName); + } + ts.forEachChild(node, findCalls); + } + findCalls(ast); + + const premiumCalls = [...calledMethods].filter((m) => { + const path = methods.get(m); + return path && premiumPaths.has(path); + }); + if (premiumCalls.length === 0) continue; + + const fetchText = getFetchOptionText(inst.optionsArg); + if (fetchText === 'premiumFetch') continue; + + violations.push({ + file: filePath, + line: inst.line, + column: inst.column, + varName: inst.varName, + className: inst.className, + fetchText: fetchText ?? '', + premiumCalls, + }); + } + return violations; +} + +async function main() { + const premiumPaths = await loadPremiumPaths(); + const clientClassMap = loadClientClassMap(); + const files = collectSourceFiles(); + + const violations = []; + for (const f of files) { + violations.push(...checkFile(f, clientClassMap, premiumPaths)); + } + + if (violations.length > 0) { + console.error( + `\u2717 ${violations.length} ServiceClient instantiation(s) call PREMIUM_RPC_PATHS methods without { fetch: premiumFetch }:\n`, + ); + for (const v of violations) { + const rel = relative(ROOT, v.file); + console.error(` ${rel}:${v.line}:${v.column}`); + console.error(` new ${v.className}(...) bound to \`${v.varName}\``); + console.error(` fetch option: ${v.fetchText}`); + console.error(` premium method(s) called: ${v.premiumCalls.join(', ')}`); + console.error(''); + } + console.error('Each ServiceClient that calls a method whose path is in'); + console.error('src/shared/premium-paths.ts PREMIUM_RPC_PATHS must be constructed with'); + console.error(' { fetch: premiumFetch }'); + console.error('imported from @/services/premium-fetch.\n'); + console.error('Why: globalThis.fetch sends no auth header, so signed-in browser pros'); + console.error('without a WORLDMONITOR_API_KEY get a 401 the generated client swallows'); + console.error('into the empty fallback. premiumFetch injects WM key / Clerk bearer when'); + console.error('available and no-ops safely otherwise — safe to use even on a client whose'); + console.error('other methods target public paths (see src/services/supply-chain/index.ts).\n'); + console.error('If a single class needs both gated and ungated calls, split into two'); + console.error('instances — one with premiumFetch (used for premium methods) and one with'); + console.error('globalThis.fetch (used for public methods only). See src/services/trade/'); + console.error('index.ts for the publicClient + premiumClient pattern.\n'); + console.error('Reference: HIGH(new) #1 in #3242 review — SupplyChainServiceClient was'); + console.error('constructed with globalThis.fetch and pro users saw silent empty country-'); + console.error('products + multi-sector-cost-shock panels until commit 01518c3c.'); + process.exit(1); + } + + console.log( + `\u2713 premium-fetch parity clean: ${clientClassMap.size} ServiceClient classes scanned, ${premiumPaths.size} premium paths checked, ${files.length} src/ files analyzed.`, + ); +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/scripts/enforce-rate-limit-policies.mjs b/scripts/enforce-rate-limit-policies.mjs index 78e2cff8d..784a5d2b7 100644 --- a/scripts/enforce-rate-limit-policies.mjs +++ b/scripts/enforce-rate-limit-policies.mjs @@ -1,4 +1,4 @@ -#!/usr/bin/env node +#!/usr/bin/env -S npx tsx /** * Validates every key in ENDPOINT_RATE_POLICIES (server/_shared/rate-limit.ts) * is a real gateway route by checking the OpenAPI specs generated from protos. @@ -8,50 +8,54 @@ * `/api/sanctions/v1/lookup-sanction-entity`, so the 30/min limit never * applied and the endpoint fell through to the 600/min global limiter). * - * Runs in the same pre-push + CI context as lint:api-contract. + * Runs in the same pre-push + CI context as lint:api-contract. Invoked via + * `tsx` so it can import the policy object straight from the TS source + * (#3278) — the previous regex-parse implementation would silently break if + * the source object literal was reformatted. */ import { readFileSync, readdirSync } from 'node:fs'; import { join } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { parse as parseYaml } from 'yaml'; const ROOT = new URL('..', import.meta.url).pathname; const OPENAPI_DIR = join(ROOT, 'docs/api'); const RATE_LIMIT_SRC = join(ROOT, 'server/_shared/rate-limit.ts'); -function extractPolicyKeys() { - const src = readFileSync(RATE_LIMIT_SRC, 'utf8'); - const match = src.match(/ENDPOINT_RATE_POLICIES:\s*Record<[^>]+>\s*=\s*\{([\s\S]*?)\n\};/); - if (!match) { - throw new Error('Could not locate ENDPOINT_RATE_POLICIES in rate-limit.ts'); +async function extractPolicyKeys() { + // Dynamic import via the file URL — works under tsx (the shebang) which + // transparently transpiles TS. Importing the live object means any reformat + // of the source literal can never desync the lint from the runtime. + const mod = await import(pathToFileURL(RATE_LIMIT_SRC).href); + if (!mod.ENDPOINT_RATE_POLICIES || typeof mod.ENDPOINT_RATE_POLICIES !== 'object') { + throw new Error( + `${RATE_LIMIT_SRC} no longer exports ENDPOINT_RATE_POLICIES — the lint relies on it (#3278).`, + ); } - const block = match[1]; - const keys = []; - // Match quoted keys: '/api/...' or "/api/..." - const keyRe = /['"](\/api\/[^'"]+)['"]\s*:/g; - let m; - while ((m = keyRe.exec(block)) !== null) { - keys.push(m[1]); - } - return keys; + return Object.keys(mod.ENDPOINT_RATE_POLICIES); } function extractRoutesFromOpenApi() { + // Parse the OpenAPI YAML rather than regex-scrape for top-level `paths:` + // keys — the earlier `/^\s{4}(\/api\/[^\s:]+):/gm` hard-coded 4-space + // indent, so any YAML formatter change (2-space indent, flow style, line + // folding) would silently drop routes and let policy-drift slip through + // (#3287 greptile nit 3). const routes = new Set(); const files = readdirSync(OPENAPI_DIR).filter((f) => f.endsWith('.openapi.yaml')); for (const file of files) { - const yaml = readFileSync(join(OPENAPI_DIR, file), 'utf8'); - // OpenAPI paths section — each route is a top-level key under `paths:` - // indented 4 spaces. Strip trailing colon. - const pathRe = /^\s{4}(\/api\/[^\s:]+):/gm; - let m; - while ((m = pathRe.exec(yaml)) !== null) { - routes.add(m[1]); + const doc = parseYaml(readFileSync(join(OPENAPI_DIR, file), 'utf8')); + const paths = doc?.paths; + if (!paths || typeof paths !== 'object') continue; + for (const route of Object.keys(paths)) { + if (route.startsWith('/api/')) routes.add(route); } } return routes; } -function main() { - const keys = extractPolicyKeys(); +async function main() { + const keys = await extractPolicyKeys(); const routes = extractRoutesFromOpenApi(); const missing = keys.filter((k) => !routes.has(k)); @@ -73,4 +77,7 @@ function main() { console.log(`✓ rate-limit policies clean: ${keys.length} policies validated against ${routes.size} gateway routes.`); } -main(); +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/server/_shared/rate-limit.ts b/server/_shared/rate-limit.ts index 8e67bda3b..a96fa5688 100644 --- a/server/_shared/rate-limit.ts +++ b/server/_shared/rate-limit.ts @@ -77,7 +77,11 @@ interface EndpointRatePolicy { window: Duration; } -const ENDPOINT_RATE_POLICIES: Record = { +// Exported so scripts/enforce-rate-limit-policies.mjs can import it directly +// (#3278) instead of regex-parsing this file. Internal callers should keep +// using checkEndpointRateLimit / hasEndpointRatePolicy below — the export is +// for tooling, not new runtime callers. +export const ENDPOINT_RATE_POLICIES: Record = { '/api/news/v1/summarize-article-cache': { limit: 3000, window: '60 s' }, '/api/intelligence/v1/classify-event': { limit: 600, window: '60 s' }, // Legacy /api/sanctions-entity-search rate limit was 30/min per IP. Preserve diff --git a/server/worldmonitor/military/v1/list-military-flights.ts b/server/worldmonitor/military/v1/list-military-flights.ts index 35c9c154c..d4ad20eb8 100644 --- a/server/worldmonitor/military/v1/list-military-flights.ts +++ b/server/worldmonitor/military/v1/list-military-flights.ts @@ -147,15 +147,40 @@ function staleToProto(f: StaleFlight): ListMilitaryFlightsResponse['flights'][nu }; } +// Negative cache for the stale Redis read — mirrors the legacy +// /api/military-flights handler's NEG_TTL=30_000ms. When the live fetch fails +// AND the stale key is also empty/unparseable, suppress further Redis reads +// of REDIS_STALE_KEY for STALE_NEG_TTL_MS so we don't hammer Redis once per +// request during sustained relay+seed outages. Per-isolate (Vercel Edge state), +// which is fine — each warm isolate gets its own 30s suppression window. +const STALE_NEG_TTL_MS = 30_000; +let staleNegUntil = 0; + +// Test seam — exposed for unit tests that need to drive the suppression +// window without sleeping. Not exported from the module's public API. +export function _resetStaleNegativeCacheForTests(): void { + staleNegUntil = 0; +} + async function fetchStaleFallback(): Promise { + const now = Date.now(); + if (now < staleNegUntil) return null; try { const raw = (await getRawJson(REDIS_STALE_KEY)) as StalePayload | null; - if (!raw || !Array.isArray(raw.flights) || raw.flights.length === 0) return null; + if (!raw || !Array.isArray(raw.flights) || raw.flights.length === 0) { + staleNegUntil = now + STALE_NEG_TTL_MS; + return null; + } const flights = raw.flights .map(staleToProto) .filter((f): f is NonNullable => f != null); - return flights.length > 0 ? flights : null; + if (flights.length === 0) { + staleNegUntil = now + STALE_NEG_TTL_MS; + return null; + } + return flights; } catch { + staleNegUntil = now + STALE_NEG_TTL_MS; return null; } } diff --git a/server/worldmonitor/shipping/v2/register-webhook.ts b/server/worldmonitor/shipping/v2/register-webhook.ts index 0373cec40..22e09a6ce 100644 --- a/server/worldmonitor/shipping/v2/register-webhook.ts +++ b/server/worldmonitor/shipping/v2/register-webhook.ts @@ -66,12 +66,16 @@ export async function registerWebhook( ]); } - // Proto default int32 is 0 — treat 0 as "unset" to preserve the legacy - // default of 50 when the caller omits alertThreshold. - const alertThreshold = req.alertThreshold > 0 ? req.alertThreshold : 50; + // alert_threshold is `optional int32` (#3242 followup #4) — undefined means + // the partner omitted the field, so apply the legacy default of 50. An + // explicit 0 is preserved (deliver every alert). The 0..100 range is + // normally enforced by buf.validate at the wire layer, but we re-enforce + // it here so direct handler calls (internal jobs, test harnesses, future + // transports that bypass buf.validate) can't store out-of-range values. + const alertThreshold = req.alertThreshold ?? 50; if (alertThreshold < 0 || alertThreshold > 100) { throw new ValidationError([ - { field: 'alertThreshold', description: 'alertThreshold must be a number between 0 and 100' }, + { field: 'alertThreshold', description: 'alertThreshold must be between 0 and 100' }, ]); } diff --git a/server/worldmonitor/shipping/v2/webhook-shared.ts b/server/worldmonitor/shipping/v2/webhook-shared.ts index 880950326..596e42e93 100644 --- a/server/worldmonitor/shipping/v2/webhook-shared.ts +++ b/server/worldmonitor/shipping/v2/webhook-shared.ts @@ -4,8 +4,24 @@ export const WEBHOOK_TTL = 86400 * 30; // 30 days export const VALID_CHOKEPOINT_IDS = new Set(CHOKEPOINT_REGISTRY.map(c => c.id)); // Private IP ranges + known cloud metadata hostnames blocked at registration. -// DNS rebinding is not mitigated here (no DNS resolution in edge runtime); the -// delivery worker must re-resolve and re-check before sending. +// +// DNS rebinding is NOT mitigated by isBlockedCallbackUrl below — the Vercel +// Edge runtime can't resolve hostnames before the request goes out. Defense +// against a hostname that returns a public IP at registration time and a +// private IP later (or different IPs per resolution) MUST happen in the +// delivery worker that actually POSTs to the callback URL: +// +// 1. Re-validate the URL with isBlockedCallbackUrl right before each send. +// 2. Resolve the hostname to its current IP via dns.promises.lookup +// (Node runtime — Edge can't do this). +// 3. Verify the resolved IP is not in PRIVATE_HOSTNAME_PATTERNS or +// BLOCKED_METADATA_HOSTNAMES. +// 4. Issue the fetch using the resolved IP with the Host header preserved +// so TLS still validates against the original hostname. +// +// As of the #3242 followup audit, no delivery worker for shipping/v2 webhooks +// exists in this repo — tracked in issue #3288. Anyone landing delivery code +// MUST import the patterns + sets above and apply steps 1–3 before each send. export const PRIVATE_HOSTNAME_PATTERNS = [ /^localhost$/i, /^127\.\d+\.\d+\.\d+$/, diff --git a/src/generated/client/worldmonitor/shipping/v2/service_client.ts b/src/generated/client/worldmonitor/shipping/v2/service_client.ts index 8039b582d..f7a5be043 100644 --- a/src/generated/client/worldmonitor/shipping/v2/service_client.ts +++ b/src/generated/client/worldmonitor/shipping/v2/service_client.ts @@ -39,7 +39,7 @@ export interface BypassOption { export interface RegisterWebhookRequest { callbackUrl: string; chokepointIds: string[]; - alertThreshold: number; + alertThreshold?: number; } export interface RegisterWebhookResponse { diff --git a/src/generated/server/worldmonitor/shipping/v2/service_server.ts b/src/generated/server/worldmonitor/shipping/v2/service_server.ts index 8483af986..2124ed242 100644 --- a/src/generated/server/worldmonitor/shipping/v2/service_server.ts +++ b/src/generated/server/worldmonitor/shipping/v2/service_server.ts @@ -39,7 +39,7 @@ export interface BypassOption { export interface RegisterWebhookRequest { callbackUrl: string; chokepointIds: string[]; - alertThreshold: number; + alertThreshold?: number; } export interface RegisterWebhookResponse { diff --git a/src/services/correlation-engine/engine.ts b/src/services/correlation-engine/engine.ts index 0d271640a..6ba897679 100644 --- a/src/services/correlation-engine/engine.ts +++ b/src/services/correlation-engine/engine.ts @@ -9,6 +9,7 @@ import type { } from './types'; import { haversineKm } from '@/utils/distance'; import { IntelligenceServiceClient } from '@/generated/client/worldmonitor/intelligence/v1/service_client'; +import { premiumFetch } from '@/services/premium-fetch'; import { hasPremiumAccess } from '@/services/panel-gating'; const LLM_SCORE_THRESHOLD = 60; @@ -30,8 +31,11 @@ export class CorrelationEngine { private llmInFlight = 0; constructor() { - // Use '' base URL — requests go to current origin, same as other panels - this.intelligenceClient = new IntelligenceServiceClient(''); + // Use '' base URL — requests go to current origin, same as other panels. + // premiumFetch — deductSituation is in PREMIUM_RPC_PATHS. globalThis.fetch + // (the generated default) would 401 signed-in browser pros so the LLM + // assessment never lands. See #3242 review HIGH(new) #1 for the bug class. + this.intelligenceClient = new IntelligenceServiceClient('', { fetch: premiumFetch }); } registerAdapter(adapter: DomainAdapter): void { diff --git a/src/services/economic/index.ts b/src/services/economic/index.ts index 67d1b77f4..2ff75e0bf 100644 --- a/src/services/economic/index.ts +++ b/src/services/economic/index.ts @@ -8,6 +8,7 @@ */ import { getRpcBaseUrl } from '@/services/rpc-client'; +import { premiumFetch } from '@/services/premium-fetch'; import { EconomicServiceClient, ApiError, @@ -53,7 +54,14 @@ import { toApiUrl } from '@/services/runtime'; // ---- Client + Circuit Breakers ---- -const client = new EconomicServiceClient(getRpcBaseUrl(), { fetch: (...args) => globalThis.fetch(...args) }); +// premiumFetch for the whole client: 1 of ~16 methods (getNationalDebt) targets a +// PREMIUM_RPC_PATHS path. globalThis.fetch here would 401 signed-in browser pros +// on getNationalDebt with no WORLDMONITOR_API_KEY (gateway runs validateApiKey +// with forceKey=true on premium paths). premiumFetch no-ops safely when no +// credentials are available, so the public methods (FRED, BLS, energy, BIS, +// EU, oil) keep working unchanged. See src/services/supply-chain/index.ts for +// the same pattern + #3242 review HIGH(new) #1 for the bug class this prevents. +const client = new EconomicServiceClient(getRpcBaseUrl(), { fetch: premiumFetch }); const WB_BREAKERS_WARN_THRESHOLD = 50; const wbBreakers = new Map>>(); diff --git a/src/services/sanctions-pressure.ts b/src/services/sanctions-pressure.ts index cc59afd82..83ffa6826 100644 --- a/src/services/sanctions-pressure.ts +++ b/src/services/sanctions-pressure.ts @@ -1,5 +1,6 @@ import { createCircuitBreaker } from '@/utils'; import { getRpcBaseUrl } from '@/services/rpc-client'; +import { premiumFetch } from '@/services/premium-fetch'; import { getHydratedData } from '@/services/bootstrap'; import { SanctionsServiceClient, @@ -54,7 +55,10 @@ export interface SanctionsPressureResult { entries: SanctionsEntry[]; } -const client = new SanctionsServiceClient(getRpcBaseUrl(), { fetch: (...args) => globalThis.fetch(...args) }); +// premiumFetch — listSanctionsPressure (the only method called here) is in +// PREMIUM_RPC_PATHS. See src/services/supply-chain/index.ts for the pattern +// and #3242 review HIGH(new) #1 for the bug class this prevents. +const client = new SanctionsServiceClient(getRpcBaseUrl(), { fetch: premiumFetch }); const breaker = createCircuitBreaker({ name: 'Sanctions Pressure', cacheTtlMs: 30 * 60 * 1000, diff --git a/tests/redis-caching.test.mjs b/tests/redis-caching.test.mjs index 77bc9eec1..c813ab05a 100644 --- a/tests/redis-caching.test.mjs +++ b/tests/redis-caching.test.mjs @@ -878,4 +878,74 @@ describe('military flights bbox behavior', { concurrency: 1 }, () => { restoreEnv(); } }); + + // #3277 — fetchStaleFallback NEG_TTL parity with the legacy + // /api/military-flights handler. Without the negative cache, a sustained + // relay+seed outage would Redis-hammer the stale key on every request. + it('suppresses stale Redis read for 30s after a stale-key miss (NEG_TTL parity)', async () => { + const { module, cleanup } = await importListMilitaryFlights(); + module._resetStaleNegativeCacheForTests(); + + const restoreEnv = withEnv({ + UPSTASH_REDIS_REST_URL: 'https://redis.test', + UPSTASH_REDIS_REST_TOKEN: 'token', + LOCAL_API_MODE: undefined, + WS_RELAY_URL: undefined, + VERCEL_ENV: undefined, + VERCEL_GIT_COMMIT_SHA: undefined, + }); + const originalFetch = globalThis.fetch; + + const staleGetCalls = []; + globalThis.fetch = async (url) => { + const raw = String(url); + if (raw.includes('/get/')) { + if (raw.includes('military%3Aflights%3Astale%3Av1')) { + staleGetCalls.push(raw); + } + // Both keys empty — drives cachedFetchJson to call the fetcher + // (which returns null because no relay) and then the handler falls + // through to fetchStaleFallback (which returns null because stale + // is also empty → arms the negative cache). + return jsonResponse({ result: null }); + } + throw new Error(`Unexpected fetch URL: ${raw}`); + }; + + try { + const ctx = { request: new Request('https://wm.test/api/military/v1/list-military-flights') }; + + // Call 1 — live empty + stale empty. Stale key MUST be read once, + // and the negative cache MUST be armed for the next 30s. + const r1 = await module.listMilitaryFlights(ctx, request); + assert.deepEqual(r1.flights, [], 'no live, no stale → empty response'); + assert.equal(staleGetCalls.length, 1, 'first call reads stale key once'); + + // Call 2 — within the 30s negative-cache window. Live cache may be + // re-checked but the stale key MUST NOT be re-read. + staleGetCalls.length = 0; + const r2 = await module.listMilitaryFlights(ctx, request); + assert.deepEqual(r2.flights, [], 'still empty during negative-cache window'); + assert.equal( + staleGetCalls.length, + 0, + 'second call within NEG_TTL window must not re-read stale key', + ); + + // Reset the negative cache (simulates wall-clock advance past 30s) → + // stale read should resume. + module._resetStaleNegativeCacheForTests(); + const r3 = await module.listMilitaryFlights(ctx, request); + assert.deepEqual(r3.flights, []); + assert.equal( + staleGetCalls.length, + 1, + 'after negative-cache reset, stale key is re-read', + ); + } finally { + cleanup(); + globalThis.fetch = originalFetch; + restoreEnv(); + } + }); }); diff --git a/tests/shipping-v2-handler.test.mjs b/tests/shipping-v2-handler.test.mjs index b0ac3ee29..efe84c0df 100644 --- a/tests/shipping-v2-handler.test.mjs +++ b/tests/shipping-v2-handler.test.mjs @@ -207,12 +207,27 @@ describe('ShippingV2Service handlers', () => { ); }); + // alert_threshold 0..100 range is enforced primarily by buf.validate at + // the wire layer. The handler re-enforces it so direct invocations + // (internal jobs, test harnesses, future transports) can't store out-of- + // range values — cheap invariant-at-the-boundary (#3287 review nit 1). it('rejects alertThreshold > 100 with ValidationError', async () => { await assert.rejects( () => registerWebhook(proCtx(), { callbackUrl: 'https://hooks.example.com/wm', chokepointIds: [], - alertThreshold: 150, + alertThreshold: 9999, + }), + (err) => err instanceof ValidationError && err.violations[0].field === 'alertThreshold', + ); + }); + + it('rejects alertThreshold < 0 with ValidationError', async () => { + await assert.rejects( + () => registerWebhook(proCtx(), { + callbackUrl: 'https://hooks.example.com/wm', + chokepointIds: [], + alertThreshold: -1, }), (err) => err instanceof ValidationError && err.violations[0].field === 'alertThreshold', ); @@ -246,7 +261,22 @@ describe('ShippingV2Service handlers', () => { assert.equal(pipeline[2][2], String(86400 * 30)); }); - it('alertThreshold 0 (proto default) coerces to legacy default 50', async () => { + it('alertThreshold omitted (undefined) applies the legacy default of 50', async () => { + const calls = stubRedisOk(); + await registerWebhook(proCtx(), { + callbackUrl: 'https://hooks.example.com/wm', + chokepointIds: [], + // alertThreshold omitted — proto3 `optional int32` arrives as undefined + }); + const record = JSON.parse(calls[0][0][2]); + assert.equal(record.alertThreshold, 50); + }); + + it('alertThreshold explicit 0 is preserved (deliver every alert)', async () => { + // #3242 followup #4 — proto3 `optional` lets the handler distinguish + // "partner explicitly sent 0" from "partner omitted the field". The + // pre-fix handler coerced both to 50, silently dropping the partner's + // intent to receive every disruption. const calls = stubRedisOk(); await registerWebhook(proCtx(), { callbackUrl: 'https://hooks.example.com/wm', @@ -254,7 +284,7 @@ describe('ShippingV2Service handlers', () => { alertThreshold: 0, }); const record = JSON.parse(calls[0][0][2]); - assert.equal(record.alertThreshold, 50); + assert.equal(record.alertThreshold, 0); }); it('empty chokepointIds subscribes to the full CHOKEPOINT_REGISTRY', async () => {