mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
main
149 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
5c955691a9 |
feat(energy-atlas): live tanker map layer + contract (parity PR 3, plan U7-U8) (#3402)
* feat(energy-atlas): live tanker map layer + contract (PR 3, plan U7-U8)
Lands the third and final parity-push surface — per-vessel tanker positions
inside chokepoint bounding boxes, refreshed every 60s. Closes the visual
gap with peer reference energy-intel sites for the live AIS tanker view.
Per docs/plans/2026-04-25-003-feat-energy-parity-pushup-plan.md PR 3.
Codex-approved through 8 review rounds against origin/main @
|
||
|
Elie Habib
|
7c0c08ad89 |
feat(energy-atlas): seed-side countries[] denorm on disruptions + CountryDeepDive row (§R #5 = B) (#3377)
* feat(energy-atlas): seed-side countries[] denorm + CountryDeepDive row (§R #5 = B)
Per plan §R/#5 decision B: denormalise countries[] at seed time on each
disruption event so CountryDeepDivePanel can filter events per country
without an asset-registry round trip. Schema join (pipeline/storage
→ event.assetId) happens once in the weekly cron, not on every panel
render. The alternative (client-side join) was rejected because it
couples UI logic to asset-registry internals and duplicates the join
for every surface that wants a per-country filter.
Changes:
- `proto/.../list_energy_disruptions.proto`: add `repeated string
countries = 15` to EnergyDisruptionEntry with doc comment tying it
to the plan decision and the always-non-empty invariant.
- `scripts/_energy-disruption-registry.mjs`:
• Load pipeline-gas + pipeline-oil + storage-facilities registries
once per seed cycle; index by id.
• `deriveCountriesForEvent()` resolves assetId to {fromCountry,
toCountry, transitCountries} (pipeline) or {country} (storage),
deduped + alpha-sorted so byte-diff stability holds.
• `buildPayload()` attaches the computed countries[] to every
event before writing.
• `validateRegistry()` now requires non-empty countries[] of
ISO2 codes. Combined with the seeder's `emptyDataIsFailure:
true`, this surfaces orphaned assetIds loudly — the next cron
tick fails validation and seed-meta stays stale, tripping
health alarms.
- `scripts/data/energy-disruptions.json`: fix two orphaned assetIds
that the new join caught:
• `cpc-force-majeure-2022`: `cpc-pipeline` → `cpc` (matches the
entry in pipelines-oil.json).
• `pdvsa-designation-2019`: `ve-petrol-2026-q1` (non-existent) →
`venezuela-anzoategui-puerto-la-cruz`.
- `server/.../list-energy-disruptions.ts`: project countries[] into
the RPC response via coerceStringArray. Legacy pre-denorm rows
surface as empty array (always present on wire, length 0 => old).
- `src/components/CountryDeepDivePanel.ts`: add 4th Atlas row —
"Energy disruptions in {iso2}" — filtered by `iso2 ∈ countries[]`.
Failure is silent; EnergyDisruptionsPanel (upcoming) is the
primary disruption surface.
- `tests/energy-disruptions-registry.test.mts`: switch to validating
the buildPayload output (post-denorm), add §R #5 B invariant
tests, plus a raw-JSON invariant ensuring curators don't hand-edit
countries[] (it's derived, not declared).
Proto regen note: `make generate` currently fails with a duplicate
openapi plugin collision in buf.gen.yaml (unrelated bug — 3 plugin
entries emit to the same out dir). Worked around by temporarily
trimming buf.gen.yaml to just the TS plugins for this regen. Added
only the `countries: string[]` wire field to both service_client and
service_server; no other generated-file drift in this PR.
* chore(proto): regenerate openapi specs for countries[] field
Runs `make generate` with the sebuf v0.11.1 plugin now correctly
resolved via the PATH fix (cherry-picked from fix/makefile-generate-path-prefix).
The new `countries` field on EnergyDisruptionEntry propagates into:
- docs/api/SupplyChainService.openapi.yaml (primary per-service spec)
- docs/api/SupplyChainService.openapi.json (machine-readable variant)
- docs/api/worldmonitor.openapi.yaml (consolidated bundle)
No TypeScript drift beyond the already-committed service_client.ts /
service_server.ts updates in
|
||
|
Sebastien Melki
|
e68a7147dd |
chore(api): sebuf migration follow-ups (post-#3242) (#3287)
* chore(api-manifest): rewrite brief-why-matters reason as proper internal-helper justification Carried in from #3248 merge as a band-aid (called out in #3242 review followup checklist item 7). The endpoint genuinely belongs in internal-helper — RELAY_SHARED_SECRET-bearer auth, cron-only caller, never reached by dashboards or partners. Same shape constraint as api/notify.ts. Replaces the apologetic "filed here to keep the lint green" framing with a proper structural justification: modeling it as a generated service would publish internal cron plumbing as user-facing API surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lint): premium-fetch parity check for ServiceClients (closes #3279) Adds scripts/enforce-premium-fetch.mjs — AST-walks src/, finds every `new <ServiceClient>(...)` (variable decl OR `this.foo =` assignment), tracks which methods each instance actually calls, and fails if any called method targets a path in src/shared/premium-paths.ts PREMIUM_RPC_PATHS without `{ fetch: premiumFetch }` on the constructor. Per-call-site analysis (not class-level) keeps the trade/index.ts pattern clean — publicClient with globalThis.fetch + premiumClient with premiumFetch on the same TradeServiceClient class — since publicClient never calls a premium method. Wired into: - npm run lint:premium-fetch - .husky/pre-push (right after lint:rate-limit-policies) - .github/workflows/lint-code.yml (right after lint:api-contract) Found and fixed three latent instances of the HIGH(new) #1 class from #3242 review (silent 401 → empty fallback for signed-in browser pros): - src/services/correlation-engine/engine.ts — IntelligenceServiceClient built with no fetch option called deductSituation. LLM-assessment overlay on convergence cards never landed for browser pros without a WM key. - src/services/economic/index.ts — EconomicServiceClient with globalThis.fetch called getNationalDebt. National-debt panel rendered empty for browser pros. - src/services/sanctions-pressure.ts — SanctionsServiceClient with globalThis.fetch called listSanctionsPressure. Sanctions-pressure panel rendered empty for browser pros. All three swap to premiumFetch (single shared client, mirrors the supply-chain/index.ts justification — premiumFetch no-ops safely on public methods, so the public methods on those clients keep working). Verification: - lint:premium-fetch clean (34 ServiceClient classes, 28 premium paths, 466 src/ files analyzed) - Negative test: revert any of the three to globalThis.fetch → exit 1 with file:line and called-premium-method names - typecheck + typecheck:api clean - lint:api-contract / lint:rate-limit-policies / lint:boundaries clean - tests/sanctions-pressure.test.mjs + premium-fetch.test.mts: 16/16 pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(military): fetchStaleFallback NEG_TTL=30s parity (closes #3277) The legacy /api/military-flights handler had NEG_TTL = 30_000ms — a short suppression window after a failed live + stale read so we don't Redis-hammer the stale key during sustained relay+seed outages. Carried into the sebuf list-military-flights handler: - Module-scoped `staleNegUntil` timestamp (per-isolate on Vercel Edge, which is fine — each warm isolate gets its own 30s suppression window). - Set whenever fetchStaleFallback returns null (key missing, parse fail, empty array after staleToProto filter, or thrown error). - Checked at the entry of fetchStaleFallback before doing the Redis read. - Test seam `_resetStaleNegativeCacheForTests()` exposed for unit tests. Test pinned in tests/redis-caching.test.mjs: drives a stale-empty cycle three times — first read hits Redis, second within window doesn't, after test-only reset it does again. Verified: 18/18 redis-caching tests pass, typecheck:api clean, lint:premium-fetch clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lint): rate-limit-policies regex → import() (closes #3278) The previous lint regex-parsed ENDPOINT_RATE_POLICIES from the source file. That worked because the literal happens to fit a single line per key today, but a future reformat (multi-line key wrap, formatter swap, etc.) would silently break the lint without breaking the build — exactly the failure mode that's worse than no lint at all. Fix: - Export ENDPOINT_RATE_POLICIES from server/_shared/rate-limit.ts. - Convert scripts/enforce-rate-limit-policies.mjs to async + dynamic import() of the policy object directly. Same TS module that the gateway uses at runtime → no source-of-truth drift possible. - Run via tsx (already a dev dep, used by test:data) so the .mjs shebang can resolve a .ts import. - npm script swapped to `tsx scripts/...`. .husky/pre-push uses `npm run lint:rate-limit-policies` so no hook change needed. Verified: - Clean: 6 policies / 182 gateway routes. - Negative test (rename a key to the original sanctions typo /api/sanctions/v1/lookup-entity): exit 1 with the same incident- attributed remedy message as before. - Reformat test (split a single-line entry across multiple lines): still passes — the property is what's read, not the source layout. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(shipping/v2): alertThreshold: 0 preserved; drop dead validation branch (#3242 followup) Before: alert_threshold was a plain int32. proto3 scalar default is 0, so the handler couldn't distinguish "partner explicitly sent 0 (deliver every disruption)" from "partner omitted the field (apply legacy default 50)" — both arrived as 0 and got coerced to 50 by `> 0 ? : 50`. Silent intent-drop for any partner who wanted every alert. The subsequent `alertThreshold < 0` branch was also unreachable after that coercion. After: - Proto field is `optional int32 alert_threshold` — TS type becomes `alertThreshold?: number`, so omitted = undefined and explicit 0 stays 0. - Handler uses `req.alertThreshold ?? 50` — undefined → 50, any number passes through unchanged. - Dead `< 0 || > 100` runtime check removed; buf.validate `int32.gte = 0, int32.lte = 100` already enforces the range at the wire layer. Partner wire contract: identical for the omit-field and 1..100 cases. Only behavioural change is explicit 0 — previously impossible to request, now honored per proto3 optional semantics. Scoped `buf generate --path worldmonitor/shipping/v2` to avoid the full- regen `@ts-nocheck` drift Seb documented in the #3242 PR comments. Re-applied `@ts-nocheck` on the two regenerated files manually. Tests: - `alertThreshold 0 coerces to 50` flipped to `alertThreshold 0 preserved`. - New test: `alertThreshold omitted (undefined) applies legacy default 50`. - `rejects > 100` test removed — proto/wire validation handles it; direct handler calls intentionally bypass wire and the handler no longer carries a redundant runtime range check. Verified: 18/18 shipping-v2-handler tests pass, typecheck + typecheck:api clean, all 4 custom lints clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(shipping/v2): document missing webhook delivery worker + DNS-rebinding contract (#3242 followup) #3242 followup checklist item 6 from @koala73 — sanity-check that the delivery worker honors the re-resolve-and-re-check contract that isBlockedCallbackUrl explicitly delegates to it. Audit finding: no delivery worker for shipping/v2 webhooks exists in this repo. Grep across the entire tree (excluding generated/dist) shows the only readers of webhook:sub:* records are the registration / inspection / rotate-secret handlers themselves. No code reads them and POSTs to the stored callbackUrl. The delivery worker is presumed to live in Railway (separate repo) or hasn't been built yet — neither is auditable from this repo. Refreshes the comment block at the top of webhook-shared.ts to: - explicitly state DNS rebinding is NOT mitigated at registration - spell out the four-step contract the delivery worker MUST follow (re-validate URL, dns.lookup, re-check resolved IP against patterns, fetch with resolved IP + Host header preserved) - flag the in-repo gap so anyone landing delivery code can't miss it Tracking the gap as #3288 — acceptance there is "delivery worker imports the patterns + helpers from webhook-shared.ts and applies the four steps before each send." Action moves to wherever the delivery worker actually lives (Railway likely). No code change. Tests + lints unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci(lint): add rate-limit-policies step (greptile P1 #3287) Pre-push hook ran lint:rate-limit-policies but the CI workflow did not, so fork PRs and --no-verify pushes bypassed the exact drift check the lint was added to enforce (closes #3278). Adding it right after lint:api-contract so it runs in the same context the lint was designed for. * refactor(lint): premium-fetch regex → import() + loop classRe (greptile P2 #3287) Two fragilities greptile flagged on enforce-premium-fetch.mjs: 1. loadPremiumPaths regex-parsed src/shared/premium-paths.ts with /'(\/api\/[^']+)'/g — same class of silent drift we just removed from enforce-rate-limit-policies in #3278. Reformatting the source Set (double quotes, spread, helper-computed entries) would drop paths from the lint while leaving the runtime untouched. Fix: flip the shebang to `#!/usr/bin/env -S npx tsx` and dynamic-import PREMIUM_RPC_PATHS directly, mirroring the rate-limit pattern. package.json lint:premium-fetch now invokes via tsx too so the npm-script path matches direct execution. 2. loadClientClassMap ran classRe.exec once, silently dropping every ServiceClient after the first if a file ever contained more than one. Current codegen emits one class per file so this was latent, but a template change would ship un-linted classes. Fix: collect every class-open match with matchAll, slice each class body with the next class's start as the boundary, and scan methods per-body so method-to-class binding stays correct even with multiple classes per file. Verification: - lint:premium-fetch clean (34 classes / 28 premium paths / 466 files — identical counts to pre-refactor, so no coverage regression). - Negative test: revert src/services/economic/index.ts to globalThis.fetch → exit 1 with file:line, bound var name, and premium method list (getNationalDebt). Restore → clean. - lint:rate-limit-policies still clean. * fix(shipping/v2): re-add alertThreshold handler range guard (greptile nit 1 #3287) Wire-layer buf.validate enforces 0..100, but direct handler invocation (internal jobs, test harnesses, future transports) bypasses it. Cheap invariant-at-the-boundary — rejects < 0 or > 100 with ValidationError before the record is stored. Tests: restored the rejects-out-of-range cases that were dropped when the branch was (correctly) deleted as dead code on the previous commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lint): premium-fetch method-regex → TS AST (greptile nits 2+5 #3287) loadClientClassMap: The method regex `async (\w+)\s*\([^)]*\)\s*:\s*Promise<[^>]+>\s*\{\s*let path = "..."` assumed (a) no nested `)` in arg types, (b) no nested `>` in the return type, (c) `let path = "..."` as the literal first statement. Any codegen template shift would silently drop methods with the lint still passing clean — the same silent-drift class #3287 just closed on the premium-paths side. Now walks the service_client.ts AST, matches `export class *ServiceClient`, iterates `MethodDeclaration` members, and reads the first `let path: string = '...'` variable statement as a StringLiteral. Tolerant to any reformatting of arg/return types or method shape. findCalls scope-blindness: Added limitation comment — the walker matches `<varName>.<method>()` anywhere in the file without respecting scope. Two constructions in different function scopes sharing a var name merge their called-method sets. No current src/ file hits this; the lint errs cautiously (flags both instances). Keeping the walker simple until scope-aware binding is needed. webhook-shared.ts: Inlined issue reference (#3288) so the breadcrumb resolves without bouncing through an MDX that isn't in the diff. Verification: - lint:premium-fetch clean — 34 classes / 28 premium paths / 489 files. Pre-refactor: 34 / 28 / 466. Class + path counts identical; file bump is from the main-branch rebase, not the refactor. - Negative test: revert src/services/economic/index.ts premiumFetch → globalThis.fetch. Lint exits 1 at `src/services/economic/index.ts:64:7` with `premium method(s) called: getNationalDebt`. Restore → clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lint): rate-limit OpenAPI regex → yaml parser (greptile nit 3 #3287) Input side (ENDPOINT_RATE_POLICIES) was flipped to live `import()` in |
||
|
Elie Habib
|
34dfc9a451 |
fix(news): ground LLM surfaces on real RSS description end-to-end (#3370)
* feat(news/parser): extract RSS/Atom description for LLM grounding (U1)
Add description field to ParsedItem, extract from the first non-empty of
description/content:encoded (RSS) or summary/content (Atom), picking the
longest after HTML-strip + entity-decode + whitespace-normalize. Clip to
400 chars. Reject empty, <40 chars after strip, or normalize-equal to the
headline — downstream consumers fall back to the cleaned headline on '',
preserving current behavior for feeds without a description.
CDATA end is anchored to the closing tag so internal ]]> sequences do not
truncate the match. Preserves cached rss:feed:v1 row compatibility during
the 1h TTL bleed since the field is additive.
Part of fix: pipe RSS description end-to-end so LLM surfaces stop
hallucinating named actors (docs/plans/2026-04-24-001-...).
Covers R1, R7.
* feat(news/story-track): persist description on story:track:v1 HSET (U2)
Append description to the story:track:v1 HSET only when non-empty. Additive
— no key version bump. Old rows and rows from feeds without a description
return undefined on HGETALL, letting downstream readers fall back to the
cleaned headline (R6).
Extract buildStoryTrackHsetFields as a pure helper so the inclusion gate is
unit-testable without Redis.
Update the contract comment in cache-keys.ts so the next reader of the
schema sees description as an optional field.
Covers R2, R6.
* feat(proto): NewsItem.snippet + SummarizeArticleRequest.bodies (U3)
Add two additive proto fields so the article description can ride to every
LLM-adjacent consumer without a breaking change:
- NewsItem.snippet (field 12): RSS/Atom description, HTML-stripped,
≤400 chars, empty when unavailable. Wired on toProtoItem.
- SummarizeArticleRequest.bodies (field 8): optional article bodies
paired 1:1 with headlines for prompt grounding. Empty array is today's
headline-only behavior.
Regenerated TS client/server stubs and OpenAPI YAML/JSON via sebuf v0.11.1
(PATH=~/go/bin required — Homebrew's protoc-gen-openapiv3 is an older
pre-bundle-mode build that collides on duplicate emission).
Pre-emptive bodies:[] placeholders at the two existing SummarizeArticle
call sites in src/services/summarization.ts; U6 replaces them with real
article bodies once SummarizeArticle handler reads the field.
Covers R3, R5.
* feat(brief/digest): forward RSS description end-to-end through brief envelope (U4)
Digest accumulator reader (seed-digest-notifications.mjs::buildDigest) now
plumbs the optional `description` field off each story:track:v1 HGETALL into
the digest story object. The brief adapter (brief-compose.mjs::
digestStoryToUpstreamTopStory) prefers the real RSS description over the
cleaned headline; when the upstream row has no description (old rows in the
48h bleed, feeds that don't carry one), we fall back to the cleaned headline
so today behavior is preserved (R6).
This is the upstream half of the description cache path. U5 lands the LLM-
side grounding + cache-prefix bump so Gemini actually sees the article body
instead of hallucinating a named actor from the headline.
Covers R4 (upstream half), R6.
* feat(brief/llm): RSS grounding + sanitisation + 4 cache prefix bumps (U5)
The actual fix for the headline-only named-actor hallucination class:
Gemini 2.5 Flash now receives the real article body as grounding context,
so it paraphrases what the article says instead of filling role-label
headlines from parametric priors ("Iran's new supreme leader" → "Ali
Khamenei" was the 2026-04-24 reproduction; with grounding, it becomes
the actual article-named actor).
Changes:
- buildStoryDescriptionPrompt interpolates a `Context: <body>` line
between the metadata block and the "One editorial sentence" instruction
when description is non-empty AND not normalise-equal to the headline.
Clips to 400 chars as a second belt-and-braces after the U1 parser cap.
No Context line → identical prompt to pre-fix (R6 preserved).
- sanitizeStoryForPrompt extended to cover `description`. Closes the
asymmetry where whyMatters was sanitised and description wasn't —
untrusted RSS bodies now flow through the same injection-marker
neutraliser before prompt interpolation. generateStoryDescription wraps
the story in sanitizeStoryForPrompt before calling the builder,
matching generateWhyMatters.
- Four cache prefixes bumped atomically to evict pre-grounding rows:
scripts/lib/brief-llm.mjs:
brief:llm:description:v1 → v2 (Railway, description path)
brief:llm:whymatters:v2 → v3 (Railway, whyMatters fallback)
api/internal/brief-why-matters.ts:
brief:llm:whymatters:v6 → v7 (edge, primary)
brief:llm:whymatters:shadow:v4 → shadow:v5 (edge, shadow)
hashBriefStory already includes description in the 6-field material
(v5 contract) so identity naturally drifts; the prefix bump is the
belt-and-braces that guarantees a clean cold-start on first tick.
- Tests: 8 new + 2 prefix-match updates on tests/brief-llm.test.mjs.
Covers Context-line injection, empty/dup-of-headline rejection,
400-char clip, sanitisation of adversarial descriptions, v2 write,
and legacy-v1 row dark (forced cold-start).
Covers R4 + new sanitisation requirement.
* feat(news/summarize): accept bodies + bump summary cache v5→v6 (U6)
SummarizeArticle now grounds on per-headline article bodies when callers
supply them, so the dashboard "News summary" path stops hallucinating
across unrelated headlines when the upstream RSS carried context.
Three coordinated changes:
1. SummarizeArticleRequest handler reads req.bodies, sanitises each entry
through sanitizeForPrompt (same trust treatment as geoContext — bodies
are untrusted RSS text), clips to 400 chars, and pads to the headlines
length so pair-wise identity is stable.
2. buildArticlePrompts accepts optional bodies and interleaves a
` Context: <body>` line under each numbered headline that has a
non-empty body. Skipped in translate mode (headline[0]-only) and when
all bodies are empty — yielding a byte-identical prompt to pre-U6
for every current caller (R6 preserved).
3. summary-cache-key bumps CACHE_VERSION v5→v6 so the pre-grounding rows
(produced from headline-only prompts) cold-start cleanly. Extends
canonicalizeSummaryInputs + buildSummaryCacheKey with a pair-wise
bodies segment `:bd<hash>`; the prefix is `:bd` rather than `:b` to
avoid colliding with `:brief:` when pattern-matching keys. Translate
mode is headline[0]-only and intentionally does not shift on bodies.
Dedup reorder preserved: the handler re-pairs bodies to the deduplicated
top-5 via findIndex, so layout matches without breaking cache identity.
New tests: 7 on buildArticlePrompts (bodies interleave, partial fill,
translate-mode skip, clip, short-array tolerance), 8 on
buildSummaryCacheKey (pair-wise sort, cache-bust on body drift, translate
skip). Existing summary-cache-key assertions updated v5→v6.
Covers R3, R4.
* feat(consumers): surface RSS snippet across dashboard, email, relay, MCP + audit (U7)
Thread the RSS description from the ingestion path (U1-U5) into every
user-facing LLM-adjacent surface. Audit the notification producers so
RSS-origin and domain-origin events stay on distinct contracts.
Dashboard (proto snippet → client → panel):
- src/types/index.ts NewsItem.snippet?:string (client-side field).
- src/app/data-loader.ts proto→client mapper propagates p.snippet.
- src/components/NewsPanel.ts renders snippet as a truncated (~200 chars,
word-boundary ellipsis) `.item-snippet` line under each headline.
- NewsPanel.currentBodies tracks per-headline bodies paired 1:1 with
currentHeadlines; passed as options.bodies to generateSummary so the
server-side SummarizeArticle LLM grounds on the article body.
Summary plumbing:
- src/services/summarization.ts threads bodies through SummarizeOptions
→ generateSummary → runApiChain → tryApiProvider; cache key now includes
bodies (via U6's buildSummaryCacheKey signature).
MCP world-brief:
- api/mcp.ts pairs headlines with their RSS snippets and POSTs `bodies`
to /api/news/v1/summarize-article so the MCP tool surface is no longer
starved.
Email digest:
- scripts/seed-digest-notifications.mjs plain-text formatDigest appends
a ~200-char truncated snippet line under each story; HTML formatDigestHtml
renders a dim-grey description div between title and meta. Both gated
on non-empty description (R6 — empty → today's behavior).
Real-time alerts:
- src/services/breaking-news-alerts.ts BreakingAlert gains optional
description; checkBatchForBreakingAlerts reads item.snippet; dispatchAlert
includes `description` in the /api/notify payload when present.
Notification relay:
- scripts/notification-relay.cjs formatMessage gated on
NOTIFY_RELAY_INCLUDE_SNIPPET=1 (default off). When on, RSS-origin
payloads render a `> <snippet>` context line under the title. When off
or payload.description absent, output is byte-identical to pre-U7.
Audit (RSS vs domain):
- tests/notification-relay-payload-audit.test.mjs enforces file-level
@notification-source tags on every producer, rejects `description:` in
domain-origin payload blocks, and verifies the relay codepath gates
snippet rendering under the flag.
- Tag added to ais-relay.cjs (domain), seed-aviation.mjs (domain),
alert-emitter.mjs (domain), breaking-news-alerts.ts (rss).
Deferred (plan explicitly flags): InsightsPanel + cluster-producer
plumbing (bodies default to [] — will unlock gradually once news:insights:v1
producer also carries primarySnippet).
Covers R5, R6.
* docs+test: grounding-path note + bump pinned CACHE_VERSION v5→v6 (U8)
Final verification for the RSS-description-end-to-end fix:
- docs/architecture.mdx — one-paragraph "News Grounding Pipeline"
subsection tracing parser → story:track:v1.description → NewsItem.snippet
→ brief / SummarizeArticle / dashboard / email / relay / MCP, with the
empty-description R6 fallback rule called out explicitly.
- tests/summarize-reasoning.test.mjs — Fix-4 static-analysis pin updated
to match the v6 bump from U6. Without this the summary cache bump silently
regressed CI's pinned-version assertion.
Final sweep (2026-04-24):
- grep -rn 'brief:llm:description:v1' → only in the U5 legacy-row test
simulation (by design: proves the v2 bump forces cold-start).
- grep -rn 'brief:llm:whymatters:v2/v6/shadow:v4' → no live references.
- grep -rn 'summary:v5' → no references.
- CACHE_VERSION = 'v6' in src/utils/summary-cache-key.ts.
- Full tsx --test sweep across all tests/*.test.{mjs,mts}: 6747/6747 pass.
- npm run typecheck + typecheck:api: both clean.
Covers R4, R6, R7.
* fix(rss-description): address /ce:review findings before merge
14 fixes from structured code review across 13 reviewer personas.
Correctness-critical (P1 — fixes that prevent R6/U7 contract violations):
- NewsPanel signature covers currentBodies so view-mode toggles that leave
headlines identical but bodies different now invalidate in-flight summaries.
Without this, switching renderItems → renderClusters mid-summary let a
grounded response arrive under a stale (now-orphaned) cache key.
- summarize-article.ts re-pairs bodies with headlines BEFORE dedup via a
single zip-sanitize-filter-dedup pass. Previously bodies[] was indexed by
position in light-sanitized headlines while findIndex looked up the
full-sanitized array — any headline that sanitizeHeadlines emptied
mispaired every subsequent body, grounding the LLM on the wrong story.
- Client skips the pre-chain cache lookup when bodies are present, since
client builds keys from RAW bodies while server sanitizes first. The
keys diverge on injection content, which would silently miss the
server's authoritative cache every call.
Test + audit hardening:
- Legacy v1 eviction test now uses the real hashBriefStory(story()) suffix
instead of a literal "somehash", so a bug where the reader still queried
the v1 prefix at the real key would actually be caught.
- tests/summary-cache-key.test.mts adds 400-char clip identity coverage so
the canonicalizer's clip and any downstream clip can't silently drift.
- tests/news-rss-description-extract.test.mts renames the well-formed
CDATA test and adds a new test documenting the malformed-]]> fallback
behavior (plain regex captures, article content survives).
Safe_auto cleanups:
- Deleted dead SNIPPET_PUSH_MAX constant in notification-relay.cjs.
- BETA-mode groq warm call now passes bodies, warming the right cache slot.
- seed-digest shares a local normalize-equality helper for description !=
headline comparison, matching the parser's contract.
- Pair-wise sort in summary-cache-key tie-breaks on body so duplicate
headlines produce stable order across runs.
- buildSummaryCacheKey gained JSDoc documenting the client/server contract
and the bodies parameter semantics.
- MCP get_world_brief tool description now mentions RSS article-body
grounding so calling agents see the current contract.
- _shared.ts `opts.bodies![i]!` double-bang replaced with `?? ''`.
- extractRawTagBody regexes cached in module-level Map, mirroring the
existing TAG_REGEX_CACHE pattern.
Deferred to follow-up (tracked for PR description / separate issue):
- Promote shared MAX_BODY constant across the 5 clip sites
- Promote shared truncateForDisplay helper across 4 render sites
- Collapse NewsPanel.{currentHeadlines, currentBodies} → Array<{title, snippet}>
- Promote sanitizeStoryForPrompt to shared/brief-llm-core.js
- Split list-feed-digest.ts parser helpers into sibling -utils.ts
- Strengthen audit test: forward-sweep + behavioral gate test
Tests: 6749/6749 pass. Typecheck clean on both configs.
* fix(summarization): thread bodies through browser T5 path (Codex #2)
Addresses the second of two Codex-raised findings on PR #3370:
The PR threaded bodies through the server-side API provider chain
(Ollama → Groq → OpenRouter → /api/news/v1/summarize-article) but the
local browser T5 path at tryBrowserT5 was still summarising from
headlines alone. In BETA_MODE that ungrounded path runs BEFORE the
grounded server providers; in normal mode it remains the last
fallback. Whenever T5-small won, the dashboard summary surface
regressed to the headline-only path — the exact hallucination class
this PR exists to eliminate.
Fix: tryBrowserT5 accepts an optional `bodies` parameter and
interleaves each body with its paired headline via a `headline —
body` separator in the combined text (clipped to 200 chars per body
to stay within T5-small's ~512-token context window). All three call
sites (BETA warm, BETA cold, normal-mode fallback) now pass the
bodies threaded down from generateSummary options.bodies.
When bodies is empty/omitted, the combined text is byte-identical to
pre-fix (R6 preserved).
On Codex finding #1 (story:track:v1 additive-only HSET keeps a body
from an earlier mention of the same normalized title), declining to
change. The current rule — "if this mention has a body, overwrite;
otherwise leave the prior body alone" — is defensible: a body from
mention A is not falsified by mention B being body-less (a wire
reprint doesn't invalidate the original source's body). A feed that
publishes a corrected headline creates a new normalized-title hash,
so no stale body carries forward. The failure window is narrow (live
story evolving while keeping the same title through hours of
body-less wire reprints) and the 7-day STORY_TTL is the backstop.
Opening a follow-up issue to revisit semantics if real-world evidence
surfaces a stale-grounding case.
* fix(story-track): description always-written to overwrite stale bodies (Codex #1)
Revisiting Codex finding #1 on PR #3370 after re-review. The previous
response declined the fix with reasoning; on reflection the argument
was over-defending the current behavior.
Problem: buildStoryTrackHsetFields previously wrote `description` only
when non-empty. Because story:track:v1 rows are collapsed by
normalized-title hash, an earlier mention's body would persist for up
to STORY_TTL (7 days) on subsequent body-less mentions of the same
story. Consumers reading `track.description` via HGETALL could not
distinguish "this mention's body" from "some mention's body from the
last week," silently grounding brief / whyMatters / SummarizeArticle
LLMs on text the current mention never supplied. That violates the
grounding contract advertised to every downstream surface in this PR.
Fix: HSET `description` unconditionally on every mention — empty
string when the current item has no body, real body when it does. An
empty value overwrites any prior mention's body so the row is always
authoritative for the current cycle. Consumers continue to treat
empty description as "fall back to cleaned headline" (R6 preserved).
The 7-day STORY_TTL and normalized-title hash semantics are unchanged.
Trade-off accepted: a valid body from Feed A (NYT) is wiped when Feed
B (AP body-less wire reprint) arrives for the same normalized title,
even though Feed A's body is factually correct. Rationale: the
alternative — keeping Feed A's body indefinitely — means the user
sees Feed A's body attributed (by proximity) to an AP mention at a
later timestamp, which is at minimum misleading and at worst carries
retracted/corrected details. Honest absence beats unlabeled presence.
Tests: new stale-body overwrite sequence test (T0 body → T1 empty →
T2 new body), existing "writes description when non-empty" preserved,
existing "omits when empty" inverted to "writes empty, overwriting."
cache-keys.ts contract comment updated to mark description as
always-written rather than optional.
|
||
|
Sebastien Melki
|
fcbb8bc0a1 |
feat(proto): unified OpenAPI bundle via sebuf v0.11.0 (#3341)
* feat(proto): generate unified worldmonitor.openapi.yaml bundle Adds a third protoc-gen-openapiv3 invocation that merges every service into a single docs/api/worldmonitor.openapi.yaml spanning all 68 RPCs, using the new bundle support shipped in sebuf 0.11.0 (SebastienMelki/sebuf#158). Per-service YAML/JSON files are untouched and continue to back the Mintlify docs in docs/docs.json. The bundle runs with strategy: all and bundle_only=true so only the aggregate file is emitted, avoiding duplicate-output conflicts with the existing per-service generator. Requires protoc-gen-openapiv3 >= v0.11.0 locally: go install github.com/SebastienMelki/sebuf/cmd/protoc-gen-openapiv3@v0.11.0 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(proto): bump sebuf to v0.11.0 and document unified OpenAPI bundle - Makefile: SEBUF_VERSION v0.7.0 → v0.11.0 (required for bundle support). - proto/buf.gen.yaml: point bundle_server at https://api.worldmonitor.app. - CONTRIBUTING.md: new "OpenAPI Output" section covering per-service specs vs the unified worldmonitor.openapi.yaml bundle, plus a note that all three sebuf plugins must be installed from the pinned version. - AGENTS.md: clarify that `make generate` also produces the unified spec and requires sebuf v0.11.0. - CHANGELOG.md: Unreleased entry announcing the bundle and version bump. Also regenerates the bundle with the updated server URL. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(codegen): regenerate TS client/server with sebuf v0.11.0 Mechanical output of the bumped protoc-gen-ts-client and protoc-gen-ts-server. Two behavioral improvements roll in from sebuf: - Proto enum fields now use the proper `*_UNSPECIFIED` sentinel in default-value checks instead of the empty string, so generated query-string serializers correctly omit enum params only when they actually equal the proto default. - `repeated string` query params now serialize via `forEach(v => params.append(...))` instead of being coerced through `String(req.field)`, matching the existing `parseStringArray()` contract on the server side. All files also drop the `// @ts-nocheck` header that earlier sebuf versions emitted — 0.11.0 output type-checks cleanly under our tsconfig. No hand edits. Reproduce with `make install-plugins && make generate`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(proto): bump sebuf v0.11.0 → v0.11.1, realign tests with repeated-param wire format - Bump SEBUF_VERSION to v0.11.1, pulling in the OpenAPI fix for repeated scalar query params (SebastienMelki/sebuf#161). `repeated string` fields now emit `type: array` + `items.type: string` + `style: form` + `explode: true` instead of `type: string`, so SDK generators consuming the unified bundle produce correct array clients. - Regenerate all 12 OpenAPI specs (unified bundle + Aviation, Economic, Infrastructure, Market, Trade per-service). TS client/server codegen is byte-identical to v0.11.0 — only the OpenAPI emitter was out of sync. - Update three tests that asserted the pre-v0.11 comma-joined wire format (`symbols=AAPL,MSFT`) to match the current repeated-param form (`symbols=AAPL&symbols=MSFT`) produced by `params.append(...)`: - tests/market-service-symbol-casing.test.mjs (2 cases: getAll) - tests/stock-analysis-history.test.mts - tests/stock-backtest.test.mts Locally: test:data 6619/6619 pass, typecheck clean, lint exit 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Apply suggestions from code review Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Elie Habib
|
7f83e1e0c3 |
chore: remove dormant proactive-intelligence agent (superseded by digest) (#3325)
* chore: remove dormant proactive-intelligence agent (superseded by digest) PR #2889 merged a Phase 4 "Proactive Intelligence Agent" in 2026-04 with 588 lines of code and a PR body explicitly requiring a 6h Railway cron service. That service was never provisioned — no Dockerfile, no Railway entry, no health-registry key, all 7 test-plan checkboxes unchecked. In the meantime the daily Intelligence Brief shipped via scripts/seed-digest-notifications.mjs (PR #3321 and earlier), covering the same "personalized editorial brief across all channels" use-case at a different cadence (30m rather than 6h). The proactive agent's landscape-diff trigger was speculative; the digest is the shipped equivalent. This PR retires the dormant code and scrubs the aspirational "post-launch classifier" references that docs + comments have been quietly carrying: - Deleted scripts/proactive-intelligence.mjs (588 lines). - scripts/_energy-disruption-registry.mjs, scripts/seed-fuel-shortages.mjs, scripts/_fuel-shortage-registry.mjs, src/shared/shortage-evidence.ts: dropped "proactive-intelligence.mjs will extend this registry / classifier output" comments. Registries are curated-only; no classifier exists. - docs/methodology/disruptions.mdx: replaced "post-launch classifier" prose with the accurate "curated-only" description of how the event log is maintained. - docs/api-notifications.mdx: envelope version is shared across **two** producers now (notification-relay, seed-digest-notifications), not three. - scripts/notification-relay.cjs: one cross-producer comment updated. - proto/worldmonitor/supply_chain/v1/list_energy_disruptions.proto + list_fuel_shortages.proto: same aspirational wording scrubbed. - docs/api/SupplyChainService.openapi.{yaml,json} auto-regenerated via `make generate` — text-only description updates, no schema changes. Net: -626 lines, +36 lines. No runtime behavior change. 6573/6573 unit tests pass locally. * fix(proto): scrub stale ListFuelShortages RPC comment (PR #3325 review) Reviewer caught a stale "classifier-extended post-launch" comment on the ListFuelShortages RPC method in service.proto that this PR's initial pass missed — I fixed the message-definition comment in list_fuel_shortages.proto but not the RPC-method comment in service.proto, which propagates into the published OpenAPI operation description. - proto/worldmonitor/supply_chain/v1/service.proto: rewrite the ListFuelShortages RPC comment to match the curated-only framing used elsewhere in this PR. - docs/api/SupplyChainService.openapi.{yaml,json}: auto-regenerated via `make generate`. Text-only operation-description update; no schema / contract changes. No runtime impact. Other `classifier` references remaining in the OpenAPI are legitimate schema field names (classifierVersion, classifierConfidence) and an unrelated auto-revision-log trigger enum value, both of which describe real on-row fields that existed before this cleanup. |
||
|
Elie Habib
|
84ee2beb3e |
feat(energy): Energy Atlas end-to-end — pipelines + storage + shortages + disruptions + country drill-down (#3294)
* feat(energy): pipeline registries (gas + oil) — evidence-based schema
Day 6 of the Energy Atlas Release 1 plan (Week 2). First curated asset
registry for the atlas — the real gap vs GEF.
## Curated data (critical assets only, not global completeness)
scripts/data/pipelines-gas.json — 12 critical gas lines:
Nord Stream 1/2 (offline; Swedish EEZ sabotage 2022; EU sanctions refs),
TurkStream, Yamal–Europe (offline; Polish counter-sanctions),
Brotherhood/Soyuz (offline; Ukraine transit expired 2024-12-31),
Power of Siberia, Dolphin, Medgaz, TAP, TANAP,
Central Asia–China, Langeled.
scripts/data/pipelines-oil.json — 12 critical oil lines:
Druzhba North/South (N offline per EU 2022/879; S under landlocked
derogation), CPC, ESPO (+ price-cap sanction ref), BTC, TAPS,
Habshan–Fujairah (Hormuz bypass), Keystone, Kirkuk–Ceyhan (offline
since 2023 ICC ruling), Baku–Supsa, Trans-Mountain (TMX expansion
May 2024), ESPO spur to Daqing.
Scope note: 75+ each is Week 2b work via GEM bulk import. Today's cut
is curated from first-hand operator disclosures + regulator filings so
I can stand behind every evidence field.
## Evidence-based schema (not conclusion labels)
Per docs/methodology/pipelines.mdx: no bare `sanctions_blocked` field.
Every pipeline carries an evidence bundle with `physicalState`,
`physicalStateSource`, `operatorStatement`, `commercialState`,
`sanctionRefs[]`, `lastEvidenceUpdate`, `classifierVersion`,
`classifierConfidence`. The public badge (`flowing|reduced|offline|
disputed`) is derived server-side from this bundle at read time.
## Seeder
scripts/seed-pipelines.mjs — single process publishes BOTH keys
(energy:pipelines:{gas,oil}:v1) via two runSeed() calls. Tiny datasets
(<20KB each) so co-location is cheap and guarantees classifierVersion
consistency.
Conventions followed (worldmonitor-bootstrap-registration skill):
- TTL 21d = 3× weekly cadence (gold-standard per
feedback_seeder_gold_standard.md)
- maxStaleMin 20_160 = 2× cadence (health-maxstalemin-write-cadence skill)
- sourceVersion + schemaVersion + recordCount + declareRecords wired
(seed-contract-foundation)
- Zero-case explicitly NOT allowed — MIN_PIPELINES_PER_REGISTRY=8 floor
## Health registration (dual, per feedback_two_health_endpoints_must_match)
- api/health.js: BOOTSTRAP_KEYS adds pipelinesGas + pipelinesOil;
SEED_META adds both with maxStaleMin=20_160.
- api/seed-health.js: mirror entries with intervalMin=10_080 (maxStaleMin/2).
## Bundle registration
scripts/seed-bundle-energy-sources.mjs adds a single Pipelines entry
(not two) because seed-pipelines.mjs publishes both keys in one run —
listing oil separately would double-execute. Monitoring of the oil key
staleness happens in api/health.js instead.
## Tests (tests/pipelines-registry.test.mts)
17 passing node:test assertions covering:
- Schema validation (both registries pass validateRegistry)
- Identity resolution (no id collisions, id matches object key)
- Country ISO2 normalization (from/to/transit all match /^[A-Z]{2}$/)
- Endpoint geometry within Earth bounds
- Evidence rigor: non-flowing badges require at least one supporting
evidence source (operator statement / sanctionRefs / ais-relay /
satellite / press)
- ClassifierConfidence in 0..1
- Commodity/capacity pairing (gas uses capacityBcmYr, oil uses
capacityMbd — mixing = test fail)
- validateRegistry rejects: empty object, null, no-evidence fixtures,
below-floor counts
Typecheck clean (both tsconfig.json and tsconfig.api.json).
Next: Day 7 will add list-pipelines / get-pipeline-detail RPCs in
supply-chain/v1. Day 8 ships PipelineStatusPanel with DeckGL PathLayer
consuming the registry.
* fix(energy): split seed-pipelines.mjs into two entry points — runSeed hard-exits
High finding from PR review. scripts/seed-pipelines.mjs called runSeed()
twice in one process and awaited Promise.all. But runSeed() in
scripts/_seed-utils.mjs hard-exits via process.exit on ~9 terminal paths
(lines 816, 820, 839, 888, 917, 989, plus fetch-retry 946, fatal 859,
skipped-lock 81). The first runSeed to reach any terminal path exits the
entire node process, so the second runSeed's resolve never fires — only
one of energy:pipelines:{gas,oil}:v1 would ever be written.
Since the bundle scheduled seed-pipelines.mjs exactly once, and both
api/health.js and api/seed-health.js expect both keys populated, the
other registry would stay permanently EMPTY/STALE after deploy.
Fix: split into two entry-point scripts around a shared utility.
- scripts/_pipeline-registry.mjs (NEW, was seed-pipelines.mjs) — shared
helpers ONLY. Exports GAS_CANONICAL_KEY, OIL_CANONICAL_KEY,
PIPELINES_TTL_SECONDS, MAX_STALE_MIN, buildGasPayload, buildOilPayload,
validateRegistry, recordCount, declareRecords. Underscore prefix marks
it as non-entry-point (matches _seed-utils.mjs / _seed-envelope-source.mjs
convention).
- scripts/seed-pipelines-gas.mjs (NEW) — imports from the shared module,
single runSeed('energy','pipelines-gas',…) call.
- scripts/seed-pipelines-oil.mjs (NEW) — same shape, oil.
- scripts/seed-bundle-energy-sources.mjs — register BOTH seeders (not one).
- scripts/seed-pipelines.mjs — deleted.
- tests/pipelines-registry.test.mts — update import path to the shared
module. All 17 tests still pass.
Typecheck clean (both configs). Tests pass. No other consumers import
from the deleted script.
* fix(energy): complete pipeline bootstrap registration per 4-file checklist
High finding from PR review. My earlier PR description claimed
worldmonitor-bootstrap-registration was complete, but I only touched two
of the four registries (api/health.js + api/seed-health.js). The bootstrap
hydration payload itself (api/bootstrap.js) and the shared cache-keys
registry (server/_shared/cache-keys.ts) still had no entry for either
pipeline key, so any consumer that reads bootstrap data would see
pipelinesGas/pipelinesOil as missing on first load.
Files updated this commit:
- api/bootstrap.js — KEYS map + SLOW_KEYS set both gain pipelinesGas +
pipelinesOil. Placed next to sprPolicies (same curated-registry cadence
and tier). Slow tier is correct: weekly cron, not needed on first paint.
- server/_shared/cache-keys.ts — PIPELINES_GAS_KEY + PIPELINES_OIL_KEY
exported constants (matches SPR_POLICIES_KEY pattern), BOOTSTRAP_KEYS map
entries, and BOOTSTRAP_TIERS entries (both 'slow').
Not touched (intentional):
- server/gateway.ts — pipeline data is free-tier per the Energy Atlas
plan; no PREMIUM_RPC_PATHS entry required. Energy Atlas monetization
hooks (scenario runner, MCP tools, subscriptions) are Release 2.
Full 4-file checklist now complete:
✅ server/_shared/cache-keys.ts (this commit)
✅ api/bootstrap.js (this commit)
✅ api/health.js (earlier in PR)
✅ api/seed-health.js (earlier in PR — dual-registry rule)
Typecheck clean (both configs).
* feat(energy): ListPipelines + GetPipelineDetail RPCs with evidence-derived badges
Day 7 of the Energy Atlas Release 1 plan (Week 2). Exposes the pipeline
registries (shipped in Day 6) via two supply-chain RPCs and ships the
evidence-to-badge derivation server-side.
## Proto
proto/worldmonitor/supply_chain/v1/list_pipelines.proto — new:
- ListPipelinesRequest { commodity_type?: 'gas' | 'oil' }
- ListPipelinesResponse { pipelines[], fetched_at, classifier_version, upstream_unavailable }
- GetPipelineDetailRequest { pipeline_id (required, query-param) }
- GetPipelineDetailResponse { pipeline?, revisions[], fetched_at, unavailable }
- PipelineEntry — wire shape mirroring scripts/data/pipelines-{gas,oil}.json
+ a server-derived public_badge field
- PipelineEvidence, OperatorStatement, SanctionRef, LatLon, PipelineRevisionEntry
service.proto adds both rpc methods with HTTP_METHOD_GET + path bindings:
/api/supply-chain/v1/list-pipelines
/api/supply-chain/v1/get-pipeline-detail
`make generate` regenerated src/generated/{client,server}/… + docs/api/
OpenAPI json/yaml.
## Evidence-derivation
server/worldmonitor/supply-chain/v1/_pipeline-evidence.ts — new.
derivePublicBadge(evidence) → 'flowing' | 'reduced' | 'offline' | 'disputed'
is deterministic + versioned (DERIVER_VERSION='badge-deriver-v1').
Rules (first match wins):
1. offline + sanctionRef OR expired/suspended commercial → offline
2. offline + operator statement → offline
3. offline + only press/ais/satellite → disputed (single-source negative claim)
4. reduced → reduced
5. flowing → flowing
6. unknown / malformed → disputed
Staleness guard: non-flowing badges on >14d-old evidence demote to
disputed. Flowing is the optimistic default — stale "still flowing" is
safer than stale "offline". Matches seed-pipelines-{gas,oil}.mjs maxStaleMin.
Tests (tests/pipeline-evidence-derivation.test.mts) — 15 passing cases
covering happy paths, disputed fallbacks, staleness guard, versioning.
## Handlers
server/worldmonitor/supply-chain/v1/list-pipelines.ts
- Reads energy:pipelines:{gas,oil}:v1 via getCachedJson.
- projectPipeline() narrows the Upstash `unknown` into PipelineEntry
shape + calls derivePublicBadge.
- Honors commodity_type filter (skip the opposite registry's Redis read
when the client pre-filters).
- Returns upstream_unavailable=true when BOTH registries miss.
server/worldmonitor/supply-chain/v1/get-pipeline-detail.ts
- Scans both registries by id (ids are globally unique per
tests/pipelines-registry.test.mts).
- Empty revisions[] for now; auto-revision log wires up in Week 3.
handler.ts registers both into supplyChainHandler.
## Gateway
server/gateway.ts adds 'static' cache-tier for both new RPC paths
(registry is slow-moving; 'static' matches the other read-mostly
supply-chain endpoints).
## Consumer wiring
Not in this commit — PipelineStatusPanel (Day 8) is what will call
listPipelines/getPipelineDetail via the generated client. pipelinesGas
+ pipelinesOil stay in PENDING_CONSUMERS until Day 8.
Typecheck clean (both configs). 15 new tests + 17 registry tests all pass.
* feat(energy): PipelineStatusPanel — evidence-backed status table + drawer
Day 8 of the Energy Atlas Release 1 plan. First consumer of the Day 6–7
registries + RPCs.
## What this PR adds
- src/components/PipelineStatusPanel.ts — new panel (id=pipeline-status).
* Bootstrap-hydrates from pipelinesGas + pipelinesOil for instant first
paint; falls through to listPipelines() RPC if bootstrap misses.
Background re-fetch runs on every render so a classifier-version bump
between bootstrap stamp and first view produces a visible update.
* Table rows sorted non-flowing-first (offline / reduced / disputed
before flowing) — what an atlas reader cares about.
* Click-to-expand drawer calls getPipelineDetail() lazily — operator
statements, sanction refs (with clickable source URLs), commercial
state, classifier version + confidence %, capacity + route metadata.
* publicBadge color-chip palette matches the methodology doc.
* Attribution footer with GEM (CC-BY 4.0) credit + classifier version.
- src/components/index.ts — barrel export.
- src/app/panel-layout.ts — import + createPanel('pipeline-status', …).
- src/config/panels.ts — ENERGY_PANELS adds 'pipeline-status' at priority 1.
## PENDING_CONSUMERS cleanup
tests/bootstrap.test.mjs — removes 'pipelinesGas' + 'pipelinesOil' from
the allowlist. The invariant "every bootstrap key has a getHydratedData
consumer" now enforces real wiring for these keys: the panel literally
calls getHydratedData('pipelinesGas') and getHydratedData('pipelinesOil').
Future regressions that remove the consumer will fail pre-push.
## Consumer contract verified
- 67 tests pass including bootstrap.test.mjs consumer coverage check.
- Typecheck clean.
- No DeckGL PathLayer in this commit — existing 'pipelines-layer' has a
separate data source, so modifying DeckGLMap.ts to overlay evidence-
derived badges on the map is a follow-up commit to avoid clobbering.
## Out of scope for Day 8 (next steps on same PR)
- DeckGL PathLayer integration (color pipelines on the main map by
publicBadge, click-to-open this drawer) — Day 8b commit.
- Storage facility registry + StorageFacilityMapPanel — Days 9-10.
* fix(energy): PipelineStatusPanel bootstrap path — client-side badge derivation
High finding from PR review. The Day-8 panel crashed on first paint
whenever bootstrap hydration succeeded, because:
- Bootstrap hydrates raw scripts/data/pipelines-{gas,oil}.json verbatim.
- That JSON does NOT include publicBadge — that field is only added by
the server handler's projectPipeline() in list-pipelines.ts.
- PipelineStatusPanel passed raw entries into badgeChip(), which called
badgeLabel(undefined).charAt(0) → TypeError.
The background RPC refresh that would have repaired the data never ran
because the panel threw before reaching it. So the exact bootstrap path
newly wired in commit
|
||
|
Sebastien Melki
|
58e42aadf9 |
chore(api): enforce sebuf contract + migrate drifting endpoints (#3207) (#3242)
* chore(api): enforce sebuf contract via exceptions manifest (#3207) Adds api/api-route-exceptions.json as the single source of truth for non-proto /api/ endpoints, with scripts/enforce-sebuf-api-contract.mjs gating every PR via npm run lint:api-contract. Fixes the root-only blind spot in the prior allowlist (tests/edge-functions.test.mjs), which only scanned top-level *.js files and missed nested paths and .ts endpoints — the gap that let api/supply-chain/v1/country-products.ts and friends drift under proto domain URL prefixes unchallenged. Checks both directions: every api/<domain>/v<N>/[rpc].ts must pair with a generated service_server.ts (so a deleted proto fails CI), and every generated service must have an HTTP gateway (no orphaned generated code). Manifest entries require category + reason + owner, with removal_issue mandatory for temporary categories (deferred, migration-pending) and forbidden for permanent ones. .github/CODEOWNERS pins the manifest to @SebastienMelki so new exceptions don't slip through review. The manifest only shrinks: migration-pending entries (19 today) will be removed as subsequent commits in this PR land each migration. * refactor(maritime): migrate /api/ais-snapshot → maritime/v1.GetVesselSnapshot (#3207) The proto VesselSnapshot was carrying density + disruptions but the frontend also needed sequence, relay status, and candidate_reports to drive the position-callback system. Those only lived on the raw relay passthrough, so the client had to keep hitting /api/ais-snapshot whenever callbacks were registered and fall back to the proto RPC only when the relay URL was gone. This commit pushes all three missing fields through the proto contract and collapses the dual-fetch-path into one proto client call. Proto changes (proto/worldmonitor/maritime/v1/): - VesselSnapshot gains sequence, status, candidate_reports. - GetVesselSnapshotRequest gains include_candidates (query: include_candidates). Handler (server/worldmonitor/maritime/v1/get-vessel-snapshot.ts): - Forwards include_candidates to ?candidates=... on the relay. - Separate 5-min in-memory caches for the candidates=on and candidates=off variants; they have very different payload sizes and should not share a slot. - Per-request in-flight dedup preserved per-variant. Frontend (src/services/maritime/index.ts): - fetchSnapshotPayload now calls MaritimeServiceClient.getVesselSnapshot directly with includeCandidates threaded through. The raw-relay path, SNAPSHOT_PROXY_URL, DIRECT_RAILWAY_SNAPSHOT_URL and LOCAL_SNAPSHOT_FALLBACK are gone — production already routed via Vercel, the "direct" branch only ever fired on localhost, and the proto gateway covers both. - New toLegacyCandidateReport helper mirrors toDensityZone/toDisruptionEvent. api/ais-snapshot.js deleted; manifest entry removed. Only reduced the codegen scope to worldmonitor.maritime.v1 (buf generate --path) — regenerating the full tree drops // @ts-nocheck from every client/server file and surfaces pre-existing type errors across 30+ unrelated services, which is not in scope for this PR. Shape-diff vs legacy payload: - disruptions / density: proto carries the same fields, just with the GeoCoordinates wrapper and enum strings (remapped client-side via existing toDisruptionEvent / toDensityZone helpers). - sequence, status.{connected,vessels,messages}: now populated from the proto response — was hardcoded to 0/false in the prior proto fallback. - candidateReports: same shape; optional numeric fields come through as 0 instead of undefined, which the legacy consumer already handled. * refactor(sanctions): migrate /api/sanctions-entity-search → LookupSanctionEntity (#3207) The proto docstring already claimed "OFAC + OpenSanctions" coverage but the handler only fuzzy-matched a local OFAC Redis index — narrower than the legacy /api/sanctions-entity-search, which proxied OpenSanctions live (the source advertised in docs/api-proxies.mdx). Deleting the legacy without expanding the handler would have been a silent coverage regression for external consumers. Handler changes (server/worldmonitor/sanctions/v1/lookup-entity.ts): - Primary path: live search against api.opensanctions.org/search/default with an 8s timeout and the same User-Agent the legacy edge fn used. - Fallback path: the existing OFAC local fuzzy match, kept intact for when OpenSanctions is unreachable / rate-limiting. - Response source field flips between 'opensanctions' (happy path) and 'ofac' (fallback) so clients can tell which index answered. - Query validation tightened: rejects q > 200 chars (matches legacy cap). Rate limiting: - Added /api/sanctions/v1/lookup-entity to ENDPOINT_RATE_POLICIES at 30/min per IP — matches the legacy createIpRateLimiter budget. The gateway already enforces per-endpoint policies via checkEndpointRateLimit. Docs: - docs/api-proxies.mdx — dropped the /api/sanctions-entity-search row (plus the orphaned /api/ais-snapshot row left over from the previous commit in this PR). - docs/panels/sanctions-pressure.mdx — points at the new RPC URL and describes the OpenSanctions-primary / OFAC-fallback semantics. api/sanctions-entity-search.js deleted; manifest entry removed. * refactor(military): migrate /api/military-flights → ListMilitaryFlights (#3207) Legacy /api/military-flights read a pre-baked Redis blob written by the seed-military-flights cron and returned flights in a flat app-friendly shape (lat/lon, lowercase enums, lastSeenMs). The proto RPC takes a bbox, fetches OpenSky live, classifies server-side, and returns nested GeoCoordinates + MILITARY_*_TYPE_* enum strings + lastSeenAt — same data, different contract. fetchFromRedis in src/services/military-flights.ts was doing nothing sebuf-aware. Renamed it to fetchViaProto and rewrote to: - Instantiate MilitaryServiceClient against getRpcBaseUrl(). - Iterate MILITARY_QUERY_REGIONS (PACIFIC + WESTERN) in parallel — same regions the desktop OpenSky path and the seed cron already use, so dashboard coverage tracks the analytic pipeline. - Dedup by hexCode across regions. - Map proto → app shape via new mapProtoFlight helper plus three reverse enum maps (AIRCRAFT_TYPE_REVERSE, OPERATOR_REVERSE, CONFIDENCE_REVERSE). The seed cron (scripts/seed-military-flights.mjs) stays put: it feeds regional-snapshot mobility, cross-source signals, correlation, and the health freshness check (api/health.js: 'military:flights:v1'). None of those read the legacy HTTP endpoint; they read the Redis key directly. The proto handler uses its own per-bbox cache keys under the same prefix, so dashboard traffic no longer races the seed cron's blob — the two paths diverge by a small refresh lag, which is acceptable. Docs: dropped the /api/military-flights row from docs/api-proxies.mdx. api/military-flights.js deleted; manifest entry removed. Shape-diff vs legacy: - f.location.{latitude,longitude} → f.lat, f.lon - f.aircraftType: MILITARY_AIRCRAFT_TYPE_TANKER → 'tanker' via reverse map - f.operator: MILITARY_OPERATOR_USAF → 'usaf' via reverse map - f.confidence: MILITARY_CONFIDENCE_LOW → 'low' via reverse map - f.lastSeenAt (number) → f.lastSeen (Date) - f.enrichment → f.enriched (with field renames) - Extra fields registration / aircraftModel / origin / destination / firstSeenAt now flow through where proto populates them. * fix(supply-chain): thread includeCandidates through chokepoint status (#3207) Caught by tsconfig.api.json typecheck in the pre-push hook (not covered by the plain tsc --noEmit run that ran before I pushed the ais-snapshot commit). The chokepoint status handler calls getVesselSnapshot internally with a static no-auth request — now required to include the new includeCandidates bool from the proto extension. Passing false: server-internal callers don't need per-vessel reports. * test(maritime): update getVesselSnapshot cache assertions (#3207) The ais-snapshot migration replaced the single cachedSnapshot/cacheTimestamp pair with a per-variant cache so candidates-on and candidates-off payloads don't evict each other. Pre-push hook surfaced that tests/server-handlers still asserted the old variable names. Rewriting the assertions to match the new shape while preserving the invariants they actually guard: - Freshness check against slot TTL. - Cache read before relay call. - Per-slot in-flight dedup. - Stale-serve on relay failure (result ?? slot.snapshot). * chore(proto): restore // @ts-nocheck on regenerated maritime files (#3207) I ran 'buf generate --path worldmonitor/maritime/v1' to scope the proto regen to the one service I was changing (to avoid the toolchain drift that drops @ts-nocheck from 60+ unrelated files — separate issue). But the repo convention is the 'make generate' target, which runs buf and then sed-prepends '// @ts-nocheck' to every generated .ts file. My scoped command skipped the sed step. The proto-check CI enforces the sed output, so the two maritime files need the directive restored. * refactor(enrichment): decomm /api/enrichment/{company,signals} legacy edge fns (#3207) Both endpoints were already ported to IntelligenceService: - getCompanyEnrichment (/api/intelligence/v1/get-company-enrichment) - listCompanySignals (/api/intelligence/v1/list-company-signals) No frontend callers of the legacy /api/enrichment/* paths exist. Removes: - api/enrichment/company.js, signals.js, _domain.js - api-route-exceptions.json migration-pending entries (58 remain) - docs/api-proxies.mdx rows for /api/enrichment/{company,signals} - docs/architecture.mdx reference updated to the IntelligenceService RPCs Verified: typecheck, typecheck:api, lint:api-contract (89 files / 58 entries), lint:boundaries, tests/edge-functions.test.mjs (136 pass), tests/enrichment-caching.test.mjs (14 pass — still guards the intelligence/v1 handlers), make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(leads): migrate /api/{contact,register-interest} → LeadsService (#3207) New leads/v1 sebuf service with two POST RPCs: - SubmitContact → /api/leads/v1/submit-contact - RegisterInterest → /api/leads/v1/register-interest Handler logic ported 1:1 from api/contact.js + api/register-interest.js: - Turnstile verification (desktop sources bypass, preserved) - Honeypot (website field) silently accepts without upstream calls - Free-email-domain gate on SubmitContact (422 ApiError) - validateEmail (disposable/offensive/typo-TLD/MX) on RegisterInterest - Convex writes via ConvexHttpClient (contactMessages:submit, registerInterest:register) - Resend notification + confirmation emails (HTML templates unchanged) Shared helpers moved to server/_shared/: - turnstile.ts (getClientIp + verifyTurnstile) - email-validation.ts (disposable/offensive/MX checks) Rate limits preserved via ENDPOINT_RATE_POLICIES: - submit-contact: 3/hour per IP (was in-memory 3/hr) - register-interest: 5/hour per IP (was in-memory 5/hr; desktop sources previously capped at 2/hr via shared in-memory map — now 5/hr like everyone else, accepting the small regression in exchange for Upstash-backed global limiting) Callers updated: - pro-test/src/App.tsx contact form → new submit-contact path - src-tauri/sidecar/local-api-server.mjs cloud-fallback rewrites /api/register-interest → /api/leads/v1/register-interest when proxying; keeps local path for older desktop builds - src/services/runtime.ts isKeyFreeApiTarget allows both old and new paths through the WORLDMONITOR_API_KEY-optional gate Tests: - tests/contact-handler.test.mjs rewritten to call submitContact handler directly; asserts on ValidationError / ApiError - tests/email-validation.test.mjs + tests/turnstile.test.mjs point at the new server/_shared/ modules Deleted: api/contact.js, api/register-interest.js, api/_ip-rate-limit.js, api/_turnstile.js, api/_email-validation.js, api/_turnstile.test.mjs. Manifest entries removed (58 → 56). Docs updated (api-platform, api-commerce, usage-rate-limits). Verified: npm run typecheck + typecheck:api + lint:api-contract (88 files / 56 entries) + lint:boundaries pass; full test:data (5852 tests) passes; make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(pro-test): rebuild bundle for leads/v1 contact form (#3207) Updates the enterprise contact form to POST to /api/leads/v1/submit-contact (old path /api/contact removed in the previous commit). Bundle is rebuilt from pro-test/src/App.tsx source change in |
||
|
Elie Habib
|
502bd4472c |
docs(resilience): sync methodology/proto/widget to 6-domain + 3-pillar reality (#3264)
Brings every user-facing surface into alignment with the live resilience scorer. Zero behavior change: overall_score is still the 6-domain weighted aggregate, schemaVersion is still 2.0 default, and every existing test continues to pass. Surfaces touched: - proto + OpenAPI: rewrote the ResiliencePillar + schema_version descriptions. 2.0 is correctly documented as default; shaped-but-empty language removed. - Widget: added missing recovery: 'Recovery' label (was rendering literal lowercase recovery before), retitled footer data-version chip from Data to Seed date so it is clear the value reflects the static seed bundle not every live input, rewrote help tooltip for 6 domains and 3 pillars and called out the 0.25 recovery weight. - Methodology doc: domains-and-weights table now carries all 6 rows with actual code weights (0.17/0.15/0.11/0.19/0.13/0.25), Recovery section header weight corrected from 1.0 to 0.25, new Pillar-combined score activation (pending) section with the measured Spearman 0.9935, top-5 movers, and the activation checklist. - documentation.mdx + features.mdx: product blurbs updated from 5 domains and 13 dimensions to 6 domains and 19 dimensions grouped into 3 pillars. - Tests: recovery-label regression pin, Seed date label pin, clarified pillar-schema degenerate-input semantics. New scaffolding for defensibility: - docs/snapshots/resilience-ranking-2026-04-21.json frozen published tables artifact with methodology metadata and commit SHA. - docs/snapshots/resilience-pillar-sensitivity-2026-04-21.json live Redis capture (52-country sample) combining sensitivity stability with the current-vs-proposed Spearman comparison. - scripts/freeze-resilience-ranking.mjs refresh script. - scripts/compare-resilience-current-vs-proposed.mjs comparison script. - tests/resilience-ranking-snapshot.test.mts 13 assertions auto discovered from any resilience-ranking-YYYY-MM-DD.json in snapshots. Verification: npm run typecheck:all clean, 390/390 resilience tests pass. Follow-up: pillar-combined score activation. The sensitivity artifact shows rank-preservation Spearman 0.9935 and no ceiling effects, which clears the methodological bar. Blocker is messaging because every country drops ~13 points under the penalty, so activation PR ships with re-anchored release-gate bands, refreshed frozen ranking, and a v2.0 methodology note. |
||
|
Elie Habib
|
96fca1dc2b |
fix(supply-chain): popup-keyed history re-query + dataAvailable flag (#3187)
* fix(supply-chain): popup-keyed history re-query + dataAvailable flag for partial coverage Two P1 findings on #3185 post-merge review: 1. MapPopup cross-chokepoint history contamination Popup's async history resolve re-queried [data-transit-chart] without a cpId key. User opens popup A → fetch starts for cpA; user opens popup B before it resolves → cpA's history mounts into cpB's chart container. Fix: add data-transit-chart-id keyed by cpId; re-query by it on resolve. Mirrors SupplyChainPanel's existing data-chart-cp-id pattern. 2. Partial portwatch coverage still looked healthy Previous fix emits all 13 canonical summaries (zero-state fill for missing IDs) and records pwCovered in seed-meta, but: - get-chokepoint-status still zero-filled missing chokepoints and cached the response as healthy — panel rendered silent empty rows. - api/health.js only degrades on recordCount=0, so 10/13 partial read as OK despite the UI hiding entire chokepoints. Fix: - proto: TransitSummary.data_available (field 12). Writer tags with Boolean(cpData). Status RPC passes through; defaults true for pre-fix payloads (absence = covered). - Status RPC writes seed-meta recordCount as covered count (not shape size), and flips response-level upstreamUnavailable on partial. - api/health.js: new minRecordCount field on SEED_META entries + new COVERAGE_PARTIAL status (warn rollup). chokepoints entry declares minRecordCount: 13. recordCount < 13 → COVERAGE_PARTIAL. - Client (panel + popup): skip stats/chart rendering when !dataAvailable; show "Transit data unavailable (upstream partial)" microcopy so users understand the gap. 5759/5759 data tests pass. Typecheck + typecheck:api clean. * fix(supply-chain): guarantee Simulate Closure button exits Computing state User reports "Simulate Closure does nothing beyond write Computing…" — the button sticks at Computing forever. Two causes: 1. Scenario worker appears down (0 scenario-result:* keys in Redis in the last 24h of 24h-TTL). Railway-side — separate intervention needed to redeploy scripts/scenario-worker.mjs. 2. Client leaked the "Computing…" state on multiple exit paths: - signal.aborted early-return inside the poll loop never reset the button. Second click fired abort on first → first returned without resetting → button stayed "Computing…" until next render. - !this.content.isConnected early-return also skipped reset (less user-visible but same class of bug). - catch block swallowed AbortError without resetting. - POST /run had no hard timeout — a hanging edge function left the button in Computing indefinitely. Fix: - resetButton(text) helper touches the btn only if still connected; applied in every exit path (abort, timeout, post-success, catch). - AbortSignal.any([caller, AbortSignal.timeout(20_000)]) on POST /run. - console.error on failure so Simulate Closure errors surface in ops. - Error message includes "scenario worker may be down" on loop timeout so operators see the right suspect. Backend observations (for follow-up): - Hormuz backend is healthy (/api/health chokepoints OK, 13 records, 1 min old; live RPC has hormuz_strait.riskLevel=critical, wow=-22, flowEstimate present; GetChokepointHistory returns 174 entries). User-reported "Hormuz empty" is likely browser/CDN stale cache from before PR #3185; hard refresh should resolve. - scenario-worker.mjs has zero result keys in 24h. Railway service needs verification/redeployment. * fix(scenario): wrong Upstash RPUSH format silently broke every Simulate Closure Railway scenario-worker log shows every job failing field validation since at least 03:06Z today: [scenario-worker] Job failed field validation, discarding: ["{\"jobId\":\"scenario:1776535792087:cynxx5v4\",... The leading [" in the payload is the smoking gun. api/scenario/v1/run.ts was POSTing to /rpush/{key} with body `[payload]`, expecting Upstash to unpack the array and push one string value. Upstash does NOT parse that form — it stored the literal `["{...}"]` string as a single list value. Worker BLMOVEs the literal string → JSON.parse → array → destructure `{jobId, scenarioId, iso2}` on an array returns undefined for all three → every job discarded without writing a result. Client poll returns `pending` for the full 60s timeout, then (on the prior client code path) leaked the stuck "Computing…" button state indefinitely. Fix: use the standard Upstash REST command format — POST to the base URL with body `["RPUSH", key, value]`. Matches scripts/ais-relay.cjs upstashLpush. After this, the scenario-queue:pending list stores the raw payload string, BLMOVE returns the payload, JSON.parse gives the object, validation passes, computeScenario runs, result key gets written, client poll sees `done`. Zero result keys existed in prod Redis in the last 24h (24h TTL on scenario-result:*) — confirms the fix addresses the production outage. |
||
|
Elie Habib
|
3c47c1b222 |
fix(supply-chain): split chokepoint transit data + close silent zero-state cache (#3185)
* fix(supply-chain): split chokepoint transit data + close silent zero-state cache
Production supply-chain panel was rendering 13 empty chokepoints because
the getChokepointStatus RPC silently cached zero-state for 5 minutes:
1. supply_chain:transit-summaries:v1 grew to ~500 KB (180d × 13 × 14 fields
of history per chokepoint).
2. REDIS_OP_TIMEOUT_MS is 1.5 s. Vercel Sydney edge → Upstash for a 500 KB
GET consistently exceeded the budget; getCachedJson caught the AbortError
and returned null.
3. The 500 KB portwatch fallback read hit the same timeout.
4. summaries = {} → every summaries[cp.id] was undefined → 13 chokepoints
got the zero-state default → cached as a non-null success response for
REDIS_CACHE_TTL (5 min) instead of NEG_SENTINEL (120 s).
Fix (one PR, per docs/plans/chokepoint-rpc-payload-split.md):
- ais-relay.cjs: split seedTransitSummaries output.
- supply_chain:transit-summaries:v1 — compact (~30 KB, no history).
- supply_chain:transit-summaries:history:v1:{id} — per chokepoint
(~35 KB each, 13 keys). Both under the 1.5 s Redis read budget.
- New RPC GetChokepointHistory: lazy-loaded on card expand.
- get-chokepoint-status.ts: drop the 500 KB portwatch/corridorrisk/
chokepoint_transits fallback reads. Treat a null transit-summaries
read as upstreamUnavailable=true so cachedFetchJson writes NEG_SENTINEL
(2 min) instead of a 5-min zero-state pin. Omit history from the
response (proto field stays declared; empty array).
- server/_shared/redis.ts: tag AbortError timeouts with [REDIS-TIMEOUT]
key=… timeoutMs=… so log drains / Sentry-Vercel integration pick up
large-payload timeouts instead of them being silently swallowed.
- SupplyChainPanel.ts + MapPopup.ts: lazy-fetch history on card expand
via fetchChokepointHistory; session-scoped cache; graceful "History
unavailable" on empty/error. PRO gating on the map popup unchanged.
- Gateway: cache-tier entry for /get-chokepoint-history (slow).
- Tests: regression guards for upstreamUnavailable gate + per-id key
shape + handler wiring + proto query annotations.
Audit included in plan: no other RPC consumer read stacks >200 KB
besides displacement:summary:v1:2026 (724 KB, same risk, flagged for
follow-up PR). wildfire:fires:v1 at 1.7 MB loads via bootstrap (3 s
timeout, different path) — monitor but out of scope.
Expected impact:
- supply_chain:chokepoints:v4 payload drops from ~508 KB to <100 KB.
- supply_chain:transit-summaries:v1 drops from ~502 KB to <50 KB.
- RPC Redis reads stay well under 1.5 s in the hot path.
- Silent zero-state pinning is now impossible: null reads → 2-min neg
cache → self-heal on next relay tick.
* fix(supply-chain): address PR #3185 review — stop caching empty/error + fix partial coverage
Two P1 regressions caught in review:
1. Client cache poisoning on empty/error (MapPopup.ts, SupplyChainPanel.ts)
Empty-array is truthy in JS, so MapPopup's `!cached && !inflight` branch
never fired once we cached []. Neither `cached && cached.length` fired
either — popup stuck on "Loading transit history..." for the session.
SupplyChainPanel had the explicit `cached && !cached.length` branch but
still never retried, so the same transient became session-sticky there too.
Fix: cache ONLY non-empty successful responses. Empty/error show the
"History unavailable" placeholder but leave the cache untouched, so the
next re-expand retries. The /get-chokepoint-history gateway tier is
"slow" (5-min CF edge cache) → retries stay cheap.
2. Partial portwatch coverage treated as healthy (ais-relay.cjs)
seedTransitSummaries iterated Object.entries(pw), so if seed-portwatch
dropped N of 13 chokepoints (ArcGIS reject/empty), summaries had <13 keys.
get-chokepoint-status upstreamUnavailable fires only on fully-empty
summaries, so the N missing chokepoints fell through to zero-state rows
that got pinned in cache for 5 minutes.
Fix: iterate CANONICAL_IDS (Object.keys(CHOKEPOINT_THREAT_LEVELS)) and
fill zero-state for any ID missing from pw. Shape is consistently 13
keys. Track pwCovered → envelope + seed-meta recordCount reflect real
upstream coverage (not shape size), so health.js can distinguish 13/13
healthy from 10/13 partial. Warn-log on shortfall.
Tests: new regression guards
- panel must NOT cache empty arrays (historyCache.set with []).
- writer must iterate CANONICAL_IDS, not Object.entries(pw).
- seed-meta recordCount binds to pwCovered.
5718/5718 data tests pass. typecheck + typecheck:api clean.
|
||
|
Elie Habib
|
dcf73385ca |
fix(scoring): rebalance formula weights severity 55%, corroboration 15% (#3144)
* fix(scoring): rebalance formula weights severity 55%, corroboration 15%
PR A of the scoring recalibration plan (docs/plans/2026-04-17-002).
The v2 shadow-log recalibration (690 items, Pearson 0.413) showed the
formula compresses scores into a narrow 30-70 range, making the 85
critical gate unreachable and the 65 high gate marginal. Root cause:
corroboration at 30% weight penalizes breaking single-source news
(the most important alerts) while severity at 40% doesn't separate
critical from high enough.
Weight change:
BEFORE: severity 0.40 + sourceTier 0.20 + corroboration 0.30 + recency 0.10
AFTER: severity 0.55 + sourceTier 0.20 + corroboration 0.15 + recency 0.10
Expected effect: critical/tier1/fresh rises from 76 to 88 (clears 85
gate). critical/tier2/fresh rises from 71 to 83 (recommend lowering
critical gate to 80 at activation time). high/tier2/fresh rises from
61 to 69 (clears 65 gate). The HIGH-CRITICAL gap widens from 10 to
14 points for same-tier items.
Also:
- Bumps shadow-log key from v2 to v3 for a clean recalibration dataset
(v2 had old-weight scores that would contaminate the 48h soak)
- Updates proto/news_item.proto formula comment to reflect new weights
- Updates cache-keys.ts documentation
No cache migration needed: the classify cache stores {level, category},
not scores. Scores are computed at read time from the stored level +
the formula, so new digest requests immediately produce new scores.
Gates remain OFF. After 48h of v3 data, re-run:
node scripts/shadow-score-report.mjs
node scripts/shadow-score-rank.mjs sample 25
🤖 Generated with Claude Opus 4.6 via Claude Code + Compound Engineering v2.49.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: regenerate proto OpenAPI docs for weight rebalance
* fix(scoring): bump SHADOW_SCORE_LOG_KEY export to v3
The exported constant in cache-keys.ts was left at v2 while the relay's
local constant was bumped to v3. Anyone importing the export (or grep-
discovering it) would get a stale key. Architecture review flagged this.
* fix(scoring): update test + stale comments for shadow-log v3
Review found the regression test still asserted v2 key, causing CI
failure. Also fixed stale v1/v2 references in report script header,
default-key comment, report title render, and shouldNotify docstring.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
Elie Habib
|
e32d9b631c |
feat(market): Hyperliquid perp positioning flow as leading indicator (#3074)
* feat(market): Hyperliquid perp positioning flow as leading indicator Adds a 4-component composite (funding × volume × OI × basis) "positioning stress" score for ~14 perps spanning crypto (BTC/ETH/SOL), tokenized gold (PAXG), commodity perps (WTI, Brent, Gold, Silver, Pt, Pd, Cu, NatGas), and FX perps (EUR, JPY). Polls Hyperliquid /info every 5min via Railway cron; publishes a single self-contained snapshot with embedded sparkline arrays (60 samples = 5h history). Surfaces as a new "Perp Flow" tab in CommoditiesPanel with separate Commodities / FX sections. Why: existing CFTC COT is weekly + US-centric; market quotes are price-only. Hyperliquid xyz: perps give 24/7 global positioning data that has been shown to lead spot moves on commodities and FX by minutes-to-hours. Implementation: - scripts/seed-hyperliquid-flow.mjs — pure scoring math, symbol whitelist, content-type + schema validation, prior-state read via readSeedSnapshot(), warmup contract (first run / post-outage zeroes vol/OI deltas), missing-symbol carry-forward, $500k/24h min-notional guard to suppress thin xyz: noise. TTL 2700s (9× cadence). - proto/worldmonitor/market/v1/get_hyperliquid_flow.proto + service.proto registration; make generate regenerated client/server bindings. - server/worldmonitor/market/v1/get-hyperliquid-flow.ts — getCachedJson reader matching get-cot-positioning.ts seeded-handler pattern. - server/gateway.ts cache-tier entry (medium). - api/health.js: hyperliquidFlow registered with maxStaleMin:15 (3× cadence) + transitional ON_DEMAND_KEYS gate for the first ~7 days of bake-in. - api/seed-health.js mirror with intervalMin:5. - scripts/seed-bundle-market-backup.mjs entry (NIXPACKS auto-redeploy on scripts/** watch). - src/components/MarketPanel.ts: CommoditiesPanel grows a Perp Flow tab + fetchHyperliquidFlow() RPC method; OI Δ1h derived from sparkOi tail. - src/App.ts: prime via primeVisiblePanelData() + recurring refresh via refreshScheduler.scheduleRefresh() at 5min cadence (panel does NOT own setInterval; matches the App.ts:1251 lifecycle convention). - 28 unit tests covering scoring parity, warmup flag, min-notional guard, schema rejection, missing-symbol carry-forward, post-outage cold start, sparkline cap, alert threshold. Tests: test:data 5169/5169, hyperliquid-flow-seed 28/28, route-cache-tier 5/5, typecheck + typecheck:api green. One pre-existing test:sidecar failure (cloud-fallback origin headers) is unrelated and reproduces on origin/main. * fix(hyperliquid-flow): address review feedback — volume baseline window, warmup lifecycle, error logging Two real correctness bugs and four review nits from PR #3074 review pass. Correctness fixes: 1. Volume baseline was anchored to the OLDEST 12 samples, not the newest. sparkVol is newest-at-tail (shiftAndAppend), so slice(0, 12) pinned the rolling mean to the first hour of data forever once len >= 12. Volume scoring would drift further from current conditions each poll. Switched to slice(-VOLUME_BASELINE_MIN_SAMPLES) so the baseline tracks the most recent window. Regression test added. 2. Warmup flag flipped to false on the second successful poll while volume scoring still needed 12+ samples to activate. UI told users warmup lasted ~1h but the badge disappeared after 5 min. Tied per-asset warmup to real baseline readiness (coldStart OR vol samples < 12 OR prior OI missing). Snapshot-level warmup = any asset still warming. Three new tests cover the persist-through-baseline-build, clear-once-ready, and missing-OI paths. Review nits: - Handler: bare catch swallowed Redis/parse errors; now logs err.message. - Panel: bare catch in fetchHyperliquidFlow hid RPC 500s; now logs. - MarketPanel.ts: deleted hand-rolled RawHyperliquidAsset; mapHyperliquidFlowResponse now takes GetHyperliquidFlowResponse from the generated client so proto drift fails compilation instead of silently. - Seeder: added @ts-check + JSDoc on computeAsset per type-safety rule. - validateUpstream: MAX_UPSTREAM_UNIVERSE=2000 cap bounds memory. - buildSnapshot: logs unknown xyz: perps upstream (once per run) so ops sees when Hyperliquid adds markets we could whitelist. Tests: 37/37 green; typecheck + typecheck:api clean. * fix(hyperliquid-flow): wire bootstrap hydration per AGENTS.md mandate Greptile review caught that AGENTS.md:187 mandates new data sources be wired into bootstrap hydration. Plan had deferred this on "lazy deep-dive signal" grounds, but the project convention is binding. - server/_shared/cache-keys.ts: add hyperliquidFlow to BOOTSTRAP_CACHE_KEYS + BOOTSTRAP_TIERS ('slow' — non-blocking, page-load-parallel). - api/bootstrap.js: add to inlined BOOTSTRAP_CACHE_KEYS + SLOW_KEYS so bootstrap.test.mjs canonical-mirror assertions pass. - src/components/MarketPanel.ts: - Import getHydratedData from @/services/bootstrap. - New mapHyperliquidFlowSeed() normalizes the raw seed-JSON shape (numeric fields) into HyperliquidFlowView. The RPC mapper handles the proto shape (string-encoded numbers); bootstrap emits the raw blob. - fetchHyperliquidFlow now reads hydrated data first, renders immediately, then refreshes from RPC — mirrors FearGreedPanel pattern. Tests: 72/72 green (bootstrap + cache-tier + hyperliquid-flow-seed). |
||
|
Elie Habib
|
0cdfddc885 |
feat(gold): central-bank reserves via IMF IFS (PR C) (#3038)
* feat(gold): central-bank gold reserves via IMF IFS (PR C) * fix(gold): prefer ounces indicator over USD in IMF IFS candidate list * fix(gold): align seed-health interval with monthly IMF cadence + drop ALG dup Review findings on PR #3038: - api/seed-health.js: intervalMin was 1440 (1 day), which flags stale at 2880min (48h) — contradicted health.js maxStaleMin=44640 (~31 days) and would false-alarm within 2 days on a monthly data source. Bumped to 22320 so both endpoints agree at ~31 days. - seed-gold-cb-reserves ISO3_NAMES: dropped duplicate ALG entry (World Bank variant); DZA is canonical ISO 3166-1 alpha-3 and stays. |
||
|
Elie Habib
|
a8b85e52c8 |
feat(gold): SPDR GLD physical holdings flows (PR B) (#3037)
* feat(gold): SPDR GLD physical holdings flows (PR B) * fix(gold): strip UTF-8 BOM from SPDR CSV header (greptile P2 #3037) |
||
|
Elie Habib
|
ee66b6b5c2 |
feat(gold): Gold Intelligence v2 — positioning depth, returns, drivers (#3034)
* feat(gold): richer Gold Intelligence panel with positioning, returns, drivers * fix(gold): restore leveragedFunds fields and derive P/S netPct in legacy fallback Review catch on PR #3034: 1. seed-cot.mjs stopped emitting leveragedFundsLong/Short during the v2 refactor, which would zero out the Leveraged Funds bars in the existing CotPositioningPanel on the next seed run. Re-read lev_money_* from the TFF rows and keep the fields on the output (commodity rows don't have this breakdown, stay at 0). 2. get-gold-intelligence legacy fallback hardcoded producerSwap.netPct to 0, meaning a pre-v2 COT snapshot rendered a neutral 0% Producer/Swap bar on deploy until seed-cot reran. Derive netPct from dealerLong/dealerShort (same formula as the v2 seeder). OI share stays 0 because open_interest wasn't captured pre-migration; clearly documented now. Tests: added two regression guards (leveragedFunds preserved for TFF, commodity rows emit 0 for those fields). * fix(gold): make enrichment layer monitored and honest about freshness Review catch on PR #3034: - seed-commodity-quotes now writes seed-meta:market:gold-extended via writeExtraKeyWithMeta on every successful run. Partial / failed fetches skip BOTH the data write and the meta bump, so health correctly reports STALE_SEED instead of masking a broken Yahoo fetch with a green check. - Require both gold (core) AND at least one driver/silver before writing, so a half-successful run doesn't overwrite healthy prior data with a degraded payload. - Handler no longer stamps updatedAt with new Date() when the enrichment key is missing. Emits empty string so the panel's freshness indicator shows "Updated —" with a dim dot, matching reality — enrichment is missing, not fresh. - Health: goldExtended entry in STANDALONE_KEYS + SEED_META (maxStaleMin 30, matching commodity quotes), and seed-health.js advertises the domain so upstream monitors pick it up. The panel already gates session/returns/drivers sections on presence, so legacy panels without the enrichment layer stay fully functional. |
||
|
Elie Habib
|
281a7c0728 | chore: regenerate MarketService OpenAPI specs for GetGoldIntelligence (#3011) | ||
|
Elie Habib
|
793d7df9dc | feat(energy-crisis): add IEA 2026 Energy Crisis Policy Response Tracker panel and seeder (#3008) | ||
|
Elie Habib
|
c26ae6b827 | feat(energy): add Oil Inventories panel with SVG charts (#3003) | ||
|
Elie Habib
|
c72251178c |
feat(route-explorer): Sprint 4 — strategic-product impact tab + get-route-impact RPC (#2996)
* feat(route-explorer): Sprint 4 — strategic-product impact tab
Adds the Impact tab to the Route Explorer, powered by a new
get-route-impact RPC that returns strategic-product trade data for
any country pair.
Backend:
- New proto get_route_impact.proto with GetRouteImpact{Request,Response}
+ StrategicProduct message
- New handler server/worldmonitor/supply-chain/v1/get-route-impact.ts:
reads comtrade:bilateral-hs4:{iso2}:v1 store, computes lane value for
selected HS2, top 5 strategic products by value with chokepoint
exposure, resilience score (server-side from Redis), dependency flags
- Cache key ROUTE_IMPACT_KEY in cache-keys.ts (NOT in BOOTSTRAP_KEYS)
- Gateway + premium-paths registered as slow-browser premium RPC
- Client wrapper fetchRouteImpact in supply-chain/index.ts
Impact tab UI:
- CountryImpactTab.ts: strategic products table (top 5 by value),
lane value card for selected HS2, hs2InSeededUniverse banner when
HS2 is not in the 14 seeded sectors, comtradeSource states
(missing/empty/bilateral-hs4), drill-sideways on product row click
- LeftRail.updateDependencyFlags: renders flags from Impact response
with color-coded badges (compound_risk/single_source/diversifiable)
Data flow:
- fetchImpact fires in parallel with fetchResilience after lane data
loads, generation-scoped
- Impact response updates left-rail flags + resilience score
- Drill-sideways: clicking a product row switches the explorer's HS2
and re-queries all tabs
Server-side resilience:
- get-route-impact reads resilience:score:v8:{iso2} from Redis directly
so the data is available for future email briefs without client calls
Plan: docs/plans/2026-04-11-001-feat-worldwide-route-explorer-plan.md
* fix(route-explorer): real exposure score for flags + tabstrip sync on drill
P1: computeDependencyFlags hardcoded primaryExposure=80 whenever any
chokepoint existed, fabricating SINGLE_CORRIDOR_CRITICAL without using
real exposure data. Replaced with computeRealExposureScore that uses the
same route-cluster overlap logic as get-sector-dependency, computing the
actual exposure percentage before comparing against the >80 threshold.
P2: handleDrillSideways set state.tab=1 directly without going through
setTab(), leaving the tabstrip visually and semantically on Impact while
content showed Current. Now calls setTab(1) which updates both the
tabstrip active state and aria-selected.
* fix(route-explorer): guard resilience overwrite + normalize HS2 filter
P1: fetchImpact could zero the left-rail resilience score when
get-route-impact returned resilienceScore=0 (Redis miss fallback),
overwriting a valid score set by the concurrent fetchResilience call.
Now only applies the server-side score when it is actually > 0.
P2: HS4-to-HS2 matching used a redundant dual-condition filter
(hs4ToHs2 + startsWith) that masked a potential normalization bug.
Simplified to normalize hs2 once via parseInt then use a single
hs4ToHs2 comparison.
|
||
|
Elie Habib
|
e070a97c3d |
Phase 3 PR2: Weekly regional briefs (LLM seeder + RPC) (#2989)
* feat(intelligence): weekly regional briefs (Phase 3 PR2) Phase 3 PR2 of the Regional Intelligence Model. Adds LLM-powered weekly intelligence briefs per region, completing the core feature set. ## New seeder: scripts/seed-regional-briefs.mjs Standalone weekly cron script (not part of the 6h derived-signals bundle). For each non-global region: 1. Read the latest snapshot via two-hop Redis read 2. Read recent regime transitions from the history log (#2981) 3. Call the LLM once per region with regime trajectory + balance + triggers + narrative context 4. Write structured brief to intelligence:regional-briefs:v1:weekly:{region} with 8-day TTL (survives one missed weekly run) Reuses the same injectable-callLlm + parse-validation + provider-chain pattern from narrative.mjs and weekly-brief.mjs. ## New module: scripts/regional-snapshot/weekly-brief.mjs generateWeeklyBrief(region, snapshot, transitions, opts?) -> { region_id, generated_at, period_start, period_end, situation_recap, regime_trajectory, key_developments[], risk_outlook, provider, model } buildBriefPrompt() — pure prompt builder parseBriefJson() — JSON parser with prose-extraction fallback emptyBrief() — canonical empty shape Global region is skipped. Provider chain: Groq -> OpenRouter. Validate callback ensures only parseable responses pass (narrative.mjs PR #2960 review fix pattern). ## Proto + RPC: GetRegionalBrief proto/worldmonitor/intelligence/v1/get_regional_brief.proto - GetRegionalBriefRequest { region_id } - GetRegionalBriefResponse { brief: RegionalBrief } - RegionalBrief { region_id, generated_at, period_start, period_end, situation_recap, regime_trajectory, key_developments[], risk_outlook, provider, model } ## Server handler server/worldmonitor/intelligence/v1/get-regional-brief.ts Simple getCachedJson read + adaptBrief snake->camel adapter. Returns upstreamUnavailable: true on Redis failure so the gateway skips caching (matching the get-regime-history pattern from #2981). ## Premium gating + cache tier src/shared/premium-paths.ts + server/gateway.ts RPC_CACHE_TIER ## Tests — 27 new unit tests buildBriefPrompt (5): region/balance/transitions/narrative rendered, empty transitions handled, missing fields tolerated parseBriefJson (5): valid JSON, garbage, all-empty, cap at 5, prose extraction generateWeeklyBrief (6): success, global skip, LLM fail, garbage, exception, period_start/end delta emptyBrief (2): region_id + empty fields handler (4): key prefix, adapter export, upstreamUnavailable, registration security (2): premium path + cache tier proto (3): RPC declared, import wired, RegionalBrief fields ## Verification - npm run test:data: 4651/4651 pass - npm run typecheck + typecheck:api: clean - biome lint: clean * fix(intelligence): address 3 review findings on #2989 P2 #1 — no consumer surface for GetRegionalBrief Acknowledged. The consumer is the RegionalIntelligenceBoard panel, which will call GetRegionalBrief and render a weekly brief block. This wiring is Phase 3 PR3 (UI) scope — the RPC + Redis key are the delivery mechanism, not the end surface. No code change in this commit; the RPC is ready for the panel to consume. P2 #2 — readRecentTransitions collapses failure to [] readRecentTransitions returned [] on Redis/network failure, which is indistinguishable from a genuinely quiet week. The LLM then generates a brief claiming "no regime transitions" when in reality the upstream is down — fabricating false input. Fix: return null on failure. The seeder skips the region with a clear log message when transitions is null, so the brief is never written with unreliable input. Empty array [] now only means genuinely no transitions in the 7-day window. P2 #3 — parseBriefJson accepts briefs the seeder rejects parseBriefJson treated non-empty key_developments as valid even if situation_recap was empty. The seeder gate only writes when brief.situation_recap is truthy. That mismatch means the validator pass + provider-fallback logic could accept a response that the seeder then silently drops. Fix: require situation_recap in parseBriefJson for valid=true, matching the seeder gate. Now both checks agree on what constitutes a usable brief, and the provider-fallback chain correctly falls through when a provider returns a brief with developments but no recap. * fix(intelligence): TTL path-segment fix + seed-meta always-write (Greptile P1+P2 on #2989) P1 — TTL silently not applied (briefs never expire) Upstash REST ignores query-string SET options (?EX=N). The correct form is path-segment: /set/{key}/{value}/EX/{seconds}. Without this fix every brief persists indefinitely and Redis storage grows unboundedly across weekly runs. P2 — seed-meta not written when all regions skipped writeExtraKeyWithMeta was gated on generated > 0. If every region was skipped (no snapshot yet, or LLM failed), seed-meta was never written, making the seeder indistinguishable from "never ran" in health tooling. Now writes seed-meta whenever failed === 0, carrying regionsSkipped count. P2 #3 (validate gate) — already fixed in previous commit (parseBriefJson now requires situation_recap for valid=true). * fix(intelligence): register regional-briefs in health.js SEED_META + STANDALONE_KEYS (review P2 on #2989) * fix(intelligence): register regional-briefs in api/seed-health.js (review P2 on #2989) * fix(intelligence): raise brief TTL to 15 days to cover missed weekly cycle (review P2 on #2989) * fix(intelligence): distinguish missing-key from Redis-error + coverage-gated health (review P2s on #2989) P2 #1 — false upstreamUnavailable before first seed getCachedJson returns null for both "key missing" and "Redis failed", so the handler was advertising an outage for every region before the first weekly seed ran. Switched to getRawJson (throws on Redis errors) so null = genuinely missing key → clean empty 200, and thrown error = upstream failure → upstreamUnavailable: true for gateway no-store. P2 #2 — partial run hides coverage loss in health The seed-meta was written with generated count even if only 1 of 7 regions produced a brief. /api/health treats any positive recordCount as healthy, so broad regional failure was invisible to operators. Fix: recordCount is set to 0 when generated < ceil(expectedRegions/2). This makes /api/health report EMPTY_DATA for severely partial runs while still writing seed-meta (so the seeder is confirmed to have run). coverageOk flag in the summary payload lets operators drill into the exact coverage state. * fix(intelligence): tighten coverage gate to expectedRegions-1 (review P2 on #2989) |
||
|
Elie Habib
|
39d5199ae0 |
feat(resilience): three-pillar schema + schemaVersion v2.0 feature flag (Phase 2 T2.1) (#2977)
* feat(resilience): three-pillar schema + schemaVersion v2.0 feature flag (Phase 2 T2.1) Ships the Phase 2 T2.1 schema slice of the country-resilience reference grade upgrade plan. Adds the three-pillar response shape (StructuralReadiness, LiveShockExposure, RecoveryCapacity) as a new `pillars` field on GetResilienceScoreResponse alongside a `schemaVersion` string field, both gated behind the RESILIENCE_SCHEMA_V2_ENABLED env flag (default false). This PR is schema + plumbing only. Pillars ship with score=0, coverage=0; real aggregation lands in PR 4 (T2.3). No behavior change at the v1 default, which preserves widget / map / Country Brief compatibility for one release cycle per the plan. What this PR commits - Proto: new ResiliencePillar message (id, score, weight, coverage, domains). New `pillars` repeated field + `schema_version` string field on GetResilienceScoreResponse. No renumbering or mutation of existing fields. - Generated TS: regenerated service_server.ts, service_client.ts, and OpenAPI JSON/YAML via `make generate`. - New module server/worldmonitor/resilience/v1/_pillar-membership.ts: declarative PILLAR_DOMAINS map + PILLAR_WEIGHTS map + ordered iteration list. Single source of truth for the pillar structure that PR 4 will import. Note: pillar membership uses the runtime ResilienceDomainId values (kebab-case domain ids that already ship in v1), not the long-form pillar names from the plan example. - structural-readiness (0.40): economic, infrastructure, social-governance - live-shock-exposure (0.35): energy, health-food - recovery-capacity (0.25): empty until PR 3 adds the new dimensions - Response builder: new buildPillarList helper emits shaped-but-empty pillars when the v2 flag is on, empty array when off. Response literal fallback paths in _shared.ts and the LOCKED_PREVIEW fixture in resilience-widget-utils.ts updated to include pillars: [] and schemaVersion: '1.0' to satisfy the generated TS types. - Tests: 13 new pillar-schema unit cases (membership invariants, weight sum=1.0, disjoint sets, empty recovery pillar, buildPillarList flag-off / flag-on / shuffled-order / partial-domain-set) + 3 response-shape cases on the release-gate test pinning the v1 default shape and the new field presence on the wire. What is deliberately NOT in this PR - No aggregation logic: score/coverage on pillars stay 0 until PR 4. - No cache key bump: schema is additive with proto3 defaults. - No changes to overallScore/baselineScore/stressScore (parallel for one release cycle). - No new seeders or dimensions (PR 3 / T2.2b). - No tiering registry changes (PR 2 / T2.2a). - No widget rendering (Phase 3 T3.6). Verified - make generate clean, new ResiliencePillar interface in regenerated client + server + OpenAPI artifacts - typecheck + typecheck:api clean - tests/resilience-pillar-schema.test.mts: 13/13 passing - tests/resilience-release-gate.test.mts: 14/14 passing (3 new T2.1 cases + 11 prior) - full resilience suite: 283/283 passing - npm run test:data: 4539/4539 passing - npm run lint: exit 0 * fix(resilience): cache-flag decoupling + freshness error-status guard (#2977 P1+P2) Two Greptile review findings addressed: P1: RESILIENCE_SCHEMA_V2_ENABLED changed the cached response shape but the cache key did not encode the flag state. Flipping the flag on a warm cache served stale v1.0 payloads until the 6h TTL expired. Fix: always compute and cache the v2 superset (with pillars and schemaVersion='2.0'). Apply the flag as a response-time gate: when off, strip pillars to [] and downgrade schemaVersion to '1.0' before returning. This decouples the cache from the flag and makes flag flips take effect immediately without waiting for TTL expiry. P2: readFreshnessMap in _dimension-freshness.ts trusted fetchedAt without checking status. The resilience-static seeder writes fetchedAt: Date.now() on BOTH success and error paths (status: 'ok' vs 'error'), so a failed seed run that preserved old data via extendExistingTtl made the freshness badges show 'fresh' for what is actually stale data. Fix: skip seed-meta entries where status !== 'ok'. When the meta is skipped, the dimension has no freshness data and classifies as stale, matching api/health.js behavior. Added a test case that verifies error-status entries are excluded from the freshness map. |
||
|
Elie Habib
|
822eef0fa6 |
feat(supply-chain): Sprint 1 — Route Explorer wrapper RPC (#2980)
* feat(supply-chain): Sprint 1 — Route Explorer wrapper RPC
Adds an internal wrapper around the vendor-only route-intelligence
compute so the upcoming Route Explorer UI can call it from a browser
PRO session instead of forcing an X-WorldMonitor-Key API gate.
Backend:
- New proto get-route-explorer-lane.proto with GetRouteExplorerLane{Request,Response}
- New handler server/worldmonitor/supply-chain/v1/get-route-explorer-lane.ts
- New static lookup tables _route-explorer-static-tables.ts:
TRANSIT_DAYS_BY_ROUTE_ID, FREIGHT_USD_BY_CARGO_TYPE,
BYPASS_CORRIDOR_GEOMETRY_BY_ID — covers all 5 land-bridge corridors
plus every sea-alternative corridor with hand-curated coordinates
- Wired into supply-chain handler.ts service dispatcher
- Cache key ROUTE_EXPLORER_LANE_KEY in cache-keys.ts (NOT in BOOTSTRAP_KEYS)
- Gateway entry: PREMIUM_RPC_PATHS + RPC_CACHE_TIER 'slow-browser'
- Premium path entry in src/shared/premium-paths.ts so browser PRO auth attaches
Response contract enriches route-intelligence with:
- primaryRouteGeometry polyline from TRADE_ROUTES (lon/lat pairs)
- fromPort/toPort coords on every bypass option so the client can call
MapContainer.setBypassRoutes directly without geometry lookups
- status: 'active' | 'proposed' | 'unavailable' derived from corridor notes
to honestly label kra_canal_future and black_sea_western_ports
- estTransitDaysRange + estFreightUsdPerTeuRange from static tables
- noModeledLane: true when origin/destination clusters share no routes
Client wrapper fetchRouteExplorerLane added to src/services/supply-chain/index.ts.
Tests: tests/route-explorer-lane.test.mts — 30-query smoke matrix
(10 country pairs × 3 HS2 codes), structural assertions only, no
hard-coded transit/cost values. Test exposes a pure computeLane()
function with an injectable status map so it does not need Redis.
Gap report (from smoke run): 12 of 30 queries fall back to a synthetic
primaryRouteId because the destination's port cluster has no shared route
with the origin (US-JP, ZA-IN, CL-CN, TR-DE × 3 HS2 each). These pairs
return noModeledLane:true; Sprint 3 will render an empty-state for them.
Plan: docs/plans/2026-04-11-001-feat-worldwide-route-explorer-plan.md
* fix(route-explorer): address PR #2980 review findings
P1: bypass warRiskTier was hard-coded to WAR_RISK_TIER_NORMAL, dropping
the live risk signal from chokepoint status. Now derived from the
statusMap via the corridor's primaryChokepointId.
P2: freight fallback in emptyResponse and client-side empty payload used
a cargo-agnostic container range for all cargo types. Removed the ranges
entirely from fallback/noModeledLane responses; they are only present
when the lane is actually modeled.
Suggestion: when noModeledLane is true, the response now returns empty
primaryRouteId, empty geometry, empty exposures, empty bypasses, and
omits transit/freight ranges. Previously it returned plausible-looking
synthetic data from the origin's first route which could mislead the UI.
Tests updated to assert the noModeledLane contract: empty fields when
the flag is set, non-empty ranges only when the lane is modeled.
* fix(route-explorer): cargo-aware route ranking + bypass waypoint risk
P1: primary route selection was order-dependent, picking whichever
shared route the origin cluster listed first. Mixed clusters like
CN/JP could return an energy lane for a container request. Now ranks
shared routes by cargo-category compatibility (container→container,
tanker→energy, bulk→bulk, roro→container) before selecting.
P1: bypass warRiskTier was copied from the primary chokepoint instead
of derived from the corridor's own waypointChokepointIds. This
overstated risk for alternatives like Cape of Good Hope whose waypoints
may have a lower risk tier. Now uses max-tier across waypoint
chokepoints, matching get-bypass-options.ts logic.
Suggestion: placeholder corridors with addedTransitDays=0 (like
gibraltar_no_bypass, cape_of_good_hope_is_bypass) are now filtered out.
Previously they could surface as active alternatives.
Regression tests added:
- CN→JP tanker: asserts energy route is selected over container route
- CN→DE with faked Suez=CRITICAL / Cape=NORMAL: asserts Cape bypass
shows NORMAL, not CRITICAL
- ES→EG: asserts zero-transit-day placeholders are excluded
* fix(route-explorer): scope exposures to primary route + narrow placeholder filter
P1: chokepointExposures and bypassOptions were computed from the full
sharedRoutes set, mixing data from energy/container corridors into a
single response. Now scoped to the cargo-ranked primaryRouteId only,
matching the proto contract that exposures are "on the primary route."
P2: the addedTransitDays === 0 filter was too broad and removed
kra_canal_future (a proposed bypass with real modeling). Narrowed to an
explicit PLACEHOLDER_CORRIDOR_IDS set (gibraltar_no_bypass,
cape_of_good_hope_is_bypass) so proposed zero-day corridors survive and
are surfaced with CORRIDOR_STATUS_PROPOSED.
Regression tests:
- chokepointExposures follow primaryRouteId (CN->JP container)
- kra_canal_future appears as CORRIDOR_STATUS_PROPOSED for Malacca routes
- placeholder filter still excludes explicit placeholders
* fix(route-explorer): address PR #2980 review comments
1. Unavailable corridors without waypoints (e.g. black_sea_western_ports)
now derive WAR_RISK_TIER_WAR_ZONE from their CORRIDOR_STATUS_UNAVAILABLE
status, instead of returning WAR_RISK_TIER_UNSPECIFIED. Corridors with
waypointChokepointIds still use max-tier across those waypoints.
2. Added fixture test with non-empty status map (suez=75/HIGH,
malacca=30/ELEVATED) so disruptionScore and warRiskTier assertions are
not trivially satisfied by the empty-map default path.
3. Documented the single-chokepoint bypass design gap in the test gap report:
bypassOptions only cover the primary chokepoint; multi-chokepoint routes
show exposure for all but bypass guidance for only the top one. Sprint 3
will decide whether to expand to top-N or add a UI hint.
|
||
|
Elie Habib
|
19d67cea94 |
Phase 3 PR1: Regime drift history (writer + RPC) (#2981)
* feat(intelligence): regime drift history (Phase 3 PR1)
Phase 3 PR1 of the Regional Intelligence Model. Adds an append-only
regime transition log per region plus a premium-gated RPC to read it.
## What landed
### New writer module: scripts/regional-snapshot/regime-history.mjs
Single public entry point:
recordRegimeTransition(region, snapshot, diff, opts?)
-> { recorded, entry, pushed, trimmed }
Pure builder + Redis-ops orchestrator + dependency-injected publisher.
Flow:
1. buildTransitionEntry() returns null when diff has no regime_changed
(steady-state snapshots produce no entry — pure transition stream)
2. publishTransitionWithOps() LPUSHes onto
intelligence:regime-history:v1:{region}, then LTRIMs to keep the
most recent REGIME_HISTORY_MAX (100) entries
3. defaultPublisher binds real Upstash REST calls; tests inject an
in-memory ops object for offline coverage
LTRIM failure is non-fatal — entry already landed, next cycle will
re-trim. LPUSH failure short-circuits and reports pushed=false. The
recorder NEVER throws and is wrapped in its own try/catch in the seed
loop so snapshot persist is never blocked.
### seed-regional-snapshots.mjs hook
Added a regime-history call alongside the existing alert-emitter call,
right after persistSnapshot success. Same best-effort contract:
unconditional try/catch, log warn on throw, continue main loop.
### Proto + RPC: GetRegimeHistory
proto/worldmonitor/intelligence/v1/get_regime_history.proto
- GetRegimeHistoryRequest { region_id, limit (0..100) }
- GetRegimeHistoryResponse { transitions: RegimeTransition[] }
- RegimeTransition { region_id, label, previous_label,
transitioned_at, transition_driver, snapshot_id }
region_id validated as strict kebab-case (same regex as
get-regional-snapshot). limit capped server-side at MAX_LIMIT=100,
defaulting to 50 when omitted.
Added to IntelligenceService in service.proto. Generated openapi
JSON/YAML committed via `make generate`.
### Server handler: server/worldmonitor/intelligence/v1/get-regime-history.ts
LRANGE-based read (newest-first because the writer LPUSHes). adapter
is a dedicated exported function adaptTransition(raw) for testability.
LRANGE helper is inlined here because server/_shared/redis.ts has no
list helpers yet — this is the first list-reading handler in the
intelligence service. If a second list reader lands, the helper can
be promoted to a shared util.
Empty list / Redis miss / failed JSON parse all return
{ transitions: [] } so the client can distinguish "never changed" from
"upstream broken" via the HTTP status code, not the body.
Registered in handler.ts.
### Premium gating + cache tier
src/shared/premium-paths.ts: added /api/intelligence/v1/get-regime-history
server/gateway.ts RPC_CACHE_TIER: same path with 'slow' tier (matches
route-parity contract enforced by
tests/route-cache-tier.test.mjs)
## Tests — 44 new unit tests
tests/regional-snapshot-regime-history.test.mjs (22 tests):
buildTransitionEntry (7):
- null on missing diff/region/snapshot
- returns entry on regime change
- first-ever transition (empty previous_label)
- falls back to generated_at when transitioned_at is missing
- preserves snapshot_id
publishTransitionWithOps (8):
- happy path (LPUSH + LTRIM both succeed)
- canonical key prefix
- LTRIM uses REGIME_HISTORY_MAX-1 stop
- LPUSH failure → not pushed, LTRIM not called
- LTRIM failure → pushed=true, trimmed=false (non-fatal)
- LPUSH/LTRIM throwing caught and reported
- null/empty entry → no-op
recordRegimeTransition (5):
- no-op on no regime change
- records on regime change
- publisher returning false → recorded=false
- publisher exceptions swallowed
- critical escalation labels preserved
module constants (2): key prefix + max are valid
tests/get-regime-history.test.mts (22 tests):
adaptTransition (4):
- all fields snake → camel
- missing fields → empty/zero defaults
- first-ever transition shape preserved
- non-numeric transitioned_at → 0
handler structural checks (7): canonical key prefix, LRANGE usage,
adapter export, handler export signature, MAX_LIMIT cap matches
writer, missing-region short-circuit, malformed-entry filter
intelligence handler registration (2): import + registration
security wiring (2): premium path + cache-tier entry
proto definition (7): RPC method declared, import wired, request
shape, kebab regex, limit bounds, RegimeTransition fields,
response shape
## Verification
- node --test tests/regional-snapshot-regime-history.test.mjs: 22/22 pass
- npx tsx --test tests/get-regime-history.test.mts: 22/22 pass
- npm run test:data: 4621/4621 pass
- npm run typecheck: clean
- npm run typecheck:api: clean
- biome lint on touched files: clean
## Deferred to future iterations
- Phase 3 PR2: weekly regional briefs LLM seeder (consumes regime history
to highlight drift events in the weekly summary)
- Phase 3 PR3: UI block in RegionalIntelligenceBoard for regime drift
timeline (can ride alongside or after PR2)
- Drift analytics: % of last N days spent in each regime, transition
frequency rolling window, regime cycle detection
- Alert triggers on drift cycles (e.g., "thrashed between regimes 3 times
in 7 days")
* fix(intelligence): address 2 review findings on #2981
P2 #1 — transition_driver always empty in the live path
buildRegimeState(balance, previousLabel, '') at Step 11 passed an empty
driver because the diff hasn't been computed yet. The regime-history
recorder reads snapshot.regime.transition_driver which was therefore
always '' in production, despite tests exercising synthetic fixtures
with a populated driver.
Fix: after Step 15 derives triggerReason via inferTriggerReason(diff),
backfill regime.transition_driver = triggerReason when a genuine regime
change occurred. This ensures both the persisted snapshot's regime block
AND the regime-history entry carry the real driver (e.g., 'regime_shift',
'trigger_activation', 'corridor_break').
Added 2 regression tests: populated driver flows through, and pre-fix
empty-driver snapshots remain back-compatible.
P2 #2 — Redis failure returns cached false-empty history
get-regime-history.ts returned 200 {transitions:[]} on LRANGE failure.
The gateway caches 200 GET responses at the slow tier, so a transient
Upstash outage would be pinned as a false-empty history until the cache
TTL expired.
Fix: when redisLrange returns null (Redis unavailable or credentials
missing), the response now includes upstreamUnavailable: true in the
body. The gateway already checks for this flag in the response body
(line 434) and sets Cache-Control: no-store, so transient failures are
not cached.
Added 1 structural test asserting the upstreamUnavailable flag is set.
Verification:
- 24/24 writer tests, 23/23 handler tests, 4624/4624 full suite pass
- npm run typecheck: clean
- biome lint on touched files: clean
* fix(intelligence): correct misleading 'log once per region' comment (Greptile P2)
|
||
|
Elie Habib
|
75b3b0026b |
feat(resilience): dimension freshness propagation (T1.5 propagation pass) (#2961)
* feat(resilience): dimension freshness propagation (T1.5 propagation pass) Ships the Phase 1 T1.5 propagation pass of the country-resilience reference-grade upgrade plan. PR #2947 shipped the staleness classifier foundation (classifyStaleness, cadence taxonomy, three staleness levels) and explicitly deferred the dimension-level propagation. This PR consumes the classifier and surfaces per dimension freshness on the ResilienceDimension response. What this PR commits - Proto: new DimensionFreshness message + `freshness` field on ResilienceDimension (last_observed_at_ms, staleness string). - New module server/worldmonitor/resilience/v1/_dimension-freshness.ts that reads seed-meta values for every sourceKey in INDICATOR_REGISTRY and aggregates the worst staleness + oldest fetchedAt across the constituent indicators of each dimension. - scoreAllDimensions decorates each dimension score with its freshness result before returning. The 13 dimension scorer function bodies are untouched: aggregation is a decoration pass at the caller level so this PR stays mechanical. - Response builder: _shared.ts buildDimensionList propagates the freshness field to the proto output. - Tests: 10 classifyDimensionFreshness + readFreshnessMap cases in a new test file + response-shape case on the release-gate test. Aggregation rules - last_observed_at_ms: MIN fetchedAt across the dimension's indicators (oldest signal = most conservative bound). 0 when no signal has ever been observed. - staleness: MAX staleness level across the dimension's indicators (stale > aging > fresh). Empty string when the dimension has no indicators in the registry (defensive path). What is deliberately NOT in this PR - No changes to the 13 individual dimension scorer function bodies. Per-signal freshness inside scorers is a future enhancement. - No widget rendering of the freshness badge (T1.6 full grid, PR 3). - No cache key bump: additive int64/string fields with zero defaults. Verified - make generate clean, new interface in regenerated types - typecheck + typecheck:api clean - tests/resilience-dimension-freshness.test.mts all new cases pass - tests/resilience-*.test.mts full suite pass - test:data clean - lint exits 0 on touched files * fix(resilience): resolve templated sourceKeys to real seed-meta (#2961 P1) Greptile P1 finding on PR #2961: readFreshnessMap() assumed every INDICATOR_REGISTRY sourceKey could be fetched as seed-meta:<sourceKey>, but most entries use placeholder templates like resilience:static:{ISO2}, energy:mix:v1:{ISO2}, and displacement:summary:v1:{year}. Those produce literal lookups like seed-meta:resilience:static:{ISO2} which don't exist in Redis, so the freshness map missed every templated entry and classifyDimensionFreshness marked the affected dimensions stale even with healthy seeds. Most Phase 1 T1.5 freshness badges were broken on arrival. Fix: two-layer resolution in _dimension-freshness.ts. Layer 1 stripTemplateTokens: drop :{placeholder} and :* segments. 'resilience:static:{ISO2}' -> 'resilience:static' 'resilience:static:*' -> 'resilience:static' 'energy:mix:v1:{ISO2}' -> 'energy:mix:v1' 'displacement:summary:v1:{year}' -> 'displacement:summary:v1' Layer 2 stripTrailingVersion: strip trailing :v\d+, mirroring writeExtraKeyWithMeta + runSeed() in scripts/_seed-utils.mjs which never persist the trailing version in seed-meta keys. Handles cyber:threats:v2, infra:outages:v1, unrest:events:v1, conflict:ucdp-events:v1, sanctions:country-counts:v1, and the displacement v1 case above. Layer 3 SOURCE_KEY_META_OVERRIDES: explicit table for drift cases where the two strips still do not match the real seed-meta key. Verified against api/seed-health.js, api/health.js, and scripts/seed-*. Drift cases covered: economic:imf:macro -> economic:imf-macro economic:bis:eer -> economic:bis economic:energy:v1:all -> economic:energy-prices energy:mix -> economic:owid-energy-mix energy:gas-storage -> energy:gas-storage-countries news:threat:summary -> news:threat-summary intelligence:social:reddit -> intelligence:social-reddit readFreshnessMap now deduplicates reads by resolved meta key (so the 15+ resilience:static indicators share one Redis read) and projects per-meta-key results back onto per-sourceKey map entries so classifyDimensionFreshness can keep its existing interface. Regression coverage: - stripTemplateTokens cases for {ISO2}, {year}, and *. - stripTrailingVersion cases for :v1 / :v2 suffixes. - Embedded :v1 carve-out (trade:restrictions:v1:tariff-overview:50 stays unchanged because :v1 is not trailing). - Override cases for the seven drift entries. - Integration test that proves every resilience:static:* / {ISO2} registry entry resolves to the same seed-meta and is marked fresh when that one key has a recent fetchedAt. - healthPublicService end-to-end test: classifies fresh when seed-meta:resilience:static is recent (was stale before the fix). - Registry-coverage assertion: every INDICATOR_REGISTRY sourceKey must resolve to a seed-meta key that either lives in api/seed-health.js, api/health.js, or the test's KNOWN_SEEDS_NOT_IN_HEALTH allowlist (which covers the four seeds written by writeExtraKeyWithMeta / runSeed that no health monitor tracks yet: trade:restrictions, trade:barriers, sanctions:country-counts, economic:energy-prices). Fails loudly if a future registry entry introduces an unknown sourceKey. Note on P1 #2 (scoreCurrencyExternal absence-branch delete): that is PR #2964's scope (T1.7 source-failure wiring), not #2961 (T1.5 propagation pass). #2961 never claimed to delete the fallback branch; no test in this branch expects the new IMPUTE.bisEer fallback. The reviewer conflated the two stacked PRs. #2964 owns the delete. |
||
|
Elie Habib
|
dca2e1ca3c |
feat(resilience): expose imputationClass on ResilienceDimension (T1.7 schema pass) (#2959)
* feat(resilience): expose imputationClass on ResilienceDimension (T1.7 schema pass) Ships the Phase 1 T1.7 schema pass of the country-resilience reference grade upgrade plan. PR #2944 shipped the classifier table foundation (ImputationClass type, ImputationEntry interface, IMPUTATION/IMPUTE tagged with four semantic classes) and explicitly deferred the schema propagation. This PR lands that propagation so downstream consumers can distinguish "country is stable" from "country is unmonitored" from "upstream is down" from "structurally not applicable" on a per-dimension basis. What this PR commits - Proto: new imputation_class string field on ResilienceDimension (empty string = dimension has any observed data; otherwise one of stable-absence, unmonitored, source-failure, not-applicable). - Generated TS types: regenerated service_server.ts and service_client.ts via make generate. - Scorer: ResilienceDimensionScore carries ImputationClass | null. WeightedMetric carries an optional imputationClass that imputation paths populate. weightedBlend aggregates the dominant class by weight when the dimension is fully imputed, returns null otherwise. - All IMPUTE.* early-return paths propagate the class from the table (IMPUTE.bisEer, IMPUTE.wtoData, IMPUTE.ipcFood, IMPUTE.unhcrDisplacement). - Response builder: _shared.ts buildDimensionList passes the class through to the ResilienceDimension proto field. - Tests: weightedBlend aggregation semantics (5 cases), dimension-level propagation from IMPUTE tables, serialized response includes the field. What is deliberately NOT in this PR - No widget icon rendering (T1.6 full grid, PR 3 of 5) - No source-failure seed-meta consultation (PR 4 of 5) - No freshness field (T1.5 propagation, PR 2 of 5) - No cache key bump: the new field is empty-string default, existing cached responses continue to deserialize cleanly Verified - make generate clean - npm run typecheck + typecheck:api clean - tests/resilience-dimension-scorers.test.mts all passing (existing + new) - tests/resilience-*.test.mts + test:data suite passing (4361 tests) - npm run lint exits 0 * fix(resilience): normalize cached score responses on read (#2959 P2) Greptile P2 finding on PR #2959: cachedFetchJson and getCachedResilienceScores return pre-change payloads verbatim, so a resilience:score:v7 entry written before this PR lands lacks the imputationClass field. Downstream consumers that read dim.imputationClass get undefined for up to 6 hours until the cache TTL expires. Fix: add normalizeResilienceScoreResponse helper that defaults missing optional fields in place and apply it at both read sites. Defaults imputationClass to empty string, matching the proto3 default for the new imputation_class field. - ensureResilienceScoreCached applies the normalizer after cachedFetchJson returns. - getCachedResilienceScores applies it after each successful JSON.parse on the pipeline result. - Two new test cases: stale payload without imputationClass gets defaulted, present values are preserved. - Not bumping the cache key: stale-read defaults are safe, the key bump would invalidate every cached score for a 6-hour cold-start cycle. The normalizer is extensible when PR #2961 adds freshness to the same payload. P3 finding (broken docs reference) verified invalid: the proto comment points to docs/methodology/country-resilience-index.mdx, which IS the current file. The .md predecessor was renamed in PR #2945 (T1.3 methodology doc promotion to CII parity). No change needed to the comment. * fix(resilience): bump score cache key v7 to v8, drop normalizer (#2959 P2) Second fixup for the Greptile P2 finding on #2959. The previous fixup ( |
||
|
Elie Habib
|
7da202c25d |
Phase 1 PR1: RegionalSnapshot proto + RPC handler (#2951)
* feat(intelligence): add RegionalSnapshot proto definition
Defines the canonical RegionalSnapshot wire format for Phase 1 of the
Regional Intelligence Model. Mirrors the TypeScript contract in
shared/regions.types.d.ts that Phase 0 landed with.
New proto file: proto/worldmonitor/intelligence/v1/get_regional_snapshot.proto
Messages:
- RegionalSnapshot (13 top-level fields matching the spec)
- SnapshotMeta (11 fields including snapshot_id, narrative_provider,
narrative_model, trigger_reason, snapshot_confidence, missing_inputs,
stale_inputs, valid_until, versions)
- RegimeState (label + transition history)
- BalanceVector (7 axes: 4 pressures + 3 buffers + net_balance + decomposed
drivers)
- BalanceDriver (axis, magnitude, evidence_ids, orientation)
- ActorState (leverage_score, role, domains, delta, evidence_ids)
- LeverageEdge (actor-to-actor directed influence)
- ScenarioSet + ScenarioLane (per-horizon distribution normalizing to 1.0)
- TransmissionPath (typed fields: severity, confidence, latency_hours,
magnitude range, asset class, template provenance)
- TriggerLadder + Trigger + TriggerThreshold (structured operator/value/
window/baseline)
- MobilityState + AirspaceStatus + FlightCorridorStress + AirportNodeStatus
- EvidenceItem (typed origin for the trust trail)
- RegionalNarrative + NarrativeSection (LLM-synthesized text with
evidence_ids on every section)
RPC: GetRegionalSnapshot(GetRegionalSnapshotRequest) -> GetRegionalSnapshotResponse
- GET /api/intelligence/v1/get-regional-snapshot
- region_id validated as lowercase kebab via buf.validate regex
- No other parameters; the handler reads canonical state
Generated code committed alongside:
- src/generated/client/worldmonitor/intelligence/v1/service_client.ts
- src/generated/server/worldmonitor/intelligence/v1/service_server.ts
- docs/api/IntelligenceService.openapi.{json,yaml}
The generated TypeScript types use camelCase per standard buf codegen, while
Phase 0 persists snapshots in Redis using the snake_case shape from
shared/regions.types.d.ts. The handler lands in a follow-up commit with a
localized snake_case -> camelCase adapter so Phase 0 code stays frozen.
Spec: docs/internal/pro-regional-intelligence-upgrade.md
* feat(intelligence): get-regional-snapshot RPC handler
Reads canonical persisted RegionalSnapshot for a region via the two-hop
lookup pattern established by the Phase 0 persist layer:
1. GET intelligence:snapshot:v1:{region}:latest -> snapshot_id
2. GET intelligence:snapshot-by-id:v1:{snapshot_id} -> full snapshot JSON
Returns empty response (snapshot omitted) when:
- No latest pointer exists (seed has never run or unknown region)
- Latest pointer references a pruned or TTL-expired snapshot
- Snapshot JSON is malformed
The handler does NOT recompute on miss. One writer (the seed bundle),
canonical reads. Matches the architecture commitment in the spec.
Includes a full snake_case -> camelCase adapter so the persisted Phase 0
shape (shared/regions.types.d.ts) maps cleanly onto the camelCase proto
wire format generated by buf. The adapter is the single bridge between
the two shapes; Phase 0 code stays frozen. Adapter handles every nested
message: SnapshotMeta, RegimeState, BalanceVector (+pressures/buffers
drivers), ActorState, LeverageEdge, ScenarioSet (+lanes +transmissions),
TransmissionPath, TriggerLadder (+triggers +thresholds), MobilityState
(+airspace +flight corridors +airports), EvidenceItem, RegionalNarrative
(+5 sections +watch items).
Wiring:
- Registered on intelligenceHandler in handler.ts
- Added to PREMIUM_RPC_PATHS (src/shared/premium-paths.ts) so the
gateway enforces Pro subscription or API key
- Added to RPC_CACHE_TIER with 'slow' tier (300s browser, 1800s edge)
matching similar premium intelligence RPCs
Not in this PR:
- LLM narrative generator (follow-up PR2, wires into snapshot writer)
- RegionalIntelligenceBoard panel UI (follow-up PR3)
- ENDPOINT_ENTITLEMENTS tier-specific enforcement (PREMIUM_RPC_PATHS
alone is the Pro gate; only stock-analysis endpoints currently use
tier-specific enforcement)
* test(intelligence): unit tests for get-regional-snapshot adapter + structural checks
29 tests across 5 suites covering:
adaptSnapshot (18 tests): real unit tests of the snake_case -> camelCase
adapter with synthetic persisted snapshots. Covers every nested message
(SnapshotMeta, RegimeState, BalanceVector with 7 axes + decomposed drivers,
ActorState, LeverageEdge, ScenarioSet with nested lanes and transmissions,
TriggerLadder with all 3 buckets + TriggerThreshold, MobilityState with
airspace/flights/airports, EvidenceItem, RegionalNarrative with all 5
sections + watch_items). Also asserts empty-default behavior when
nested fields are missing.
Handler structural checks (8 tests): validates import of getCachedJson,
canonical key prefixes, two-hop lookup ordering, empty-response fallbacks
on missing pointer or malformed snapshot, and export signature matching
the service interface.
Registration (2 tests): confirms getRegionalSnapshot is imported and
registered on the intelligenceHandler object.
Security wiring (2 tests): confirms the endpoint is in PREMIUM_RPC_PATHS
and RPC_CACHE_TIER with 'slow' tier.
Proto definition (3 tests): confirms the RPC method declaration, region_id
validation regex, RegionalSnapshot top-level field layout, and
BalanceVector 7-axis declaration.
* fix(intelligence): address Greptile P2 review findings on #2951
Two P2 findings from Greptile on the RegionalSnapshot proto+RPC PR.
1) region_id regex permitted trailing and consecutive hyphens
Old: ^[a-z][a-z0-9-]*$ — accepted "mena-", "east-asia-", "foo--bar"
New: ^[a-z][a-z0-9]*(-[a-z0-9]+)*$ — strict kebab-case, every hyphen must be
followed by at least one alphanumeric character. Regenerated openapi JSON/YAML
via `make generate`. Test assertion updated to match.
2) RPC_CACHE_TIER entry looked like dead code for premium paths
Greptile flagged that `isPremium` short-circuits the tier lookup to
'slow-browser' before RPC_CACHE_TIER is consulted, so the entry is never read
at runtime. Kept the entry because `tests/route-cache-tier.test.mjs` enforces
a parity contract requiring every generated GET route to have an explicit
tier. Added a NOTE comment in gateway.ts explaining the policy, and updated
the security-wiring test with a rationale comment so future maintainers know
the entry is intentional documentation, not a stale wire.
|
||
|
Elie Habib
|
46c35e6073 | feat(breadth): add market breadth history chart (#2932) | ||
|
Elie Habib
|
55c9c36de2 |
feat(stocks): add insider transaction tracking to stock analysis panel (#2928)
* feat(stocks): add insider transaction tracking to stock analysis panel Shows 6-month insider buy/sell activity from Finnhub: total buys, sells, net value, and recent named-exec transactions. Gracefully skips when FINNHUB_API_KEY is unavailable. * fix: add cache tier entry for get-insider-transactions route * fix(stocks): add insider RPC to premium paths + fix empty/stale states * fix(stocks): add insider RPC to premium paths + fix empty/stale states - Add /api/market/v1/get-insider-transactions to PREMIUM_RPC_PATHS - Return unavailable:false with empty transactions when Finnhub has no data (panel shows "No insider transactions" instead of "unavailable") - Mark stale insider data on refresh failures to avoid showing outdated info - Update test to match new empty-data behavior * fix(stocks): unblock stock-analysis render and surface exercise-only insider activity - loadStockAnalysis no longer awaits loadInsiderDataForPanel before panel.renderAnalyses. The insider fetch now fires in parallel after the primary render at both the cached-snapshot and live-fetch call sites. When insider data arrives, loadInsiderDataForPanel re-renders the panel so the section fills in asynchronously without holding up the analyst report on a secondary Finnhub RPC. - Add transaction code 'M' (exercise / conversion of derivative) to the allowed set in get-insider-transactions so symbols whose only recent Form 4 activity is option/RSU exercises no longer appear as "No insider transactions in the last 6 months". Exercises do not contribute to buys/sells dollar totals because transactionPrice is the strike price, not a market transaction. - Panel table now uses a neutral (dim) color for non-buy/non-sell rows (M rows) instead of the buy/sell green/red binary. - Tests cover: exercise-only activity producing non-empty transactions with zero buys/sells, and blended P/S/M activity preserving all three rows. * fix(stocks): prevent cached insider fetch from clobbering live render - Cached-path insider enrichment only runs when no live fetch is coming - Added generation counter to guard against concurrent loadStockAnalysis calls - Stale insider fetches now no-op instead of reverting panel state * fix(stocks): hide transient insider-unavailable flash and zero out strike-derived values - renderInsiderSection returns empty string when insider data is not yet fetched, so the transient "Insider data unavailable" card no longer flashes on initial render before the RPC completes - Exercise rows (code M) now carry value: 0 on the server and render a dash placeholder in the Value cell, matching how the buy/sell totals already exclude strike-derived dollar amounts * fix(stocks): exclude non-market Form 4 codes (A/D/F) from insider buy/sell totals Form 4 codes A (grant/award), D (disposition to issuer), and F (tax/exercise payment) are not open-market trades and should not drive insider conviction totals. Only P (open-market purchase) and S (open-market sale) now feed the buy/sell aggregates. A/D/F rows are still surfaced in the transaction list alongside M (exercise) with value zeroed out so the panel does not look empty. |
||
|
Elie Habib
|
2b189b77b6 |
feat(stocks): add dividend growth analysis to stock analysis panel (#2927)
* feat(stocks): add dividend growth analysis to stock analysis panel Shows yield, 5Y CAGR, frequency (quarterly/monthly/annual), payout ratio, and ex-dividend date. Hidden for non-dividend stocks. Data from Yahoo Finance dividend history. * fix(stocks): bump cache key + fix partial-year CAGR + remove misleading avg yield * fix(stocks): hydrate payout ratio, drop dead five-year yield, currency-aware dividend rate - Add fetchPayoutRatio helper that calls Yahoo quoteSummary's summaryDetail module in parallel with the dividend chart fetch and returns the raw 0-1 payout ratio (or undefined when missing/non-positive). The chart endpoint alone never returns payoutRatio, which is why it was hardcoded to 0. - Make payout_ratio optional in proto and DividendProfile so a missing value is undefined instead of 0; remove five_year_avg_dividend_yield entirely (proto reserved 51) since it was always returned as 0 and never wired up. - StockAnalysisPanel.renderDividendProfile now omits the Payout Ratio cell unless the value is present and > 0, formats it as (raw * 100).toFixed(1)%, and renders the dividend rate via Intl.NumberFormat with item.currency so EUR/GBP/JPY tickers no longer get a hardcoded "$" prefix. - Bump live cache key v2 -> v3 so any cached snapshots persisted with the old shape are refetched once. - Tests cover: payoutRatio populated from summaryDetail, payoutRatio undefined when summaryDetail returns 500 or raw=0, dividend response shape no longer contains fiveYearAvgDividendYield. * fix(stocks): bump persisted history store to v3 to rotate pre-PR snapshots Live analyze-stock cache was already bumped to v3, but the persisted history store (premium-stock-store.ts) still used v2 keys for index/item lookups. Pre-PR snapshots without the new dividend fields could pass the age-only freshness check and suppress a live refetch, leaving the new dividend section missing for up to 15 minutes. Bumping the persisted store keys to v3 makes old snapshots invisible. The data loader sees an empty history, triggers a live refetch, and repopulates under the new v3 keys. Old v2 keys expire via TTL. * fix(stocks): compute dividend CAGR correctly for quarterly/semiannual/annual payers Previously computeDividendCagr() required at least 10 distinct dividend months for a year to count as "full", which excluded every non-monthly dividend payer (quarterly = 4 months, semiannual = 2, annual = 1). CAGR therefore collapsed to 0/N/A for most ordinary dividend stocks. The new check uses calendar position: any year strictly earlier than the current calendar year is treated as complete, and the current in-progress year is excluded to avoid penalizing stocks whose next payment has not yet landed. * test(stocks): pass analystData to buildAnalysisResponse after rebase onto #2926 |
||
|
Elie Habib
|
889fa62849 |
feat(stocks): add analyst consensus + price targets to stock analysis panel (#2926)
* feat(stocks): add analyst consensus + price targets to stock analysis panel Shows recommendation trend (strongBuy/buy/hold/sell), price target range (high/low/median vs current), and recent upgrade/downgrade actions with firm names. Data from Yahoo Finance quoteSummary. * chore: regenerate proto types and OpenAPI docs * fix(stocks): fallback median to mean + use stock currency for price targets * fix(stocks): drop fake $0 price targets and force refetch for pre-rollout snapshots - Make PriceTarget high/low/mean/median/current optional in proto so partial Yahoo financialData payloads stop materializing as $0.00 cells in the panel. - fetchYahooAnalystData now passes undefined (via optionalPositive) when a field is missing or non-positive, instead of coercing to 0. - StockAnalysisPanel.renderPriceTarget skips Low/High cells entirely when the upstream value is missing and falls back to a Median + Analysts view. - Add field-presence freshness check in stock-analysis-history: snapshots written before the analyst-revisions rollout (no analystConsensus and no priceTarget) are now classified as stale even when their generatedAt is inside the freshness window, so the data loader forces a live refetch. - Tests cover undefined targets path, missing financialData path, and the three field-presence freshness branches. * fix(stocks): preserve fresh snapshots on partial refetch + accept median-only targets - loadStockAnalysis now merges still-fresh cached symbols with refetched live results so a partial refetch does not shrink the rendered watchlist - renderAnalystConsensus accepts median-only price targets (not just mean) |
||
|
Elie Habib
|
a742537ae5 |
feat(supply-chain): Sprint D — GetSectorDependency RPC + vendor route-intelligence API + webhooks (#2905)
* feat(supply-chain): Sprint D — GetSectorDependency RPC + vendor route-intelligence API + webhooks * fix(supply-chain): move bypass-corridors + chokepoint-registry to server/_shared to fix api/ boundary violations * fix(supply-chain): webhooks — persist secret, fix sub-resource routing, add ownership check * fix(supply-chain): address PR #2905 review findings - Use SHA-256(apiKey) for ownerTag instead of last-12-chars (unambiguous ownership) - Implement GET /api/v2/shipping/webhooks list route via per-owner Redis Set index - Tighten SSRF: https-only, expanded metadata hostname blocklist, document DNS rebinding edge-runtime limitation - Fix get-sector-dependency.ts stale src/config/ imports → server/_shared/ (Greptile P1) * fix(supply-chain): getSectorDependency returns blank primaryChokepointId for landlocked countries computeExposures() previously mapped over all of CHOKEPOINT_REGISTRY even when nearestRouteIds was empty, producing a full array of score-0 entries in registry insertion order. The caller's exposures[0] then picked the first registry entry (Suez) as the "primary" chokepoint despite primaryChokepointExposure = 0. LI, AD, SM, BT and other landlocked countries were all silently assigned a fake chokepoint. Fix: guard at the top of computeExposures() -- return [] when input is empty so primaryChokepointId stays '' and primaryChokepointExposure stays 0. |
||
|
Elie Habib
|
ce30a48664 |
feat(resilience): add rankStable flag to ranking items (#2879)
* feat(resilience): add rankStable flag to ranking items Countries with score interval width <= 8 (p95-p05) are flagged as rankStable=true, indicating robust ranking under weight perturbation. Read from batch-computed intervals in Redis. * fix(resilience): guard inverted intervals + scope fetch to scored countries 1. isRankStable rejects negative width (malformed p05 > p95) 2. fetchIntervals scoped to cachedScores.keys() instead of all countries * fix(resilience): raw key read for intervals + bump ranking cache to v8 * fix(resilience): remove duplicate ScoreInterval interface after rebase ScoreInterval is now generated in service_server.ts (from PR #2877). Remove the local duplicate and re-export the generated type. |
||
|
Elie Habib
|
1af73975b9 |
feat(energy): SPR policy classification layer (#2881)
* feat(energy): add SPR policy classification layer with 66-country registry Static JSON registry classifying strategic petroleum reserve regimes for 66 countries (all IEA members + major producers/consumers). Integrates into energy profile handler, shock model limitations, analyst context, spine seeder, and CDP UI. - scripts/data/spr-policies.json: 66-entry registry with regime, source, asOf - scripts/seed-spr-policies.mjs: seeder following chokepoint-baselines pattern - Proto fields 51-59 on GetCountryEnergyProfileResponse - Handler reads SPR registry from Redis, populates proto fields - Shock model adds fuel-mode-gated SPR limitations for non-IEA gov SPR - Analyst context refactored to accumulator pattern (IEA + SPR parts) - CDP UI: SPR badge for non-IEA government_spr, muted text for spare_capacity - Spine integration: SPR fields in shockInputs + hasSprPolicy coverage flag - Cache keys, health, bootstrap, seed-health registrations - Tests: registry shape, ISO2, regime enum, required entries, no estimatedFillPct * fix(energy): remove SPR from bootstrap (server-only); narrow SPR hasAny gate to renderable regimes * feat(energy): render "no known SPR" risk note for countries with regime=none * fix(energy): human-readable SPR regime labels; parallelize spine+registry reads in analyst |
||
|
Elie Habib
|
0a1b74a9b2 |
feat(resilience): add score confidence intervals via batch Monte Carlo (#2877)
* feat(resilience): add score confidence intervals via batch Monte Carlo
Weekly cron perturbs domain weights ±10% across 100 draws per country,
stores p05/p95 in Redis. Score handler reads intervals and includes
them in the API response as ScoreInterval { p05, p95 }.
Proto field 14 (score_interval) added to GetResilienceScoreResponse.
* chore: regenerate proto types and OpenAPI docs for ScoreInterval
* fix(resilience): add seed-meta + lock + fix interval cache + percentile formula
1. Write seed-meta:resilience:intervals for health monitoring
2. Add distributed lock to prevent concurrent cron overlap
3. Move scoreInterval read outside 6h score cache boundary
4. Fix percentile index from floor to ceil-1 (nearest-rank)
* fix(health): add resilience:intervals to health + seed-health registries
* fix(seed): skip seed-meta on no-op runs + register intervals in health check
|
||
|
Elie Habib
|
23ed4eba44 |
fix(supply-chain): address all code review findings from PR #2873 (#2878)
* fix(supply-chain): address all code review findings from PR #2873 - Rename costIncreasePct → supplyDeficitPct (semantic correction) - Add primaryChokepointWarRiskTier to GetBypassOptionsResponse - Consolidate ThreatLevel/threatLevelToWarRiskTier into _insurance-tier.ts - Replace inline CpEntry/ChokepointStatusCacheEntry with ChokepointInfo - Add outer cachedFetchJson wrapper (3 serial Redis reads → 1 on warm path) - Add hs2 validation guard matching sibling handler pattern - Extract CHOKEPOINT_STATUS_KEY constant; eliminate string literal duplication - Add SCORE_RISK_WEIGHT/SCORE_COST_WEIGHT named constants; clamp liveScore ≥ 0 - Add Math.max(0,...) to liveScore for sub-1.0 cost multiplier corridors - Fix closurePct: req.closurePct ?? 100 (was || which falsy-coalesced zero) - Type fetchBypassOptions cargoType as CargoType (was implicit string) - Add exhaustiveness check to threatLevelToInsurancePremiumBps switch - Move TIER_RANK to module level in _insurance-tier.ts - Update WIDGET_PRO_SYSTEM_PROMPT with both new PRO RPCs * fix(supply-chain): fix supplyDeficitPct averaging and coverageDays sentinel - Remove .filter(d > 0) from productDeficits: zero-deficit products have demand and must stay in the denominator to avoid overstating the average - Clamp coverageDays = Math.max(0, effectiveCoverDays): prevents -1 net-exporter sentinel from leaking into the public API response - Update proto comment: document 0 for net exporters - Add test assertions for both contracts * chore(api-docs): regenerate OpenAPI docs for coverage_days comment update * refactor(supply-chain): use CHOKEPOINT_STATUS_KEY in chokepoint-status writer The key was extracted to cache-keys.ts in the previous commit but the primary writer (getChokepointStatus) and BOOTSTRAP_CACHE_KEYS still embedded the raw string literal. Import the constant at both sites to complete the refactor. * test: update supply-chain-v2 assertions for CHOKEPOINT_STATUS_KEY refactor Handler now imports CHOKEPOINT_STATUS_KEY as REDIS_CACHE_KEY from cache-keys.ts rather than defining a local constant. BOOTSTRAP_CACHE_KEYS also references the constant. Update source-string assertions to match the new patterns. * fix: keep BOOTSTRAP_CACHE_KEYS.chokepoints as string literal bootstrap.test.mjs enforces string-literal values in BOOTSTRAP_CACHE_KEYS via regex. CHOKEPOINT_STATUS_KEY is used in handler imports and is the primary dedup win; the static registry entry stays as-is per test contract. |
||
|
Elie Habib
|
bd07829518 |
feat(supply-chain): Sprint 2 — bypass corridor intelligence + cost shock engine (#2873)
* feat(supply-chain): Sprint 2 — bypass corridor intelligence + cost shock engine - src/config/bypass-corridors.ts: ~40 bypass corridors for all 13 chokepoints - server/supply-chain/v1/get-bypass-options.ts: PRO-gated RPC, live bypass scoring from chokepoint status cache - server/supply-chain/v1/get-country-cost-shock.ts: PRO-gated RPC, war risk premium BPS + energy coverage days (HS 27) - server/supply-chain/v1/_insurance-tier.ts: pure function, Lloyd's JWC threat → premium BPS - gateway.ts + premium-paths.ts: registered both RPCs as slow-browser + PRO-gated - src/services/supply-chain/index.ts: fetchBypassOptions + fetchCountryCostShock client methods - proto: GetBypassOptions + GetCountryCostShock messages + service registrations - tests/supply-chain-sprint2.test.mjs: 61 tests covering all new components Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> * fix(cost-shock): call computeEnergyShockScenario directly instead of reading wrong cache key The old code read from `energy:shock:${iso2}:${chokepointId}:v1` which never matches the actual v2 cache key written by compute-energy-shock.ts. Fix by calling computeEnergyShockScenario() directly (it handles v2 caching internally) and mapping effectiveCoverDays + crude product deficitPct to the response fields. * fix(cost-shock): average refined product deficitPct instead of looking for non-existent 'crude' product --------- Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> |
||
|
Elie Habib
|
6e401ad02f |
feat(supply-chain): Global Shipping Intelligence — Sprint 0 + Sprint 1 (#2870)
* feat(supply-chain): Sprint 0 — chokepoint registry, HS2 sectors, war_risk_tier
- src/config/chokepoint-registry.ts: single source of truth for all 13
canonical chokepoints with displayName, relayName, portwatchName,
corridorRiskName, baselineId, shockModelSupported, routeIds, lat/lon
- src/config/hs2-sectors.ts: static dictionary for all 99 HS2 chapters
with category, shockModelSupported (true only for HS27), cargoType
- server/worldmonitor/supply-chain/v1/_chokepoint-ids.ts: migrated to
derive CANONICAL_CHOKEPOINTS from chokepoint-registry; no data duplication
- src/config/geo.ts + src/types/index.ts: added chokepointId field to
StrategicWaterway interface and all 13 STRATEGIC_WATERWAYS entries
- src/components/MapPopup.ts: switched chokepoint matching from fragile
name.toLowerCase() to direct chokepointId === id comparison
- server/worldmonitor/intelligence/v1/_shock-compute.ts: migrated from old
IDs (hormuz/malacca/babelm) to canonical IDs (hormuz_strait/malacca_strait/
bab_el_mandeb); same for CHOKEPOINT_LNG_EXPOSURE
- proto/worldmonitor/supply_chain/v1/supply_chain_data.proto: added
WarRiskTier enum + war_risk_tier field (field 16) on ChokepointInfo
- get-chokepoint-status.ts: populates warRiskTier from ChokepointConfig.threatLevel
via new threatLevelToWarRiskTier() helper (FREE field, no PRO gate)
* feat(supply-chain): Sprint 1 — country chokepoint exposure index + sector ring
S1.1: scripts/shared/country-port-clusters.json
~130 country → {nearestRouteIds, coastSide} mappings derived from trade route
waypoints; covers all 6 seeded Comtrade reporters plus major trading nations.
S1.2: scripts/seed-hs2-chokepoint-exposure.mjs
Daily cron seeder. Pure computation — reads country-port-clusters.json,
scores each country against CHOKEPOINT_REGISTRY route overlap, writes
supply-chain:exposure:{iso2}:{hs2}:v1 keys + seed-meta (24h TTL).
S1.3: RPC get-country-chokepoint-index (PRO-gated, request-varying)
- proto: GetCountryChokepointIndexRequest/Response + ChokepointExposureEntry
- handler: isCallerPremium gate; cachedFetchJson 24h; on-demand for any iso2
- cache-keys.ts: CHOKEPOINT_EXPOSURE_KEY(iso2, hs2) constant
- health.js: chokepointExposure SEED_META entry (48h threshold)
- gateway.ts: slow-browser cache tier
- service client: fetchCountryChokepointIndex() exported
S1.4: Chokepoint popup HS2 sector ring chart (PRO-gated)
Static trade-sector breakdown (IEA/UNCTAD estimates) per 9 major chokepoints.
SVG donut ring + legend shown for PRO users; blurred lockout + gate-hit
analytics for free users. Wired into renderWaterwayPopup().
🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.com/claude-code) + Compound Engineering v2.49.0
Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
* fix(tests): update energy-shock-v2 tests to use canonical chokepoint IDs
CHOKEPOINT_EXPOSURE and CHOKEPOINT_LNG_EXPOSURE keys were migrated from
short IDs (hormuz, malacca, babelm) to canonical registry IDs
(hormuz_strait, malacca_strait, bab_el_mandeb) in Sprint 0.
Test fixtures were not updated at the time; fix them now.
* fix(tests): update energy-shock-seed chokepoint ID to canonical form
VALID_CHOKEPOINTS changed to canonical IDs in Sprint 0; the seed test
that checks valid IDs was not updated alongside it.
* fix(cache-keys): reword JSDoc comment to avoid confusing bootstrap test regex
The comment "NOT in BOOTSTRAP_CACHE_KEYS" caused the bootstrap.test.mjs
regex to match the comment rather than the actual export declaration,
resulting in 0 entries found. Rephrase to "excluded from bootstrap".
* fix(supply-chain): address P1 review findings for chokepoint exposure index
- Add get-country-chokepoint-index to PREMIUM_RPC_PATHS (CDN bypass)
- Validate iso2/hs2 params before Redis key construction (cache injection)
- Fix seeder TTL to 172800s (2× interval) and extend TTL on skipped lock
- Fix CHOKEPOINT_EXPOSURE_SEED_META_KEY to match seeder write key
- Render placeholder sectors behind blur gate (DOM data leakage)
- Document get-country-chokepoint-index in widget agent system prompts
* fix(lint): resolve Biome CI failures
- Add biome.json overrides to silence noVar in HTML inline scripts,
disable linting for public/ vendor/build artifacts and pro-test/
- Remove duplicate NG and MW keys from country-port-clusters.json
- Use import attributes (with) instead of deprecated assert syntax
* fix(build): drop JSON import attribute — esbuild rejects `with` syntax
---------
Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
|
||
|
Elie Habib
|
cffdcf8052 |
fix(energy): V6 review findings (7 issues across 5 PRs) (#2861)
* fix(energy): V6 review findings — flow availability, ember proto bool, LNG stale/blocking, fixture accuracy
- Fix 1: Only show "Flow data unavailable" for chokepoints with PortWatch
flow coverage (hormuz, malacca, suez, bab_el_mandeb), not all corridors
- Fix 2: Correct proto comment on data_available field 9 to document
gas mode and both mode behavior
- Fix 3: Add ember_available bool field 50 to GetCountryEnergyProfile proto,
set server-side from spine.electricity or direct Ember key fallback
- Fix 4: Ember fallback reads energy:ember:v1:{code} when spine exists but
has no electricity block (or fossilShare is absent)
- Fix 6: Add IEA upstream fixture matching actual API response shape,
with golden test parsing through seeder parseRecord/buildIndex
- Fix 7: Add PortWatch ArcGIS fixture with all attributes.* fields used
by buildHistory, with golden test validating parsed output
* fix(energy): add emberAvailable to energy gate; use real buildHistory in portwatch test
* fix(energy): add Ember render block to renderEnergyProfile for Ember-only countries
* chore: regenerate OpenAPI specs after proto comment update
|
||
|
Elie Habib
|
75e9c22dd3 |
feat(resilience): populate dataVersion field from seed-meta timestamp (#2865)
* feat(resilience): populate dataVersion field from seed-meta timestamp Sets dataVersion to the ISO date of the most recent static bundle seed, making the data vintage visible to API consumers. * fix(resilience): bump score cache to v7 for dataVersion field addition |
||
|
Elie Habib
|
c35b3618e0 |
feat(fires): flag possible explosions in satellite thermal detections (#2850)
* feat(fires): flag possible explosions in satellite thermal detections Adds possibleExplosion field (FRP >80 MW + brightness >380 K) to fire detections, surfacing non-fire thermal signatures that may indicate strikes or explosions. Seeder computes the flag, panel shows inline badge per region and summary alert when explosions are detected. * refactor(fires): extract brightness/frp locals to avoid double-parse |
||
|
Elie Habib
|
c826003e5f |
feat(energy): LNG and gas shock path (V5-4) (#2822)
* feat(energy): LNG gas shock path with GIE storage buffer (V5-4) * fix(energy): gas path zero-LNG handling, oil gate bypass, live flow scaling, coverage semantics * fix(energy): gas-only coverage respects degraded; zero oil fields in gas mode |
||
|
Elie Habib
|
f53c05599a |
feat(resilience): baseline vs stress scoring engine (#2821)
* feat(resilience): baseline vs stress scoring engine Splits the resilience index into structural capacity (baselineScore) and active disruption (stressScore) using the dimension type tags from RESILIENCE_DIMENSION_TYPES (baseline/stress/mixed). overallScore = baselineScore * (1 - stressFactor) where stressFactor is clamped to [0, 0.5]. Mixed dimensions contribute to both scores. Proto fields 10-12 added (baseline_score, stress_score, stress_factor). Widget updated to display baseline/stress breakdown. Cache keys bumped v4 -> v5 for atomic rollout. * fix(resilience): bump history key to v2 for baseline/stress formula change The overallScore formula changed from domain-weighted-sum to baselineScore * (1 - stressFactor). Old history entries are incomparable, causing fake change30d drops of -20 to -30 points. Versioned history key starts a clean series. |
||
|
Elie Habib
|
80b24d8686 |
feat(energy): shock model v2 — live flow ratios, coverage, limitations (V5-3) (#2810)
* feat(energy): shock model v2 — live flow ratios, coverage, limitations (V5-3) Replace static CHOKEPOINT_EXPOSURE multipliers with live PortWatch flow ratios from energy:chokepoint-flows:v1. Add proto fields 11-19: per-source coverage flags (jodi_oil_coverage, comtrade_coverage, iea_stocks_coverage, portwatch_coverage), coverage_level, limitations[], degraded, chokepoint_confidence, live_flow_ratio. Expand ISO2_TO_COMTRADE from 6 to 150+ countries via _comtrade-reporters.ts. Partial-coverage path proxies Gulf share at 40%. Unsupported countries return structured metadata instead of opaque string. data_available (field 9) preserved for backward compat. * fix(energy): correct chokepoint-flows key shape in shock handler (V5-3) energy:chokepoint-flows:v1 is a flat object keyed by canonicalId (hormuz_strait, bab_el_mandeb, suez, malacca_strait), not an object with a chokepoints[] array. The wrong shape caused degraded=true and liveFlowRatio=null for every request, silently falling back to static CHOKEPOINT_EXPOSURE multipliers even when live PortWatch data was available. Fixes: ChokepointEntry interface, typecast, cpEntry lookup, degraded check. * fix(energy): 4 review fixes for shock v2 handler (V5-3) 1. Cache key v1→v2: response shape changed (fields 11-19 added), old v1 cache entries would be served without new coverage fields. 2. IEA unknown vs zero: when ieaStocksCoverage=false, assessment now shows "IEA cover: unknown" instead of "0 days" to avoid conflating missing data with real zero stock. 3. liveFlowRatio 0.0 truthiness: changed `if (liveFlowRatio)` to `if (liveFlowRatio != null)` — a blocked chokepoint (ratio=0.0) is now shown, not hidden as "no live data". 4. Badge cleared on request start: coverageBadge is now reset before each shock compute request so a failed request doesn't leave the previous result's badge visible. * fix(energy): 3 remaining review fixes for shock v2 (V5-3) 1. Flow column: gate on portwatchCoverage (bool) not liveFlowRatio!=null — proto double defaults to 0, so null-check was always true and degraded responses showed a misleading "Flow: 0%" column. 2. Degraded cache TTL: 5min when degraded=true, 1h when live data present — limits how long stale degraded state persists after PortWatch recovers. 3. IEA anomaly cover days: zero daysOfCover when ieaStocksCoverage=false so anomalous IEA data no longer contributes to effectiveCoverDays; panel IEA cover row also gated on ieaStocksCoverage. * fix(energy): netExporter from anomalous IEA + zero-days panel display (V5-3) 1. netExporter: gated on ieaStocksCoverage — anomalous IEA rows no longer drive the net-exporter assessment branch when coverage=false. 2. Panel IEA cover: removed effectiveCoverDays>0 guard so a real zero (reserves exhausted under scenario) renders as "0 days" instead of being silently hidden as if there were no IEA data. * fix(energy): handle net-exporter sentinel (-1) in IEA cover panel row * fix(energy): NaN guard on flowRatio; optional live_flow_ratio; coverageLevel includes IEA+degraded * fix(energy): narrow Gulf-share proxy to Comtrade-only; NaN guard computeGulfShare; EMPTY liveFlowRatio undefined * fix(energy): tighten ieaStocksCoverage null guard; cache key varies by degraded state * fix(energy): harden IEA/PortWatch input validation; reduce shock cache TTL * fix(energy): add null narrowing for daysOfCover to satisfy strict TS |
||
|
Elie Habib
|
1937dbf844 |
feat(portwatch): Maritime Activity section in CountryDeepDivePanel (PR C) (#2805)
* feat(portwatch): Maritime Activity section in CountryDeepDivePanel (PR C) - Add get_country_port_activity.proto with PortActivityEntry + CountryPortActivityResponse messages - Register GetCountryPortActivity RPC in service.proto with HTTP GET /get-country-port-activity path - Run make generate to produce updated service_client.ts and service_server.ts - Implement get-country-port-activity.ts handler: countries guard, top-5 slice, trendDelta→tankerCallsPrev mapping - Register handler in intelligence handler.ts and gateway.ts slow cache tier - Add CountryPortActivityData interface and updateMaritimeActivity? method to CountryBriefPanel - Implement updateMaritimeActivity in CountryDeepDivePanel: table with 5 ports, anomaly badge, trend color, IMF PortWatch footer - Add getCountryPortActivity call in country-intel.ts with stale guard - Add maritime-activity CMD+K entry in commands.ts - 29 source-string assertions in tests/country-port-activity.test.mjs (all pass) Task: PR C * fix(portwatch): pass trendDelta directly, add runtime trend tests |
||
|
Elie Habib
|
2edcdeee06 |
refactor(resilience): remove Cronbach alpha, add imputationShare confidence (#2787)
* refactor(resilience): remove Cronbach alpha, add imputationShare confidence Remove cronbach_alpha from proto (field 5 reserved) and all response builders. Replace with imputationShare (field 9): the fraction of weighted score from imputed (not observed) data. lowConfidence now triggers on averageCoverage < 0.55 or imputationShare > 0.40, replacing the unstable Cronbach-based gate. weightedBlend() return type extended with observedWeight/imputedWeight for provenance tracking through the scoring pipeline. * fix(resilience): version cache key + fix IMF proxy imputation classification 1. Bump resilience score cache key to v2 to avoid serving stale cached responses missing imputationShare after deploy. 2. Add explicit `imputed` flag to WeightedMetric so proxy data (real IMF inflation with lower certaintyCoverage) is classified as observed, not imputed. Only synthetic absence-based scores count toward imputationShare. |
||
|
Elie Habib
|
a09f49ff9c |
feat(supply-chain): energy flow estimates per chokepoint (mb/d card row) (#2780)
* feat(supply-chain): energy flow estimates per chokepoint (mb/d card row) - Add FlowEstimate proto message + ChokepointInfo field 15; regenerate stubs - Add baselineId mapping to _chokepoint-ids.ts (7 of 13 chokepoints) - Add relayId to seed-chokepoint-baselines.mjs CHOKEPOINTS entries - New seed-chokepoint-flows.mjs: reads portwatch + baselines, computes 7d tanker avg vs 90d baseline, outputs flow_ratio and current_mbd; prefers DWT (capTanker) when available; flags disruption if last 3 days each below 0.85 threshold; writes energy:chokepoint-flows:v1 (TTL 3d) - get-chokepoint-status.ts: parallel-reads flows key, attaches flowEstimate - SupplyChainPanel: compact card gains mb/d row (red <85%, amber <95%) - 19 new unit tests for flow computation and seeder contract * fix(chokepoint-flows): base useDwt on 90d baseline window, not recent 7 days Zero recent capTanker is the disruption signal, not a reason to fall back to vessel counts. Switching metrics during peak disruption caused the seeder to report a higher (less accurate) flow estimate exactly when oil-flow collapse is most acute. useDwt is now locked to whether the baseline window has DWT data -- stable across disruption events. Adds regression test covering DWT-collapse scenario. * fix(chokepoint-flows): require majority DWT coverage in baseline before activating DWT mode capBaselineSum > 0 would activate DWT on a single non-zero day during partial data roll-out, pulling down the baseline average via zero-filled gaps. Now requires >= ceil(prev90.length / 2) days with DWT data. ArcGIS data is all-or-nothing per chokepoint in practice, so this guard catches edge cases without affecting normal operation. |
||
|
Elie Habib
|
190095ca89 |
feat(supply-chain): stacked vessel-type transit chart with 7d MA, DWT tab, zoom (#2777)
* feat(supply-chain): stacked vessel-type transit chart with 7d MA, DWT tab, zoom - Update TransitDayCount proto (fields 6-14): container, dry_bulk, general_cargo, roro, cap_* DWT capacity fields; regenerate TS types - Rewrite transit-chart.ts: 5-type stacked bar (container/dryBulk/ generalCargo/roro/tanker), 7d MA dashed overlay, Transit Calls / Trade Volume tab toggle, 1m/3m/6m zoom buttons, richer tooltip - SupplyChainPanel: enlarge chart placeholder min-height 120->200px * fix(transit-chart): stop control clicks bubbling + track source div in destroy - stopPropagation on controls container prevents tab/zoom button clicks from collapsing the chokepoint card - source div now tracked as this.source and cleaned up in destroy(), preventing duplicate attribution lines on repeated remounts * fix(transit-chart): import from generated client, reuse data in onMouseMove - Import TransitDayCount from generated client stub instead of server layer; keeps src/ imports within src/ - onMouseMove: reuse already-bound data array for MA computation instead of calling visibleData() again on every mouse event |
||
|
Elie Habib
|
b9b552cfcd |
feat(energy): product supply shock scenario RPC (Phase 4 PR C) (#2768)
* feat(energy): ComputeEnergyShockScenario RPC + country brief shock UI (Phase 4 PR C) Adds on-demand product supply shock scenario computation from JODI Oil, Comtrade and IEA stocks data. * fix(tests): add runtime + intelligence-client stubs to resilience harness ResilienceWidget imports @/services/runtime which dynamically imports @/services/widget-store. Without stubbing runtime, esbuild bundled the chain and failed on loadFromStorage not exported by the utils stub. * fix(energy): dataAvailable requires Comtrade data; gate shock widget on jodiOilAvailable - dataAvailable now requires both jodiOil and comtradeHasData, eliminating the contradiction of returning true with an "insufficient data" assessment - Collapsed redundant !hasComtradeData branch into the unified !dataAvailable guard - Gate renderShockScenarioWidget() behind data.jodiOilAvailable in CountryDeepDivePanel to avoid rendering a widget that will always return dataAvailable: false * fix(energy): zero-import hasData=false; extract pure shock-compute for real unit coverage - `totalImports === 0` in `computeGulfShare` now returns `hasData: false` so the handler correctly falls through to the "insufficient data" branch instead of treating empty Comtrade rows as usable Gulf-share data - Extract `clamp`, `computeGulfShare`, `computeEffectiveCoverDays`, `buildAssessment`, `GULF_PARTNER_CODES`, and `CHOKEPOINT_EXPOSURE` into `_shock-compute.ts` - Handler delegates pure computation to imported functions; `getGulfCrudeShare` still owns Redis I/O and calls `computeGulfShare(flows)` for the math - Tests now import the real functions via `.js` ESM extension; all 24 test cases exercise actual production logic (was previously reimplemented inline) * fix(energy): handle net-exporter in shock assessment; fix tautological chokepoint tests * fix(energy): move net-exporter branch before low-Gulf-share check in buildAssessment A net exporter with gulfCrudeShare < 0.1 (e.g. Norway) incorrectly received "low Gulf crude dependence" instead of "net oil exporter". Adds regression test to cover the ordering case. |
||
|
Elie Habib
|
e0dc630ed5 |
feat(energy): days of cover global view (Phase 4 PR B) (#2767)
* feat(energy): days of cover analysis key + EnergyComplexPanel oil stocks section - seed-iea-oil-stocks.mjs exports buildOilStocksAnalysis and writes energy:oil-stocks-analysis:v1 via afterPublish hook after main index - Rankings sorted by daysOfCover desc (net-exporters last), vsObligation, obligationMet, regional summaries (Europe/Asia-Pacific/North America) - EnergyComplexPanel.setOilStocksAnalysis() renders IEA member table with below-obligation badges, rank, days vs 90d obligation, regional summary rows - Health monitoring: seed-meta:energy:oil-stocks-analysis (42d maxStaleMin) - Gateway cache tier: static (monthly seed data) - 13 new tests covering sorting, exclusions, regional rollups, obligation logic * feat(energy): add proto + regenerate service for oil stocks analysis RPC - Add get_oil_stocks_analysis.proto with OilStocksAnalysisMember, OilStocksRegionalSummary sub-messages, and GetOilStocksAnalysisResponse - Use proto3 optional fields for nullable int32 (daysOfCover, vsObligation, avgDays, minDays) avoiding google.protobuf.wrappers complexity - Regenerate service_client.ts + service_server.ts via make generate - Update handler fallback and panel null-safety guards for optional fields - Regenerated OpenAPI docs include getOilStocksAnalysis endpoint * fix(energy): preserve oil-stocks-analysis TTL via extraKeys; fix seed-meta TTL to exceed health threshold - Move ANALYSIS_KEY into ANALYSIS_EXTRA_KEY in extraKeys so runSeed() extends its TTL on fetch failure or validation skip (was only written in afterPublish, leaving the key unprotected on the sad path) - afterPublish now writes only the seed-meta for ANALYSIS_KEY with a 50-day TTL (Math.max(86400*50, TTL_SECONDS)) — exceeds the health maxStaleMin threshold - Add optional metaTtlSeconds param to writeExtraKeyWithMeta() (backward-compat, defaults to existing 7-day value for all other callers) - Update health.js oilStocksAnalysis maxStaleMin from 42d to 50d to stay below the new seed-meta TTL and avoid false stale/missing reports * fix(energy): preserve seed-meta:oil-stocks-analysis TTL via extraKeys on seeder failure |