mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
main
24 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
34dfc9a451 |
fix(news): ground LLM surfaces on real RSS description end-to-end (#3370)
* feat(news/parser): extract RSS/Atom description for LLM grounding (U1)
Add description field to ParsedItem, extract from the first non-empty of
description/content:encoded (RSS) or summary/content (Atom), picking the
longest after HTML-strip + entity-decode + whitespace-normalize. Clip to
400 chars. Reject empty, <40 chars after strip, or normalize-equal to the
headline — downstream consumers fall back to the cleaned headline on '',
preserving current behavior for feeds without a description.
CDATA end is anchored to the closing tag so internal ]]> sequences do not
truncate the match. Preserves cached rss:feed:v1 row compatibility during
the 1h TTL bleed since the field is additive.
Part of fix: pipe RSS description end-to-end so LLM surfaces stop
hallucinating named actors (docs/plans/2026-04-24-001-...).
Covers R1, R7.
* feat(news/story-track): persist description on story:track:v1 HSET (U2)
Append description to the story:track:v1 HSET only when non-empty. Additive
— no key version bump. Old rows and rows from feeds without a description
return undefined on HGETALL, letting downstream readers fall back to the
cleaned headline (R6).
Extract buildStoryTrackHsetFields as a pure helper so the inclusion gate is
unit-testable without Redis.
Update the contract comment in cache-keys.ts so the next reader of the
schema sees description as an optional field.
Covers R2, R6.
* feat(proto): NewsItem.snippet + SummarizeArticleRequest.bodies (U3)
Add two additive proto fields so the article description can ride to every
LLM-adjacent consumer without a breaking change:
- NewsItem.snippet (field 12): RSS/Atom description, HTML-stripped,
≤400 chars, empty when unavailable. Wired on toProtoItem.
- SummarizeArticleRequest.bodies (field 8): optional article bodies
paired 1:1 with headlines for prompt grounding. Empty array is today's
headline-only behavior.
Regenerated TS client/server stubs and OpenAPI YAML/JSON via sebuf v0.11.1
(PATH=~/go/bin required — Homebrew's protoc-gen-openapiv3 is an older
pre-bundle-mode build that collides on duplicate emission).
Pre-emptive bodies:[] placeholders at the two existing SummarizeArticle
call sites in src/services/summarization.ts; U6 replaces them with real
article bodies once SummarizeArticle handler reads the field.
Covers R3, R5.
* feat(brief/digest): forward RSS description end-to-end through brief envelope (U4)
Digest accumulator reader (seed-digest-notifications.mjs::buildDigest) now
plumbs the optional `description` field off each story:track:v1 HGETALL into
the digest story object. The brief adapter (brief-compose.mjs::
digestStoryToUpstreamTopStory) prefers the real RSS description over the
cleaned headline; when the upstream row has no description (old rows in the
48h bleed, feeds that don't carry one), we fall back to the cleaned headline
so today behavior is preserved (R6).
This is the upstream half of the description cache path. U5 lands the LLM-
side grounding + cache-prefix bump so Gemini actually sees the article body
instead of hallucinating a named actor from the headline.
Covers R4 (upstream half), R6.
* feat(brief/llm): RSS grounding + sanitisation + 4 cache prefix bumps (U5)
The actual fix for the headline-only named-actor hallucination class:
Gemini 2.5 Flash now receives the real article body as grounding context,
so it paraphrases what the article says instead of filling role-label
headlines from parametric priors ("Iran's new supreme leader" → "Ali
Khamenei" was the 2026-04-24 reproduction; with grounding, it becomes
the actual article-named actor).
Changes:
- buildStoryDescriptionPrompt interpolates a `Context: <body>` line
between the metadata block and the "One editorial sentence" instruction
when description is non-empty AND not normalise-equal to the headline.
Clips to 400 chars as a second belt-and-braces after the U1 parser cap.
No Context line → identical prompt to pre-fix (R6 preserved).
- sanitizeStoryForPrompt extended to cover `description`. Closes the
asymmetry where whyMatters was sanitised and description wasn't —
untrusted RSS bodies now flow through the same injection-marker
neutraliser before prompt interpolation. generateStoryDescription wraps
the story in sanitizeStoryForPrompt before calling the builder,
matching generateWhyMatters.
- Four cache prefixes bumped atomically to evict pre-grounding rows:
scripts/lib/brief-llm.mjs:
brief:llm:description:v1 → v2 (Railway, description path)
brief:llm:whymatters:v2 → v3 (Railway, whyMatters fallback)
api/internal/brief-why-matters.ts:
brief:llm:whymatters:v6 → v7 (edge, primary)
brief:llm:whymatters:shadow:v4 → shadow:v5 (edge, shadow)
hashBriefStory already includes description in the 6-field material
(v5 contract) so identity naturally drifts; the prefix bump is the
belt-and-braces that guarantees a clean cold-start on first tick.
- Tests: 8 new + 2 prefix-match updates on tests/brief-llm.test.mjs.
Covers Context-line injection, empty/dup-of-headline rejection,
400-char clip, sanitisation of adversarial descriptions, v2 write,
and legacy-v1 row dark (forced cold-start).
Covers R4 + new sanitisation requirement.
* feat(news/summarize): accept bodies + bump summary cache v5→v6 (U6)
SummarizeArticle now grounds on per-headline article bodies when callers
supply them, so the dashboard "News summary" path stops hallucinating
across unrelated headlines when the upstream RSS carried context.
Three coordinated changes:
1. SummarizeArticleRequest handler reads req.bodies, sanitises each entry
through sanitizeForPrompt (same trust treatment as geoContext — bodies
are untrusted RSS text), clips to 400 chars, and pads to the headlines
length so pair-wise identity is stable.
2. buildArticlePrompts accepts optional bodies and interleaves a
` Context: <body>` line under each numbered headline that has a
non-empty body. Skipped in translate mode (headline[0]-only) and when
all bodies are empty — yielding a byte-identical prompt to pre-U6
for every current caller (R6 preserved).
3. summary-cache-key bumps CACHE_VERSION v5→v6 so the pre-grounding rows
(produced from headline-only prompts) cold-start cleanly. Extends
canonicalizeSummaryInputs + buildSummaryCacheKey with a pair-wise
bodies segment `:bd<hash>`; the prefix is `:bd` rather than `:b` to
avoid colliding with `:brief:` when pattern-matching keys. Translate
mode is headline[0]-only and intentionally does not shift on bodies.
Dedup reorder preserved: the handler re-pairs bodies to the deduplicated
top-5 via findIndex, so layout matches without breaking cache identity.
New tests: 7 on buildArticlePrompts (bodies interleave, partial fill,
translate-mode skip, clip, short-array tolerance), 8 on
buildSummaryCacheKey (pair-wise sort, cache-bust on body drift, translate
skip). Existing summary-cache-key assertions updated v5→v6.
Covers R3, R4.
* feat(consumers): surface RSS snippet across dashboard, email, relay, MCP + audit (U7)
Thread the RSS description from the ingestion path (U1-U5) into every
user-facing LLM-adjacent surface. Audit the notification producers so
RSS-origin and domain-origin events stay on distinct contracts.
Dashboard (proto snippet → client → panel):
- src/types/index.ts NewsItem.snippet?:string (client-side field).
- src/app/data-loader.ts proto→client mapper propagates p.snippet.
- src/components/NewsPanel.ts renders snippet as a truncated (~200 chars,
word-boundary ellipsis) `.item-snippet` line under each headline.
- NewsPanel.currentBodies tracks per-headline bodies paired 1:1 with
currentHeadlines; passed as options.bodies to generateSummary so the
server-side SummarizeArticle LLM grounds on the article body.
Summary plumbing:
- src/services/summarization.ts threads bodies through SummarizeOptions
→ generateSummary → runApiChain → tryApiProvider; cache key now includes
bodies (via U6's buildSummaryCacheKey signature).
MCP world-brief:
- api/mcp.ts pairs headlines with their RSS snippets and POSTs `bodies`
to /api/news/v1/summarize-article so the MCP tool surface is no longer
starved.
Email digest:
- scripts/seed-digest-notifications.mjs plain-text formatDigest appends
a ~200-char truncated snippet line under each story; HTML formatDigestHtml
renders a dim-grey description div between title and meta. Both gated
on non-empty description (R6 — empty → today's behavior).
Real-time alerts:
- src/services/breaking-news-alerts.ts BreakingAlert gains optional
description; checkBatchForBreakingAlerts reads item.snippet; dispatchAlert
includes `description` in the /api/notify payload when present.
Notification relay:
- scripts/notification-relay.cjs formatMessage gated on
NOTIFY_RELAY_INCLUDE_SNIPPET=1 (default off). When on, RSS-origin
payloads render a `> <snippet>` context line under the title. When off
or payload.description absent, output is byte-identical to pre-U7.
Audit (RSS vs domain):
- tests/notification-relay-payload-audit.test.mjs enforces file-level
@notification-source tags on every producer, rejects `description:` in
domain-origin payload blocks, and verifies the relay codepath gates
snippet rendering under the flag.
- Tag added to ais-relay.cjs (domain), seed-aviation.mjs (domain),
alert-emitter.mjs (domain), breaking-news-alerts.ts (rss).
Deferred (plan explicitly flags): InsightsPanel + cluster-producer
plumbing (bodies default to [] — will unlock gradually once news:insights:v1
producer also carries primarySnippet).
Covers R5, R6.
* docs+test: grounding-path note + bump pinned CACHE_VERSION v5→v6 (U8)
Final verification for the RSS-description-end-to-end fix:
- docs/architecture.mdx — one-paragraph "News Grounding Pipeline"
subsection tracing parser → story:track:v1.description → NewsItem.snippet
→ brief / SummarizeArticle / dashboard / email / relay / MCP, with the
empty-description R6 fallback rule called out explicitly.
- tests/summarize-reasoning.test.mjs — Fix-4 static-analysis pin updated
to match the v6 bump from U6. Without this the summary cache bump silently
regressed CI's pinned-version assertion.
Final sweep (2026-04-24):
- grep -rn 'brief:llm:description:v1' → only in the U5 legacy-row test
simulation (by design: proves the v2 bump forces cold-start).
- grep -rn 'brief:llm:whymatters:v2/v6/shadow:v4' → no live references.
- grep -rn 'summary:v5' → no references.
- CACHE_VERSION = 'v6' in src/utils/summary-cache-key.ts.
- Full tsx --test sweep across all tests/*.test.{mjs,mts}: 6747/6747 pass.
- npm run typecheck + typecheck:api: both clean.
Covers R4, R6, R7.
* fix(rss-description): address /ce:review findings before merge
14 fixes from structured code review across 13 reviewer personas.
Correctness-critical (P1 — fixes that prevent R6/U7 contract violations):
- NewsPanel signature covers currentBodies so view-mode toggles that leave
headlines identical but bodies different now invalidate in-flight summaries.
Without this, switching renderItems → renderClusters mid-summary let a
grounded response arrive under a stale (now-orphaned) cache key.
- summarize-article.ts re-pairs bodies with headlines BEFORE dedup via a
single zip-sanitize-filter-dedup pass. Previously bodies[] was indexed by
position in light-sanitized headlines while findIndex looked up the
full-sanitized array — any headline that sanitizeHeadlines emptied
mispaired every subsequent body, grounding the LLM on the wrong story.
- Client skips the pre-chain cache lookup when bodies are present, since
client builds keys from RAW bodies while server sanitizes first. The
keys diverge on injection content, which would silently miss the
server's authoritative cache every call.
Test + audit hardening:
- Legacy v1 eviction test now uses the real hashBriefStory(story()) suffix
instead of a literal "somehash", so a bug where the reader still queried
the v1 prefix at the real key would actually be caught.
- tests/summary-cache-key.test.mts adds 400-char clip identity coverage so
the canonicalizer's clip and any downstream clip can't silently drift.
- tests/news-rss-description-extract.test.mts renames the well-formed
CDATA test and adds a new test documenting the malformed-]]> fallback
behavior (plain regex captures, article content survives).
Safe_auto cleanups:
- Deleted dead SNIPPET_PUSH_MAX constant in notification-relay.cjs.
- BETA-mode groq warm call now passes bodies, warming the right cache slot.
- seed-digest shares a local normalize-equality helper for description !=
headline comparison, matching the parser's contract.
- Pair-wise sort in summary-cache-key tie-breaks on body so duplicate
headlines produce stable order across runs.
- buildSummaryCacheKey gained JSDoc documenting the client/server contract
and the bodies parameter semantics.
- MCP get_world_brief tool description now mentions RSS article-body
grounding so calling agents see the current contract.
- _shared.ts `opts.bodies![i]!` double-bang replaced with `?? ''`.
- extractRawTagBody regexes cached in module-level Map, mirroring the
existing TAG_REGEX_CACHE pattern.
Deferred to follow-up (tracked for PR description / separate issue):
- Promote shared MAX_BODY constant across the 5 clip sites
- Promote shared truncateForDisplay helper across 4 render sites
- Collapse NewsPanel.{currentHeadlines, currentBodies} → Array<{title, snippet}>
- Promote sanitizeStoryForPrompt to shared/brief-llm-core.js
- Split list-feed-digest.ts parser helpers into sibling -utils.ts
- Strengthen audit test: forward-sweep + behavioral gate test
Tests: 6749/6749 pass. Typecheck clean on both configs.
* fix(summarization): thread bodies through browser T5 path (Codex #2)
Addresses the second of two Codex-raised findings on PR #3370:
The PR threaded bodies through the server-side API provider chain
(Ollama → Groq → OpenRouter → /api/news/v1/summarize-article) but the
local browser T5 path at tryBrowserT5 was still summarising from
headlines alone. In BETA_MODE that ungrounded path runs BEFORE the
grounded server providers; in normal mode it remains the last
fallback. Whenever T5-small won, the dashboard summary surface
regressed to the headline-only path — the exact hallucination class
this PR exists to eliminate.
Fix: tryBrowserT5 accepts an optional `bodies` parameter and
interleaves each body with its paired headline via a `headline —
body` separator in the combined text (clipped to 200 chars per body
to stay within T5-small's ~512-token context window). All three call
sites (BETA warm, BETA cold, normal-mode fallback) now pass the
bodies threaded down from generateSummary options.bodies.
When bodies is empty/omitted, the combined text is byte-identical to
pre-fix (R6 preserved).
On Codex finding #1 (story:track:v1 additive-only HSET keeps a body
from an earlier mention of the same normalized title), declining to
change. The current rule — "if this mention has a body, overwrite;
otherwise leave the prior body alone" — is defensible: a body from
mention A is not falsified by mention B being body-less (a wire
reprint doesn't invalidate the original source's body). A feed that
publishes a corrected headline creates a new normalized-title hash,
so no stale body carries forward. The failure window is narrow (live
story evolving while keeping the same title through hours of
body-less wire reprints) and the 7-day STORY_TTL is the backstop.
Opening a follow-up issue to revisit semantics if real-world evidence
surfaces a stale-grounding case.
* fix(story-track): description always-written to overwrite stale bodies (Codex #1)
Revisiting Codex finding #1 on PR #3370 after re-review. The previous
response declined the fix with reasoning; on reflection the argument
was over-defending the current behavior.
Problem: buildStoryTrackHsetFields previously wrote `description` only
when non-empty. Because story:track:v1 rows are collapsed by
normalized-title hash, an earlier mention's body would persist for up
to STORY_TTL (7 days) on subsequent body-less mentions of the same
story. Consumers reading `track.description` via HGETALL could not
distinguish "this mention's body" from "some mention's body from the
last week," silently grounding brief / whyMatters / SummarizeArticle
LLMs on text the current mention never supplied. That violates the
grounding contract advertised to every downstream surface in this PR.
Fix: HSET `description` unconditionally on every mention — empty
string when the current item has no body, real body when it does. An
empty value overwrites any prior mention's body so the row is always
authoritative for the current cycle. Consumers continue to treat
empty description as "fall back to cleaned headline" (R6 preserved).
The 7-day STORY_TTL and normalized-title hash semantics are unchanged.
Trade-off accepted: a valid body from Feed A (NYT) is wiped when Feed
B (AP body-less wire reprint) arrives for the same normalized title,
even though Feed A's body is factually correct. Rationale: the
alternative — keeping Feed A's body indefinitely — means the user
sees Feed A's body attributed (by proximity) to an AP mention at a
later timestamp, which is at minimum misleading and at worst carries
retracted/corrected details. Honest absence beats unlabeled presence.
Tests: new stale-body overwrite sequence test (T0 body → T1 empty →
T2 new body), existing "writes description when non-empty" preserved,
existing "omits when empty" inverted to "writes empty, overwriting."
cache-keys.ts contract comment updated to mark description as
always-written rather than optional.
|
||
|
Elie Habib
|
d75bde4e03 |
fix(agent-readiness): host-aware oauth-protected-resource endpoint (#3351)
* fix(agent-readiness): host-aware oauth-protected-resource endpoint
isitagentready.com enforces that `authorization_servers[*]` share
origin with `resource` (same-origin rule, matches Cloudflare's
mcp.cloudflare.com reference — RFC 9728 §3 permits split origins
but the scanner is stricter).
A single static file served from 3 hosts (apex/www/api) can only
satisfy one origin at a time. Replacing with an edge function that
derives both `resource` and `authorization_servers` from the
request `Host` header gives each origin self-consistent metadata.
No server-side behavior changes: api/oauth/*.js token issuer
doesn't bind tokens to a specific resource value (verified in
the previous PR's review).
* fix(agent-readiness): host-derive resource_metadata + runtime guardrails
Addresses P1/P2 review on this PR:
- api/mcp.ts (P1): WWW-Authenticate resource_metadata was still
hardcoded to apex even when the client hit api.worldmonitor.app.
Derive from request.headers.get('host') so each client gets a
pointer matching their own origin — consistent with the host-
aware edge function this PR introduces.
- api/oauth-protected-resource.ts (P2): add Vary: Host so any
intermediate cache keys by hostname (belt + suspenders on top of
Vercel's routing).
- tests/deploy-config.test.mjs (P2): replace regex-on-source with
a runtime handler invocation asserting origin-matching metadata
for apex/www/api hosts, and tighten the api/mcp.ts assertion to
require host-derived resource_metadata construction.
---------
Co-authored-by: Elie Habib <elie@worldmonitor.app>
|
||
|
Elie Habib
|
9c3c7e8657 |
fix(agent-readiness): align OAuth resource with public MCP origin (#3345)
* fix(agent-readiness): align OAuth resource with actual public MCP origin isitagentready.com's OAuth Protected Resource check enforces an origin match between the scanned host and the metadata's `resource` field (per the spirit of RFC 9728 §3). Our metadata declared `resource: "https://api.worldmonitor.app"` while the MCP endpoint is publicly served at `https://worldmonitor.app/mcp` (per vercel.json's /mcp → /api/mcp rewrite and the MCP card's transport.endpoint). Flip `resource` to `https://worldmonitor.app` across the three places that declare it: - public/.well-known/oauth-protected-resource - public/.well-known/mcp/server-card.json (authentication block) - api/mcp.ts (two WWW-Authenticate resource_metadata pointers) `authorization_servers` intentionally stays on api.worldmonitor.app — that's where /oauth/{authorize,token,register} actually live. RFC 9728 permits AS and resource to be at different origins. No server-side validation breaks: api/oauth/*.js and api/mcp.ts do not bind tokens to the old resource value. * fix(agent-readiness): align docs/tests + add MCP origin guardrail Addresses P1/P2 review on this PR. The resource-origin flip in the previous commit only moved the mismatch from apex to api unless the repo also documents apex as the canonical MCP origin. - docs/mcp.mdx: swap api.worldmonitor.app/mcp -> worldmonitor.app/mcp (OAuth endpoints stay on api.*, only the resource URL changes) - tests/mcp.test.mjs: same fixture update - tests/deploy-config.test.mjs: new guardrail block asserting that MCP transport.endpoint origin, OAuth metadata resource, MCP card authentication.resource, and api/mcp.ts resource_metadata pointers all share the same origin. Includes a regression guard that authorization_servers stays on api.worldmonitor.app (the intentional resource/AS split). |
||
|
Elie Habib
|
30ddad28d7 |
fix(seeds): upstream API drift — SPDR XLSX + IMF IRFCL + IMF-External BX/BM drop (#3076)
* fix(seeds): gold-etf XLSX migration, IRFCL dataflow, imf-external BX/BM drop Three upstream-drift regressions caught from the market-backup + imf-extended bundle logs. Root causes validated by live API probes before coding. 1. seed-gold-etf-flows: SPDR /assets/dynamic/GLD/GLD_US_archive_EN.csv now silently returns a PDF (Content-Type: application/pdf) — site migrated to api.spdrgoldshares.com/api/v1/historical-archive which serves XLSX. Swapped the CSV parser for an exceljs-based XLSX parser. Adds browser-ish Origin/Referer headers (SPDR swaps payload for PDF without them) and a Content-Type guard. Column layout: Date | Closing | ... | Tonnes | Total NAV USD. 2. seed-gold-cb-reserves: PR #3038 shipped with IMF.STA/IFS dataflow and 3-segment key M..<indicator> — both wrong. IFS isn't exposed on api.imf.org (HTTP 204). Gold-reserves data lives under IMF.STA/IRFCL with 4 dimensions (COUNTRY.INDICATOR.SECTOR.FREQUENCY). Verified live: *.IRFCLDT1_IRFCL56_FTO.*.M returns 111 series. Switched to IRFCL + IRFCLDT1_IRFCL56_FTO (fine troy ounces) and fallbacks. The valueIsOunces flag now matches _FTO suffix (keeps legacy _OZT/OUNCE detection for backward compat). 3. seed-imf-external: BX/BM (export/import LEVELS, USD) WEO coverage collapsed to ~10 countries in late 2026 — the seeder's >=190-country validate floor was failing every run. Dropped BX/BM from fetch + join; kept BCA (~209) / TM_RPCH (~189) / TX_RPCH (~190). exportsUsd / importsUsd / tradeBalanceUsd fields kept as explicit null so consumers see a deliberate gap. validate floor lowered to 180 (BCA∪TM∪TX union). Tests: 32/32 pass. Rewrote gold-etf tests to use synthetic XLSX fixtures (exceljs resolved from scripts/package.json since repo root doesn't have it). Updated imf-external tests for the new indicator set + null BX/BM contract + 180-country validate threshold. * fix(mcp): update get_country_macro description after BX/BM drop Consumer-side catch during PR #3076 validation: the MCP tool description still promised 'exports, imports, trade balance' fields that the seeder fix nulls out. LLM consumers would be directed to exportsUsd/importsUsd/ tradeBalanceUsd fields that always return null since seed-imf-external dropped BX/BM (WEO coverage collapsed to ~10 countries). Updated description to list only the indicators actually populated (currentAccountUsd, importVolumePctChg, exportVolumePctChg) with an explicit note about the null trade-level fields so LLMs don't attempt to use them. * fix(gold-cb-reserves): compute real pctOfReserves + add exceljs to root Follow-up to #3076 review. 1. pctOfReserves was hardcoded to 0 with a "IFS doesn't give us total reserves" comment. That was a lazy limitation claim — IMF IRFCL DOES expose total official reserve assets as IRFCLDT1_IRFCL65_USD parallel to the gold USD series IRFCLDT1_IRFCL56_USD. fetchCbReserves now pulls all three indicators (primary FTO tonnage + the two USD series) via Promise.allSettled and passes the USD pair to buildReservesPayload so it can compute the true gold share per country. Falls back to 0 only when the denominator is genuinely missing for that country (IRFCL coverage: 114 gold_usd, 96 total_usd series; ~15% of holders have no matched-month denominator). 3-month lookback window absorbs per-country reporting lag. 2. CI fix: tests couldn't find exceljs because scripts/package.json is not workspace-linked to the repo root — CI runs `npm ci` at root only. Added exceljs@^4.4.0 as a root devDependency. Runtime seeder continues to resolve it from scripts/node_modules via Node's upward module resolution. 3 new tests cover pct computation, missing-denominator fallback, and the 3-month lookback window. |
||
|
Elie Habib
|
cd5ed0d183 |
feat(seeds): BIS DSR + property prices (2 of 7) (#3048)
* feat(seeds): BIS DSR + property prices (2 of 7) Ships 2 of 7 BIS dataflows flagged as genuinely new signals in #3026 — the rest are redundant with IMF/WB or are low-fit global aggregates. New seeder: scripts/seed-bis-extended.mjs - WS_DSR household debt service ratio (% income, quarterly) - WS_SPP residential property prices (real index, quarterly) - WS_CPP commercial property prices (real index, quarterly) Gold-standard pattern: atomic publish + writeExtraKey for extras, retry on missing startPeriod, TTL = 3 days (3× 12h cron), runSeed drives seed-meta:economic:bis-extended. Series selection scores dimension matches (PP_VALUATION=R / UNIT_MEASURE=628 for property, DSR_BORROWERS=P / DSR_ADJUST=A for DSR), then falls back to observation count. Wired into: - bootstrap (slow tier) + cache-keys.ts - api/health.js (STANDALONE_KEYS + SEED_META, maxStaleMin = 24h) - api/mcp.ts get_economic_data tool (_cacheKeys + _freshnessChecks) - resilience macroFiscal: new householdDebtService sub-metric (weight 0.05, currentAccountPct rebalanced 0.3 → 0.25) - Housing Cycle tile on CountryDeepDivePanel (Economic Indicators card) with euro-area (XM) fallback for EU member states - seed-bundle-macro Railway cron (BIS-Extended, 12h interval) Tests: tests/bis-extended-seed.test.mjs covers CSV parsing, series selection, quarter math + YoY. Updated resilience golden-value tests for the macroFiscal weight rebalance. Closes #3026 https://claude.ai/code/session_01DDo39mPD9N2fNHtUntHDqN * fix(resilience): unblock PR #3048 on #3046 stack - rebase onto #3046; final macroFiscal weights: govRevenue 0.40, currentAccount 0.20, debtGrowth 0.20, unemployment 0.15, householdDebtService 0.05 (=1.00) - add updateHousingCycle? stub to CountryBriefPanel interface so country-intel dispatch typechecks - add HR to EURO_AREA fallback set (joined euro 2023-01-01) - seed-bis-extended: extend SPP/CPP TTLs when DSR fetch returns empty so the rejected publish does not silently expire the still-good property keys - update resilience goldens for the 5-sub-metric macroFiscal blend * fix(country-brief): housing tile renders em-dash for null change values The new Housing cycle tile used `?? 0` to default qoqChange/yoyChange/change when missing, fabricating a flat "0.0%" label (with positive-trend styling) for countries with no prior comparable period. Fetch path and builders correctly return null; the panel was coercing it. formatPctTrend now accepts null|undefined and returns an em-dash, matching how other cards surface unavailable metrics. Drop the `?? 0` fallbacks at the three housing call sites. * fix(seed-health): register economic:bis-extended seed-meta monitoring 12h Railway cron writes seed-meta:economic:bis-extended but it was missing from SEED_DOMAINS, so /api/seed-health never reported its freshness. intervalMin=720 matches maxStaleMin/2 (1440/2) from api/health.js. * fix(seed-bis-extended): decouple DSR/SPP/CPP so one fetch failure doesn't block the others Previously validate() required data.entries.length > 0 on the DSR slice after publishTransform pulled it out of the aggregate payload. If WS_DSR fetch failed but WS_SPP / WS_CPP succeeded, validate() rejected the publish → afterPublish() never ran → fresh SPP/CPP data was silently discarded and only the old snapshots got a TTL bump. This treats the three datasets as independent: - SPP and CPP are now published (or have their existing TTLs extended) as side-effects of fetchAll(), per-dataset. A failure in one never affects the others. - DSR continues to flow through runSeed's canonical-key path. When DSR is empty, publishTransform yields { entries: [] } so atomicPublish skips the canonical write (preserving the old DSR snapshot); runSeed's skipped branch extends its TTL and refreshes seed-meta. Shape B (one runSeed call, semantics changed) chosen over Shape A (three sequential runSeed calls) because runSeed owns the lock + process.exit lifecycle and can't be safely called three times in a row, and Shape B keeps the single aggregate seed-meta:economic:bis-extended key that health.js already monitors. Tests cover both failure modes: - DSR empty + SPP/CPP healthy → SPP/CPP written, DSR TTL extended - DSR healthy + SPP/CPP empty → DSR written, SPP/CPP TTLs extended * fix(health): per-dataset seed-meta for BIS DSR/SPP/CPP Health was pointing bisDsr / bisPropertyResidential / bisPropertyCommercial at the shared seed-meta:economic:bis-extended key, which runSeed refreshes on every run (including its validation-failed "skipped" branch). A DSR-only outage therefore left bisDsr reporting fresh in api/health.js while the resilience scorer consumed stale/missing economic:bis:dsr:v1 data. Write a dedicated seed-meta key per dataset ONLY when that dataset actually published fresh entries. The aggregate bis-extended key stays as a "seeder ran" signal in api/seed-health.js. * fix(seed-bis-extended): write DSR seed-meta only after atomicPublish succeeds Previously fetchAll() wrote seed-meta:economic:bis-dsr inline before runSeed/atomicPublish ran. If atomicPublish then failed (Redis hiccup, validate rejection, etc.), seed-meta was already bumped — health would report DSR fresh while the canonical key was stale. Move the DSR seed-meta write into a dsrAfterPublish callback passed to runSeed via the existing afterPublish hook, which fires only after a successful canonical publish. SPP/CPP paths already used this ordering inside publishDatasetIndependently; this brings DSR in line. Adds a regression test exercising dsrAfterPublish with mocked Upstash: populated DSR -> single SET on seed-meta key; null/empty DSR -> zero Redis calls. * fix(resilience): per-dataset BIS seed-meta keys in freshness overrides SOURCE_KEY_META_OVERRIDES previously collapsed economic:bis:dsr:v1 and both property-* sourceKeys onto the aggregate seed-meta:economic:bis-extended key. api/health.js (SEED_META) writes per-dataset keys (seed-meta:economic:bis-dsr / bis-property-residential / bis-property-commercial), so a DSR-only outage showed stale in /api/health but the resilience dimension freshness code still reported macroFiscal inputs as fresh. Map each BIS sourceKey to its dedicated seed-meta key to match health.js. The aggregate bis-extended key is still written by the seeder and read by api/seed-health.js as a "seeder ran" signal, so it is retained upstream. * fix(bis): prefer households in DSR + per-dataset freshness in MCP Greptile review catches on #3048: 1. buildDsr() was selecting DSR_BORROWERS='P' (private non-financial) while the UI labels it "Household DSR" and resilience scoring uses it as `householdDebtService`. Changed to 'H' (households). Countries without an H series now get dropped rather than silently mislabeled. 2. api/mcp.ts get_economic_data still read only the aggregate seed-meta:economic:bis-extended for freshness. If DSR goes stale while SPP/CPP keep publishing, MCP would report the BIS block as fresh even though one of its returned keys is stale. Swapped to the three per-dataset seed-meta keys (bis-dsr, bis-property-residential, bis-property-commercial), matching the fix already applied to /api/health and the resilience dimension-freshness pipeline. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Elie Habib
|
f5d8ff9458 |
feat(seeds): Eurostat house prices + quarterly debt + industrial production (#3047)
* feat(seeds): Eurostat house prices + quarterly debt + industrial production Adds three new Eurostat overlay seeders covering all 27 EU members plus EA20 and EU27_2020 aggregates (issue #3028): - prc_hpi_a (annual house price index, 10y sparkline, TTL 35d) key: economic:eurostat:house-prices:v1 complements BIS WS_SPP (#3026) for the Housing cycle tile - gov_10q_ggdebt (quarterly gov debt %GDP, 8q sparkline, TTL 14d) key: economic:eurostat:gov-debt-q:v1 upgrades National Debt card cadence from annual IMF to quarterly for EU - sts_inpr_m (monthly industrial production, 12m sparkline, TTL 5d) key: economic:eurostat:industrial-production:v1 feeds "Real economy pulse" sparkline on Economic Indicators card Shared JSON-stat parser in scripts/_eurostat-utils.mjs handles the EL/GR and EA20 geo quirks and returns full time series for sparklines. Wires each seeder into bootstrap (SLOW_KEYS), health registries (keys + seed-meta thresholds matched to cadence), macro seed bundle, cache-keys shared module, and the MCP tool registry (get_eu_housing_cycle, get_eu_quarterly_gov_debt, get_eu_industrial_production). MCP tool count updated to 31. Tests cover JSON-stat parsing, sparkline ordering, EU-only coverage gating (non-EU geos return null so panels never render blank tiles), validator thresholds, and registry wiring across all surfaces. https://claude.ai/code/session_01Tgm6gG5yUMRoc2LRAKvmza * fix(bootstrap): register new Eurostat keys in tiers, defer consumers Adds eurostatHousePrices/GovDebtQ/IndProd to BOOTSTRAP_TIERS ('slow') to match SLOW_KEYS in api/bootstrap.js, and lists them as PENDING_CONSUMERS in the hydration coverage test (panel wiring lands in follow-up). * fix(eurostat): raise seeder coverage thresholds to catch partial publishes The three Eurostat overlay seeders (house prices, quarterly gov debt, monthly industrial production) all validated with makeValidator(10) against a fixed 29-geo universe (EU27 + EA20 + EU27_2020). A bad run returning only 10-15 geos would pass validation and silently publish a snapshot missing most of the EU. Raise thresholds to near-complete coverage, with a small margin for geos with patchy reporting: - house prices (annual): 10 -> 24 - gov debt (quarterly): 10 -> 24 - industrial prod (monthly): 10 -> 22 (monthly is slightly patchier) Add a guard test that asserts every overlay seeder keeps its threshold >=22 so this regression can't reappear. * fix(seed-health): register 3 Eurostat seed-meta entries house-prices, gov-debt-q, industrial-production were wired in api/health.js SEED_META but missing from api/seed-health.js SEED_DOMAINS, so /api/seed-health would not surface their freshness. intervalMin = health.js maxStaleMin / 2 per convention. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Elie Habib
|
71a6309503 |
feat(seeds): expand IMF WEO coverage — growth, labor, external themes (#3027) (#3046)
* feat(seeds): expand IMF WEO coverage — growth, labor, external themes (#3027) Adds three new SDMX-3.0 seeders alongside the existing imf-macro seeder to surface 15+ additional WEO indicators across ~210 countries at zero incremental API cost. Bundled into seed-bundle-imf-extended.mjs on the same monthly Railway cron cadence. Seeders + Redis keys: - seed-imf-growth.mjs → economic:imf:growth:v1 NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP - seed-imf-labor.mjs → economic:imf:labor:v1 LUR (unemployment), LP (population) - seed-imf-external.mjs → economic:imf:external:v1 BX, BM, BCA, TM_RPCH, TX_RPCH (+ derived trade balance) - seed-imf-macro.mjs extended with PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP All four seeders share the 35-day TTL (monthly WEO release) and ~210 country coverage via the same imfSdmxFetchIndicator helper. Wiring: - api/bootstrap.js, api/health.js, server/_shared/cache-keys.ts — register new keys, mark them slow-tier, add SEED_META freshness thresholds matching the imfMacro entry (70d = 2× monthly cadence) - server/worldmonitor/resilience/v1/_dimension-freshness.ts — override entries for the dash-vs-colon seed-meta key shape - _indicator-registry.ts — add LUR as a 4th macroFiscal sub-metric (enrichment tier, weight 0.15); rebalance govRevenuePct (0.5→0.4) and currentAccountPct (0.3→0.25) so weights still sum to 1.0 - _dimension-scorers.ts — read economic:imf:labor:v1 in scoreMacroFiscal, normalize LUR with goalposts 3% (best) → 25% (worst); null-tolerant so weightedBlend redistributes when labor data is unavailable - api/mcp.ts — new get_country_macro tool bundling all four IMF keys with a single freshness check; describes per-country fields including growth/inflation/labor/BOP for LLM-driven country screening - src/services/imf-country-data.ts — bootstrap-cached client + pure buildImfEconomicIndicators helper - src/app/country-intel.ts — async-fetch the IMF bundle on country selection and merge real GDP growth, CPI inflation, unemployment, and GDP/capita rows into the Economic Indicators card; bumps card cap from 3 → 6 rows to fit live signals + IMF context Tests: - tests/seed-imf-extended.test.mjs — 13 unit tests across the three new seeders' pure helpers (canonical keys, ISO3→ISO2 mapping, aggregate filtering, derived savings-investment gap & trade balance, validate thresholds) - tests/imf-country-data.test.mts — 6 tests for the panel rendering helper, including stagflation flag and high-unemployment trend - tests/resilience-dimension-scorers.test.mts — new LUR sub-metric test (tight vs slack labor); existing scoreMacroFiscal coverage assertions updated for the new 4-metric weight split - tests/helpers/resilience-fixtures.mts — labor fixture for NO/US/YE so the existing macroFiscal ordering test still resolves the LUR weight - tests/bootstrap.test.mjs — register imfGrowth/imfLabor/imfExternal as pending consumers (matching imfMacro) - tests/mcp.test.mjs — bump tools/list count 28 → 29 https://claude.ai/code/session_018enRzZuRqaMudKsLD5RLZv * fix(resilience): update macroFiscal goldens for LUR weight rebalance Recompute pinned fixture values after adding labor-unemployment as 4th macroFiscal sub-metric (weight rebalance in _indicator-registry). Also align seed-imf-external tradeBalance to a single reference year to avoid mixing ex/im values from different WEO vintages. * fix(seeds): tighten IMF coverage gates to reject partial snapshots IMF WEO growth/labor/external indicators report ~210 countries for healthy runs. Previous thresholds (150/100/150) let a bad IMF run overwrite a good snapshot with dozens of missing countries and still pass validation. Raise all three to >=190, matching the pattern of sibling seeders and leaving a ~20-country margin for indicators with slightly narrower reporting. Labor validator unions LUR + population (LP), so healthy coverage tracks LP (~210), not LUR (~100) — the old 100 threshold was based on a misread of the union logic. * fix(seed-health): register imf-growth/labor/external seed-meta keys Missing SEED_DOMAINS entries meant the 3 new IMF WEO seeders (growth, labor, external) had no /api/seed-health visibility. intervalMin=50400 matches health.js maxStaleMin/2 (100800/2) — same monthly WEO cadence as imf-macro. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Elie Habib
|
d1cb0e3c10 |
feat(sectors): add P/E valuation benchmarking to sector heatmap (#2929)
* feat(sectors): add P/E valuation benchmarking to sector heatmap Trailing/forward P/E, beta, and returns for 12 sector ETFs from Yahoo Finance. Horizontal bar chart color-coded by valuation level plus sortable table. Extends existing sector data pipeline. * fix(sectors): clear stale valuations on empty refresh + document cache behavior * fix(sectors): force valuation rollout for cached + breaker-persisted bootstraps - Bumped market:sectors bootstrap key v1 -> v2 so stale 24h slow-tier payloads without the new valuations field are invisible to returning users on next page load - Versioned the fetchSectors circuit-breaker (name -> "Sector Summary v2") so old localStorage/IndexedDB entries predating this PR cannot be returned as stale via the SWR path - shouldCache now requires the valuations field to be present on the cached response, not just a non-empty sectors array - loadMarkets no longer clears the valuations tab when a hydrated or fresh payload lacks the field; prior render is left intact, matching the finding's requirement - Defensive check: hydrated payloads without valuations fall through to a live fetch instead of rendering an empty valuations tab * fix(stocks): correct beta3Year source and null YTD color in sector P/E view - scripts/ais-relay.cjs: beta3Year lives on defaultKeyStatistics (ks), not summaryDetail (sd); the previous fallback was a silent no-op. - src/components/MarketPanel.ts: null ytdReturn now renders with var(--text-dim) instead of var(--red); the '--' placeholder no longer looks like a loss. Addresses greptile review on PR #2929. |
||
|
Fayez Bast
|
8609ad1384 |
feat: " climate disasters alerts seeders " (#2550)
* Revert "Revert "feat(climate): add climate disasters seed + ListClimateDisast…"
This reverts commit
|
||
|
Elie Habib
|
88923388f4 |
feat(climate): expand get_climate_data MCP tool with all data layers (#2690)
* feat(climate): expand get_climate_data MCP tool with air quality, news, and layer filter - Add climate:air-quality:v1 and climate:news-intelligence:v1 to _cacheKeys - Add freshnessChecks for air-quality (180min) and climate-news (90min) - Add layer/region input parameters for filtered queries - Update description to cover all 6 climate data layers Note: climateAirQuality and climateZoneNormals are NOT added to bootstrap because they lack getHydratedData consumers in src/. They are already monitored via health.js BOOTSTRAP_KEYS/SEED_META. climate:disasters:v1 does not exist as a key; natural disaster data is served by the existing natural:events:v1 key with its own MCP tool. Closes #2474 * fix(review): remove dead layer/region params from get_climate_data Cache-read MCP tools never receive params at runtime (executeTool dispatches with no arguments). The layer/region schema was accepted but silently ignored, creating a broken API contract. |
||
|
Fayez Bast
|
02f55dc584 |
feat(climate): add ocean ice indicators seed and RPC (#2652)
* feat(climate): add ocean ice indicators seed and RPC * fix(review): restore MCP maxStaleMin, widen health threshold, harden sea level parser, type globe.gl shim - Restore get_climate_data _maxStaleMin to 2880 (was accidentally lowered to 1440) - Bump oceanIce SEED_META maxStaleMin from 1440 to 2880 (2× daily interval, tolerates one missed run) - Add fallback regex patterns for NASA sea level overlay HTML parsing - Replace globe.gl GlobeInstance `any` with typed interface (index sig stays `any` for Three.js compat) * fix(review): merge prior cache on partial failures, fix fallback regex, omit trend without baseline - P1: fetchOceanIceData() now reads prior cache and merges last-known-good indicators when any upstream source fails, preventing partial overwrites from erasing previously healthy data - P1: sea level fallback regex now requires "current" context to avoid matching the historical 1993 baseline rate instead of the current rate - P2: classifyArcticTrend() returns null (omitted from payload) when no climatology baseline exists, instead of misleadingly labeling as "average" - Added tests for all three fixes * fix(review): merge prior cache by source field group, not whole object Prior-cache merge was too coarse: Object.assign(payload, priorCache) reintroduced stale arctic_extent_anomaly_mkm2 and arctic_trend from prior cache when sea-ice succeeded but intentionally omitted those fields (no climatology baseline), and an unrelated source like OHC or sea level failed in the same run. Fix: define per-source field groups (seaIce, seaLevel, ohc, sst). Only fall back to prior cache fields for groups whose source failed entirely. When a source succeeds, only its returned fields appear in the payload, even if it omits fields it previously provided. Added test covering the exact combined case: sea-ice climatology unavailable + unrelated source failure + prior-cache merge enabled. --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
bb4f8dcb12 |
feat(climate): add WMO normals seeding and CO2 monitoring (#2531)
* feat(climate): add WMO normals seeding and CO2 monitoring * fix(climate): skip missing normals per-zone and align anomaly tooltip copy * fix(climate): remove normals from bootstrap and harden health/cache key wiring * feat(climate): version anomaly cache to v2, harden seed freshness, and align CO2/normal baselines |
||
|
Elie Habib
|
ae4010a795 |
Revert "feat(climate): add climate disasters seed + ListClimateDisasters RPC with snake_case Redis payload (#2535)" (#2544)
This reverts commit
|
||
|
Fayez Bast
|
e2dea9440d | feat(climate): add climate disasters seed + ListClimateDisasters RPC with snake_case Redis payload (#2535) | ||
|
Elie Habib
|
8aee4d340e |
feat(intelligence): GetCountryRisk RPC + MCP tool for per-country risk scores (#2502)
* feat(intelligence): GetCountryRisk RPC for per-country risk intelligence Adds a new fast Redis-read RPC that consolidates CII score, travel advisory level, and OFAC sanctions exposure into a single per-country response. Replaces the need to call GetRiskScores (all-countries) and filter client-side. Wired to MCP as get_country_risk tool (no LLM, ~200ms, good for agent screening). - proto/intelligence/v1/get_country_risk.proto (new) - server/intelligence/v1/get-country-risk.ts (reads 3 pre-seeded Redis keys) - gateway.ts: slow cache tier - api/mcp.ts: RpcToolDef with 8s timeout - tests/mcp.test.mjs: update tool count 27→28 * fix(intelligence): upstream-unavailable signal, fetchedAt from CII, drop redundant catch P1: return upstreamUnavailable:true when all Redis reads are null — prevents CDN from caching false-negative sanctions/risk responses during Redis outages. P2: fetchedAt now uses cii.computedAt (actual data age) instead of request time. P2: removed redundant .catch(() => null) — getCachedJson already swallows errors. * fix(intelligence): accurate OFAC counts and country names for GetCountryRisk P1: sanctions:pressure:v1.countries is a top-12 slice — switch to a new sanctions:country-counts:v1 key (ISO2→count across ALL 40K+ OFAC entries). Written by seed-sanctions-pressure.mjs in afterPublish alongside entity index. P1: trigger upstreamUnavailable:true when sanctions key alone is missing, preventing false-negative sanctionsActive:false from being cached by CDN. P2: advisory seeder now writes byCountryName (ISO2→display name) derived from country-names.json reverse map. Handler uses it as fallback so countries outside TIER1_COUNTRIES (TH, CO, BD, IT...) get proper names. |
||
|
Elie Habib
|
ba54dc12d7 |
feat(commodity): gold layer enhancements (#2464)
* feat(commodity): add gold layer enhancements from fork review Enrich the commodity variant with learnings from Yazan-Abuawwad/gold-monitor fork: - Add 10 missing gold mines to MINING_SITES: Muruntau (world's largest open-pit gold mine), Kibali (DRC), Sukhoi Log (Russia, development), Ahafo (Ghana), Loulo-Gounkoto (Mali), South Deep (SA), Kumtor (Kyrgyzstan), Yanacocha (Peru), Cerro Negro (Argentina), Tropicana (Australia). Covers ~40% of top-20 global mines previously absent. - Add XAUUSD=X spot gold and 9 FX pairs (EUR, GBP, JPY, CNY, INR, AUD, CHF, CAD, TRY) to shared/commodities.json. All =X symbols auto-seeded via existing seedCommodityQuotes() — no new seeder needed. Registered in YAHOO_ONLY_SYMBOLS in both _shared.ts and ais-relay.cjs. - Add XAU/FX tab to CommoditiesPanel showing gold priced in 10 currencies. Computed live from GC=F * FX rates. Commodity variant only. - Fix InsightsPanel brief title: commodity variant now shows "⛏️ COMMODITY BRIEF" instead of "🌍 WORLD BRIEF". - Route commodity variant daily market brief to commodity feed categories (commodity-news, gold-silver, mining-news, energy, critical-minerals) via new newsCategories option on BuildDailyMarketBriefOptions. - Add Gold Silver Worlds + FX Empire Gold direct RSS feeds to gold-silver panel (9 sources total, up from 7). * fix(commodity): address review findings from PR #2464 - Fix USDCHF=X multiply direction: was true (wrong), now false (USD/CHF is USD-per-CHF convention) - Fix newsCategories augments BRIEF_NEWS_CATEGORIES instead of replacing (preserves macro/Fed context in commodity brief) - Add goldsilverworlds.com + www.fxempire.com to RSS allowlist (api + shared + scripts/shared) - Rename "Metals" tab label conditionally: commodity variant gets "Metals", others keep "Commodities" - Reset _tab to "commodities" when hasXau becomes false (prevent stale XAU tab re-activation) - Add Number.isFinite() guard in _renderXau() before computing xauPrice - Narrow fxMap filter to =X symbols only - Collapse redundant two-branch number formatter to Math.round().toLocaleString() - Remove XAUUSD=X from shared/commodities.json: seeded but never displayed (saves 150ms/cycle) * feat(mcp): add get_commodity_geo tool and update get_market_data description * fix(commodity): correct USDCHF direction, replace headline categories, restore dep overrides * fix(commodity): empty XAU grid fallback and restore FRED timeout to 20s * fix(commodity): remove XAU/USD from MCP description, revert Metals tab label * fix(commodity): remove dead XAUUSD=X from YAHOO_ONLY_SYMBOLS XAU widget uses GC=F as base price, not XAUUSD=X. Symbol was never seeded (not in commodities.json) and never referenced in the UI. |
||
|
Elie Habib
|
380b495be8 |
feat(mcp): live airspace + maritime tools; fix OAuth consent UI (#2442)
* fix(oauth): fix CSS arrow bullets + add MCP branding to consent page
- CSS content:'\2192' (not HTML entity which doesn't work in CSS)
- Rename logo/title to "WorldMonitor MCP" on both consent and error pages
- Inject real news headlines into get_country_brief to prevent hallucination
Fetches list-feed-digest (4s budget), passes top-15 headlines as ?context=
to get-country-intel-brief; brief timeout reduced to 24s to stay under Edge ceiling
* feat(mcp): add get_airspace + get_maritime_activity live query tools
New tools answer real-time positional questions via existing bbox RPCs:
- get_airspace: civilian ADS-B (OpenSky) + military flights over any country
parallel-fetches track-aircraft + list-military-flights, capped at 100 each
- get_maritime_activity: AIS density zones + disruptions for a country's waters
calls get-vessel-snapshot with country bbox
Country → bounding box resolved via shared/country-bboxes.json (167 entries,
generated from public/data/countries.geojson by scripts/generate-country-bboxes.cjs).
Both API calls use 8s AbortSignal.timeout; get_airspace uses Promise.allSettled
so one failure doesn't block the other.
* docs: fix markdown lint in airspace/maritime plan (blank lines around lists)
* fix(oauth): use literal → in CSS content (\2192 is invalid JS octal in ESM)
* fix(hooks): extend bundle check to api/oauth/ subdirectory (was api/*.js, now uses find)
* fix(mcp): address P1 review findings from PR 2442
- JSON import: add 'with { type: json }' so node --test works without tsx loader
- get_airspace: surface upstream failures; partial outage => partial:true+warnings,
total outage => throw (prevents misleading zero-aircraft response)
- pre-push hook: add #!/usr/bin/env bash shebang (was no shebang, ran as /bin/sh
on Linux CI/contributors; process substitution + [[ ]] require bash)
* fix(mcp): replace JSON import attribute with TS module for Vercel compat
Vercel's esbuild bundler does not support `with { type: 'json' }` import
attributes, causing builds to fail with "Expected ';' but found 'with'".
Fix: generate shared/country-bboxes.ts (typed TS module) alongside the
existing JSON file. The TS import has no attributes and bundles cleanly
with all esbuild versions.
Also extend the pre-push bundle check to include api/*.ts root-level files
so this class of error is caught locally before push.
* fix(mcp): reduce get_country_brief timing budget to 24 s (6 s Edge margin)
Digest pre-fetch: 4 s → 2 s (cached endpoint, silent fallback on miss)
Brief call: 24 s → 22 s
Total worst-case: 24 s vs Vercel Edge 30 s hard kill — was 28 s (2 s margin)
* test(mcp): add coverage for get_airspace and get_maritime_activity
9 new tests:
- get_airspace: happy path, unknown code, partial failure (mil down),
total failure (-32603), type=civilian skips military fetch
- get_maritime_activity: happy path, unknown code, API failure (-32603),
empty snapshot handled gracefully
Also fixes import to use .ts extension so Node --test resolver finds the
country-bboxes module (tsx resolves .ts directly; .js alias only works
under moduleResolution:bundler at typecheck time)
* fix(mcp): use .js + .d.ts for country-bboxes — Vercel rejects .ts imports
Vercel edge bundler refuses .ts extension imports even from .ts edge
functions. Plain .js is the only safe runtime import for edge functions.
Pattern: generate shared/country-bboxes.js (pure ESM, no TS syntax) +
shared/country-bboxes.d.ts (type declaration). TypeScript uses the .d.ts
for tuple types at check time; Vercel and Node --test load the .js at
runtime. The previous .ts module is removed.
* test(mcp): update tool count to 26 (main added search_flights + search_flight_prices_by_date)
|
||
|
Elie Habib
|
d01469ba9c |
feat(aviation): add SearchGoogleFlights and SearchGoogleDates RPCs (#2446)
* feat(aviation): add SearchGoogleFlights and SearchGoogleDates RPCs
Port Google Flights internal API from fli Python library as two new
aviation service RPCs routed through the Railway relay.
- Proto: SearchGoogleFlights and SearchGoogleDates messages and RPCs
- Relay: handleGoogleFlightsSearch and handleGoogleFlightsDates handlers
with JSONP parsing, 61-day chunking for date ranges, cabin/stops/sort mappers
- Server handlers: forward params to relay google-flights endpoints
- gateway.ts: no-store for flights, medium cache for dates
* feat(mcp): expose search_flights and search_flight_prices_by_date tools
* test(mcp): update tool count to 24 after adding search_flights and search_flight_prices_by_date
* fix(aviation): address PR review issues in Google Flights RPCs
P1: airline filtering — use gfParseAirlines() in relay (handles comma-joined
string from codegen) and parseStringArray() in server handlers
P1: partial chunk failure now sets degraded: true instead of silently
returning incomplete data as success; relay includes partial: true flag
P2: round-trip date search validates trip_duration > 0 before proceeding;
returns 400 when is_round_trip=true and duration is absent/zero
P2: relay mappers accept user-friendly aliases ('0'/'1' for max_stops,
'price'/'departure' for sort_by) alongside symbolic enum values;
MCP tool docs updated to match
* fix(aviation): use cachedFetchJson in searchGoogleDates for stampede protection
Medium cache tier (10 min) requires Redis-level coalescing to prevent
concurrent requests from all hitting the relay before cache warms.
Cache key includes all request params (sorted airlines for stable keys).
* fix(aviation): always use getAll() for airlines in relay; add multi-airline tests
The OR short-circuit (get() || getAll()) meant get() returned the first
airline value (truthy), so getAll() never ran and only one airline was
forwarded to Google. Fix: unconditionally use getAll().
Tests cover: multi-airline repeated params, single airline, empty array,
comma-joined string from codegen, partial degraded flag propagation.
|
||
|
Elie Habib
|
51f7b7cf6d |
feat(mcp): full OAuth 2.1 compliance — Authorization Code + PKCE + DCR (#2432)
* feat(mcp): full OAuth 2.1 compliance — Authorization Code + PKCE + DCR Adds the complete OAuth 2.1 flow required by claude.ai (and any MCP 2025-03-26 client) on top of the existing client_credentials path. New endpoints: - POST /oauth/register — Dynamic Client Registration (RFC 7591) Strict allowlist: claude.ai/claude.com callbacks + localhost only. 90-day sliding TTL on client records (no unbounded Redis growth). - GET/POST /oauth/authorize — Consent page + authorization code issuance CSRF nonce binding, X-Frame-Options: DENY, HTML-escapes all metadata, shows exact redirect hostname, Origin validation (must be our domain). Stores full SHA-256 of API key (not fingerprint) in auth code. - authorization_code + refresh_token grants in /oauth/token Both use GETDEL for atomic single-use consumption (no race condition). Refresh tokens carry family_id for future family invalidation. Returns 401 invalid_client when DCR client is expired (triggers re-reg). Security improvements: - verifyPkceS256() validates code_verifier format before any SHA-256 work; returns null=invalid_request vs false=invalid_grant. - Full SHA-256 (64 hex) stored for new OAuth tokens; legacy client_credentials keeps 16-char fingerprint (backward compat). - Discovery doc: only authorization_code + refresh_token advertised. - Protected resource metadata: /.well-known/oauth-protected-resource - WWW-Authenticate headers include resource_metadata param. - HEAD /mcp returns 200 (Anthropic probe compatibility). - Origin validation on POST /mcp: claude.ai/claude.com + absent allowed. - ping method + tool annotations (readOnlyHint, openWorldHint). - api/oauth/ subdir added to edge-function module isolation scan. * fix(oauth): distinguish Redis unavailable from key-miss; fix retry nonce; extend client TTL on token use P1: redisGetDel and redisGet in token.js and authorize.js now throw on transport/HTTP errors instead of swallowing them as null. Callers catch and return 503 so clients know to retry, not discard valid codes/tokens. Key-miss (result=null from Redis) still returns null as before. P2a: Invalid API key retry path now generates and stores a fresh nonce before re-rendering the consent form. Previously the new nonce was never persisted, causing the next submit to fail with "Session Expired" and forcing the user to restart the entire OAuth flow on a single typo. P2b: token.js now extends the client TTL (EXPIRE, fire-and-forget) after a successful client lookup in both authorization_code and refresh_token paths. CLIENT_TTL_SECONDS was defined but unused — clients that only refresh tokens would expire after 90 days despite continuous use. * fix(oauth): atomic nonce consumption via GETDEL; fail closed on nonce storage failure P2a: Nonce storage result is now checked before rendering the consent page (both initial GET and invalid-key retry path). If redisSet returns false (storage unavailable), we return a 503-style error page instead of rendering a form the user cannot submit successfully. P2b: CSRF nonce is now consumed atomically via GETDEL instead of a read-then-fire-and-forget-delete. Two concurrent POST submits can no longer both pass validation before the delete lands, and the delete is no longer vulnerable to edge runtime isolate teardown. |
||
|
Elie Habib
|
14a31c4283 |
feat(mcp): OAuth 2.0 Authorization Server for claude.ai connector (#2418)
* feat(mcp): add OAuth 2.0 Authorization Server for claude.ai connector Implements spec-compliant MCP authentication so claude.ai's remote connector (which requires OAuth Client ID + Secret, no custom headers) can authenticate. - public/.well-known/oauth-authorization-server: RFC 8414 discovery document - api/oauth/token.js: client_credentials grant, issues UUID Bearer token in Redis TTL 3600s - api/_oauth-token.js: resolveApiKeyFromBearer() looks up token in Redis - api/mcp.ts: 3-tier auth (Bearer OAuth first, then ?key=, then X-WorldMonitor-Key); switch to getPublicCorsHeaders; surface error messages in catch - vercel.json: rewrite /oauth/token, exclude oauth from SPA, CORS headers - tests: update SPA no-cache pattern Supersedes PR #2417. Usage: URL=worldmonitor.app/mcp, Client ID=worldmonitor, Client Secret=<API key> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: fix markdown lint in OAuth plan (blank lines around lists) * fix(oauth): address all P1+P2 code review findings for MCP OAuth endpoint - Add per-IP rate limiting (10 req/min) to /oauth/token via Upstash slidingWindow - Return HTTP 401 + WWW-Authenticate header when Bearer token is invalid/expired - Add Cache-Control: no-store + Pragma: no-cache to token response (RFC 6749 §5.1) - Simplify _oauth-token.js to delegate to readJsonFromUpstash (removes duplicated Redis boilerplate) - Remove dead code from token.js: parseBasicAuth, JSON body path, clientId/issuedAt fields - Add Content-Type: application/json header for /.well-known/oauth-authorization-server - Remove response_types_supported (only applies to authorization endpoint, not client_credentials) Closes: todos 075, 076, 077, 078, 079 🤖 Generated with claude-sonnet-4-6 via Claude Code (https://claude.ai/claude-code) + Compound Engineering v2.40.0 Co-Authored-By: claude-sonnet-4-6 (200K context) <noreply@anthropic.com> * chore(review): fresh review findings — todos 081-086, mark 075/077/078/079 complete * fix(mcp): remove ?key= URL param auth + mask internal errors - Remove ?key= query param auth path — API keys in URLs appear in Vercel/CF access logs, browser history, Referer headers. OAuth client_credentials (same PR) already covers clients that cannot set custom headers. Only two auth paths remain: Bearer OAuth and X-WorldMonitor-Key header. - Revert err.message disclosure: catch block was accidentally exposing internal service URLs/IPs via err.message. Restore original hardcoded string, add console.error for server-side visibility. Resolves: todos 081, 082 * fix(oauth): resolve all P2/P3 review findings (todos 076, 080, 083-086) - 076: no-credentials path in mcp.ts now returns HTTP 401 + WWW-Authenticate instead of rpcError (200) - 080: store key fingerprint (sha256 first 16 hex chars) in Redis, not plaintext key - 083: replace Array.includes() with timingSafeIncludes() (constant-time HMAC comparison) in token.js and mcp.ts - 084: resolveApiKeyFromBearer uses direct fetch that throws on Redis errors (500 not 401 on infra failure) - 085: token.js imports getClientIp, getPublicCorsHeaders, jsonResponse from shared helpers; removes local duplicates - 086: mcp.ts auth chain restructured to check Bearer header first, passes token string to resolveApiKeyFromBearer (eliminates double header read + unconditional await) * test(mcp): update auth test to expect HTTP 401 for missing credentials Align with todo 076 fix: no-credentials path now returns 401 + WWW-Authenticate instead of JSON-RPC 200 response. Also asserts WWW-Authenticate header presence. * chore: mark todos 076, 080, 083-086 complete * fix(mcp): harden OAuth error paths and fix rate limit cross-user collision - Wrap resolveApiKeyFromBearer() in try/catch in mcp.ts; Redis/network errors now return 503 + Retry-After: 5 instead of crashing the handler - Wrap storeToken() fetch in try/catch in oauth/token.js; network errors return false so the existing if (!stored) path returns 500 cleanly - Re-key token endpoint rate limit by sha256(clientSecret).slice(0,8) instead of IP; prevents cross-user 429s when callers share Anthropic's shared outbound IPs (Claude remote MCP connector) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Elie Habib
|
e9cbbc65d5 |
fix(mcp): use public CORS (*) so claude.ai and external MCP clients connect (#2416)
getCorsHeaders() restricts ACAO to worldmonitor.app origins only. claude.ai sends Origin: https://claude.ai which fails the isDisallowedOrigin check and gets ACAO: https://worldmonitor.app back — browser rejects it. MCP is secured by API key (validateApiKey forceKey:true), not by origin. Switch to getPublicCorsHeaders() (ACAO: *) and remove isDisallowedOrigin block so any authenticated MCP client (claude.ai, Claude Desktop, agents) can connect. |
||
|
Elie Habib
|
274e6c705e |
fix(mcp): fix get_country_brief snake_case + get_world_brief provider (#2412)
* fix(mcp): fix get_country_brief snake_case + get_world_brief provider Two bugs in api/mcp.ts: 1. get_country_brief: sent `countryCode` (camelCase) to get-country-intel-brief but the sebuf proto deserializer expects `country_code` (snake_case). Result: req.countryCode was always empty string → returned empty brief. 2. get_world_brief: hardcoded `provider: 'groq'` in summarize-article body. GROQ was failing → fallback:true, empty summary. Fix: switch to `openrouter` which is the reliable production provider. * fix(mcp): address review findings — 5 fixes 1. get_world_brief: groq → openrouter (groq failing, no fallback chain) 2. get_world_brief: tighten timeout 8s+20s → 6s+18s to stay under 30s edge ceiling 3. get_country_brief: countryCode → country_code (proto deserializer expects snake_case) 4. analyze_situation: expose framework field in schema + forward to deduct-situation 5. tools/call error handler: surface err.message instead of swallowing all details |
||
|
Elie Habib
|
7100b56fdc |
feat(mcp): add 5 AI inference tools to MCP server (#2387)
* chore: redeploy to pick up WORLDMONITOR_VALID_KEYS fix * feat(mcp): add 5 AI inference tools — WorldBrief, CountryBrief, DeductionPanel, Forecasts, Social Add RpcToolDef discriminated union alongside CacheToolDef so the MCP can support both Redis cache reads and live LLM inference calls. New tools (22 total, up from 17): - get_social_velocity: Reddit geopolitical signals (cache read, intelligence:social:reddit:v1) - get_world_brief: fetches live headlines via list-feed-digest then calls summarize-article LLM; optional geo_context to focus the brief - get_country_brief: calls get-country-intel-brief LLM endpoint; supports analytical framework injection (PR #2380 compatible) - analyze_situation: calls deduct-situation for open-ended AI geopolitical reasoning with query + optional context - generate_forecasts: calls get-forecasts for fresh AI forecast generation (distinct from get_forecast_predictions which reads pre-computed cache) The _execute path makes internal fetch calls using new URL(req.url).origin and forwards the PRO API key header. * fix(mcp): address Greptile review — timeouts, User-Agent, test coverage - generate_forecasts: reduce AbortSignal timeout 60s → 25s (Vercel Edge hard ceiling ~30s) - get_world_brief: reduce timeouts 10s+30s → 8s+20s (stays under 30s) - Add User-Agent header to all _execute internal RPC fetches - Move get_social_velocity out of AI inference comment block (it's a cache tool) - Assert _execute not leaked in tools/list alongside existing _cacheKeys check |
||
|
Elie Habib
|
bca939c423 |
feat(mcp): PRO MCP server — WorldMonitor data via Model Context Protocol (#2382)
* feat(mcp): PRO MCP server — WorldMonitor data via Model Context Protocol Adds api/mcp.ts: a Vercel edge function implementing MCP Streamable HTTP transport (protocol 2025-03-26) for PRO API key holders. PRO users point Claude Desktop (or any MCP client) at https://api.worldmonitor.app/mcp with their X-WorldMonitor-Key header and get 17 tools covering all major WorldMonitor data domains. - Auth: validateApiKey(forceKey:true) — returns JSON-RPC -32001 on failure - Rate limit: 60 calls/min per API key via Upstash sliding window (rl:mcp prefix) - Tools read from existing Redis bootstrap cache (no upstream API calls) - Each response includes cached_at (ISO timestamp) and stale (boolean) - tools/list served from in-memory registry (<500ms, no Redis) - Tool calls with warm cache respond in <800ms - 11 tests covering auth, rate limit, protocol, tools/list, tools/call * fix(mcp): handle Redis fetch throws + fix confusing label derivation P1: wrap executeTool call in tools/call handler with try-catch so TimeoutError/TypeError from AbortSignal.timeout in readJsonFromUpstash returns JSON-RPC -32603 instead of an unhandled rejection → 500. Mirrors the same guard already on the rate-limiter call above it. P2: walk backwards through cache key segments, skipping version tags (v\d+), bare numbers, 'stale', and 'sebuf' suffixes to find the first meaningful label. Fixes 'economic:fred:v1:FEDFUNDS:0' → 'FEDFUNDS' and 'risk:scores:sebuf:stale:v1' / 'theater_posture:sebuf:stale:v1' both resolving to 'stale' (which collides with the top-level stale flag). Adds test for P1 scenario (12 tests total, all pass). |