mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
34dfc9a451f3f50f5235fa10b17521b0954a25ba
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
34dfc9a451 |
fix(news): ground LLM surfaces on real RSS description end-to-end (#3370)
* feat(news/parser): extract RSS/Atom description for LLM grounding (U1)
Add description field to ParsedItem, extract from the first non-empty of
description/content:encoded (RSS) or summary/content (Atom), picking the
longest after HTML-strip + entity-decode + whitespace-normalize. Clip to
400 chars. Reject empty, <40 chars after strip, or normalize-equal to the
headline — downstream consumers fall back to the cleaned headline on '',
preserving current behavior for feeds without a description.
CDATA end is anchored to the closing tag so internal ]]> sequences do not
truncate the match. Preserves cached rss:feed:v1 row compatibility during
the 1h TTL bleed since the field is additive.
Part of fix: pipe RSS description end-to-end so LLM surfaces stop
hallucinating named actors (docs/plans/2026-04-24-001-...).
Covers R1, R7.
* feat(news/story-track): persist description on story:track:v1 HSET (U2)
Append description to the story:track:v1 HSET only when non-empty. Additive
— no key version bump. Old rows and rows from feeds without a description
return undefined on HGETALL, letting downstream readers fall back to the
cleaned headline (R6).
Extract buildStoryTrackHsetFields as a pure helper so the inclusion gate is
unit-testable without Redis.
Update the contract comment in cache-keys.ts so the next reader of the
schema sees description as an optional field.
Covers R2, R6.
* feat(proto): NewsItem.snippet + SummarizeArticleRequest.bodies (U3)
Add two additive proto fields so the article description can ride to every
LLM-adjacent consumer without a breaking change:
- NewsItem.snippet (field 12): RSS/Atom description, HTML-stripped,
≤400 chars, empty when unavailable. Wired on toProtoItem.
- SummarizeArticleRequest.bodies (field 8): optional article bodies
paired 1:1 with headlines for prompt grounding. Empty array is today's
headline-only behavior.
Regenerated TS client/server stubs and OpenAPI YAML/JSON via sebuf v0.11.1
(PATH=~/go/bin required — Homebrew's protoc-gen-openapiv3 is an older
pre-bundle-mode build that collides on duplicate emission).
Pre-emptive bodies:[] placeholders at the two existing SummarizeArticle
call sites in src/services/summarization.ts; U6 replaces them with real
article bodies once SummarizeArticle handler reads the field.
Covers R3, R5.
* feat(brief/digest): forward RSS description end-to-end through brief envelope (U4)
Digest accumulator reader (seed-digest-notifications.mjs::buildDigest) now
plumbs the optional `description` field off each story:track:v1 HGETALL into
the digest story object. The brief adapter (brief-compose.mjs::
digestStoryToUpstreamTopStory) prefers the real RSS description over the
cleaned headline; when the upstream row has no description (old rows in the
48h bleed, feeds that don't carry one), we fall back to the cleaned headline
so today behavior is preserved (R6).
This is the upstream half of the description cache path. U5 lands the LLM-
side grounding + cache-prefix bump so Gemini actually sees the article body
instead of hallucinating a named actor from the headline.
Covers R4 (upstream half), R6.
* feat(brief/llm): RSS grounding + sanitisation + 4 cache prefix bumps (U5)
The actual fix for the headline-only named-actor hallucination class:
Gemini 2.5 Flash now receives the real article body as grounding context,
so it paraphrases what the article says instead of filling role-label
headlines from parametric priors ("Iran's new supreme leader" → "Ali
Khamenei" was the 2026-04-24 reproduction; with grounding, it becomes
the actual article-named actor).
Changes:
- buildStoryDescriptionPrompt interpolates a `Context: <body>` line
between the metadata block and the "One editorial sentence" instruction
when description is non-empty AND not normalise-equal to the headline.
Clips to 400 chars as a second belt-and-braces after the U1 parser cap.
No Context line → identical prompt to pre-fix (R6 preserved).
- sanitizeStoryForPrompt extended to cover `description`. Closes the
asymmetry where whyMatters was sanitised and description wasn't —
untrusted RSS bodies now flow through the same injection-marker
neutraliser before prompt interpolation. generateStoryDescription wraps
the story in sanitizeStoryForPrompt before calling the builder,
matching generateWhyMatters.
- Four cache prefixes bumped atomically to evict pre-grounding rows:
scripts/lib/brief-llm.mjs:
brief:llm:description:v1 → v2 (Railway, description path)
brief:llm:whymatters:v2 → v3 (Railway, whyMatters fallback)
api/internal/brief-why-matters.ts:
brief:llm:whymatters:v6 → v7 (edge, primary)
brief:llm:whymatters:shadow:v4 → shadow:v5 (edge, shadow)
hashBriefStory already includes description in the 6-field material
(v5 contract) so identity naturally drifts; the prefix bump is the
belt-and-braces that guarantees a clean cold-start on first tick.
- Tests: 8 new + 2 prefix-match updates on tests/brief-llm.test.mjs.
Covers Context-line injection, empty/dup-of-headline rejection,
400-char clip, sanitisation of adversarial descriptions, v2 write,
and legacy-v1 row dark (forced cold-start).
Covers R4 + new sanitisation requirement.
* feat(news/summarize): accept bodies + bump summary cache v5→v6 (U6)
SummarizeArticle now grounds on per-headline article bodies when callers
supply them, so the dashboard "News summary" path stops hallucinating
across unrelated headlines when the upstream RSS carried context.
Three coordinated changes:
1. SummarizeArticleRequest handler reads req.bodies, sanitises each entry
through sanitizeForPrompt (same trust treatment as geoContext — bodies
are untrusted RSS text), clips to 400 chars, and pads to the headlines
length so pair-wise identity is stable.
2. buildArticlePrompts accepts optional bodies and interleaves a
` Context: <body>` line under each numbered headline that has a
non-empty body. Skipped in translate mode (headline[0]-only) and when
all bodies are empty — yielding a byte-identical prompt to pre-U6
for every current caller (R6 preserved).
3. summary-cache-key bumps CACHE_VERSION v5→v6 so the pre-grounding rows
(produced from headline-only prompts) cold-start cleanly. Extends
canonicalizeSummaryInputs + buildSummaryCacheKey with a pair-wise
bodies segment `:bd<hash>`; the prefix is `:bd` rather than `:b` to
avoid colliding with `:brief:` when pattern-matching keys. Translate
mode is headline[0]-only and intentionally does not shift on bodies.
Dedup reorder preserved: the handler re-pairs bodies to the deduplicated
top-5 via findIndex, so layout matches without breaking cache identity.
New tests: 7 on buildArticlePrompts (bodies interleave, partial fill,
translate-mode skip, clip, short-array tolerance), 8 on
buildSummaryCacheKey (pair-wise sort, cache-bust on body drift, translate
skip). Existing summary-cache-key assertions updated v5→v6.
Covers R3, R4.
* feat(consumers): surface RSS snippet across dashboard, email, relay, MCP + audit (U7)
Thread the RSS description from the ingestion path (U1-U5) into every
user-facing LLM-adjacent surface. Audit the notification producers so
RSS-origin and domain-origin events stay on distinct contracts.
Dashboard (proto snippet → client → panel):
- src/types/index.ts NewsItem.snippet?:string (client-side field).
- src/app/data-loader.ts proto→client mapper propagates p.snippet.
- src/components/NewsPanel.ts renders snippet as a truncated (~200 chars,
word-boundary ellipsis) `.item-snippet` line under each headline.
- NewsPanel.currentBodies tracks per-headline bodies paired 1:1 with
currentHeadlines; passed as options.bodies to generateSummary so the
server-side SummarizeArticle LLM grounds on the article body.
Summary plumbing:
- src/services/summarization.ts threads bodies through SummarizeOptions
→ generateSummary → runApiChain → tryApiProvider; cache key now includes
bodies (via U6's buildSummaryCacheKey signature).
MCP world-brief:
- api/mcp.ts pairs headlines with their RSS snippets and POSTs `bodies`
to /api/news/v1/summarize-article so the MCP tool surface is no longer
starved.
Email digest:
- scripts/seed-digest-notifications.mjs plain-text formatDigest appends
a ~200-char truncated snippet line under each story; HTML formatDigestHtml
renders a dim-grey description div between title and meta. Both gated
on non-empty description (R6 — empty → today's behavior).
Real-time alerts:
- src/services/breaking-news-alerts.ts BreakingAlert gains optional
description; checkBatchForBreakingAlerts reads item.snippet; dispatchAlert
includes `description` in the /api/notify payload when present.
Notification relay:
- scripts/notification-relay.cjs formatMessage gated on
NOTIFY_RELAY_INCLUDE_SNIPPET=1 (default off). When on, RSS-origin
payloads render a `> <snippet>` context line under the title. When off
or payload.description absent, output is byte-identical to pre-U7.
Audit (RSS vs domain):
- tests/notification-relay-payload-audit.test.mjs enforces file-level
@notification-source tags on every producer, rejects `description:` in
domain-origin payload blocks, and verifies the relay codepath gates
snippet rendering under the flag.
- Tag added to ais-relay.cjs (domain), seed-aviation.mjs (domain),
alert-emitter.mjs (domain), breaking-news-alerts.ts (rss).
Deferred (plan explicitly flags): InsightsPanel + cluster-producer
plumbing (bodies default to [] — will unlock gradually once news:insights:v1
producer also carries primarySnippet).
Covers R5, R6.
* docs+test: grounding-path note + bump pinned CACHE_VERSION v5→v6 (U8)
Final verification for the RSS-description-end-to-end fix:
- docs/architecture.mdx — one-paragraph "News Grounding Pipeline"
subsection tracing parser → story:track:v1.description → NewsItem.snippet
→ brief / SummarizeArticle / dashboard / email / relay / MCP, with the
empty-description R6 fallback rule called out explicitly.
- tests/summarize-reasoning.test.mjs — Fix-4 static-analysis pin updated
to match the v6 bump from U6. Without this the summary cache bump silently
regressed CI's pinned-version assertion.
Final sweep (2026-04-24):
- grep -rn 'brief:llm:description:v1' → only in the U5 legacy-row test
simulation (by design: proves the v2 bump forces cold-start).
- grep -rn 'brief:llm:whymatters:v2/v6/shadow:v4' → no live references.
- grep -rn 'summary:v5' → no references.
- CACHE_VERSION = 'v6' in src/utils/summary-cache-key.ts.
- Full tsx --test sweep across all tests/*.test.{mjs,mts}: 6747/6747 pass.
- npm run typecheck + typecheck:api: both clean.
Covers R4, R6, R7.
* fix(rss-description): address /ce:review findings before merge
14 fixes from structured code review across 13 reviewer personas.
Correctness-critical (P1 — fixes that prevent R6/U7 contract violations):
- NewsPanel signature covers currentBodies so view-mode toggles that leave
headlines identical but bodies different now invalidate in-flight summaries.
Without this, switching renderItems → renderClusters mid-summary let a
grounded response arrive under a stale (now-orphaned) cache key.
- summarize-article.ts re-pairs bodies with headlines BEFORE dedup via a
single zip-sanitize-filter-dedup pass. Previously bodies[] was indexed by
position in light-sanitized headlines while findIndex looked up the
full-sanitized array — any headline that sanitizeHeadlines emptied
mispaired every subsequent body, grounding the LLM on the wrong story.
- Client skips the pre-chain cache lookup when bodies are present, since
client builds keys from RAW bodies while server sanitizes first. The
keys diverge on injection content, which would silently miss the
server's authoritative cache every call.
Test + audit hardening:
- Legacy v1 eviction test now uses the real hashBriefStory(story()) suffix
instead of a literal "somehash", so a bug where the reader still queried
the v1 prefix at the real key would actually be caught.
- tests/summary-cache-key.test.mts adds 400-char clip identity coverage so
the canonicalizer's clip and any downstream clip can't silently drift.
- tests/news-rss-description-extract.test.mts renames the well-formed
CDATA test and adds a new test documenting the malformed-]]> fallback
behavior (plain regex captures, article content survives).
Safe_auto cleanups:
- Deleted dead SNIPPET_PUSH_MAX constant in notification-relay.cjs.
- BETA-mode groq warm call now passes bodies, warming the right cache slot.
- seed-digest shares a local normalize-equality helper for description !=
headline comparison, matching the parser's contract.
- Pair-wise sort in summary-cache-key tie-breaks on body so duplicate
headlines produce stable order across runs.
- buildSummaryCacheKey gained JSDoc documenting the client/server contract
and the bodies parameter semantics.
- MCP get_world_brief tool description now mentions RSS article-body
grounding so calling agents see the current contract.
- _shared.ts `opts.bodies![i]!` double-bang replaced with `?? ''`.
- extractRawTagBody regexes cached in module-level Map, mirroring the
existing TAG_REGEX_CACHE pattern.
Deferred to follow-up (tracked for PR description / separate issue):
- Promote shared MAX_BODY constant across the 5 clip sites
- Promote shared truncateForDisplay helper across 4 render sites
- Collapse NewsPanel.{currentHeadlines, currentBodies} → Array<{title, snippet}>
- Promote sanitizeStoryForPrompt to shared/brief-llm-core.js
- Split list-feed-digest.ts parser helpers into sibling -utils.ts
- Strengthen audit test: forward-sweep + behavioral gate test
Tests: 6749/6749 pass. Typecheck clean on both configs.
* fix(summarization): thread bodies through browser T5 path (Codex #2)
Addresses the second of two Codex-raised findings on PR #3370:
The PR threaded bodies through the server-side API provider chain
(Ollama → Groq → OpenRouter → /api/news/v1/summarize-article) but the
local browser T5 path at tryBrowserT5 was still summarising from
headlines alone. In BETA_MODE that ungrounded path runs BEFORE the
grounded server providers; in normal mode it remains the last
fallback. Whenever T5-small won, the dashboard summary surface
regressed to the headline-only path — the exact hallucination class
this PR exists to eliminate.
Fix: tryBrowserT5 accepts an optional `bodies` parameter and
interleaves each body with its paired headline via a `headline —
body` separator in the combined text (clipped to 200 chars per body
to stay within T5-small's ~512-token context window). All three call
sites (BETA warm, BETA cold, normal-mode fallback) now pass the
bodies threaded down from generateSummary options.bodies.
When bodies is empty/omitted, the combined text is byte-identical to
pre-fix (R6 preserved).
On Codex finding #1 (story:track:v1 additive-only HSET keeps a body
from an earlier mention of the same normalized title), declining to
change. The current rule — "if this mention has a body, overwrite;
otherwise leave the prior body alone" — is defensible: a body from
mention A is not falsified by mention B being body-less (a wire
reprint doesn't invalidate the original source's body). A feed that
publishes a corrected headline creates a new normalized-title hash,
so no stale body carries forward. The failure window is narrow (live
story evolving while keeping the same title through hours of
body-less wire reprints) and the 7-day STORY_TTL is the backstop.
Opening a follow-up issue to revisit semantics if real-world evidence
surfaces a stale-grounding case.
* fix(story-track): description always-written to overwrite stale bodies (Codex #1)
Revisiting Codex finding #1 on PR #3370 after re-review. The previous
response declined the fix with reasoning; on reflection the argument
was over-defending the current behavior.
Problem: buildStoryTrackHsetFields previously wrote `description` only
when non-empty. Because story:track:v1 rows are collapsed by
normalized-title hash, an earlier mention's body would persist for up
to STORY_TTL (7 days) on subsequent body-less mentions of the same
story. Consumers reading `track.description` via HGETALL could not
distinguish "this mention's body" from "some mention's body from the
last week," silently grounding brief / whyMatters / SummarizeArticle
LLMs on text the current mention never supplied. That violates the
grounding contract advertised to every downstream surface in this PR.
Fix: HSET `description` unconditionally on every mention — empty
string when the current item has no body, real body when it does. An
empty value overwrites any prior mention's body so the row is always
authoritative for the current cycle. Consumers continue to treat
empty description as "fall back to cleaned headline" (R6 preserved).
The 7-day STORY_TTL and normalized-title hash semantics are unchanged.
Trade-off accepted: a valid body from Feed A (NYT) is wiped when Feed
B (AP body-less wire reprint) arrives for the same normalized title,
even though Feed A's body is factually correct. Rationale: the
alternative — keeping Feed A's body indefinitely — means the user
sees Feed A's body attributed (by proximity) to an AP mention at a
later timestamp, which is at minimum misleading and at worst carries
retracted/corrected details. Honest absence beats unlabeled presence.
Tests: new stale-body overwrite sequence test (T0 body → T1 empty →
T2 new body), existing "writes description when non-empty" preserved,
existing "omits when empty" inverted to "writes empty, overwriting."
cache-keys.ts contract comment updated to mark description as
always-written rather than optional.
|
||
|
Elie Habib
|
dcf73385ca |
fix(scoring): rebalance formula weights severity 55%, corroboration 15% (#3144)
* fix(scoring): rebalance formula weights severity 55%, corroboration 15%
PR A of the scoring recalibration plan (docs/plans/2026-04-17-002).
The v2 shadow-log recalibration (690 items, Pearson 0.413) showed the
formula compresses scores into a narrow 30-70 range, making the 85
critical gate unreachable and the 65 high gate marginal. Root cause:
corroboration at 30% weight penalizes breaking single-source news
(the most important alerts) while severity at 40% doesn't separate
critical from high enough.
Weight change:
BEFORE: severity 0.40 + sourceTier 0.20 + corroboration 0.30 + recency 0.10
AFTER: severity 0.55 + sourceTier 0.20 + corroboration 0.15 + recency 0.10
Expected effect: critical/tier1/fresh rises from 76 to 88 (clears 85
gate). critical/tier2/fresh rises from 71 to 83 (recommend lowering
critical gate to 80 at activation time). high/tier2/fresh rises from
61 to 69 (clears 65 gate). The HIGH-CRITICAL gap widens from 10 to
14 points for same-tier items.
Also:
- Bumps shadow-log key from v2 to v3 for a clean recalibration dataset
(v2 had old-weight scores that would contaminate the 48h soak)
- Updates proto/news_item.proto formula comment to reflect new weights
- Updates cache-keys.ts documentation
No cache migration needed: the classify cache stores {level, category},
not scores. Scores are computed at read time from the stored level +
the formula, so new digest requests immediately produce new scores.
Gates remain OFF. After 48h of v3 data, re-run:
node scripts/shadow-score-report.mjs
node scripts/shadow-score-rank.mjs sample 25
🤖 Generated with Claude Opus 4.6 via Claude Code + Compound Engineering v2.49.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: regenerate proto OpenAPI docs for weight rebalance
* fix(scoring): bump SHADOW_SCORE_LOG_KEY export to v3
The exported constant in cache-keys.ts was left at v2 while the relay's
local constant was bumped to v3. Anyone importing the export (or grep-
discovering it) would get a stale key. Architecture review flagged this.
* fix(scoring): update test + stale comments for shadow-log v3
Review found the regression test still asserted v2 key, causing CI
failure. Also fixed stale v1/v2 references in report script header,
default-key comment, report title render, and shouldNotify docstring.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
Elie Habib
|
6c017998d3 |
feat(e3): story persistence tracking (#2620)
* feat(e3): story persistence tracking
Adds cross-cycle story tracking layer to the RSS digest pipeline:
- Proto: StoryMeta message + StoryPhase enum on NewsItem (fields 9-11).
importanceScore and corroborationCount stubs added for E1.
- list-feed-digest.ts: builds corroboration map across ALL items before
truncation; batch-reads existing story:track hashes from Redis; writes
HINCRBY/HSET/HSETNX/SADD/EXPIRE per story in 80-story pipeline chunks;
attaches StoryMeta (firstSeen, mentionCount, sourceCount, phase) to
each proto item using read-back data.
- cache-keys.ts: STORY_TRACK_KEY_PREFIX, STORY_SOURCES_KEY_PREFIX,
DIGEST_ACCUMULATOR_KEY_PREFIX, STORY_TRACKING_TTL_S.
- src/types/index.ts: StoryMeta, StoryPhase, NewsItem extended.
- data-loader.ts: protoItemToNewsItem maps STORY_PHASE_* → client phase.
- NewsPanel.ts: BREAKING/DEVELOPING/ONGOING phase badges in item rows.
New story first appearance: phase=BREAKING. After 2 mentions within 2h:
DEVELOPING. After 6+ mentions or >2h: SUSTAINED. If score drops below
50% of peak: FADING (used by E1; defaults to SUSTAINED for now).
Redis keys per story (48h TTL):
story:track:v1:<hash16> → hash (firstSeen,lastSeen,mentionCount,...)
story:sources:v1:<hash16> → set (feed names, for cross-source count)
* fix(e3): correct storyMeta staleness and mentionCount semantics
P1 — storyMeta was always one cycle behind because storyTracks was read
before writeStoryTracking ran. Fix: keep read-before-write but compute
storyMeta from merged in-memory state (stale.mentionCount + 1, fresh
sourceCount from corroborationMap). New stories get mentionCount=1 and
phase=BREAKING in the same cycle they first appear — no extra Redis
round-trip needed.
P2 — mentionCount incremented once per item occurrence, so a story seen
in 3 sources in its first cycle was immediately stored as mentionCount=3.
Fix: deduplicate by titleHash in writeStoryTracking so each unique story
gets exactly one HINCRBY per digest cycle regardless of source count.
SADD still collects all sources for the set key.
* fix(e3): Unicode hash collision, ALERT badge regression, FADING comment
P1 — normalizeTitle used [^\w\s] without the u flag; \w is ASCII-only
so every Arabic/CJK/Cyrillic title stripped to "" and shared one Redis
hash. Fixed: use /[^\p{L}\p{N}\s]/gu (Unicode property escapes require
the u flag).
P1 — ALERT badge was gated on !item.storyMeta, suppressing the indicator
for any tracked story regardless of isAlert. Phase and alert are
orthogonal signals; ALERT now renders unconditionally when isAlert=true.
P2 — FADING branch is intentionally inactive until E1 ships real scores
(currentScore/peakScore placeholder 0 via HSETNX). Added comment to
document the intentional ordering.
* fix(news-alerts): skip sustained/fading stories in breaking alert selectBest
Sustained and fading story phases are already well-covered by the feed;
only breaking and developing phases warrant a banner interrupt. Items
without storyMeta (phase unspecified) pass through unchanged.
Fixes gap C from docs/plans/2026-04-02-003-fix-news-alerts-pr-gaps-plan.md
* fix(e3): remove rebase artifacts from list-feed-digest
Removes a stray closing brace, duplicate ASCII normalizeTitle
(Unicode-aware version from the fix commit is correct), and
a leftover storyPhase assignment that references a removed field.
All typecheck and typecheck:api pass clean.
|
||
|
Elie Habib
|
8d8cf56ce2 |
feat(scoring): composite importance score + story tracking infrastructure (#2604)
* feat(scoring): add composite importance score + story tracking infrastructure
- Extract SOURCE_TIERS/getSourceTier to server/_shared/source-tiers.ts so
server handlers can import it without pulling in client-only modules;
src/config/feeds.ts re-exports for backward compatibility
- Add story tracking Redis key helpers to cache-keys.ts
(story:track:v1, story:sources:v1, story:peak:v1, digest:accumulator:v1)
- Export SEVERITY_SCORES from _classifier.ts for server-side score math
- Add upstashPipeline() to redis.ts for arbitrary batched Redis writes
- Add importanceScore/corroborationCount/storyPhase fields to proto,
generated TS, and src/types NewsItem
- Add StoryMeta message and StoryPhase enum to proto
- In list-feed-digest.ts:
- Build corroboration map across full corpus BEFORE per-category truncation
- Compute importanceScore (severity 40% + tier 20% + corroboration 30%
+ recency 10%) per item
- Sort by importanceScore desc before truncating at MAX_ITEMS_PER_CATEGORY
- Write story:track / story:sources / story:peak / digest:accumulator
to Redis in 80-story pipeline batches after each digest build
Score gate in notification-relay.cjs follows in the next PR (shadow mode,
behind IMPORTANCE_SCORE_LIVE flag). RELAY_GATES_READY removal of
/api/notify comes after 48h shadow comparison confirms parity.
* fix(scoring): add storyPhase field + regenerate proto types
- Add storyPhase to ParsedItem and toProtoItem (defaults UNSPECIFIED)
- Regenerate service_server.ts: required fields, StoryPhase type relocated
- Regenerate service_client.ts and OpenAPI docs from buf generate
- Fix typecheck:api error on missing required storyPhase in NewsItem
* fix(scoring): address all code review findings from PR #2604
P1:
- await writeStoryTracking instead of fire-and-forget to prevent
silent data loss on edge isolate teardown
- remove duplicate upstashPipeline; use existing runRedisPipeline
- strip non-https links before Redis write (XSS prevention)
- implement storyPhase read path: HGETALL batch + computePhase()
so BREAKING/DEVELOPING/SUSTAINED/FADING badges are now live
P2/P3:
- extend STORY_TTL 48h → 7 days (sustained stories no longer reset)
- extract SCORE_WEIGHTS named constants with rationale comment
- move SEVERITY_SCORES out of _classifier.ts into list-feed-digest.ts
- add normalizeTitle comment referencing todo #102
- pre-compute title hashes once, share between phase read + write
* fix(scoring): correct enrichment-before-scoring and write-before-read ordering
Two sequencing bugs:
1. enrichWithAiCache ran after truncation (post-slice), so items whose
threat level was upgraded by the LLM cache could have already been
cut from the top-20, and downgraded items kept inflated scores.
Fix: enrich ALL items from the full corpus before scoring, so
importanceScore always uses the final post-LLM classification level.
2. Phase HGETALL read happened before writeStoryTracking, meaning
first-time stories had no Redis entry and always returned UNSPECIFIED
instead of BREAKING, and all existing stories lagged one cycle behind.
Fix: write tracking first, then read back for phase assignment.
|
||
|
Elie Habib
|
110ab402c4 |
feat(intelligence): analytical framework selector for AI panels (#2380)
* feat(frameworks): add settings section and import modal - Add Analysis Frameworks group to preferences-content.ts between Intelligence and Media sections - Per-panel active framework display (read-only, 4 panels) - Skill library list with built-in badge, Rename and Delete actions for imported frameworks - Import modal with two tabs: From agentskills.io (fetch + preview) and Paste JSON - All error cases handled inline: network, domain validation, missing instructions, invalid JSON, duplicate name, instructions too long, rate limit - Add api/skills/fetch-agentskills.ts edge function (proxy to agentskills.io) - Add analysis-framework-store.ts (loadFrameworkLibrary, saveImportedFramework, deleteImportedFramework, renameImportedFramework, getActiveFrameworkForPanel) - Add fw-* CSS classes to main.css matching dark panel aesthetic * feat(panels): wire analytical framework store into InsightsPanel, CountryDeepDive, DailyMarketBrief, DeductionPanel - InsightsPanel: append active framework to geoContext in updateFromClient(); subscribe in constructor, unsubscribe in destroy() - CountryIntelManager: pass framework as query param to fetchCountryIntelBrief(); subscribe to re-open brief on framework change; unsubscribe in destroy() - DataLoaderManager: add dailyBriefGeneration counter for stale-result guard; pass frameworkAppend to buildDailyMarketBrief(); subscribe to framework changes to force refresh; unsubscribe in destroy() - daily-market-brief service: add frameworkAppend? field to BuildDailyMarketBriefOptions; append to extendedContext before summarize call - DeductionPanel: append active framework to geoContext in handleSubmit() before RPC call * feat(frameworks): add FrameworkSelector UI component - Create FrameworkSelector component with premium/locked states - Premium: select dropdown with all framework options, change triggers setActiveFrameworkForPanel - Locked: disabled select + PRO badge, click calls showGatedCta(FREE_TIER) - InsightsPanel: adds asterisk note (client-generated analysis hint) - Wire into InsightsPanel, DailyMarketBriefPanel, DeductionPanel (via this.header) - Wire into CountryDeepDivePanel header right-side (no Panel base, panel=null) - Add framework-selector CSS to main.css * fix(frameworks): make new proto fields optional in generated types * fix(frameworks): extract firstMsg to satisfy strict null checks in tsconfig.api.json * fix(docs): add blank lines around lists/headings to pass markdownlint * fix(frameworks): add required proto string fields to call sites after make generate * chore(review): add code review todos 041-057 for PR #2380 7 review agents (TypeScript, Security, Architecture, Performance, Simplicity, Agent-Native, Learnings) identified 17 findings across 5 P1, 8 P2, and 4 P3 categories. |
||
|
Elie Habib
|
80cb7d5aa7 |
fix(cache): digest TTL alignment + slow-browser tier + feedStatuses trim (#1798)
* fix(cache): align Redis digest + RSS feed TTLs to CF CDN TTL
RSS feed TTL 600s → 3600s; digest TTL 900s → 3600s.
CF CDN caches at 3600s, so Redis expiring earlier caused every hourly
CF revalidation to hit a cold origin and run the full buildDigest()
pipeline (75 feeds, up to 25s). Aligning both to 3600s ensures CF
revalidation gets a warm Redis hit and returns immediately.
* fix(cache): emit only non-ok feedStatuses; update proto comment + make generate
Digest was emitting 'ok' for every successful feed (~50 entries, ~1-2KB
per response). No in-repo client reads feedStatuses values. Changed to
only emit 'empty' and 'timeout'; absent key implies ok.
Updated proto comment to document the absence-implies-ok contract and
ran make generate to regenerate docs/api/ OpenAPI files.
* fix(cache): add slow-browser tier; move digest route to it
New 'slow-browser' tier is identical to 'slow' but adds max-age=300,
letting browsers skip the network for 5 minutes. Without max-age,
browsers ignore s-maxage and send conditional If-None-Match on every
20-min poll — each costing 1 billable edge request even for 304s.
Scoped only to list-feed-digest (a safe polling endpoint). Premium
user-triggered endpoints (analyze-stock, backtest-stock) stay on 'slow'
where browser caching is inappropriate.
* test: regression tests for feedStatuses and slow-browser tier
- digest-no-reclassify: assert buildDigest does not write 'ok' to feedStatuses
- route-cache-tier: include slow-browser in tier regex; assert slow-browser
has max-age and slow tier does not
* fix(cache): add variant to per-feed RSS cache key
rss:feed:v1:${url} was shared across variants even though classifyByKeyword()
bakes variant-specific threat/category labels into the cached ParsedItem[].
Feeds shared between full and tech variants (Verge, Ars, HN, etc.) had
whichever variant populated the cache first control the other variant's
classifications for the full 3600s TTL — turning a pre-existing 10-minute
bleed-through into a 1-hour accuracy bug for the tech dashboard.
Fix: key is now rss:feed:v1:${variant}:${url}.
* fix(cache): bypass browser HTTP cache on digest fetch
max-age=300 on the slow-browser tier lets browsers serve the digest
from their HTTP cache for up to 5 minutes, including on explicit
in-app refresh (window.location.reload) or page reload after a
breaking event. Users would see stale data until the TTL expired.
Add cache: 'no-cache' to tryFetchDigest() so every fetch revalidates
against CF edge. CF returns 304 (minimal cost) when data is unchanged,
or 200 with the current digest. s-maxage and CF-level caching are
unaffected; max-age still benefits browser back/forward cache.
* fix(cache): 15-min consistent TTL + degrade guard for digest
Issue 1 — TTL alignment: Redis digest TTL reverted to 900s (from 3600).
slow-browser tier reduced from s-maxage=1800/CDN=3600 to s-maxage=900 on
both sides, matching the Redis TTL. The freshness window is now consistently
15 minutes across Redis, Vercel edge, and CF CDN. max-age=300 (browser
local) is kept to avoid unnecessary revalidations on tab switch.
Issue 2 — Cache poisoning: replaced cachedFetchJson in listFeedDigest with
explicit getCachedJson/setCachedJson. After buildDigest(), if total items
across all categories is 0 the response is treated as degraded: Redis write
is skipped and markNoCacheResponse(ctx.request) is called so the gateway
sets Cache-Control: no-store instead of the normal tier headers. This
prevents a transient bad run from poisoning Redis and browser/CDN for the
full TTL. Error paths also call markNoCacheResponse.
|
||
|
Elie Habib
|
4de2f74210 |
feat: move EONET/GDACS to server-side with Redis caching (#983)
* feat: move EONET/GDACS to server-side with Redis caching and bootstrap hydration Browser-direct fetches to eonet.gsfc.nasa.gov and gdacs.org caused CORS errors and had no server-side caching. This moves both to the standard Vercel edge → cachedFetchJson → Redis → bootstrap hydration pattern. - Add proto definitions for NaturalService with ListNaturalEvents RPC - Create server handler merging EONET + GDACS with 30min Redis TTL - Add Vercel edge function at /api/natural/v1/list-natural-events - Register naturalEvents in bootstrap SLOW_KEYS for CDN hydration - Replace browser-direct fetches with RPC client + circuit breaker - Delete src/services/gdacs.ts (logic moved server-side) * fix: restore @ts-nocheck on generated files stripped by buf generate |
||
|
Nicolas Gomes Ferreira Dos Santos
|
ee27b91c8f |
refactor(proto): consolidate SummarizeArticleResponse status fields (#813)
* refactor(proto): consolidate SummarizeArticleResponse status fields Replace redundant boolean/string status fields (cached, skipped, error, error_type, reason) with a SummarizeStatus enum and a single status_detail string. This addresses L-8 lint issue by reducing field count and making the response status unambiguous. - Add SummarizeStatus enum (UNSPECIFIED, SUCCESS, CACHED, SKIPPED, ERROR) - Replace cached/skipped booleans with status enum field - Merge error/error_type/reason into statusDetail string - Reserve old field numbers (4, 7, 8, 9, 10) for wire compatibility - Update server handlers and client code to use new fields - Regenerate TypeScript types and OpenAPI docs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(proto): restore error and errorType fields for programmatic error handling Keep the new SummarizeStatus enum and statusDetail for consolidated status tracking, but restore the separate error and errorType fields (proto field numbers 9, 10) to preserve structured error information for downstream consumers that need to programmatically handle errors. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
363cf5e71c |
perf(api): convert POST RPCs to GET for CDN caching (#795)
* perf(api): convert classify-event to GET and add summarize-article cache endpoint for CDN caching classify-event (7.9M calls/wk) was POST — bypassing all CDN caching. Converting to GET with static cache tier (1hr) enables Cloudflare edge caching. Degraded responses (no API key, empty title, errors) are marked no-cache to prevent caching error states. summarize-article has repeated headlines too large for URL params. Added a new GetSummarizeArticleCache GET endpoint that looks up Redis by a deterministic cache key. Client computes key via shared buildSummaryCacheKey(), tries GET first (CDN-cacheable), falls back to existing POST on miss. Shared module ensures client/server key parity. * fix(types): wire missing DeductSituation and ListGulfQuotes RPCs, fix tsc errors - Added DeductSituation RPC to intelligence/v1/service.proto (messages existed, RPC declaration was missing) - Added ListGulfQuotes proto + RPC to market/v1/service.proto (handler existed, proto was missing) - Fixed scrapedAt type mismatch in conflict/index.ts (int64 → string) - Added @ts-nocheck to generated files with codegen type bugs - Regenerated all sebuf client/server code * fix(types): fix int64→string type mismatch in list-iran-events.ts |
||
|
Elie Habib
|
3ec97c7ac6 |
feat(news): server-side feed aggregation to reduce edge invocations by ~95% (#622)
Phase 1: Force CDN caching on rss-proxy (s-maxage=300 for 2xx, short
TTL for errors) — fixes bug where upstream no-cache headers were passed
through verbatim, defeating Vercel CDN.
Phase 2: Add ListFeedDigest RPC that aggregates all feeds server-side
into a single Redis-cached response. Client makes 1 request instead of
~90 per cycle. Includes circuit breaker with persistent cache fallback,
per-feed AI reclassification, and headline ingestion parity.
Phase 3: Increase polling interval from 5min to 7min to offset CDN
cache alignment.
New files:
- proto/worldmonitor/news/v1/list_feed_digest.proto
- server/worldmonitor/news/v1/{_feeds,_classifier,list-feed-digest}.ts
- src/services/ai-classify-queue.ts (extracted from rss.ts)
|
||
|
Sebastien Melki
|
c939cc6296 |
Proto-first API rebuild: sebuf contracts, handlers, gateway, and generated docs (#106)
* docs: initialize sebuf integration project with codebase map Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: add project config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: complete project research * docs: define v1 requirements (34 requirements, 8 categories) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: create roadmap (8 phases) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(01): capture phase context * docs(state): record phase 1 context session * docs(01): research phase domain - buf toolchain, sebuf codegen, proto patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(01-proto-foundation): create phase plan * chore(01-01): configure buf toolchain with buf.yaml, buf.gen.yaml, buf.lock - buf.yaml v2 with STANDARD+COMMENTS lint, FILE+PACKAGE+WIRE_JSON breaking, deps on protovalidate and sebuf - buf.gen.yaml configures protoc-gen-ts-client, protoc-gen-ts-server, protoc-gen-openapiv3 plugins - buf.lock generated with resolved dependency versions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(01-01): add shared core proto type definitions - geo.proto: GeoCoordinates with lat/lng validation, BoundingBox for spatial queries - time.proto: TimeRange with google.protobuf.Timestamp start/end - pagination.proto: cursor-based PaginationRequest (1-100 page_size) and PaginationResponse - i18n.proto: LocalizableString for pre-localized upstream API strings - identifiers.proto: typed ID wrappers (HotspotID, EventID, ProviderID) for cross-domain refs - general_error.proto: GeneralError with RateLimited, UpstreamDown, GeoBlocked, MaintenanceMode All files pass buf lint (STANDARD+COMMENTS) and buf build with zero errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(01-01): complete buf toolchain and core proto types plan - SUMMARY.md documents 2 tasks, 9 files created, 2 deviations auto-fixed - STATE.md updated: plan 1/2 in phase 1, decisions recorded - ROADMAP.md updated: phase 01 in progress (1/2 plans) - REQUIREMENTS.md updated: PROTO-01, PROTO-02, PROTO-03 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(01-01): use int64 epoch millis instead of google.protobuf.Timestamp User preference: all time fields use int64 (Unix epoch milliseconds) instead of google.protobuf.Timestamp for simpler serialization and JS interop. Applied to TimeRange and MaintenanceMode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(01-02): create test domain proto files with core type imports - Add test_item.proto with GeoCoordinates import and int64 timestamps - Add get_test_items.proto with TimeRange and Pagination imports - Add service.proto with HTTP annotations for TestService - All proto files pass buf lint and buf build * feat(01-02): run buf generate and create Makefile for code generation pipeline - Add Makefile with generate, lint, clean, install, check, format, breaking targets - Update buf.gen.yaml with managed mode and paths=source_relative for correct output paths - Generate TypeScript client (TestServiceClient class) at src/generated/client/ - Generate TypeScript server (TestServiceHandler interface) at src/generated/server/ - Generate OpenAPI 3.1.0 specs (JSON + YAML) at docs/api/ - Core type imports (GeoCoordinates, TimeRange, Pagination) flow through to generated output * docs(01-02): complete test domain code generation pipeline plan - Create 01-02-SUMMARY.md with pipeline validation results - Update STATE.md: phase 1 complete, 2/2 plans done, new decisions recorded - Update ROADMAP.md: phase 1 marked complete (2/2) - Update REQUIREMENTS.md: mark PROTO-04 and PROTO-05 complete * docs(phase-01): complete phase execution and verification * test(01): complete UAT - 6 passed, 0 issues * feat(2A): define all 17 domain proto packages with generated clients, servers, and OpenAPI specs Remove test domain protos (Phase 1 scaffolding). Add core enhancements (severity.proto, country.proto, expanded identifiers.proto). Define all 17 domain services: seismology, wildfire, climate, conflict, displacement, unrest, military, aviation, maritime, cyber, market, prediction, economic, news, research, infrastructure, intelligence. 79 proto files producing 34 TypeScript files and 34 OpenAPI specs. buf lint clean, tsc clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2B): add server runtime phase context and handoff checkpoint Prepare Phase 2B with full context file covering deliverables, key reference files, generated code patterns, and constraints. Update STATE.md with resume pointer. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2B): research phase domain * docs(2B): create phase plan * feat(02-01): add shared server infrastructure (router, CORS, error mapper) - router.ts: Map-based route matcher from RouteDescriptor[] arrays - cors.ts: TypeScript port of api/_cors.js with POST/OPTIONS methods - error-mapper.ts: onError callback handling ApiError, network, and unknown errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(02-01): implement seismology handler as first end-to-end proof - Implements SeismologyServiceHandler from generated server types - Fetches USGS M4.5+ earthquake GeoJSON feed and transforms to proto-shaped Earthquake[] - Maps all fields: id, place, magnitude, depthKm, location, occurredAt (String), sourceUrl Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(02-01): complete server infrastructure plan - SUMMARY.md with task commits, decisions, and self-check - STATE.md updated: position, decisions, session info - REQUIREMENTS.md: SERVER-01, SERVER-02, SERVER-06 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(02-02): create Vercel catch-all gateway, tsconfig.api.json, and typecheck:api script - api/[[...path]].ts mounts seismology routes via catch-all with CORS on every response path - tsconfig.api.json extends base config without vite/client types for edge runtime - package.json adds typecheck:api script * feat(02-02): add Vite dev server plugin for sebuf API routes - sebufApiPlugin() intercepts /api/{domain}/v1/* in dev mode - Uses dynamic imports to lazily load handler modules inside configureServer - Converts Connect IncomingMessage to Web Standard Request - CORS headers applied to all plugin responses (200, 204, 403, 404) - Falls through to existing proxy rules for non-sebuf /api/* paths * docs(02-02): complete gateway integration plan - SUMMARY.md documenting catch-all gateway + Vite plugin implementation - STATE.md updated: Phase 2B complete, decisions recorded - ROADMAP.md updated: Phase 02 marked complete (2/2 plans) - REQUIREMENTS.md: SERVER-03, SERVER-04, SERVER-05 marked complete * docs(02-server-runtime): create gap closure plan for SERVER-05 Tauri sidecar * feat(02-03): add esbuild compilation step for sebuf sidecar gateway bundle - Create scripts/build-sidecar-sebuf.mjs that bundles api/[[...path]].ts into a single ESM .js file - Add build:sidecar-sebuf npm script and chain it into the main build command - Install esbuild as explicit devDependency - Gitignore the compiled api/[[...path]].js build artifact Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(02-03): verify sidecar discovery and annotate SERVER-05 gap closure - Confirm compiled bundle handler returns status 200 for POST requests - Add gap closure note to SERVER-05 in REQUIREMENTS.md - Verify typecheck:api and full build pipeline pass without regressions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(02-03): complete sidecar sebuf bundle plan - Create 02-03-SUMMARY.md documenting esbuild bundle compilation - Update STATE.md with plan 03 position, decisions, and metrics - Update ROADMAP.md plan progress (3/3 plans complete) - Annotate SERVER-05 gap closure in REQUIREMENTS.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-02): complete phase execution * docs(2C): capture seismology migration phase context Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(state): record phase 2C context session Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2C): research seismology migration phase Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2C): create seismology migration phase plan * feat(2C-01): annotate all int64 time fields with INT64_ENCODING_NUMBER - Vendor sebuf/http/annotations.proto locally with Int64Encoding extension (50010) - Remove buf.build/sebmelki/sebuf BSR dep, use local vendored proto instead - Add INT64_ENCODING_NUMBER annotation to 34 time fields across 20 proto files - Regenerate all TypeScript client and server code (time fields now `number` not `string`) - Fix seismology handler: occurredAt returns number directly (no String() wrapper) - All non-time int64 fields (displacement counts, population) left as string - buf lint, buf generate, tsc, and sidecar build all pass with zero errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2C-01): complete INT64_ENCODING_NUMBER plan - Create 2C-01-SUMMARY.md with execution results and deviations - Update STATE.md: plan 01 complete, int64 blocker resolved, new decisions - Update ROADMAP.md: mark 2C-01 plan complete - Update REQUIREMENTS.md: mark CLIENT-01 complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(lint): exclude .planning/ from markdownlint GSD planning docs use formatting that triggers MD032 -- these are machine-generated and not user-facing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2C-02): rewrite earthquake adapter to use SeismologyServiceClient and adapt all consumers to proto types - Replace legacy fetch/circuit-breaker adapter with port/adapter wrapping SeismologyServiceClient - Update 7 consuming files to import Earthquake from @/services/earthquakes (the port) - Adapt all field accesses: lat/lon -> location?.latitude/longitude, depth -> depthKm, time -> occurredAt, url -> sourceUrl - Remove unused filterByTime from Map.ts (only called for earthquakes, replaced with inline filter) - Update e2e test data to proto Earthquake shape Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore(2C-02): delete legacy earthquake endpoint, remove Vite proxy, clean API_URLS config - Delete api/earthquakes.js (legacy Vercel edge function proxying USGS) - Remove /api/earthquake Vite dev proxy (sebufApiPlugin handles seismology now) - Remove API_URLS.earthquakes entry from base config (no longer referenced) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2C-02): complete seismology client wiring plan - Create 2C-02-SUMMARY.md with execution results - Update STATE.md: phase 2C complete, decisions, metrics - Update ROADMAP.md: mark 2C-02 and phase 2C complete - Mark requirements CLIENT-02, CLIENT-04, CLEAN-01, CLEAN-02 complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2C): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2D): create wildfire migration phase plan * feat(2D-01): enhance FireDetection proto and implement wildfire handler - Add region (field 8) and day_night (field 9) to FireDetection proto - Regenerate TypeScript client and server types - Implement WildfireServiceHandler with NASA FIRMS CSV proxy - Fetch all 9 monitored regions in parallel via Promise.allSettled - Graceful degradation to empty list when API key is missing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2D-01): wire wildfire routes into gateway and rebuild sidecar - Import createWildfireServiceRoutes and wildfireHandler in catch-all - Mount wildfire routes alongside seismology in allRoutes array - Rebuild sidecar-sebuf bundle with wildfire endpoint included Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2D-01): complete wildfire handler plan - Create 2D-01-SUMMARY.md with execution results - Update STATE.md position to 2D plan 01 complete - Update ROADMAP.md with 2D progress (1/2 plans) - Mark DOMAIN-01 requirement complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2D-02): create wildfires service module and rewire all consumers - Add src/services/wildfires/index.ts with fetchAllFires, computeRegionStats, flattenFires, toMapFires - Rewire App.ts to import from @/services/wildfires with proto field mappings - Rewire SatelliteFiresPanel.ts to import FireRegionStats from @/services/wildfires - Update signal-aggregator.ts source comment * chore(2D-02): delete legacy wildfire endpoint and service module - Remove api/firms-fires.js (replaced by api/server/worldmonitor/wildfire/v1/handler.ts) - Remove src/services/firms-satellite.ts (replaced by src/services/wildfires/index.ts) - Zero dangling references confirmed - Full build passes (tsc, vite, sidecar) * docs(2D-02): complete wildfire consumer wiring plan - Create 2D-02-SUMMARY.md with execution results - Update STATE.md: phase 2D complete, progress ~52% - Update ROADMAP.md: phase 2D plan progress * docs(phase-2D): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2E): research climate migration domain Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2E): create phase plan * feat(2E-01): implement climate handler with 15-zone monitoring and baseline comparison - Create ClimateServiceHandler with 15 hardcoded monitored zones matching legacy - Parallel fetch from Open-Meteo Archive API via Promise.allSettled - 30-day baseline comparison: last 7 days vs preceding baseline - Null filtering with paired data points, minimum 14-point threshold - Severity classification (normal/moderate/extreme) and type (warm/cold/wet/dry/mixed) - 1-decimal rounding for tempDelta and precipDelta - Proto ClimateAnomaly mapping with GeoCoordinates Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2E-01): wire climate routes into gateway and rebuild sidecar - Import createClimateServiceRoutes and climateHandler in catch-all gateway - Mount climate routes alongside seismology and wildfire - Rebuild sidecar-sebuf bundle with climate routes included Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2E-01): complete climate handler plan - Create 2E-01-SUMMARY.md with execution results - Update STATE.md: position to 2E plan 01, add decisions - Update ROADMAP.md: mark 2E-01 complete, update progress table Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2E-02): rewrite climate service module and rewire all consumers - Replace src/services/climate.ts with src/services/climate/index.ts directory module - Port/adapter pattern: ClimateServiceClient maps proto shapes to legacy consumer shapes - Rewire ClimateAnomalyPanel, DeckGLMap, MapContainer, country-instability, conflict-impact - All 6 consumers import ClimateAnomaly from @/services/climate instead of @/types - Drop dead getSeverityColor function, keep getSeverityIcon and formatDelta - Fix minSeverity required param in listClimateAnomalies call * chore(2E-02): delete legacy climate endpoint and remove dead types - Delete api/climate-anomalies.js (replaced by sebuf climate handler) - Remove ClimateAnomaly and AnomalySeverity from src/types/index.ts - Full build passes with zero errors * docs(2E-02): complete climate client wiring plan - Create 2E-02-SUMMARY.md with execution results - Update STATE.md: phase 2E complete, decisions, session continuity - Update ROADMAP.md: phase 2E progress * docs(phase-2E): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2F): research prediction migration domain * docs(2F): create prediction migration phase plans * feat(2F-01): implement prediction handler with Gamma API proxy - PredictionServiceHandler proxying Gamma API with 8s timeout - Maps events/markets to proto PredictionMarket with 0-1 yesPrice scale - Graceful degradation: returns empty markets on any failure (Cloudflare expected) - Supports category-based events endpoint and default markets endpoint Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2F-01): wire prediction routes into gateway - Import createPredictionServiceRoutes and predictionHandler - Mount prediction routes in allRoutes alongside seismology, wildfire, climate - Sidecar bundle rebuilt successfully (21.2 KB) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2F-01): complete prediction handler plan - SUMMARY.md with handler implementation details and deviation log - STATE.md updated to 2F in-progress position with decisions - ROADMAP.md updated to 1/2 plans complete for phase 2F - REQUIREMENTS.md marked DOMAIN-02 complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2F-02): create prediction service module and rewire all consumers - Create src/services/prediction/index.ts preserving all business logic from polymarket.ts - Replace strategy 4 (Vercel edge) with PredictionServiceClient in polyFetch - Update barrel export from polymarket to prediction in services/index.ts - Rewire 7 consumers to import PredictionMarket from @/services/prediction - Fix 3 yesPrice bugs: CountryIntelModal (*100), App.ts search (*100), App.ts snapshot (1-y) - Drop dead code getPolymarketStatus() Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore(2F-02): delete legacy endpoint and remove dead types - Delete api/polymarket.js (replaced by sebuf handler) - Delete src/services/polymarket.ts (replaced by src/services/prediction/index.ts) - Remove PredictionMarket interface from src/types/index.ts (now in prediction module) - Type check and sidecar build both pass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2F-02): complete prediction consumer wiring plan Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2F): complete phase execution * docs(phase-2F): fix roadmap plan counts and completion status Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2G): research displacement migration phase * docs(2G): create displacement migration phase plans * feat(2G-01): implement displacement handler with UNHCR API pagination and aggregation - 40-entry COUNTRY_CENTROIDS map for geographic coordinates - UNHCR Population API pagination (10,000/page, 25-page guard) - Year fallback: current year to current-2 until data found - Per-country origin + asylum aggregation with unified merge - Global totals computation across all raw records - Flow corridor building sorted by refugees, capped by flowLimit - All int64 fields returned as String() per proto types - Graceful empty response on any failure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2G-01): wire displacement routes into gateway and rebuild sidecar - Import createDisplacementServiceRoutes and displacementHandler - Mount displacement routes alongside seismology, wildfire, climate, prediction - Sidecar bundle rebuilt with displacement included (31.0 KB) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2G-01): complete displacement handler plan - SUMMARY.md with execution metrics and decisions - STATE.md updated to 2G phase position - ROADMAP.md updated with 2G-01 plan progress Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2G-02): create displacement service module and rewire all consumers - Create src/services/displacement/index.ts as port/adapter using DisplacementServiceClient - Map proto int64 strings to numbers and GeoCoordinates to flat lat/lon - Preserve circuit breaker, presentation helpers (getDisplacementColor, formatPopulation, etc.) - Rewire App.ts, DisplacementPanel, MapContainer, DeckGLMap, conflict-impact, country-instability - Delete legacy src/services/unhcr.ts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore(2G-02): delete legacy endpoint and remove dead displacement types - Delete api/unhcr-population.js (replaced by displacement handler from 2G-01) - Remove DisplacementFlow, CountryDisplacement, UnhcrSummary from src/types/index.ts - All consumers now import from @/services/displacement - Sidecar rebuild, tsc, and full Vite build pass clean Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2G-02): complete displacement consumer wiring plan - SUMMARY.md with 2 task commits, decisions, deviation documentation - STATE.md updated: phase 2G complete, 02/02 plans done - ROADMAP.md updated with plan progress Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2G): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2H): research aviation migration phase * docs(2H): create aviation migration phase plans * feat(2H-01): implement aviation handler with FAA XML parsing and simulated delays - Install fast-xml-parser for server-side XML parsing (edge-compatible) - Create AviationServiceHandler with FAA NASSTATUS XML fetch and parse - Enrich US airports with MONITORED_AIRPORTS metadata (lat, lon, name, icao) - Generate simulated delays for non-US airports with rush-hour weighting - Map short-form strings to proto enums (FlightDelayType, FlightDelaySeverity, etc.) - Wrap flat lat/lon into GeoCoordinates for proto response - Graceful empty alerts on any upstream failure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2H-01): wire aviation routes into gateway and rebuild sidecar - Mount createAviationServiceRoutes in catch-all gateway alongside 5 existing domains - Import aviationHandler for route wiring - Rebuild sidecar-sebuf bundle with aviation routes included Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2H-01): complete aviation handler plan - Create 2H-01-SUMMARY.md with execution results - Update STATE.md position to 2H-01 with aviation decisions - Update ROADMAP.md progress for phase 2H (1/2 plans) - Mark DOMAIN-08 requirement as complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2H-02): create aviation service module and rewire all consumers - Create src/services/aviation/index.ts as port/adapter wrapping AviationServiceClient - Map proto enum strings to short-form (severity, delayType, region, source) - Unwrap GeoCoordinates to flat lat/lon, convert epoch-ms updatedAt to Date - Preserve circuit breaker with identical name string - Rewire Map, DeckGLMap, MapContainer, MapPopup, map-harness to import from @/services/aviation - Update barrel export: flights -> aviation - Delete legacy src/services/flights.ts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore(2H-02): delete legacy endpoint and remove dead aviation types - Delete api/faa-status.js (replaced by aviation handler in 2H-01) - Remove FlightDelaySource, FlightDelaySeverity, FlightDelayType, AirportRegion, AirportDelayAlert from src/types/index.ts - Preserve MonitoredAirport with inlined region type union - Full build (tsc + vite + sidecar) passes with zero errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2H-02): complete aviation consumer wiring plan Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2H): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2I): research phase domain * docs(2I): create phase plan * feat(2I-01): implement ResearchServiceHandler with 3 RPCs - arXiv XML parsing with fast-xml-parser (ignoreAttributes: false for attributes) - GitHub trending repos with primary + fallback API URLs - Hacker News Firebase API with 2-step fetch and bounded concurrency (10) - All RPCs return empty arrays on failure (graceful degradation) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2I-01): mount research routes in gateway and rebuild sidecar - Import createResearchServiceRoutes and researchHandler in catch-all gateway - Add research routes to allRoutes array (after aviation) - Sidecar bundle rebuilt (116.6 KB) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2I-01): complete research handler plan - SUMMARY.md with self-check passed - STATE.md updated to phase 2I, plan 01 of 02 - ROADMAP.md updated with plan 2I-01 complete - REQUIREMENTS.md: DOMAIN-05 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2I-02): create research service module and delete legacy code - Add src/services/research/index.ts with fetchArxivPapers, fetchTrendingRepos, fetchHackernewsItems backed by ResearchServiceClient - Re-export proto types ArxivPaper, GithubRepo, HackernewsItem (no enum mapping needed) - Circuit breakers wrap all 3 client calls with empty-array fallback - Delete legacy API endpoints: api/arxiv.js, api/github-trending.js, api/hackernews.js - Delete legacy service files: src/services/arxiv.ts, src/services/github-trending.ts, src/services/hackernews.ts - Remove arxiv, githubTrending, hackernews entries from API_URLS and REFRESH_INTERVALS in config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2I-02): complete research consumer wiring plan - SUMMARY.md documenting service module creation and 6 legacy file deletions - STATE.md updated: phase 2I complete, decisions recorded - ROADMAP.md updated: phase 2I marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2I): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2J): complete unrest migration research * docs(2J): create unrest migration phase plans * feat(2J-01): implement UnrestServiceHandler with ACLED + GDELT dual-fetch - Create handler with listUnrestEvents RPC proxying ACLED API and GDELT GEO API - ACLED fetch uses Bearer auth from ACLED_ACCESS_TOKEN env var, returns empty on missing token - GDELT fetch returns GeoJSON protest events with no auth required - Deduplication uses 0.5-degree grid + date key, preferring ACLED over GDELT on collision - Severity classification and event type mapping ported from legacy protests.ts - Sort by severity (high first) then recency (newest first) - Graceful degradation: returns empty events on any upstream failure * feat(2J-01): mount unrest routes in gateway and rebuild sidecar - Import createUnrestServiceRoutes and unrestHandler in catch-all gateway - Add unrest service routes to allRoutes array - Sidecar bundle rebuilt to include unrest endpoint - RPC routable at POST /api/unrest/v1/list-unrest-events * docs(2J-01): complete unrest handler plan - Create 2J-01-SUMMARY.md with execution results and self-check - Update STATE.md with phase 2J position, decisions, session continuity - Update ROADMAP.md with plan 01 completion status * feat(2J-02): create unrest service module with proto-to-legacy type mapping - Full adapter maps proto UnrestEvent to legacy SocialUnrestEvent shape - 4 enum mappers: severity, eventType, sourceType, confidence - fetchProtestEvents returns ProtestData with events, byCountry, highSeverityCount, sources - getProtestStatus infers ACLED configuration from response event sources - Circuit breaker wraps client call with empty fallback * feat(2J-02): update services barrel, remove vite proxies, delete legacy files - Services barrel: protests -> unrest re-export - Vite proxy entries removed: /api/acled, /api/gdelt-geo - Legacy files deleted: api/acled.js, api/gdelt-geo.js, src/services/protests.ts - Preserved: api/acled-conflict.js (conflict domain), SocialUnrestEvent type * docs(2J-02): complete unrest service module plan - SUMMARY.md created with full adapter pattern documentation - STATE.md updated: 2J-02 complete, decisions recorded - ROADMAP.md updated: Phase 2J marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2J): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2K): complete conflict migration research * docs(2K): create phase plan * feat(2K-01): implement ConflictServiceHandler with 3 RPCs - listAcledEvents proxies ACLED API for battles/explosions/violence with Bearer auth - listUcdpEvents discovers UCDP GED API version dynamically, fetches backward with 365-day trailing window - getHumanitarianSummary proxies HAPI API with ISO-2 to ISO-3 country mapping - All RPCs have graceful degradation returning empty on failure * feat(2K-01): mount conflict routes in gateway and rebuild sidecar - Add createConflictServiceRoutes and conflictHandler imports to catch-all gateway - Spread conflict routes into allRoutes array (3 RPC endpoints) - Rebuild sidecar bundle with conflict endpoints included * docs(2K-01): complete conflict handler plan - Create 2K-01-SUMMARY.md with execution details and self-check - Update STATE.md: position to 2K-01, add 5 decisions - Update ROADMAP.md: mark 2K-01 complete (1/2 plans done) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2K-02): create conflict service module with 4-shape proto-to-legacy type mapping - Port/adapter mapping AcledConflictEvent -> ConflictEvent, UcdpViolenceEvent -> UcdpGeoEvent, HumanitarianCountrySummary -> HapiConflictSummary - UCDP classifications derived heuristically from GED events (deaths/events thresholds -> war/minor/none) - deduplicateAgainstAcled ported exactly with haversine + date + fatality matching - 3 circuit breakers for 3 RPCs, exports 5 functions + 2 group helpers + all legacy types * feat(2K-02): rewire consumer imports and delete 9 legacy conflict files - App.ts consolidated from 4 direct imports to single @/services/conflict import - country-instability.ts consolidated from 3 type imports to single ./conflict import - Deleted 4 API endpoints: acled-conflict.js, ucdp-events.js, ucdp.js, hapi.js - Deleted 4 service files: conflicts.ts, ucdp.ts, ucdp-events.ts, hapi.ts - Deleted 1 dead code file: conflict-impact.ts - UcdpGeoEvent preserved in src/types/index.ts (scope guard for map components) * docs(2K-02): complete conflict service module plan - SUMMARY.md with 4-shape proto adapter, consumer consolidation, 9 legacy deletions - STATE.md updated: Phase 2K complete (2/2 plans), progress ~100% - ROADMAP.md updated: Phase 2K marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2K): complete conflict migration phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2L): research maritime migration phase domain * docs(2L): create maritime migration phase plans * feat(2L-01): implement MaritimeServiceHandler with 2 RPCs - getVesselSnapshot proxies WS relay with wss->https URL conversion - Maps density/disruptions to proto shape with GeoCoordinates nesting - Disruption type/severity mapped from lowercase to proto enums - listNavigationalWarnings proxies NGA MSI broadcast warnings API - NGA military date parsing (081653Z MAY 2024) to epoch ms - Both RPCs gracefully degrade to empty on upstream failure - No caching (client-side polling manages refresh intervals) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2L-01): mount maritime routes in gateway and rebuild sidecar - Import createMaritimeServiceRoutes and maritimeHandler - Add maritime routes to allRoutes array in catch-all gateway - Sidecar bundle rebuilt (148.0 KB) with maritime endpoints - RPCs routable at /api/maritime/v1/get-vessel-snapshot and /api/maritime/v1/list-navigational-warnings Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2L-01): complete maritime handler plan - SUMMARY.md with 2 task commits documented - STATE.md updated to 2L phase, plan 01/02 complete - ROADMAP.md progress updated for phase 2L - REQUIREMENTS.md: DOMAIN-06 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2L-02): create maritime service module with hybrid fetch and polling/callback preservation - Port/adapter wrapping MaritimeServiceClient for proto RPC path - Full polling/callback architecture preserved from legacy ais.ts - Hybrid fetch: proto RPC for snapshot-only, raw WS relay for candidates - Proto-to-legacy type mapping for AisDisruptionEvent and AisDensityZone - Exports fetchAisSignals, initAisStream, disconnectAisStream, getAisStatus, isAisConfigured, registerAisCallback, unregisterAisCallback, AisPositionData * feat(2L-02): rewire consumer imports and delete 3 legacy maritime files - cable-activity.ts: fetch NGA warnings via MaritimeServiceClient.listNavigationalWarnings() with NgaWarning shape reconstruction from proto fields - military-vessels.ts: imports updated from './ais' to './maritime' - Services barrel: updated from './ais' to './maritime' - desktop-readiness.ts: service/api references updated to maritime handler paths - Deleted: api/ais-snapshot.js, api/nga-warnings.js, src/services/ais.ts - AisDisruptionEvent/AisDensityZone/AisDisruptionType preserved in src/types/index.ts * docs(2L-02): complete maritime service module plan - SUMMARY.md with hybrid fetch pattern, polling/callback preservation, 3 legacy files deleted - STATE.md updated: phase 2L complete, 5 decisions recorded - ROADMAP.md updated: 2L plans marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: bind globalThis.fetch in all sebuf service clients Generated sebuf clients store globalThis.fetch as a class property, then call it as this.fetchFn(). This loses the window binding and throws "Illegal invocation" in browsers. Pass { fetch: fetch.bind(globalThis) } to all 11 client constructors. Also includes vite.config.ts with all 10 migrated domain handlers registered in the sebuf dev server plugin. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate cyber + economic domains to sebuf (12/17) Cyber (Phase 2M): - Create handler aggregating 5 upstream sources (Feodo, URLhaus, C2Intel, OTX, AbuseIPDB) with dedup, GeoIP hydration, country centroid fallback - Create service module with CyberServiceClient + circuit breaker - Delete api/cyber-threats.js, api/cyber-threats.test.mjs, src/services/cyber-threats.ts Economic (Phase 2N) — consolidates 3 legacy services: - Create handler with 3 RPCs: getFredSeries (FRED API), listWorldBankIndicators (World Bank API), getEnergyPrices (EIA API) - Create unified service module replacing fred.ts, oil-analytics.ts, worldbank.ts - Preserve all exported functions/types for EconomicPanel and TechReadinessPanel - Delete api/fred-data.js, api/worldbank.js, src/services/fred.ts, src/services/oil-analytics.ts, src/services/worldbank.ts Both domains registered in vite.config.ts and api/[[...path]].ts. TypeScript check and vite build pass cleanly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate infrastructure domain to sebuf (13/17) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate market domain to sebuf (14/17) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate news domain to sebuf (15/17) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate intelligence domain to sebuf (16/17) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate military domain to sebuf (17/17) All 17 domains now have sebuf handlers registered in the gateway. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate intelligence services to sebuf client Rewire pizzint.ts, cached-risk-scores.ts, and threat-classifier.ts to use IntelligenceServiceClient instead of legacy /api/ fetch calls. Handler now preserves raw threat level in subcategory field. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate military theater posture to sebuf client Rewire cached-theater-posture.ts to use MilitaryServiceClient instead of legacy /api/theater-posture fetch. Adds theater metadata map for proto→legacy TheaterPostureSummary adapter. UI gracefully falls back to total counts when per-type breakdowns aren't available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: rewire country-intel to sebuf client Replace legacy fetch('/api/country-intel') with typed IntelligenceServiceClient.getCountryIntelBrief() RPC call in App.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate stablecoin-markets to sebuf (market domain) Add ListStablecoinMarkets RPC to market service. Port CoinGecko stablecoin peg-health logic from api/stablecoin-markets.js into the market handler. Rewire StablecoinPanel to use typed sebuf client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate etf-flows to sebuf (market domain) Add ListEtfFlows RPC to market service. Port Yahoo Finance BTC spot ETF flow estimation logic from api/etf-flows.js into the market handler. Rewire ETFFlowsPanel to use typed sebuf client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate worldpop-exposure to sebuf (displacement domain) Add GetPopulationExposure RPC to displacement service. Port country population data and radius-based exposure estimation from api/worldpop-exposure.js into the displacement handler. Rewire population-exposure.ts to use typed sebuf client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove superseded legacy edge functions Delete 4 legacy api/*.js files that are now fully replaced by sebuf handlers: - api/stablecoin-markets.js -> market/ListStablecoinMarkets - api/etf-flows.js -> market/ListEtfFlows - api/worldpop-exposure.js -> displacement/GetPopulationExposure - api/classify-batch.js -> intelligence/ClassifyEvent Remaining legacy files are still actively used by client code (stock-index, opensky, gdelt-doc, rss-proxy, summarize endpoints, macro-signals, tech-events) or are shared utilities. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: delete dead legacy files and unused API_URLS config Remove coingecko.js, debug-env.js, cache-telemetry.js, _cache-telemetry.js (all zero active consumers). Delete unused API_URLS export from base config. Update desktop-readiness market-panel metadata to reference sebuf paths. Remove dead CoinGecko dev proxy from vite.config.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate stock-index and opensky to sebuf - Add GetCountryStockIndex RPC to market domain (Yahoo Finance + cache) - Fill ListMilitaryFlights stub in military handler (OpenSky with bounding box) - Rewire App.ts stock-index fetch to MarketServiceClient.getCountryStockIndex() - Delete api/stock-index.js and api/opensky.js edge functions - OpenSky client path unchanged (relay primary, vite proxy for dev) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * wip: sebuf legacy migration paused at phase 3/10 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(03): capture phase context * docs(state): record phase 3 context session * docs(03): research phase domain * docs(03): create phase plan — 5 plans in 2 waves * feat(03-01): commit wingbits migration (step 3) -- 3 RPCs added to military domain - Add GetAircraftDetails, GetAircraftDetailsBatch, GetWingbitsStatus RPCs - Rewire src/services/wingbits.ts to use MilitaryServiceClient - Update desktop-readiness.ts routes to match new RPC paths - Delete legacy api/wingbits/ edge functions (3 files) - Regenerate military service client/server TypeScript + OpenAPI docs * feat(03-02): add SummarizeArticle proto and implement handler - Create summarize_article.proto with request/response messages - Add SummarizeArticle RPC to NewsService proto - Implement full handler with provider dispatch (ollama/groq/openrouter) - Port cache key builder, deduplication, prompt builder, think-token stripping - Inline Upstash Redis helpers for edge-compatible caching * feat(03-01): migrate gdelt-doc to intelligence RPC + delete _ip-rate-limit.js - Add SearchGdeltDocuments RPC to IntelligenceService proto - Implement searchGdeltDocuments handler (port from api/gdelt-doc.js) - Rewire src/services/gdelt-intel.ts to use IntelligenceServiceClient - Delete legacy api/gdelt-doc.js edge function - Delete dead api/_ip-rate-limit.js (zero importers) - Regenerate intelligence service client/server TypeScript + OpenAPI docs * feat(03-02): rewire summarization client to NewsService RPC, delete 4 legacy files - Replace direct fetch to /api/{provider}-summarize with NewsServiceClient.summarizeArticle() - Preserve identical fallback chain: ollama -> groq -> openrouter -> browser T5 - Delete api/groq-summarize.js, api/ollama-summarize.js, api/openrouter-summarize.js - Delete api/_summarize-handler.js and api/_summarize-handler.test.mjs - Update desktop-readiness.ts to reference new sebuf route * feat(03-03): rewire MacroSignalsPanel to EconomicServiceClient + delete legacy - Replace fetch('/api/macro-signals') with EconomicServiceClient.getMacroSignals() - Add mapProtoToData() to convert proto optional fields to null for rendering - Delete legacy api/macro-signals.js edge function * feat(03-04): add ListTechEvents proto, city-coords data, and handler - Create list_tech_events.proto with TechEvent, TechEventCoords messages - Add ListTechEvents RPC to ResearchService proto - Extract 360-city geocoding table to api/data/city-coords.ts - Implement listTechEvents handler with ICS+RSS parsing, curated events, dedup, filtering - Regenerate TypeScript client/server from proto Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(03-01): complete wingbits + GDELT doc migration plan - Create 03-01-SUMMARY.md with execution results - Update STATE.md with plan 01 completion, steps 3-4 done - Update ROADMAP.md plan progress (2/5 plans complete) - Mark DOMAIN-10 requirement complete * docs(03-02): complete summarization migration plan - Create 03-02-SUMMARY.md with execution results - Update STATE.md position to step 6/10 - Update ROADMAP.md plan progress - Mark DOMAIN-09 requirement complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(03-04): rewire TechEventsPanel and App to ResearchServiceClient, delete legacy - Replace fetch('/api/tech-events') with ResearchServiceClient.listTechEvents() in TechEventsPanel - Replace fetch('/api/tech-events') with ResearchServiceClient.listTechEvents() in App.loadTechEvents() - Delete legacy api/tech-events.js (737 lines) - TypeScript compiles cleanly with no references to legacy endpoint Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(03-03): complete macro-signals migration plan - Create 03-03-SUMMARY.md with execution results - Mark DOMAIN-04 requirement complete in REQUIREMENTS.md * docs(03-04): complete tech-events migration plan - Add 03-04-SUMMARY.md with execution results - Update STATE.md: advance to plan 5/step 8, add decisions - Update ROADMAP.md: 4/5 plans complete for phase 03 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(03-05): add temporal baseline protos + handler with Welford's algorithm - GetTemporalBaseline RPC: anomaly detection with z-score thresholds - RecordBaselineSnapshot RPC: batch update via Welford's online algorithm - Inline mgetJson helper for Redis batch reads - Inline getCachedJson/setCachedJson Redis helpers - Generated TypeScript client/server + OpenAPI docs * feat(03-05): migrate temporal-baseline + tag non-JSON + final cleanup - Rewire temporal-baseline.ts to InfrastructureServiceClient RPCs - Delete api/temporal-baseline.js (migrated to sebuf handler) - Delete api/_upstash-cache.js (no importers remain) - Tag 6 non-JSON edge functions with // Non-sebuf: comment header - Update desktop-readiness.ts: fix stale cloudflare-outages reference * docs(03-05): complete temporal-baseline + non-JSON tagging + final cleanup plan - SUMMARY.md with Welford algorithm migration details - STATE.md updated: Phase 3 complete (100%) - ROADMAP.md updated: 5/5 plans complete * chore(03): delete orphaned ollama-summarize test after RPC migration Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-3): complete phase execution * docs(v1): create milestone audit report Audits all 13 phases of the v1 sebuf integration milestone. 12/13 phases verified (2L maritime missing VERIFICATION.md). 25/34 requirements satisfied, 6 superseded, 2 partial, 1 unsatisfied (CLEAN-03). All 17 domains wired end-to-end. Integration check passes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(v1): update audit — mark CLEAN-03/04 + MIGRATE-* as superseded CLEAN-03 superseded by port/adapter architecture (internal types intentionally decoupled from proto wire types). MIGRATE-01-05 superseded by direct cutover approach. DOMAIN-03 checkbox updated. Milestone status: tech_debt (no unsatisfied requirements). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(roadmap): add gap closure phase 4 — v1 milestone cleanup Closes all audit gaps: CLIENT-03 circuit breaker coverage, DOMAIN-03/06 verification gaps, documentation staleness, orphaned code cleanup. Fixes traceability table phase assignments to match actual roadmap phases. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore(03): commit generated NewsService OpenAPI specs + checkpoint update SummarizeArticle RPC was added during Phase 3 plan 02 but generated OpenAPI specs were not staged with that commit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(04): research phase domain * docs(04): create phase plan * docs(04-01): fix ROADMAP.md Phase 3 staleness and delete .continue-here.md - Change Phase 3 heading from IN PROGRESS to COMPLETE - Check off plans 03-03, 03-04, 03-05 (all were already complete) - Delete stale .continue-here.md (showed task 3/10 in_progress but all 10 done) * feat(04-02): add circuit breakers to seismology, wildfire, climate, maritime - Seismology: wrap listEarthquakes in breaker.execute with empty-array fallback - Wildfire: replace manual try/catch with breaker.execute for listFireDetections - Climate: replace manual try/catch with breaker.execute for listClimateAnomalies - Maritime: wrap proto getVesselSnapshot RPC in snapshotBreaker.execute, preserve raw relay fallback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(04-02): add circuit breakers to news summarization and GDELT intelligence - Summarization: wrap newsClient.summarizeArticle in summaryBreaker.execute for both tryApiProvider and translateText - GDELT: wrap client.searchGdeltDocuments in gdeltBreaker.execute, replace manual try/catch - Fix: include all required fields (tokens, reason, error, errorType, query) in fallback objects - CLIENT-03 fully satisfied: all 17 domains have circuit breaker coverage Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(04-01): create 2L-VERIFICATION.md, fix desktop-readiness.ts, complete service barrel - Create retroactive 2L-VERIFICATION.md with 12/12 must-haves verified - Fix map-layers-core and market-panel stale file refs in desktop-readiness.ts - Fix opensky-relay-cloud stale refs (api/opensky.js deleted) - Add missing barrel re-exports: conflict, displacement, research, wildfires, climate - Skip military/intelligence/news barrels (would cause duplicate exports) - TypeScript compiles cleanly with zero errors * docs(04-02): complete circuit breaker coverage plan - SUMMARY.md: 6 domains covered, CLIENT-03 satisfied, 1 deviation (fallback type fix) - STATE.md: Phase 4 plan 02 complete, position and decisions updated - ROADMAP.md: Phase 04 marked complete (2/2 plans) - REQUIREMENTS.md: CLIENT-03 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(04-01): complete documentation fixes plan - Create 04-01-SUMMARY.md with execution results - Update STATE.md with Plan 01 completion and decisions - Update ROADMAP.md: Plan 04-01 checked, progress 1/2 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-04): complete phase verification and fix tracking gaps ROADMAP.md Phase 4 status updated to Complete, 04-02 checkbox checked, progress table finalized. REQUIREMENTS.md coverage summary updated (27 complete, 0 partial/pending). STATE.md reflects verified phase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(military): migrate USNI fleet tracker to sebuf RPC Port all USNI Fleet Tracker parsing logic from api/usni-fleet.js into MilitaryService.GetUSNIFleetReport RPC with proto definitions, inline Upstash caching (6h fresh / 7d stale), and client adapter mapping. Deletes legacy edge function and _upstash-cache.js dependency. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: add proto validation annotations and split all 17 handler files into per-RPC modules Phase A: Added buf.validate constraints to ~25 proto files (~130 field annotations including required IDs, score ranges, coordinate bounds, page size limits). Phase B: Split all 17 domain handler.ts files into per-RPC modules with thin re-export handler.ts files. Extracted shared Redis cache helpers to api/server/_shared/redis.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add ADDING_ENDPOINTS guide and update Contributing section All JSON endpoints must use sebuf — document the complete workflow for adding RPCs to existing services and creating new services, including proto conventions, validation annotations, and generated OpenAPI docs. Update DOCUMENTATION.md Contributing section to reference the new guide and remove the deprecated "Adding a New API Proxy" pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add blank lines before lists to pass markdown lint (MD032) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: disambiguate duplicate city keys in city-coords Vercel's TypeScript check treats duplicate object keys as errors (TS1117). Rename 'san jose' (Costa Rica) -> 'san jose cr' and 'cambridge' (UK) -> 'cambridge uk' to avoid collision. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: resolve Vercel deployment errors — relocate hex-db + fix TS strict-null - Move military-hex-db.js next to handler (fixes Edge Function unsupported module) - Fix strict-null TS errors across 12 handler files (displacement, economic, infrastructure, intelligence, market, military, research, wildfire) - Add process declare to wildfire handler, prefix unused vars, cast types Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: convert military-hex-db to .ts for Edge Function compatibility Vercel Edge bundler can't resolve .js data modules from .ts handlers. Also remove unused _region variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: inline military hex db as packed string to avoid Edge Function module error Vercel Edge bundler can't resolve separate data modules. Inline 20K hex IDs as a single concatenated string, split into Set at runtime. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove broken export { process } from 4 _shared files declare const is stripped in JS output, making export { process } reference nothing. No consumers import it — each handler file has its own declare. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: move server/ out of api/ to fix Vercel catch-all routing Vercel's file-based routing was treating api/server/**/*.ts as individual API routes, overriding the api/[[...path]].ts catch-all for multi-segment paths like /api/infrastructure/v1/list-service-statuses (3 segments). Moving to server/ at repo root removes the ambiguity. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: rename catch-all to [...path] — [[...path]] is Next.js-only syntax Vercel's native edge function routing only supports [...path] for multi-segment catch-all matching. The [[...path]] double-bracket syntax is a Next.js feature and was only matching single-segment paths. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add dynamic segment route for multi-segment API paths Vercel's native file-based routing for non-Next.js projects doesn't support [...path] catch-all matching multiple segments. Use explicit api/[domain]/v1/[rpc].ts which matches /api/{domain}/v1/{rpc} via standard single-segment dynamic routing that Vercel fully supports. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove conflicting [...path] catch-all — replaced by [domain]/v1/[rpc] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: widen CORS pattern to match all Vercel preview URL formats Preview URLs use elie-ab2dce63 not elie-habib-projects as the team slug. Broaden pattern to elie-[a-z0-9]+ to cover both. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update sidecar build script for new api/[domain]/v1/[rpc] path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: trigger Vercel rebuild for all variants Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #106 review — critical bugs, hardening, and cleanup Fixes from @koala73's code review: Critical: - C-1: Add max-size eviction (2048 cap) to GeoIP in-memory cache - C-2: Move type/source/severity filters BEFORE .slice(pageSize) in cyber handler - C-3: Atomic SET with EX in Redis helper (single Upstash REST call) - C-4: Add AbortSignal.timeout(30s) to LLM fetch in summarize-article High: - H-1: Add top-level try/catch in gateway with CORS-aware 500 response - H-3: Sanitize error messages — generic text for 5xx, passthrough for 4xx only - H-4: Add timeout (10s) + Redis cache (5min) to seismology handler - H-5: Add null guards (optional chaining) in seismology USGS feature mapping - H-6: Race OpenSky + Wingbits with Promise.allSettled instead of sequential fallback - H-8: Add Redis cache (5min TTL) to infrastructure service-status handler Medium: - M-12/M-13: Fix HAPI summary field mappings (iso3 from countryCode, internallyDisplaced) Infrastructure: - R-1: Remove .planning/ from git tracking, add to .gitignore - Port UCDP parallel page fetching from main branch (#198) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #106 review issues — hash, CORS, router, cache, LLM, GeoIP Fixes/improvements from PR #106 code review tracking issues: - #180: Replace 32-bit hash (Java hashCode/DJB2) with unified FNV-1a 52-bit hash in server/_shared/hash.ts, greatly reducing collision probability - #182: Cache router construction in Vite dev plugin — build once, invalidate on HMR changes to server/ files - #194: Add input length limits for LLM prompt injection (headlines 500 chars, title 500 chars, geoContext 2000 chars, max 10 headlines) - #195/#196: GeoIP AbortController — cancel orphaned background workers on timeout instead of letting them fire after response is sent - #198: Port UCDP partial-result caching from main — 10min TTL for partial results vs 6hr for complete, with in-memory fallback cache Proto codegen regenerated for displacement + conflict int64_encoding changes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: restore fast-xml-parser dependency needed by sebuf handlers Main branch removed fast-xml-parser in v2.5.1 (legacy edge functions no longer needed it), but sebuf handlers in aviation/_shared.ts and research/list-arxiv-papers.ts still import it for XML API parsing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: fix stale paths, version badge, and local-backend-audit for sebuf - ADDING_ENDPOINTS.md: fix handler paths from api/server/ to server/, fix import depth (4 levels not 5), fix gateway extension (.js not .ts) - DOCUMENTATION.md: update version badge 2.1.4 -> 2.5.1, fix broken ROADMAP.md links to .planning/ROADMAP.md, fix handler path reference - COMMUNITY-PROMOTION-GUIDE.md: add missing v2.5.1 to version table - local-backend-audit.md: rewrite for sebuf architecture — replace all stale api/*.js references with sebuf domain handler paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: use make commands, add generation-before-push warning, bump sebuf to v0.7.0 - ADDING_ENDPOINTS.md: replace raw `cd proto && buf ...` with `make check`, `make generate`, `make install`; add warning that `make generate` must run before pushing proto changes (links to #200) - Makefile: bump sebuf plugin versions from v0.6.0 to v0.7.0 - PR description also updated to use make commands Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: make install installs everything, document setup - Makefile: `make install` now installs buf, sebuf plugins, npm deps, and proto deps in one command; pin buf and sebuf versions as variables - ADDING_ENDPOINTS.md: updated prerequisites to show `make install` - DOCUMENTATION.md: updated Installation section with `make install` and generation reminder Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #106 re-review issues — timeouts, iso3, pagination, CORS, dedup - NEW-1: HAPI handler now returns ISO-3 code (via ISO2_TO_ISO3 lookup) instead of ISO-2 - NEW-3: Aviation FAA fetch now has AbortSignal.timeout(15s) - NEW-4: Climate Open-Meteo fetch now has AbortSignal.timeout(20s) - NEW-5: Wildfire FIRMS fetch now has AbortSignal.timeout(15s) - NEW-6: Seismology now respects pagination.pageSize (default 500) - NEW-9: Gateway wraps getCorsHeaders() in try/catch with safe fallback - NEW-10: Tech events dedup key now includes start year to avoid dropping yearly variants Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #106 round 2 — seismology crash, tsconfig, type contracts - Fix _req undefined crash in seismology handler (renamed to req) - Cache full earthquake set, slice on read (avoids cache pollution) - Add server/ to tsconfig.api.json includes (catches type errors at build) - Remove String() wrappers on numeric proto fields in displacement/HAPI - Fix hashString re-export not available locally in news/_shared.ts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: loadMarkets heatmap regression, stale test, Makefile playwright - Add finnhub_skipped + skip_reason to ListMarketQuotesResponse proto - Wire skipped signal through handler → adapter → App.loadMarkets - Fix circuit breaker cache conflating different symbol queries (cacheTtlMs: 0) - Use dynamic fetch wrapper so e2e test mocks intercept correctly - Update e2e test mocks from old endpoints to sebuf proto endpoints - Delete stale summarization-chain.test.mjs (imports deleted files) - Add install-playwright target to Makefile Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add missing proto fields to emptyStockFallback (build fix) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(quick-1): fix country fallback, ISO-2 contract, and proto field semantics - BLOCKING-1: Return undefined when country has no ISO3 mapping instead of wrong country data - BLOCKING-2: country_code field now returns ISO-2 per proto contract - MEDIUM-1: Rename proto fields from humanitarian to conflict-event semantics (populationAffected -> conflictEventsTotal, etc.) - Update client service adapter to use new field names - Regenerate TypeScript types from updated proto Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(quick-1): add in-memory cache + in-flight dedup to AIS vessel snapshot - HIGH-1: 10-second TTL cache matching client poll interval - Concurrent requests share single upstream fetch (in-flight dedup) - Follows same pattern as get-macro-signals.ts cache - No change to RPC response shape Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(quick-1): stub RPCs throw UNIMPLEMENTED, remove hardcoded politics, add tests - HIGH-2: listNewsItems, summarizeHeadlines, listMilitaryVessels now throw UNIMPLEMENTED - LOW-1: Replace hardcoded "Donald Trump" with date-based dynamic LLM context - LOW-1 extended: Also fix same issue in intelligence/get-country-intel-brief.ts (Rule 2) - MEDIUM-2: Add tests/server-handlers.test.mjs with 20 tests covering all review items Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(quick-2): remove 3 dead stub RPCs (ListNewsItems, SummarizeHeadlines, ListMilitaryVessels) - Delete proto definitions, handler stubs, and generated code for dead RPCs - Clean _shared.ts: remove tryGroq, tryOpenRouter, buildPrompt, dead constants - Remove 3 UNIMPLEMENTED stub tests from server-handlers.test.mjs - Regenerate proto codegen (buf generate) and OpenAPI docs - SummarizeArticle and all other RPCs remain intact Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #106 review findings (stale snapshot, iso naming, scoring, test import) - Serve stale AIS snapshot on relay failure instead of returning undefined - Rename HapiConflictSummary.iso3 → iso2 to match actual ISO-2 content - Fix HAPI fallback scoring: use weight 3 for combined political violence (civilian targeting is folded in, was being underweighted at 0) - Extract deduplicateHeadlines to shared .mjs so tests import production code Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add coverage for PR106 iso2 and fallback regressions --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |