mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
main
43 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
2f5445284b |
fix(brief): single canonical synthesis brain — eliminate email/brief lead divergence (#3396)
* feat(brief-llm): canonical synthesis prompt + v3 cache key
Extends generateDigestProse to be the single source of truth for
brief executive-summary synthesis (canonicalises what was previously
split between brief-llm's generateDigestProse and seed-digest-
notifications.mjs's generateAISummary). Ports Brain B's prompt
features into buildDigestPrompt:
- ctx={profile, greeting, isPublic} parameter (back-compat: 4-arg
callers behave like today)
- per-story severity uppercased + short-hash prefix [h:XXXX] so the
model can emit rankedStoryHashes for stable re-ranking
- profile lines + greeting opener appear only when ctx.isPublic !== true
validateDigestProseShape gains optional rankedStoryHashes (≥4-char
strings, capped to MAX_STORIES_PER_USER × 2). v2-shaped rows still
pass — field defaults to [].
hashDigestInput v3:
- material includes profile-SHA, greeting bucket, isPublic flag,
per-story hash
- isPublic=true substitutes literal 'public' for userId in the cache
key so all share-URL readers of the same (date, sensitivity, pool)
hit ONE cache row (no PII in public cache key)
Adds generateDigestProsePublic(stories, sensitivity, deps) wrapper —
no userId param by design — for the share-URL surface.
Cache prefix bumped brief:llm:digest:v2 → v3. v2 rows expire on TTL.
Per the v1→v2 precedent (see hashDigestInput comment), one-tick cost
on rollout is acceptable for cache-key correctness.
Tests: 72/72 passing in tests/brief-llm.test.mjs (8 new for the v3
behaviors), full data suite 6952/6952.
Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Step 1, Codex-approved (5 rounds).
* feat(brief): envelope v3 — adds digest.publicLead for share-URL surface
Bumps BRIEF_ENVELOPE_VERSION 2 → 3. Adds optional
BriefDigest.publicLead — non-personalised executive lead generated
by generateDigestProsePublic (already in this branch from the
previous commit) for the public share-URL surface. Personalised
`lead` is the canonical synthesis for authenticated channels;
publicLead is its profile-stripped sibling so api/brief/public/*
never serves user-specific content (watched assets/regions).
SUPPORTED_ENVELOPE_VERSIONS = [1, 2, 3] keeps v1 + v2 envelopes
in the 7-day TTL window readable through the rollout — the
composer only ever writes the current version, but readers must
tolerate older shapes that haven't expired yet. Same rollout
pattern used at the v1 → v2 bump.
Renderer changes (server/_shared/brief-render.js):
- ALLOWED_DIGEST_KEYS gains 'publicLead' (closed-key-set still
enforced; v2 envelopes pass because publicLead === undefined is
the v2 shape).
- assertBriefEnvelope: new isNonEmptyString check on publicLead
when present. Type contract enforced; absence is OK.
Tests (tests/brief-magazine-render.test.mjs):
- New describe block "v3 publicLead field": v3 envelope renders;
malformed publicLead rejected; v2 envelope still passes; ad-hoc
digest keys (e.g. synthesisLevel) still rejected — confirming
the closed-key-set defense holds for the cron-local-only fields
the orchestrator must NOT persist.
- BRIEF_ENVELOPE_VERSION pin updated 2 → 3 with rollout-rationale
comment.
Test results: 182 brief-related tests pass; full data suite
6956/6956.
Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Step 2, Codex Round-3 Medium #2.
* feat(brief): synthesis splice + rankedStoryHashes pre-cap re-order
Plumbs the canonical synthesis output (lead, threads, signals,
publicLead, rankedStoryHashes from generateDigestProse) through the
pure composer so the orchestration layer can hand pre-resolved data
into envelope.digest. Composer stays sync / no I/O — Codex Round-2
High #2 honored.
Changes:
scripts/lib/brief-compose.mjs:
- digestStoryToUpstreamTopStory now emits `hash` (the digest story's
stable identifier, falls back to titleHash when absent). Without
this, rankedStoryHashes from the LLM has nothing to match against.
- composeBriefFromDigestStories accepts opts.synthesis = {lead,
threads, signals, rankedStoryHashes?, publicLead?}. When passed,
splices into envelope.digest after the stub is built. Partial
synthesis (e.g. only `lead` populated) keeps stub defaults for the
other fields — graceful degradation when L2 fallback fires.
shared/brief-filter.js:
- filterTopStories accepts optional rankedStoryHashes. New helper
applyRankedOrder re-orders stories by short-hash prefix match
BEFORE the cap is applied, so the model's editorial judgment of
importance survives MAX_STORIES_PER_USER. Stable for ties; stories
not in the ranking come after in original order. Empty/missing
ranking is a no-op (legacy callers unchanged).
shared/brief-filter.d.ts:
- filterTopStories signature gains rankedStoryHashes?: string[].
- UpstreamTopStory gains hash?: unknown (carried through from
digestStoryToUpstreamTopStory).
Tests added (tests/brief-from-digest-stories.test.mjs):
- synthesis substitutes lead/threads/signals/publicLead.
- legacy 4-arg callers (no synthesis) keep stub lead.
- partial synthesis (only lead) keeps stub threads/signals.
- rankedStoryHashes re-orders pool before cap.
- short-hash prefix match (model emits 8 chars; story carries full).
- unranked stories go after in original order.
Test results: 33/33 in brief-from-digest-stories; 182/182 across all
brief tests; full data suite 6956/6956.
Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Step 3, Codex Round-2 Low + Round-2 High #2.
* feat(brief): single canonical synthesis per user; rewire all channels
Restructures the digest cron's per-user compose + send loops to
produce ONE canonical synthesis per user per issueSlot — the lead
text every channel (email HTML, plain-text, Telegram, Slack,
Discord, webhook) and the magazine show is byte-identical. This
eliminates the "two-brain" divergence that was producing different
exec summaries on different surfaces (observed 2026-04-25 0802).
Architecture:
composeBriefsForRun (orchestration):
- Pre-annotates every eligible rule with lastSentAt + isDue once,
before the per-user pass. Same getLastSentAt helper the send loop
uses so compose + send agree on lastSentAt for every rule.
composeAndStoreBriefForUser (per-user):
- Two-pass winner walk: try DUE rules first (sortedDue), fall back
to ALL eligible rules (sortedAll) for compose-only ticks.
Preserves today's dashboard refresh contract for weekly /
twice_daily users on non-due ticks (Codex Round-4 High #1).
- Within each pass, walk by compareRules priority and pick the
FIRST candidate with a non-empty pool — mirrors today's behavior
at scripts/seed-digest-notifications.mjs:1044 and prevents the
"highest-priority but empty pool" edge case (Codex Round-4
Medium #2).
- Three-level synthesis fallback chain:
L1: generateDigestProse(fullPool, ctx={profile,greeting,!public})
L2: generateDigestProse(envelope-sized slice, ctx={})
L3: stub from assembleStubbedBriefEnvelope
Distinct log lines per fallback level so ops can quantify
failure-mode distribution.
- Generates publicLead in parallel via generateDigestProsePublic
(no userId param; cache-shared across all share-URL readers).
- Splices synthesis into envelope via composer's optional
`synthesis` arg (Step 3); rankedStoryHashes re-orders the pool
BEFORE the cap so editorial importance survives MAX_STORIES.
- synthesisLevel stored in the cron-local briefByUser entry — NOT
persisted in the envelope (renderer's assertNoExtraKeys would
reject; Codex Round-2 Medium #5).
Send loop:
- Reads lastSentAt via shared getLastSentAt helper (single source
of truth with compose flow).
- briefLead = brief?.envelope?.data?.digest?.lead — the canonical
lead. Passed to buildChannelBodies (text/Telegram/Slack/Discord),
injectEmailSummary (HTML email), and sendWebhook (webhook
payload's `summary` field). All-channel parity (Codex Round-1
Medium #6).
- Subject ternary reads cron-local synthesisLevel: 1 or 2 →
"Intelligence Brief", 3 → "Digest" (preserves today's UX for
fallback paths; Codex Round-1 Missing #5).
Removed:
- generateAISummary() — the second LLM call that produced the
divergent email lead. ~85 lines.
- AI_SUMMARY_CACHE_TTL constant — no longer referenced. The
digest:ai-summary:v1:* cache rows expire on their existing 1h
TTL (no cleanup pass).
Helpers added:
- getLastSentAt(rule) — extracted Upstash GET for digest:last-sent
so compose + send both call one source of truth.
- buildSynthesisCtx(rule, nowMs) — formats profile + greeting for
the canonical synthesis call. Preserves all today's prefs-fetch
failure-mode behavior.
Composer:
- compareRules now exported from scripts/lib/brief-compose.mjs so
the cron can sort each pass identically to groupEligibleRulesByUser.
Test results: full data suite 6962/6962 (was 6956 pre-Step 4; +6
new compose-synthesis tests from Step 3).
Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Steps 4 + 4b. Codex-approved (5 rounds).
* fix(brief-render): public-share lead fail-safe — never leak personalised lead
Public-share render path (api/brief/public/[hash].ts → renderer
publicMode=true) MUST NEVER serve the personalised digest.lead
because that string can carry profile context — watched assets,
saved-region names, etc. — written by generateDigestProse with
ctx.profile populated.
Previously: redactForPublic redacted user.name and stories.whyMatters
but passed digest.lead through unchanged. Codex Round-2 High
(security finding).
Now (v3 envelope contract):
- redactForPublic substitutes digest.lead = digest.publicLead when
the v3 envelope carries one (generated by generateDigestProsePublic
with profile=null, cache-shared across all public readers).
- When publicLead is absent (v2 envelope still in TTL window OR v3
envelope where publicLead generation failed), redactForPublic sets
digest.lead to empty string.
- renderDigestGreeting: when lead is empty, OMIT the <blockquote>
pull-quote entirely. Page still renders complete (greeting +
horizontal rule), just without the italic lead block.
- NEVER falls back to the original personalised lead.
assertBriefEnvelope still validates publicLead's contract (when
present, must be a non-empty string) BEFORE redactForPublic runs,
so a malformed publicLead throws before any leak risk.
Tests added (tests/brief-magazine-render.test.mjs):
- v3 envelope renders publicLead in pull-quote, personalised lead
text never appears.
- v2 envelope (no publicLead) omits pull-quote; rest of page
intact.
- empty-string publicLead rejected by validator (defensive).
- private render still uses personalised lead.
Test results: 68 brief-magazine-render tests pass; full data suite
remains green from prior commit.
Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Step 5, Codex Round-2 High (security).
* feat(digest): brief lead parity log + extra acceptance tests
Adds the parity-contract observability line and supplementary
acceptance tests for the canonical synthesis path.
Parity log (per send, after successful delivery):
[digest] brief lead parity user=<id> rule=<v>:<s>:<lang>
synthesis_level=<1|2|3> exec_len=<n> brief_lead_len=<n>
channels_equal=<bool> public_lead_len=<n>
When channels_equal=false an extra WARN line fires —
"PARITY REGRESSION user=… — email lead != envelope lead." Sentry's
existing console-breadcrumb hook lifts this without an explicit
captureMessage call. Plan acceptance criterion A5.
Tests added (tests/brief-llm.test.mjs, +9):
- generateDigestProsePublic: two distinct callers with identical
(sensitivity, story-pool) hit the SAME cache row (per Codex
Round-2 Medium #4 — "no PII in public cache key").
- public + private writes never collide on cache key (defensive).
- greeting bucket change re-keys the personalised cache (Brain B
parity).
- profile change re-keys the personalised cache.
- v3 cache prefix used (no v2 writes).
Test results: 77/77 in brief-llm; full data suite 6971/6971
(was 6962 pre-Step-7; +9 new public-cache tests).
Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Steps 6 (partial) + 7. Acceptance A5, A6.g, A6.f.
* test(digest): backfill A6.h/i/l/m acceptance tests via helper extraction
* fix(brief): close two correctness regressions on multi-rule + public surface
Two findings from human review of the canonical-synthesis PR:
1. Public-share redaction leaked personalised signals + threads.
The new prompt explicitly personalises both `lead` and `signals`
("personalise lead and signals"), but redactForPublic only
substituted `lead` — leaving `signals` and `threads` intact.
Public renderer's hasSignals gate would emit the signals page
whenever `digest.signals.length > 0`, exposing watched-asset /
region phrasing to anonymous readers. Same privacy bug class
the original PR was meant to close, just on different fields.
2. Multi-rule users got cross-pool lead/storyList mismatch.
composeAndStoreBriefForUser picks ONE winning rule for the
canonical envelope. The send loop then injected that ONE
`briefLead` into every due rule's channel body — even though
each rule's storyList came from its own (per-rule) digest pool.
Multi-rule users (e.g. `full` + `finance`) ended up with email
bodies leading on geopolitics while listing finance stories.
Cross-rule editorial mismatch reintroduced after the cross-
surface fix.
Fix 1 — public signals + threads:
- Envelope shape: BriefDigest gains `publicSignals?: string[]` +
`publicThreads?: BriefThread[]` (sibling fields to publicLead).
Renderer's ALLOWED_DIGEST_KEYS extended; assertBriefEnvelope
validates them when present.
- generateDigestProsePublic already returned a full prose object
(lead + signals + threads) — orchestration now captures all
three instead of just `.lead`. Composer splices each into its
envelope slot.
- redactForPublic substitutes:
digest.lead ← publicLead (or empty → omits pull-quote)
digest.signals ← publicSignals (or empty → omits signals page)
digest.threads ← publicThreads (or category-derived stub via
new derivePublicThreadsStub helper — never
falls back to the personalised threads)
- New tests cover all three substitutions + their fail-safes.
Fix 2 — per-rule synthesis in send loop:
- Each due rule independently calls runSynthesisWithFallback over
ITS OWN pool + ctx. Channel body lead is internally consistent
with the storyList (both from the same pool).
- Cache absorbs the cost: when this is the winner rule, the
synthesis hits the cache row written during the compose pass
(same userId/sensitivity/pool/ctx) — no extra LLM call. Only
multi-rule users with non-overlapping pools incur additional
LLM calls.
- magazineUrl still points at the winner's envelope (single brief
per user per slot — `(userId, issueSlot)` URL contract). Channel
lead vs magazine lead may differ for non-winner rule sends;
documented as acceptable trade-off (URL/key shape change to
support per-rule magazines is out of scope for this PR).
- Parity log refined: adds `winner_match=<bool>` field. The
PARITY REGRESSION warning now fires only when winner_match=true
AND the channel lead differs from the envelope lead (the actual
contract regression). Non-winner sends with legitimately
different leads no longer spam the alert.
Test results:
- tests/brief-magazine-render.test.mjs: 75/75 (+7 new for public
signals/threads + validator + private-mode-ignores-public-fields)
- Full data suite: 6995/6995 (was 6988; +7 net)
- typecheck + typecheck:api: clean
Plan: docs/plans/2026-04-25-002-fix-brief-email-two-brain-divergence-plan.md
Addresses 2 review findings on PR #3396 not anticipated in the
5-round Codex review.
* fix(brief): unify compose+send window, fall through filter-rejection
Address two residual risks in PR #3396 (single-canonical-brain refactor):
Risk 1 — canonical lead synthesized from a fixed 24h pool while the
send loop ships stories from `lastSentAt ?? 24h`. For weekly users
that meant a 24h-pool lead bolted onto a 7d email body — the same
cross-surface divergence the refactor was meant to eliminate, just in
a different shape. Twice-daily users hit a 12h-vs-24h variant.
Fix: extract the window formula to `digestWindowStartMs(lastSentAt,
nowMs, defaultLookbackMs)` in digest-orchestration-helpers.mjs and
call it from BOTH the compose path's digestFor closure AND the send
loop. The compose path now derives windowStart per-candidate from
`cand.lastSentAt`, identical to what the send loop will use for that
rule. Removed the now-unused BRIEF_STORY_WINDOW_MS constant.
Side-effect: digestFor now receives the full annotated candidate
(`cand`) instead of just the rule, so it can reach `cand.lastSentAt`.
Backwards-compatible at the helper level — pickWinningCandidateWithPool
forwards `cand` instead of `cand.rule`.
Cache memo hit rate drops since lastSentAt varies per-rule, but
correctness > a few extra Upstash GETs.
Risk 2 — pickWinningCandidateWithPool returned the first candidate
with a non-empty raw pool as winner. If composeBriefFromDigestStories
then dropped every story (URL/headline/shape filters), the caller
bailed without trying lower-priority candidates. Pre-PR behaviour was
to keep walking. This regressed multi-rule users whose top-priority
rule's pool happens to be entirely filter-rejected.
Fix: optional `tryCompose(cand, stories)` callback on
pickWinningCandidateWithPool. When provided, the helper calls it after
the non-empty pool check; falsy return → log filter-rejected and walk
to the next candidate; truthy → returns `{winner, stories,
composeResult}` so the caller can reuse the result. Without the
callback, legacy semantics preserved (existing tests + callers
unaffected).
Caller composeAndStoreBriefForUser passes a no-synthesis compose call
as tryCompose — cheap pure-JS, no I/O. Synthesis only runs once after
the winner is locked in, so the perf cost is one extra compose per
filter-rejected candidate, no extra LLM round-trips.
Tests:
- 10 new cases in tests/digest-orchestration-helpers.test.mjs
covering: digestFor receiving full candidate; tryCompose
fall-through to lower-priority; all-rejected returns null;
composeResult forwarded; legacy semantics without tryCompose;
digestWindowStartMs lastSentAt-vs-default branches; weekly +
twice-daily window parity assertions; epoch-zero ?? guard.
- Updated tests/digest-cache-key-sensitivity.test.mjs static-shape
regex to match the new `cand.rule.sensitivity` cache-key shape
(intent unchanged: cache key MUST include sensitivity).
Stacked on PR #3396 — targets feat/brief-two-brain-divergence.
|
||
|
Elie Habib
|
9c14820c69 |
fix(digest): brief filter-drop instrumentation + cache-key correctness (#3387)
* fix(digest): include sensitivity in digestFor cache key
buildDigest filters by rule.sensitivity BEFORE dedup, but digestFor
memoized only on (variant, lang, windowStart). Stricter-sensitivity
users in a shared bucket inherited the looser populator's pool,
producing the wrong story set and defeating downstream topic-grouping
adjacency once filterTopStories re-applied sensitivity.
Solution 1 from docs/plans/2026-04-24-004-fix-brief-topic-adjacency-defects-plan.md.
* feat(digest): instrument per-user filterTopStories drops
Adds an optional onDrop metrics callback to filterTopStories and threads
it through composeBriefFromDigestStories. The seeder aggregates counts
per composed brief and emits one structured log line per user per tick:
[digest] brief filter drops user=<id> sensitivity=<s> in=<count>
dropped_severity=<n> dropped_url=<n> dropped_headline=<n>
dropped_shape=<n> out=<count>
Decides whether the conditional Solution 3 (post-filter regroup) is
warranted by quantifying how often post-group filter drops puncture
multi-member topics in production. No behaviour change for callers
that omit onDrop.
Solution 0 from docs/plans/2026-04-24-004-fix-brief-topic-adjacency-defects-plan.md.
* fix(digest): close two Sol-0 instrumentation gaps from code review
Review surfaced two P2 gaps in the filter-drop telemetry that weakened
its diagnostic purpose for Sol-3 gating:
1. Cap-truncation silent drop: filterTopStories broke on
`out.length >= maxStories` BEFORE the onDrop emit sites, so up to
(DIGEST_MAX_ITEMS - MAX_STORIES_PER_USER) stories per user were
invisible. Added a 'cap' reason to DropMetricsFn and emit one event
per skipped story so `in - out - sum(dropped_*) == 0` reconciles.
2. Wipeout invisibility: composeAndStoreBriefForUser only logged drop
stats for the WINNING candidate. When every candidate composed to
null, the log line never fired — exactly the wipeout case Sol-0
was meant to surface. Now tracks per-candidate drops and emits an
aggregate `outcome=wipeout` line covering all attempts.
Also tightens the digest-cache-key sensitivity regex test to anchor
inside the cache-key template literal (it would otherwise match the
unrelated `chosenCandidate.sensitivity ?? 'high'` in the new log line).
PR review residuals from
docs/plans/2026-04-24-004-fix-brief-topic-adjacency-defects-plan.md
ce-code-review run 20260424-232911-37a2d5df.
* chore: ignore .context/ ce-code-review run artifacts
The ce-code-review skill writes per-run artifacts (reviewer JSON,
synthesis.md, metadata.json) under .context/compound-engineering/.
These are local-only — neither tracked nor linted.
* fix(digest): emit per-attempt filter-drop rows, not per-user
Addresses two PR #3387 review findings:
- P2: Earlier candidates that composed to null (wiped out by post-group
filtering) had their dropStats silently discarded when a later
candidate shipped — exactly the signal Sol-0 was meant to surface.
- P3: outcome=wipeout row was labeled with allCandidateDrops[0]
.sensitivity, misleading when candidates within one user have
different sensitivities.
Fix: emit one structured row per attempted candidate, tagged with that
candidate's own sensitivity and variant. Outcome is shipped|rejected.
A wipeout is now detectable as "all rows for this user are rejected
within the tick" — no aggregate-row ambiguity. Removes the
allCandidateDrops accumulator entirely.
* fix(digest): align composeBriefFromDigestStories sensitivity default to 'high'
Addresses PR #3387 review (P2): composeBriefFromDigestStories defaulted
to `?? 'all'` while buildDigest, the digestFor cache key, and the new
per-attempt log line all default to `?? 'high'`. The mismatch is
harmless in production (the live cron path pre-filters the pool) but:
- A non-prefiltered caller with undefined sensitivity would silently
ship medium/low stories.
- Per-attempt telemetry labels the attempt as `sensitivity=high` while
compose actually applied 'all' — operators are misled.
Aligning compose to 'high' makes the four sites agree and the telemetry
honest. Production output is byte-identical (input pool was already
'high'-filtered upstream).
Adds 3 regression tests asserting the new default: critical/high admitted,
medium/low dropped, and onDrop fires reason=severity for the dropped
levels (locks in alignment with per-attempt telemetry).
* fix(digest): align remaining sensitivity defaults to 'high'
Addresses PR #3387 review (P2 + P3): three more sites still defaulted
missing sensitivity to 'all' while compose/buildDigest/cache/log now
treat it as 'high'.
P2 — compareRules (scripts/lib/brief-compose.mjs:35-36): the rank
function used to default to 'all', placing legacy undefined-sensitivity
rules FIRST in the candidate order. Compose then applied a 'high'
filter to them, shipping a narrow brief while an explicit 'all' rule
for the same user was never tried. Aligned to 'high' so the rank
matches what compose actually applies.
P3 — enrichBriefEnvelopeWithLLM (scripts/lib/brief-llm.mjs:526):
the digest prompt and cache key still used 'all' for legacy rules,
misleading personalization ("Reader sensitivity level: all" while the
brief contains only critical/high stories) and busting the cache for
legacy vs explicit-'all' rows that should share entries.
Also aligns the @deprecated composeBriefForRule (line 164) for
consistency, since tests still import it.
3 new regression tests in tests/brief-composer-rule-dedup.test.mjs
lock in the new ranking: explicit 'all' beats undefined-sensitivity,
undefined-sensitivity ties with explicit 'high' (decided by updatedAt),
and groupEligibleRulesByUser candidate order respects the rank.
6853/6853 tests pass (was 6850 → +3).
|
||
|
Elie Habib
|
425507d15a |
fix(brief): category-gated context + RELEVANCE RULE to stop formulaic grounding (#3281)
* fix(brief): category-gated context + RELEVANCE RULE to stop formulaic grounding
Shadow-diff of 15 v2 pairs (2026-04-22) showed the analyst pattern-
matching the loudest context numbers — VIX 19.50, top forecast
probability, MidEast FX stress 77 — into every story regardless of
editorial fit. A Rwanda humanitarian story about refugees cited VIX;
an aviation story cited a forecast probability.
Root cause: every story got the same 6-bundle context block, so the
LLM had markets / forecasts / macro in-hand and the "cite a specific
fact" instruction did the rest.
Two-layer fix:
1. STRUCTURAL — sectionsForCategory() maps the story's category to
an editorially-relevant subset of bundles. Humanitarian stories
don't see marketData / forecasts / macroSignals; diplomacy gets
riskScores only; market/energy gets markets+forecasts but drops
riskScores. The model physically cannot cite what it wasn't
given. Unknown categories fall back to all six (backcompat).
2. PROMPT — WHY_MATTERS_ANALYST_SYSTEM_V2 adds a RELEVANCE RULE
that explicitly permits grounding in headline/description
actors when no context fact is a clean fit, and bans dragging
off-topic market metrics into humanitarian/aviation/diplomacy
stories. The prompt footer (inline, per-call) restates the
same guardrail — models follow inline instructions more
reliably than system-prompt constraints on longer outputs.
Cache keys bumped to invalidate the formulaic v5 output: endpoint
v5 to v6, shadow v3 to v4. Adds 11 unit tests pinning the 5
policies + default fallback + humanitarian structural guarantee +
market policy does-see-markets + guardrail footer presence.
Observability: endpoint now logs policyLabel per call so operators
can confirm in Vercel logs that humanitarian/aviation stories are
NOT seeing marketData without dumping the full prompt.
* test(brief): address greptile P2 — sync MAX_BODY_BYTES + add parseWhyMattersV2 coverage
Greptile PR #3281 review raised two P2 test-quality issues:
1. Test-side MAX_BODY_BYTES mirror was still 4096 — the endpoint
was bumped to 8192 in PR #3269 (v2 output + description). With
the stale constant, a payload in the 4097–8192 range was
accepted by the real endpoint but looked oversize in the test
mirror, letting the body-cap invariant silently drift. Fixed
by syncing to 8192 + bumping the bloated fixture to 10_000
bytes so a future endpoint-cap bump doesn't silently
re-invalidate the assertion.
2. parseWhyMattersV2 (the only output-validation gate on the
analyst path) had no dedicated unit tests. Adds 11 targeted
cases covering: valid 2 and 3 sentence output, 100/500 char
bounds (incl. boundary assertions), all 6 banned preamble
phrases, section-label leaks (SITUATION/ANALYSIS/Watch),
markdown leakage (#, -, *, 1.), stub echo rejection, smart/
plain quote stripping, non-string defensive branch, and
whitespace-only strings.
Suite size: 50 to 61 tests, all green.
* fix(brief): add aviation policy to sectionsForCategory (PR #3281 review P1)
Reviewer caught that aviation was named in WHY_MATTERS_ANALYST_SYSTEM_V2's
RELEVANCE RULE as a category banned from off-topic market metrics, but
had no matching regex entry in CATEGORY_SECTION_POLICY. So 'Aviation
Incident' / 'Airspace Closure' / 'Plane Crash' / 'Drone Incursion' all
fell through to DEFAULT_SECTIONS and still got all 6 bundles including
marketData, forecasts, and macroSignals — exactly the VIX / forecast
probability pattern the PR claimed to structurally prevent.
Reproduced on HEAD before fix:
Aviation Incident -> default
Airspace Closure -> default
Plane Crash -> default
...etc.
Fix:
1. Adds aviation policy (same 3 bundles as humanitarian/diplomacy/
tech: worldBrief, countryBrief, riskScores).
2. Adds dedicated aviation-gating test with 6 category variants.
3. Adds meta-invariant test: every category named in the system
prompt's RELEVANCE RULE MUST have a structural policy entry,
asserting policyLabel !== 'default'. If someone adds a new
category name to the prompt in the future, this test fires
until they wire up a regex — prevents soft-guard drift.
4. Removes 'Aviation Incident' from the default-fall-through test
list (it now correctly matches aviation).
No cache bump needed — v6 was published to the feature branch only a
few minutes ago, no production entries have been written yet.
|
||
|
Elie Habib
|
ec35cf4158 |
feat(brief): analyst prompt v2 — multi-sentence, grounded, story description (#3269)
* feat(brief): analyst prompt v2 — multi-sentence, grounded, includes story description
Shadow-diff of 12 prod stories on 2026-04-21 showed v1 analyst output
indistinguishable from legacy Gemini: identical single-sentence
abstraction ("destabilize / systemic / sovereign risk repricing") with
no named actors, metrics, or dates — in several cases Gemini was MORE
specific.
Root cause: 18–30 word cap compressed context specifics out.
v2 loosens three dials at once so we can settle the A/B:
1. New system prompt WHY_MATTERS_ANALYST_SYSTEM_V2 — 2–3 sentences,
40–70 words, implicit SITUATION→ANALYSIS→(optional) WATCH arc,
MUST cite one specific named actor / metric / date / place from
the context. Analyst path only; gemini path stays on v1.
2. New parser parseWhyMattersV2 — accepts 100–500 chars, rejects
preamble boilerplate + leaked section labels + markdown.
3. Story description plumbed through — endpoint body accepts optional
story.description (≤ 1000 chars, body cap bumped 4 KB → 8 KB).
Cron forwards it when upstream has one (skipped when it equals the
headline — no new signal).
Cache + shadow bumped v3 → v4 / v1 → v2 so fresh output lands on the
first post-deploy cron tick. maxTokens 180 → 260 for ~3× output length.
If shadow-diff 24h after deploy still shows no delta vs gemini, kill
is BRIEF_WHY_MATTERS_PRIMARY=gemini on Vercel (instant, no redeploy).
Tests: 6059 pass (was 6022 + 37 new). typecheck × 2 clean.
* fix(brief): stop truncating v2 multi-sentence output + description in cache hash
Two P1s caught in PR #3269 review.
P1a — cron reparsed endpoint output with v1 single-sentence parser,
silently dropping sentences 2+3 of v2 analyst output. The endpoint had
ALREADY validated the string (parseWhyMattersV2 for analyst path;
parseWhyMatters for gemini). Re-parsing with v1 took only the first
sentence — exact regression #3269 was meant to fix.
Fix: trust the endpoint. Replace re-parse with bounds check (30–500
chars) + stub-echo reject. Added regression test asserting multi-
sentence output reaches the envelope unchanged.
P1b — `story.description` flowed into the analyst prompt but NOT into
the cache hash. Two requests with identical core fields but different
descriptions collided on one cache slot → second caller got prose
grounded in the FIRST caller's description.
Fix: add `description` as the 6th field of `hashBriefStory`. Bump
endpoint cache v4→v5 and shadow v2→v3 so buggy 5-field entries are
dropped. Updated the parity sentinel in brief-llm-core.test.mjs to
match 6-field semantics. Added regression tests covering different-
descriptions-differ and present-vs-absent-differ.
Tests: 6083 pass. typecheck × 2 clean.
|
||
|
Elie Habib
|
2f19d96357 |
feat(brief): route whyMatters through internal analyst-context endpoint (#3248)
* feat(brief): route whyMatters through internal analyst-context endpoint
The brief's "why this is important" callout currently calls Gemini on
only {headline, source, threatLevel, category, country} with no live
state. The LLM can't know whether a ceasefire is on day 2 or day 50,
that IMF flagged >90% gas dependency in UAE/Qatar/Bahrain, or what
today's forecasts look like. Output is generic prose instead of the
situational analysis WMAnalyst produces when given live context.
This PR adds an internal Vercel edge endpoint that reuses a trimmed
variant of the analyst context (country-brief, risk scores, top-3
forecasts, macro signals, market data — no GDELT, no digest-search)
and ships it through a one-sentence LLM call with the existing
WHY_MATTERS_SYSTEM prompt. The endpoint owns its own Upstash cache
(v3 prefix, 6h TTL), supports a shadow mode that runs both paths in
parallel for offline diffing, and is auth'd via RELAY_SHARED_SECRET.
Three-layer graceful degradation (endpoint → legacy Gemini-direct →
stub) keeps the brief shipping on any failure.
Env knobs:
- BRIEF_WHY_MATTERS_PRIMARY=analyst|gemini (default: analyst; typo → gemini)
- BRIEF_WHY_MATTERS_SHADOW=0|1 (default: 1; only '0' disables)
- BRIEF_WHY_MATTERS_SHADOW_SAMPLE_PCT=0..100 (default: 100)
- BRIEF_WHY_MATTERS_ENDPOINT_URL (Railway, optional override)
Cache keys:
- brief:llm:whymatters:v3:{hash16} — envelope {whyMatters, producedBy,
at}, 6h TTL. Endpoint-owned.
- brief:llm:whymatters:shadow:v1:{hash16} — {analyst, gemini, chosen,
at}, 7d TTL. Fire-and-forget.
- brief:llm:whymatters:v2:{hash16} — legacy. Cron's fallback path
still reads/writes during the rollout window; expires in ≤24h.
Tests: 6022 pass (existing 5915 + 12 core + 36 endpoint + misc).
typecheck + typecheck:api + biome on changed files clean.
Plan (Codex-approved after 4 rounds):
docs/plans/2026-04-21-001-feat-brief-why-matters-analyst-endpoint-plan.md
* fix(brief): address /ce:review round 1 findings on PR #3248
Fixes 5 findings from multi-agent review, 2 of them P1:
- #241 P1: `.gitignore !api/internal/**` was too broad — it re-included
`.env`, `.env.local`, and any future secret file dropped into that
directory. Narrowed to explicit source extensions (`*.ts`, `*.js`,
`*.mjs`) so parent `.env` / secrets rules stay in effect inside
api/internal/.
- #242 P1: `Dockerfile.digest-notifications` did not COPY
`shared/brief-llm-core.js` + `.d.ts`. Cron would have crashed at
container start with ERR_MODULE_NOT_FOUND. Added alongside
brief-envelope + brief-filter COPY lines.
- #243 P2: Cron dropped the endpoint's source/producedBy ground-truth
signal, violating PR #3247's own round-3 memory
(feedback_gate_on_ground_truth_not_configured_state.md). Added
structured log at the call site: `[brief-llm] whyMatters source=<src>
producedBy=<pb> hash=<h>`. Endpoint response now includes `hash` so
log + shadow-record pairs can be cross-referenced.
- #244 P2: Defense-in-depth prompt-injection hardening. Story fields
flowed verbatim into both LLM prompts, bypassing the repo's
sanitizeForPrompt convention. Added sanitizeStoryFields helper and
applied in both analyst and gemini paths.
- #245 P2: Removed redundant `validate` option from callLlmReasoning.
With only openrouter configured in prod, a parse-reject walked the
provider chain, then fell through to the other path (same provider),
then the cron's own fallback (same model) — 3x billing on one reject.
Post-call parseWhyMatters check already handles rejection cleanly.
Deferred to P3 follow-ups (todos 246-248): singleflight, v2 sunset,
misc polish (country-normalize LOC, JSDoc pruning, shadow waitUntil,
auto-sync mirror, context-assembly caching).
Tests: 6022 pass. typecheck + typecheck:api clean.
* fix(brief-why-matters): ctx.waitUntil for shadow write + sanitize legacy fallback
Two P2 findings on PR #3248:
1. Shadow record was fire-and-forget without ctx.waitUntil on an Edge
function. Vercel can terminate the isolate after response return,
so the background redisPipeline write completes unreliably — i.e.
the rollout-validation signal the shadow keys were supposed to
provide was flaky in production.
Fix: accept an optional EdgeContext 2nd arg. Build the shadow
promise up front (so it starts executing immediately) then register
it with ctx.waitUntil when present. Falls back to plain unawaited
execution when ctx is absent (local harness / tests).
2. scripts/lib/brief-llm.mjs legacy fallback path called
buildWhyMattersPrompt(story) on raw fields with no sanitization.
The analyst endpoint sanitizes before its own prompt build, but
the fallback is exactly what runs when the endpoint misses /
errors — so hostile headlines / sources reached the LLM verbatim
on that path.
Fix: local sanitizeStoryForPrompt wrapper imports sanitizeForPrompt
from server/_shared/llm-sanitize.js (existing pattern — see
scripts/seed-digest-notifications.mjs:41). Wraps story fields
before buildWhyMattersPrompt. Cache key unchanged (hash is over raw
story), so cache parity with the analyst endpoint's v3 entries is
preserved.
Regression guard: new test asserts the fallback prompt strips
"ignore previous instructions", "### Assistant:" line prefixes, and
`<|im_start|>` tokens when injection-crafted fields arrive.
Typecheck + typecheck:api clean. 6023 / 6023 data tests pass.
* fix(digest-cron): COPY server/_shared/llm-sanitize into digest-notifications image
Reviewer P1 on PR #3248: my previous commit (
|
||
|
Elie Habib
|
ecd56d4212 |
feat(feeds): add IRNA, Mehr, Jerusalem Post, Ynetnews to middleeast (#3236)
* feat(feeds): add IRNA, Mehr, Jerusalem Post, Ynetnews to middleeast Four direct-RSS sources verified from a clean IP and absent everywhere in the repo (src/config/feeds.ts, scripts/seed-*, ais-relay.cjs, RSS allowlist). Closes the highest-ROI Iran / Israel domestic-press gap from the ME source audit (PR #3226) with zero infra changes. - IRNA https://en.irna.ir/rss - Mehr News https://en.mehrnews.com/rss - Jerusalem Post https://www.jpost.com/rss/rssfeedsheadlines.aspx - Ynetnews https://www.ynetnews.com/Integration/StoryRss3089.xml Propaganda-risk metadata: - IRNA + Mehr tagged high / Iran state-affiliated (join Press TV). - JPost + Ynetnews tagged low with knownBiases for transparency. RSS allowlist updated in all three mirrors (shared/, scripts/shared/, api/_rss-allowed-domains.js) per the byte-identical mirror contract enforced by tests/edge-functions.test.mjs. Deferred (separate PRs): - Times of Israel: already in allowlist; was removed from feeds for cloud-IP 403. Needs Decodo routing. - IDF Spokesperson: idf.il has no direct RSS endpoint; needs scraper. - Tasnim / Press TV RSS / Israel Hayom: known cloud-IP blocks. - WAM / SPA / KUNA / QNA / BNA: public RSS endpoints are dead; sites migrated to SPAs or gate with 403. Plan doc (PR #3226) overstated the gap: it audited only feeds.ts and missed that travel advisories + US Embassy alerts are already covered by scripts/seed-security-advisories.mjs. NOTAM claim in that doc is also wrong: we use ICAO's global NOTAM API, not FAA. * fix(feeds): enable IRNA, Mehr, Jerusalem Post, Ynetnews by default Reviewer on #3236 flagged that adding the four new ME feeds to FULL_FEEDS.middleeast alone leaves them disabled on first run, because App.ts:661 persists computeDefaultDisabledSources() output derived from DEFAULT_ENABLED_SOURCES. Users would have to manually re-enable via Settings > Sources, defeating the purpose of broadening the default ME mix. Add the four new sources to DEFAULT_ENABLED_SOURCES.middleeast so they ship on by default. Placement keeps them adjacent to their peers (IRNA / Mehr with the other Iran sources, JPost / Ynetnews after Haaretz). Risk/slant tags already in SOURCE_PROPAGANDA_RISK ensure downstream digest dedup + summarization weights them correctly. * style(feeds): move JPost + Ynetnews under Low-risk section header Greptile on #3236 flagged that both entries are risk: 'low' but were inserted above the `// Low risk - Independent with editorial standards` comment header, making the section boundary misleading for future contributors. Shift them under the header where they belong. No runtime change; cosmetic ordering only. |
||
|
Elie Habib
|
38e6892995 |
fix(brief): per-run slot URL so same-day digests link to distinct briefs (#3205)
* fix(brief): per-run slot URL so same-day digests link to distinct briefs
Digest emails at 8am and 1pm on the same day pointed to byte-identical
magazine URLs because the URL was keyed on YYYY-MM-DD in the user tz.
Each compose run overwrote the single daily envelope in place, and the
composer rolling 24h story window meant afternoon output often looked
identical to morning. Readers clicking an older email got whatever the
latest cron happened to write.
Slot format is now YYYY-MM-DD-HHMM (local tz, per compose run). The
magazine URL, carousel URLs, and Redis key all carry the slot, and each
digest dispatch gets its own frozen envelope that lives out the 7d TTL.
envelope.data.date stays YYYY-MM-DD for rendering "19 April 2026".
The digest cron also writes a brief:latest:{userId} pointer (7d TTL,
overwritten each compose) so the dashboard panel and share-url endpoint
can locate the most recent brief without knowing the slot. The
previous date-probing strategy does not work once keys carry HHMM.
No back-compat for the old YYYY-MM-DD format: the verifier rejects it,
the composer only ever writes the new shape, and any in-flight
notifications signed under the old format will 403 on click. Acceptable
at the rollout boundary per product decision.
* fix(brief): carve middleware bot allowlist to accept slot-format carousel path
BRIEF_CAROUSEL_PATH_RE in middleware.ts was still matching only the
pre-slot YYYY-MM-DD segment, so every slot-based carousel URL emitted
by the digest cron (YYYY-MM-DD-HHMM) would miss the social allowlist
and fall into the generic bot gate. Telegram/Slack/Discord/LinkedIn
image fetchers would 403 on sendMediaGroup, breaking previews for the
new digest links.
CI missed this because tests/middleware-bot-gate.test.mts still
exercised the old /YYYY-MM-DD/ path shape. Swap the fixture to the
slot format and add a regression asserting the pre-slot shape is now
rejected, so legacy links cannot silently leak the allowlist after
the rollout.
* fix(brief): preserve caller-requested slot + correct no-brief share-url error
Two contract bugs in the slot rollout that silently misled callers:
1. GET /api/latest-brief?slot=X where X has no envelope was returning
{ status: 'composing', issueDate: <today UTC> } — which reads as
"today's brief is composing" instead of "the specific slot you
asked about doesn't exist". A caller probing a known historical
slot would get a completely unrelated "today" signal. Now we echo
the requested slot back (issueSlot + issueDate derived from its
date portion) when the caller supplied ?slot=, and keep the
UTC-today placeholder only for the no-param path.
2. POST /api/brief/share-url with no slot and no latest-pointer was
falling into the generic invalid_slot_shape 400 branch. That is
not an input-shape problem; it is "no brief exists yet for this
user". Return 404 brief_not_found — the same code the
existing-envelope check returns — so callers get one coherent
contract: either the brief exists and is shareable, or it doesn't
and you get 404.
|
||
|
Elie Habib
|
81536cb395 |
feat(brief): source links, LLM descriptions, strip suffix (envelope v2) (#3181)
* feat(brief): source links, LLM descriptions, strip publisher suffix (envelope v2) Three coordinated fixes to the magazine content pipeline. 1. Headlines were ending with " - AP News" / " | Reuters" etc. because the composer passed RSS titles through verbatim. Added stripHeadlineSuffix() in brief-compose.mjs, conservative case- insensitive match only when the trailing token equals primarySource, so a real subtitle that happens to contain a dash still survives. 2. Story descriptions were the headline verbatim. Added generateStoryDescription to brief-llm.mjs, plumbed into enrichBriefEnvelopeWithLLM: one additional LLM call per story, cached 24h on a v1 key covering headline, source, severity, category, country. Cache hits are revalidated via parseStoryDescription so a bad row cannot flow to the envelope. Falls through to the cleaned headline on any failure. 3. Source attribution was plain text, no outgoing link. Bumped BRIEF_ENVELOPE_VERSION to 2, added BriefStory.sourceUrl. The composer now plumbs story:track:v1.link through digestStoryToUpstreamTopStory, UpstreamTopStory.primaryLink, filterTopStories, BriefStory.sourceUrl. The renderer wraps the Source line in an anchor with target=_blank, rel=noopener noreferrer, and UTM params (utm_source=worldmonitor, utm_medium=brief, utm_campaign=<issueDate>, utm_content=story- <rank>). UTM appending is idempotent, publisher-attributed URLs keep their own utm_source. Envelope validation gains a validateSourceUrl step (https/http only, no userinfo credentials, parseable absolute URL). Stories without a valid upstream link are dropped by filterTopStories rather than shipping with an unlinked source. Tests: 30 renderer tests to 38; new assertions cover UTM presence on every anchor, HTML-escaping of ampersands in hrefs, pre-existing UTM preservation, and all four validator rejection modes. New composer tests cover suffix stripping, link plumb-through, and v2 drop-on-no- link behaviour. New LLM tests for generateStoryDescription cover cache hit/miss, revalidation of bad rows, 24h TTL, and null-on- failure. * fix(brief): v1 back-compat window on renderer + consolidate story hash helper Two P1/P2 review findings on #3181. P1 (v1 back-compat). Bumping BRIEF_ENVELOPE_VERSION 1 to 2 made every v1 envelope still resident in Redis under the 7-day TTL fail assertBriefEnvelope. The hosted /api/brief route would 404 "expired" and the /api/latest-brief preview would downgrade to "composing", breaking already-issued links from the preceding week. Fix: renderer now accepts SUPPORTED_ENVELOPE_VERSIONS = Set([1, 2]) on READ. BRIEF_ENVELOPE_VERSION stays at 2 and is the only version the composer ever writes. BriefStory.sourceUrl is required when version === 2 and absent on v1; when rendering a v1 story the source line degrades to plain text (no anchor), matching pre-v2 appearance. When the TTL window passes the set can shrink to [2] in a follow-up. P2 (hash dedup). hashStoryDescription was byte-identical to hashStory, inviting silent drift if one prompt gains a field the other forgets. Consolidated into hashBriefStory. Cache key separation remains via the distinct prefixes (brief:llm:whymatters:v2:/brief:llm:description:v1:). Tests: adds 3 v1 back-compat assertions (plain source line, field validation still runs, defensive sourceUrl check), updates the version-mismatch assertion to match the new supported-set message. 161/161 pass (was 158). Full test:data 5706/5706. |
||
|
Elie Habib
|
45da551d17 |
feat(brief): per-user composer writing brief:{userId}:{issueDate} (Phase 3a) (#3154)
* feat(brief): per-user composer writing brief:{userId}:{issueDate} (Phase 3a)
Phase 3a of docs/plans/2026-04-17-003. Produces the Redis-resident
envelopes that Phases 1 (renderer) and 2 (edge routes) already know
how to serve, so after this ships the end-to-end read path works
with real data.
Files:
- shared/brief-filter.{js,d.ts}: pure helpers. normaliseThreatLevel
maps upstream 'moderate' -> 'medium' (contract pinned the union in
Phase 1). filterTopStories applies sensitivity thresholds and caps
at maxStories. assembleStubbedBriefEnvelope builds a full envelope
with stubbed greeting/lead/threads/signals and runs it through the
renderer's assertBriefEnvelope so no malformed envelope is ever
persisted. issueDateInTz computes per-user local date via Intl
with UTC fallback.
- scripts/seed-brief-composer.mjs: Railway cron. Reads
news:insights:v1 once, fetches enabled alert rules via the
existing /relay/digest-rules endpoint (same set
seed-digest-notifications uses), then for each rule computes the
user's local issue date, filters stories, assembles an envelope,
and SETEX brief:{userId}:{issueDate} with 7-day TTL. Respects
aiDigestEnabled opt-in. Honours SIGTERM. Exits non-zero when >5%
of rules fail so Railway surfaces structural breakage.
- Dockerfile.seed-brief-composer: standalone container. Copies the
minimum set (composer + shared/ contract + renderer validator +
Upstash helper + seed-envelope unwrapper).
- tests/brief-filter.test.mjs: 22 pure-function tests covering
severity normalisation (including 'moderate' alias), sensitivity
thresholds, story cap, empty-title drop, envelope assembly passes
the strict renderer validator, tz-aware date math across +UTC/-UTC
offsets with a bad-timezone fallback.
Out of scope for this PR:
- LLM-generated whyMatters / lead / signals (Phase 3b).
- brief_ready event fan-out to notification-relay (Phase 3c).
- Dashboard panel that consumes /api/latest-brief (Phase 4).
Pre-merge runbook:
1. Create a new Railway service from Dockerfile.seed-brief-composer.
2. Set env vars (UPSTASH_*, CONVEX_URL, RELAY_SHARED_SECRET) — reuse
the values already in the digest service.
3. Add a cron schedule (suggested: hourly at :05 so it lands between
the insights-seeder tick and the digest cron).
4. Verify first run: check service logs for
"[brief-composer] Done: success=X ..." and a reader's
/api/latest-brief should stop returning 'composing' within one
cron cycle.
Tests: 72/72 (22 brief-filter + 30 render + 20 HMAC). Typecheck +
lint clean. Composer script parses with node --check.
* fix(brief): aiDigestEnabled default + per-user rule dedupe
Addresses two fourth-round review findings on PR #3154.
1. aiDigestEnabled default parity (todo 224). Composer was checking
`!rule.aiDigestEnabled`, which skips legacy rules that predate the
optional field. The rest of the codebase defaults it to true
(seed-digest-notifications.mjs:914 uses `!== false`;
notifications-settings.ts:228 uses `?? true`; the Convex setter
defaults to true). Flipped the composer to `=== false` so only an
explicit opt-out skips the brief.
2. Multi-variant last-write-wins (todo 225). alertRules are
(userId, variant)-scoped but the brief key is user-scoped
(brief:{userId}:{issueDate}). Users with the full+finance+tech
variants all enabled would produce three competing writes with a
nondeterministic survivor. Added dedupeRulesByUser() that picks
one rule per user: prefers 'full' variant, then most permissive
sensitivity (all > high > critical), tie-breaking on earliest
updatedAt for stability across input reordering. Logs the
occurrence so we can see how often users have multi-variant
configs.
Also hardened against future regressions:
- Moved env-var guards + main() call behind an isMain() check
(feedback_seed_isMain_guard). Previously, importing the script
from a test would fire process.exit(0) on the
BRIEF_COMPOSER_ENABLED=0 branch and kill the test runner. Tests
now load the file cleanly.
- Exported dedupeRulesByUser so the tests can exercise the selection
logic directly.
- The new tests/brief-composer-rule-dedup.test.mjs includes a
cross-module assertion that seed-digest-notifications.mjs still
reads `rule.aiDigestEnabled !== false`. If the digest cron ever
drifts, this test fails loud — the brief and digest must agree on
who is eligible.
Tests: 83/83 (was 72; +6 dedupe cases + 5 aiDigestEnabled parity
cases). Typecheck + lint clean.
* fix(brief): dedupe order + failure-rate denominator
Addresses two fifth-round review findings on PR #3154.
1. Dedupe was picking a preferred variant BEFORE checking whether it
could actually emit a brief (todo 226). A user with
aiDigestEnabled=false on 'full' but true on 'finance' got skipped
entirely; same for a user with sensitivity='critical' on 'full'
that filters to zero stories while 'finance' has matching content.
Replaced dedupeRulesByUser with groupEligibleRulesByUser: pre-
filters opted-out rules, then returns ALL eligible variants per
user in preference order. The main loop walks candidates and
takes the first one whose story filter produces non-empty content.
Fallback is cheap (story filter is pure) and preserves the 'full'-
first + most-permissive-sensitivity tie-breakers from before.
dedupeRulesByUser is kept as a thin wrapper for the existing tests;
new tests exercise the group+fallback path directly (opt-out +
opt-in sibling, all-opted-out drop, ordering stability).
2. Failure gate denominator drifted from numerator (todo 227). After
dedupe, `failed` counts per-user but the gate still compared to
pre-dedupe rules.length. 60 rules → 10 users → 2 failed writes =
20% real failure hidden behind a 60-rule denominator.
Fix: denominator is now eligibleUserCount (Map size after
group-and-filter). Log line reports rules + eligible_users +
success + skipped_empty + failed + duration so ops can see the
full shape.
Tests: 86/86 (was 83; +3 new: opt-out+sibling, all-opted-out drop,
candidate-ordering). Typecheck clean, node --check clean, biome clean.
* fix(brief): body-POST SETEX + attempted-only failure denominator
Addresses two sixth-round review findings on PR #3154.
1. Upstash SETEX (todo 228). The previous write path URL-encoded the
full envelope into /setex/{key}/{ttl}/{payload} which can blow
past proxy/edge/Node HTTP request-target limits for realistic
12-story briefs (5-20 KB JSON). Switched to body-POST via the
existing `redisPipeline` helper — same transport every other
write in the repo uses. Per-command error surface is preserved:
the wrapper throws on null pipeline response or on a {error}
entry in the result array.
2. Failure-rate denominator (todo 229). Earlier round switched
denominator from pre-dedupe rules.length to eligibleUserCount,
but the numerator only counts users that actually reached a
write attempt. skipped_empty users inflate eligibleUserCount
without being able to fail, so 4/4 failed writes against 100
eligible (96 skipped_empty) reads as 4% and silently passes.
Denominator is now `success + failed` (attempted writes only).
Extracted shouldExitNonZero({success, failed}) so the denominator
contract lives in a pure function with 7 test cases:
- 0 failures → no exit
- 100% failure on small volume → exits
- 1/20 at exact 5% threshold → exits (documented boundary)
- 1/50 below threshold → no exit
- 2/10 above Math.max(1) floor → exits
- 1/1 single isolated failure → exits
- 0 attempted (no signal) → no exit
Tests: 93/93 (was 86; +7 threshold cases). Typecheck + lint clean.
|
||
|
Elie Habib
|
66ca645571 |
feat(brief): WorldMonitor Brief magazine renderer + envelope contract (Phase 1) (#3150)
* feat(brief): add WorldMonitor Brief magazine renderer + envelope contract
Phase 1 of the WorldMonitor Brief plan (docs/plans/2026-04-17-003-feat-
worldmonitor-brief-magazine-plan.md). Establishes the integration
boundary between the future per-user composer and every consumer
surface (hosted edge route, dashboard panel, email teaser, carousel,
Tauri reader).
- shared/brief-envelope.{d.ts,js}: BriefEnvelope + BRIEF_ENVELOPE_VERSION
- shared/render-brief-magazine.{d.ts,js}: pure (envelope) -> HTML
- tests/brief-magazine-render.test.mjs: 28 shape + chrome + leak tests
Page sequence is derived from data (no hardcoded counts). Threads
split into 03a/03b when more than six; Signals page is omitted when
signals is empty; story palette alternates light/dark by index
parity. Forbidden-field guard asserts importanceScore / primaryLink /
pubDate / model + provider names never appear in rendered HTML.
No runtime impact: purely additive, no consumers yet.
* fix(brief): address code review findings on PR #3150
Addresses all seven review items from todos/205..211.
P1 (merge blockers):
- todo 205: forbidden-field test no longer matches free-text content
like 'openai' / 'claude' / 'gemini' (false-positives on legitimate
stories). Narrowed to JSON-key structural tokens
('"importanceScore":', '"_seed":', etc.) and added a sentinel-
poisoning test that injects values into non-data envelope fields and
asserts they never appear in output.
- todo 206: drop 'moderate' from BriefThreatLevel union (synonym of
'medium'). Four-value ladder: critical | high | medium | low. Added
THREAT_LABELS map + HIGHLIGHTED_LEVELS set so display label and
highlight rule live together instead of inline char-case + hardcoded
comparison.
- todo 207: replace the minimal validator with assertBriefEnvelope()
that walks every required field (user.name/tz, date YYYY-MM-DD shape,
digest.greeting/lead/numbers.{clusters,multiSource,surfaced}, threads
array + per-element shape, signals array of strings, per-story
required fields + threatLevel enum). Throws with field-path message
on first miss. Adds nine negative tests covering common omissions.
- todo 208: envelope carries version guard. assertBriefEnvelope throws
when envelope.version !== BRIEF_ENVELOPE_VERSION with a message
naming both observed and expected versions.
P2 (should-fix, now included):
- todo 209: drop the _seed wrapper and make BriefEnvelope a flat
{ version, issuedAt, data } shape. A per-user brief is not a global
seed; reusing _seed invited mis-application of seed invariants
(SEED_META pairing, TTL rotation). Locked down before Phase 3
composer bakes the shape in.
- todo 210: move renderer from shared/render-brief-magazine.{js,d.ts}
to server/_shared/brief-render.{js,d.ts}. The 740-line template
doesn't belong on the shared/ mirror hot path (Vercel+Railway). Keep
only the envelope contract in shared/. Import path updated in tests.
- todo 211: logo SVG now defined once per document as a <symbol> and
referenced via <use> at each placement (~8 refs/doc). Drops ~7KB
from rendered output (~26% total size reduction on small inputs).
Tests pass (26/26), typecheck clean, lint clean, mirror check (169/169)
unaffected.
* fix(brief): enforce closed-key contract + surfaced-count invariant
Addresses two additional P1 review findings on PR #3150.
1. Validator rejects extra keys at every level (envelope root, data,
user, digest, numbers, each thread, each story). Previously the
forbidden-field rule was documented in the .d.ts and proven only at
render time via a sentinel test — a producer could still persist
importanceScore, primaryLink, pubDate, briefModel, _seed, fetchedAt,
etc. into the Redis-resident envelope and every downstream consumer
(edge route, dashboard panel preview, email teaser, carousel) would
accept them. The validator is the only place the invariant can live.
2. Validator now asserts digest.numbers.surfaced === stories.length.
The renderer uses both values — surfaced on the at-a-glance page,
stories.length for the cover blurb and page count — so allowing
them to disagree produced a self-contradictory brief that the old
validator silently passed.
Tests: 30/30 (was 26). New negative tests cover the strict-keys
rejection at root / data / numbers / story[i] levels and the surfaced
mismatch. The earlier sentinel-poisoning test is superseded — the
strict-keys check catches the same class of bug earlier and harder.
|
||
|
Elie Habib
|
7aa8dd1bf2 |
fix(scoring): relay recomputes importanceScore post-LLM + shadow-log v2 + parity test (#3069)
* fix(scoring): relay recomputes importanceScore post-LLM + shadow-log v2 + parity test
Before this change, classified rss_alert events published by ais-relay carried
a stale importanceScore: the digest computed it from the keyword-level threat
before the LLM upgrade, and the relay republished that value unchanged. Shadow
log (2,850 entries / 7 days) had Pearson 0.31 vs human rating with zero events
reaching the ≥85 critical gate — the score being measured was the keyword
fallback, not the AI classification.
Fixes:
- ais-relay.cjs: recompute importanceScore locally from the post-LLM level
using an exact mirror of the digest formula (SEVERITY_SCORES, SCORE_WEIGHTS,
SOURCE_TIERS, formula). Publish includes corroborationCount for downstream
shadow-log enrichment.
- notification-relay.cjs: delete the duplicate shadowLogScore() call that
produced ~50% near-duplicate pairs. Move shadow log to v2 key with
JSON-encoded members carrying severity, source, corroborationCount,
variant — future calibration cycles get cleaner data.
- shadow-score-{report,rank}.mjs: parse both v2 JSON and legacy v1 string
members; default to v2, override via SHADOW_SCORE_KEY env.
- _classifier.ts: narrow keyword additions — blockade, siege, sanction
(singular), escalation → HIGH; evacuation orders (plural) → CRITICAL.
- tests/importance-score-parity.test.mjs: extracts tier map and formula from
both TS digest and CJS relay sources, asserts identical output across 12
sample cases. Catches any future drift.
- tests/relay-importance-recompute.test.mjs + notification-relay-shadow-log
.test.mjs: regression tests for the publish path and single-write discipline.
Gates remain OFF. After deploy, collect 48h of fresh shadow:score-log:v2
data, re-run scripts/shadow-score-rank.mjs for calibration, then set final
IMPORTANCE_SCORE_MIN / high / critical thresholds before enabling
IMPORTANCE_SCORE_LIVE=1.
See docs/internal/scoringDiagnostic.md (local) for full diagnosis.
🤖 Generated with Claude Sonnet 4.6 via Claude Code + Compound Engineering v2.49.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(scoring): PR #3069 review amendments — revert risky keywords + extract SOURCE_TIERS
Addresses review findings on PR #3069 (todos/193 through 204).
BLOCKING (P1):
- Revert 5 keyword additions in _classifier.ts. Review showed `escalation`,
`sanction`, `siege`, `blockade`, `evacuation orders` fire HIGH on
`de-escalation`, `sanctioned`, `besieged`, `blockaded` (substring matches),
and the plural `evacuation orders` is already covered by the singular.
Classifier work will land in a separate PR with phrase-based rules.
- Delete dead `digestImportanceScore` field from relay allTitles metadata
(written in two places, read nowhere).
IMPORTANT (P2):
- Extract SOURCE_TIERS to shared/source-tiers.{json,cjs} using the
existing shared/rss-allowed-domains.* precedent. Dockerfile.relay
already `COPY shared/` (whole dir), so no infra change. Deletes
255-line inline duplicate from ais-relay.cjs; TS digest imports the
same JSON via resolveJsonModule. Tier-map parity is now structural.
- Simplify parity test — tier extraction no longer needed. SEVERITY_SCORES
+ SCORE_WEIGHTS + scoring function parity retained across 12 cases
plus an unknown-level defensiveness case. Deleted no-op regex replace
(`.replace(/X/g, 'X')`). Fixed misleading recency docstring.
- Pipeline shadow log: ZADD + ZREMRANGEBYSCORE + belt+suspenders EXPIRE
now go in a single Upstash /pipeline POST (~50% RT reduction, no
billing delta).
- Bounded ZRANGE in shadow-score-report.mjs (20k cap + warn if reached).
- Bump outbound webhook envelope v1→v2 to signal the new
`corroborationCount` field on rss_alert payloads.
- Restore rss_alert eventType gate at shadowLogScore caller (skip
promise cost for non-rss events).
- Align ais-relay scorer comment with reality: it has ONE intentional
deviation from digest (`?? 0` on severity for defensiveness, returning
0 vs NaN on unknown levels). Documented + tested.
P3:
- Narrow loadEnv in scripts to only UPSTASH_REDIS_REST_* (was setting
any UPPER_UNDERSCORE env var from .env.local).
- Escape markdown specials in rating-sheet.md title embeds.
Pre-existing activation blockers NOT fixed here (tracked as todos 196,
203): `/api/notify` accepts arbitrary importanceScore from authenticated
Pro users, and notification channel bodies don't escape mrkdwn/Discord
markup. Both must close before `IMPORTANCE_SCORE_LIVE=1`.
Net: -614 lines (more deleted than added). 26 regression assertions pass.
npm run typecheck, typecheck:api, test:data all pass.
🤖 Generated with Claude Sonnet 4.6 via Claude Code + Compound Engineering v2.49.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(scoring): mirror source-tiers.{json,cjs} into scripts/shared/
The `scripts-shared-mirror` enforcement in tests/edge-functions.test.mjs
requires every *.json and *.cjs in shared/ to have a byte-identical copy
in scripts/shared/ (Railway rootDirectory=scripts deploy bundle cannot
reach repo-root shared/). Last commit added shared/source-tiers.{json,cjs}
without mirroring them. CI caught it.
* fix(scoring): revert webhook envelope to v1 + log shadow-log pipeline failures
Two P1/P2 review findings on PR #3069:
P1: Bumping webhook envelope to version: '2' was a unilateral breaking
change — the other webhook producers (proactive-intelligence.mjs:407,
seed-digest-notifications.mjs:736) still emit v1, so the same endpoint
would receive mixed envelope versions per event type. A consumer
validating `version === '1'` would break specifically on realtime
rss_alert deliveries while proactive_brief and digest events kept
working. Revert to '1' and document why — `corroborationCount` is an
additive payload field, backwards-compatible for typical consumers;
strict consumers using `additionalProperties: false` should be handled
via a coordinated version bump across all producers in a separate PR.
P2: The new shadow-log /pipeline write swallowed all errors silently
(no resp.ok check, no per-command error inspection), a regression from
the old upstashRest() path which at least logged non-2xx. Since the
48h recalibration cycle depends on shadow:score-log:v2 filling with
clean data, a bad auth token or malformed pipeline body would leave
operators staring at an empty ZSET with no signal. Now logs HTTP
failures and per-command pipeline errors.
* docs(scoring): fix stale v1 references + clarify two-copy source-tiers mirror
Two follow-up review findings on PR #3069:
P2 — Source-tier "single source of truth" comments were outdated.
PR #3069 ships TWO JSON copies (shared/source-tiers.json for Vercel edge +
main relay container, scripts/shared/source-tiers.json for Railway services
using rootDirectory=scripts). Comments in server/_shared/source-tiers.ts
and scripts/ais-relay.cjs now explicitly document the mirror setup and
point at the two tests that enforce byte-identity: the existing
scripts-shared-mirror test (tests/edge-functions.test.mjs:37-48) and a
new explicit cross-check in tests/importance-score-parity.test.mjs.
Adding the assertion here is belt-and-suspenders: if edge-functions.test.mjs
is ever narrowed, the parity test still catches drift.
P3 — Stale v1 references in shared metadata/docs. The actual writer
moved to shadow:score-log:v2 in notification-relay.cjs, but
server/_shared/cache-keys.ts:23-31 still documented v1 and exported the
v1 string. No runtime impact (the export has zero importers — relay
uses its own local const) but misleading. Updated the doc block to
explain both v1 (legacy, self-pruning) and v2 (current), bumped the
constant to v2, and added a comment that notification-relay.cjs is the
owner. Header comment in scripts/shadow-score-report.mjs now documents
the SHADOW_SCORE_KEY override path too.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Elie Habib
|
08a5eb77a5 |
fix(intelligence): region-scope signals and chokepoint evidence (#2952)
* fix(intelligence): region-scope signals and chokepoint evidence Two review findings on PR #2940 caused MENA/SSA snapshots to silently drop broad cross-source signals and leak foreign chokepoint events into every region's evidence chain. P1 - theater label matching seed-cross-source-signals.mjs normalizes raw values to broad display labels like "Middle East" and "Sub-Saharan Africa". The consumer side compared these against region.theaters kebab IDs (levant, persian-gulf, horn-of-africa). "middle east" does not substring-match any of those, so every MENA signal emitted with the broad label was silently dropped from coercive_pressure and the evidence chain. Same for SSA. Added signalAliases per region and a shared isSignalInRegion helper in shared/geography.js. Both balance-vector.mjs and evidence-collector.mjs now route signals through the helper, which normalizes both sides and matches against theater IDs or region aliases. P2 - chokepoint region leak evidence-collector.mjs:62 iterated every chokepoint in the payload without filtering by regionId, so Taiwan Strait, Baltic, and Panama threat events surfaced in MENA and SSA evidence chains. Now derives the allowed chokepoint ID set from getRegionCorridors(regionId) and skips anything not owned by the region. Added 15 unit tests covering: broad-label matching, kebab/spaced input, cross-region rejection, and the chokepoint filter for MENA/East Asia/ Europe/SSA. * fix(intelligence): address Greptile P2 review findings on #2952 Two P2 findings from Greptile on the region-scoping PR. 1) Drop 'eu' short alias from europe.signalAliases `isSignalInRegion` uses substring matching, and bare 'eu' would match any theater label containing those two letters ('fuel', 'neutral zone', 'feudal'). Replaced with 'european union' which is long enough to be unambiguous. No seed currently emits a bare 'eu' label, so this is pure hardening. 2) Make THEATERS.corridorIds live data via getRegionCorridors union horn-of-africa declared corridorIds: ['babelm'] and caribbean declared corridorIds: ['panama'], but `getRegionCorridors` only consulted CORRIDORS.theaterId — so those entries were dead data. After yesterday's region-scoped chokepoint filter, Bab el-Mandeb threat events landed ONLY in MENA evidence (via the primary red-sea theater), never in SSA, even though the corridor physically borders Djibouti/Eritrea. Same for Panama missing from LatAm. `getRegionCorridors` now unions direct theater membership (via CORRIDORS.theaterId) with indirect claims (via THEATERS.corridorIds), de-duplicated by corridor id. This reflects geopolitical reality: - MENA + SSA both see babelm threat events - NA + LatAm both see panama threat events Scoring impact: SSA maritime_access now weighs babelm (weight 0.9), LatAm maritime_access now weighs panama (weight 0.6). These were missing buffers under the pre-fix model. Added regression tests for both new paths. The existing "SSA evidence has no chokepoints" test was inverted to assert SSA now DOES include babelm (and excludes MENA/East Asia corridors). |
||
|
Elie Habib
|
2dd28e186d | Phase 0 PR2: Forecast region filter quick win (#2942) | ||
|
Elie Habib
|
7dfdc819a9 | Phase 0: Regional Intelligence snapshot writer foundation (#2940) | ||
|
Elie Habib
|
b5d83241b4 |
fix(trade): expand tariff + trade coverage to 130+ countries (#2913)
* fix(trade): expand tariff coverage from 15 to 130+ countries MAJOR_REPORTERS (15 countries) replaced with ALL_REPORTERS (130+ WTO members). Tariffs, trade restrictions, and barriers now seeded for every WTO member economy, not just the original 15. WTO API calls batched in groups of 30 to avoid URL length limits while keeping total request count under 5. Added shared/un-to-iso2.json (239 entries) for UN M49 to ISO2 mapping used by the cache key builder. Removed hardcoded WTO_MEMBER_CODES lookup; uses WTO API response field ReportingEconomy for country names instead. * fix(trade): ESM-compatible file loading + fix row scope in buildFlowRecords 1. Replace require()/CJS __dirname with ESM imports (readFileSync, dirname, fileURLToPath). The .mjs module can't use require(), so un-to-iso2.json was never loaded and REPORTER_ISO2 fell back to the 20-country hardcoded subset. 2. Fix ReferenceError in buildFlowRecords: row.ReportingEconomy was accessed outside the for-of loop scope. Extract reporter/partner names during row iteration and use them in the map callback. 3. Add reporterName/partnerName to parseFlowRows output so buildFlowRecords has access to WTO economy names. * fix(trade): load ALL 239 reporters from un-to-iso2.json, remove caps 1. ALL_REPORTERS now derived from Object.keys(un-to-iso2.json) at startup: 239 economies, no manual list to maintain. Covers NZ, Jamaica, Dominican Republic, Panama, and every other country. 2. Removed .slice(0, 50) caps on trade barriers and restrictions so all reporter data is published. 3. Cleaned up duplicate ESM imports; single __dirname derivation. * fix(trade): batch WTO barriers/restrictions in groups of 30 The WTO API returns HTTP 400 when all 239 reporter codes are passed in a single r= parameter. Barriers and restrictions now batch in groups of 30 with 1s delay between batches, matching the tariff trends pattern. Also removed stale duplicate variable declarations. * fix: add un-to-iso2.json to scripts/shared (sync test) * fix(trade): fetch WTO reporter list from API instead of UN codes The WTO API has its own reporter code system (288 economies) that doesn't match UN M49 1:1. Sending un-to-iso2.json codes caused HTTP 400 on 7/8 batches. Now fetches /v1/reporters at startup to get the exact set of codes the API accepts. Falls back to un-to-iso2.json if the reporter endpoint fails. * fix(trade): compute REPORTER_ISO2 lazily after fetchWtoReporters() REPORTER_ISO2 was computed at module load when ALL_REPORTERS was still []. Changed to getReporterIso2() function that evaluates after ALL_REPORTERS is populated by fetchWtoReporters(). |
||
|
Elie Habib
|
4053a5119a |
feat(energy): surface coal and TTF gas spot prices in analyst context (#2715)
* feat(intelligence): surface coal and TTF gas spot prices in analyst context - Add TTF=F (TTF Natural Gas) to scripts/shared/commodities.json after NG=F - Add coalSpotPrice and gasSpotTtf fields to AnalystContext interface - Add extractCommodityQuote and buildSpotCommodityLine helpers in chat-analyst-context.ts - Derive coal (MTF=F) and TTF (TTF=F) from already-fetched commodities bootstrap, no extra Redis calls - Gate on SPOT_ENERGY_DOMAINS (economic, geo, all) - Register both fields in SOURCE_LABELS with CoalSpot/GasTTF labels - Add coalSpotPrice and gasSpotTtf to DOMAIN_SECTIONS for geo and economic in chat-analyst-prompt.ts - Push Coal Spot Price and TTF Gas Price sections into context when present * fix(energy): use correct unit for Newcastle coal (/t not /MWh) buildSpotCommodityLine hardcoded /MWh for all commodities. Coal is priced per metric ton. Added denominator parameter with /MWh default, pass /t for Newcastle coal. |
||
|
Elie Habib
|
02555671f2 |
refactor: consolidate country name/code mappings into single canonical sources (#2676)
* refactor(country-maps): consolidate country name/ISO maps Expand shared/country-names.json from 265 to 309 entries by merging geojson names, COUNTRY_ALIAS_MAP, upstream API variants (World Bank, WHO, UN, FAO), and seed-correlation extras. Add ISO3 map generator (generate-iso3-maps.cjs) producing iso3-to-iso2.json (239 entries) and iso2-to-iso3.json (239 entries) with TWN and XKX supplements. Add build-country-names.cjs for reproducible expansion from all sources. Sync scripts/shared/ copies for edge-function test compatibility. * refactor: consolidate country name/code mappings into single canonical sources Eliminates fragmented country mapping across the repo. Every feature (resilience, conflict, correlation, intelligence) was maintaining its own partial alias map. Data consolidation: - Expand shared/country-names.json from 265 to 302 entries covering World Bank, WHO, UN, FAO, and correlation script naming variants - Generate shared/iso3-to-iso2.json (239 entries) and shared/iso2-to-iso3.json from countries.geojson + supplements (Taiwan TWN, Kosovo XKX) Consumer migrations: - _country-resolver.mjs: delete COUNTRY_ALIAS_MAP (37 entries), replace 2MB geojson parse with 5KB iso3-to-iso2.json - conflict/_shared.ts: replace 33-entry ISO2_TO_ISO3 literal - seed-conflict-intel.mjs: replace 20-entry ISO2_TO_ISO3 literal - _dimension-scorers.ts: replace geojson-based ISO3 construction - get-risk-scores.ts: replace 31-entry ISO3_TO_ISO2 literal - seed-correlation.mjs: replace 102-entry COUNTRY_NAME_TO_ISO2 and 90-entry ISO3_TO_ISO2, use resolveIso2() from canonical resolver, lower short-alias threshold to 2 chars with word boundary matching, export matchCountryNamesInText(), add isMain guard Tests: - New tests/country-resolver.test.mjs with structural validation, parity regression for all 37 old aliases, ISO3 bidirectional consistency, and Taiwan/Kosovo assertions - Updated resilience seed test for new resolver signature Net: -190 lines, 0 hardcoded country maps remaining * fix: normalize raw text before country name matching Text matchers (geo-extract, seed-security-advisories, seed-correlation) were matching normalized keys against raw text containing diacritics and punctuation. "Curaçao", "Timor-Leste", "Hong Kong S.A.R." all failed to resolve after country-names.json keys were normalized. Fix: apply NFKD + diacritic stripping + punctuation normalization to input text before matching, same transform used on the keys. Also add "hong kong" and "sao tome" as short-form keys for bigram headline matching in geo-extract. * fix: remove 'u s' alias that caused US/VI misattribution 'u s' in country-names.json matched before 'u s virgin islands' in geo-extract's bigram scanner, attributing Virgin Islands headlines to US. Removed since 'usa', 'united states', and the uppercase US expansion already cover the United States. |
||
|
Elie Habib
|
ba54dc12d7 |
feat(commodity): gold layer enhancements (#2464)
* feat(commodity): add gold layer enhancements from fork review Enrich the commodity variant with learnings from Yazan-Abuawwad/gold-monitor fork: - Add 10 missing gold mines to MINING_SITES: Muruntau (world's largest open-pit gold mine), Kibali (DRC), Sukhoi Log (Russia, development), Ahafo (Ghana), Loulo-Gounkoto (Mali), South Deep (SA), Kumtor (Kyrgyzstan), Yanacocha (Peru), Cerro Negro (Argentina), Tropicana (Australia). Covers ~40% of top-20 global mines previously absent. - Add XAUUSD=X spot gold and 9 FX pairs (EUR, GBP, JPY, CNY, INR, AUD, CHF, CAD, TRY) to shared/commodities.json. All =X symbols auto-seeded via existing seedCommodityQuotes() — no new seeder needed. Registered in YAHOO_ONLY_SYMBOLS in both _shared.ts and ais-relay.cjs. - Add XAU/FX tab to CommoditiesPanel showing gold priced in 10 currencies. Computed live from GC=F * FX rates. Commodity variant only. - Fix InsightsPanel brief title: commodity variant now shows "⛏️ COMMODITY BRIEF" instead of "🌍 WORLD BRIEF". - Route commodity variant daily market brief to commodity feed categories (commodity-news, gold-silver, mining-news, energy, critical-minerals) via new newsCategories option on BuildDailyMarketBriefOptions. - Add Gold Silver Worlds + FX Empire Gold direct RSS feeds to gold-silver panel (9 sources total, up from 7). * fix(commodity): address review findings from PR #2464 - Fix USDCHF=X multiply direction: was true (wrong), now false (USD/CHF is USD-per-CHF convention) - Fix newsCategories augments BRIEF_NEWS_CATEGORIES instead of replacing (preserves macro/Fed context in commodity brief) - Add goldsilverworlds.com + www.fxempire.com to RSS allowlist (api + shared + scripts/shared) - Rename "Metals" tab label conditionally: commodity variant gets "Metals", others keep "Commodities" - Reset _tab to "commodities" when hasXau becomes false (prevent stale XAU tab re-activation) - Add Number.isFinite() guard in _renderXau() before computing xauPrice - Narrow fxMap filter to =X symbols only - Collapse redundant two-branch number formatter to Math.round().toLocaleString() - Remove XAUUSD=X from shared/commodities.json: seeded but never displayed (saves 150ms/cycle) * feat(mcp): add get_commodity_geo tool and update get_market_data description * fix(commodity): correct USDCHF direction, replace headline categories, restore dep overrides * fix(commodity): empty XAU grid fallback and restore FRED timeout to 20s * fix(commodity): remove XAU/USD from MCP description, revert Metals tab label * fix(commodity): remove dead XAUUSD=X from YAHOO_ONLY_SYMBOLS XAU widget uses GC=F as base price, not XAUUSD=X. Symbol was never seeded (not in commodities.json) and never referenced in the UI. |
||
|
Elie Habib
|
380b495be8 |
feat(mcp): live airspace + maritime tools; fix OAuth consent UI (#2442)
* fix(oauth): fix CSS arrow bullets + add MCP branding to consent page
- CSS content:'\2192' (not HTML entity which doesn't work in CSS)
- Rename logo/title to "WorldMonitor MCP" on both consent and error pages
- Inject real news headlines into get_country_brief to prevent hallucination
Fetches list-feed-digest (4s budget), passes top-15 headlines as ?context=
to get-country-intel-brief; brief timeout reduced to 24s to stay under Edge ceiling
* feat(mcp): add get_airspace + get_maritime_activity live query tools
New tools answer real-time positional questions via existing bbox RPCs:
- get_airspace: civilian ADS-B (OpenSky) + military flights over any country
parallel-fetches track-aircraft + list-military-flights, capped at 100 each
- get_maritime_activity: AIS density zones + disruptions for a country's waters
calls get-vessel-snapshot with country bbox
Country → bounding box resolved via shared/country-bboxes.json (167 entries,
generated from public/data/countries.geojson by scripts/generate-country-bboxes.cjs).
Both API calls use 8s AbortSignal.timeout; get_airspace uses Promise.allSettled
so one failure doesn't block the other.
* docs: fix markdown lint in airspace/maritime plan (blank lines around lists)
* fix(oauth): use literal → in CSS content (\2192 is invalid JS octal in ESM)
* fix(hooks): extend bundle check to api/oauth/ subdirectory (was api/*.js, now uses find)
* fix(mcp): address P1 review findings from PR 2442
- JSON import: add 'with { type: json }' so node --test works without tsx loader
- get_airspace: surface upstream failures; partial outage => partial:true+warnings,
total outage => throw (prevents misleading zero-aircraft response)
- pre-push hook: add #!/usr/bin/env bash shebang (was no shebang, ran as /bin/sh
on Linux CI/contributors; process substitution + [[ ]] require bash)
* fix(mcp): replace JSON import attribute with TS module for Vercel compat
Vercel's esbuild bundler does not support `with { type: 'json' }` import
attributes, causing builds to fail with "Expected ';' but found 'with'".
Fix: generate shared/country-bboxes.ts (typed TS module) alongside the
existing JSON file. The TS import has no attributes and bundles cleanly
with all esbuild versions.
Also extend the pre-push bundle check to include api/*.ts root-level files
so this class of error is caught locally before push.
* fix(mcp): reduce get_country_brief timing budget to 24 s (6 s Edge margin)
Digest pre-fetch: 4 s → 2 s (cached endpoint, silent fallback on miss)
Brief call: 24 s → 22 s
Total worst-case: 24 s vs Vercel Edge 30 s hard kill — was 28 s (2 s margin)
* test(mcp): add coverage for get_airspace and get_maritime_activity
9 new tests:
- get_airspace: happy path, unknown code, partial failure (mil down),
total failure (-32603), type=civilian skips military fetch
- get_maritime_activity: happy path, unknown code, API failure (-32603),
empty snapshot handled gracefully
Also fixes import to use .ts extension so Node --test resolver finds the
country-bboxes module (tsx resolves .ts directly; .js alias only works
under moduleResolution:bundler at typecheck time)
* fix(mcp): use .js + .d.ts for country-bboxes — Vercel rejects .ts imports
Vercel edge bundler refuses .ts extension imports even from .ts edge
functions. Plain .js is the only safe runtime import for edge functions.
Pattern: generate shared/country-bboxes.js (pure ESM, no TS syntax) +
shared/country-bboxes.d.ts (type declaration). TypeScript uses the .d.ts
for tuple types at check time; Vercel and Node --test load the .js at
runtime. The previous .ts module is removed.
* test(mcp): update tool count to 26 (main added search_flights + search_flight_prices_by_date)
|
||
|
Elie Habib
|
c431397599 |
fix(grocery-basket): fix JP sites, add TRY/EGP/INR floors, evict bad routes (#2324)
The bilateral outlier gate (previous commit) catches bad prices AFTER the
fact. This commit prevents them from being accepted in the first place and
forces re-scraping on the very next seed run.
Japan sites (grocery-basket.json):
Before: kakaku.com, price.com — price comparison aggregators that show
per-gram/per-serving prices in LLM summaries, not per-kg shelf prices.
price.com is not even Japan-specific.
After: seiyu.co.jp, life.co.jp, aeon-net.com — actual Japanese supermarket
chains with clear per-unit JPY shelf prices.
CURRENCY_MIN additions (seed-grocery-basket.mjs):
TRY: 10 — Turkish shelf prices are always ≥ 10 TRY; the bad values
(2.75–5.61 TRY) were per-100g sub-unit matches.
EGP: 5 — Egyptian shelf prices are ≥ 5 EGP; 2.95/3.25 EGP were fractional.
INR: 12 — Indian shelf prices are ≥ 12 INR; 10 INR potatoes was stale.
One-time eviction (_migration:bad-prices-v1):
All JP routes — stale since sites changed; forces fresh searches on new domains.
TR: sugar, eggs, milk, oil — confirmed sub-unit price scrapes.
EG: salt, bread, milk — confirmed too-cheap scrapes.
IN: potatoes, milk — confirmed too-cheap scrapes.
Next seed run re-fetches all evicted items from EXA/Firecrawl from scratch.
|
||
|
Elie Habib
|
3085e154b1 |
fix(grocery-index): canola oil, tighter caps, outlier gate, sticky col, wow badge (#2234)
* fix(grocery-index): switch oil to canola, tighten caps, fix scroll + wow badge - Change oil item query from "sunflower cooking oil" to "canola oil 1L" and lower ITEM_USD_MAX.oil from 15 to 10 (canola tops ~$7 globally) - Evict all *:oil learned routes on seed startup since product changed - Tighten ITEM_USD_MAX caps: sugar 8->3.5, pasta 4->3.5, milk 8->5, flour 8->4.5, bread 8->6, salt 5->2.5 to prevent organic/bulk mismatches - Add 4x median cross-country outlier gate with Redis learned route eviction (catches France sugar $6.94 which was 4.75x median $1.46) - Strengthen validateFn: reject seed if <5 countries have >=40% coverage - Fix sticky first column in gb-scroll so item names don't scroll under prices - Add missing .gb-wow CSS rule so WoW badges render in total row * fix(grocery-index): one-time oil migration guard, WoW version gate, main.css dedupe P1: Gate oil route eviction behind a Redis migration sentinel (_migration:canola-oil-v1) so it only fires once and learned canola routes persist across subsequent weekly runs. P1: Add BASKET_VERSION=2 constant; suppress WoW when prevSnapshot.basketVersion differs to prevent bogus deltas on the first canola seed run comparing against a sunflower baseline. P2: Update main.css gb-item-col, gb-item-name, and gb-wow rules to match panels.css intent. The more-specific .gb-table selectors and later cascade position caused main.css to override sticky positioning, min-width: 110px, and gb-wow sizing back to old values. |
||
|
Elie Habib
|
663a58bf80 |
fix(market): route sectors/commodities to correct RPC endpoints (#2198)
* fix(fear-greed): add undici to scripts/package.json (ERR_MODULE_NOT_FOUND on Railway) * fix(market): route sectors/commodities to correct RPC endpoints fetchMultipleStocks called listMarketQuotes which reads market:stocks-bootstrap:v1. Sector ETFs (XLK, XLF...) and commodity futures (GC=F, CL=F...) are NOT in that key, so the live-fetch fallback always returned empty after the one-shot bootstrap hydration was consumed, causing panels to show "data temporarily unavailable" on every reload. Fix: add fetchSectors() -> getSectorSummary (reads market:sectors:v1) and fetchCommodityQuotes() -> listCommodityQuotes (reads market:commodities-bootstrap:v1), each with their own circuit breaker and persistent cache. Remove useCommodityBreaker option from fetchMultipleStocks which no longer serves commodities. * feat(heatmap): show friendly sector names instead of ETF tickers The relay seeds name:ticker into Redis (market:sectors:v1), so the heatmap showed XLK/XLF/etc which is non-intuitive for most users. Fix: build a sectorNameMap from shared/sectors.json (keyed by symbol) and apply it in both the hydrated and live fetch paths. Also update sectors.json names from ultra-short aliases (Tech, Finance) to clearer labels (Technology, Financials, Health Care, etc). Closes #2194 * sync scripts/shared/sectors.json * feat(heatmap): show ticker + sector name side by side Each tile now shows: XLK <- dim ticker (for professionals) Technology <- full sector name (for laymen) +1.23% Sector names updated: Tech→Technology, Finance→Financials, Health→Health Care, Real Est→Real Estate, Comms→Comm. Svcs, etc. Refs #2194 |
||
|
Elie Habib
|
1d28c352da |
feat(commodities): expand tracking to 23 symbols — agriculture and coal (#2135)
* feat(commodities): expand tracking to cover agricultural and coal futures Adds 9 new commodity symbols to cover the price rally visible in our intelligence feeds: Newcastle Coal (MTF=F), Wheat (ZW=F), Corn (ZC=F), Soybeans (ZS=F), Rough Rice (ZR=F), Coffee (KC=F), Sugar (SB=F), Cocoa (CC=F), and Cotton (CT=F). Also fixes ais-relay seeder to use display names from commodities.json instead of raw symbols, so seeded data is self-consistent. * fix(commodities): gold standard cache, 3-col grid, cleanup - Add upstashExpire on zero-quotes failure path so bootstrap key TTL extends during Yahoo outages (gold standard pattern) - Remove unreachable fallback in retry loop (COMMODITY_META always has the symbol since it mirrors COMMODITY_SYMBOLS) - Switch commodities panel to 3-column grid (19 items → ~7 rows vs 10) |
||
|
Elie Habib
|
549084fbca |
fix(relay): add missing USNI regions and remove dead NZ safetravel feed (#1969)
* fix(relay): add missing USNI regions and remove dead NZ safetravel feed USNI fleet tracker warns on unknown regions "Tasman Sea" and "Eastern Atlantic" — add coords to USNI_REGION_COORDS map. safetravel.govt.nz/news/feed redirects to /404 on every request; remove from advisory feeds and RSS allowed domains. * fix: remove safetravel.govt.nz from edge function allowed domains copy * fix: remove safetravel.govt.nz from shared allowed domains copy |
||
|
Elie Habib
|
a8f8c0aa61 |
feat(economic): Middle East grocery basket price index (#1904)
* feat(economic): add ME grocery basket price index Adds a grocery basket price comparison panel for 9 Middle East countries (UAE, KSA, Qatar, Kuwait, Bahrain, Oman, Egypt, Jordan, Lebanon) using EXA AI to discover prices from regional e-commerce sites (Carrefour, Lulu, Noon, Amazon) and Yahoo Finance for FX rates. - proto: ListGroceryBasketPrices RPC with CountryBasket/GroceryItemPrice messages - seed: seed-grocery-basket.mjs, 90 EXA calls/run, 150ms delay, hardcoded FX fallbacks for pegged GCC currencies, 6h TTL - handler: seed-only RPC reading economic:grocery-basket:v1 - gateway: static cache tier for the new route - bootstrap/health: groceryBasket key in SLOW tier, 720min stale threshold - frontend: GroceryBasketPanel with scrollable table, cheapest/priciest column highlighting, styles moved to panels.css - panel disabled by default until seed is run on Railway * fix(generated): restore @ts-nocheck in economic service codegen * fix(grocery-basket): tighten seed health staleness and seed script robustness - Set maxStaleMin to 360 (6h) matching CACHE_TTL so health alerts on first missed run - Use ?? over || for FX fallback to handle 0-value rates correctly - Add labeled regex patterns with bare-number warning in extractPrice - Replace conditional delay logic with unconditional per-item sleep * fix(grocery-basket): fix EXA API format and price extraction after live validation - Use contents.summary format (not top-level summary) — previous format returned no data - Support EXA_API_KEYS (comma-separated) in addition to EXA_API_KEY - Extract price from plain-text summary string (EXA returns text, not JSON schema) - Remove bare-number fallback — too noisy (matched "500" from "pasta 500g" as SAR 500) - Fix LBP FX rate zero-guard: use fallback when Yahoo returns 0 for ultra-low-value currencies Validated locally: 9 countries seeded, Redis write confirmed, ~111s runtime * fix(grocery-basket): validate extracted currency against expected country currency - matchPrice now returns the currency code alongside the price - extractPrice rejects results where currency != expected country currency (prevents AED prices from being treated as JOD prices on gcc.luluhypermarket.com) - Tighten item queries (white granulated sugar, spaghetti pasta, etc.) to reduce irrelevant product matches like Stevia on sugar queries - Replace Jordan's gcc.luluhypermarket.com (GCC-only) with carrefour.jo + ounasdelivery.com - Sync scripts/shared/grocery-basket.json * feat(bigmac): add Big Mac Index seed + drop grocery basket includeDomains Grocery basket: - Remove includeDomains restriction — EXA neural search finds better sources than hardcoded domain lists; currency validation prevents contamination - Tighten query strings (supermarket retail price suffix) Big Mac seed (scripts/seed-bigmac.mjs): - Two-tier search: specialist sites (theburgerindex.com, eatmyindex.com) first, fall back to open search for countries without per-country indexed pages - Handle thousands-separator prices (480,000 LBP) - Accept USD prices from cost-of-living index sites as fallback - Exclude ranking/average pages (Numbeo country_price_rankings, Expatistan) - Validated live: 7/9 countries with confirmed prices UAE=19AED, KSA=19SAR, QAR=22QAR, KWD=1.4KWD, EGP=135EGP, JOD=3JOD, LBP=480kLBP * feat(economic): expand grocery basket to 24 global countries, drop Big Mac tier-2 search Grocery basket: extend coverage from 9 MENA to 24 countries across all regions (US, UK, DE, FR, JP, CN, IN, AU, CA, BR, MX, ZA, TR, NG, KR, SG, PK, AE, SA, EG, KE, AR, ID, PH). Add FX fallbacks and fxSymbols for all 23 new currencies. CCY regex in seed script updated to match all supported currency codes. Big Mac: remove tier-2 open search (too noisy, non-specialist pages report combo prices or global averages). Specialist sites only (theburgerindex.com, eatmyindex.com) for clean per-country data. * feat(bigmac): wire Big Mac Index RPC, proto, bootstrap, health Add ListBigMacPrices RPC end-to-end: - proto/list_bigmac_prices.proto: BigMacCountryPrice + request/response - service.proto: register ListBigMacPrices endpoint (GET /list-bigmac-prices) - buf generate: regenerate service_server.ts + all client stubs - server/list-bigmac-prices.ts: seed-only handler reads economic:bigmac:v1 - handler.ts: wire listBigMacPrices into EconomicServiceHandler - api/bootstrap.js: bigmac key in BOOTSTRAP_CACHE_KEYS + SLOW_KEYS - api/health.js: bigmac key in BOOTSTRAP_KEYS + SEED_META (maxStaleMin: 1440) - _bootstrap-cache-key-refs.ts: groceryBasket + bigmac refs * feat(bigmac): add BigMacPanel + register in panel layout BigMacPanel renders a country-by-country Big Mac price table sorted by USD price (cheapest/most expensive highlighted). Wired into bootstrap hydration, refresh scheduler, and panel registry. Registered in panels.ts (enabled: false, to be flipped once seed data is verified). Also updates grocery basket i18n from ME-specific to global wording. * fix(bigmac): register bigmac in cache-keys and RPC_CACHE_TIER Add bigmac to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in server/_shared/cache-keys.ts, and to RPC_CACHE_TIER (static tier) in gateway.ts. Both were caught by bootstrap and RPC tier parity tests. * fix(generated): restore @ts-nocheck in all generated service files after buf regenerate buf generate does not emit @ts-nocheck. Previous convention restores it manually post-generate to suppress strict type errors in generated code. * fix(grocery-basket): restore includeDomains per country, add userLocation, fix currency symbol parsing Root cause of 0-item countries (UK, JP, IN, NG): removing includeDomains caused EXA neural search to return USD-priced global comparison pages (Numbeo, Tridge, Expatistan) which currency validation correctly rejected. Fixes: - Add per-country sites[] in grocery-basket.json (researched local supermarket/retailer domains: tesco.com, kaufland.de, bigbasket.com, etc.) - Pass includeDomains: country.sites to restrict EXA to local retailers - Pass userLocation: country.code (ISO) to bias results to target country - Add currency symbol fallback regex (£→GBP, €→EUR, ¥→JPY, ₹→INR, ₩→KRW, ₦→NGN, R$→BRL) — sites like BigBasket use ₹ not INR - Summary query now explicitly requests ISO currency code - Simplify item queries (drop country name — context from domains) Smoke test results: UK sugar → GBP 1.09 (tesco.com) ✓ IN rice → ₹66 (bigbasket.com) ✓ JP rice → JPY 500 (kakaku.com) ✓ * fix(grocery-basket): add Firecrawl fallback, parallel items, bulk caps, currency floors - Add Firecrawl as JS-SPA fallback after EXA (handles noon.com, coupang, daraz, tokopedia, lazada) - Parallelize item fetching per country with 200ms stagger: runtime 38min to ~4.5min - Add CURRENCY_MIN floors (NGN:50, IDR:500, KRW:1000, etc.) to reject product codes as prices - Add ITEM_USD_MAX bulk caps (sugar:8USD, salt:5USD, rice:6USD, etc.) applied to both EXA and Firecrawl - Fix SA: use noon.com/saudi-en + carrefour.com.sa (removes luluhypermarket cross-country pollution) - Fix EG: use carrefouregypt.com + spinneys.com.eg + seoudi.com (removes GCC luluhypermarket) - Expand sites for DE, MX, ZA, TR, NG, KR, IN, PK, AR, ID, PH to improve coverage - Sync scripts/shared/grocery-basket.json with shared/grocery-basket.json * fix(grocery-basket): address PR review comments P1+P2 P1 - fix ranking with incomplete data: only include countries with >=70% item coverage (>=7/10) in cheapest/mostExpensive ranking — prevents a country with 4/10 items appearing cheapest due to missing data P1 - fix regex false-match on pack sizes / weights: try currency-first pattern (SAR 8.99) before number-first to avoid matching pack counts; use matchAll and take last match P2 - mark seed-miss responses as non-cacheable: add upstream_unavailable to proto + return true on empty seed so gateway sets Cache-Control: no-store on cold deploy * fix(generated): update EconomicService OpenAPI docs for upstream_unavailable field |
||
|
Elie Habib
|
c0bf784d21 |
feat(finance): crypto sectors heatmap + DeFi/AI/Alt token panels + expanded crypto news (#1900)
* feat(market): add crypto sectors heatmap and token panels (DeFi, AI, Other) backend - Add shared/crypto-sectors.json, defi-tokens.json, ai-tokens.json, other-tokens.json configs - Add scripts/seed-crypto-sectors.mjs and seed-token-panels.mjs seed scripts - Add proto messages for ListCryptoSectors, ListDefiTokens, ListAiTokens, ListOtherTokens - Add change7d field (field 6) to CryptoQuote proto message - Run buf generate to produce updated TypeScript bindings - Add server handlers for all 4 new RPCs reading from seeded Redis cache - Wire handlers into marketHandler and register cache keys with BOOTSTRAP_TIERS=slow - Wire seedCryptoSectors and seedTokenPanels into ais-relay.cjs seedAllMarketData loop * feat(panels): add crypto sectors heatmap and token panels (DeFi, AI, Other) - Add TokenData interface to src/types/index.ts - Wire ListCryptoSectorsResponse/ListDefiTokensResponse/ListAiTokensResponse/ListOtherTokensResponse into market service with circuit breakers and hydration fallbacks - Add CryptoHeatmapPanel, TokenListPanel, DefiTokensPanel, AiTokensPanel, OtherTokensPanel to MarketPanel.ts - Register 4 new panels in panels.ts FINANCE_PANELS and cryptoDigital category - Instantiate new panels in panel-layout.ts - Load data in data-loader.ts loadMarkets() alongside existing crypto fetch * fix(crypto-panels): resolve test failures and type errors post-review - Add @ts-nocheck to regenerated market service_server/client (matches repo convention) - Add 4 new RPC routes to RPC_CACHE_TIER in gateway.ts (route-cache-tier test) - Sync scripts/shared/ with shared/ for new token/sector JSON configs - Restore non-market generated files to origin/main state (avoid buf version diff) * fix(crypto-panels): address code review findings (P1-P3) - ais-relay seedTokenPanels: add empty-guard before Redis write to prevent overwriting cached data when all IDs are unresolvable - server _feeds.ts: sync 4 missing crypto feeds (Wu Blockchain, Messari, NFT News, Stablecoin Policy) with client-side feeds.ts - data-loader: expose panel refs outside try block so catch can call showRetrying(); log error instead of swallowing silently - MarketPanel: replace hardcoded English error strings with t() calls (failedSectorData / failedCryptoData) to honour user locale - seed-token-panels.mjs: remove unused getRedisCredentials import - cache-keys.ts: one BOOTSTRAP_TIERS entry per line for consistency * fix(crypto-panels): three correctness fixes for RSS proxy, refresh, and Redis write visibility - api/_rss-allowed-domains.js: add 7 new crypto domains (decrypt.co, blockworks.co, thedefiant.io, bitcoinmagazine.com, www.dlnews.com, cryptoslate.com, unchainedcrypto.com) so rss-proxy.js accepts the new finance feeds instead of rejecting them as disallowed hosts - src/App.ts: add crypto-heatmap/defi-tokens/ai-tokens/other-tokens to the periodic markets refresh viewport condition so panels on screen continue receiving live updates, not just the initial load - ais-relay seedTokenPanels: capture upstashSet return values and log PARTIAL if any Redis write fails, matching seedCryptoSectors pattern |
||
|
Elie Habib
|
0dae526a4b |
feat(markets): add NSE/BSE India market support (#1863)
* feat(config): add NSE and BSE (India) market support (#1102) * fix(india-markets): wire NSE/BSE symbols into stocks.json so seed fetches them - Add 20 India symbols (^NSEI, ^BSESN, 18x .NS equities) to shared/stocks.json - Mark all .NS symbols + indices as yahooOnly (Finnhub does not support NSE) - Remove orphan src/config/india-markets.ts; stocks.json is the seed source of truth * fix(india-markets): sync scripts/shared/stocks.json mirror * fix(ci): exclude scripts/data/ and scripts/node_modules/ from unicode safety scan --------- Co-authored-by: lspassos1 <lspassos@icloud.com> |
||
|
Elie Habib
|
8c7c03b29d |
feat: expand commodities from 6 to 14 symbols (#1776)
Metals: add Platinum (PL=F), Palladium (PA=F), Aluminum (ALI=F) Energy: add Brent Crude (BZ=F), Gasoline RBOB (RB=F), Heating Oil (HO=F) Strategic: add Uranium ETF (URA), Lithium & Battery ETF (LIT) Config-only change. Relay auto-fetches all symbols on next deploy. Grouped by category: metals first, then energy, then strategic proxies. |
||
|
Steven J. Miklovic
|
6e32a346c3 |
Add Greek news channels & feed (#1602)
* Add Greek news channels * Add ERT and SKAI hlsUrl to LiveNewsPanel.ts --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
f336418c17 |
feat(advisories): gold standard migration for security advisories (#1637)
* feat(advisories): gold standard migration for security advisories Move security advisories from client-side RSS fetching (24 feeds per page load) to Railway cron seed with Redis-read-only Vercel handler. - Add seed script fetching via relay RSS proxy with domain allowlist - Add ListSecurityAdvisories proto, handler, and RPC cache tier - Add bootstrap hydration key for instant page load - Rewrite client service: bootstrap -> RPC fallback, no browser RSS - Wire health.js, seed-health.js, and dataSize tracking * fix(advisories): empty RPC returns ok:true, use full country map P1 fixes from Codex review: - Return ok:true for empty-but-successful RPC responses so the panel clears to empty instead of stuck loading on cold environments - Replace 50-entry hardcoded country map with 251-entry shared config generated from the project GeoJSON + aliases, matching coverage of the old client-side nameToCountryCode matcher * fix(advisories): add Cote d'Ivoire and other missing country aliases Adds 14 missing aliases including "cote d ivoire" (US State Dept title format), common article-prefixed names (the Bahamas, the Gambia), and alternative official names (Czechia, Eswatini, Cabo Verde, Timor-Leste). * fix(proto): inject @ts-nocheck via Makefile generate target buf generate does not emit @ts-nocheck, but tsc strict mode rejects the generated code. Adding a post-generation sed step in the Makefile ensures both CI proto-freshness (make generate + diff) and CI typecheck (tsc --noEmit) pass consistently. |
||
|
Elie Habib
|
5d19ce45c7 |
fix(feeds): remove dead feed domains from RSS allowed-domains list (#1626)
PR #1596 removed the feeds but left the domains in the allowlist. The relay still accepted proxy requests for these 403-blocked domains from clients with cached old bundles. Removed: - breakingdefense.com (403) - www.arabnews.com (403) - www.aei.org (403) - mymodernmet.com (403) Updated all 3 copies: shared/, scripts/shared/, api/ |
||
|
Elie Habib
|
d4088fede5 |
fix(feeds): update dead RSS feed URLs (#1575)
- a16z: a16z.com/feed/ -> www.a16z.news/feed - First Round Review: /feed.xml -> /articles/rss - RAND: Google News proxy -> rand.org/pubs/articles.xml (direct) - Add www.a16z.news to allowed domains |
||
|
Elie Habib
|
cad24d8817 |
fix(predictions): move prediction-tags.json into scripts/data/ for Railway (#1518)
Railway deploys with rootDirectory=scripts/, so ../shared/ resolves to /shared/ which doesn't exist. Move the canonical file to scripts/data/ and update all four consumers. |
||
|
RaulC
|
bfcf7d88ec |
chore(predictions): extract shared tags and remove unused open_interest (#1512)
- Move GEOPOLITICAL_TAGS, TECH_TAGS, FINANCE_TAGS, and EXCLUDE_KEYWORDS to shared/prediction-tags.json so seed, RPC handler, and client all reference a single source of truth - Remove open_interest proto field (always 0 for Polymarket, never displayed in UI) and corresponding openInterest assignments Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
09fc20fbdf |
fix: remove smartraveller.gov.au from RSS allowed domains (#1273)
Persistent relay timeouts from smartraveller.gov.au. Remove both bare and www variants from all three allowed-domains files. |
||
|
Elie Habib
|
dd127447c0 |
refactor: consolidate duplicated market data lists into shared JSON configs (#1212)
Adding a new item (crypto, ETF, stablecoin, gulf symbol, etc.) previously required editing 2-4 files because the same list was hardcoded independently in seed scripts, RPC handlers, and frontend config. Following the proven shared/crypto.json pattern, extract 6 new shared JSON configs so each list has a single source of truth. New shared configs: - shared/stablecoins.json (ids + coinpaprika mappings) - shared/etfs.json (BTC spot ETF tickers + issuers) - shared/gulf.json (GCC indices, currencies, oil benchmarks) - shared/sectors.json (sector ETF symbols + names) - shared/commodities.json (VIX, gold, oil, gas, silver, copper) - shared/stocks.json (market symbols + yahoo-only set) All seed scripts, RPC handlers, and frontend config now import from these shared JSON files instead of maintaining independent copies. |
||
|
Nicolas Dos Santos
|
7b9426299d |
fix: Tech Readiness toggle, Crypto top 10, FIRMS API key check (#1132, #979, #997) (#1135)
* fix: three panel issues — Tech Readiness toggle, Crypto top 10, FIRMS key check 1. #1132 — Add tech-readiness to FULL_PANELS so it appears in the Settings toggle list for Full/Geopolitical variant users. 2. #979 — Expand crypto panel from 4 coins to top 10 by market cap (BTC, ETH, USDT, BNB, SOL, XRP, USDC, ADA, DOGE, TRX) across client config, server metadata, CoinPaprika fallback map, and seed script. 3. #997 — Check isFeatureAvailable('nasaFirms') before loading FIRMS data. When the API key is missing, show a clear "not configured" message instead of the generic "No fire data available". Closes #1132, closes #979, closes #997 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: replace stablecoins with AVAX/LINK, remove duplicate key, revert FIRMS change - Replace USDT/USDC (stablecoins pegged ~$1) with AVAX and LINK - Remove duplicate 'usd-coin' key in COINPAPRIKA_ID_MAP - Add CoinPaprika fallback IDs for avalanche-2 and chainlink - Revert FIRMS API key gating (handled differently now) - Add sync comments across the 3 crypto config locations * fix: update AIS relay + seed CoinPaprika fallback for all 10 coins The AIS relay (primary seeder) still had the old 4-coin list. The seed script's CoinPaprika fallback map was also missing the new coins. Both now have all 10 entries. * refactor: DRY crypto config into shared/crypto.json Single source of truth for crypto IDs, metadata, and CoinPaprika fallback mappings. All 4 consumers now import from shared/crypto.json: - src/config/markets.ts (client) - server/worldmonitor/market/v1/_shared.ts (server) - scripts/seed-crypto-quotes.mjs (seed script) - scripts/ais-relay.cjs (primary relay seeder) Adding a new coin now requires editing only shared/crypto.json. * chore: fix pre-existing markdown lint errors in README.md Add blank lines between headings and lists per MD022/MD032 rules. * fix: correct CoinPaprika XRP mapping and add crypto config test - Fix xrp-ripple → xrp-xrp (current CoinPaprika id) - Add tests/crypto-config.test.mjs: validates every coin has meta, coinpaprika mapping, unique symbols, no stablecoins, and valid id format — bad fallback ids now fail fast * test: validate CoinPaprika ids against live API The regex-only check wouldn't have caught the xrp-ripple typo. New test fetches /v1/coins from CoinPaprika and asserts every configured id exists. Gracefully skips if API is unreachable. * fix(test): handle network failures in CoinPaprika API validation Wrap fetch in try-catch so DNS failures, timeouts, and rate limits skip gracefully instead of failing the test suite. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
f56f84d6f9 |
fix(webcams): MTV Lebanon as live stream, not RSS (reverts #1121) (#1122)
* Revert "feat(feeds): add MTV Lebanon News YouTube feed to Middle East region (#1121)"
This reverts commit
|
||
|
Elie Habib
|
5bee51ab79 | feat(feeds): add MTV Lebanon News YouTube feed to Middle East region (#1121) | ||
|
JYR-AI
|
6745f47305 |
Variant/commodity (#1040)
* commod variants * mining map layers complete * metal news feed * commod variant final * readme update * fix: clean up commodity variant for merge readiness - Remove duplicate FEEDS definition (central feeds.ts is source of truth) - Remove duplicate inline ALLOWED_DOMAINS in rss-proxy.js (use shared module) - Add 14 commodity RSS domains to shared/rss-allowed-domains.json - Remove invalid geopoliticalBoundaries property (not in MapLayers type) - Fix broken mobile-map-integration-harness imports - Remove Substack credit link from app header - Rename i18n key commod → commodity - Extract mineralColor() helper for DRY color mapping - Add XSS-safe tooltips for mining sites, processing plants, commodity ports - Add missing interface fields (annualOutput, materials, capacityTpa, annualVolumeMt) - Comment out unused COMMODITY_MINERS export - Isolate commodity DeckGL changes from unrelated basemap refactor * fix: hide commodity variant from selector until testing complete Only show the commodity option in the variant switcher when the user is already on the commodity variant (same pattern as happy variant). Other variants (full, tech, finance) won't see the commodity link. --------- Co-authored-by: jroachell <jianyin.roachell@siriusxm.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
fe44cbe182 |
feat: expand happy variant RSS feeds, render GPS jamming as hexagons, fix layer toggle repaint (#1070)
- Add 7 RSS feed categories for happy variant: positive, science, nature, health, inspiring, community with sources like Mongabay, Yes! Magazine, Shareable, Conservation Optimism, My Modern Met, GNN subcategories - Switch GPS jamming layer from ScatterplotLayer (circles) to H3HexagonLayer (tessellated hexagons) matching industry standard - Fix layer toggle not visually updating by adding triggerRepaint() after all deck.gl setProps() calls — MapboxOverlay in interleaved mode requires MapLibre to repaint for changes to appear |
||
|
Mert Efe Şensoy
|
f771114522 |
feat: aviation monitoring layer with flight tracking, airline intel panel, and news feeds (#907)
* feat: Implement comprehensive aviation monitoring service with flight search, status, news, and tracking. * feat: Introduce Airline Intelligence Panel with aviation data tabs, map components, and localization. * feat: Implement DeckGL-based map for advanced visualization, D3/SVG fallback, i18n support, and aircraft tracking. * Update server/worldmonitor/aviation/v1/get-carrier-ops.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update server/worldmonitor/aviation/v1/search-flight-prices.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update server/worldmonitor/aviation/v1/track-aircraft.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update server/worldmonitor/aviation/v1/get-airport-ops-summary.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update proto/worldmonitor/aviation/v1/position_sample.proto Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update server/worldmonitor/aviation/v1/list-airport-flights.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update proto/worldmonitor/aviation/v1/price_quote.proto Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: Add server-side endpoints for aviation news and aircraft tracking, and introduce a new DeckGLMap component for map visualization. * Update server/worldmonitor/aviation/v1/list-airport-flights.ts The cache key for listAirportFlights excludes limit, but the upstream fetch/simulated generator uses limit to determine how many flights to return. If the first request within TTL uses a small limit, larger subsequent requests will be incorrectly capped until cache expiry. Include limit (or a normalized bucket/max) in cacheKey, or always fetch/cache a fixed max then slice per request. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update server/worldmonitor/aviation/v1/get-flight-status.ts getFlightStatus accepts origin, but cacheKey does not include it. This can serve cached results from an origin-less query to an origin-filtered query (or vice versa). Add origin (normalized) to the cache key or apply filtering after fetch to ensure cache correctness. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: Implement DeckGL map for advanced visualization and new aviation data services. * fix(aviation): prevent cache poisoning and keyboard shortcut in inputs - get-carrier-ops: move minFlights filter post-cache to avoid cache fragmentation (different callers sharing cached full result) - AviationCommandBar: guard Ctrl+J shortcut so it does not fire when focus is inside an INPUT or TEXTAREA element Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: introduce AviationCommandBar component for parsing user commands, fetching aviation data, and displaying results. * feat: Implement aircraft tracking service with OpenSky and simulated data sources. * feat: introduce DeckGLMap component for WebGL-accelerated map visualizations using deck.gl and maplibre-gl. * fix(aviation): address code review findings for PR #907 Proto: add missing (sebuf.http.query) annotations on all GET request fields across 6 proto files; add currency/market fields to SearchFlightPricesRequest. Server: add parseStringArray to aviation _shared.ts and apply to get-airport-ops-summary, get-carrier-ops, list-aviation-news handlers to prevent crash on comma-separated query params; remove leaked API token from URL params in travelpayouts_data; fix identical simulated flight statuses in list-airport-flights; remove unused endDate var; normalize cache key entity casing in list-aviation-news. Client: refactor AirlineIntelPanel to extend Panel base class and register in DEFAULT_PANELS for full/tech/finance variants; fix AviationCommandBar reference leak with proper destroy() cleanup in panel-layout; rename priceUsd→priceAmount in display type and all usages; change auto-refresh to call refresh() instead of loadOps(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: introduce aviation command bar component with aircraft tracking and flight information services. * feat: Add `AirlineIntelPanel` component for displaying airline operations, flights, carriers, tracking, news, and prices in a tabbed interface. * feat: Add endpoints for listing airport flights and fetching aviation news. * Update proto/worldmonitor/aviation/v1/search_flight_prices.proto Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: Add server endpoint for listing airport flights and client-side MapPopup types and utilities. * feat: Introduce MapPopup component with support for various data types and responsive positioning for map features. * feat: Add initial English localization file (en.json). * fix(aviation): address PR review findings across aviation stack - Add User-Agent header to Travelpayouts provider (server convention) - Use URLSearchParams for API keys instead of raw URL interpolation - Add input length validation on flightNumber (max 10 chars) - Replace regex XML parsing with fast-xml-parser in aviation news - Fix (f as any)._airport type escape with typed Map<FI, string> - Extract DEFAULT_WATCHED_AIRPORTS constant from hardcoded arrays - Use event delegation for AirlineIntelPanel price search listener - Add bootstrap hydration key for flight delays - Bump OpenSky cache TTL to 120s (anonymous tier rate limit) - Match DeckGLMap aircraft poll interval to server cache (120s) - Fix GeoJSON polygon winding order (shoelace check + auto-reversal) * docs: add aviation env vars to .env.example AVIATIONSTACK_API, ICAO_API_KEY, TRAVELPAYOUTS_API_TOKEN * feat: Add aviation news listing API and introduce shared RSS allowed domains. * fix: add trailing newline to rss-allowed-domains.json, remove unused ringIsClockwise --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
898ac7b1c4 |
perf(rss): route RSS direct to Railway, skip Vercel middleman (#961)
* perf(rss): route RSS direct to Railway, skip Vercel middleman
Vercel /api/rss-proxy has 65% error rate (207K failed invocations/12h).
Route browser RSS requests directly to Railway (proxy.worldmonitor.app)
via Cloudflare CDN, eliminating Vercel as middleman.
- Add VITE_RSS_DIRECT_TO_RELAY feature flag (default off) for staged rollout
- Centralize RSS proxy URL in rssProxyUrl() with desktop/dev/prod routing
- Make Railway /rss public (skip auth, keep rate limiting with CF-Connecting-IP)
- Add wildcard *.worldmonitor.app CORS + always emit Vary: Origin on /rss
- Extract ~290 RSS domains to shared/rss-allowed-domains.cjs (single source of truth)
- Convert Railway domain check to Set for O(1) lookups
- Remove rss-proxy from KEYED_CLOUD_API_PATTERN (no longer needs API key header)
- Add edge function test for shared domain list import
* fix(edge): replace node:module with JSON import for edge-compatible RSS domains
api/_rss-allowed-domains.js used createRequire from node:module which is
unsupported in Vercel Edge Runtime, breaking all edge functions (including
api/gpsjam). Replaced with JSON import attribute syntax that works in both
esbuild (Vercel build) and Node.js 22+ (tests).
Also fixed middleware.ts TS18048 error where VARIANT_OG[variant] could be
undefined.
* test(edge): add guard against node: built-in imports in api/ files
Scans ALL api/*.js files (including _ helpers) for node: module imports
which are unsupported in Vercel Edge Runtime. This would have caught the
createRequire(node:module) bug before it reached Vercel.
* fix(edge): inline domain array and remove NextResponse reference
- Replace `import ... with { type: 'json' }` in _rss-allowed-domains.js
with inline array — Vercel esbuild doesn't support import attributes
- Replace `NextResponse.next()` with bare `return` in middleware.ts —
NextResponse was never imported
* ci(pre-push): add esbuild bundle check and edge function tests
The pre-push hook now catches Vercel build failures locally:
- esbuild bundles each api/*.js entrypoint (catches import attribute
syntax, missing modules, and other bundler errors)
- runs edge function test suite (node: imports, module isolation)
|