Elie Habib 9c14820c69 fix(digest): brief filter-drop instrumentation + cache-key correctness (#3387)
* fix(digest): include sensitivity in digestFor cache key

buildDigest filters by rule.sensitivity BEFORE dedup, but digestFor
memoized only on (variant, lang, windowStart). Stricter-sensitivity
users in a shared bucket inherited the looser populator's pool,
producing the wrong story set and defeating downstream topic-grouping
adjacency once filterTopStories re-applied sensitivity.

Solution 1 from docs/plans/2026-04-24-004-fix-brief-topic-adjacency-defects-plan.md.

* feat(digest): instrument per-user filterTopStories drops

Adds an optional onDrop metrics callback to filterTopStories and threads
it through composeBriefFromDigestStories. The seeder aggregates counts
per composed brief and emits one structured log line per user per tick:

  [digest] brief filter drops user=<id> sensitivity=<s> in=<count>
    dropped_severity=<n> dropped_url=<n> dropped_headline=<n>
    dropped_shape=<n> out=<count>

Decides whether the conditional Solution 3 (post-filter regroup) is
warranted by quantifying how often post-group filter drops puncture
multi-member topics in production. No behaviour change for callers
that omit onDrop.

Solution 0 from docs/plans/2026-04-24-004-fix-brief-topic-adjacency-defects-plan.md.

* fix(digest): close two Sol-0 instrumentation gaps from code review

Review surfaced two P2 gaps in the filter-drop telemetry that weakened
its diagnostic purpose for Sol-3 gating:

1. Cap-truncation silent drop: filterTopStories broke on
   `out.length >= maxStories` BEFORE the onDrop emit sites, so up to
   (DIGEST_MAX_ITEMS - MAX_STORIES_PER_USER) stories per user were
   invisible. Added a 'cap' reason to DropMetricsFn and emit one event
   per skipped story so `in - out - sum(dropped_*) == 0` reconciles.

2. Wipeout invisibility: composeAndStoreBriefForUser only logged drop
   stats for the WINNING candidate. When every candidate composed to
   null, the log line never fired — exactly the wipeout case Sol-0
   was meant to surface. Now tracks per-candidate drops and emits an
   aggregate `outcome=wipeout` line covering all attempts.

Also tightens the digest-cache-key sensitivity regex test to anchor
inside the cache-key template literal (it would otherwise match the
unrelated `chosenCandidate.sensitivity ?? 'high'` in the new log line).

PR review residuals from
docs/plans/2026-04-24-004-fix-brief-topic-adjacency-defects-plan.md
ce-code-review run 20260424-232911-37a2d5df.

* chore: ignore .context/ ce-code-review run artifacts

The ce-code-review skill writes per-run artifacts (reviewer JSON,
synthesis.md, metadata.json) under .context/compound-engineering/.
These are local-only — neither tracked nor linted.

* fix(digest): emit per-attempt filter-drop rows, not per-user

Addresses two PR #3387 review findings:

- P2: Earlier candidates that composed to null (wiped out by post-group
  filtering) had their dropStats silently discarded when a later
  candidate shipped — exactly the signal Sol-0 was meant to surface.
- P3: outcome=wipeout row was labeled with allCandidateDrops[0]
  .sensitivity, misleading when candidates within one user have
  different sensitivities.

Fix: emit one structured row per attempted candidate, tagged with that
candidate's own sensitivity and variant. Outcome is shipped|rejected.
A wipeout is now detectable as "all rows for this user are rejected
within the tick" — no aggregate-row ambiguity. Removes the
allCandidateDrops accumulator entirely.

* fix(digest): align composeBriefFromDigestStories sensitivity default to 'high'

Addresses PR #3387 review (P2): composeBriefFromDigestStories defaulted
to `?? 'all'` while buildDigest, the digestFor cache key, and the new
per-attempt log line all default to `?? 'high'`. The mismatch is
harmless in production (the live cron path pre-filters the pool) but:

- A non-prefiltered caller with undefined sensitivity would silently
  ship medium/low stories.
- Per-attempt telemetry labels the attempt as `sensitivity=high` while
  compose actually applied 'all' — operators are misled.

Aligning compose to 'high' makes the four sites agree and the telemetry
honest. Production output is byte-identical (input pool was already
'high'-filtered upstream).

Adds 3 regression tests asserting the new default: critical/high admitted,
medium/low dropped, and onDrop fires reason=severity for the dropped
levels (locks in alignment with per-attempt telemetry).

* fix(digest): align remaining sensitivity defaults to 'high'

Addresses PR #3387 review (P2 + P3): three more sites still defaulted
missing sensitivity to 'all' while compose/buildDigest/cache/log now
treat it as 'high'.

P2 — compareRules (scripts/lib/brief-compose.mjs:35-36): the rank
function used to default to 'all', placing legacy undefined-sensitivity
rules FIRST in the candidate order. Compose then applied a 'high'
filter to them, shipping a narrow brief while an explicit 'all' rule
for the same user was never tried. Aligned to 'high' so the rank
matches what compose actually applies.

P3 — enrichBriefEnvelopeWithLLM (scripts/lib/brief-llm.mjs:526):
the digest prompt and cache key still used 'all' for legacy rules,
misleading personalization ("Reader sensitivity level: all" while the
brief contains only critical/high stories) and busting the cache for
legacy vs explicit-'all' rows that should share entries.

Also aligns the @deprecated composeBriefForRule (line 164) for
consistency, since tests still import it.

3 new regression tests in tests/brief-composer-rule-dedup.test.mjs
lock in the new ranking: explicit 'all' beats undefined-sensitivity,
undefined-sensitivity ties with explicit 'high' (decided by updatedAt),
and groupEligibleRulesByUser candidate order respects the rank.

6853/6853 tests pass (was 6850 → +3).
2026-04-25 00:23:29 +04:00

World Monitor

Real-time global intelligence dashboard — AI-powered news aggregation, geopolitical monitoring, and infrastructure tracking in a unified situational awareness interface.

GitHub stars GitHub forks Discord License: AGPL v3 TypeScript Last commit Latest release

Web App  Tech Variant  Finance Variant  Commodity Variant  Happy Variant

Download Windows  Download macOS ARM  Download macOS Intel  Download Linux

Documentation  ·  Releases  ·  Contributing

World Monitor Dashboard


What It Does

  • 500+ curated news feeds across 15 categories, AI-synthesized into briefs
  • Dual map engine — 3D globe (globe.gl) and WebGL flat map (deck.gl) with 45 data layers
  • Cross-stream correlation — military, economic, disaster, and escalation signal convergence
  • Country Intelligence Index — composite risk scoring across 12 signal categories
  • Finance radar — 92 stock exchanges, commodities, crypto, and 7-signal market composite
  • Local AI — run everything with Ollama, no API keys required
  • 5 site variants from a single codebase (world, tech, finance, commodity, happy)
  • Native desktop app (Tauri 2) for macOS, Windows, and Linux
  • 21 languages with native-language feeds and RTL support

For the full feature list, architecture, data sources, and algorithms, see the documentation.


Quick Start

git clone https://github.com/koala73/worldmonitor.git
cd worldmonitor
npm install
npm run dev

Open localhost:5173. No environment variables required for basic operation.

For variant-specific development:

npm run dev:tech       # tech.worldmonitor.app
npm run dev:finance    # finance.worldmonitor.app
npm run dev:commodity  # commodity.worldmonitor.app
npm run dev:happy      # happy.worldmonitor.app

See the self-hosting guide for deployment options (Vercel, Docker, static).


Tech Stack

Category Technologies
Frontend Vanilla TypeScript, Vite, globe.gl + Three.js, deck.gl + MapLibre GL
Desktop Tauri 2 (Rust) with Node.js sidecar
AI/ML Ollama / Groq / OpenRouter, Transformers.js (browser-side)
API Contracts Protocol Buffers (92 protos, 22 services), sebuf HTTP annotations
Deployment Vercel Edge Functions (60+), Railway relay, Tauri, PWA
Caching Redis (Upstash), 3-tier cache, CDN, service worker

Full stack details in the architecture docs.


Flight Data

Flight data provided gracefully by Wingbits, the most advanced ADS-B flight data solution.


Data Sources

WorldMonitor aggregates 65+ external data sources across geopolitics, finance, energy, climate, aviation, cyber, military, infrastructure, and news intelligence. See the full data sources catalog for providers, feed tiers, and collection methods.


Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines.

npm run typecheck        # Type checking
npm run build:full       # Production build

License

AGPL-3.0 for non-commercial use. Commercial license required for any commercial use.

Use Case Allowed?
Personal / research / educational Yes
Self-hosted (non-commercial) Yes, with attribution
Fork and modify (non-commercial) Yes, share source under AGPL-3.0
Commercial use / SaaS / rebranding Requires commercial license

See LICENSE for full terms. For commercial licensing, contact the maintainer.

Copyright (C) 2024-2026 Elie Habib. All rights reserved.


Author

Elie HabibGitHub

Contributors

Security Acknowledgments

We thank the following researchers for responsibly disclosing security issues:

  • Cody Richard — Disclosed three security findings covering IPC command exposure, renderer-to-sidecar trust boundary analysis, and fetch patch credential injection architecture (2026)

See our Security Policy for responsible disclosure guidelines.


worldmonitor.app  ·  docs.worldmonitor.app  ·  finance.worldmonitor.app  ·  commodity.worldmonitor.app

Star History

Star History Chart
Description
Mirrored from GitHub
Readme AGPL-3.0 380 MiB
Languages
TypeScript 49.4%
JavaScript 46.8%
CSS 2.9%
HTML 0.4%
Rust 0.2%
Other 0.1%