mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
8cca8d19e3
* feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup
Prereq for the re-export-share Comtrade seeder (plan 2026-04-24-003),
usable by any cohort seeder whose consumer needs bundle-level freshness.
Two coupled changes:
1. `_bundle-runner.mjs` injects `BUNDLE_RUN_STARTED_AT_MS` into every
spawned child. All siblings in a single bundle run share one value
(captured at `runBundle` start, not spawn time). Consumers use this
to detect stale peer keys — if a peer's seed-meta predates the
current bundle run, fall back to a hard default rather than read
a cohort-peer's last-week output.
2. `_seed-utils.mjs::runSeed` registers a `process.once('SIGTERM')`
handler that releases the acquired lock and extends existing-data
TTL before exiting 143. `_bundle-runner.mjs` sends SIGTERM on
section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without
this handler the `finally` path never runs on SIGKILL, leaving
the 30-min acquireLock reservation in place until its own TTL
expires — the next cron tick silently skips the resource.
Regression guard memory: `bundle-runner-sigkill-leaks-child-lock` (PR
#3128 root cause).
Tests added:
- bundle-runner env injection (value within run bounds)
- sibling sections share the same timestamp (critical for the
consumer freshness guard)
- runSeed SIGTERM path: exit 143 + cleanup log
- process.once contract: second SIGTERM does not re-enter handler
* fix(seed): address P1/P2 review findings on SIGTERM + bundle contracts
Addresses PR #3384 review findings (todos 256, 257, 259, 260):
#256 (P1) — SIGTERM handler narrowed to fetch phase only. Was installed
at runSeed entry and armed through every `process.exit` path; could
race `emptyDataIsFailure: true` strict-floor exits (IMF-External,
WB-bulk) and extend seed-meta TTL when the contract forbids it —
silently re-masking 30-day outages. Now the handler is attached
immediately before `withRetry(fetchFn)` and removed in a try/finally
that covers all fetch-phase exit branches.
#257 (P1) — `BUNDLE_RUN_STARTED_AT_MS` now has a first-class helper.
Exported `getBundleRunStartedAtMs()` from `_seed-utils.mjs` with JSDoc
describing the bundle-freshness contract. Fleet-wide helper so the
next consumer seeder imports instead of rediscovering the idiom.
#259 (P2) — SIGTERM cleanup runs `Promise.allSettled` on disjoint-key
ops (`releaseLock` + `extendExistingTtl`). Serialising compounded
Upstash latency during the exact failure mode (Redis degraded) this
handler exists to handle, risking breach of the 5s SIGKILL grace.
#260 (P2) — `_bundle-runner.mjs` asserts topological order on
optional `dependsOn` section field. Throws on unknown-label refs and
on deps appearing at a later index. Fleet-wide contract replacing
the previous prose-comment ordering guarantee.
Tests added/updated:
- New: SIGTERM handler removed after fetchFn completes (narrowed-scope
contract — post-fetch SIGTERM must NOT trigger TTL extension)
- New: dependsOn unknown-label + out-of-order + happy-path (3 tests)
Full test suite: 6,866 tests pass (+4 net).
* fix(seed): getBundleRunStartedAtMs returns null outside a bundle run
Review follow-up: the earlier `Math.floor(Date.now()/1000)*1000` fallback
regressed standalone (non-bundle) runs. A consumer seeder invoked
manually just after its peer wrote `fetchedAt = (now - 5s)` would see
`bundleStartMs = Date.now()`, reject the perfectly-fresh peer envelope
as "stale", and fall back to defaults — defeating the point of the
peer-read path outside the bundle.
Returning null when `BUNDLE_RUN_STARTED_AT_MS` is unset/invalid keeps
the freshness gate scoped to its real purpose (across-bundle-tick
staleness) and lets standalone runs skip the gate entirely. Consumers
check `bundleStartMs != null` before applying the comparison; see the
companion `seed-sovereign-wealth.mjs` change on the stacked PR.
* test(seed): SIGTERM cleanup test now verifies Redis DEL + EXPIRE calls
Greptile review P2 on PR #3384: the existing test only asserted exit
code + log line, not that the Redis ops were actually issued. The
log claim was ahead of the test.
Fixture now logs every Upstash fetch call's shape (EVAL / pipeline-
EXPIRE / other) to stderr. Test asserts:
- >=1 EVAL op was issued during SIGTERM cleanup (releaseLock Lua
script on the lock key)
- >=1 pipeline-EXPIRE op was issued (extendExistingTtl on canonical
+ seed-meta keys)
- The EVAL body carries the runSeed-generated runId (proves it's
THIS run's release, not a phantom op)
- The EXPIRE pipeline touches both the canonicalKey AND the
seed-meta key (proves the keys[] array was built correctly
including the extraKeys merge path)
Full test suite: 6,866 tests pass, typecheck clean.
* feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read
Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md
Motivating case. Before this PR, the SWF `rawMonths` denominator for
the `sovereignFiscalBuffer` dimension used GROSS annual imports for
every country. For re-export hubs (goods transiting without domestic
settlement), this structurally under-reports resilience: UAE's 2023
$941B of imports include $334B of transit flow that never represents
domestic consumption. Net imports = gross × (1 − reexport_share).
The previous (PR 3A) design flattened a hand-curated YAML into Redis;
the YAML shipped empty and never populated, so the correction never
applied and the cohort audit showed no movement.
Gap #2 (this PR). Two coupled changes to make the correction actually
apply:
1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`).
Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and
`flowCode=M` (imports) per cohort member, compute share = RX/M at
the latest co-populated year, clamp to [0.05, 0.95], publish the
envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription
key never reaches URL/logs/Redis. `maxRecords=250000` cap with
truncation detection. Sequential + retry-on-429 with backoff.
Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0):
['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200
with zero RX rows — Comtrade doesn't expose RX for those reporters.
2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`).
Swaps `loadReexportShareByCountry()` (YAML) for
`loadReexportShareFromRedis()` (Redis key written by #1). Guarded
by bundle-run freshness: if the sibling Reexport-Share seeder's
`seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the
prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback
to gross imports rather than apply last-month's stale share.
Health registries. Both new keys registered in BOTH `api/health.js`
SEED_META (60-day alert threshold) and `api/seed-health.js`
SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match.
Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share
timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min
worst-case). Ordering preserved: Reexport-Share before Sovereign-
Wealth so the SWF seeder reads a freshly-written key in the same
cron tick.
Deletions. YAML + loader + 7 obsolete loader tests removed; single
source of truth is now Comtrade → Redis.
Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm)
which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed
SIGTERM cleanup. This PR's bundle-freshness guard depends on
that env variable.
Tests (19 new, 7 deleted, +12 net):
- Pure math: parseComtradeFlowResponse, computeShareFromFlows,
clampShare, declareRecords + credential-leak source scan (15)
- Integration (Gap #2 regression guards): SWF seeder loadReexport
ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5)
- Health registry dual-registry drift guard — scoped to this PR's
keys, respecting pre-existing asymmetry (4)
- Bundle-ordering + timeout assertions (2)
Phase 0 cohort validation committed to plan. Full test suite
passes: 6,881 tests.
* fix(resilience): address P1/P2 review findings — adopt shared helpers, pin freshness boundary
Addresses PR #3385 review findings:
#257 (P1) consumer — `seed-sovereign-wealth.mjs` imports the shared
`getBundleRunStartedAtMs` helper from `_seed-utils.mjs` (added in the
prereq commit) instead of its own `getBundleStartMs`. Single source of
truth for the bundle-freshness contract.
#258 (P2) — `seed-recovery-reexport-share.mjs` isMain guard uses the
canonical `pathToFileURL(process.argv[1]).href === import.meta.url`
form instead of basename-suffix matching. Handles symlinks, case-
different paths on macOS HFS+, and Windows path separators without
string munging.
#260 (P2) consumer — Sovereign-Wealth declares `dependsOn:
['Reexport-Share']` in the bundle spec. `_bundle-runner.mjs` (prereq
commit) now enforces topological order on load and throws on
violation — replaces the previous prose-comment ordering contract.
#261 (P2) — added a test to `tests/seed-sovereign-wealth-reads-redis-
reexport-share.test.mts` pinning the inclusive-boundary semantic:
`fetchedAtMs === bundleStartMs` must be treated as FRESH. Guards
against a future refactor to `<=` that would silently reject peers
writing at the very first millisecond of the bundle run.
Rebased onto updated prereq. Full test suite: 6,886 tests pass (+5 net).
* fix(resilience): freshness gate skipped in standalone mode; meta still required
Review catch: the previous `bundleStartMs = Date.now()` fallback made
standalone/manual `seed-sovereign-wealth.mjs` runs ALWAYS reject any
previously-seeded re-export-share meta as "stale" — even when the
operator ran the Reexport seeder milliseconds beforehand. Defeated
the point of the peer-read path outside the bundle.
With `getBundleRunStartedAtMs()` now returning null outside a bundle
(companion commit on the prereq branch), the consumer only applies
the freshness gate when `bundleStartMs != null`. Standalone runs
accept any `fetchedAt` — the operator is responsible for ordering.
Two guards survive the change:
- Meta MUST exist (absence = peer-outage fail-safe, both modes)
- In-bundle: meta MUST be at or after `BUNDLE_RUN_STARTED_AT_MS`
Two new tests pin both modes:
- standalone: accepts meta written 10 min before this process started
- standalone: still rejects missing meta (peer-outage fail-safe
survives gate bypass)
Rebased onto updated prereq. Full test suite: 6,888 tests (+2 net).
* fix(resilience): filter world-aggregate Comtrade rows + skip final-retry sleep
Greptile review of PR #3385 flagged two P2s in the Comtrade seeder.
Finding #3 (parseComtradeFlowResponse double-count risk):
`cmdCode=TOTAL` without a partner filter currently returns only
world-aggregate rows in practice — but `parseComtradeFlowResponse`
summed every row unconditionally. A future refactor adding per-
partner querying would silently double-count (world-aggregate row +
partner-level rows for the same year), cutting the derived share in
half with no test signal.
Fix: explicit `partnerCode ∈ {'0', 0, null/undefined}` filter. Matches
current empirical behavior (aggregate-only responses) and makes the
construct robust to a future partner-level query.
Finding #4 (wasted backoff on final retry):
429 and 5xx branches slept `backoffMs` before `continue`, but on
`attempt === RETRY_MAX_ATTEMPTS` the loop condition fails immediately
after — the sleep was pure waste. Added early-return (parallel to the
existing pattern in the network-error catch branch) so the final
attempt exits the retry loop at the first non-success response
without extra latency.
Tests:
- 3 new `parseComtradeFlowResponse` variants: world-only filter,
numeric-0 partnerCode shape, rows without partnerCode field
- Existing tests updated: the double-count assertion replaced with
a "per-partner rows must NOT sum into the world-aggregate total"
assertion that pins the new contract
Rebased onto updated prereq. Full test suite: 6,890 tests (+2 net).
World Monitor
Real-time global intelligence dashboard — AI-powered news aggregation, geopolitical monitoring, and infrastructure tracking in a unified situational awareness interface.
-0078D4?style=for-the-badge&logo=windows&logoColor=white){kind=icon}
-FCC624?style=for-the-badge&logo=linux&logoColor=black){kind=icon}
Documentation · Releases · Contributing
What It Does
- 500+ curated news feeds across 15 categories, AI-synthesized into briefs
- Dual map engine — 3D globe (globe.gl) and WebGL flat map (deck.gl) with 45 data layers
- Cross-stream correlation — military, economic, disaster, and escalation signal convergence
- Country Intelligence Index — composite risk scoring across 12 signal categories
- Finance radar — 92 stock exchanges, commodities, crypto, and 7-signal market composite
- Local AI — run everything with Ollama, no API keys required
- 5 site variants from a single codebase (world, tech, finance, commodity, happy)
- Native desktop app (Tauri 2) for macOS, Windows, and Linux
- 21 languages with native-language feeds and RTL support
For the full feature list, architecture, data sources, and algorithms, see the documentation.
Quick Start
git clone https://github.com/koala73/worldmonitor.git
cd worldmonitor
npm install
npm run dev
Open localhost:5173. No environment variables required for basic operation.
For variant-specific development:
npm run dev:tech # tech.worldmonitor.app
npm run dev:finance # finance.worldmonitor.app
npm run dev:commodity # commodity.worldmonitor.app
npm run dev:happy # happy.worldmonitor.app
See the self-hosting guide for deployment options (Vercel, Docker, static).
Tech Stack
| Category | Technologies |
|---|---|
| Frontend | Vanilla TypeScript, Vite, globe.gl + Three.js, deck.gl + MapLibre GL |
| Desktop | Tauri 2 (Rust) with Node.js sidecar |
| AI/ML | Ollama / Groq / OpenRouter, Transformers.js (browser-side) |
| API Contracts | Protocol Buffers (92 protos, 22 services), sebuf HTTP annotations |
| Deployment | Vercel Edge Functions (60+), Railway relay, Tauri, PWA |
| Caching | Redis (Upstash), 3-tier cache, CDN, service worker |
Full stack details in the architecture docs.
Flight Data
Flight data provided gracefully by Wingbits, the most advanced ADS-B flight data solution.
Data Sources
WorldMonitor aggregates 65+ external data sources across geopolitics, finance, energy, climate, aviation, cyber, military, infrastructure, and news intelligence. See the full data sources catalog for providers, feed tiers, and collection methods.
Contributing
Contributions welcome! See CONTRIBUTING.md for guidelines.
npm run typecheck # Type checking
npm run build:full # Production build
License
AGPL-3.0 for non-commercial use. Commercial license required for any commercial use.
| Use Case | Allowed? |
|---|---|
| Personal / research / educational | Yes |
| Self-hosted (non-commercial) | Yes, with attribution |
| Fork and modify (non-commercial) | Yes, share source under AGPL-3.0 |
| Commercial use / SaaS / rebranding | Requires commercial license |
See LICENSE for full terms. For commercial licensing, contact the maintainer.
Copyright (C) 2024-2026 Elie Habib. All rights reserved.
Author
Elie Habib — GitHub
Contributors
Security Acknowledgments
We thank the following researchers for responsibly disclosing security issues:
- Cody Richard — Disclosed three security findings covering IPC command exposure, renderer-to-sidecar trust boundary analysis, and fetch patch credential injection architecture (2026)
See our Security Policy for responsible disclosure guidelines.
worldmonitor.app · docs.worldmonitor.app · finance.worldmonitor.app · commodity.worldmonitor.app