mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-05-08 16:22:00 +02:00
636ace7b2c0ccbbda2c6c8a7d8a6509e6568e340
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
39931456a1 |
feat(forecast): add structured scenario pipeline and trace export (#1646)
* feat(forecast): add AI Forecasts prediction module (Pro-tier)
MiroFish-inspired prediction engine that generates structured forecasts
across 6 domains (conflict, market, supply chain, political, military,
infrastructure) using existing WorldMonitor data streams.
- Proto definitions for ForecastService with GetForecasts RPC
- Dedicated seed script (seed-forecasts.mjs) with 6 domain detectors,
cross-domain cascade resolver, prediction market calibration, and
trend detection via prior snapshot comparison
- Premium-gated RPC handler (PREMIUM_RPC_PATHS enforcement)
- Lazy-loaded ForecastPanel with domain filters, probability bars,
trend arrows, signal evidence, and cascade links
- Health monitoring integration (seed-meta freshness tracking)
- Refresh scheduler with API key guard
* test(forecast): add 47 unit tests for forecast detectors and utilities
Covers forecastId, normalize, resolveCascades, calibrateWithMarkets,
computeTrends, and smoke tests for all 6 domain detectors. Exports
testable functions from seed script with direct-run guard.
* fix(forecast): domain mismatch 'infra' vs 'infrastructure', add panel category
- Seed script used 'infra' but ForecastPanel filtered on 'infrastructure',
causing Infra tab to show zero results
- Added 'forecast' to intelligence category in PANEL_CATEGORY_MAP
* fix(forecast): move CSS to one-time injection, improve type safety
- P2: Move style block from setContent to one-time document.head injection
to prevent CSS accumulation on repeated renders
- P3: Replace +toFixed(3) with Math.round for readability in seed script
- P3: Use Forecast type instead of any[] in RPC handler filter
* fix(forecast): handle sebuf proto data shapes from Redis
Detectors now normalize CII scores from server-side proto format
(combinedScore, TREND_DIRECTION_RISING, region) to uniform shape.
Outage severity handles proto enum format (SEVERITY_LEVEL_HIGH).
Added confidence floor of 0.3 for single-source predictions.
Verified against live Redis: 2 predictions generated (Iran infra
shutdown, IL political instability).
* feat(forecast): unlock AI Forecasts on web, lock desktop only (trial)
- Remove forecast RPC from PREMIUM_RPC_PATHS (web access is free)
- Panel locked on desktop only (same as oref-sirens/telegram-intel)
- Remove API key guards from data-loader and refresh scheduler
- Web users get full access during trial period
* chore: regenerate proto types with make generate
Re-ran make generate after rebasing on main. Plugin v0.7.0 dropped
@ts-nocheck from output, added it back to all 50 generated files.
Fixed 4 type errors from proto codegen changes:
- MarketSource enum -> string union type
- TemporalAnomalyProto -> TemporalAnomaly rename
- webcam lastUpdated number -> string
* chore: add proto freshness check to pre-push hook
Runs make generate before push and compares checksums of generated files.
If proto types are stale, blocks push with instructions to regenerate.
Skips gracefully if buf CLI is not installed.
* fix(forecast): use chokepoints v4 key, include ciiContribution in unrest
- P1: Switch chokepoints input from stale v2 to active v4 Redis key,
matching bootstrap.js and cache-keys.ts
- P2: Add ciiContribution to unrest component fallback chain in
normalizeCiiEntry so political detector reads the correct sebuf field
* feat(forecast): Phase 2 LLM scenario enrichment + confidence model
MiroFish-inspired enhancements:
- LLM scenario narratives via Groq/OpenRouter (narrative-only, no numeric
adjustment). Evidence-grounded prompts with mandatory signal citation
and few-shot examples from MiroFish's SECTION_SYSTEM_PROMPT_TEMPLATE.
- Top-4 predictions batched into single LLM call for cost efficiency.
- News context from newsInsights attached to all predictions for LLM
prompt grounding (NOT in signals, cannot affect confidence).
- Deterministic confidence model: source diversity via SIGNAL_TO_SOURCE
mapping (deduplicates cii+cii_delta, theater+indicators) + calibration
agreement from prediction market drift. Floor 0.2, ceiling 1.0.
- Output validation: rejects scenarios without signal references.
- Truncated JSON repair for small model output.
- Structured JSON logging for LLM calls.
- Redis cache for LLM scenarios (1h TTL).
- 23 new tests (70 total), all passing.
- Live-tested: OpenRouter gemini-2.5-flash produces evidence-grounded
scenario narratives from real WorldMonitor data.
* feat(forecast): Phase 3 multi-perspective scenarios, projections, data-driven cascades
MiroFish-inspired enhancements:
- Multi-perspective LLM analysis: top-2 predictions get strategic,
regional, and contrarian viewpoints via combined LLM call
- Probability projections: domain-specific decay curves (h24/d7/d30)
anchored to timeHorizon so probability equals projections[timeHorizon]
- Data-driven cascade rules: moved from hardcoded array to JSON config
(scripts/data/cascade-rules.json) with schema validation, named
predicate evaluators, unknown key rejection, and fallback to defaults
- 4 new cascade paths: infrastructure->supply_chain, infrastructure->market
(both requiresSeverity:total), conflict->political, political->market
- Proto: added Perspectives and Projections messages to Forecast
- ForecastPanel: renders projections row and conditional perspectives toggle
- 89 tests (19 new), all passing
- Live-tested: OpenRouter produces perspectives from real data
* feat(forecast): Phase 4 data utilization + entity graph
Fixes data gaps that prevented 4 of 6 detectors from firing:
- Input normalizers: chokepoint v4 shape + GPS hexes-to-zones mapping
- Chokepoint warm-ping (production-only, requires WM_API_BASE_URL)
- Lowered CII conflict threshold from 70 to 60, gated on level=high|critical
4 new standalone detectors:
- UCDP conflict zones (10+ events per country)
- Cyber threat concentration (5+ threats per country)
- GPS jamming in maritime shipping zones (5 regions)
- Prediction markets as signals (60-90% probability markets)
Entity-relationship graph (file-based, 38 nodes):
- Countries, theaters, commodities, chokepoints, alliances
- Alias table resolves both ISO codes and display names
- Graph cascade discovery links predictions across entities
Result: 51 predictions (up from 1-2), spanning conflict, infrastructure,
and supply chain domains. 112 tests, all passing.
* fix(forecast): redis cache format, signal source mapping, type safety
Fresh-eyes audit fixes:
- BUG: redisSet used wrong Upstash API format (POST body with {value,ex}
instead of command array ['SET',key,value,'EX',ttl]). LLM cache writes
were silently failing, causing fresh LLM calls every run.
- BUG: prediction_market signal type missing from SIGNAL_TO_SOURCE,
inflating confidence for market-derived predictions.
- CLEANUP: Remove unnecessary (f as any) casts in ForecastPanel since
generated Forecast type already has projections/perspectives fields.
- CLEANUP: Bump health maxStaleMin from 60 to 90 to avoid false STALE
alerts when LLM calls add latency to seed runs.
* feat(forecast): headline-entity matching with news corroboration signals
Uses entity graph aliases to match headlines to predictions by
country/theater (excludes commodity/infrastructure nodes to prevent
false positives). Predictions with matching headlines get a
news_corroboration signal visible in the panel.
Also fixes buildUserPrompt to merge unique headlines from ALL
predictions in the LLM batch (was only reading preds[0].newsContext).
Live-tested: 13 of 51 predictions now have corroborating headlines
(Iran, Israel, Syria, Ukraine, etc). 116 tests, all passing.
* feat(forecast): add country-codes.json for headline-entity matching
56 countries with ISO codes, full names, and scoring keywords (extracted
from src/config/countries.ts + UCDP-relevant additions). Used by
attachNewsContext for richer headline matching via getSearchTermsForRegion
which combines country-codes + entity graph + keyword aliases.
14/57 predictions now have news corroboration (limited by headline
coverage, not matching quality: only 8 headlines currently available).
* feat(forecast): read 300 headlines from news digest instead of 8
Read news:digest:v1:full:en (300 headlines across 16 categories) instead
of just news:insights:v1 topStories (8 headlines). Fallback to topStories
if digest is unavailable.
Result: news corroboration jumped from 25% to 64% (38/59 predictions).
* fix(forecast): handle parenthetical country names in headline matching
Strip suffixes like '(Zaire)', '(Burma)', '(Soviet Union)' from UCDP
region names before matching against country-codes.json. Also use
includes() for reverse name lookup to catch partial matches.
Corroboration: 64% -> 69% (41/59). Remaining 18 unmatched are countries
with no current English-language news coverage.
* fix(forecast): cache validated LLM output, add digest test, log cache errors
Fresh-eyes audit fixes:
- Combined LLM cache now stores only validated items (was caching raw
unvalidated output, serving potentially invalid scenarios on cache hit)
- redisSet logs warnings on failure (was silently swallowing all errors)
- Added digest-based test for attachNewsContext (primary path was untested)
- Fixed test arity: attachNewsContext(preds, news, digest) with 3 params
* fix(forecast): remove dead confidenceFromSources, reduce warm-ping timeout
- P2: Remove confidenceFromSources (dead code, computeConfidence overwrites
all initial confidence values). Inline the formula in original detectors.
- P3: Reduce warm-ping timeout from 30s to 15s (non-critical step)
- P3: Add trial status comment on forecast panel config
* fix(forecast): resolve ISO codes to country names, fix market detector, safe pre-push
P1 fixes from code review:
- CII ISO codes (IL, IR) now resolved to full country names (Israel, Iran)
via country-codes.json. Prevents substring false positives (IL matching
Chile) in event correlation. Uses word-boundary regex for matching.
- Market detector CII-to-theater mapping now uses entity graph traversal
instead of broken theater-name substring matching. Iran correctly maps
to Middle East theater via graph links.
- Pre-push hook no longer runs destructive git checkout on proto freshness
failure. Reports mismatch and exits without modifying worktree.
* feat(forecast): add structured scenario pipeline and trace export
* fix(forecast): hydrate bootstrap and trim generated drift
* fix(forecast): keep required supply-chain contract updates
* fix(ci): add forecasts to cache-keys registry and regenerate proto
Add forecasts entry to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in
cache-keys.ts to match api/bootstrap.js. Regenerate SupplyChain proto
to fix duplicate TransitDayCount and add riskSummary/riskReportAction.
|
||
|
Elie Habib
|
45f5e5a457 |
feat(forecast): AI Forecasts prediction module (#1579)
* feat(forecast): add AI Forecasts prediction module (Pro-tier)
MiroFish-inspired prediction engine that generates structured forecasts
across 6 domains (conflict, market, supply chain, political, military,
infrastructure) using existing WorldMonitor data streams.
- Proto definitions for ForecastService with GetForecasts RPC
- Dedicated seed script (seed-forecasts.mjs) with 6 domain detectors,
cross-domain cascade resolver, prediction market calibration, and
trend detection via prior snapshot comparison
- Premium-gated RPC handler (PREMIUM_RPC_PATHS enforcement)
- Lazy-loaded ForecastPanel with domain filters, probability bars,
trend arrows, signal evidence, and cascade links
- Health monitoring integration (seed-meta freshness tracking)
- Refresh scheduler with API key guard
* test(forecast): add 47 unit tests for forecast detectors and utilities
Covers forecastId, normalize, resolveCascades, calibrateWithMarkets,
computeTrends, and smoke tests for all 6 domain detectors. Exports
testable functions from seed script with direct-run guard.
* fix(forecast): domain mismatch 'infra' vs 'infrastructure', add panel category
- Seed script used 'infra' but ForecastPanel filtered on 'infrastructure',
causing Infra tab to show zero results
- Added 'forecast' to intelligence category in PANEL_CATEGORY_MAP
* fix(forecast): move CSS to one-time injection, improve type safety
- P2: Move style block from setContent to one-time document.head injection
to prevent CSS accumulation on repeated renders
- P3: Replace +toFixed(3) with Math.round for readability in seed script
- P3: Use Forecast type instead of any[] in RPC handler filter
* fix(forecast): handle sebuf proto data shapes from Redis
Detectors now normalize CII scores from server-side proto format
(combinedScore, TREND_DIRECTION_RISING, region) to uniform shape.
Outage severity handles proto enum format (SEVERITY_LEVEL_HIGH).
Added confidence floor of 0.3 for single-source predictions.
Verified against live Redis: 2 predictions generated (Iran infra
shutdown, IL political instability).
* feat(forecast): unlock AI Forecasts on web, lock desktop only (trial)
- Remove forecast RPC from PREMIUM_RPC_PATHS (web access is free)
- Panel locked on desktop only (same as oref-sirens/telegram-intel)
- Remove API key guards from data-loader and refresh scheduler
- Web users get full access during trial period
* chore: regenerate proto types with make generate
Re-ran make generate after rebasing on main. Plugin v0.7.0 dropped
@ts-nocheck from output, added it back to all 50 generated files.
Fixed 4 type errors from proto codegen changes:
- MarketSource enum -> string union type
- TemporalAnomalyProto -> TemporalAnomaly rename
- webcam lastUpdated number -> string
* fix(forecast): use chokepoints v4 key, include ciiContribution in unrest
- P1: Switch chokepoints input from stale v2 to active v4 Redis key,
matching bootstrap.js and cache-keys.ts
- P2: Add ciiContribution to unrest component fallback chain in
normalizeCiiEntry so political detector reads the correct sebuf field
* feat(forecast): Phase 2 LLM scenario enrichment + confidence model
MiroFish-inspired enhancements:
- LLM scenario narratives via Groq/OpenRouter (narrative-only, no numeric
adjustment). Evidence-grounded prompts with mandatory signal citation
and few-shot examples from MiroFish's SECTION_SYSTEM_PROMPT_TEMPLATE.
- Top-4 predictions batched into single LLM call for cost efficiency.
- News context from newsInsights attached to all predictions for LLM
prompt grounding (NOT in signals, cannot affect confidence).
- Deterministic confidence model: source diversity via SIGNAL_TO_SOURCE
mapping (deduplicates cii+cii_delta, theater+indicators) + calibration
agreement from prediction market drift. Floor 0.2, ceiling 1.0.
- Output validation: rejects scenarios without signal references.
- Truncated JSON repair for small model output.
- Structured JSON logging for LLM calls.
- Redis cache for LLM scenarios (1h TTL).
- 23 new tests (70 total), all passing.
- Live-tested: OpenRouter gemini-2.5-flash produces evidence-grounded
scenario narratives from real WorldMonitor data.
* feat(forecast): Phase 3 multi-perspective scenarios, projections, data-driven cascades
MiroFish-inspired enhancements:
- Multi-perspective LLM analysis: top-2 predictions get strategic,
regional, and contrarian viewpoints via combined LLM call
- Probability projections: domain-specific decay curves (h24/d7/d30)
anchored to timeHorizon so probability equals projections[timeHorizon]
- Data-driven cascade rules: moved from hardcoded array to JSON config
(scripts/data/cascade-rules.json) with schema validation, named
predicate evaluators, unknown key rejection, and fallback to defaults
- 4 new cascade paths: infrastructure->supply_chain, infrastructure->market
(both requiresSeverity:total), conflict->political, political->market
- Proto: added Perspectives and Projections messages to Forecast
- ForecastPanel: renders projections row and conditional perspectives toggle
- 89 tests (19 new), all passing
- Live-tested: OpenRouter produces perspectives from real data
* feat(forecast): Phase 4 data utilization + entity graph
Fixes data gaps that prevented 4 of 6 detectors from firing:
- Input normalizers: chokepoint v4 shape + GPS hexes-to-zones mapping
- Chokepoint warm-ping (production-only, requires WM_API_BASE_URL)
- Lowered CII conflict threshold from 70 to 60, gated on level=high|critical
4 new standalone detectors:
- UCDP conflict zones (10+ events per country)
- Cyber threat concentration (5+ threats per country)
- GPS jamming in maritime shipping zones (5 regions)
- Prediction markets as signals (60-90% probability markets)
Entity-relationship graph (file-based, 38 nodes):
- Countries, theaters, commodities, chokepoints, alliances
- Alias table resolves both ISO codes and display names
- Graph cascade discovery links predictions across entities
Result: 51 predictions (up from 1-2), spanning conflict, infrastructure,
and supply chain domains. 112 tests, all passing.
* fix(forecast): redis cache format, signal source mapping, type safety
Fresh-eyes audit fixes:
- BUG: redisSet used wrong Upstash API format (POST body with {value,ex}
instead of command array ['SET',key,value,'EX',ttl]). LLM cache writes
were silently failing, causing fresh LLM calls every run.
- BUG: prediction_market signal type missing from SIGNAL_TO_SOURCE,
inflating confidence for market-derived predictions.
- CLEANUP: Remove unnecessary (f as any) casts in ForecastPanel since
generated Forecast type already has projections/perspectives fields.
- CLEANUP: Bump health maxStaleMin from 60 to 90 to avoid false STALE
alerts when LLM calls add latency to seed runs.
* feat(forecast): headline-entity matching with news corroboration signals
Uses entity graph aliases to match headlines to predictions by
country/theater (excludes commodity/infrastructure nodes to prevent
false positives). Predictions with matching headlines get a
news_corroboration signal visible in the panel.
Also fixes buildUserPrompt to merge unique headlines from ALL
predictions in the LLM batch (was only reading preds[0].newsContext).
Live-tested: 13 of 51 predictions now have corroborating headlines
(Iran, Israel, Syria, Ukraine, etc). 116 tests, all passing.
* feat(forecast): add country-codes.json for headline-entity matching
56 countries with ISO codes, full names, and scoring keywords (extracted
from src/config/countries.ts + UCDP-relevant additions). Used by
attachNewsContext for richer headline matching via getSearchTermsForRegion
which combines country-codes + entity graph + keyword aliases.
14/57 predictions now have news corroboration (limited by headline
coverage, not matching quality: only 8 headlines currently available).
* feat(forecast): read 300 headlines from news digest instead of 8
Read news:digest:v1:full:en (300 headlines across 16 categories) instead
of just news:insights:v1 topStories (8 headlines). Fallback to topStories
if digest is unavailable.
Result: news corroboration jumped from 25% to 64% (38/59 predictions).
* fix(forecast): handle parenthetical country names in headline matching
Strip suffixes like '(Zaire)', '(Burma)', '(Soviet Union)' from UCDP
region names before matching against country-codes.json. Also use
includes() for reverse name lookup to catch partial matches.
Corroboration: 64% -> 69% (41/59). Remaining 18 unmatched are countries
with no current English-language news coverage.
* fix(forecast): cache validated LLM output, add digest test, log cache errors
Fresh-eyes audit fixes:
- Combined LLM cache now stores only validated items (was caching raw
unvalidated output, serving potentially invalid scenarios on cache hit)
- redisSet logs warnings on failure (was silently swallowing all errors)
- Added digest-based test for attachNewsContext (primary path was untested)
- Fixed test arity: attachNewsContext(preds, news, digest) with 3 params
* fix(forecast): remove dead confidenceFromSources, reduce warm-ping timeout
- P2: Remove confidenceFromSources (dead code, computeConfidence overwrites
all initial confidence values). Inline the formula in original detectors.
- P3: Reduce warm-ping timeout from 30s to 15s (non-critical step)
- P3: Add trial status comment on forecast panel config
* fix(forecast): resolve ISO codes to country names, fix market detector, safe pre-push
P1 fixes from code review:
- CII ISO codes (IL, IR) now resolved to full country names (Israel, Iran)
via country-codes.json. Prevents substring false positives (IL matching
Chile) in event correlation. Uses word-boundary regex for matching.
- Market detector CII-to-theater mapping now uses entity graph traversal
instead of broken theater-name substring matching. Iran correctly maps
to Middle East theater via graph links.
- Pre-push hook no longer runs destructive git checkout on proto freshness
failure. Reports mismatch and exits without modifying worktree.
|
||
|
Elie Habib
|
cad24d8817 |
fix(predictions): move prediction-tags.json into scripts/data/ for Railway (#1518)
Railway deploys with rootDirectory=scripts/, so ../shared/ resolves to /shared/ which doesn't exist. Move the canonical file to scripts/data/ and update all four consumers. |
||
|
Elie Habib
|
1262e79b38 |
fix: remove data files from git tracking (#1114)
* data(iran): import 100 events + add 27 geocoded locations Import latest LiveUAMap events (March 5-6, 2026) covering US-Israeli strikes on Iran, Iranian retaliatory attacks on Gulf states and Israel, and regional diplomatic developments. New LOCATION_COORDS: Paveh, Parchin, Rasht, Khorramabad, Damavand, Parand, Javanrud, Basra, Karbala, Nakhchivan, Koya, Elad, Juffair, Hodeidah, Sana'a, Ma'ameer, Pakdasht, Alborz, Khor al-Zubair, Prince Sultan AB, Ben Gurion, Tel Nof, Azerbaijan, Yemen. * fix: remove seed script and event data from git tracking These files are already in .gitignore but were committed previously. Event data belongs in Redis only, not in the repo. |
||
|
Elie Habib
|
309eeea6fc |
feat: add 24 geocoder locations and auto-rotate CDN cache buster for Iran events (#1047)
- Add missing locations to seed script: Bukan, Saqqez, Sardasht, Marivan, Baneh, Sulaymaniyah, Riffa, Al-Kharj, Al-Jawf, Mehrabad, Mahallati, Tehransar, Borujerdi, Incirlik, Aqaba, Ashkelon, Jerusalem, Sri Lanka, Tabriz, Yazd, Hatay, Najaf, Hazmieh - Replace hardcoded ?_v=9 cache-bust with time-based rotation (2-min buckets) so CDN cache refreshes automatically after Redis imports - Update iran-events-latest.json with Mar 5 import data (100 events) |
||
|
Elie Habib
|
07aca2c396 |
feat(conflict): seed 100 Iran events + add 20 geocoding locations (#899)
- Import latest LiveUAMap Iran events (100 events, March 2026) - Add missing LOCATION_COORDS: Khomein, Markazi, Kashan, Qom, Ahvaz, Dezful, Khorramshahr, Ilam, Laar, Kermanshah, Fujairah, Hermel, Amman, Jeddah, Dhahran, Al Minhad, Galilee, Evin - Bump cache-bust param _v=8 → _v=9 to bypass stale CDN/IndexedDB |
||
|
Elie Habib
|
9c5ad83651 |
feat(conflict): seed 100 Iran war events and expand geocoder (#792)
Add 26 new locations to seed script geocoder (Beersheba, Akrotiri, Bandar Abbas, Kerman, Natanz, Beirut, Baalbek, Ras Tanura, Ras Laffan, Quneitra, etc.) and bump CDN cache-bust _v=7 → _v=8. |
||
|
Elie Habib
|
b279e881a2 |
feat(rag): worker-side vector store with opt-in Headline Memory (#675)
* Add Security Advisories panel with government travel alerts (#460) * feat: add Security Advisories panel with government travel advisory feeds Adds a new panel aggregating travel/security advisories from official government foreign affairs agencies (US State Dept, AU DFAT Smartraveller, UK FCDO, NZ MFAT). Advisories are categorized by severity level (Do Not Travel, Reconsider, Caution, Normal) with filter tabs by source country. Includes summary counts, auto-refresh, and persistent caching via the existing data-freshness system. * chore: update package-lock.json * fix: event delegation, localization, and cleanup for SecurityAdvisories panel P1 fixes: - Use event delegation on this.content (bound once in constructor) instead of direct addEventListener after each innerHTML replacement — prevents memory leaks and stale listener issues on re-render - Use setContent() consistently instead of mixing with this.content.innerHTML - Add securityAdvisories translations to all 16 non-English locale files (panels name, component strings, common.all key) - Revert unrelated package-lock.json version bump P2 fixes: - Deduplicate loadSecurityAdvisories — loadIntelligenceData now calls the shared method instead of inlining duplicate fetch+set logic - Add Accept header to fetch calls for better content negotiation * feat(advisories): add US embassy alerts, CDC, ECDC, and WHO health feeds Adds 21 new advisory RSS feeds: - 13 US Embassy per-country security alerts (TH, AE, DE, UA, MX, IN, PK, CO, PL, BD, IT, DO, MM) - CDC Travel Notices - 5 ECDC feeds (epidemiological, threats, risk assessments, avian flu, publications) - 2 WHO feeds (global news, Africa emergencies) Panel gains a Health filter tab for CDC/ECDC/WHO sources. All new domains added to RSS proxy allowlist. i18n "health" key added across all 17 locales. * feat(cache): add negative-result caching to cachedFetchJson (#466) When upstream APIs return errors (HTTP 403, 429, timeout), fetchers return null. Previously null results were not cached, causing repeated request storms against broken APIs every refresh cycle. Now caches a sentinel value ('__WM_NEG__') with a short 2-minute TTL on null results. Subsequent requests within that window get null immediately without hitting upstream. Thrown errors (transient) skip sentinel caching and retry immediately. Also filters sentinels from getCachedJsonBatch pipeline reads and fixes theater posture coalescing test (expected 2 OpenSky fetches for 2 theater query regions, not 1). * feat: convert 52 API endpoints from POST to GET for edge caching (#468) * feat: convert 52 API endpoints from POST to GET for edge caching Convert all cacheable sebuf RPC endpoints to HTTP GET with query/path parameters, enabling CDN edge caching to reduce costs. Flatten nested request types (TimeRange, PaginationRequest, BoundingBox) into scalar query params. Add path params for resource lookups (GetFredSeries, GetHumanitarianSummary, GetCountryStockIndex, GetCountryIntelBrief, GetAircraftDetails). Rewrite router with hybrid static/dynamic matching for path param support. Kept as POST: SummarizeArticle, ClassifyEvent, RecordBaselineSnapshot, GetAircraftDetailsBatch, RegisterInterest. Generated with sebuf v0.9.0 (protoc-gen-ts-client, protoc-gen-ts-server). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add rate_limited field to market response protos The rateLimited field was hand-patched into generated files on main but never declared in the proto definitions. Regenerating wiped it out, breaking the build. Now properly defined in both ListEtfFlowsResponse and ListMarketQuotesResponse protos. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove accidentally committed .planning files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat: add Cloudflare edge caching infrastructure for api.worldmonitor.app (#471) Route web production RPC traffic through api.worldmonitor.app via fetch interceptor (installWebApiRedirect). Add default Cache-Control headers (s-maxage=300, stale-while-revalidate=60) on GET 200 responses, with no-store override for real-time endpoints (vessel snapshot). Update CORS to allow GET method. Skip Vercel bot middleware for API subdomain using hostname check (non-spoofable, replacing CF-Ray header approach). Update desktop cloud fallback to route through api.worldmonitor.app. * fix(beta): eagerly load T5-small model when beta mode is enabled BETA_MODE now couples the badge AND model loading — the summarization-beta model starts loading on startup instead of waiting for the first summarization call. * fix: move 5 path-param endpoints to query params for Vercel routing (#472) Vercel's `api/[domain]/v1/[rpc].ts` captures one dynamic segment. Path params like `/get-humanitarian-summary/SA` add an extra segment that has no matching route file, causing 404 on both OPTIONS preflight and direct requests. These endpoints were broken in production. Changes: - Remove `{param}` from 5 service.proto HTTP paths - Add `(sebuf.http.query)` annotations to request message fields - Update generated client/server code to use URLSearchParams - Update OpenAPI specs (YAML + JSON) to declare query params - Add early-return guards in 4 handlers for missing required params - Add happy.worldmonitor.app to runtime.ts redirect hosts Affected endpoints: - GET /api/conflict/v1/get-humanitarian-summary?country_code=SA - GET /api/economic/v1/get-fred-series?series_id=T10Y2Y&limit=120 - GET /api/market/v1/get-country-stock-index?country_code=US - GET /api/intelligence/v1/get-country-intel-brief?country_code=US - GET /api/military/v1/get-aircraft-details?icao24=a12345 * fix(security-advisories): route feeds through RSS proxy to avoid CORS blocks (#473) - Advisory feeds were fetched directly from the browser, hitting CORS on all 21 feeds (US State Dept, AU Smartraveller, US Embassies, ECDC, CDC, WHO). Route through /api/rss-proxy on web, keep proxyUrl for desktop. - Fix double slash in ECDC Avian Influenza URL (323//feed → 323/feed) - Add feeds.news24.com to RSS proxy allowlist (was returning 403) * feat(cache): tiered edge Cache-Control aligned to upstream TTLs (#474) * fix: move 5 path-param endpoints to query params for Vercel routing Vercel's `api/[domain]/v1/[rpc].ts` captures one dynamic segment. Path params like `/get-humanitarian-summary/SA` add an extra segment that has no matching route file, causing 404 on both OPTIONS preflight and direct requests. These endpoints were broken in production. Changes: - Remove `{param}` from 5 service.proto HTTP paths - Add `(sebuf.http.query)` annotations to request message fields - Update generated client/server code to use URLSearchParams - Update OpenAPI specs (YAML + JSON) to declare query params - Add early-return guards in 4 handlers for missing required params - Add happy.worldmonitor.app to runtime.ts redirect hosts Affected endpoints: - GET /api/conflict/v1/get-humanitarian-summary?country_code=SA - GET /api/economic/v1/get-fred-series?series_id=T10Y2Y&limit=120 - GET /api/market/v1/get-country-stock-index?country_code=US - GET /api/intelligence/v1/get-country-intel-brief?country_code=US - GET /api/military/v1/get-aircraft-details?icao24=a12345 * feat(cache): add tiered edge Cache-Control aligned to upstream TTLs Replace flat s-maxage=300 with 5 tiers (fast/medium/slow/static/no-store) mapped per-endpoint to respect upstream Redis TTLs. Adds stale-if-error resilience headers and X-No-Cache plumbing for future degraded responses. X-Cache-Tier debug header gated behind ?_debug query param. * fix(tech): use rss() for CISA feed, drop build from pre-push hook (#475) - CISA Advisories used dead rss.worldmonitor.app domain (404), switch to rss() helper - Remove Vite build from pre-push hook (tsc already catches errors) * fix(desktop): enable click-to-play YouTube embeds + CISA feed fixes (#476) * fix(tech): use rss() for CISA feed, drop build from pre-push hook - CISA Advisories used dead rss.worldmonitor.app domain (404), switch to rss() helper - Remove Vite build from pre-push hook (tsc already catches errors) * fix(desktop): enable click-to-play for YouTube embeds in WKWebView WKWebView blocks programmatic autoplay in cross-origin iframes regardless of allow attributes, Permissions-Policy, mute-first retries, or secure context. Documented all 10 approaches tested in docs/internal/. Changes: - Switch sidecar embed origin from 127.0.0.1 to localhost (secure context) - Add MutationObserver + retry chain as best-effort autoplay attempts - Use postMessage('*') to fix tauri://localhost cross-origin messaging - Make sidecar play overlay non-interactive (pointer-events:none) - Fix .webcam-iframe pointer-events:none blocking clicks in grid view - Add expand button to grid cells for switching to single view on desktop - Add http://localhost:* to CSP frame-src in index.html and tauri.conf.json * fix(gateway): convert stale POST requests to GET for backwards compat (#477) Stale cached client bundles still send POST to endpoints converted to GET in PR #468, causing 404s. The gateway now parses the POST JSON body into query params and retries the match as GET. * feat(proxy): add Cloudflare edge caching for proxy.worldmonitor.app (#478) Add CDN-Cache-Control headers to all proxy endpoints so Cloudflare can cache responses at the edge independently of browser Cache-Control: - RSS: 600s edge + stale-while-revalidate=300 (browser: 300s) - UCDP: 3600s edge (matches browser) - OpenSky: 15s edge (browser: 30s) for fresher flight data - WorldBank: 1800s/86400s edge (matches browser) - Polymarket: 120s edge (matches browser) - Telegram: 10s edge (matches browser) - AIS snapshot: 2s edge (matches browser) Also fixes: - Vary header merging: sendCompressed/sendPreGzipped now merge existing Vary: Origin instead of overwriting, preventing cross-origin cache poisoning at the edge - Stale fallback responses (OpenSky, WorldBank, Polymarket, RSS) now set Cache-Control: no-store + CDN-Cache-Control: no-store to prevent edge caching of degraded responses - All no-cache branches get CDN-Cache-Control: no-store - /opensky-reset gets no-store (state-changing endpoint) * fix(sentry): add noise filters for 4 unresolved issues (#479) - Tighten AbortError filter to match "AbortError: The operation was aborted" - Filter "The user aborted a request" (normal navigation cancellation) - Filter UltraViewer service worker injection errors (/uv/service/) - Filter Huawei WebView __isInQueue__ injection * feat: configurable VITE_WS_API_URL + harden POST→GET shim (#480) * fix(gateway): harden POST→GET shim with scalar guard and size limit - Only convert string/number/boolean values to query params (skip objects, nested arrays, __proto__ etc.) to prevent prototype pollution vectors - Skip body parsing for Content-Length > 1MB to avoid memory pressure * feat: make API base URL configurable via VITE_WS_API_URL Replace hardcoded api.worldmonitor.app with VITE_WS_API_URL env var. When empty, installWebApiRedirect() is skipped entirely — relative /api/* calls stay on the same domain (local installs). When set, browser fetch is redirected to that URL. Also adds VITE_WS_API_URL and VITE_WS_RELAY_URL hostnames to APP_HOSTS allowlist dynamically. * fix(analytics): use greedy regex in PostHog ingest rewrites (#481) Vercel's :path* wildcard doesn't match trailing slashes that PostHog SDK appends (e.g. /ingest/s/?compression=...), causing 404s. Switch to :path(.*) which matches all path segments including trailing slashes. Ref: PostHog/posthog#17596 * perf(proxy): increase AIS snapshot edge TTL from 2s to 10s (#482) With 20k requests/30min (60% of proxy traffic) and per-PoP caching, a 2s edge TTL expires before the next request from the same PoP arrives, resulting in near-zero cache hits. 10s allows same-PoP dedup while keeping browser TTL at 2s for fresh vessel positions. * fix(markets): commodities panel showing stocks instead of commodities (#483) The shared circuit breaker (cacheTtlMs: 0) cached the stocks response, then the stale-while-revalidate path returned that cached stocks data for the subsequent commodities fetch. Skip SWR when caching is disabled. * feat(gateway): complete edge cache tier coverage + degraded-response policy (#484) - Add 11 missing GET routes to RPC_CACHE_TIER map (8 slow, 3 medium) - Add response-headers side-channel (WeakMap) so handlers can signal X-No-Cache without codegen changes; wire into military-flights and positive-geo-events handlers on upstream failure - Add env-controlled per-endpoint tier override (CACHE_TIER_OVERRIDE_*) for incident response rollback - Add VITE_WS_API_URL hostname allowlist (*.worldmonitor.app + localhost) - Fix fetch.bind(globalThis) in positive-events-geo.ts (deferred lambda) - Add CI test asserting every generated GET route has an explicit cache tier entry (prevents silent default-tier drift) * chore: bump version to 2.5.20 + changelog Covers PRs #452–#484: Cloudflare edge caching, commodities SWR fix, security advisories panel, settings redesign, 52 POST→GET migrations. * fix(rss): remove stale indianewsnetwork.com from proxy allowlist (#486) Feed has no <pubDate> fields and latest content is from April 2022. Not referenced in any feed config — only in the proxy domain allowlist. * feat(i18n): add Korean (한국어) localization (#487) - Add ko.json with all 1606 translation keys matching en.json structure - Register 'ko' in SUPPORTED_LANGUAGES, LANGUAGES display array, and locale map - Korean appears as 🇰🇷 한국어 in the language dropdown * feat: add Polish tv livestreams (#488) * feat(rss): add Axios (api.axios.com/feed) as US news source (#494) Add api.axios.com to proxy allowlist and CSP connect-src, register Axios feed under US category as Tier 2 mainstream source. * perf: bootstrap endpoint + polling optimization (#495) * perf: bootstrap endpoint + polling optimization (phases 3-4) Replace 15+ individual RPC calls on startup with a single /api/bootstrap batch call that fetches pre-cached data from Redis. Consolidate 6 panel setInterval timers into the central RefreshScheduler for hidden-tab awareness (10x multiplier) and adaptive backoff (up to 4x for unchanged data). Convert IntelligenceGapBadge from 10s polling to event-driven updates with 60s safety fallback. * fix(bootstrap): inline Redis + cache keys in edge function Vercel Edge Functions cannot resolve cross-directory TypeScript imports from server/_shared/. Inline getCachedJsonBatch and BOOTSTRAP_CACHE_KEYS directly in api/bootstrap.js. Add sync test to ensure inlined keys stay in sync with the canonical server/_shared/cache-keys.ts registry. * test: add Edge Function module isolation guard for all api/*.js files Prevents any Edge Function from importing from ../server/ or ../src/ which breaks Vercel builds. Scans all 12 non-helper Edge Functions. * fix(bootstrap): read unprefixed cache keys on all environments Preview deploys set VERCEL_ENV=preview which caused getKeyPrefix() to prefix Redis keys with preview:<sha>:, but handlers only write to unprefixed keys on production. Bootstrap is a read-only consumer of production cache — always read unprefixed keys. * fix(bootstrap): wire sectors hydration + add coverage guard - Wire getHydratedData('sectors') in data-loader to skip Yahoo Finance fetch when bootstrap provides sector data - Add test ensuring every bootstrap key has a getHydratedData consumer — prevents adding keys without wiring them * fix(server): resolve 25 TypeScript errors + add server typecheck to CI - _shared.ts: remove unused `delay` variable - list-etf-flows.ts: add missing `rateLimited` field to 3 return literals - list-market-quotes.ts: add missing `rateLimited` field to 4 return literals - get-cable-health.ts: add non-null assertions for regex groups and array access - list-positive-geo-events.ts: add non-null assertion for array index - get-chokepoint-status.ts: add required fields to request objects - CI: run `typecheck:api` (tsconfig.api.json) alongside `typecheck` to catch server/ TS errors before merge * feat(military): server-side military bases 125K + rate limiting (#496) * feat(military): server-side military bases with 125K entries + rate limiting (#485) Migrate military bases from 224 static client-side entries to 125,380 server-side entries stored in Redis GEO sorted sets, served via bbox-filtered GEOSEARCH endpoint with server-side clustering. Data pipeline: - Pizzint/Polyglobe: 79,156 entries (Supabase extraction) - OpenStreetMap: 45,185 entries - MIRTA: 821 entries - Curated strategic: 218 entries - 277 proximity duplicates removed Server: - ListMilitaryBases RPC with GEOSEARCH + HMGET + tier/filter/clustering - Antimeridian handling (split bbox queries) - Blue-green Redis deployment with atomic version pointer switch - geoSearchByBox() + getHashFieldsBatch() helpers in redis.ts Security: - @upstash/ratelimit: 60 req/min sliding window per IP - IP spoofing fix: prioritize x-real-ip (Vercel-injected) over x-forwarded-for - Require API key for non-browser requests (blocks unauthenticated curl/scripts) - Input validation: allowlisted types/kinds, regex country, clamped bbox/zoom Frontend: - Viewport-driven loading with bbox quantization + debounce - Server-side grid clustering at low zoom levels - Enriched popup with kind, category badges (airforce/naval/nuclear/space) - Static 224 bases kept as search fallback + initial render * fix(military): fallback to production Redis keys in preview deployments Preview deployments prefix Redis keys with `preview:{sha}:` but military bases data is seeded to unprefixed (production) keys. When the prefixed `military:bases:active` key is missing, fall back to the unprefixed key and use raw (unprefixed) keys for geo/meta lookups. * fix: remove unused 'remaining' destructure in rate-limit (TS6133) * ci: add typecheck:api to pre-push hook to catch server-side TS errors * debug(military): add X-Bases-Debug response header for preview diagnostics * fix(bases): trigger initial server fetch on map load fetchServerBases() was only called on moveend — if the user never panned/zoomed, the API was never called and only the 224 static fallback bases showed. * perf(military): debounce base fetches + upgrade edge cache to static tier (#497) - Add 300ms debounce on moveend to prevent rapid pan flooding - Fixes stale-bbox bug where pendingFetch returns old viewport data - Upgrade edge cache tier from medium (5min) to static (1hr) — bases are static infrastructure, aligned with server-side cachedFetchJson TTL - Keep error logging in catch blocks for production diagnostics * fix(cyber): make GeoIP centroid fallback jitter deterministic (#498) Replace Math.random() jitter with DJB2 hash seeded by the threat indicator (IP/URL), so the same threat always maps to the same coordinates across requests while different threats from the same country still spread out. Closes #203 Co-authored-by: Chris Chen <fuleinist@users.noreply.github.com> * fix: use cross-env for Windows-compatible npm scripts (#499) Replace direct `VAR=value command` syntax with cross-env/cross-env-shell so dev, build, test, and desktop scripts work on Windows PowerShell/CMD. Co-authored-by: facusturla <facusturla@users.noreply.github.com> * feat(live-news): add CBC News to optional North America channels (#502) YouTube handle @CBCNews with fallback video ID 5vfaDsMhCF4. * fix(bootstrap): harden hydration cache + polling review fixes (#504) - Filter null/undefined values before storing in hydration cache to prevent future consumers using !== undefined from misinterpreting null as valid data - Debounce wm:intelligence-updated event handler via requestAnimationFrame to coalesce rapid alert generation into a single render pass - Include alert IDs in StrategicRiskPanel change fingerprint so content changes are detected even when alert count stays the same - Replace JSON.stringify change detection in ServiceStatusPanel with lightweight name:status fingerprint - Document max effective refresh interval (40x base) in scheduler * fix(geo): tokenization-based keyword matching to prevent false positives (#503) * fix(geo): tokenization-based keyword matching to prevent false positives Replace String.includes() with tokenization-based Set.has() matching across the geo-tagging pipeline. Prevents false positives like "assad" matching inside "ambassador" and "hts" matching inside "rights". - Add src/utils/keyword-match.ts as single source of truth - Decompose possessives/hyphens ("Assad's" → includes "assad") - Support multi-word phrase matching ("white house" as contiguous) - Remove false-positive-prone DC keywords ('house', 'us ') - Update 9 consumer files across geo-hub, map, CII, and asset systems - Add 44 tests covering false positives, true positives, edge cases Co-authored-by: karim <mirakijka@gmail.com> Fixes #324 * fix(geo): add inflection suffix matching + fix test imports Address code review feedback: P1a: Add suffix-aware matching for plurals and demonyms so existing keyword lists don't regress (houthi→houthis, ukraine→ukrainian, iran→iranian, israel→israeli, russia→russian, taiwan→taiwanese). Uses curated suffix list + e-dropping rule to avoid false positives. P1b: Expand conflictTopics arrays in DeckGLMap and Map with demonym forms so "Iranian senate..." correctly registers as conflict topic. P2: Replace inline test functions with real module import via tsx. Tests now exercise the production keyword-match.ts directly. * fix: wire geo-keyword tests into test:data command The .mts test file wasn't covered by `node --test tests/*.test.mjs`. Add `npx tsx --test tests/*.test.mts` so test:data runs both suites. * fix: cross-platform test:data + pin tsx in devDependencies - Use tsx as test runner for both .mjs and .mts (single invocation) - Removes ; separator which breaks on Windows cmd.exe - Add tsx to devDependencies so it works in offline/CI environments * fix(geo): multi-word demonym matching + short-keyword suffix guard - Add wordMatches() for suffix-aware phrase matching so "South Korean" matches keyword "south korea" and "North Korean" matches "north korea" - Add MIN_SUFFIX_KEYWORD_LEN=4 guard so short keywords like "ai", "us", "hts" only do exact-match (prevents "ais"→"ai", "uses"→"us" false positives) - Add 5 new tests covering both fixes (58 total, all passing) * fix(geo): support plural demonyms in keyword matching Add compound suffixes (ians, eans, ans, ns, is) to handle plural demonym forms like "Iranians"→"iran", "Ukrainians"→"ukraine", "Russians"→"russia", "Israelis"→"israel". Adds 5 new tests (63 total). --------- Co-authored-by: karim <mirakijka@gmail.com> * chore: strip 61 debug console.log calls from 20 service files (#501) * chore: strip 61 debug console.log calls from services Remove development/tracing console.log statements from 20 files. These add noise to production browser consoles and increase bundle size. Preserved: all console.error (error handling) and console.warn (warnings). Preserved: debug-gated logs in runtime.ts (controlled by verbose flag). Removed: debugInjectTestEvents() from geo-convergence.ts (test-only code). Removed: logSummary()/logReport() methods that were pure console.log wrappers. * fix: remove orphaned stubs and remaining debug logs from stripped services - Remove empty logReport() method and unused startTime variable (parallel-analysis.ts) - Remove orphaned console.group/console.groupEnd pair (parallel-analysis.ts) - Remove empty logSignalSummary() export (signal-aggregator.ts) - Remove logSignalSummary import/call and 3 remaining console.logs (InsightsPanel.ts) - Remove no-op logDirectFetchBlockedOnce() and dead infrastructure (prediction/index.ts) * fix: generalize Vercel preview origin regex + include filters in bases cache key (#506) - api/_api-key.js: preview URL pattern was user-specific (-elie-), rejecting other collaborators' Vercel preview deployments. Generalized to match any worldmonitor-*.vercel.app origin. - military-bases.ts: client cache key only checked bbox/zoom, ignoring type/kind/country filters. Switching filters without panning returned stale results. Unified into single cacheKey string. * fix(prediction): filter stale/expired markets from Polymarket panel (#507) Prediction panel was showing expired markets (e.g. "Will US strike Iran on Feb 9" at 0%). Root causes: no active/archived API filters, no end_date_min param, no client-side expiry guard, and sub-market selection picking highest volume before filtering expired ones. - Add active=true, archived=false, end_date_min API params to all 3 Gamma API call sites (events, markets, probe) - Pre-filter sub-markets by closed/expired BEFORE volume selection in both fetchPredictions() and fetchCountryMarkets() - Add defense-in-depth isExpired() client-side filter on final results - Propagate endDate through all market object paths including sebuf fallback - Show expiry date in PredictionPanel UI with new .prediction-meta layout - Add "closes" i18n key to all 18 locale files - Add endDate to server handler GammaMarket/GammaEvent interfaces and map to proto closesAt field * fix(relay): guard proxy handlers against ERR_HTTP_HEADERS_SENT crash (#509) Polymarket and World Bank proxy handlers had unguarded res.writeHead() calls in error/timeout callbacks that race with the response callback. When upstream partially responds then times out, both paths write headers → process crash. Replace 5 raw writeHead+end calls with safeEnd() which checks res.headersSent before writing. * feat(breaking-news): add active alert banner with audio for critical/high RSS items (#508) RSS items classified as critical/high threat now trigger a full-width breaking news banner with audio alert, auto-dismiss (60s/30s by severity), visibility-aware timer pause, dedup, and a toggle in the Intelligence Findings dropdown. * fix(sentry): filter Android OEM WebView bridge injection errors (#510) Add ignoreErrors pattern for LIDNotifyId, onWebViewAppeared, and onGetWiFiBSSID — native bridge functions injected by Lenovo/Huawei device SDKs into Chrome Mobile WebView. No stack frames in our code. * chore: add validated telegram channels list (global + ME + Iran + cyber) (#249) * feat(conflict): add Iran Attacks map layer + strip debug logs (#511) * chore: strip 61 debug console.log calls from services Remove development/tracing console.log statements from 20 files. These add noise to production browser consoles and increase bundle size. Preserved: all console.error (error handling) and console.warn (warnings). Preserved: debug-gated logs in runtime.ts (controlled by verbose flag). Removed: debugInjectTestEvents() from geo-convergence.ts (test-only code). Removed: logSummary()/logReport() methods that were pure console.log wrappers. * fix: remove orphaned stubs and remaining debug logs from stripped services - Remove empty logReport() method and unused startTime variable (parallel-analysis.ts) - Remove orphaned console.group/console.groupEnd pair (parallel-analysis.ts) - Remove empty logSignalSummary() export (signal-aggregator.ts) - Remove logSignalSummary import/call and 3 remaining console.logs (InsightsPanel.ts) - Remove no-op logDirectFetchBlockedOnce() and dead infrastructure (prediction/index.ts) * feat(conflict): add Iran Attacks map layer Adds a new Iran-focused conflict events layer that aggregates real-time events, geocodes via 40-city lookup table, caches 15min in Redis, and renders as a toggleable DeckGL ScatterplotLayer with severity coloring. - New proto + codegen for ListIranEvents RPC - Server handler with HTML parsing, city geocoding, category mapping - Frontend service with circuit breaker - DeckGL ScatterplotLayer with severity-based color/size - MapPopup with sanitized source links - iranAttacks toggle across all variants, harnesses, and URL state * fix: resolve bootstrap 401 and 429 rate limiting on page init (#512) Same-origin browser requests don't send Origin header (per CORS spec), causing validateApiKey to reject them. Extract origin from Referer as fallback. Increase rate limit from 60 to 200 req/min to accommodate the ~50 requests fired during page initialization. * fix(relay): prevent Polymarket OOM via request deduplication (#513) Concurrent Polymarket requests for the same cache key each fired independent https.get() calls. With 12 categories × multiple clients, 740 requests piled up in 10s, all buffering response bodies → 4.1GB heap → OOM crash on Railway. Fix: in-flight promise map deduplicates concurrent requests to the same cache key. 429/error responses are negative-cached for 30s to prevent retry storms. * fix(threat-classifier): add military/conflict keyword gaps and news-to-conflict bridge (#514) Breaking news headlines like "Israel's strike on Iran" were classified as info level because the keyword classifier lacked standalone conflict phrases. Additionally, the conflict instability score depended solely on ACLED data (1-7 day lag) with no bridge from real-time breaking news. - Add 3 critical + 18 high contextual military/conflict keywords - Preserve threat classification on semantically merged clusters - Add news-derived conflict floor when ACLED/HAPI report zero signal - Upsert news events by cluster ID to prevent duplicates - Extract newsEventIndex to module-level Map for serialization safety * fix(breaking-news): let critical alerts bypass global cooldown and replace HIGH alerts (#516) Global cooldown (60s) was blocking critical alerts when a less important HIGH alert fired from an earlier RSS batch. Added priority-aware cooldown so critical alerts always break through. Banner now auto-dismisses HIGH alerts when a CRITICAL arrives. Added Iran/strikes keywords to classifier. * fix(rate-limit): increase sliding window to 300 req/min (#515) App init fires many concurrent classify-event, summarize-article, and record-baseline-snapshot calls, exhausting the 200/min limit and causing 429s. Bump to 300 as a temporary measure while client-side batching is implemented. * fix(breaking-news): fix fake pubDate fallback and filter noisy think-tank alerts (#517) Two bugs causing stale CrisisWatch article to fire as breaking alert: 1. Non-standard pubDate format ("Friday, February 27, 2026 - 12:38") failed to parse → fallback was `new Date()` (NOW) → day-old articles appeared as "just now" and passed recency gate on every fetch 2. Tier 3+ sources (think tanks) firing alerts on keyword-only matches like "War" in policy analysis titles — too noisy for breaking alerts Fix: parsePubDate() handles non-standard formats and falls back to epoch (not now). Tier 3+ sources require LLM classification to fire. * fix: make iran-events handler read-only from Redis (#518) Remove server-side LiveUAMap scraper (blocked by Cloudflare 403 on Vercel IPs). Handler now reads pre-populated Redis cache pushed from local browser scraping. Change cache tier from slow to fast to prevent CDN from serving stale empty responses for 30+ minutes. * fix(relay): Polymarket circuit breaker + concurrency limiter (OOM fix) (#519) * fix(rate-limit): increase sliding window to 300 req/min App init fires many concurrent classify-event, summarize-article, and record-baseline-snapshot calls, exhausting the 200/min limit and causing 429s. Bump to 300 as a temporary measure while client-side batching is implemented. * fix(relay): add Polymarket circuit breaker + concurrency limiter to prevent OOM Railway relay OOM crash: 280 Polymarket 429 errors in 8s, heap hit 3.7GB. Multiple unique cache keys bypassed per-key dedup, flooding upstream. - Circuit breaker: trips after 5 consecutive failures, 60s cooldown - Concurrent upstream limiter: max 3 simultaneous requests - Negative cache TTL: 30s → 60s to reduce retry frequency - Upstream slot freed on response.on('end'), not headers, preventing body buffer accumulation past the concurrency cap * fix(relay): guard against double-finalization on Polymarket timeout request.destroy() in timeout handler also fires request.on('error'), causing double decrement of polymarketActiveUpstream (counter goes negative, disabling concurrency cap) and double circuit breaker trip. Add finalized guard so decrement + failure accounting happens exactly once per request regardless of which error path fires first. * fix(threat-classifier): stagger AI classification requests to avoid Groq 429 (#520) flushBatch() fired up to 20 classifyEvent RPCs simultaneously via Promise.all, instantly hitting Groq's ~30 req/min rate limit. - Sequential execution with 2s min-gap between requests (~28 req/min) - waitForGap() enforces hard floor + jitter across batch boundaries - batchInFlight guard prevents concurrent flush loops - 429/5xx: requeue failed job (with retry cap) + remaining untouched jobs - Queue cap at 100 items with warn on overflow * fix(relay): regenerate package-lock.json with telegram dependency The lockfile was missing resolved entries for the telegram package, causing Railway to skip installation despite it being in package.json. * chore: trigger deploy to flush CDN cache for iran-events endpoint * Revert "fix(relay): regenerate package-lock.json with telegram dependency" This reverts commit |
||
|
Elie Habib
|
3d2c638a72 |
feat(military): server-side military bases 125K + rate limiting (#496)
* feat(military): server-side military bases with 125K entries + rate limiting (#485) Migrate military bases from 224 static client-side entries to 125,380 server-side entries stored in Redis GEO sorted sets, served via bbox-filtered GEOSEARCH endpoint with server-side clustering. Data pipeline: - Pizzint/Polyglobe: 79,156 entries (Supabase extraction) - OpenStreetMap: 45,185 entries - MIRTA: 821 entries - Curated strategic: 218 entries - 277 proximity duplicates removed Server: - ListMilitaryBases RPC with GEOSEARCH + HMGET + tier/filter/clustering - Antimeridian handling (split bbox queries) - Blue-green Redis deployment with atomic version pointer switch - geoSearchByBox() + getHashFieldsBatch() helpers in redis.ts Security: - @upstash/ratelimit: 60 req/min sliding window per IP - IP spoofing fix: prioritize x-real-ip (Vercel-injected) over x-forwarded-for - Require API key for non-browser requests (blocks unauthenticated curl/scripts) - Input validation: allowlisted types/kinds, regex country, clamped bbox/zoom Frontend: - Viewport-driven loading with bbox quantization + debounce - Server-side grid clustering at low zoom levels - Enriched popup with kind, category badges (airforce/naval/nuclear/space) - Static 224 bases kept as search fallback + initial render * fix(military): fallback to production Redis keys in preview deployments Preview deployments prefix Redis keys with `preview:{sha}:` but military bases data is seeded to unprefixed (production) keys. When the prefixed `military:bases:active` key is missing, fall back to the unprefixed key and use raw (unprefixed) keys for geo/meta lookups. * fix: remove unused 'remaining' destructure in rate-limit (TS6133) * ci: add typecheck:api to pre-push hook to catch server-side TS errors * debug(military): add X-Bases-Debug response header for preview diagnostics * fix(bases): trigger initial server fetch on map load fetchServerBases() was only called on moveend — if the user never panned/zoomed, the API was never called and only the 224 static fallback bases showed. |