mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-05-13 18:46:21 +02:00
5841acd06d830df14ebc8fa2f1a80eeecabfb2b0
37 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
41faeea4b3 | Scan recent commits for likely bugs (#1480) | ||
|
Elie Habib
|
651cd3d08b |
feat(desktop): sidecar cloud proxy, domain handlers, and panel fixes (#1454)
* feat(desktop): compile domain handlers + add in-memory sidecar cache The sidecar was broken for all 23 sebuf/RPC domain routes because the build script (build-sidecar-handlers.mjs) never existed on main while package.json already referenced it. This adds the missing script and an in-memory TTL+LRU cache so the sidecar doesn't need Upstash Redis. - Add scripts/build-sidecar-handlers.mjs (esbuild multi-entry, 23 domains) - Add server/_shared/sidecar-cache.ts (500 entries, 50MB max, lazy sweep) - Modify redis.ts getCachedJson/setCachedJson to use dynamic import for sidecar cache when LOCAL_API_MODE=tauri-sidecar (zero cost on Vercel Edge) - Update tauri.conf.json beforeDevCommand to compile handlers - Add gitignore pattern for compiled api/*/v1/[rpc].js * fix(desktop): gate premium panel fetches and open footer links in browser Skip oref-sirens and telegram-intel HTTP requests on desktop when WORLDMONITOR_API_KEY is not present. Use absolute URLs for footer links on desktop so the Tauri external link handler opens them in the system browser instead of navigating within the webview. * fix(desktop): cloud proxy, bootstrap timeouts, and panel data fixes - Set Origin header on cloud proxy requests (fixes 401 from API key validator) - Strip If-None-Match/If-Modified-Since headers (fixes stale 304 responses) - Add cloud-preferred routing for market/economic/news/infrastructure/research - Enable cloud fallback via LOCAL_API_CLOUD_FALLBACK env var in main.rs - Increase bootstrap timeouts on desktop (8s/12s vs 3s/5s) for sidecar proxy hops - Force per-feed RSS fallback on desktop (server digest has fewer categories) - Add finance feeds to commodity variant (client + server) - Remove desktop diagnostics from ServiceStatusPanel (show cloud statuses only) - Restore DeductionPanel CSS from PR #1162 - Deduplicate repeated sidecar error logs |
||
|
Elie Habib
|
2a7d7fc3fe |
fix: standardize brand name to "World Monitor" with space (#1463)
Replace "WorldMonitor" with "World Monitor" in all user-facing display text across blog posts, docs, layouts, structured data, footer, offline page, and X-Title headers. Technical identifiers (User-Agent strings, X-WorldMonitor-Key headers, @WorldMonitorApp handle, function names) are preserved unchanged. Also adds anchors color to Mintlify docs config to fix blue link color in dark mode. |
||
|
Elie Habib
|
0b3762f55e |
fix(cache): align CDN and client cache TTLs with freshness thresholds (#1320)
Move theaterPosture from SLOW (2h CDN) to FAST tier (20min/10min after PR #1314) so military posture data stays fresh. Increase risk scores breaker TTL to 30min to match health.js maxStaleMin, and reduce localStorage staleness from 24h to 1h to prevent stale risk data in UI. |
||
|
Elie Habib
|
595e3dbb86 |
feat: premium finance stock analysis suite (#1268)
* Add premium finance stock analysis suite * docs: link premium finance from README Add Premium Stock Analysis entry to the Finance & Markets section with a link to docs/PREMIUM_FINANCE.md. * fix: address review feedback on premium finance suite - Chunk Redis pipelines into batches of 200 (Upstash limit) - Add try-catch around cachedFetchJson in backtest handler - Log warnings on Redis pipeline HTTP failures - Include name in analyze-stock cache key to avoid collisions - Change analyze-stock and backtest-stock gateway cache to 'slow' - Add dedup guard for concurrent ledger generation - Add SerpAPI date pre-filter (tbs=qdr:d/w) - Extract sanitizeSymbol to shared module - Extract buildEmptyAnalysisResponse helper - Fix RSI to use Wilder's smoothing (matches TradingView) - Add console.warn for daily brief summarization errors - Fall back to stale data in loadStockBacktest on error - Make daily-market-brief premium on all platforms - Use word boundaries for short token headline matching - Add stock-analysis 15-min refresh interval - Stagger stock-analysis and backtest requests (200ms) - Rename signalTone to stockSignalTone |
||
|
Elie Habib
|
d6c9176213 |
Revert "fix(scripts): sync package-lock.json with h3-js dependency (#1254)" (#1256)
This reverts commit
|
||
|
Elie Habib
|
4816e27d3c |
fix(scripts): sync package-lock.json with h3-js dependency (#1254)
* Add premium stock analysis for finance variant * fix(scripts): sync package-lock.json with h3-js dependency Railway npm ci requires lock file in sync with package.json. * fix(market): narrow undefined check for TS strict null safety |
||
|
Elie Habib
|
3d2a4e4df4 |
fix(rate-limit): prioritize cf-connecting-ip over x-real-ip for correct per-user rate limiting (#1241)
With Cloudflare proxy in front of Vercel, x-real-ip contains the CF edge server IP (shared across all users behind the same edge), not the real client IP. This caused all users to share a single rate-limit bucket, triggering 429 storms on classify-event. Also adds exponential backoff (60s→120s→240s→5min cap) and queue trimming on repeated 429s to prevent client-side hammering. |
||
|
Elie Habib
|
cd7d3b7501 |
perf(baseline): move temporal baseline for news+fires to server-side (#1194)
* perf(baseline): move temporal baseline for news+fires to server-side Every browser client was calling record-baseline-snapshot (POST) + get-temporal-baseline (GET) on every data refresh from 5 call sites. With N concurrent users this created N identical writes and ~5N reads per cycle — causing 429 rate limiting and statistically biased baselines. Phase 1 moves news and satellite_fires to server-side computation: - New ListTemporalAnomalies RPC reads counts from existing Redis keys (news:insights:v1, wildfire:fires:v1), computes anomalies against v2 baselines, applies Welford update (1 sample/cycle), caches 15min - Bootstrap hydration delivers anomalies on page load (zero extra calls) - Client refreshes via RPC every 10min (1 cached call vs 5N before) - Remaining 3 types (military_flights, vessels, ais_gaps) stay client-side - Owner-guarded distributed lock prevents concurrent computation - All reads/writes use prefix-aware getCachedJson/setCachedJson Expected ~60% reduction in baseline-related Vercel invocations. * fix(temporal): per-invocation lock owner and immediate refresh on cold cache P1: When bootstrap has no temporal anomaly data (cold cache, expired snapshot, fresh deploy), fire refreshTemporalBaseline() immediately instead of waiting up to 10 minutes for the scheduled refresh. P2: Generate lockOwner per invocation via makeLockOwner() instead of once at module load. Prevents warm edge isolates from cross-releasing each other's locks when one invocation outlives the 30s TTL. * fix(temporal): use TTL-only lock instead of TOCTOU GET-then-DEL release The non-atomic GET→check-owner→DEL release was vulnerable to a race where the TTL expires between GET and DEL, allowing a new lock holder to be evicted. Simplify by relying solely on the 30s TTL for lock expiry — the computation completes well within that window. |
||
|
Elie Habib
|
d58c18b3fd |
fix(news): remove overly strict rate limit on summarize-article (#1171)
* fix(news): remove overly strict per-endpoint rate limit on summarize-article The 20 req/60s per-IP limit was causing 429s for legitimate users. The frontend fallback chain (Ollama → Groq → OpenRouter) can consume 3 requests per attempt, and server-side Redis caching already prevents redundant LLM calls. The global 600/60s rate limit remains as protection. * fix(intelligence): give classify-event its own 600/60s rate limit The classify-event endpoint was sharing the global 600/60s rate limit with all other endpoints, causing 429s during page loads when summarize-article, bootstrap, and classify calls competed for the same budget. Giving it a dedicated 600/60s limit isolates it from the global pool. Existing client-side throttling (80/min via canQueueAiClassification) ensures real traffic stays well under this. |
||
|
Elie Habib
|
e48162e9c0 |
fix(rate-limit): add per-endpoint rate limits for summarize endpoints (#1161)
Cache lookups (summarize-article-cache) shared the global 600/60s IP rate limit with expensive LLM calls (summarize-article), causing users to burn rate limit budget on cheap Redis reads. - Add ENDPOINT_RATE_POLICIES map with per-endpoint limits - summarize-article POST: 20 req/60s (protect LLM quota) - summarize-article-cache GET: 3000 req/60s (generous for browsing) - Two-phase gateway check: endpoint-specific first, global fallback - Endpoints with specific policy skip global limiter entirely - Path normalization prevents trailing-slash bypass |
||
|
Elie Habib
|
a6b7c771ac |
fix(economic): seed all WB indicators on Railway, never call WB API from frontend (#1159)
* fix(economic): seed all WB indicators on Railway, never call WB API from frontend Extends seed-wb-indicators.mjs to pre-compute progress data (4 indicators) and renewable energy data (EG.ELC.RNEW.ZS) alongside tech readiness rankings. Frontend callers (progress-data.ts, renewable-energy-data.ts, getTechReadinessRankings, getCountryComparison) now read exclusively from bootstrap/Redis seed data. Zero Vercel Edge → World Bank API calls remain. * fix: address code review findings (P1+P2) - Fix triple JSON.parse in seed verification (P1) - Graceful fallback for renewable data fetch failure (P2) - Use Map lookup instead of Array.find in progress-data (P2) - Update regression test for bootstrap-only getTechReadinessRankings (P2) |
||
|
Elie Habib
|
3bfb167226 |
fix(health): write seed-meta on cache hits (throttled 5min) (#1138)
Previously seed-meta was only written on cache misses, so RPC-populated keys like minerals/giving never got seed-meta if the data was already cached. Now writeSeedMeta fires on cache hits too, throttled to one Redis SET per 5 minutes per key to avoid spam. |
||
|
Elie Habib
|
077da6b024 |
feat(health): auto seed-meta freshness tracking for all RPC handlers (#1127)
cachedFetchJson now auto-writes seed-meta:{key} on every fresh fetch,
enabling freshness tracking for all standalone RPC-populated keys
without manual instrumentation. Health endpoint checks staleness for
all monitored keys using configurable maxStaleMin thresholds.
|
||
|
Elie Habib
|
4cc99629be |
fix(security): harden cache keys against injection and hash collision (WM-2026-001/002) (#1103)
- CVE-1: search-gdelt-documents — add 500-char query cap + SHA-256 hash for cache key (was raw user input, no length limit) - CVE-2: get-country-intel-brief — replace FNV-1a hashString() with sha256Hex() for attacker-controlled context param - BONUS: summary-cache-key — remove .slice(0,6) truncation that collapsed hash space from 2^52 to ~2.18B values - Harden: classify-event + deduct-situation — same sha256Hex() migration - Add sha256Hex() utility using crypto.subtle (Edge/Vercel/Node 18+) Reported by: Cody Richard (Sstrickys) via responsible disclosure |
||
|
Elie Habib
|
7ecb7f06b8 |
fix: wire bootstrap hydration for 8 missing data sources (#1065)
Register 6 new seeds in bootstrap.js (crypto, gulf, stablecoin, unrest, iran, ucdp) and wire getHydratedData() in 7 service files. Also adds hydration for 2 previously-registered keys (cyberThreats, flightDelays) that had no frontend consumer. Syncs cache-keys.ts with bootstrap.js for test parity. Cyber hydration correctly maps through toCyberThreat() to convert proto enum strings to friendly types. |
||
|
Elie Habib
|
c7942b800a |
feat: Railway CII seed + bootstrap hydration for instant panel render (#984)
* fix: add circuit breaker + bootstrap to CII risk scores Same pattern as theater posture (#948): replace fragile in-memory cache + manual persistent-cache with circuit breaker (SWR, IndexedDB, cooldown) and bootstrap hydration. Eliminates learning-mode delay on cold start and survives RPC failures without blanking the panel. * fix: add localStorage sync prime for CII risk scores getCachedScores() is called synchronously by country-intel.ts as a fallback during learning mode. Without localStorage priming, the breaker's async IndexedDB hydration hasn't run yet and returns null. - Add shape validator (isValidCiiEntry) for untrusted localStorage data - Add loadFromStorage/saveToStorage with 24h staleness ceiling - Prime breaker synchronously at module load from localStorage - Skip priming for empty cii arrays to avoid cached-empty trap - Save to localStorage on both bootstrap and RPC success paths * feat: Railway CII seed + bootstrap hydration for instant panel render - Add 8-source CII seed to Railway (ACLED, UCDP, outages, climate, cyber, fires, GPS, Iran strikes) - Neuter Vercel handler to read-only (returns Railway-seeded cache, never recomputes) - Register riskScores in bootstrap FAST tier for CDN-cached delivery - Add early CII hydration in data-loader before intelligence signals - Add CIIPanel.renderFromCached() for instant render from bootstrap data - Refactor cached-risk-scores.ts: circuit breaker + localStorage sync prime + bootstrap hydration - Progressive enhancement: cached render → full 18-source local recompute (no spinner) * fix: remove duplicate riskScores key in BOOTSTRAP_TIERS after merge |
||
|
Elie Habib
|
1743b5c289 |
fix: add circuit breaker + bootstrap to CII risk scores (#980)
* fix: add circuit breaker + bootstrap to CII risk scores Same pattern as theater posture (#948): replace fragile in-memory cache + manual persistent-cache with circuit breaker (SWR, IndexedDB, cooldown) and bootstrap hydration. Eliminates learning-mode delay on cold start and survives RPC failures without blanking the panel. * fix: add localStorage sync prime for CII risk scores getCachedScores() is called synchronously by country-intel.ts as a fallback during learning mode. Without localStorage priming, the breaker's async IndexedDB hydration hasn't run yet and returns null. - Add shape validator (isValidCiiEntry) for untrusted localStorage data - Add loadFromStorage/saveToStorage with 24h staleness ceiling - Prime breaker synchronously at module load from localStorage - Skip priming for empty cii arrays to avoid cached-empty trap - Save to localStorage on both bootstrap and RPC success paths |
||
|
Elie Habib
|
02a4a52673 |
fix: strategic risk overview loses sources after idle (#948) (#968)
Add circuit breaker + IndexedDB persistence + bootstrap hydration to theater posture fetching — the only major panel without these resilience layers. Replaces fragile in-memory cache (15-min TTL) and destructive localStorage (30-min hard-delete) with the standard 3-tier pattern used by all other panels. |
||
|
Elie Habib
|
c2f17dec45 |
fix(supply-chain): resolve P1 threat zeroing and P2 geo-first misclassification (#964)
* enhance supply chain panel * fix(supply-chain): resolve P1 threat zeroing and P2 geo-first misclassification P1: threat baseline is now always applied regardless of config staleness — stale config only adds a review-recommended note, never zeros the score. P2: resolveChokepointId now checks text evidence first and only falls back to proximity when text has no confident match. Adds regression test: text "Bab el-Mandeb" with location near Suez correctly resolves to bab_el_mandeb. --------- Co-authored-by: fayez bast <fayezbast15@gmail.com> |
||
|
Elie Habib
|
2e93e0e8ed |
perf(bootstrap): tier slow/fast data for ~46% CDN egress reduction (#838)
Split bootstrap endpoint into slow-changing (1h TTL: BIS rates, minerals, sectors, etc.) and fast-changing (10min TTL: earthquakes, outages, macro signals, etc.) tiers via ?tier=slow|fast query param. Frontend fetches both tiers in parallel with shared 800ms timeout. Partial failure is graceful — panels fall through to individual RPCs. Backward compatible: no ?tier= param returns all keys at s-maxage=600. Also removes orphaned ucdpEvents key (no getHydratedData consumer). |
||
|
Elie Habib
|
5b33947909 |
refactor: deduplicate clampInt into server/_shared/constants
Move clampInt to server/_shared/constants.ts as a single shared export. Remove 4 duplicate inline definitions from research handlers and re-export from cyber/_shared.ts for backward compatibility. Simplify tech-events limit/days clamping by removing redundant guards. |
||
|
Elie Habib
|
37f07a6af2 |
fix(prod): CORS fallback, rate-limit bump, RSS proxy allowlist (#814)
- Add wildcard CORS headers in vercel.json for /api/* routes so Vercel infra 500s (which bypass edge function code) still include CORS headers - Bump rate limit from 300 to 600 req/60s in both rate-limit files to accommodate dashboard init burst (~30-40 parallel requests) - Add smartraveller.gov.au (bare + www) to Railway relay RSS allowlist - Add redirect hostname validation in fetchWithRedirects to prevent SSRF via open redirects on allowed domains |
||
|
Nicolas Gomes Ferreira Dos Santos
|
f5b2d4c82c |
refactor(server): consolidate declare const process into shared env.d.ts (#752)
The same ambient process declaration was duplicated across 35 server files. Move it to a single server/env.d.ts file that tsconfig.api.json automatically includes. Addresses #197 (L-15). Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Nicolas Gomes Ferreira Dos Santos
|
74d159cda9 |
fix(redis): add error logging and reduce timeouts for edge functions (#749)
* fix(redis): add error logging and reduce timeouts for edge functions - Add console.warn to 5 silent catch blocks for Redis operation failures - Add .catch() to cachedFetchJson/cachedFetchJsonWithMeta to log fetcher exceptions - Reduce Redis GET/SET timeouts from 3s to 1.5s (edge function budget awareness) - Keep pipeline operation timeouts at 5s (geoSearchByBox, getHashFieldsBatch) Addresses #197 (L-4, L-5). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(redis): preserve error propagation and use safe error formatting - Change .catch() on cachedFetchJson/cachedFetchJsonWithMeta to log-and-rethrow instead of swallowing errors — preserves existing contract where callers rely on rejection for fallback tiers (list-ucdp-events, get-macro-signals) - Restore getCachedJsonBatch to pipeline timeout (5s) — it's a pipeline operation like geoSearchByBox/getHashFieldsBatch, not a simple GET/SET - Replace unsafe (err as Error).message with errMsg() helper that handles non-Error thrown values safely - Extract REDIS_OP_TIMEOUT_MS and REDIS_PIPELINE_TIMEOUT_MS constants --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
699576e727 |
fix(rate-limit): increase sliding window to 300 req/min (#515)
App init fires many concurrent classify-event, summarize-article, and record-baseline-snapshot calls, exhausting the 200/min limit and causing 429s. Bump to 300 as a temporary measure while client-side batching is implemented. |
||
|
Elie Habib
|
524e3d4c25 |
fix: resolve bootstrap 401 and 429 rate limiting on page init (#512)
Same-origin browser requests don't send Origin header (per CORS spec), causing validateApiKey to reject them. Extract origin from Referer as fallback. Increase rate limit from 60 to 200 req/min to accommodate the ~50 requests fired during page initialization. |
||
|
Elie Habib
|
3d2c638a72 |
feat(military): server-side military bases 125K + rate limiting (#496)
* feat(military): server-side military bases with 125K entries + rate limiting (#485) Migrate military bases from 224 static client-side entries to 125,380 server-side entries stored in Redis GEO sorted sets, served via bbox-filtered GEOSEARCH endpoint with server-side clustering. Data pipeline: - Pizzint/Polyglobe: 79,156 entries (Supabase extraction) - OpenStreetMap: 45,185 entries - MIRTA: 821 entries - Curated strategic: 218 entries - 277 proximity duplicates removed Server: - ListMilitaryBases RPC with GEOSEARCH + HMGET + tier/filter/clustering - Antimeridian handling (split bbox queries) - Blue-green Redis deployment with atomic version pointer switch - geoSearchByBox() + getHashFieldsBatch() helpers in redis.ts Security: - @upstash/ratelimit: 60 req/min sliding window per IP - IP spoofing fix: prioritize x-real-ip (Vercel-injected) over x-forwarded-for - Require API key for non-browser requests (blocks unauthenticated curl/scripts) - Input validation: allowlisted types/kinds, regex country, clamped bbox/zoom Frontend: - Viewport-driven loading with bbox quantization + debounce - Server-side grid clustering at low zoom levels - Enriched popup with kind, category badges (airforce/naval/nuclear/space) - Static 224 bases kept as search fallback + initial render * fix(military): fallback to production Redis keys in preview deployments Preview deployments prefix Redis keys with `preview:{sha}:` but military bases data is seeded to unprefixed (production) keys. When the prefixed `military:bases:active` key is missing, fall back to the unprefixed key and use raw (unprefixed) keys for geo/meta lookups. * fix: remove unused 'remaining' destructure in rate-limit (TS6133) * ci: add typecheck:api to pre-push hook to catch server-side TS errors * debug(military): add X-Bases-Debug response header for preview diagnostics * fix(bases): trigger initial server fetch on map load fetchServerBases() was only called on moveend — if the user never panned/zoomed, the API was never called and only the 224 static fallback bases showed. |
||
|
Elie Habib
|
98d231595e |
perf: bootstrap endpoint + polling optimization (#495)
* perf: bootstrap endpoint + polling optimization (phases 3-4)
Replace 15+ individual RPC calls on startup with a single /api/bootstrap
batch call that fetches pre-cached data from Redis. Consolidate 6 panel
setInterval timers into the central RefreshScheduler for hidden-tab
awareness (10x multiplier) and adaptive backoff (up to 4x for unchanged
data). Convert IntelligenceGapBadge from 10s polling to event-driven
updates with 60s safety fallback.
* fix(bootstrap): inline Redis + cache keys in edge function
Vercel Edge Functions cannot resolve cross-directory TypeScript imports
from server/_shared/. Inline getCachedJsonBatch and BOOTSTRAP_CACHE_KEYS
directly in api/bootstrap.js. Add sync test to ensure inlined keys stay
in sync with the canonical server/_shared/cache-keys.ts registry.
* test: add Edge Function module isolation guard for all api/*.js files
Prevents any Edge Function from importing from ../server/ or ../src/
which breaks Vercel builds. Scans all 12 non-helper Edge Functions.
* fix(bootstrap): read unprefixed cache keys on all environments
Preview deploys set VERCEL_ENV=preview which caused getKeyPrefix() to
prefix Redis keys with preview:<sha>:, but handlers only write to
unprefixed keys on production. Bootstrap is a read-only consumer of
production cache — always read unprefixed keys.
* fix(bootstrap): wire sectors hydration + add coverage guard
- Wire getHydratedData('sectors') in data-loader to skip Yahoo Finance
fetch when bootstrap provides sector data
- Add test ensuring every bootstrap key has a getHydratedData consumer
— prevents adding keys without wiring them
* fix(server): resolve 25 TypeScript errors + add server typecheck to CI
- _shared.ts: remove unused `delay` variable
- list-etf-flows.ts: add missing `rateLimited` field to 3 return literals
- list-market-quotes.ts: add missing `rateLimited` field to 4 return literals
- get-cable-health.ts: add non-null assertions for regex groups and array access
- list-positive-geo-events.ts: add non-null assertion for array index
- get-chokepoint-status.ts: add required fields to request objects
- CI: run `typecheck:api` (tsconfig.api.json) alongside `typecheck` to catch
server/ TS errors before merge
|
||
|
Elie Habib
|
5289a0cedb |
feat(gateway): complete edge cache tier coverage + degraded-response policy (#484)
- Add 11 missing GET routes to RPC_CACHE_TIER map (8 slow, 3 medium) - Add response-headers side-channel (WeakMap) so handlers can signal X-No-Cache without codegen changes; wire into military-flights and positive-geo-events handlers on upstream failure - Add env-controlled per-endpoint tier override (CACHE_TIER_OVERRIDE_*) for incident response rollback - Add VITE_WS_API_URL hostname allowlist (*.worldmonitor.app + localhost) - Fix fetch.bind(globalThis) in positive-events-geo.ts (deferred lambda) - Add CI test asserting every generated GET route has an explicit cache tier entry (prevents silent default-tier drift) |
||
|
Elie Habib
|
5f908c0929 |
feat(cache): add negative-result caching to cachedFetchJson (#466)
When upstream APIs return errors (HTTP 403, 429, timeout), fetchers
return null. Previously null results were not cached, causing repeated
request storms against broken APIs every refresh cycle.
Now caches a sentinel value ('__WM_NEG__') with a short 2-minute TTL
on null results. Subsequent requests within that window get null
immediately without hitting upstream. Thrown errors (transient) skip
sentinel caching and retry immediately.
Also filters sentinels from getCachedJsonBatch pipeline reads and fixes
theater posture coalescing test (expected 2 OpenSky fetches for 2
theater query regions, not 1).
|
||
|
Elie Habib
|
b0071d1a14 |
fix: eliminate cache stampede across all server handlers (#389)
* fix: use cachedFetchJson for theater posture to prevent Wingbits API stampede The theater posture handler used manual getCachedJson/setCachedJson which has no request coalescing. During the window between a cache miss and the cache write completing (~15s Wingbits timeout), every concurrent request bypassed the cache and fired its own POST to Wingbits with 9 bounding boxes — causing ~1500 requests/min. Switch to cachedFetchJson which deduplicates concurrent in-flight requests via a shared promise, so only one upstream fetch runs at a time. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: migrate all handlers to cachedFetchJson, fix metadata labeling with cachedFetchJsonWithMeta - Migrate 48 server handlers from manual getCachedJson/setCachedJson to cachedFetchJson, eliminating cache stampede vulnerability across the entire codebase (concurrent cache misses now coalesce into one fetch) - Add cachedFetchJsonWithMeta to redis.ts — returns { data, source } atomically, fixing TOCTOU race between separate cache check and fetch - Wire summarize-article.ts and get-usni-fleet-report.ts to use cachedFetchJsonWithMeta for correct provider/cached metadata - Add try/catch to 7 handlers that lost graceful degradation - Add 4 tests for WithMeta source labeling: cache hit, fresh miss, coalesced followers, and TOCTOU regression protection * chore: trigger PR sync --------- Co-authored-by: Mikael Sundberg <mikael.sundberg82@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
EFK
|
9e2432f098 |
fix: infra cost optimizations round 2 — polling, TTLs, ACLED dedup (#275)
* fix: skip AIS polling when browser tab is backgrounded Add two layers of defence to avoid wasting Railway relay bandwidth on hidden tabs: 1. document.hidden guard in pollSnapshot() — skips fetch when tab is not visible (interval keeps ticking so first tick after focus polls immediately) 2. visibilitychange listener — pauses/resumes the interval timer entirely so no callbacks fire while backgrounded Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: align cache TTLs with upstream data source refresh rates Climate anomalies: 30min → 3h Source: Open-Meteo Archive API uses ERA5 reanalysis data with a 2–7 day lag (https://open-meteo.com/en/docs/historical-weather-api). 30-min polling gains nothing against week-old data. Fire detections: 30min → 1h Source: NASA FIRMS VIIRS_SNPP_NRT refreshes approximately every 3 hours (https://firms.modaps.eosdis.nasa.gov/usfs/active_fire/). Polling twice per refresh cycle still catches every update. Combined: ~66% fewer Redis reads on these two endpoints. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: deduplicate ACLED API calls via shared cached fetch layer Three RPC handlers (conflict, unrest, intelligence) each made independent ACLED API calls with their own auth/timeout/parsing logic. Extract a shared fetchAcledCached() in server/_shared/acled.ts that: - Centralises auth, timeout, and error handling - Caches by query params so overlapping date ranges share results - Eliminates ~120 lines of duplicated fetch boilerplate - Reduces ACLED API calls when multiple handlers run concurrently Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add structural tests for round 2 infra optimizations 17 tests covering: TTL alignment (climate 3h, fire 1h), ACLED shared cache layer (fetchAcledCached, cache key derivation, consumer imports, no duplicated ACLED_API_URL), and maritime AIS visibility guard (document.hidden check, pausePolling, resumePolling, visibilitychange listener wiring). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: rename round 2 test file to ttl-acled-ais-guards Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #275 review findings --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
dc7a1ae61a |
fix: infrastructure cost optimizations across caching, polling, and batching (#283)
* fix: add request coalescing to Redis cache layer Concurrent cache misses for the same key now share a single upstream fetch instead of each triggering redundant API calls. This eliminates duplicate work within Edge Function invocations under burst traffic. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: reduce AIS polling frequency from 10s to 30s Vessel positions do not change meaningfully in 10 seconds at sea. Reduces Railway relay requests by 66% with negligible UX impact. Stale threshold bumped to 45s to match the new interval. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: quantize military flights bbox cache keys to 1-degree grid Precise bounding box coordinates caused near-zero cache hit rate since every map pan/zoom produced a unique key. Snapping to a 1-degree grid lets nearby viewports share cache entries, dramatically reducing redundant OpenSky API calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: parallelize ETF chart fetches instead of sequential await loop The loop awaited each ETF chart fetch individually, blocking on every Yahoo gate delay. Using Promise.allSettled lets all 10 fetches queue concurrently through the Yahoo gate, cutting wall time from ~12s to ~6s. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add Redis pipeline batch GET to reduce round-trips Add getCachedJsonBatch() using the Upstash pipeline API to fetch multiple keys in a single HTTP call. Refactor aircraft details batch handler from 20 sequential GETs to 1 pipelined request. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add structural tests for Redis caching optimizations 18 tests covering: cachedFetchJson request coalescing (in-flight dedup, cache-before-fetch ordering, cleanup), getCachedJsonBatch pipeline API, aircraft batch handler pipeline usage, bbox grid quantization (1-degree step, expanded fetch bbox), and ETF parallel fetch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: enforce military bbox contract and add behavioral cache tests --------- Co-authored-by: Elias El Khoury <efk@anghami.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Elie Habib
|
63a4c9ab9c |
feat: Upstash Redis shared caching + cache key contamination fixes (#232)
* fix(sentry): add noise filters for 5 non-actionable error patterns Filter dynamic import alt phrasing, script parse errors, maplibre style/WebGL crashes, and CustomEvent promise rejections. Also fix beforeSend to catch short Firefox null messages like "E is null". * fix: cache write race, settings stale key status, yahoo gate concurrency P1: Replace async background thread cache write with synchronous fs::write to prevent out-of-order writes and dirty flag cleared before persistence. P2: Add WorldMonitorTab.refresh() called after loadDesktopSecrets() so the API key badge reflects actual keychain state. P3: Replace timestamp-based Yahoo gate with promise queue to ensure sequential execution under concurrent callers. * feat: add Upstash Redis shared caching to all RPC handlers + fix cache key contamination - Add Redis L2 cache (getCachedJson/setCachedJson) to 28 RPC handlers across all service domains (market, conflict, cyber, economic, etc.) - Fix 10 P1 cache key contamination bugs where under-specified keys caused cross-request data pollution (e.g. filtered requests returning unfiltered cached data) - Restructure list-internet-outages to cache-then-filter pattern so country/timeRange filters always apply after cache read - Add write_lock mutex to PersistentCache in main.rs to prevent desktop cache write-race conditions - Document FMP (Financial Modeling Prep) as Yahoo Finance fallback TODO in market/v1/_shared.ts * fix: cache-key contamination and PizzINT/GDELT partial-failure regression - tech-events: fetch with limit=0 and cache full result, apply limit slice after cache read to prevent low-limit requests poisoning cache - pizzint: restore try-catch around PizzINT fetch so GDELT tension pairs are still returned when PizzINT API is down * fix: remove extra closing brace in pizzint try-catch * fix: recompute conferenceCount/mappableCount after limit slice * fix: bypass WM API key gate for registration endpoint /api/register-interest must reach cloud without a WorldMonitor API key, otherwise desktop users can never register (circular dependency). |
||
|
Elie Habib
|
a2bcfeede4 |
fix(market): harden Yahoo Finance resilience and UCDP retry logic
- Fix proto.price falsy bug: price of 0 was treated as null - Replace global lastSuccessfulResults with per-symbol-set Map to prevent stock data leaking into commodity fallback - Add yahooGate (600ms) to serialize Yahoo requests and avoid IP rate limits - Add per-symbol-set cache key in server handler to isolate stock/commodity/sector calls - Clear UCDP circuit breaker cache on empty responses to prevent 10-min lockout - Add UCDP retry loop (3 attempts, 15s apart) on cold start - Delay ETF panel initial fetch by 8s to reduce Yahoo contention on startup |
||
|
Sebastien Melki
|
c939cc6296 |
Proto-first API rebuild: sebuf contracts, handlers, gateway, and generated docs (#106)
* docs: initialize sebuf integration project with codebase map Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: add project config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: complete project research * docs: define v1 requirements (34 requirements, 8 categories) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: create roadmap (8 phases) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(01): capture phase context * docs(state): record phase 1 context session * docs(01): research phase domain - buf toolchain, sebuf codegen, proto patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(01-proto-foundation): create phase plan * chore(01-01): configure buf toolchain with buf.yaml, buf.gen.yaml, buf.lock - buf.yaml v2 with STANDARD+COMMENTS lint, FILE+PACKAGE+WIRE_JSON breaking, deps on protovalidate and sebuf - buf.gen.yaml configures protoc-gen-ts-client, protoc-gen-ts-server, protoc-gen-openapiv3 plugins - buf.lock generated with resolved dependency versions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(01-01): add shared core proto type definitions - geo.proto: GeoCoordinates with lat/lng validation, BoundingBox for spatial queries - time.proto: TimeRange with google.protobuf.Timestamp start/end - pagination.proto: cursor-based PaginationRequest (1-100 page_size) and PaginationResponse - i18n.proto: LocalizableString for pre-localized upstream API strings - identifiers.proto: typed ID wrappers (HotspotID, EventID, ProviderID) for cross-domain refs - general_error.proto: GeneralError with RateLimited, UpstreamDown, GeoBlocked, MaintenanceMode All files pass buf lint (STANDARD+COMMENTS) and buf build with zero errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(01-01): complete buf toolchain and core proto types plan - SUMMARY.md documents 2 tasks, 9 files created, 2 deviations auto-fixed - STATE.md updated: plan 1/2 in phase 1, decisions recorded - ROADMAP.md updated: phase 01 in progress (1/2 plans) - REQUIREMENTS.md updated: PROTO-01, PROTO-02, PROTO-03 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(01-01): use int64 epoch millis instead of google.protobuf.Timestamp User preference: all time fields use int64 (Unix epoch milliseconds) instead of google.protobuf.Timestamp for simpler serialization and JS interop. Applied to TimeRange and MaintenanceMode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(01-02): create test domain proto files with core type imports - Add test_item.proto with GeoCoordinates import and int64 timestamps - Add get_test_items.proto with TimeRange and Pagination imports - Add service.proto with HTTP annotations for TestService - All proto files pass buf lint and buf build * feat(01-02): run buf generate and create Makefile for code generation pipeline - Add Makefile with generate, lint, clean, install, check, format, breaking targets - Update buf.gen.yaml with managed mode and paths=source_relative for correct output paths - Generate TypeScript client (TestServiceClient class) at src/generated/client/ - Generate TypeScript server (TestServiceHandler interface) at src/generated/server/ - Generate OpenAPI 3.1.0 specs (JSON + YAML) at docs/api/ - Core type imports (GeoCoordinates, TimeRange, Pagination) flow through to generated output * docs(01-02): complete test domain code generation pipeline plan - Create 01-02-SUMMARY.md with pipeline validation results - Update STATE.md: phase 1 complete, 2/2 plans done, new decisions recorded - Update ROADMAP.md: phase 1 marked complete (2/2) - Update REQUIREMENTS.md: mark PROTO-04 and PROTO-05 complete * docs(phase-01): complete phase execution and verification * test(01): complete UAT - 6 passed, 0 issues * feat(2A): define all 17 domain proto packages with generated clients, servers, and OpenAPI specs Remove test domain protos (Phase 1 scaffolding). Add core enhancements (severity.proto, country.proto, expanded identifiers.proto). Define all 17 domain services: seismology, wildfire, climate, conflict, displacement, unrest, military, aviation, maritime, cyber, market, prediction, economic, news, research, infrastructure, intelligence. 79 proto files producing 34 TypeScript files and 34 OpenAPI specs. buf lint clean, tsc clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2B): add server runtime phase context and handoff checkpoint Prepare Phase 2B with full context file covering deliverables, key reference files, generated code patterns, and constraints. Update STATE.md with resume pointer. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2B): research phase domain * docs(2B): create phase plan * feat(02-01): add shared server infrastructure (router, CORS, error mapper) - router.ts: Map-based route matcher from RouteDescriptor[] arrays - cors.ts: TypeScript port of api/_cors.js with POST/OPTIONS methods - error-mapper.ts: onError callback handling ApiError, network, and unknown errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(02-01): implement seismology handler as first end-to-end proof - Implements SeismologyServiceHandler from generated server types - Fetches USGS M4.5+ earthquake GeoJSON feed and transforms to proto-shaped Earthquake[] - Maps all fields: id, place, magnitude, depthKm, location, occurredAt (String), sourceUrl Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(02-01): complete server infrastructure plan - SUMMARY.md with task commits, decisions, and self-check - STATE.md updated: position, decisions, session info - REQUIREMENTS.md: SERVER-01, SERVER-02, SERVER-06 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(02-02): create Vercel catch-all gateway, tsconfig.api.json, and typecheck:api script - api/[[...path]].ts mounts seismology routes via catch-all with CORS on every response path - tsconfig.api.json extends base config without vite/client types for edge runtime - package.json adds typecheck:api script * feat(02-02): add Vite dev server plugin for sebuf API routes - sebufApiPlugin() intercepts /api/{domain}/v1/* in dev mode - Uses dynamic imports to lazily load handler modules inside configureServer - Converts Connect IncomingMessage to Web Standard Request - CORS headers applied to all plugin responses (200, 204, 403, 404) - Falls through to existing proxy rules for non-sebuf /api/* paths * docs(02-02): complete gateway integration plan - SUMMARY.md documenting catch-all gateway + Vite plugin implementation - STATE.md updated: Phase 2B complete, decisions recorded - ROADMAP.md updated: Phase 02 marked complete (2/2 plans) - REQUIREMENTS.md: SERVER-03, SERVER-04, SERVER-05 marked complete * docs(02-server-runtime): create gap closure plan for SERVER-05 Tauri sidecar * feat(02-03): add esbuild compilation step for sebuf sidecar gateway bundle - Create scripts/build-sidecar-sebuf.mjs that bundles api/[[...path]].ts into a single ESM .js file - Add build:sidecar-sebuf npm script and chain it into the main build command - Install esbuild as explicit devDependency - Gitignore the compiled api/[[...path]].js build artifact Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(02-03): verify sidecar discovery and annotate SERVER-05 gap closure - Confirm compiled bundle handler returns status 200 for POST requests - Add gap closure note to SERVER-05 in REQUIREMENTS.md - Verify typecheck:api and full build pipeline pass without regressions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(02-03): complete sidecar sebuf bundle plan - Create 02-03-SUMMARY.md documenting esbuild bundle compilation - Update STATE.md with plan 03 position, decisions, and metrics - Update ROADMAP.md plan progress (3/3 plans complete) - Annotate SERVER-05 gap closure in REQUIREMENTS.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-02): complete phase execution * docs(2C): capture seismology migration phase context Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(state): record phase 2C context session Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2C): research seismology migration phase Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2C): create seismology migration phase plan * feat(2C-01): annotate all int64 time fields with INT64_ENCODING_NUMBER - Vendor sebuf/http/annotations.proto locally with Int64Encoding extension (50010) - Remove buf.build/sebmelki/sebuf BSR dep, use local vendored proto instead - Add INT64_ENCODING_NUMBER annotation to 34 time fields across 20 proto files - Regenerate all TypeScript client and server code (time fields now `number` not `string`) - Fix seismology handler: occurredAt returns number directly (no String() wrapper) - All non-time int64 fields (displacement counts, population) left as string - buf lint, buf generate, tsc, and sidecar build all pass with zero errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2C-01): complete INT64_ENCODING_NUMBER plan - Create 2C-01-SUMMARY.md with execution results and deviations - Update STATE.md: plan 01 complete, int64 blocker resolved, new decisions - Update ROADMAP.md: mark 2C-01 plan complete - Update REQUIREMENTS.md: mark CLIENT-01 complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(lint): exclude .planning/ from markdownlint GSD planning docs use formatting that triggers MD032 -- these are machine-generated and not user-facing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2C-02): rewrite earthquake adapter to use SeismologyServiceClient and adapt all consumers to proto types - Replace legacy fetch/circuit-breaker adapter with port/adapter wrapping SeismologyServiceClient - Update 7 consuming files to import Earthquake from @/services/earthquakes (the port) - Adapt all field accesses: lat/lon -> location?.latitude/longitude, depth -> depthKm, time -> occurredAt, url -> sourceUrl - Remove unused filterByTime from Map.ts (only called for earthquakes, replaced with inline filter) - Update e2e test data to proto Earthquake shape Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore(2C-02): delete legacy earthquake endpoint, remove Vite proxy, clean API_URLS config - Delete api/earthquakes.js (legacy Vercel edge function proxying USGS) - Remove /api/earthquake Vite dev proxy (sebufApiPlugin handles seismology now) - Remove API_URLS.earthquakes entry from base config (no longer referenced) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2C-02): complete seismology client wiring plan - Create 2C-02-SUMMARY.md with execution results - Update STATE.md: phase 2C complete, decisions, metrics - Update ROADMAP.md: mark 2C-02 and phase 2C complete - Mark requirements CLIENT-02, CLIENT-04, CLEAN-01, CLEAN-02 complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2C): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2D): create wildfire migration phase plan * feat(2D-01): enhance FireDetection proto and implement wildfire handler - Add region (field 8) and day_night (field 9) to FireDetection proto - Regenerate TypeScript client and server types - Implement WildfireServiceHandler with NASA FIRMS CSV proxy - Fetch all 9 monitored regions in parallel via Promise.allSettled - Graceful degradation to empty list when API key is missing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2D-01): wire wildfire routes into gateway and rebuild sidecar - Import createWildfireServiceRoutes and wildfireHandler in catch-all - Mount wildfire routes alongside seismology in allRoutes array - Rebuild sidecar-sebuf bundle with wildfire endpoint included Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2D-01): complete wildfire handler plan - Create 2D-01-SUMMARY.md with execution results - Update STATE.md position to 2D plan 01 complete - Update ROADMAP.md with 2D progress (1/2 plans) - Mark DOMAIN-01 requirement complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2D-02): create wildfires service module and rewire all consumers - Add src/services/wildfires/index.ts with fetchAllFires, computeRegionStats, flattenFires, toMapFires - Rewire App.ts to import from @/services/wildfires with proto field mappings - Rewire SatelliteFiresPanel.ts to import FireRegionStats from @/services/wildfires - Update signal-aggregator.ts source comment * chore(2D-02): delete legacy wildfire endpoint and service module - Remove api/firms-fires.js (replaced by api/server/worldmonitor/wildfire/v1/handler.ts) - Remove src/services/firms-satellite.ts (replaced by src/services/wildfires/index.ts) - Zero dangling references confirmed - Full build passes (tsc, vite, sidecar) * docs(2D-02): complete wildfire consumer wiring plan - Create 2D-02-SUMMARY.md with execution results - Update STATE.md: phase 2D complete, progress ~52% - Update ROADMAP.md: phase 2D plan progress * docs(phase-2D): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2E): research climate migration domain Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2E): create phase plan * feat(2E-01): implement climate handler with 15-zone monitoring and baseline comparison - Create ClimateServiceHandler with 15 hardcoded monitored zones matching legacy - Parallel fetch from Open-Meteo Archive API via Promise.allSettled - 30-day baseline comparison: last 7 days vs preceding baseline - Null filtering with paired data points, minimum 14-point threshold - Severity classification (normal/moderate/extreme) and type (warm/cold/wet/dry/mixed) - 1-decimal rounding for tempDelta and precipDelta - Proto ClimateAnomaly mapping with GeoCoordinates Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2E-01): wire climate routes into gateway and rebuild sidecar - Import createClimateServiceRoutes and climateHandler in catch-all gateway - Mount climate routes alongside seismology and wildfire - Rebuild sidecar-sebuf bundle with climate routes included Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2E-01): complete climate handler plan - Create 2E-01-SUMMARY.md with execution results - Update STATE.md: position to 2E plan 01, add decisions - Update ROADMAP.md: mark 2E-01 complete, update progress table Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2E-02): rewrite climate service module and rewire all consumers - Replace src/services/climate.ts with src/services/climate/index.ts directory module - Port/adapter pattern: ClimateServiceClient maps proto shapes to legacy consumer shapes - Rewire ClimateAnomalyPanel, DeckGLMap, MapContainer, country-instability, conflict-impact - All 6 consumers import ClimateAnomaly from @/services/climate instead of @/types - Drop dead getSeverityColor function, keep getSeverityIcon and formatDelta - Fix minSeverity required param in listClimateAnomalies call * chore(2E-02): delete legacy climate endpoint and remove dead types - Delete api/climate-anomalies.js (replaced by sebuf climate handler) - Remove ClimateAnomaly and AnomalySeverity from src/types/index.ts - Full build passes with zero errors * docs(2E-02): complete climate client wiring plan - Create 2E-02-SUMMARY.md with execution results - Update STATE.md: phase 2E complete, decisions, session continuity - Update ROADMAP.md: phase 2E progress * docs(phase-2E): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2F): research prediction migration domain * docs(2F): create prediction migration phase plans * feat(2F-01): implement prediction handler with Gamma API proxy - PredictionServiceHandler proxying Gamma API with 8s timeout - Maps events/markets to proto PredictionMarket with 0-1 yesPrice scale - Graceful degradation: returns empty markets on any failure (Cloudflare expected) - Supports category-based events endpoint and default markets endpoint Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2F-01): wire prediction routes into gateway - Import createPredictionServiceRoutes and predictionHandler - Mount prediction routes in allRoutes alongside seismology, wildfire, climate - Sidecar bundle rebuilt successfully (21.2 KB) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2F-01): complete prediction handler plan - SUMMARY.md with handler implementation details and deviation log - STATE.md updated to 2F in-progress position with decisions - ROADMAP.md updated to 1/2 plans complete for phase 2F - REQUIREMENTS.md marked DOMAIN-02 complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2F-02): create prediction service module and rewire all consumers - Create src/services/prediction/index.ts preserving all business logic from polymarket.ts - Replace strategy 4 (Vercel edge) with PredictionServiceClient in polyFetch - Update barrel export from polymarket to prediction in services/index.ts - Rewire 7 consumers to import PredictionMarket from @/services/prediction - Fix 3 yesPrice bugs: CountryIntelModal (*100), App.ts search (*100), App.ts snapshot (1-y) - Drop dead code getPolymarketStatus() Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore(2F-02): delete legacy endpoint and remove dead types - Delete api/polymarket.js (replaced by sebuf handler) - Delete src/services/polymarket.ts (replaced by src/services/prediction/index.ts) - Remove PredictionMarket interface from src/types/index.ts (now in prediction module) - Type check and sidecar build both pass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2F-02): complete prediction consumer wiring plan Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2F): complete phase execution * docs(phase-2F): fix roadmap plan counts and completion status Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2G): research displacement migration phase * docs(2G): create displacement migration phase plans * feat(2G-01): implement displacement handler with UNHCR API pagination and aggregation - 40-entry COUNTRY_CENTROIDS map for geographic coordinates - UNHCR Population API pagination (10,000/page, 25-page guard) - Year fallback: current year to current-2 until data found - Per-country origin + asylum aggregation with unified merge - Global totals computation across all raw records - Flow corridor building sorted by refugees, capped by flowLimit - All int64 fields returned as String() per proto types - Graceful empty response on any failure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2G-01): wire displacement routes into gateway and rebuild sidecar - Import createDisplacementServiceRoutes and displacementHandler - Mount displacement routes alongside seismology, wildfire, climate, prediction - Sidecar bundle rebuilt with displacement included (31.0 KB) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2G-01): complete displacement handler plan - SUMMARY.md with execution metrics and decisions - STATE.md updated to 2G phase position - ROADMAP.md updated with 2G-01 plan progress Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2G-02): create displacement service module and rewire all consumers - Create src/services/displacement/index.ts as port/adapter using DisplacementServiceClient - Map proto int64 strings to numbers and GeoCoordinates to flat lat/lon - Preserve circuit breaker, presentation helpers (getDisplacementColor, formatPopulation, etc.) - Rewire App.ts, DisplacementPanel, MapContainer, DeckGLMap, conflict-impact, country-instability - Delete legacy src/services/unhcr.ts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore(2G-02): delete legacy endpoint and remove dead displacement types - Delete api/unhcr-population.js (replaced by displacement handler from 2G-01) - Remove DisplacementFlow, CountryDisplacement, UnhcrSummary from src/types/index.ts - All consumers now import from @/services/displacement - Sidecar rebuild, tsc, and full Vite build pass clean Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2G-02): complete displacement consumer wiring plan - SUMMARY.md with 2 task commits, decisions, deviation documentation - STATE.md updated: phase 2G complete, 02/02 plans done - ROADMAP.md updated with plan progress Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2G): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2H): research aviation migration phase * docs(2H): create aviation migration phase plans * feat(2H-01): implement aviation handler with FAA XML parsing and simulated delays - Install fast-xml-parser for server-side XML parsing (edge-compatible) - Create AviationServiceHandler with FAA NASSTATUS XML fetch and parse - Enrich US airports with MONITORED_AIRPORTS metadata (lat, lon, name, icao) - Generate simulated delays for non-US airports with rush-hour weighting - Map short-form strings to proto enums (FlightDelayType, FlightDelaySeverity, etc.) - Wrap flat lat/lon into GeoCoordinates for proto response - Graceful empty alerts on any upstream failure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2H-01): wire aviation routes into gateway and rebuild sidecar - Mount createAviationServiceRoutes in catch-all gateway alongside 5 existing domains - Import aviationHandler for route wiring - Rebuild sidecar-sebuf bundle with aviation routes included Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2H-01): complete aviation handler plan - Create 2H-01-SUMMARY.md with execution results - Update STATE.md position to 2H-01 with aviation decisions - Update ROADMAP.md progress for phase 2H (1/2 plans) - Mark DOMAIN-08 requirement as complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2H-02): create aviation service module and rewire all consumers - Create src/services/aviation/index.ts as port/adapter wrapping AviationServiceClient - Map proto enum strings to short-form (severity, delayType, region, source) - Unwrap GeoCoordinates to flat lat/lon, convert epoch-ms updatedAt to Date - Preserve circuit breaker with identical name string - Rewire Map, DeckGLMap, MapContainer, MapPopup, map-harness to import from @/services/aviation - Update barrel export: flights -> aviation - Delete legacy src/services/flights.ts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore(2H-02): delete legacy endpoint and remove dead aviation types - Delete api/faa-status.js (replaced by aviation handler in 2H-01) - Remove FlightDelaySource, FlightDelaySeverity, FlightDelayType, AirportRegion, AirportDelayAlert from src/types/index.ts - Preserve MonitoredAirport with inlined region type union - Full build (tsc + vite + sidecar) passes with zero errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2H-02): complete aviation consumer wiring plan Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2H): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2I): research phase domain * docs(2I): create phase plan * feat(2I-01): implement ResearchServiceHandler with 3 RPCs - arXiv XML parsing with fast-xml-parser (ignoreAttributes: false for attributes) - GitHub trending repos with primary + fallback API URLs - Hacker News Firebase API with 2-step fetch and bounded concurrency (10) - All RPCs return empty arrays on failure (graceful degradation) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2I-01): mount research routes in gateway and rebuild sidecar - Import createResearchServiceRoutes and researchHandler in catch-all gateway - Add research routes to allRoutes array (after aviation) - Sidecar bundle rebuilt (116.6 KB) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2I-01): complete research handler plan - SUMMARY.md with self-check passed - STATE.md updated to phase 2I, plan 01 of 02 - ROADMAP.md updated with plan 2I-01 complete - REQUIREMENTS.md: DOMAIN-05 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2I-02): create research service module and delete legacy code - Add src/services/research/index.ts with fetchArxivPapers, fetchTrendingRepos, fetchHackernewsItems backed by ResearchServiceClient - Re-export proto types ArxivPaper, GithubRepo, HackernewsItem (no enum mapping needed) - Circuit breakers wrap all 3 client calls with empty-array fallback - Delete legacy API endpoints: api/arxiv.js, api/github-trending.js, api/hackernews.js - Delete legacy service files: src/services/arxiv.ts, src/services/github-trending.ts, src/services/hackernews.ts - Remove arxiv, githubTrending, hackernews entries from API_URLS and REFRESH_INTERVALS in config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2I-02): complete research consumer wiring plan - SUMMARY.md documenting service module creation and 6 legacy file deletions - STATE.md updated: phase 2I complete, decisions recorded - ROADMAP.md updated: phase 2I marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2I): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2J): complete unrest migration research * docs(2J): create unrest migration phase plans * feat(2J-01): implement UnrestServiceHandler with ACLED + GDELT dual-fetch - Create handler with listUnrestEvents RPC proxying ACLED API and GDELT GEO API - ACLED fetch uses Bearer auth from ACLED_ACCESS_TOKEN env var, returns empty on missing token - GDELT fetch returns GeoJSON protest events with no auth required - Deduplication uses 0.5-degree grid + date key, preferring ACLED over GDELT on collision - Severity classification and event type mapping ported from legacy protests.ts - Sort by severity (high first) then recency (newest first) - Graceful degradation: returns empty events on any upstream failure * feat(2J-01): mount unrest routes in gateway and rebuild sidecar - Import createUnrestServiceRoutes and unrestHandler in catch-all gateway - Add unrest service routes to allRoutes array - Sidecar bundle rebuilt to include unrest endpoint - RPC routable at POST /api/unrest/v1/list-unrest-events * docs(2J-01): complete unrest handler plan - Create 2J-01-SUMMARY.md with execution results and self-check - Update STATE.md with phase 2J position, decisions, session continuity - Update ROADMAP.md with plan 01 completion status * feat(2J-02): create unrest service module with proto-to-legacy type mapping - Full adapter maps proto UnrestEvent to legacy SocialUnrestEvent shape - 4 enum mappers: severity, eventType, sourceType, confidence - fetchProtestEvents returns ProtestData with events, byCountry, highSeverityCount, sources - getProtestStatus infers ACLED configuration from response event sources - Circuit breaker wraps client call with empty fallback * feat(2J-02): update services barrel, remove vite proxies, delete legacy files - Services barrel: protests -> unrest re-export - Vite proxy entries removed: /api/acled, /api/gdelt-geo - Legacy files deleted: api/acled.js, api/gdelt-geo.js, src/services/protests.ts - Preserved: api/acled-conflict.js (conflict domain), SocialUnrestEvent type * docs(2J-02): complete unrest service module plan - SUMMARY.md created with full adapter pattern documentation - STATE.md updated: 2J-02 complete, decisions recorded - ROADMAP.md updated: Phase 2J marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2J): complete phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2K): complete conflict migration research * docs(2K): create phase plan * feat(2K-01): implement ConflictServiceHandler with 3 RPCs - listAcledEvents proxies ACLED API for battles/explosions/violence with Bearer auth - listUcdpEvents discovers UCDP GED API version dynamically, fetches backward with 365-day trailing window - getHumanitarianSummary proxies HAPI API with ISO-2 to ISO-3 country mapping - All RPCs have graceful degradation returning empty on failure * feat(2K-01): mount conflict routes in gateway and rebuild sidecar - Add createConflictServiceRoutes and conflictHandler imports to catch-all gateway - Spread conflict routes into allRoutes array (3 RPC endpoints) - Rebuild sidecar bundle with conflict endpoints included * docs(2K-01): complete conflict handler plan - Create 2K-01-SUMMARY.md with execution details and self-check - Update STATE.md: position to 2K-01, add 5 decisions - Update ROADMAP.md: mark 2K-01 complete (1/2 plans done) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2K-02): create conflict service module with 4-shape proto-to-legacy type mapping - Port/adapter mapping AcledConflictEvent -> ConflictEvent, UcdpViolenceEvent -> UcdpGeoEvent, HumanitarianCountrySummary -> HapiConflictSummary - UCDP classifications derived heuristically from GED events (deaths/events thresholds -> war/minor/none) - deduplicateAgainstAcled ported exactly with haversine + date + fatality matching - 3 circuit breakers for 3 RPCs, exports 5 functions + 2 group helpers + all legacy types * feat(2K-02): rewire consumer imports and delete 9 legacy conflict files - App.ts consolidated from 4 direct imports to single @/services/conflict import - country-instability.ts consolidated from 3 type imports to single ./conflict import - Deleted 4 API endpoints: acled-conflict.js, ucdp-events.js, ucdp.js, hapi.js - Deleted 4 service files: conflicts.ts, ucdp.ts, ucdp-events.ts, hapi.ts - Deleted 1 dead code file: conflict-impact.ts - UcdpGeoEvent preserved in src/types/index.ts (scope guard for map components) * docs(2K-02): complete conflict service module plan - SUMMARY.md with 4-shape proto adapter, consumer consolidation, 9 legacy deletions - STATE.md updated: Phase 2K complete (2/2 plans), progress ~100% - ROADMAP.md updated: Phase 2K marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-2K): complete conflict migration phase execution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2L): research maritime migration phase domain * docs(2L): create maritime migration phase plans * feat(2L-01): implement MaritimeServiceHandler with 2 RPCs - getVesselSnapshot proxies WS relay with wss->https URL conversion - Maps density/disruptions to proto shape with GeoCoordinates nesting - Disruption type/severity mapped from lowercase to proto enums - listNavigationalWarnings proxies NGA MSI broadcast warnings API - NGA military date parsing (081653Z MAY 2024) to epoch ms - Both RPCs gracefully degrade to empty on upstream failure - No caching (client-side polling manages refresh intervals) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2L-01): mount maritime routes in gateway and rebuild sidecar - Import createMaritimeServiceRoutes and maritimeHandler - Add maritime routes to allRoutes array in catch-all gateway - Sidecar bundle rebuilt (148.0 KB) with maritime endpoints - RPCs routable at /api/maritime/v1/get-vessel-snapshot and /api/maritime/v1/list-navigational-warnings Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(2L-01): complete maritime handler plan - SUMMARY.md with 2 task commits documented - STATE.md updated to 2L phase, plan 01/02 complete - ROADMAP.md progress updated for phase 2L - REQUIREMENTS.md: DOMAIN-06 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(2L-02): create maritime service module with hybrid fetch and polling/callback preservation - Port/adapter wrapping MaritimeServiceClient for proto RPC path - Full polling/callback architecture preserved from legacy ais.ts - Hybrid fetch: proto RPC for snapshot-only, raw WS relay for candidates - Proto-to-legacy type mapping for AisDisruptionEvent and AisDensityZone - Exports fetchAisSignals, initAisStream, disconnectAisStream, getAisStatus, isAisConfigured, registerAisCallback, unregisterAisCallback, AisPositionData * feat(2L-02): rewire consumer imports and delete 3 legacy maritime files - cable-activity.ts: fetch NGA warnings via MaritimeServiceClient.listNavigationalWarnings() with NgaWarning shape reconstruction from proto fields - military-vessels.ts: imports updated from './ais' to './maritime' - Services barrel: updated from './ais' to './maritime' - desktop-readiness.ts: service/api references updated to maritime handler paths - Deleted: api/ais-snapshot.js, api/nga-warnings.js, src/services/ais.ts - AisDisruptionEvent/AisDensityZone/AisDisruptionType preserved in src/types/index.ts * docs(2L-02): complete maritime service module plan - SUMMARY.md with hybrid fetch pattern, polling/callback preservation, 3 legacy files deleted - STATE.md updated: phase 2L complete, 5 decisions recorded - ROADMAP.md updated: 2L plans marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: bind globalThis.fetch in all sebuf service clients Generated sebuf clients store globalThis.fetch as a class property, then call it as this.fetchFn(). This loses the window binding and throws "Illegal invocation" in browsers. Pass { fetch: fetch.bind(globalThis) } to all 11 client constructors. Also includes vite.config.ts with all 10 migrated domain handlers registered in the sebuf dev server plugin. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate cyber + economic domains to sebuf (12/17) Cyber (Phase 2M): - Create handler aggregating 5 upstream sources (Feodo, URLhaus, C2Intel, OTX, AbuseIPDB) with dedup, GeoIP hydration, country centroid fallback - Create service module with CyberServiceClient + circuit breaker - Delete api/cyber-threats.js, api/cyber-threats.test.mjs, src/services/cyber-threats.ts Economic (Phase 2N) — consolidates 3 legacy services: - Create handler with 3 RPCs: getFredSeries (FRED API), listWorldBankIndicators (World Bank API), getEnergyPrices (EIA API) - Create unified service module replacing fred.ts, oil-analytics.ts, worldbank.ts - Preserve all exported functions/types for EconomicPanel and TechReadinessPanel - Delete api/fred-data.js, api/worldbank.js, src/services/fred.ts, src/services/oil-analytics.ts, src/services/worldbank.ts Both domains registered in vite.config.ts and api/[[...path]].ts. TypeScript check and vite build pass cleanly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate infrastructure domain to sebuf (13/17) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate market domain to sebuf (14/17) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate news domain to sebuf (15/17) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate intelligence domain to sebuf (16/17) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate military domain to sebuf (17/17) All 17 domains now have sebuf handlers registered in the gateway. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate intelligence services to sebuf client Rewire pizzint.ts, cached-risk-scores.ts, and threat-classifier.ts to use IntelligenceServiceClient instead of legacy /api/ fetch calls. Handler now preserves raw threat level in subcategory field. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate military theater posture to sebuf client Rewire cached-theater-posture.ts to use MilitaryServiceClient instead of legacy /api/theater-posture fetch. Adds theater metadata map for proto→legacy TheaterPostureSummary adapter. UI gracefully falls back to total counts when per-type breakdowns aren't available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: rewire country-intel to sebuf client Replace legacy fetch('/api/country-intel') with typed IntelligenceServiceClient.getCountryIntelBrief() RPC call in App.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate stablecoin-markets to sebuf (market domain) Add ListStablecoinMarkets RPC to market service. Port CoinGecko stablecoin peg-health logic from api/stablecoin-markets.js into the market handler. Rewire StablecoinPanel to use typed sebuf client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate etf-flows to sebuf (market domain) Add ListEtfFlows RPC to market service. Port Yahoo Finance BTC spot ETF flow estimation logic from api/etf-flows.js into the market handler. Rewire ETFFlowsPanel to use typed sebuf client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate worldpop-exposure to sebuf (displacement domain) Add GetPopulationExposure RPC to displacement service. Port country population data and radius-based exposure estimation from api/worldpop-exposure.js into the displacement handler. Rewire population-exposure.ts to use typed sebuf client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove superseded legacy edge functions Delete 4 legacy api/*.js files that are now fully replaced by sebuf handlers: - api/stablecoin-markets.js -> market/ListStablecoinMarkets - api/etf-flows.js -> market/ListEtfFlows - api/worldpop-exposure.js -> displacement/GetPopulationExposure - api/classify-batch.js -> intelligence/ClassifyEvent Remaining legacy files are still actively used by client code (stock-index, opensky, gdelt-doc, rss-proxy, summarize endpoints, macro-signals, tech-events) or are shared utilities. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: delete dead legacy files and unused API_URLS config Remove coingecko.js, debug-env.js, cache-telemetry.js, _cache-telemetry.js (all zero active consumers). Delete unused API_URLS export from base config. Update desktop-readiness market-panel metadata to reference sebuf paths. Remove dead CoinGecko dev proxy from vite.config.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: migrate stock-index and opensky to sebuf - Add GetCountryStockIndex RPC to market domain (Yahoo Finance + cache) - Fill ListMilitaryFlights stub in military handler (OpenSky with bounding box) - Rewire App.ts stock-index fetch to MarketServiceClient.getCountryStockIndex() - Delete api/stock-index.js and api/opensky.js edge functions - OpenSky client path unchanged (relay primary, vite proxy for dev) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * wip: sebuf legacy migration paused at phase 3/10 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(03): capture phase context * docs(state): record phase 3 context session * docs(03): research phase domain * docs(03): create phase plan — 5 plans in 2 waves * feat(03-01): commit wingbits migration (step 3) -- 3 RPCs added to military domain - Add GetAircraftDetails, GetAircraftDetailsBatch, GetWingbitsStatus RPCs - Rewire src/services/wingbits.ts to use MilitaryServiceClient - Update desktop-readiness.ts routes to match new RPC paths - Delete legacy api/wingbits/ edge functions (3 files) - Regenerate military service client/server TypeScript + OpenAPI docs * feat(03-02): add SummarizeArticle proto and implement handler - Create summarize_article.proto with request/response messages - Add SummarizeArticle RPC to NewsService proto - Implement full handler with provider dispatch (ollama/groq/openrouter) - Port cache key builder, deduplication, prompt builder, think-token stripping - Inline Upstash Redis helpers for edge-compatible caching * feat(03-01): migrate gdelt-doc to intelligence RPC + delete _ip-rate-limit.js - Add SearchGdeltDocuments RPC to IntelligenceService proto - Implement searchGdeltDocuments handler (port from api/gdelt-doc.js) - Rewire src/services/gdelt-intel.ts to use IntelligenceServiceClient - Delete legacy api/gdelt-doc.js edge function - Delete dead api/_ip-rate-limit.js (zero importers) - Regenerate intelligence service client/server TypeScript + OpenAPI docs * feat(03-02): rewire summarization client to NewsService RPC, delete 4 legacy files - Replace direct fetch to /api/{provider}-summarize with NewsServiceClient.summarizeArticle() - Preserve identical fallback chain: ollama -> groq -> openrouter -> browser T5 - Delete api/groq-summarize.js, api/ollama-summarize.js, api/openrouter-summarize.js - Delete api/_summarize-handler.js and api/_summarize-handler.test.mjs - Update desktop-readiness.ts to reference new sebuf route * feat(03-03): rewire MacroSignalsPanel to EconomicServiceClient + delete legacy - Replace fetch('/api/macro-signals') with EconomicServiceClient.getMacroSignals() - Add mapProtoToData() to convert proto optional fields to null for rendering - Delete legacy api/macro-signals.js edge function * feat(03-04): add ListTechEvents proto, city-coords data, and handler - Create list_tech_events.proto with TechEvent, TechEventCoords messages - Add ListTechEvents RPC to ResearchService proto - Extract 360-city geocoding table to api/data/city-coords.ts - Implement listTechEvents handler with ICS+RSS parsing, curated events, dedup, filtering - Regenerate TypeScript client/server from proto Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(03-01): complete wingbits + GDELT doc migration plan - Create 03-01-SUMMARY.md with execution results - Update STATE.md with plan 01 completion, steps 3-4 done - Update ROADMAP.md plan progress (2/5 plans complete) - Mark DOMAIN-10 requirement complete * docs(03-02): complete summarization migration plan - Create 03-02-SUMMARY.md with execution results - Update STATE.md position to step 6/10 - Update ROADMAP.md plan progress - Mark DOMAIN-09 requirement complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(03-04): rewire TechEventsPanel and App to ResearchServiceClient, delete legacy - Replace fetch('/api/tech-events') with ResearchServiceClient.listTechEvents() in TechEventsPanel - Replace fetch('/api/tech-events') with ResearchServiceClient.listTechEvents() in App.loadTechEvents() - Delete legacy api/tech-events.js (737 lines) - TypeScript compiles cleanly with no references to legacy endpoint Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(03-03): complete macro-signals migration plan - Create 03-03-SUMMARY.md with execution results - Mark DOMAIN-04 requirement complete in REQUIREMENTS.md * docs(03-04): complete tech-events migration plan - Add 03-04-SUMMARY.md with execution results - Update STATE.md: advance to plan 5/step 8, add decisions - Update ROADMAP.md: 4/5 plans complete for phase 03 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(03-05): add temporal baseline protos + handler with Welford's algorithm - GetTemporalBaseline RPC: anomaly detection with z-score thresholds - RecordBaselineSnapshot RPC: batch update via Welford's online algorithm - Inline mgetJson helper for Redis batch reads - Inline getCachedJson/setCachedJson Redis helpers - Generated TypeScript client/server + OpenAPI docs * feat(03-05): migrate temporal-baseline + tag non-JSON + final cleanup - Rewire temporal-baseline.ts to InfrastructureServiceClient RPCs - Delete api/temporal-baseline.js (migrated to sebuf handler) - Delete api/_upstash-cache.js (no importers remain) - Tag 6 non-JSON edge functions with // Non-sebuf: comment header - Update desktop-readiness.ts: fix stale cloudflare-outages reference * docs(03-05): complete temporal-baseline + non-JSON tagging + final cleanup plan - SUMMARY.md with Welford algorithm migration details - STATE.md updated: Phase 3 complete (100%) - ROADMAP.md updated: 5/5 plans complete * chore(03): delete orphaned ollama-summarize test after RPC migration Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-3): complete phase execution * docs(v1): create milestone audit report Audits all 13 phases of the v1 sebuf integration milestone. 12/13 phases verified (2L maritime missing VERIFICATION.md). 25/34 requirements satisfied, 6 superseded, 2 partial, 1 unsatisfied (CLEAN-03). All 17 domains wired end-to-end. Integration check passes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(v1): update audit — mark CLEAN-03/04 + MIGRATE-* as superseded CLEAN-03 superseded by port/adapter architecture (internal types intentionally decoupled from proto wire types). MIGRATE-01-05 superseded by direct cutover approach. DOMAIN-03 checkbox updated. Milestone status: tech_debt (no unsatisfied requirements). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(roadmap): add gap closure phase 4 — v1 milestone cleanup Closes all audit gaps: CLIENT-03 circuit breaker coverage, DOMAIN-03/06 verification gaps, documentation staleness, orphaned code cleanup. Fixes traceability table phase assignments to match actual roadmap phases. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore(03): commit generated NewsService OpenAPI specs + checkpoint update SummarizeArticle RPC was added during Phase 3 plan 02 but generated OpenAPI specs were not staged with that commit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(04): research phase domain * docs(04): create phase plan * docs(04-01): fix ROADMAP.md Phase 3 staleness and delete .continue-here.md - Change Phase 3 heading from IN PROGRESS to COMPLETE - Check off plans 03-03, 03-04, 03-05 (all were already complete) - Delete stale .continue-here.md (showed task 3/10 in_progress but all 10 done) * feat(04-02): add circuit breakers to seismology, wildfire, climate, maritime - Seismology: wrap listEarthquakes in breaker.execute with empty-array fallback - Wildfire: replace manual try/catch with breaker.execute for listFireDetections - Climate: replace manual try/catch with breaker.execute for listClimateAnomalies - Maritime: wrap proto getVesselSnapshot RPC in snapshotBreaker.execute, preserve raw relay fallback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(04-02): add circuit breakers to news summarization and GDELT intelligence - Summarization: wrap newsClient.summarizeArticle in summaryBreaker.execute for both tryApiProvider and translateText - GDELT: wrap client.searchGdeltDocuments in gdeltBreaker.execute, replace manual try/catch - Fix: include all required fields (tokens, reason, error, errorType, query) in fallback objects - CLIENT-03 fully satisfied: all 17 domains have circuit breaker coverage Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(04-01): create 2L-VERIFICATION.md, fix desktop-readiness.ts, complete service barrel - Create retroactive 2L-VERIFICATION.md with 12/12 must-haves verified - Fix map-layers-core and market-panel stale file refs in desktop-readiness.ts - Fix opensky-relay-cloud stale refs (api/opensky.js deleted) - Add missing barrel re-exports: conflict, displacement, research, wildfires, climate - Skip military/intelligence/news barrels (would cause duplicate exports) - TypeScript compiles cleanly with zero errors * docs(04-02): complete circuit breaker coverage plan - SUMMARY.md: 6 domains covered, CLIENT-03 satisfied, 1 deviation (fallback type fix) - STATE.md: Phase 4 plan 02 complete, position and decisions updated - ROADMAP.md: Phase 04 marked complete (2/2 plans) - REQUIREMENTS.md: CLIENT-03 marked complete Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(04-01): complete documentation fixes plan - Create 04-01-SUMMARY.md with execution results - Update STATE.md with Plan 01 completion and decisions - Update ROADMAP.md: Plan 04-01 checked, progress 1/2 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(phase-04): complete phase verification and fix tracking gaps ROADMAP.md Phase 4 status updated to Complete, 04-02 checkbox checked, progress table finalized. REQUIREMENTS.md coverage summary updated (27 complete, 0 partial/pending). STATE.md reflects verified phase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(military): migrate USNI fleet tracker to sebuf RPC Port all USNI Fleet Tracker parsing logic from api/usni-fleet.js into MilitaryService.GetUSNIFleetReport RPC with proto definitions, inline Upstash caching (6h fresh / 7d stale), and client adapter mapping. Deletes legacy edge function and _upstash-cache.js dependency. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: add proto validation annotations and split all 17 handler files into per-RPC modules Phase A: Added buf.validate constraints to ~25 proto files (~130 field annotations including required IDs, score ranges, coordinate bounds, page size limits). Phase B: Split all 17 domain handler.ts files into per-RPC modules with thin re-export handler.ts files. Extracted shared Redis cache helpers to api/server/_shared/redis.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add ADDING_ENDPOINTS guide and update Contributing section All JSON endpoints must use sebuf — document the complete workflow for adding RPCs to existing services and creating new services, including proto conventions, validation annotations, and generated OpenAPI docs. Update DOCUMENTATION.md Contributing section to reference the new guide and remove the deprecated "Adding a New API Proxy" pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add blank lines before lists to pass markdown lint (MD032) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: disambiguate duplicate city keys in city-coords Vercel's TypeScript check treats duplicate object keys as errors (TS1117). Rename 'san jose' (Costa Rica) -> 'san jose cr' and 'cambridge' (UK) -> 'cambridge uk' to avoid collision. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: resolve Vercel deployment errors — relocate hex-db + fix TS strict-null - Move military-hex-db.js next to handler (fixes Edge Function unsupported module) - Fix strict-null TS errors across 12 handler files (displacement, economic, infrastructure, intelligence, market, military, research, wildfire) - Add process declare to wildfire handler, prefix unused vars, cast types Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: convert military-hex-db to .ts for Edge Function compatibility Vercel Edge bundler can't resolve .js data modules from .ts handlers. Also remove unused _region variable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: inline military hex db as packed string to avoid Edge Function module error Vercel Edge bundler can't resolve separate data modules. Inline 20K hex IDs as a single concatenated string, split into Set at runtime. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove broken export { process } from 4 _shared files declare const is stripped in JS output, making export { process } reference nothing. No consumers import it — each handler file has its own declare. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: move server/ out of api/ to fix Vercel catch-all routing Vercel's file-based routing was treating api/server/**/*.ts as individual API routes, overriding the api/[[...path]].ts catch-all for multi-segment paths like /api/infrastructure/v1/list-service-statuses (3 segments). Moving to server/ at repo root removes the ambiguity. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: rename catch-all to [...path] — [[...path]] is Next.js-only syntax Vercel's native edge function routing only supports [...path] for multi-segment catch-all matching. The [[...path]] double-bracket syntax is a Next.js feature and was only matching single-segment paths. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add dynamic segment route for multi-segment API paths Vercel's native file-based routing for non-Next.js projects doesn't support [...path] catch-all matching multiple segments. Use explicit api/[domain]/v1/[rpc].ts which matches /api/{domain}/v1/{rpc} via standard single-segment dynamic routing that Vercel fully supports. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove conflicting [...path] catch-all — replaced by [domain]/v1/[rpc] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: widen CORS pattern to match all Vercel preview URL formats Preview URLs use elie-ab2dce63 not elie-habib-projects as the team slug. Broaden pattern to elie-[a-z0-9]+ to cover both. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update sidecar build script for new api/[domain]/v1/[rpc] path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: trigger Vercel rebuild for all variants Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #106 review — critical bugs, hardening, and cleanup Fixes from @koala73's code review: Critical: - C-1: Add max-size eviction (2048 cap) to GeoIP in-memory cache - C-2: Move type/source/severity filters BEFORE .slice(pageSize) in cyber handler - C-3: Atomic SET with EX in Redis helper (single Upstash REST call) - C-4: Add AbortSignal.timeout(30s) to LLM fetch in summarize-article High: - H-1: Add top-level try/catch in gateway with CORS-aware 500 response - H-3: Sanitize error messages — generic text for 5xx, passthrough for 4xx only - H-4: Add timeout (10s) + Redis cache (5min) to seismology handler - H-5: Add null guards (optional chaining) in seismology USGS feature mapping - H-6: Race OpenSky + Wingbits with Promise.allSettled instead of sequential fallback - H-8: Add Redis cache (5min TTL) to infrastructure service-status handler Medium: - M-12/M-13: Fix HAPI summary field mappings (iso3 from countryCode, internallyDisplaced) Infrastructure: - R-1: Remove .planning/ from git tracking, add to .gitignore - Port UCDP parallel page fetching from main branch (#198) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #106 review issues — hash, CORS, router, cache, LLM, GeoIP Fixes/improvements from PR #106 code review tracking issues: - #180: Replace 32-bit hash (Java hashCode/DJB2) with unified FNV-1a 52-bit hash in server/_shared/hash.ts, greatly reducing collision probability - #182: Cache router construction in Vite dev plugin — build once, invalidate on HMR changes to server/ files - #194: Add input length limits for LLM prompt injection (headlines 500 chars, title 500 chars, geoContext 2000 chars, max 10 headlines) - #195/#196: GeoIP AbortController — cancel orphaned background workers on timeout instead of letting them fire after response is sent - #198: Port UCDP partial-result caching from main — 10min TTL for partial results vs 6hr for complete, with in-memory fallback cache Proto codegen regenerated for displacement + conflict int64_encoding changes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: restore fast-xml-parser dependency needed by sebuf handlers Main branch removed fast-xml-parser in v2.5.1 (legacy edge functions no longer needed it), but sebuf handlers in aviation/_shared.ts and research/list-arxiv-papers.ts still import it for XML API parsing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: fix stale paths, version badge, and local-backend-audit for sebuf - ADDING_ENDPOINTS.md: fix handler paths from api/server/ to server/, fix import depth (4 levels not 5), fix gateway extension (.js not .ts) - DOCUMENTATION.md: update version badge 2.1.4 -> 2.5.1, fix broken ROADMAP.md links to .planning/ROADMAP.md, fix handler path reference - COMMUNITY-PROMOTION-GUIDE.md: add missing v2.5.1 to version table - local-backend-audit.md: rewrite for sebuf architecture — replace all stale api/*.js references with sebuf domain handler paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: use make commands, add generation-before-push warning, bump sebuf to v0.7.0 - ADDING_ENDPOINTS.md: replace raw `cd proto && buf ...` with `make check`, `make generate`, `make install`; add warning that `make generate` must run before pushing proto changes (links to #200) - Makefile: bump sebuf plugin versions from v0.6.0 to v0.7.0 - PR description also updated to use make commands Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: make install installs everything, document setup - Makefile: `make install` now installs buf, sebuf plugins, npm deps, and proto deps in one command; pin buf and sebuf versions as variables - ADDING_ENDPOINTS.md: updated prerequisites to show `make install` - DOCUMENTATION.md: updated Installation section with `make install` and generation reminder Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #106 re-review issues — timeouts, iso3, pagination, CORS, dedup - NEW-1: HAPI handler now returns ISO-3 code (via ISO2_TO_ISO3 lookup) instead of ISO-2 - NEW-3: Aviation FAA fetch now has AbortSignal.timeout(15s) - NEW-4: Climate Open-Meteo fetch now has AbortSignal.timeout(20s) - NEW-5: Wildfire FIRMS fetch now has AbortSignal.timeout(15s) - NEW-6: Seismology now respects pagination.pageSize (default 500) - NEW-9: Gateway wraps getCorsHeaders() in try/catch with safe fallback - NEW-10: Tech events dedup key now includes start year to avoid dropping yearly variants Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #106 round 2 — seismology crash, tsconfig, type contracts - Fix _req undefined crash in seismology handler (renamed to req) - Cache full earthquake set, slice on read (avoids cache pollution) - Add server/ to tsconfig.api.json includes (catches type errors at build) - Remove String() wrappers on numeric proto fields in displacement/HAPI - Fix hashString re-export not available locally in news/_shared.ts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: loadMarkets heatmap regression, stale test, Makefile playwright - Add finnhub_skipped + skip_reason to ListMarketQuotesResponse proto - Wire skipped signal through handler → adapter → App.loadMarkets - Fix circuit breaker cache conflating different symbol queries (cacheTtlMs: 0) - Use dynamic fetch wrapper so e2e test mocks intercept correctly - Update e2e test mocks from old endpoints to sebuf proto endpoints - Delete stale summarization-chain.test.mjs (imports deleted files) - Add install-playwright target to Makefile Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add missing proto fields to emptyStockFallback (build fix) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(quick-1): fix country fallback, ISO-2 contract, and proto field semantics - BLOCKING-1: Return undefined when country has no ISO3 mapping instead of wrong country data - BLOCKING-2: country_code field now returns ISO-2 per proto contract - MEDIUM-1: Rename proto fields from humanitarian to conflict-event semantics (populationAffected -> conflictEventsTotal, etc.) - Update client service adapter to use new field names - Regenerate TypeScript types from updated proto Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(quick-1): add in-memory cache + in-flight dedup to AIS vessel snapshot - HIGH-1: 10-second TTL cache matching client poll interval - Concurrent requests share single upstream fetch (in-flight dedup) - Follows same pattern as get-macro-signals.ts cache - No change to RPC response shape Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(quick-1): stub RPCs throw UNIMPLEMENTED, remove hardcoded politics, add tests - HIGH-2: listNewsItems, summarizeHeadlines, listMilitaryVessels now throw UNIMPLEMENTED - LOW-1: Replace hardcoded "Donald Trump" with date-based dynamic LLM context - LOW-1 extended: Also fix same issue in intelligence/get-country-intel-brief.ts (Rule 2) - MEDIUM-2: Add tests/server-handlers.test.mjs with 20 tests covering all review items Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(quick-2): remove 3 dead stub RPCs (ListNewsItems, SummarizeHeadlines, ListMilitaryVessels) - Delete proto definitions, handler stubs, and generated code for dead RPCs - Clean _shared.ts: remove tryGroq, tryOpenRouter, buildPrompt, dead constants - Remove 3 UNIMPLEMENTED stub tests from server-handlers.test.mjs - Regenerate proto codegen (buf generate) and OpenAPI docs - SummarizeArticle and all other RPCs remain intact Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR #106 review findings (stale snapshot, iso naming, scoring, test import) - Serve stale AIS snapshot on relay failure instead of returning undefined - Rename HapiConflictSummary.iso3 → iso2 to match actual ISO-2 content - Fix HAPI fallback scoring: use weight 3 for combined political violence (civilian targeting is folded in, was being underweighted at 0) - Extract deduplicateHeadlines to shared .mjs so tests import production code Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add coverage for PR106 iso2 and fallback regressions --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Elie Habib <elie.habib@gmail.com> |