Commit Graph

3 Commits

Author SHA1 Message Date
Fayez Bast
9d94ad36aa feat(climate+health):add shared air quality seed and mirrored health (#2634)
* feat(climate+health):add shared air quality seed and mirrored health/climate RPCs

* feat(climate+health):add shared air quality seed and mirrored health/climate RPCs

* fix(air-quality): address review findings — TTL, seed-health, FAST_KEYS, shared meta

- Raise CACHE_TTL from 3600 to 10800 (3× the 1h cron cadence; gold standard)
- Add health:air-quality to api/seed-health.js SEED_DOMAINS so monitoring dashboard tracks freshness
- Remove climateAirQuality and healthAirQuality from FAST_KEYS (large station payloads; load in slow batch)
- Point climateAirQuality SEED_META to same meta key as healthAirQuality (same seeder run, one source of truth)

* fix(bootstrap): move air quality keys to SLOW tier — large station payloads avoid critical-path batch

* fix(air-quality): fix malformed OpenAQ URL and remove from bootstrap until panel exists

- Drop deprecated first URL attempt (parameters=pm25, order_by=lastUpdated, sort=desc);
  use correct v3 params (parameters_id=2, sort_order=desc) directly — eliminates
  guaranteed 4xx retry cycle per page on 20-page crawl
- Remove climateAirQuality and healthAirQuality from BOOTSTRAP_CACHE_KEYS, SLOW_KEYS,
  and BOOTSTRAP_TIERS — no panel consumes these yet; adding thousands of station records
  to every startup bootstrap is pure payload bloat
- Remove normalizeAirQualityPayload helpers from bootstrap.js (no longer called)
- Update service wrappers to fetch via RPC directly; re-add bootstrap hydration
  when a panel actually needs it

* fix(air-quality): raise lock TTL to 3600s to cover 20-page crawl worst case

2 OpenAQ calls × 20 pages × (30s timeout × 3 attempts) = 3600s max runtime.
Previous 600s TTL allowed concurrent cron runs on any degraded upstream.

---------

Co-authored-by: Elie Habib <elie.habib@gmail.com>
2026-04-03 10:27:37 +04:00
Elie Habib
9a277233a0 fix(disease-outbreaks): use TGH exact lat/lng for map pins, fix density and location display (#2393)
* chore: redeploy to pick up WORLDMONITOR_VALID_KEYS fix

* fix(disease-outbreaks): use TGH lat/lng for map pins, fix location, add cases + date to tooltip

Root cause: all map pins collapsed to country centroids (getCountryCentroid) even though TGH
provides exact lat/lng per alert. Deduplication further collapsed all same-disease+country
alerts into one pin. 90-day lookback of ~1,600 TGH records was being reduced to ~10 pins.

- proto: add lat, lng, cases fields to DiseaseOutbreakItem (field 10/11/12)
- seed: preserve _lat/_lng/_cases from TGH bundle per alert
- seed: trim place_name to first comma-segment to avoid "Riga, Riga, Latvia" display
- seed: TGH items skip keyword filter (already disease-curated) and deduplication
- seed: raise cap to 150 TGH + 50 WHO/CDC/ONT (from flat 50 total)
- DeckGLMap: use item.lat/lng when non-zero, fall back to country centroid
- tooltip: add date and case count (when available) below source name
- sourceVersion bumped to v6

* chore: regenerate HealthService OpenAPI docs — add lat, lng, cases fields

* fix(mcp): address Greptile P2s — coordinate falsy coercion, dedup sort order, dead tooltip branch

- Use Number.isFinite() + null-coalescing (??) instead of || 0 for lat/lng; prevents
  treating equatorial coord 0 as "missing" (even though TGH already filters !rec.lat)
- DeckGLMap: use Number.isFinite + !== 0 guard instead of falsy && check for lat/lng
- Sort otherOutbreaks by publishedAt desc BEFORE deduplication so first-seen = most recent
- Remove redundant re-sort of dedupedOthers (already sorted above)
- Simplify metaHtml: always show date (publishedAt is always set); remove dead else-branch
2026-03-28 01:54:44 +04:00
Elie Habib
1e1f377078 feat(panels): Disease Outbreaks, Shipping Stress, Social Velocity, nuclear test site enrichment (#2375)
* feat(panels): Disease Outbreaks, Shipping Stress, Social Velocity, nuclear test site monitoring

- Add HealthService proto with ListDiseaseOutbreaks RPC (WHO + ProMED RSS)
- Add GetShippingStress RPC to SupplyChainService (Yahoo Finance carrier ETFs)
- Add GetSocialVelocity RPC to IntelligenceService (Reddit r/worldnews + r/geopolitics)
- Enrich earthquake seed with Haversine nuclear test-site proximity scoring
- Add 5 nuclear test sites to NUCLEAR_FACILITIES (Punggye-ri, Lop Nur, Novaya Zemlya, Nevada NTS, Semipalatinsk)
- Add shipping stress + social velocity seed loops to ais-relay.cjs
- Add seed-disease-outbreaks.mjs Railway cron script
- Wire all new RPCs: edge functions, handlers, gateway cache tiers, health.js STANDALONE_KEYS/SEED_META

* fix(relay): apply gold standard retry/TTL-extend pattern to shipping-stress and social-velocity seeders

* fix(review): address all PR #2375 review findings

- health.js: shippingStress maxStaleMin 30→45 (3x interval), socialVelocity 20→30 (3x interval)
- health.js: remove shippingStress/diseaseOutbreaks/socialVelocity from ON_DEMAND_KEYS (relay/cron seeds, not on-demand)
- cache-keys.ts: add shippingStress, diseaseOutbreaks, socialVelocity to BOOTSTRAP_CACHE_KEYS
- ais-relay.cjs: stressScore formula 50→40 (neutral market = moderate, not elevated)
- ais-relay.cjs: fetchedAt Date.now() (consistent with other seeders)
- ais-relay.cjs: deduplicate cross-subreddit article URLs in social velocity loop
- seed-disease-outbreaks.mjs: WHO URL → specific DON RSS endpoint (not dead general news feed)
- seed-disease-outbreaks.mjs: validate() requires outbreaks.length >= 1 (reject empty array)
- seed-disease-outbreaks.mjs: stable id using hash(link) not array index
- seed-disease-outbreaks.mjs: RSS regexes use [\s\S]*? for CDATA multiline content
- seed-earthquakes.mjs: Lop Nur coordinates corrected (41.39,89.03 not 41.75,88.35)
- seed-earthquakes.mjs: sourceVersion bumped to usgs-4.5-day-nuclear-v1
- earthquake.proto: fields 8-11 marked optional (distinguish not-enriched from enriched=false/0)
- buf generate: regenerate seismology service stubs

* revert(cache-keys): don't add new keys to bootstrap without frontend consumers

* fix(panels): address all P1/P2/P3 review findings for PR #2375

- proto: add INT64_ENCODING_NUMBER annotation + sebuf import to get_shipping_stress.proto (run make generate)
- bootstrap: register shippingStress (fast), socialVelocity (fast), diseaseOutbreaks (slow) in api/bootstrap.js + cache-keys.ts
- relay: update WIDGET_SYSTEM_PROMPT with new bootstrap keys and live RPCs for health/supply-chain/intelligence
- seeder: remove broken ProMED feed URL (promedmail.org/feed/ returns HTML 404); add 500K size guard to fetchRssItems; replace private COUNTRY_CODE_MAP with shared geo-extract.mjs; remove permanently-empty location field; bump sourceVersion to who-don-rss-v2
- handlers: remove dead .catch from all 3 new RPC handlers; fix stressLevel fallback to low; fix fetchedAt fallback to 0
- services: add fetchShippingStress, disease-outbreaks.ts, social-velocity.ts with getHydratedData consumers
2026-03-27 22:33:45 +04:00