mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
a8f8c0aa61
* feat(economic): add ME grocery basket price index Adds a grocery basket price comparison panel for 9 Middle East countries (UAE, KSA, Qatar, Kuwait, Bahrain, Oman, Egypt, Jordan, Lebanon) using EXA AI to discover prices from regional e-commerce sites (Carrefour, Lulu, Noon, Amazon) and Yahoo Finance for FX rates. - proto: ListGroceryBasketPrices RPC with CountryBasket/GroceryItemPrice messages - seed: seed-grocery-basket.mjs, 90 EXA calls/run, 150ms delay, hardcoded FX fallbacks for pegged GCC currencies, 6h TTL - handler: seed-only RPC reading economic:grocery-basket:v1 - gateway: static cache tier for the new route - bootstrap/health: groceryBasket key in SLOW tier, 720min stale threshold - frontend: GroceryBasketPanel with scrollable table, cheapest/priciest column highlighting, styles moved to panels.css - panel disabled by default until seed is run on Railway * fix(generated): restore @ts-nocheck in economic service codegen * fix(grocery-basket): tighten seed health staleness and seed script robustness - Set maxStaleMin to 360 (6h) matching CACHE_TTL so health alerts on first missed run - Use ?? over || for FX fallback to handle 0-value rates correctly - Add labeled regex patterns with bare-number warning in extractPrice - Replace conditional delay logic with unconditional per-item sleep * fix(grocery-basket): fix EXA API format and price extraction after live validation - Use contents.summary format (not top-level summary) — previous format returned no data - Support EXA_API_KEYS (comma-separated) in addition to EXA_API_KEY - Extract price from plain-text summary string (EXA returns text, not JSON schema) - Remove bare-number fallback — too noisy (matched "500" from "pasta 500g" as SAR 500) - Fix LBP FX rate zero-guard: use fallback when Yahoo returns 0 for ultra-low-value currencies Validated locally: 9 countries seeded, Redis write confirmed, ~111s runtime * fix(grocery-basket): validate extracted currency against expected country currency - matchPrice now returns the currency code alongside the price - extractPrice rejects results where currency != expected country currency (prevents AED prices from being treated as JOD prices on gcc.luluhypermarket.com) - Tighten item queries (white granulated sugar, spaghetti pasta, etc.) to reduce irrelevant product matches like Stevia on sugar queries - Replace Jordan's gcc.luluhypermarket.com (GCC-only) with carrefour.jo + ounasdelivery.com - Sync scripts/shared/grocery-basket.json * feat(bigmac): add Big Mac Index seed + drop grocery basket includeDomains Grocery basket: - Remove includeDomains restriction — EXA neural search finds better sources than hardcoded domain lists; currency validation prevents contamination - Tighten query strings (supermarket retail price suffix) Big Mac seed (scripts/seed-bigmac.mjs): - Two-tier search: specialist sites (theburgerindex.com, eatmyindex.com) first, fall back to open search for countries without per-country indexed pages - Handle thousands-separator prices (480,000 LBP) - Accept USD prices from cost-of-living index sites as fallback - Exclude ranking/average pages (Numbeo country_price_rankings, Expatistan) - Validated live: 7/9 countries with confirmed prices UAE=19AED, KSA=19SAR, QAR=22QAR, KWD=1.4KWD, EGP=135EGP, JOD=3JOD, LBP=480kLBP * feat(economic): expand grocery basket to 24 global countries, drop Big Mac tier-2 search Grocery basket: extend coverage from 9 MENA to 24 countries across all regions (US, UK, DE, FR, JP, CN, IN, AU, CA, BR, MX, ZA, TR, NG, KR, SG, PK, AE, SA, EG, KE, AR, ID, PH). Add FX fallbacks and fxSymbols for all 23 new currencies. CCY regex in seed script updated to match all supported currency codes. Big Mac: remove tier-2 open search (too noisy, non-specialist pages report combo prices or global averages). Specialist sites only (theburgerindex.com, eatmyindex.com) for clean per-country data. * feat(bigmac): wire Big Mac Index RPC, proto, bootstrap, health Add ListBigMacPrices RPC end-to-end: - proto/list_bigmac_prices.proto: BigMacCountryPrice + request/response - service.proto: register ListBigMacPrices endpoint (GET /list-bigmac-prices) - buf generate: regenerate service_server.ts + all client stubs - server/list-bigmac-prices.ts: seed-only handler reads economic:bigmac:v1 - handler.ts: wire listBigMacPrices into EconomicServiceHandler - api/bootstrap.js: bigmac key in BOOTSTRAP_CACHE_KEYS + SLOW_KEYS - api/health.js: bigmac key in BOOTSTRAP_KEYS + SEED_META (maxStaleMin: 1440) - _bootstrap-cache-key-refs.ts: groceryBasket + bigmac refs * feat(bigmac): add BigMacPanel + register in panel layout BigMacPanel renders a country-by-country Big Mac price table sorted by USD price (cheapest/most expensive highlighted). Wired into bootstrap hydration, refresh scheduler, and panel registry. Registered in panels.ts (enabled: false, to be flipped once seed data is verified). Also updates grocery basket i18n from ME-specific to global wording. * fix(bigmac): register bigmac in cache-keys and RPC_CACHE_TIER Add bigmac to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in server/_shared/cache-keys.ts, and to RPC_CACHE_TIER (static tier) in gateway.ts. Both were caught by bootstrap and RPC tier parity tests. * fix(generated): restore @ts-nocheck in all generated service files after buf regenerate buf generate does not emit @ts-nocheck. Previous convention restores it manually post-generate to suppress strict type errors in generated code. * fix(grocery-basket): restore includeDomains per country, add userLocation, fix currency symbol parsing Root cause of 0-item countries (UK, JP, IN, NG): removing includeDomains caused EXA neural search to return USD-priced global comparison pages (Numbeo, Tridge, Expatistan) which currency validation correctly rejected. Fixes: - Add per-country sites[] in grocery-basket.json (researched local supermarket/retailer domains: tesco.com, kaufland.de, bigbasket.com, etc.) - Pass includeDomains: country.sites to restrict EXA to local retailers - Pass userLocation: country.code (ISO) to bias results to target country - Add currency symbol fallback regex (£→GBP, €→EUR, ¥→JPY, ₹→INR, ₩→KRW, ₦→NGN, R$→BRL) — sites like BigBasket use ₹ not INR - Summary query now explicitly requests ISO currency code - Simplify item queries (drop country name — context from domains) Smoke test results: UK sugar → GBP 1.09 (tesco.com) ✓ IN rice → ₹66 (bigbasket.com) ✓ JP rice → JPY 500 (kakaku.com) ✓ * fix(grocery-basket): add Firecrawl fallback, parallel items, bulk caps, currency floors - Add Firecrawl as JS-SPA fallback after EXA (handles noon.com, coupang, daraz, tokopedia, lazada) - Parallelize item fetching per country with 200ms stagger: runtime 38min to ~4.5min - Add CURRENCY_MIN floors (NGN:50, IDR:500, KRW:1000, etc.) to reject product codes as prices - Add ITEM_USD_MAX bulk caps (sugar:8USD, salt:5USD, rice:6USD, etc.) applied to both EXA and Firecrawl - Fix SA: use noon.com/saudi-en + carrefour.com.sa (removes luluhypermarket cross-country pollution) - Fix EG: use carrefouregypt.com + spinneys.com.eg + seoudi.com (removes GCC luluhypermarket) - Expand sites for DE, MX, ZA, TR, NG, KR, IN, PK, AR, ID, PH to improve coverage - Sync scripts/shared/grocery-basket.json with shared/grocery-basket.json * fix(grocery-basket): address PR review comments P1+P2 P1 - fix ranking with incomplete data: only include countries with >=70% item coverage (>=7/10) in cheapest/mostExpensive ranking — prevents a country with 4/10 items appearing cheapest due to missing data P1 - fix regex false-match on pack sizes / weights: try currency-first pattern (SAR 8.99) before number-first to avoid matching pack counts; use matchAll and take last match P2 - mark seed-miss responses as non-cacheable: add upstream_unavailable to proto + return true on empty seed so gateway sets Cache-Control: no-store on cold deploy * fix(generated): update EconomicService OpenAPI docs for upstream_unavailable field