1 Commits

Author	SHA1	Message	Date
Elie Habib	7711e9de03	feat(consumer-prices): add basket price monitoring domain (#1901) ... * feat(consumer-prices): add basket price monitoring domain Adds end-to-end consumer price tracking to enable inflation monitoring across key markets, starting with UAE essentials basket. - consumer-prices-core/: companion scraping service with pluggable acquisition providers (Playwright, Exa, Firecrawl, Parallel P0), config-driven retailer YAML, Postgres schema, Redis snapshots - proto/worldmonitor/consumer_prices/v1/: 6-RPC service definition - api/consumer-prices/v1/[rpc].ts: Vercel edge route - server/worldmonitor/consumer-prices/v1/: Redis-backed RPC handlers - src/services/consumer-prices/: circuit breakers + bootstrap hydration - src/components/ConsumerPricesPanel.ts: 5-tab panel (overview / categories / movers / spread / health) - scripts/seed-consumer-prices.mjs: Railway cron seed script - Wire into bootstrap, health, panels, gateway, cache-keys, locale * fix(consumer-prices): resolve all code review findings P0: populate topCategories — categoryResult was fetched but never used. Added buildTopCategories() helper with grouped CTE query that extracts current_index and week-over-week pct per category. P1 (4 fixes): - aggregate: replace N+1 getBaselinePrice loop with single batch query getBaselinePrices(ids[], date) via ANY($1) — eliminates 119 DB roundtrips per basket run - aggregate/computeValueIndex: was dividing all category floors by the same arbitrary first baseline; now uses per-item floor price with per-item baseline (same methodology as fixed index but with cheapest price) - basket-series endpoint now seeded: added buildBasketSeriesSnapshot() to worldmonitor.ts, /basket-series route in companion API, publish.ts writes 7d/30d/90d series per basket, seed script fetches and writes all three ranges - scrape: call teardownAll() after each retailer run to close Playwright browser; without this the Chromium process leaked on Railway P2 (4 fixes): - db/client: remove rejectUnauthorized: false — was bypassing TLS cert validation on all non-localhost connections - publish: seed-meta now writes { fetchedAt, recordCount } matching the format expected by _seed-utils.mjs writeExtraKeyWithMeta (was writing { fetchedAt, key }) - products: remove unused getMatchedProductsForBasket — exact duplicate of getBasketRows in aggregate.ts; never imported by anything Snapshot type overhaul: - Flatten WMOverviewSnapshot to match proto GetConsumerPriceOverviewResponse (was nested under overview:{}; handlers read flat) - All asOf fields changed from number to string (int64 → string per proto JSON) - freshnessMin/parseSuccessRate null -> 0 defaults - lastRunAt changed from epoch number to ISO string - Mover items now include currentPrice and currencyCode - emptyOverview/Movers/Spread/Freshness in seed script use String(Date.now()) * feat(consumer-prices): wire Exa search engine as acquisition backend for UAE retailers Ports the proven Exa+summary price extraction from PR #1904 (seed-grocery-basket.mjs) into consumer-prices-core as ExaSearchAdapter, replacing unvalidated Playwright CSS scraping for all three UAE retailers (Carrefour, Lulu, Noon). - New ExaSearchAdapter: discovers targets from basket YAML config (one per item), calls Exa API with contents.summary to get AI-extracted prices, uses matchPrice() regex (ISO codes + symbol fallback + CURRENCY_MIN guards) to extract AED amounts - New db/queries/matches.ts: upsertProductMatch() + getBasketItemId() for auto-linking scraped Exa results to basket items without a separate matching step - scrape.ts: selects ExaSearchAdapter when config.adapter === 'exa-search'; after insertObservation(), auto-creates canonical product and product_match (status: 'auto') so aggregate.ts can compute indices immediately without manual review - All three UAE retailer YAMLs switched to adapter: exa-search and enabled: true; CSS extraction blocks removed (not used by search adapter) - config/types.ts: adds 'exa-search' to adapter enum * fix(consumer-prices): use EXA_API_KEYS (with fallback to EXA_API_KEY) matching PR #1904 pattern * fix(consumer-prices): wire ConsumerPricesPanel in layout + fix movers limit:0 bug Addresses Codex P1 findings on PR #1901: - panel-layout.ts: import and createPanel('consumer-prices') so the panel actually renders in finance/commodity variants where it is enabled in config - consumer-prices/index.ts: limit was hardcoded 0 causing slice(0,0) to always return empty risers/fallers after bootstrap is consumed; fixed to 10 * fix(consumer-prices): add categories snapshot to close P2 gap consumer-prices:categories:ae:* was in BOOTSTRAP_KEYS but had no producer, so the Categories tab always showed upstreamUnavailable. - buildCategoriesSnapshot() in worldmonitor.ts — wraps buildTopCategories() and returns WMCategoriesSnapshot matching ListConsumerPriceCategoriesResponse - /categories route in consumer-prices-core API - publish.ts writes consumer-prices:categories:{market}:{range} for 7d/30d/90d - seed-consumer-prices.mjs fetches all three ranges from consumer-prices-core and writes them to Redis alongside the other snapshots P1 issues (snapshot structure mismatch + limit:0 movers) were already fixed in earlier commits on this branch. * fix(types): add variants? to PANEL_CATEGORY_MAP type	2026-03-20 17:08:22 +04:00