mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-26 01:24:59 +02:00
7711e9de03
* feat(consumer-prices): add basket price monitoring domain
Adds end-to-end consumer price tracking to enable inflation monitoring
across key markets, starting with UAE essentials basket.
- consumer-prices-core/: companion scraping service with pluggable
acquisition providers (Playwright, Exa, Firecrawl, Parallel P0),
config-driven retailer YAML, Postgres schema, Redis snapshots
- proto/worldmonitor/consumer_prices/v1/: 6-RPC service definition
- api/consumer-prices/v1/[rpc].ts: Vercel edge route
- server/worldmonitor/consumer-prices/v1/: Redis-backed RPC handlers
- src/services/consumer-prices/: circuit breakers + bootstrap hydration
- src/components/ConsumerPricesPanel.ts: 5-tab panel (overview /
categories / movers / spread / health)
- scripts/seed-consumer-prices.mjs: Railway cron seed script
- Wire into bootstrap, health, panels, gateway, cache-keys, locale
* fix(consumer-prices): resolve all code review findings
P0: populate topCategories — categoryResult was fetched but never used.
Added buildTopCategories() helper with grouped CTE query that extracts
current_index and week-over-week pct per category.
P1 (4 fixes):
- aggregate: replace N+1 getBaselinePrice loop with single batch query
getBaselinePrices(ids[], date) via ANY($1) — eliminates 119 DB roundtrips
per basket run
- aggregate/computeValueIndex: was dividing all category floors by the same
arbitrary first baseline; now uses per-item floor price with per-item
baseline (same methodology as fixed index but with cheapest price)
- basket-series endpoint now seeded: added buildBasketSeriesSnapshot() to
worldmonitor.ts, /basket-series route in companion API, publish.ts writes
7d/30d/90d series per basket, seed script fetches and writes all three ranges
- scrape: call teardownAll() after each retailer run to close Playwright
browser; without this the Chromium process leaked on Railway
P2 (4 fixes):
- db/client: remove rejectUnauthorized: false — was bypassing TLS cert
validation on all non-localhost connections
- publish: seed-meta now writes { fetchedAt, recordCount } matching the format
expected by _seed-utils.mjs writeExtraKeyWithMeta (was writing { fetchedAt, key })
- products: remove unused getMatchedProductsForBasket — exact duplicate of
getBasketRows in aggregate.ts; never imported by anything
Snapshot type overhaul:
- Flatten WMOverviewSnapshot to match proto GetConsumerPriceOverviewResponse
(was nested under overview:{}; handlers read flat)
- All asOf fields changed from number to string (int64 → string per proto JSON)
- freshnessMin/parseSuccessRate null -> 0 defaults
- lastRunAt changed from epoch number to ISO string
- Mover items now include currentPrice and currencyCode
- emptyOverview/Movers/Spread/Freshness in seed script use String(Date.now())
* feat(consumer-prices): wire Exa search engine as acquisition backend for UAE retailers
Ports the proven Exa+summary price extraction from PR #1904 (seed-grocery-basket.mjs)
into consumer-prices-core as ExaSearchAdapter, replacing unvalidated Playwright CSS
scraping for all three UAE retailers (Carrefour, Lulu, Noon).
- New ExaSearchAdapter: discovers targets from basket YAML config (one per item),
calls Exa API with contents.summary to get AI-extracted prices, uses matchPrice()
regex (ISO codes + symbol fallback + CURRENCY_MIN guards) to extract AED amounts
- New db/queries/matches.ts: upsertProductMatch() + getBasketItemId() for auto-linking
scraped Exa results to basket items without a separate matching step
- scrape.ts: selects ExaSearchAdapter when config.adapter === 'exa-search'; after
insertObservation(), auto-creates canonical product and product_match (status: 'auto')
so aggregate.ts can compute indices immediately without manual review
- All three UAE retailer YAMLs switched to adapter: exa-search and enabled: true;
CSS extraction blocks removed (not used by search adapter)
- config/types.ts: adds 'exa-search' to adapter enum
* fix(consumer-prices): use EXA_API_KEYS (with fallback to EXA_API_KEY) matching PR #1904 pattern
* fix(consumer-prices): wire ConsumerPricesPanel in layout + fix movers limit:0 bug
Addresses Codex P1 findings on PR #1901:
- panel-layout.ts: import and createPanel('consumer-prices') so the panel
actually renders in finance/commodity variants where it is enabled in config
- consumer-prices/index.ts: limit was hardcoded 0 causing slice(0,0) to always
return empty risers/fallers after bootstrap is consumed; fixed to 10
* fix(consumer-prices): add categories snapshot to close P2 gap
consumer-prices:categories:ae:* was in BOOTSTRAP_KEYS but had no producer,
so the Categories tab always showed upstreamUnavailable.
- buildCategoriesSnapshot() in worldmonitor.ts — wraps buildTopCategories()
and returns WMCategoriesSnapshot matching ListConsumerPriceCategoriesResponse
- /categories route in consumer-prices-core API
- publish.ts writes consumer-prices:categories:{market}:{range} for 7d/30d/90d
- seed-consumer-prices.mjs fetches all three ranges from consumer-prices-core
and writes them to Redis alongside the other snapshots
P1 issues (snapshot structure mismatch + limit:0 movers) were already fixed
in earlier commits on this branch.
* fix(types): add variants? to PANEL_CATEGORY_MAP type
1 line
20 KiB
JSON
1 line
20 KiB
JSON
{"components":{"schemas":{"BasketPoint":{"description":"BasketPoint is a single data point in a basket index time series.","properties":{"date":{"description":"date is the ISO 8601 date string (YYYY-MM-DD).","type":"string"},"index":{"description":"index is the basket index value (base = 100).","format":"double","type":"number"}},"type":"object"},"CategorySnapshot":{"description":"CategorySnapshot holds price index data for a single product category.","properties":{"coveragePct":{"description":"coverage_pct is the percentage of basket items observed for this category.","format":"double","type":"number"},"currentIndex":{"description":"current_index is the current price index value (base = 100).","format":"double","type":"number"},"itemCount":{"description":"item_count is the number of observed products in this category.","format":"int32","type":"integer"},"momPct":{"description":"mom_pct is the month-over-month percentage change.","format":"double","type":"number"},"name":{"description":"name is the human-readable category label.","type":"string"},"slug":{"description":"slug is the machine-readable category identifier (e.g. \"eggs\", \"rice\").","type":"string"},"sparkline":{"items":{"description":"sparkline is an ordered sequence of index values for the selected range.","format":"double","type":"number"},"type":"array"},"wowPct":{"description":"wow_pct is the week-over-week percentage change.","format":"double","type":"number"}},"type":"object"},"Error":{"description":"Error is returned when a handler encounters an error. It contains a simple error message that the developer can customize.","properties":{"message":{"description":"Error message (e.g., 'user not found', 'database connection failed')","type":"string"}},"type":"object"},"FieldViolation":{"description":"FieldViolation describes a single validation error for a specific field.","properties":{"description":{"description":"Human-readable description of the validation violation (e.g., 'must be a valid email address', 'required field missing')","type":"string"},"field":{"description":"The field path that failed validation (e.g., 'user.email' for nested fields). For header validation, this will be the header name (e.g., 'X-API-Key')","type":"string"}},"required":["field","description"],"type":"object"},"GetConsumerPriceBasketSeriesRequest":{"description":"GetConsumerPriceBasketSeriesRequest parameters for time series data.","properties":{"basketSlug":{"description":"basket_slug selects the basket (e.g. \"essentials-ae\").","type":"string"},"marketCode":{"description":"market_code is the ISO 3166-1 alpha-2 market identifier.","type":"string"},"range":{"description":"range is one of \"7d\", \"30d\", \"90d\", \"180d\".","type":"string"}},"type":"object"},"GetConsumerPriceBasketSeriesResponse":{"description":"GetConsumerPriceBasketSeriesResponse contains the basket index time series.","properties":{"asOf":{"description":"as_of is the Unix millisecond timestamp of the snapshot.","format":"int64","type":"string"},"basketSlug":{"description":"basket_slug echoes the requested basket.","type":"string"},"currencyCode":{"description":"currency_code is the ISO 4217 currency code.","type":"string"},"essentialsSeries":{"items":{"$ref":"#/components/schemas/BasketPoint"},"type":"array"},"marketCode":{"description":"market_code echoes the requested market.","type":"string"},"range":{"description":"range echoes the requested range.","type":"string"},"upstreamUnavailable":{"description":"upstream_unavailable is true when the companion service could not be reached.","type":"boolean"},"valueSeries":{"items":{"$ref":"#/components/schemas/BasketPoint"},"type":"array"}},"type":"object"},"GetConsumerPriceFreshnessRequest":{"description":"GetConsumerPriceFreshnessRequest parameters for the freshness RPC.","properties":{"marketCode":{"description":"market_code is the ISO 3166-1 alpha-2 market identifier.","type":"string"}},"type":"object"},"GetConsumerPriceFreshnessResponse":{"description":"GetConsumerPriceFreshnessResponse describes feed health for all retailers.","properties":{"asOf":{"description":"as_of is the Unix millisecond timestamp of the snapshot.","format":"int64","type":"string"},"marketCode":{"description":"market_code echoes the requested market.","type":"string"},"overallFreshnessMin":{"description":"overall_freshness_min is the average freshness lag across all retailers.","format":"int32","type":"integer"},"retailers":{"items":{"$ref":"#/components/schemas/RetailerFreshnessInfo"},"type":"array"},"stalledCount":{"description":"stalled_count is the number of retailers with no recent successful scrape.","format":"int32","type":"integer"},"upstreamUnavailable":{"description":"upstream_unavailable is true when the companion service could not be reached.","type":"boolean"}},"type":"object"},"GetConsumerPriceOverviewRequest":{"description":"GetConsumerPriceOverviewRequest parameters for the overview RPC.","properties":{"basketSlug":{"description":"basket_slug selects which basket to use (e.g. \"essentials-ae\").","type":"string"},"marketCode":{"description":"market_code is the ISO 3166-1 alpha-2 market identifier (e.g. \"ae\").","type":"string"}},"type":"object"},"GetConsumerPriceOverviewResponse":{"description":"GetConsumerPriceOverviewResponse contains headline basket and coverage metrics.","properties":{"asOf":{"description":"as_of is the Unix millisecond timestamp of the snapshot.","format":"int64","type":"string"},"coveragePct":{"description":"coverage_pct is the fraction of basket items with current observations.","format":"double","type":"number"},"currencyCode":{"description":"currency_code is the ISO 4217 currency for price values.","type":"string"},"essentialsIndex":{"description":"essentials_index is the fixed basket index value (base = 100).","format":"double","type":"number"},"freshnessLagMin":{"description":"freshness_lag_min is the average minutes since last observation across all retailers.","format":"int32","type":"integer"},"marketCode":{"description":"market_code echoes the requested market.","type":"string"},"momPct":{"description":"mom_pct is the month-over-month percentage change in the essentials index.","format":"double","type":"number"},"retailerSpreadPct":{"description":"retailer_spread_pct is the basket cost spread between cheapest and most expensive retailer.","format":"double","type":"number"},"topCategories":{"items":{"$ref":"#/components/schemas/CategorySnapshot"},"type":"array"},"upstreamUnavailable":{"description":"upstream_unavailable is true when the companion service could not be reached.","type":"boolean"},"valueBasketIndex":{"description":"value_basket_index is the value basket index value (base = 100).","format":"double","type":"number"},"wowPct":{"description":"wow_pct is the week-over-week percentage change in the essentials index.","format":"double","type":"number"}},"type":"object"},"ListConsumerPriceCategoriesRequest":{"description":"ListConsumerPriceCategoriesRequest parameters for category listing.","properties":{"basketSlug":{"description":"basket_slug selects the basket scope.","type":"string"},"marketCode":{"description":"market_code is the ISO 3166-1 alpha-2 market identifier.","type":"string"},"range":{"description":"range is one of \"7d\", \"30d\", \"90d\", \"180d\".","type":"string"}},"type":"object"},"ListConsumerPriceCategoriesResponse":{"description":"ListConsumerPriceCategoriesResponse holds category-level price snapshots.","properties":{"asOf":{"description":"as_of is the Unix millisecond timestamp of the snapshot.","format":"int64","type":"string"},"categories":{"items":{"$ref":"#/components/schemas/CategorySnapshot"},"type":"array"},"marketCode":{"description":"market_code echoes the requested market.","type":"string"},"range":{"description":"range echoes the requested range.","type":"string"},"upstreamUnavailable":{"description":"upstream_unavailable is true when the companion service could not be reached.","type":"boolean"}},"type":"object"},"ListConsumerPriceMoversRequest":{"description":"ListConsumerPriceMoversRequest parameters for the movers RPC.","properties":{"categorySlug":{"description":"category_slug filters to a single category when set.","type":"string"},"limit":{"description":"limit caps the number of risers and fallers returned (default 10).","format":"int32","type":"integer"},"marketCode":{"description":"market_code is the ISO 3166-1 alpha-2 market identifier.","type":"string"},"range":{"description":"range is one of \"7d\", \"30d\", \"90d\".","type":"string"}},"type":"object"},"ListConsumerPriceMoversResponse":{"description":"ListConsumerPriceMoversResponse holds the top price movers.","properties":{"asOf":{"description":"as_of is the Unix millisecond timestamp of the snapshot.","format":"int64","type":"string"},"fallers":{"items":{"$ref":"#/components/schemas/PriceMover"},"type":"array"},"marketCode":{"description":"market_code echoes the requested market.","type":"string"},"range":{"description":"range echoes the requested range.","type":"string"},"risers":{"items":{"$ref":"#/components/schemas/PriceMover"},"type":"array"},"upstreamUnavailable":{"description":"upstream_unavailable is true when the companion service could not be reached.","type":"boolean"}},"type":"object"},"ListRetailerPriceSpreadsRequest":{"description":"ListRetailerPriceSpreadsRequest parameters for the retailer spread RPC.","properties":{"basketSlug":{"description":"basket_slug selects which basket to compare across retailers.","type":"string"},"marketCode":{"description":"market_code is the ISO 3166-1 alpha-2 market identifier.","type":"string"}},"type":"object"},"ListRetailerPriceSpreadsResponse":{"description":"ListRetailerPriceSpreadsResponse holds cheapest-basket rankings.","properties":{"asOf":{"description":"as_of is the Unix millisecond timestamp of the snapshot.","format":"int64","type":"string"},"basketSlug":{"description":"basket_slug echoes the requested basket.","type":"string"},"currencyCode":{"description":"currency_code is the ISO 4217 currency code.","type":"string"},"marketCode":{"description":"market_code echoes the requested market.","type":"string"},"retailers":{"items":{"$ref":"#/components/schemas/RetailerSpread"},"type":"array"},"spreadPct":{"description":"spread_pct is the percentage difference between cheapest and most expensive retailer.","format":"double","type":"number"},"upstreamUnavailable":{"description":"upstream_unavailable is true when the companion service could not be reached.","type":"boolean"}},"type":"object"},"PriceMover":{"description":"PriceMover describes a product with a notable upward or downward price move.","properties":{"category":{"description":"category is the product category slug.","type":"string"},"changePct":{"description":"change_pct is the signed percentage change over the selected window.","format":"double","type":"number"},"currencyCode":{"description":"currency_code is the ISO 4217 currency code.","type":"string"},"currentPrice":{"description":"current_price is the latest observed price.","format":"double","type":"number"},"productId":{"description":"product_id is the retailer product identifier.","type":"string"},"retailerSlug":{"description":"retailer_slug identifies the retailer where this move was observed.","type":"string"},"title":{"description":"title is the normalized product title.","type":"string"}},"type":"object"},"RetailerFreshnessInfo":{"description":"RetailerFreshnessInfo describes the operational health of one retailer feed.","properties":{"freshnessMin":{"description":"freshness_min is minutes since last successful observation.","format":"int32","type":"integer"},"lastRunAt":{"description":"last_run_at is the Unix millisecond timestamp of the last successful scrape.","format":"int64","type":"string"},"name":{"description":"name is the retailer display name.","type":"string"},"parseSuccessRate":{"description":"parse_success_rate is the fraction of pages parsed successfully (0–1).","format":"double","type":"number"},"slug":{"description":"slug is the retailer identifier.","type":"string"},"status":{"description":"status is one of \"ok\", \"stale\", \"failed\", \"unknown\".","type":"string"}},"type":"object"},"RetailerSpread":{"description":"RetailerSpread holds the basket cost breakdown for one retailer.","properties":{"basketTotal":{"description":"basket_total is the sum of matched basket item prices at this retailer.","format":"double","type":"number"},"currencyCode":{"description":"currency_code is the ISO 4217 currency code.","type":"string"},"deltaVsCheapest":{"description":"delta_vs_cheapest is the absolute price difference vs the cheapest retailer.","format":"double","type":"number"},"deltaVsCheapestPct":{"description":"delta_vs_cheapest_pct is the percentage difference vs the cheapest retailer.","format":"double","type":"number"},"freshnessMin":{"description":"freshness_min is minutes since the last successful scrape for this retailer.","format":"int32","type":"integer"},"itemCount":{"description":"item_count is the number of matched basket items observed.","format":"int32","type":"integer"},"name":{"description":"name is the retailer display name.","type":"string"},"slug":{"description":"slug is the retailer identifier.","type":"string"}},"type":"object"},"ValidationError":{"description":"ValidationError is returned when request validation fails. It contains a list of field violations describing what went wrong.","properties":{"violations":{"description":"List of validation violations","items":{"$ref":"#/components/schemas/FieldViolation"},"type":"array"}},"required":["violations"],"type":"object"}}},"info":{"title":"ConsumerPricesService API","version":"1.0.0"},"openapi":"3.1.0","paths":{"/api/consumer-prices/v1/get-consumer-price-basket-series":{"get":{"description":"GetConsumerPriceBasketSeries retrieves the basket index time series.","operationId":"GetConsumerPriceBasketSeries","parameters":[{"description":"market_code is the ISO 3166-1 alpha-2 market identifier.","in":"query","name":"market_code","required":false,"schema":{"type":"string"}},{"description":"basket_slug selects the basket (e.g. \"essentials-ae\").","in":"query","name":"basket_slug","required":false,"schema":{"type":"string"}},{"description":"range is one of \"7d\", \"30d\", \"90d\", \"180d\".","in":"query","name":"range","required":false,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/GetConsumerPriceBasketSeriesResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"GetConsumerPriceBasketSeries","tags":["ConsumerPricesService"]}},"/api/consumer-prices/v1/get-consumer-price-freshness":{"get":{"description":"GetConsumerPriceFreshness retrieves feed freshness and coverage health per retailer.","operationId":"GetConsumerPriceFreshness","parameters":[{"description":"market_code is the ISO 3166-1 alpha-2 market identifier.","in":"query","name":"market_code","required":false,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/GetConsumerPriceFreshnessResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"GetConsumerPriceFreshness","tags":["ConsumerPricesService"]}},"/api/consumer-prices/v1/get-consumer-price-overview":{"get":{"description":"GetConsumerPriceOverview retrieves headline basket indices and coverage metrics.","operationId":"GetConsumerPriceOverview","parameters":[{"description":"market_code is the ISO 3166-1 alpha-2 market identifier (e.g. \"ae\").","in":"query","name":"market_code","required":false,"schema":{"type":"string"}},{"description":"basket_slug selects which basket to use (e.g. \"essentials-ae\").","in":"query","name":"basket_slug","required":false,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/GetConsumerPriceOverviewResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"GetConsumerPriceOverview","tags":["ConsumerPricesService"]}},"/api/consumer-prices/v1/list-consumer-price-categories":{"get":{"description":"ListConsumerPriceCategories retrieves category summaries with sparklines.","operationId":"ListConsumerPriceCategories","parameters":[{"description":"market_code is the ISO 3166-1 alpha-2 market identifier.","in":"query","name":"market_code","required":false,"schema":{"type":"string"}},{"description":"basket_slug selects the basket scope.","in":"query","name":"basket_slug","required":false,"schema":{"type":"string"}},{"description":"range is one of \"7d\", \"30d\", \"90d\", \"180d\".","in":"query","name":"range","required":false,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ListConsumerPriceCategoriesResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"ListConsumerPriceCategories","tags":["ConsumerPricesService"]}},"/api/consumer-prices/v1/list-consumer-price-movers":{"get":{"description":"ListConsumerPriceMovers retrieves the largest upward and downward item price moves.","operationId":"ListConsumerPriceMovers","parameters":[{"description":"market_code is the ISO 3166-1 alpha-2 market identifier.","in":"query","name":"market_code","required":false,"schema":{"type":"string"}},{"description":"range is one of \"7d\", \"30d\", \"90d\".","in":"query","name":"range","required":false,"schema":{"type":"string"}},{"description":"limit caps the number of risers and fallers returned (default 10).","in":"query","name":"limit","required":false,"schema":{"format":"int32","type":"integer"}},{"description":"category_slug filters to a single category when set.","in":"query","name":"category_slug","required":false,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ListConsumerPriceMoversResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"ListConsumerPriceMovers","tags":["ConsumerPricesService"]}},"/api/consumer-prices/v1/list-retailer-price-spreads":{"get":{"description":"ListRetailerPriceSpreads retrieves cheapest-basket comparisons across retailers.","operationId":"ListRetailerPriceSpreads","parameters":[{"description":"market_code is the ISO 3166-1 alpha-2 market identifier.","in":"query","name":"market_code","required":false,"schema":{"type":"string"}},{"description":"basket_slug selects which basket to compare across retailers.","in":"query","name":"basket_slug","required":false,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ListRetailerPriceSpreadsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ValidationError"}}},"description":"Validation error"},"default":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Error"}}},"description":"Error response"}},"summary":"ListRetailerPriceSpreads","tags":["ConsumerPricesService"]}}}} |