* feat(commodities): expand tracking to cover agricultural and coal futures
Adds 9 new commodity symbols to cover the price rally visible in our
intelligence feeds: Newcastle Coal (MTF=F), Wheat (ZW=F), Corn (ZC=F),
Soybeans (ZS=F), Rough Rice (ZR=F), Coffee (KC=F), Sugar (SB=F),
Cocoa (CC=F), and Cotton (CT=F).
Also fixes ais-relay seeder to use display names from commodities.json
instead of raw symbols, so seeded data is self-consistent.
* fix(commodities): gold standard cache, 3-col grid, cleanup
- Add upstashExpire on zero-quotes failure path so bootstrap key TTL
extends during Yahoo outages (gold standard pattern)
- Remove unreachable fallback in retry loop (COMMODITY_META always has
the symbol since it mirrors COMMODITY_SYMBOLS)
- Switch commodities panel to 3-column grid (19 items → ~7 rows vs 10)
* fix(seeders): apply gold standard TTL-extend+retry pattern to Aviation, NOTAM, Cyber, PositiveEvents
* feat(consumer-prices): default to All — global comparison table as landing view
- DEFAULT_MARKET = 'all' so panel opens with the global view
- 🌍 All pill added at front of market bar
- All view fetches all 9 markets in parallel via fetchAllMarketsOverview()
and renders a comparison table: Market / Index / WoW / Spread / Updated
- Clicking any market row drills into that market's full tab view
- SINGLE_MARKETS exported for use in All-view iteration
- CSS: .cp-global-table and row styles
FRED_TTL was 1h but seed runs daily — key expires 23h before next run,
causing persistent EMPTY CRIT. Same pattern as consumer-prices publish.ts
(93600s TTL). maxStaleMin bumped from 90 to 1500 to match daily cadence.
TRADE_TTL (6h) matched the cron cadence exactly, causing the customs
revenue data key to expire seconds before the next seed run, resulting
in recurring EMPTY CRITs on the health endpoint. Monthly Treasury data
doesn't need 6h TTL; 24h (CUSTOMS_TTL) matches the maxStaleMin threshold.
Adds BAMLH0A0HYM2, ICSA, MORTGAGE30US, GSCPI to FRED_SERIES array.
These were in the server allowlist and frontend config but absent from
the seeder, causing empty data in production on every request.
Closes#2041
* feat(docker): enable Redis RDB persistence, add seed-orchestrator to supervisord, copy scripts into image
* feat(orchestrator): add prefixed logger utility
* feat(orchestrator): add seed-meta read/write helpers
* feat(orchestrator): add child process runner with timeout support
* feat(orchestrator): add central seed catalog with 42 seeders
* feat(orchestrator): implement seed orchestrator with tiered scheduling
Adds scripts/seed-orchestrator.mjs with:
- classifySeeders: classify seeders into active/skipped by env vars
- buildStartupSummary: human-readable startup report
- Tiered cold start (hot/warm/cold/frozen with per-tier concurrency)
- Freshness check via seed-meta keys before running stale seeders
- Steady-state scheduling with setTimeout-based recurring timers
- Overlap protection, retry-after-60s, consecutive failure demotion
- Global concurrency cap of 5 with queue-based overflow
- Graceful shutdown on SIGTERM/SIGINT (15s drain timeout)
- Meta writing for null-metaKey seeders to seed-meta:orchestrator:{name}
* fix(seeds): use local API for warm-ping seeders in Docker mode
* fix(orchestrator): allow ACLED email+password as alternative to access token
* feat(wmsm): add seed manager CLI scaffold with help, catalog, and checks
* feat(wmsm): implement status command with freshness display
* feat(wmsm): implement schedule command with next-run estimates
* feat(wmsm): implement refresh command with single and --all modes
* feat(wmsm): implement flush and logs commands
* fix(wmsm): auto-detect docker vs podman runtime
* feat(orchestrator): extract pure scheduling functions and add test harness
* feat(orchestrator): add SEED_TURBO=real|dry mode with compressed intervals
* feat(orchestrator): add SEED_TURBO env passthrough and fix retry log message
* fix(consumer-prices): harden scrape/aggregate/publish pipeline
- scrape: treat 0-product parse as error (increments errorsCount, skips
pagesSucceeded) so noon_grocery_ae missing eggs_12/tomatoes_1kg marks
the run partial instead of completed
- publish: fix freshData gate (freshnessMin >= 0) so a scrape finishing
at exactly 0 min lag still advances seed-meta
- aggregate: wrap per-basket aggregation in try/catch so one failing
basket does not skip remaining baskets; re-throw if any failed
- seed-consumer-prices.mjs: require --force flag to prevent accidentally
stomping publish.ts 26h TTLs with short 10-60min fallback TTLs
* fix(consumer-prices): correct basket comparison with intersection + dedup
Both aggregate.ts and the retailer spread snapshot were summing ALL
matched SKUs per retailer without deduplication, making Carrefour
appear most expensive simply because it had more matched products
(31 "items" vs Noon's 20 for a 12-item basket).
Fixes:
- aggregate.ts retailer_spread_pct: deduplicate per (retailer, basketItem)
taking cheapest price, then only compare on items all retailers carry
- worldmonitor.ts buildRetailerSpreadSnapshot: same dedup + intersection
logic in SQL — one best_price per (retailer, basket_item), common_items
CTE filters to items every active retailer covers
- exa-search.ts parseListing: log whether Exa returned 0 results or
results with no extractable price, to distinguish the two failure modes
* fix(consumer-prices-panel): correct parse rate display, category names, and freshness colors
- parseSuccessRate is stored as 0-100 but UI was doing *100 again (shows 10000%)
- Category name builder converts snake_case to Title Case (Cooking_oil → Cooking Oil)
- Add missing cp-fresh--ok/warn/stale/unknown CSS classes (freshness labels had no color)
- Add border-radius to stat cards and range buttons; add font-family to range buttons
- Add padding + bottom border to cp-range-bar for visual separation
* fix(consumer-prices): gate overview spread_pct query to last 2 days
buildOverviewSnapshot queried retailer_spread_pct with no recency
filter, so ORDER BY metric_date DESC LIMIT 1 would serve an
arbitrarily old row when today's aggregate run omitted a write
(no retailer intersection). Add INTERVAL '2 days' cutoff — covers
24h cron cadence plus scheduling drift. Falls through to 0 (→ UI
shows '—') when no recent value exists.
Sanctions seed has been failing since PR #2008 with:
ERR_MODULE_NOT_FOUND: Cannot find package 'sax'
PR #2008 replaced fast-xml-parser with SAX streaming but only updated
the root package.json, not scripts/package.json which is the Railway
container manifest. Railway runs npm ci from scripts/ so sax was never
installed. Add sax ^1.6.0 and remove the now-unused fast-xml-parser.
Also raise consumer-prices SEED_META maxStaleMin from 90-120 to 1500 min.
publish.ts runs once daily at 02:30 UTC; all five consumer-prices keys
were permanently STALE_SEED for 22+ hours/day after the daily run.
1500 min (25h) = 24h cadence + 1h grace before warning.
* fix(relay): add global Wingbits fallback for callsign-only index misses
When a callsign search hits the relay but the flight isn't in any
recent viewport (index miss), the relay now attempts a worldwide
Wingbits API call filtered by callsign server-side.
Previously: index miss → immediate empty response
Now: index miss → global bbox call → filter by callsign → return matches
Fallback is graceful: timeouts/errors return empty rather than 5xx.
Matched flights are also written into the index for subsequent queries.
* fix(aviation): try Wingbits before OpenSky for callsign-only searches
Commercial flights like UAE20, THY6260 are Wingbits-exclusive and
invisible to OpenSky receivers. The old order (OpenSky first) wasted
10s+ on OpenSky global states/all (rate-limited, no callsign filter),
then returned early if any OpenSky positions existed — never reaching
Wingbits. Result: source:"none" even when the flight was visible on map.
New order for callsign-only queries:
1. Wingbits relay (index + global fallback on miss) — fast and exact
2. OpenSky skipped (no callsign filter, rate-limited, wrong source)
OpenSky and Wingbits bbox fallback paths unchanged for bbox/icao24 queries.
The global Wingbits bbox call (-80/-180/80/180) was unreliable — Wingbits
returns 401/empty for worldwide queries. Replace it with an in-memory index
populated from every successful bbox response. Callsign-only queries check the
index (fresh within 5 min) without hitting the Wingbits API.
Also fix cache key bug: callsign-only searches previously fell through to
'aviation:track:all:v1' (shared across all searches). Now keyed per callsign.
Use shorter TTLs: 60s positive, 10s negative for callsign searches.
Result: if CTN465 or SWR785B is visible in any user's viewport, the relay
index has it and CMD+K 'flight CTN465' returns it immediately.
- Extend /wingbits/track relay endpoint to accept ?callsign=EK36 without bbox,
using global bounding box (-80/-180/80/180) for worldwide aircraft lookup
- Filter by callsign in relay response loop to return only matching flights
- Server: try Wingbits for callsign-only searches (previously only tried for bbox)
- Server: return empty positions (not simulated) for explicit callsign/icao24 lookups
- Client: filter out simulated-source positions before displaying search results
Open-Meteo rate-limits concurrent requests (429s). When 10/15 zones
failed, the seed wrote 5 all-NORMAL entries to Redis, causing the panel
to show 0 anomalies (client filters out NORMAL).
- Replace Promise.allSettled with sequential loop + 200ms delay between
zone requests to avoid triggering Open-Meteo rate limits
- Add minimum quality guard: throw if fewer than 2/3 zones return data,
so runSeed extends existing TTL instead of overwriting with partial data
- Update validate() to enforce the same minimum zone count
- Note climate cron schedule in health.js
The seed was being SIGKILL'd on Railway (512MB limit) because fast-xml-parser
built a ~300MB object tree from the 120MB OFAC SDN XML download, causing both
the raw XML string and the full parsed object to coexist in heap simultaneously.
Switch to sax (already a transitive dep) with a streaming pipeline: response.body
is piped chunk-by-chunk via a TextDecoder into the SAX parser. The full XML
string is never held in memory. Reference maps (areaCodes, featureTypes,
legalBasis, locations, parties) are populated as SAX events arrive, and entries
are emitted one at a time on </SanctionsEntry>.
All DOM-traversal helpers (listify, textValue, buildEpoch, buildReferenceMaps,
buildLocationMap, extractPartyName, resolveEntityType, extractPartyCountries,
buildPartyMap, extractPrograms, extractEffectiveAt, extractNote,
buildEntriesForDocument) are removed. Output-stage pure functions (uniqueSorted,
compactNote, sortEntries, buildCountryPressure, buildProgramPressure) are kept.
Tests updated to match: removed test blocks for deleted DOM helpers, kept
coverage for the remaining pure functions (2173/2173 pass).
* feat(seeds): shared FX rate cache + BigMac WoW guards
- Extract SHARED_FX_FALLBACKS to _seed-utils.mjs as single source of truth,
eliminating duplicated FX fallback tables across seed-bigmac, seed-grocery-basket,
and seed-fx-rates
- Add getSharedFxRates() / fetchYahooFxRates() to _seed-utils.mjs so all seeds
share one Redis-cached rate set (shared:fx-rates:v1, 4h TTL) instead of each
making ~46 independent Yahoo Finance calls per run
- Add seed-fx-rates.mjs: dedicated daily Railway cron that pre-warms the shared
FX cache, saving ~90 Yahoo calls per weekly bigmac+grocery-basket cycle
- Add WoW minimum-age guard (6 days): prevents week-on-week display when previous
snapshot is less than 6 days old (fixes -98.5% France WoW on first seed run)
- Add per-country WoW anomaly filter (+-20%): nulls suspicious entries and logs
admin alert with country name and delta for Railway log monitoring
- Fix global WoW anomaly check to use unfiltered raw average so it can actually
exceed +-20% (filtered average was mathematically bounded and never triggered)
- Add USD price sanity range guard ($1.50-$12.00): drops prices from bad scrapes
before they reach Redis (would have caught the $470 France value)
- Move WOW_ANOMALY_THRESHOLD, MIN_WOW_AGE_MS, USD_MIN, USD_MAX to module scope
* fix(seed-fx): address PR review — TTL mismatch and partial write-back risk
- Extend shared:fx-rates:v1 TTL from 4h to 25h so cache stays warm
between daily cron runs (with 1h drift buffer)
- Make getSharedFxRates() read-only: remove write-back on partial cache
hit and on cache miss; only seed-fx-rates.mjs owns writes to this key,
preventing a subset consumer from silently overwriting a fuller cache
The sdn_advanced.xml file has grown to ~120MB. During XML_PARSER.parse()
both the raw string (~120MB) and the parsed JS object tree (~200-300MB)
coexist in memory, exceeding Railway's 512MB limit and causing a silent
SIGKILL with no error output in logs.
Block-scope the xml string so it is GC-eligible immediately after parse
returns, and yield to the event loop so GC can reclaim it before
buildEntriesForDocument allocates the entry objects.
Also corrects the stale ~10MB comment that was written before the file grew.
* feat(seed): add learned routes cache to grocery basket seed
Persists successful EXA/Firecrawl URL discoveries in Redis so subsequent
runs skip the expensive EXA search for known-good (country, item) pairs.
Strategy per item:
1. Direct fetch + matchPrice on learned URL (free)
2. Firecrawl on learned URL if step 1 fails (handles JS SPAs)
3. Full EXA search only when learned route fails or is absent
4. Saves newly discovered URL as learned route for next run
Safety guarantees matching the Codex review:
- isAllowedRouteHost() validates hostname against country.sites allowlist
before both saving and replaying (prevents stored-SSRF)
- tryDirectFetch() applies CURRENCY_MIN + ITEM_USD_MAX bulk-price guards
identical to the existing EXA and Firecrawl paths
- failsSinceSuccess >= 2 triggers true DEL (not TTL wait)
- SET/DEL conflict resolved: effectiveDeletes filters keys in updates;
DELs sent before SETs in pipeline
- All operations non-fatal: pipeline failures log warnings, seed continues
New exports in _seed-utils.mjs: isAllowedRouteHost, bulkReadLearnedRoutes,
bulkWriteLearnedRoutes (1 pipeline read + 1 pipeline write per run).
BigMac deferred to Phase 2 (uses EXA summaries from aggregator pages).
Estimated savings: ~63 of 90 EXA calls skipped per run at 70% hit rate.
* test(seed): extract processItemRoute for testability; add 5 integration tests
- Move item-level decision tree into processItemRoute() in _seed-utils.mjs
so it can be imported and unit-tested without triggering runSeed()
- seed-grocery-basket.mjs delegates to processItemRoute() with fetchViaExa
callback containing the existing EXA+Firecrawl block
- 5 integration tests cover: learned-hit success (EXA skipped), learned fail
+ EXA replacement, fail x2 eviction, SSRF guard (bad host blocks direct
fetch), EXA success with unlisted host (route not saved)
- Fix: move allowedHosts computation outside Promise.all (once per country)
- Fix: add [EXA->learned] log tag when new route is saved from EXA discovery
- All 21 seed-learned-routes tests pass
* fix(seed): strip path from allowedHosts entries before hostname comparison
grocery-basket.json contains "noon.com/saudi-en" for Saudi Arabia.
allowedHosts was built with only www. stripped, so the comparison
hostname === 'noon.com/saudi-en'
was always false — noon.com routes for SA were rejected or evicted
every run, preventing the cache from ever stabilizing there.
Fix: split('/')[0] after stripping www., giving bare hostname.
Add regression test: path-bearing allowlist entry matches noon.com URL.
* fix(panels): hide finance-only panels on non-finance variants
stock-analysis, stock-backtest, and daily-market-brief are FINANCE_PANELS-only
but persisted panelSettings from a previous finance/full session caused them
to appear on tech variant. Guard creation with variantPanelKeys set derived
from VARIANT_DEFAULTS[SITE_VARIANT].
* fix(pro): remove duplicate Create with AI button, gate MCP connect as pro
With unified isProUser() both widget creation buttons showed simultaneously.
Remove the basic-tier 'Create with AI' block (duplicate of 'Create Interactive
Widget'). Gate 'Connect with MCP' with isProUser() — was previously ungated.
* feat(pro): gate export (⬇) and playback (⏪) toolbar buttons for pro users
Both setupExportPanel() and setupPlaybackControl() now return early
unless isProUser() — either wm-widget-key or wm-pro-key grants access.
* revert(panels): remove incorrect variant guard on stock/daily-market panels
The variantPanelKeys.has() guards were wrong — shouldCreatePanel() already
handles variant defaults via panelSettings. Users who manually enable these
panels on any variant should see them (locked if not pro, unlocked if pro).
The guard broke cross-variant user customization.
* fix(resilience): extend TTLs and add Finnhub/FRED fallbacks for macro signals
- MACRO_TTL 1800s → 21600s (6h) so stale data serves during Yahoo outages
- MARKET_SEED_TTL 1800s → 7200s (2h) in ais-relay for sectors/quotes
- Finnhub stock/candle fallback for QQQ and XLP when Yahoo returns null
- Finnhub crypto/candle fallback for BTC (BINANCE:BTCUSDT) when Yahoo fails
- FRED DEXJPUS fallback for JPY/USD historical prices (no new API key needed)
Prevents Macro Stress and Sector Heatmap panels from showing "unavailable"
during Yahoo Finance 429 rate-limit windows from Railway IPs.
CoinGecko was silently failing on the relay (no API key, rate-limited by
the main crypto seed loop), leaving defi/ai/other-tokens keys empty.
- seed-token-panels.mjs: rewrite with CoinGecko → CoinPaprika fallback,
use runSeed() wrapper so seed-meta:market:token-panels is written
- ais-relay.cjs: add fetchTokenPanelsCoinPaprika() fallback; relay no
longer silently drops token panels on CoinGecko failure
- health.js: add defiTokens, aiTokens, otherTokens to BOOTSTRAP_KEYS;
add tokenPanels to SEED_META (maxStaleMin: 90, 30-min cron × 3)
- Railway: provision seed-token-panels cron service (*/30 * * * *,
1vCPU/1GB, watchPatterns scoped to token panel files)
* feat(economic): add WoW tracking and fix plumbing for bigmac/grocery-basket panels
Phase 1 — Fix Plumbing:
- Adjust CACHE_TTL to 10 days (864000s) for bigmac and grocery-basket seeds
- Align health.js SEED_META maxStaleMin to 10080 (7 days) for both
- Add grocery-basket and bigmac to seed-health.js SEED_DOMAINS with intervalMin: 5040
- Refactor publish.ts writeSnapshot to accept advanceSeedMeta param; only
advance seed-meta when fresh data exists (overallFreshnessMin < 120)
- Add manual-fallback-only comment to seed-consumer-prices.mjs
Phase 2 — Week-over-Week Tracking:
- Add wow_pct field to BigMacCountryPrice and CountryBasket proto messages
- Add wow_avg_pct, wow_available, prev_fetched_at to both response protos
- Regenerate client/server TypeScript from updated protos
- Add readCurrentSnapshot() helper + WoW computation to seed-bigmac.mjs
and seed-grocery-basket.mjs; write :prev key via extraKeys
- Update BigMacPanel.ts to show per-country WoW column and global avg summary
- Update GroceryBasketPanel.ts to show WoW badge on total row and basket avg summary
- Add .bm-wow-up, .bm-wow-down, .bm-wow-summary, .gb-wow CSS classes
- Fix server handlers to include new WoW fields in fallback responses
* fix(economic): guard :prev extraKey against null on first seed run; eliminate double freshness query in publish.ts
* refactor(economic): address code review findings from PR #1974
- Extract readSeedSnapshot() into _seed-utils.mjs (DRY: was duplicated
verbatim in seed-bigmac and seed-grocery-basket)
- Add FRESH_DATA_THRESHOLD_MIN constant in publish.ts (replace magic 120)
- Fix seed-consumer-prices.mjs contradictory JSDoc (remove stale
"Deployed as: Railway cron service" line that contradicted manual-only warning)
- Add i18n keys panels.bigmacWow / panels.bigmacCountry to en.json
- Replace hardcoded "WoW" / "Country" with t() calls in BigMacPanel
- Replace IIFE-in-ternary pattern with plain if blocks in BigMacPanel
and GroceryBasketPanel (P2/P3 from code review)
* fix(publish): gate advanceSeedMeta on any-retailer freshness, not average
overallFreshnessMin is the arithmetic mean across all retailers, so with
1 fresh + 2 stale retailers the average can exceed 120 min and suppress
seed-meta advancement even while fresh data is being published.
Use retailers.some(r => r.freshnessMin < 120) to correctly implement
"at least one retailer scraped within the last 2 hours."
* fix(relay): add missing USNI regions and remove dead NZ safetravel feed
USNI fleet tracker warns on unknown regions "Tasman Sea" and "Eastern
Atlantic" — add coords to USNI_REGION_COORDS map.
safetravel.govt.nz/news/feed redirects to /404 on every request; remove
from advisory feeds and RSS allowed domains.
* fix: remove safetravel.govt.nz from edge function allowed domains copy
* fix: remove safetravel.govt.nz from shared allowed domains copy
* fix(consumer-prices): restore dropped seed script, fix maxStaleMin
seed-consumer-prices.mjs was included in PR #1901 but accidentally
dropped during a subsequent squash merge. Restored verbatim from the
PR diff. The script fetches from consumer-prices-core and writes all
Redis keys (overview, categories, movers, spread, freshness, series).
Also corrects health.js maxStaleMin values from 2880 (48h) to match
actual seed TTLs: overview/categories/movers=90min, spread=120min,
freshness=30min. The previous values would mask outages for 2 days.
Needs CONSUMER_PRICES_CORE_BASE_URL set in Railway env to produce
real data. Without it the script writes empty-but-valid placeholders.
* fix(consumer-prices): switch publish job to Upstash HTTP REST client
Replaced standard redis TCP client (REDIS_URL → Valkey) with Upstash
HTTP REST calls (UPSTASH_REDIS_REST_URL + UPSTASH_REDIS_REST_TOKEN),
matching the pattern used by all other seed scripts. This is why all
consumer-prices Redis keys were permanently empty — publish.ts was
writing to Railway-internal Valkey instead of Upstash.
Railway env vars added to seed-consumer-prices service separately.
- seed-bigmac.mjs: expand from 9 ME countries to 50+ global (Americas, Europe, Asia-Pacific, ME, Africa); derive FX_SYMBOLS from COUNTRIES; expand CCY regex and FX_FALLBACKS
- ConsumerPricesPanel: show structured seeding placeholder (icon + title + sub) when upstreamUnavailable instead of bare empty text
- main.css: add cp-* styles for ConsumerPricesPanel and gb-* styles for GroceryBasketPanel (were missing entirely)
* chore(iran-seed): add minab, asaluyeh, south pars, al-ahmadi location coords
* fix(sanctions): remove redundant Total card from summary grid
Total count is already shown in the panel header badge, making the
large card duplicate information. Keep New, Vessels, Aircraft cards.
- Add migration 002: seed canonical_products, baskets, basket_items for essentials-ae
- Add migration 003: partial unique index to fix NULL uniqueness gap in canonical_products
- Add ExaSearchAdapter to scrape.ts; auto-creates canonical product→basket matches
- Fix getBasketItemId to lookup by canonical_name via JOIN (not category) to avoid
dairy collision (3 items share same category)
- Fix getOrCreateRetailer race condition with ON CONFLICT upsert
- Add per-category index writes in aggregate.ts; guard division-by-zero
- Set all publish.ts TTLs to 93600s (26h) to survive cron scheduling drift
- Set health.js maxStaleMin to 2880 (daily cron × 2) for correct staleness detection
- Remove redundant seed-consumer-prices.mjs (publish.ts writes to Redis directly)
- Add src/cli/validate.ts DB health check script
- Fix z.record() to z.record(z.string(), z.unknown()) for Zod compat
* feat(consumer-prices): add basket price monitoring domain
Adds end-to-end consumer price tracking to enable inflation monitoring
across key markets, starting with UAE essentials basket.
- consumer-prices-core/: companion scraping service with pluggable
acquisition providers (Playwright, Exa, Firecrawl, Parallel P0),
config-driven retailer YAML, Postgres schema, Redis snapshots
- proto/worldmonitor/consumer_prices/v1/: 6-RPC service definition
- api/consumer-prices/v1/[rpc].ts: Vercel edge route
- server/worldmonitor/consumer-prices/v1/: Redis-backed RPC handlers
- src/services/consumer-prices/: circuit breakers + bootstrap hydration
- src/components/ConsumerPricesPanel.ts: 5-tab panel (overview /
categories / movers / spread / health)
- scripts/seed-consumer-prices.mjs: Railway cron seed script
- Wire into bootstrap, health, panels, gateway, cache-keys, locale
* fix(consumer-prices): resolve all code review findings
P0: populate topCategories — categoryResult was fetched but never used.
Added buildTopCategories() helper with grouped CTE query that extracts
current_index and week-over-week pct per category.
P1 (4 fixes):
- aggregate: replace N+1 getBaselinePrice loop with single batch query
getBaselinePrices(ids[], date) via ANY($1) — eliminates 119 DB roundtrips
per basket run
- aggregate/computeValueIndex: was dividing all category floors by the same
arbitrary first baseline; now uses per-item floor price with per-item
baseline (same methodology as fixed index but with cheapest price)
- basket-series endpoint now seeded: added buildBasketSeriesSnapshot() to
worldmonitor.ts, /basket-series route in companion API, publish.ts writes
7d/30d/90d series per basket, seed script fetches and writes all three ranges
- scrape: call teardownAll() after each retailer run to close Playwright
browser; without this the Chromium process leaked on Railway
P2 (4 fixes):
- db/client: remove rejectUnauthorized: false — was bypassing TLS cert
validation on all non-localhost connections
- publish: seed-meta now writes { fetchedAt, recordCount } matching the format
expected by _seed-utils.mjs writeExtraKeyWithMeta (was writing { fetchedAt, key })
- products: remove unused getMatchedProductsForBasket — exact duplicate of
getBasketRows in aggregate.ts; never imported by anything
Snapshot type overhaul:
- Flatten WMOverviewSnapshot to match proto GetConsumerPriceOverviewResponse
(was nested under overview:{}; handlers read flat)
- All asOf fields changed from number to string (int64 → string per proto JSON)
- freshnessMin/parseSuccessRate null -> 0 defaults
- lastRunAt changed from epoch number to ISO string
- Mover items now include currentPrice and currencyCode
- emptyOverview/Movers/Spread/Freshness in seed script use String(Date.now())
* feat(consumer-prices): wire Exa search engine as acquisition backend for UAE retailers
Ports the proven Exa+summary price extraction from PR #1904 (seed-grocery-basket.mjs)
into consumer-prices-core as ExaSearchAdapter, replacing unvalidated Playwright CSS
scraping for all three UAE retailers (Carrefour, Lulu, Noon).
- New ExaSearchAdapter: discovers targets from basket YAML config (one per item),
calls Exa API with contents.summary to get AI-extracted prices, uses matchPrice()
regex (ISO codes + symbol fallback + CURRENCY_MIN guards) to extract AED amounts
- New db/queries/matches.ts: upsertProductMatch() + getBasketItemId() for auto-linking
scraped Exa results to basket items without a separate matching step
- scrape.ts: selects ExaSearchAdapter when config.adapter === 'exa-search'; after
insertObservation(), auto-creates canonical product and product_match (status: 'auto')
so aggregate.ts can compute indices immediately without manual review
- All three UAE retailer YAMLs switched to adapter: exa-search and enabled: true;
CSS extraction blocks removed (not used by search adapter)
- config/types.ts: adds 'exa-search' to adapter enum
* fix(consumer-prices): use EXA_API_KEYS (with fallback to EXA_API_KEY) matching PR #1904 pattern
* fix(consumer-prices): wire ConsumerPricesPanel in layout + fix movers limit:0 bug
Addresses Codex P1 findings on PR #1901:
- panel-layout.ts: import and createPanel('consumer-prices') so the panel
actually renders in finance/commodity variants where it is enabled in config
- consumer-prices/index.ts: limit was hardcoded 0 causing slice(0,0) to always
return empty risers/fallers after bootstrap is consumed; fixed to 10
* fix(consumer-prices): add categories snapshot to close P2 gap
consumer-prices:categories:ae:* was in BOOTSTRAP_KEYS but had no producer,
so the Categories tab always showed upstreamUnavailable.
- buildCategoriesSnapshot() in worldmonitor.ts — wraps buildTopCategories()
and returns WMCategoriesSnapshot matching ListConsumerPriceCategoriesResponse
- /categories route in consumer-prices-core API
- publish.ts writes consumer-prices:categories:{market}:{range} for 7d/30d/90d
- seed-consumer-prices.mjs fetches all three ranges from consumer-prices-core
and writes them to Redis alongside the other snapshots
P1 issues (snapshot structure mismatch + limit:0 movers) were already fixed
in earlier commits on this branch.
* fix(types): add variants? to PANEL_CATEGORY_MAP type
* feat(economic): add ME grocery basket price index
Adds a grocery basket price comparison panel for 9 Middle East
countries (UAE, KSA, Qatar, Kuwait, Bahrain, Oman, Egypt, Jordan,
Lebanon) using EXA AI to discover prices from regional e-commerce
sites (Carrefour, Lulu, Noon, Amazon) and Yahoo Finance for FX rates.
- proto: ListGroceryBasketPrices RPC with CountryBasket/GroceryItemPrice messages
- seed: seed-grocery-basket.mjs, 90 EXA calls/run, 150ms delay, hardcoded
FX fallbacks for pegged GCC currencies, 6h TTL
- handler: seed-only RPC reading economic:grocery-basket:v1
- gateway: static cache tier for the new route
- bootstrap/health: groceryBasket key in SLOW tier, 720min stale threshold
- frontend: GroceryBasketPanel with scrollable table, cheapest/priciest
column highlighting, styles moved to panels.css
- panel disabled by default until seed is run on Railway
* fix(generated): restore @ts-nocheck in economic service codegen
* fix(grocery-basket): tighten seed health staleness and seed script robustness
- Set maxStaleMin to 360 (6h) matching CACHE_TTL so health alerts on first missed run
- Use ?? over || for FX fallback to handle 0-value rates correctly
- Add labeled regex patterns with bare-number warning in extractPrice
- Replace conditional delay logic with unconditional per-item sleep
* fix(grocery-basket): fix EXA API format and price extraction after live validation
- Use contents.summary format (not top-level summary) — previous format returned no data
- Support EXA_API_KEYS (comma-separated) in addition to EXA_API_KEY
- Extract price from plain-text summary string (EXA returns text, not JSON schema)
- Remove bare-number fallback — too noisy (matched "500" from "pasta 500g" as SAR 500)
- Fix LBP FX rate zero-guard: use fallback when Yahoo returns 0 for ultra-low-value currencies
Validated locally: 9 countries seeded, Redis write confirmed, ~111s runtime
* fix(grocery-basket): validate extracted currency against expected country currency
- matchPrice now returns the currency code alongside the price
- extractPrice rejects results where currency != expected country currency
(prevents AED prices from being treated as JOD prices on gcc.luluhypermarket.com)
- Tighten item queries (white granulated sugar, spaghetti pasta, etc.) to reduce
irrelevant product matches like Stevia on sugar queries
- Replace Jordan's gcc.luluhypermarket.com (GCC-only) with carrefour.jo + ounasdelivery.com
- Sync scripts/shared/grocery-basket.json
* feat(bigmac): add Big Mac Index seed + drop grocery basket includeDomains
Grocery basket:
- Remove includeDomains restriction — EXA neural search finds better sources
than hardcoded domain lists; currency validation prevents contamination
- Tighten query strings (supermarket retail price suffix)
Big Mac seed (scripts/seed-bigmac.mjs):
- Two-tier search: specialist sites (theburgerindex.com, eatmyindex.com) first,
fall back to open search for countries without per-country indexed pages
- Handle thousands-separator prices (480,000 LBP)
- Accept USD prices from cost-of-living index sites as fallback
- Exclude ranking/average pages (Numbeo country_price_rankings, Expatistan)
- Validated live: 7/9 countries with confirmed prices
UAE=19AED, KSA=19SAR, QAR=22QAR, KWD=1.4KWD, EGP=135EGP, JOD=3JOD, LBP=480kLBP
* feat(economic): expand grocery basket to 24 global countries, drop Big Mac tier-2 search
Grocery basket: extend coverage from 9 MENA to 24 countries across all
regions (US, UK, DE, FR, JP, CN, IN, AU, CA, BR, MX, ZA, TR, NG, KR,
SG, PK, AE, SA, EG, KE, AR, ID, PH). Add FX fallbacks and fxSymbols
for all 23 new currencies. CCY regex in seed script updated to match
all supported currency codes.
Big Mac: remove tier-2 open search (too noisy, non-specialist pages
report combo prices or global averages). Specialist sites only
(theburgerindex.com, eatmyindex.com) for clean per-country data.
* feat(bigmac): wire Big Mac Index RPC, proto, bootstrap, health
Add ListBigMacPrices RPC end-to-end:
- proto/list_bigmac_prices.proto: BigMacCountryPrice + request/response
- service.proto: register ListBigMacPrices endpoint (GET /list-bigmac-prices)
- buf generate: regenerate service_server.ts + all client stubs
- server/list-bigmac-prices.ts: seed-only handler reads economic:bigmac:v1
- handler.ts: wire listBigMacPrices into EconomicServiceHandler
- api/bootstrap.js: bigmac key in BOOTSTRAP_CACHE_KEYS + SLOW_KEYS
- api/health.js: bigmac key in BOOTSTRAP_KEYS + SEED_META (maxStaleMin: 1440)
- _bootstrap-cache-key-refs.ts: groceryBasket + bigmac refs
* feat(bigmac): add BigMacPanel + register in panel layout
BigMacPanel renders a country-by-country Big Mac price table sorted by
USD price (cheapest/most expensive highlighted). Wired into bootstrap
hydration, refresh scheduler, and panel registry. Registered in
panels.ts (enabled: false, to be flipped once seed data is verified).
Also updates grocery basket i18n from ME-specific to global wording.
* fix(bigmac): register bigmac in cache-keys and RPC_CACHE_TIER
Add bigmac to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in
server/_shared/cache-keys.ts, and to RPC_CACHE_TIER (static tier)
in gateway.ts. Both were caught by bootstrap and RPC tier parity tests.
* fix(generated): restore @ts-nocheck in all generated service files after buf regenerate
buf generate does not emit @ts-nocheck. Previous convention restores it
manually post-generate to suppress strict type errors in generated code.
* fix(grocery-basket): restore includeDomains per country, add userLocation, fix currency symbol parsing
Root cause of 0-item countries (UK, JP, IN, NG): removing includeDomains
caused EXA neural search to return USD-priced global comparison pages
(Numbeo, Tridge, Expatistan) which currency validation correctly rejected.
Fixes:
- Add per-country sites[] in grocery-basket.json (researched local
supermarket/retailer domains: tesco.com, kaufland.de, bigbasket.com, etc.)
- Pass includeDomains: country.sites to restrict EXA to local retailers
- Pass userLocation: country.code (ISO) to bias results to target country
- Add currency symbol fallback regex (£→GBP, €→EUR, ¥→JPY, ₹→INR,
₩→KRW, ₦→NGN, R$→BRL) — sites like BigBasket use ₹ not INR
- Summary query now explicitly requests ISO currency code
- Simplify item queries (drop country name — context from domains)
Smoke test results:
UK sugar → GBP 1.09 (tesco.com) ✓
IN rice → ₹66 (bigbasket.com) ✓
JP rice → JPY 500 (kakaku.com) ✓
* fix(grocery-basket): add Firecrawl fallback, parallel items, bulk caps, currency floors
- Add Firecrawl as JS-SPA fallback after EXA (handles noon.com, coupang, daraz, tokopedia, lazada)
- Parallelize item fetching per country with 200ms stagger: runtime 38min to ~4.5min
- Add CURRENCY_MIN floors (NGN:50, IDR:500, KRW:1000, etc.) to reject product codes as prices
- Add ITEM_USD_MAX bulk caps (sugar:8USD, salt:5USD, rice:6USD, etc.) applied to both EXA and Firecrawl
- Fix SA: use noon.com/saudi-en + carrefour.com.sa (removes luluhypermarket cross-country pollution)
- Fix EG: use carrefouregypt.com + spinneys.com.eg + seoudi.com (removes GCC luluhypermarket)
- Expand sites for DE, MX, ZA, TR, NG, KR, IN, PK, AR, ID, PH to improve coverage
- Sync scripts/shared/grocery-basket.json with shared/grocery-basket.json
* fix(grocery-basket): address PR review comments P1+P2
P1 - fix ranking with incomplete data:
only include countries with >=70% item coverage (>=7/10) in
cheapest/mostExpensive ranking — prevents a country with 4/10
items appearing cheapest due to missing data
P1 - fix regex false-match on pack sizes / weights:
try currency-first pattern (SAR 8.99) before number-first to
avoid matching pack counts; use matchAll and take last match
P2 - mark seed-miss responses as non-cacheable:
add upstream_unavailable to proto + return true on empty seed
so gateway sets Cache-Control: no-store on cold deploy
* fix(generated): update EconomicService OpenAPI docs for upstream_unavailable field
