* fix(fuel-prices): switch EU source from CSV to XLSX, increase Spain timeout
EU Oil Bulletin dropped CSV format — all YYYY-MM CSV fallback URLs return 404.
EC now only publishes XLSX files with a stable document UUID. Switch to the
"Prices with taxes latest prices" XLSX (document ID is permanent, file is
updated in-place weekly). Uses SheetJS xlsx package added to scripts/package.json.
parseEUPrice updated to handle both "1234.56" (xlsx default) and European
"1.234,56" / "1,234.56" number formats with thousand separators.
Also increase Spain (minetur) AbortSignal timeout 30s to 60s. The endpoint
returns ~11K stations as one JSON blob; 30s was insufficient from Railway US.
* fix(fuel-prices): add URL logging to Brazil, Mexico, NZ fetchers for debuggability
* feat(fuel-prices): add retail fuel prices panel and seeder
- Add ListFuelPrices proto RPC with FuelPrice/FuelCountryPrice messages
- Create seed-fuel-prices.mjs seeder with 5 sources: Malaysia (data.gov.my),
Spain (minetur.gob.es), Mexico (datos.gob.mx), US EIA, EU oil bulletin CSV
- Add list-fuel-prices.ts RPC handler reading from Redis seed cache
- Wire handler into EconomicService handler.ts
- Register fuelPrices in cache-keys.ts, bootstrap.js, health.js, gateway.ts
- Add FuelPricesPanel frontend component with gasoline/diesel/source columns
- Wire panel into panel-layout.ts, App.ts (prime + refresh scheduler)
- Add panel config, command entry with fuel/gas/diesel/petrol keywords
- Add fuelPrices refresh interval (6h) in base.ts
- Add i18n keys in en.json (panels.fuelPrices + components.fuelPrices)
Task: fuel-prices
* fix(fuel-prices): add BR/NZ/UK sources, fix EU non-euro currency, review fixes
- Add fetchBrazil() — ANP CSVs (GASOLINA + DIESEL), Promise.allSettled for
independent partial results, BRL→USD via FX
- Add fetchNewZealand() — MBIE weekly-table.csv, Board price national avg, NZD→USD
- Add fetchUK_ModeA() — CMA retailer JSON feeds (Asda/BP/JET/MFG/Sainsbury's/
Morrisons), E10+B7 pence→GBP, max-date observedAt across retailers
- Fix EU non-euro members (BG/CZ/DK/HU/PL/RO/SE) using local currency FX on
EUR-denominated prices — all EU entries now use currency:'EUR'
- Fix fetchBrazil Promise.all → Promise.allSettled (partial CSV failure no
longer discards both fuels)
- Fix UK observedAt: keep latest date across retailers (not last-processed)
- Fix WoW anomaly: omit wowPct instead of setting to 0
- Lift parseEUPrice out of inner loop to module scope
- Pre-compute parseBRDate per row to avoid double-conversion
- Update infoTooltip: describe methodology without exposing source URLs
- Add BRL, NZD, GBP to FX symbols list
* fix(fuel-prices): fix 4 live data bugs found in external review
EU CSV: replace hardcoded 2024 URLs (both returning 404) with dynamic
discovery — scrape EC energy page for current CSV link, fall back to
generated YYYY-MM patterns for last 4 months.
NZ: live MBIE header has no Region column (Week,Date,Fuel,Variable,Value,
Unit,Status) — remove regionIdx guard that was forcing return []. Values
are in NZD c/L not NZD/L — divide by 100 before storing.
UK: last_updated is DD/MM/YYYY HH:mm:ss not ISO — parse to YYYY-MM-DD
before lexicographic max-date comparison; previous code stored the
seed-run date instead of the latest retailer timestamp.
Panel: source column fell back to — for diesel-only countries because it
only read gas?.source. Use (gas ?? dsl)?.source so diesel-only rows
display their source correctly.
* feat(intelligence): GDELT tone/vol timeline per topic (#2044)
* fix(gdelt-timeline): add isMain guard to seed script, fix gateway cache tier
- Wrap runSeed() call in isMain guard (process.argv[1].endsWith check) to
prevent CI failures when the seed module is imported rather than executed
directly — pre-push hook does not catch this
- Change gateway cache tier from 'medium' (20min CDN) to 'daily' (1h
browser/s-maxage=86400 CDN) to align with the 1h TIMELINE_TTL on the
per-topic tone/vol Redis keys
* fix(gdelt-timeline): TTL 1h→12h, medium cache tier, real fetchedAt, exit 0
- seed-gdelt-intel.mjs: TIMELINE_TTL 3600→43200 (12h = 2× 6h cron) so
tone/vol keys survive between cron runs instead of expiring after 1h
- seed-gdelt-intel.mjs: afterPublish wraps tone/vol as {data, fetchedAt}
so the real write timestamp is stored alongside the arrays
- get-gdelt-topic-timeline.ts: unwrap new envelope shape; fetchedAt now
reflects actual data write time instead of request time
- gateway.ts: daily→medium cache tier (CDN s-maxage=1200 matches 6h cadence)
- seed-gdelt-intel.mjs: process.exit(1)→0 to match seeder suite convention
* fix(gdelt-timeline): add GdeltTimelinePoint type cast in unwrap helper
* feat(trade): UN Comtrade strategic commodity flows seeder + RPC (#2045)
* fix(trade): correct byYear overwrite bug and anomaliesOnly upstream flag
Two bugs in the Comtrade flows feature:
1. seed-trade-flows.mjs: byYear Map used year alone as key. With flowCode=X,M
the API returns both export and import records for the same year; the second
record silently overwrote the first, causing incorrect val/wt and YoY
calculations. Fix: key by `${flowCode}:${year}` so exports and imports are
tracked separately and YoY is computed per flow direction.
2. list-comtrade-flows.ts: `if (!flows.length)` set upstreamUnavailable=true
even when Redis data was present but all records were filtered out by
anomaliesOnly=true. Fix: track dataFound separately and only set
upstreamUnavailable when no Redis keys returned data.
* fix(comtrade-flows): gold standard TTL, maxStaleMin, exit code, batch Redis fetch
- health.js: maxStaleMin 1440→2880 (2× daily interval per gold standard)
- seed-trade-flows.mjs: CACHE_TTL 86400→259200 (72h = 3× daily interval)
- seed-trade-flows.mjs: process.exit(1)→0 to match seeder suite convention
- list-comtrade-flows.ts: replace 30 getCachedJson calls with single getCachedJsonBatch pipeline
* feat(sanctions): entity lookup index + OpenSanctions search (#2042)
* fix: guard tokens[0] access in sanctions lookup
* fix: use createIpRateLimiter pattern in sanctions-entity-search
* fix: add sanctions-entity-search to allowlist and cache tier
* fix: add LookupSanctionEntity RPC to service.proto, regenerate
* fix(sanctions): strip _entityIndex/_state from main key publish, guard limit NaN
P0: seed-sanctions-pressure was writing the full _entityIndex array and _state
snapshot into sanctions:pressure:v1 because afterPublish runs after atomicPublish.
Add publishTransform to strip both fields before the main key write so the
pressure payload stays compact; afterPublish and extraKeys still receive the full
data object and write the correct separate keys.
P1: limit param in sanctions-entity-search edge function passed NaN to OpenSanctions
when a non-numeric value was supplied. Fix with Number.isFinite guard.
P2: add 200-char max length on q param to prevent oversized upstream requests.
* fix(sanctions): maxStaleMin 2x interval, no-store on entity search
health.js: 720min (1x) → 1440min (2x) for both sanctionsPressure and
sanctionsEntities. A single missed 12h cron was immediately flagging stale.
sanctions-entity-search.js: Cache-Control public → no-store. Sanctions
lookups include compliance-sensitive names in the query string; public
caching would have logged/stored these at CDN/proxy layer.
* feat(intelligence): add countryCode geo-attribution to topStories (#2051)
* fix(geo-extract): filter EU as supranational, add unigram stopwords, type countryCode in ServerInsightStory
- Map 'eu'/'europe' to 'XX' (supranational marker, returns null) instead of 'EU' which is not a valid ISO2 code and would be silently ignored by downstream CII scorer
- Add UNIGRAM_STOPWORDS set for high-false-positive single-word entries in country-names.json: chad/jordan/georgia/niger/guinea/mali/peru — these match too frequently as person names and US state names in English headlines; their country meanings are covered by unambiguous aliases (nigerian, georgian context via bigrams, etc.)
- Add countryCode: string | null and pubDate: string to ServerInsightStory TypeScript interface to match what seed-insights.mjs now writes to Redis
* fix(geo-extract): add 'us' to UNIGRAM_STOPWORDS to prevent pronoun false positives
'us' as a bare word matches almost every English headline ("give us",
"tells us", etc.). US coverage is preserved via the 'washington' and
'american' aliases in ALIAS_MAP.
* fix(geo-extract): fix US abbreviation, bigram punctuation, and scan ordering
Three issues:
1. 'us' stopword suppressed uppercase US (country). Fix: pre-process
\bUS\b → 'United States' before lowercasing; remove 'us' from stopwords.
2. Bigram matching used raw tokens so 'West Bank,' and 'Tel Aviv:' missed
their alias entries. Fix: strip punctuation from each token before
forming the bigram key.
3. Two-pass scan (all bigrams then all unigrams) meant 'United States'
bigram fired before earlier unigrams like 'Iran' in 'Iran blames US'.
Fix: single left-to-right scan with local longest-match (bigram at i
before unigram at i), preserving first-mention document order.
* refactor(intelligence): preserve digest threat fields through normalization (#2050)
* fix(intelligence): shallow-copy cluster threat, add medium to GlobeMap color map and VALID_THREAT_LEVELS
- _clustering.mjs: spread-copy threat object instead of passing reference to prevent downstream mutation from corrupting cluster data
- GlobeMap.ts: add 'medium' to threatLevel color guards (marker render + tooltip) so new digest taxonomy shows yellow/orange instead of falling through to the info-blue default
- InsightsPanel.ts: extend VALID_THREAT_LEVELS to include new taxonomy values (medium/low/info) so the safeThreat guard stays in sync
* fix(intelligence): normalize proto threat levels and use tier-sorted cluster threat
P1: digest items store THREAT_LEVEL_HIGH/MEDIUM/etc (proto enum strings from
toProtoItem). Copying item.threat verbatim caused threatLevel: 'THREAT_LEVEL_HIGH'
to land in output, missing every downstream check (THREAT_RGB, badge render,
GlobeMap color switch). Add normalizeThreat() with PROTO_TO_LEVEL map at read
time so values arrive as 'high'/'medium'/etc.
P2: cluster threatItem was found via group.find() on the unsorted insertion-order
array. A low-tier item first in the array would win over a Reuters item. Switch
to sorted.find() (already tier/date sorted for primary selection) to prefer the
highest-quality source's threat classification.
Russell 2000 (^RUT):
- Add to MARKET_SYMBOLS and YAHOO_ONLY in ais-relay.cjs
- Covers the 4th major US index missing from the panel
GSCPI (NY Fed Global Supply Chain Pressure Index):
- New seedGscpi() loop in ais-relay.cjs — fetches monthly CSV from
newyorkfed.org (no API key required), parses wide-format vintage CSV
- Stored as economic:fred:v1:GSCPI:0 in FRED-compatible format so the
existing GetFredSeriesBatch RPC serves it without any proto changes
- TTL 72h (3x 24h interval), retry 20min on failure — gold standard pattern
- Add GSCPI to ALLOWED_SERIES in get-fred-series-batch.ts
- Add gscpi to STANDALONE_KEYS + SEED_META in health.js (maxStaleMin 2880)
* feat(commodities): expand tracking to cover agricultural and coal futures
Adds 9 new commodity symbols to cover the price rally visible in our
intelligence feeds: Newcastle Coal (MTF=F), Wheat (ZW=F), Corn (ZC=F),
Soybeans (ZS=F), Rough Rice (ZR=F), Coffee (KC=F), Sugar (SB=F),
Cocoa (CC=F), and Cotton (CT=F).
Also fixes ais-relay seeder to use display names from commodities.json
instead of raw symbols, so seeded data is self-consistent.
* fix(commodities): gold standard cache, 3-col grid, cleanup
- Add upstashExpire on zero-quotes failure path so bootstrap key TTL
extends during Yahoo outages (gold standard pattern)
- Remove unreachable fallback in retry loop (COMMODITY_META always has
the symbol since it mirrors COMMODITY_SYMBOLS)
- Switch commodities panel to 3-column grid (19 items → ~7 rows vs 10)
* fix(seeders): apply gold standard TTL-extend+retry pattern to Aviation, NOTAM, Cyber, PositiveEvents
* feat(consumer-prices): default to All — global comparison table as landing view
- DEFAULT_MARKET = 'all' so panel opens with the global view
- 🌍 All pill added at front of market bar
- All view fetches all 9 markets in parallel via fetchAllMarketsOverview()
and renders a comparison table: Market / Index / WoW / Spread / Updated
- Clicking any market row drills into that market's full tab view
- SINGLE_MARKETS exported for use in All-view iteration
- CSS: .cp-global-table and row styles
FRED_TTL was 1h but seed runs daily — key expires 23h before next run,
causing persistent EMPTY CRIT. Same pattern as consumer-prices publish.ts
(93600s TTL). maxStaleMin bumped from 90 to 1500 to match daily cadence.
TRADE_TTL (6h) matched the cron cadence exactly, causing the customs
revenue data key to expire seconds before the next seed run, resulting
in recurring EMPTY CRITs on the health endpoint. Monthly Treasury data
doesn't need 6h TTL; 24h (CUSTOMS_TTL) matches the maxStaleMin threshold.
Adds BAMLH0A0HYM2, ICSA, MORTGAGE30US, GSCPI to FRED_SERIES array.
These were in the server allowlist and frontend config but absent from
the seeder, causing empty data in production on every request.
Closes#2041
* feat(docker): enable Redis RDB persistence, add seed-orchestrator to supervisord, copy scripts into image
* feat(orchestrator): add prefixed logger utility
* feat(orchestrator): add seed-meta read/write helpers
* feat(orchestrator): add child process runner with timeout support
* feat(orchestrator): add central seed catalog with 42 seeders
* feat(orchestrator): implement seed orchestrator with tiered scheduling
Adds scripts/seed-orchestrator.mjs with:
- classifySeeders: classify seeders into active/skipped by env vars
- buildStartupSummary: human-readable startup report
- Tiered cold start (hot/warm/cold/frozen with per-tier concurrency)
- Freshness check via seed-meta keys before running stale seeders
- Steady-state scheduling with setTimeout-based recurring timers
- Overlap protection, retry-after-60s, consecutive failure demotion
- Global concurrency cap of 5 with queue-based overflow
- Graceful shutdown on SIGTERM/SIGINT (15s drain timeout)
- Meta writing for null-metaKey seeders to seed-meta:orchestrator:{name}
* fix(seeds): use local API for warm-ping seeders in Docker mode
* fix(orchestrator): allow ACLED email+password as alternative to access token
* feat(wmsm): add seed manager CLI scaffold with help, catalog, and checks
* feat(wmsm): implement status command with freshness display
* feat(wmsm): implement schedule command with next-run estimates
* feat(wmsm): implement refresh command with single and --all modes
* feat(wmsm): implement flush and logs commands
* fix(wmsm): auto-detect docker vs podman runtime
* feat(orchestrator): extract pure scheduling functions and add test harness
* feat(orchestrator): add SEED_TURBO=real|dry mode with compressed intervals
* feat(orchestrator): add SEED_TURBO env passthrough and fix retry log message
* fix(consumer-prices): harden scrape/aggregate/publish pipeline
- scrape: treat 0-product parse as error (increments errorsCount, skips
pagesSucceeded) so noon_grocery_ae missing eggs_12/tomatoes_1kg marks
the run partial instead of completed
- publish: fix freshData gate (freshnessMin >= 0) so a scrape finishing
at exactly 0 min lag still advances seed-meta
- aggregate: wrap per-basket aggregation in try/catch so one failing
basket does not skip remaining baskets; re-throw if any failed
- seed-consumer-prices.mjs: require --force flag to prevent accidentally
stomping publish.ts 26h TTLs with short 10-60min fallback TTLs
* fix(consumer-prices): correct basket comparison with intersection + dedup
Both aggregate.ts and the retailer spread snapshot were summing ALL
matched SKUs per retailer without deduplication, making Carrefour
appear most expensive simply because it had more matched products
(31 "items" vs Noon's 20 for a 12-item basket).
Fixes:
- aggregate.ts retailer_spread_pct: deduplicate per (retailer, basketItem)
taking cheapest price, then only compare on items all retailers carry
- worldmonitor.ts buildRetailerSpreadSnapshot: same dedup + intersection
logic in SQL — one best_price per (retailer, basket_item), common_items
CTE filters to items every active retailer covers
- exa-search.ts parseListing: log whether Exa returned 0 results or
results with no extractable price, to distinguish the two failure modes
* fix(consumer-prices-panel): correct parse rate display, category names, and freshness colors
- parseSuccessRate is stored as 0-100 but UI was doing *100 again (shows 10000%)
- Category name builder converts snake_case to Title Case (Cooking_oil → Cooking Oil)
- Add missing cp-fresh--ok/warn/stale/unknown CSS classes (freshness labels had no color)
- Add border-radius to stat cards and range buttons; add font-family to range buttons
- Add padding + bottom border to cp-range-bar for visual separation
* fix(consumer-prices): gate overview spread_pct query to last 2 days
buildOverviewSnapshot queried retailer_spread_pct with no recency
filter, so ORDER BY metric_date DESC LIMIT 1 would serve an
arbitrarily old row when today's aggregate run omitted a write
(no retailer intersection). Add INTERVAL '2 days' cutoff — covers
24h cron cadence plus scheduling drift. Falls through to 0 (→ UI
shows '—') when no recent value exists.
Sanctions seed has been failing since PR #2008 with:
ERR_MODULE_NOT_FOUND: Cannot find package 'sax'
PR #2008 replaced fast-xml-parser with SAX streaming but only updated
the root package.json, not scripts/package.json which is the Railway
container manifest. Railway runs npm ci from scripts/ so sax was never
installed. Add sax ^1.6.0 and remove the now-unused fast-xml-parser.
Also raise consumer-prices SEED_META maxStaleMin from 90-120 to 1500 min.
publish.ts runs once daily at 02:30 UTC; all five consumer-prices keys
were permanently STALE_SEED for 22+ hours/day after the daily run.
1500 min (25h) = 24h cadence + 1h grace before warning.
* fix(relay): add global Wingbits fallback for callsign-only index misses
When a callsign search hits the relay but the flight isn't in any
recent viewport (index miss), the relay now attempts a worldwide
Wingbits API call filtered by callsign server-side.
Previously: index miss → immediate empty response
Now: index miss → global bbox call → filter by callsign → return matches
Fallback is graceful: timeouts/errors return empty rather than 5xx.
Matched flights are also written into the index for subsequent queries.
* fix(aviation): try Wingbits before OpenSky for callsign-only searches
Commercial flights like UAE20, THY6260 are Wingbits-exclusive and
invisible to OpenSky receivers. The old order (OpenSky first) wasted
10s+ on OpenSky global states/all (rate-limited, no callsign filter),
then returned early if any OpenSky positions existed — never reaching
Wingbits. Result: source:"none" even when the flight was visible on map.
New order for callsign-only queries:
1. Wingbits relay (index + global fallback on miss) — fast and exact
2. OpenSky skipped (no callsign filter, rate-limited, wrong source)
OpenSky and Wingbits bbox fallback paths unchanged for bbox/icao24 queries.
The global Wingbits bbox call (-80/-180/80/180) was unreliable — Wingbits
returns 401/empty for worldwide queries. Replace it with an in-memory index
populated from every successful bbox response. Callsign-only queries check the
index (fresh within 5 min) without hitting the Wingbits API.
Also fix cache key bug: callsign-only searches previously fell through to
'aviation:track:all:v1' (shared across all searches). Now keyed per callsign.
Use shorter TTLs: 60s positive, 10s negative for callsign searches.
Result: if CTN465 or SWR785B is visible in any user's viewport, the relay
index has it and CMD+K 'flight CTN465' returns it immediately.
- Extend /wingbits/track relay endpoint to accept ?callsign=EK36 without bbox,
using global bounding box (-80/-180/80/180) for worldwide aircraft lookup
- Filter by callsign in relay response loop to return only matching flights
- Server: try Wingbits for callsign-only searches (previously only tried for bbox)
- Server: return empty positions (not simulated) for explicit callsign/icao24 lookups
- Client: filter out simulated-source positions before displaying search results
Open-Meteo rate-limits concurrent requests (429s). When 10/15 zones
failed, the seed wrote 5 all-NORMAL entries to Redis, causing the panel
to show 0 anomalies (client filters out NORMAL).
- Replace Promise.allSettled with sequential loop + 200ms delay between
zone requests to avoid triggering Open-Meteo rate limits
- Add minimum quality guard: throw if fewer than 2/3 zones return data,
so runSeed extends existing TTL instead of overwriting with partial data
- Update validate() to enforce the same minimum zone count
- Note climate cron schedule in health.js
The seed was being SIGKILL'd on Railway (512MB limit) because fast-xml-parser
built a ~300MB object tree from the 120MB OFAC SDN XML download, causing both
the raw XML string and the full parsed object to coexist in heap simultaneously.
Switch to sax (already a transitive dep) with a streaming pipeline: response.body
is piped chunk-by-chunk via a TextDecoder into the SAX parser. The full XML
string is never held in memory. Reference maps (areaCodes, featureTypes,
legalBasis, locations, parties) are populated as SAX events arrive, and entries
are emitted one at a time on </SanctionsEntry>.
All DOM-traversal helpers (listify, textValue, buildEpoch, buildReferenceMaps,
buildLocationMap, extractPartyName, resolveEntityType, extractPartyCountries,
buildPartyMap, extractPrograms, extractEffectiveAt, extractNote,
buildEntriesForDocument) are removed. Output-stage pure functions (uniqueSorted,
compactNote, sortEntries, buildCountryPressure, buildProgramPressure) are kept.
Tests updated to match: removed test blocks for deleted DOM helpers, kept
coverage for the remaining pure functions (2173/2173 pass).
* feat(seeds): shared FX rate cache + BigMac WoW guards
- Extract SHARED_FX_FALLBACKS to _seed-utils.mjs as single source of truth,
eliminating duplicated FX fallback tables across seed-bigmac, seed-grocery-basket,
and seed-fx-rates
- Add getSharedFxRates() / fetchYahooFxRates() to _seed-utils.mjs so all seeds
share one Redis-cached rate set (shared:fx-rates:v1, 4h TTL) instead of each
making ~46 independent Yahoo Finance calls per run
- Add seed-fx-rates.mjs: dedicated daily Railway cron that pre-warms the shared
FX cache, saving ~90 Yahoo calls per weekly bigmac+grocery-basket cycle
- Add WoW minimum-age guard (6 days): prevents week-on-week display when previous
snapshot is less than 6 days old (fixes -98.5% France WoW on first seed run)
- Add per-country WoW anomaly filter (+-20%): nulls suspicious entries and logs
admin alert with country name and delta for Railway log monitoring
- Fix global WoW anomaly check to use unfiltered raw average so it can actually
exceed +-20% (filtered average was mathematically bounded and never triggered)
- Add USD price sanity range guard ($1.50-$12.00): drops prices from bad scrapes
before they reach Redis (would have caught the $470 France value)
- Move WOW_ANOMALY_THRESHOLD, MIN_WOW_AGE_MS, USD_MIN, USD_MAX to module scope
* fix(seed-fx): address PR review — TTL mismatch and partial write-back risk
- Extend shared:fx-rates:v1 TTL from 4h to 25h so cache stays warm
between daily cron runs (with 1h drift buffer)
- Make getSharedFxRates() read-only: remove write-back on partial cache
hit and on cache miss; only seed-fx-rates.mjs owns writes to this key,
preventing a subset consumer from silently overwriting a fuller cache
The sdn_advanced.xml file has grown to ~120MB. During XML_PARSER.parse()
both the raw string (~120MB) and the parsed JS object tree (~200-300MB)
coexist in memory, exceeding Railway's 512MB limit and causing a silent
SIGKILL with no error output in logs.
Block-scope the xml string so it is GC-eligible immediately after parse
returns, and yield to the event loop so GC can reclaim it before
buildEntriesForDocument allocates the entry objects.
Also corrects the stale ~10MB comment that was written before the file grew.
* feat(seed): add learned routes cache to grocery basket seed
Persists successful EXA/Firecrawl URL discoveries in Redis so subsequent
runs skip the expensive EXA search for known-good (country, item) pairs.
Strategy per item:
1. Direct fetch + matchPrice on learned URL (free)
2. Firecrawl on learned URL if step 1 fails (handles JS SPAs)
3. Full EXA search only when learned route fails or is absent
4. Saves newly discovered URL as learned route for next run
Safety guarantees matching the Codex review:
- isAllowedRouteHost() validates hostname against country.sites allowlist
before both saving and replaying (prevents stored-SSRF)
- tryDirectFetch() applies CURRENCY_MIN + ITEM_USD_MAX bulk-price guards
identical to the existing EXA and Firecrawl paths
- failsSinceSuccess >= 2 triggers true DEL (not TTL wait)
- SET/DEL conflict resolved: effectiveDeletes filters keys in updates;
DELs sent before SETs in pipeline
- All operations non-fatal: pipeline failures log warnings, seed continues
New exports in _seed-utils.mjs: isAllowedRouteHost, bulkReadLearnedRoutes,
bulkWriteLearnedRoutes (1 pipeline read + 1 pipeline write per run).
BigMac deferred to Phase 2 (uses EXA summaries from aggregator pages).
Estimated savings: ~63 of 90 EXA calls skipped per run at 70% hit rate.
* test(seed): extract processItemRoute for testability; add 5 integration tests
- Move item-level decision tree into processItemRoute() in _seed-utils.mjs
so it can be imported and unit-tested without triggering runSeed()
- seed-grocery-basket.mjs delegates to processItemRoute() with fetchViaExa
callback containing the existing EXA+Firecrawl block
- 5 integration tests cover: learned-hit success (EXA skipped), learned fail
+ EXA replacement, fail x2 eviction, SSRF guard (bad host blocks direct
fetch), EXA success with unlisted host (route not saved)
- Fix: move allowedHosts computation outside Promise.all (once per country)
- Fix: add [EXA->learned] log tag when new route is saved from EXA discovery
- All 21 seed-learned-routes tests pass
* fix(seed): strip path from allowedHosts entries before hostname comparison
grocery-basket.json contains "noon.com/saudi-en" for Saudi Arabia.
allowedHosts was built with only www. stripped, so the comparison
hostname === 'noon.com/saudi-en'
was always false — noon.com routes for SA were rejected or evicted
every run, preventing the cache from ever stabilizing there.
Fix: split('/')[0] after stripping www., giving bare hostname.
Add regression test: path-bearing allowlist entry matches noon.com URL.
* fix(panels): hide finance-only panels on non-finance variants
stock-analysis, stock-backtest, and daily-market-brief are FINANCE_PANELS-only
but persisted panelSettings from a previous finance/full session caused them
to appear on tech variant. Guard creation with variantPanelKeys set derived
from VARIANT_DEFAULTS[SITE_VARIANT].
* fix(pro): remove duplicate Create with AI button, gate MCP connect as pro
With unified isProUser() both widget creation buttons showed simultaneously.
Remove the basic-tier 'Create with AI' block (duplicate of 'Create Interactive
Widget'). Gate 'Connect with MCP' with isProUser() — was previously ungated.
* feat(pro): gate export (⬇) and playback (⏪) toolbar buttons for pro users
Both setupExportPanel() and setupPlaybackControl() now return early
unless isProUser() — either wm-widget-key or wm-pro-key grants access.
* revert(panels): remove incorrect variant guard on stock/daily-market panels
The variantPanelKeys.has() guards were wrong — shouldCreatePanel() already
handles variant defaults via panelSettings. Users who manually enable these
panels on any variant should see them (locked if not pro, unlocked if pro).
The guard broke cross-variant user customization.
* fix(resilience): extend TTLs and add Finnhub/FRED fallbacks for macro signals
- MACRO_TTL 1800s → 21600s (6h) so stale data serves during Yahoo outages
- MARKET_SEED_TTL 1800s → 7200s (2h) in ais-relay for sectors/quotes
- Finnhub stock/candle fallback for QQQ and XLP when Yahoo returns null
- Finnhub crypto/candle fallback for BTC (BINANCE:BTCUSDT) when Yahoo fails
- FRED DEXJPUS fallback for JPY/USD historical prices (no new API key needed)
Prevents Macro Stress and Sector Heatmap panels from showing "unavailable"
during Yahoo Finance 429 rate-limit windows from Railway IPs.
CoinGecko was silently failing on the relay (no API key, rate-limited by
the main crypto seed loop), leaving defi/ai/other-tokens keys empty.
- seed-token-panels.mjs: rewrite with CoinGecko → CoinPaprika fallback,
use runSeed() wrapper so seed-meta:market:token-panels is written
- ais-relay.cjs: add fetchTokenPanelsCoinPaprika() fallback; relay no
longer silently drops token panels on CoinGecko failure
- health.js: add defiTokens, aiTokens, otherTokens to BOOTSTRAP_KEYS;
add tokenPanels to SEED_META (maxStaleMin: 90, 30-min cron × 3)
- Railway: provision seed-token-panels cron service (*/30 * * * *,
1vCPU/1GB, watchPatterns scoped to token panel files)
* feat(economic): add WoW tracking and fix plumbing for bigmac/grocery-basket panels
Phase 1 — Fix Plumbing:
- Adjust CACHE_TTL to 10 days (864000s) for bigmac and grocery-basket seeds
- Align health.js SEED_META maxStaleMin to 10080 (7 days) for both
- Add grocery-basket and bigmac to seed-health.js SEED_DOMAINS with intervalMin: 5040
- Refactor publish.ts writeSnapshot to accept advanceSeedMeta param; only
advance seed-meta when fresh data exists (overallFreshnessMin < 120)
- Add manual-fallback-only comment to seed-consumer-prices.mjs
Phase 2 — Week-over-Week Tracking:
- Add wow_pct field to BigMacCountryPrice and CountryBasket proto messages
- Add wow_avg_pct, wow_available, prev_fetched_at to both response protos
- Regenerate client/server TypeScript from updated protos
- Add readCurrentSnapshot() helper + WoW computation to seed-bigmac.mjs
and seed-grocery-basket.mjs; write :prev key via extraKeys
- Update BigMacPanel.ts to show per-country WoW column and global avg summary
- Update GroceryBasketPanel.ts to show WoW badge on total row and basket avg summary
- Add .bm-wow-up, .bm-wow-down, .bm-wow-summary, .gb-wow CSS classes
- Fix server handlers to include new WoW fields in fallback responses
* fix(economic): guard :prev extraKey against null on first seed run; eliminate double freshness query in publish.ts
* refactor(economic): address code review findings from PR #1974
- Extract readSeedSnapshot() into _seed-utils.mjs (DRY: was duplicated
verbatim in seed-bigmac and seed-grocery-basket)
- Add FRESH_DATA_THRESHOLD_MIN constant in publish.ts (replace magic 120)
- Fix seed-consumer-prices.mjs contradictory JSDoc (remove stale
"Deployed as: Railway cron service" line that contradicted manual-only warning)
- Add i18n keys panels.bigmacWow / panels.bigmacCountry to en.json
- Replace hardcoded "WoW" / "Country" with t() calls in BigMacPanel
- Replace IIFE-in-ternary pattern with plain if blocks in BigMacPanel
and GroceryBasketPanel (P2/P3 from code review)
* fix(publish): gate advanceSeedMeta on any-retailer freshness, not average
overallFreshnessMin is the arithmetic mean across all retailers, so with
1 fresh + 2 stale retailers the average can exceed 120 min and suppress
seed-meta advancement even while fresh data is being published.
Use retailers.some(r => r.freshnessMin < 120) to correctly implement
"at least one retailer scraped within the last 2 hours."
* fix(relay): add missing USNI regions and remove dead NZ safetravel feed
USNI fleet tracker warns on unknown regions "Tasman Sea" and "Eastern
Atlantic" — add coords to USNI_REGION_COORDS map.
safetravel.govt.nz/news/feed redirects to /404 on every request; remove
from advisory feeds and RSS allowed domains.
* fix: remove safetravel.govt.nz from edge function allowed domains copy
* fix: remove safetravel.govt.nz from shared allowed domains copy
