- Extend /wingbits/track relay endpoint to accept ?callsign=EK36 without bbox,
using global bounding box (-80/-180/80/180) for worldwide aircraft lookup
- Filter by callsign in relay response loop to return only matching flights
- Server: try Wingbits for callsign-only searches (previously only tried for bbox)
- Server: return empty positions (not simulated) for explicit callsign/icao24 lookups
- Client: filter out simulated-source positions before displaying search results
Open-Meteo rate-limits concurrent requests (429s). When 10/15 zones
failed, the seed wrote 5 all-NORMAL entries to Redis, causing the panel
to show 0 anomalies (client filters out NORMAL).
- Replace Promise.allSettled with sequential loop + 200ms delay between
zone requests to avoid triggering Open-Meteo rate limits
- Add minimum quality guard: throw if fewer than 2/3 zones return data,
so runSeed extends existing TTL instead of overwriting with partial data
- Update validate() to enforce the same minimum zone count
- Note climate cron schedule in health.js
The seed was being SIGKILL'd on Railway (512MB limit) because fast-xml-parser
built a ~300MB object tree from the 120MB OFAC SDN XML download, causing both
the raw XML string and the full parsed object to coexist in heap simultaneously.
Switch to sax (already a transitive dep) with a streaming pipeline: response.body
is piped chunk-by-chunk via a TextDecoder into the SAX parser. The full XML
string is never held in memory. Reference maps (areaCodes, featureTypes,
legalBasis, locations, parties) are populated as SAX events arrive, and entries
are emitted one at a time on </SanctionsEntry>.
All DOM-traversal helpers (listify, textValue, buildEpoch, buildReferenceMaps,
buildLocationMap, extractPartyName, resolveEntityType, extractPartyCountries,
buildPartyMap, extractPrograms, extractEffectiveAt, extractNote,
buildEntriesForDocument) are removed. Output-stage pure functions (uniqueSorted,
compactNote, sortEntries, buildCountryPressure, buildProgramPressure) are kept.
Tests updated to match: removed test blocks for deleted DOM helpers, kept
coverage for the remaining pure functions (2173/2173 pass).
* feat(seeds): shared FX rate cache + BigMac WoW guards
- Extract SHARED_FX_FALLBACKS to _seed-utils.mjs as single source of truth,
eliminating duplicated FX fallback tables across seed-bigmac, seed-grocery-basket,
and seed-fx-rates
- Add getSharedFxRates() / fetchYahooFxRates() to _seed-utils.mjs so all seeds
share one Redis-cached rate set (shared:fx-rates:v1, 4h TTL) instead of each
making ~46 independent Yahoo Finance calls per run
- Add seed-fx-rates.mjs: dedicated daily Railway cron that pre-warms the shared
FX cache, saving ~90 Yahoo calls per weekly bigmac+grocery-basket cycle
- Add WoW minimum-age guard (6 days): prevents week-on-week display when previous
snapshot is less than 6 days old (fixes -98.5% France WoW on first seed run)
- Add per-country WoW anomaly filter (+-20%): nulls suspicious entries and logs
admin alert with country name and delta for Railway log monitoring
- Fix global WoW anomaly check to use unfiltered raw average so it can actually
exceed +-20% (filtered average was mathematically bounded and never triggered)
- Add USD price sanity range guard ($1.50-$12.00): drops prices from bad scrapes
before they reach Redis (would have caught the $470 France value)
- Move WOW_ANOMALY_THRESHOLD, MIN_WOW_AGE_MS, USD_MIN, USD_MAX to module scope
* fix(seed-fx): address PR review — TTL mismatch and partial write-back risk
- Extend shared:fx-rates:v1 TTL from 4h to 25h so cache stays warm
between daily cron runs (with 1h drift buffer)
- Make getSharedFxRates() read-only: remove write-back on partial cache
hit and on cache miss; only seed-fx-rates.mjs owns writes to this key,
preventing a subset consumer from silently overwriting a fuller cache
The sdn_advanced.xml file has grown to ~120MB. During XML_PARSER.parse()
both the raw string (~120MB) and the parsed JS object tree (~200-300MB)
coexist in memory, exceeding Railway's 512MB limit and causing a silent
SIGKILL with no error output in logs.
Block-scope the xml string so it is GC-eligible immediately after parse
returns, and yield to the event loop so GC can reclaim it before
buildEntriesForDocument allocates the entry objects.
Also corrects the stale ~10MB comment that was written before the file grew.
* feat(seed): add learned routes cache to grocery basket seed
Persists successful EXA/Firecrawl URL discoveries in Redis so subsequent
runs skip the expensive EXA search for known-good (country, item) pairs.
Strategy per item:
1. Direct fetch + matchPrice on learned URL (free)
2. Firecrawl on learned URL if step 1 fails (handles JS SPAs)
3. Full EXA search only when learned route fails or is absent
4. Saves newly discovered URL as learned route for next run
Safety guarantees matching the Codex review:
- isAllowedRouteHost() validates hostname against country.sites allowlist
before both saving and replaying (prevents stored-SSRF)
- tryDirectFetch() applies CURRENCY_MIN + ITEM_USD_MAX bulk-price guards
identical to the existing EXA and Firecrawl paths
- failsSinceSuccess >= 2 triggers true DEL (not TTL wait)
- SET/DEL conflict resolved: effectiveDeletes filters keys in updates;
DELs sent before SETs in pipeline
- All operations non-fatal: pipeline failures log warnings, seed continues
New exports in _seed-utils.mjs: isAllowedRouteHost, bulkReadLearnedRoutes,
bulkWriteLearnedRoutes (1 pipeline read + 1 pipeline write per run).
BigMac deferred to Phase 2 (uses EXA summaries from aggregator pages).
Estimated savings: ~63 of 90 EXA calls skipped per run at 70% hit rate.
* test(seed): extract processItemRoute for testability; add 5 integration tests
- Move item-level decision tree into processItemRoute() in _seed-utils.mjs
so it can be imported and unit-tested without triggering runSeed()
- seed-grocery-basket.mjs delegates to processItemRoute() with fetchViaExa
callback containing the existing EXA+Firecrawl block
- 5 integration tests cover: learned-hit success (EXA skipped), learned fail
+ EXA replacement, fail x2 eviction, SSRF guard (bad host blocks direct
fetch), EXA success with unlisted host (route not saved)
- Fix: move allowedHosts computation outside Promise.all (once per country)
- Fix: add [EXA->learned] log tag when new route is saved from EXA discovery
- All 21 seed-learned-routes tests pass
* fix(seed): strip path from allowedHosts entries before hostname comparison
grocery-basket.json contains "noon.com/saudi-en" for Saudi Arabia.
allowedHosts was built with only www. stripped, so the comparison
hostname === 'noon.com/saudi-en'
was always false — noon.com routes for SA were rejected or evicted
every run, preventing the cache from ever stabilizing there.
Fix: split('/')[0] after stripping www., giving bare hostname.
Add regression test: path-bearing allowlist entry matches noon.com URL.
* fix(panels): hide finance-only panels on non-finance variants
stock-analysis, stock-backtest, and daily-market-brief are FINANCE_PANELS-only
but persisted panelSettings from a previous finance/full session caused them
to appear on tech variant. Guard creation with variantPanelKeys set derived
from VARIANT_DEFAULTS[SITE_VARIANT].
* fix(pro): remove duplicate Create with AI button, gate MCP connect as pro
With unified isProUser() both widget creation buttons showed simultaneously.
Remove the basic-tier 'Create with AI' block (duplicate of 'Create Interactive
Widget'). Gate 'Connect with MCP' with isProUser() — was previously ungated.
* feat(pro): gate export (⬇) and playback (⏪) toolbar buttons for pro users
Both setupExportPanel() and setupPlaybackControl() now return early
unless isProUser() — either wm-widget-key or wm-pro-key grants access.
* revert(panels): remove incorrect variant guard on stock/daily-market panels
The variantPanelKeys.has() guards were wrong — shouldCreatePanel() already
handles variant defaults via panelSettings. Users who manually enable these
panels on any variant should see them (locked if not pro, unlocked if pro).
The guard broke cross-variant user customization.
* fix(resilience): extend TTLs and add Finnhub/FRED fallbacks for macro signals
- MACRO_TTL 1800s → 21600s (6h) so stale data serves during Yahoo outages
- MARKET_SEED_TTL 1800s → 7200s (2h) in ais-relay for sectors/quotes
- Finnhub stock/candle fallback for QQQ and XLP when Yahoo returns null
- Finnhub crypto/candle fallback for BTC (BINANCE:BTCUSDT) when Yahoo fails
- FRED DEXJPUS fallback for JPY/USD historical prices (no new API key needed)
Prevents Macro Stress and Sector Heatmap panels from showing "unavailable"
during Yahoo Finance 429 rate-limit windows from Railway IPs.
CoinGecko was silently failing on the relay (no API key, rate-limited by
the main crypto seed loop), leaving defi/ai/other-tokens keys empty.
- seed-token-panels.mjs: rewrite with CoinGecko → CoinPaprika fallback,
use runSeed() wrapper so seed-meta:market:token-panels is written
- ais-relay.cjs: add fetchTokenPanelsCoinPaprika() fallback; relay no
longer silently drops token panels on CoinGecko failure
- health.js: add defiTokens, aiTokens, otherTokens to BOOTSTRAP_KEYS;
add tokenPanels to SEED_META (maxStaleMin: 90, 30-min cron × 3)
- Railway: provision seed-token-panels cron service (*/30 * * * *,
1vCPU/1GB, watchPatterns scoped to token panel files)
* feat(economic): add WoW tracking and fix plumbing for bigmac/grocery-basket panels
Phase 1 — Fix Plumbing:
- Adjust CACHE_TTL to 10 days (864000s) for bigmac and grocery-basket seeds
- Align health.js SEED_META maxStaleMin to 10080 (7 days) for both
- Add grocery-basket and bigmac to seed-health.js SEED_DOMAINS with intervalMin: 5040
- Refactor publish.ts writeSnapshot to accept advanceSeedMeta param; only
advance seed-meta when fresh data exists (overallFreshnessMin < 120)
- Add manual-fallback-only comment to seed-consumer-prices.mjs
Phase 2 — Week-over-Week Tracking:
- Add wow_pct field to BigMacCountryPrice and CountryBasket proto messages
- Add wow_avg_pct, wow_available, prev_fetched_at to both response protos
- Regenerate client/server TypeScript from updated protos
- Add readCurrentSnapshot() helper + WoW computation to seed-bigmac.mjs
and seed-grocery-basket.mjs; write :prev key via extraKeys
- Update BigMacPanel.ts to show per-country WoW column and global avg summary
- Update GroceryBasketPanel.ts to show WoW badge on total row and basket avg summary
- Add .bm-wow-up, .bm-wow-down, .bm-wow-summary, .gb-wow CSS classes
- Fix server handlers to include new WoW fields in fallback responses
* fix(economic): guard :prev extraKey against null on first seed run; eliminate double freshness query in publish.ts
* refactor(economic): address code review findings from PR #1974
- Extract readSeedSnapshot() into _seed-utils.mjs (DRY: was duplicated
verbatim in seed-bigmac and seed-grocery-basket)
- Add FRESH_DATA_THRESHOLD_MIN constant in publish.ts (replace magic 120)
- Fix seed-consumer-prices.mjs contradictory JSDoc (remove stale
"Deployed as: Railway cron service" line that contradicted manual-only warning)
- Add i18n keys panels.bigmacWow / panels.bigmacCountry to en.json
- Replace hardcoded "WoW" / "Country" with t() calls in BigMacPanel
- Replace IIFE-in-ternary pattern with plain if blocks in BigMacPanel
and GroceryBasketPanel (P2/P3 from code review)
* fix(publish): gate advanceSeedMeta on any-retailer freshness, not average
overallFreshnessMin is the arithmetic mean across all retailers, so with
1 fresh + 2 stale retailers the average can exceed 120 min and suppress
seed-meta advancement even while fresh data is being published.
Use retailers.some(r => r.freshnessMin < 120) to correctly implement
"at least one retailer scraped within the last 2 hours."
* fix(relay): add missing USNI regions and remove dead NZ safetravel feed
USNI fleet tracker warns on unknown regions "Tasman Sea" and "Eastern
Atlantic" — add coords to USNI_REGION_COORDS map.
safetravel.govt.nz/news/feed redirects to /404 on every request; remove
from advisory feeds and RSS allowed domains.
* fix: remove safetravel.govt.nz from edge function allowed domains copy
* fix: remove safetravel.govt.nz from shared allowed domains copy
* fix(consumer-prices): restore dropped seed script, fix maxStaleMin
seed-consumer-prices.mjs was included in PR #1901 but accidentally
dropped during a subsequent squash merge. Restored verbatim from the
PR diff. The script fetches from consumer-prices-core and writes all
Redis keys (overview, categories, movers, spread, freshness, series).
Also corrects health.js maxStaleMin values from 2880 (48h) to match
actual seed TTLs: overview/categories/movers=90min, spread=120min,
freshness=30min. The previous values would mask outages for 2 days.
Needs CONSUMER_PRICES_CORE_BASE_URL set in Railway env to produce
real data. Without it the script writes empty-but-valid placeholders.
* fix(consumer-prices): switch publish job to Upstash HTTP REST client
Replaced standard redis TCP client (REDIS_URL → Valkey) with Upstash
HTTP REST calls (UPSTASH_REDIS_REST_URL + UPSTASH_REDIS_REST_TOKEN),
matching the pattern used by all other seed scripts. This is why all
consumer-prices Redis keys were permanently empty — publish.ts was
writing to Railway-internal Valkey instead of Upstash.
Railway env vars added to seed-consumer-prices service separately.
- seed-bigmac.mjs: expand from 9 ME countries to 50+ global (Americas, Europe, Asia-Pacific, ME, Africa); derive FX_SYMBOLS from COUNTRIES; expand CCY regex and FX_FALLBACKS
- ConsumerPricesPanel: show structured seeding placeholder (icon + title + sub) when upstreamUnavailable instead of bare empty text
- main.css: add cp-* styles for ConsumerPricesPanel and gb-* styles for GroceryBasketPanel (were missing entirely)
* chore(iran-seed): add minab, asaluyeh, south pars, al-ahmadi location coords
* fix(sanctions): remove redundant Total card from summary grid
Total count is already shown in the panel header badge, making the
large card duplicate information. Keep New, Vessels, Aircraft cards.
- Add migration 002: seed canonical_products, baskets, basket_items for essentials-ae
- Add migration 003: partial unique index to fix NULL uniqueness gap in canonical_products
- Add ExaSearchAdapter to scrape.ts; auto-creates canonical product→basket matches
- Fix getBasketItemId to lookup by canonical_name via JOIN (not category) to avoid
dairy collision (3 items share same category)
- Fix getOrCreateRetailer race condition with ON CONFLICT upsert
- Add per-category index writes in aggregate.ts; guard division-by-zero
- Set all publish.ts TTLs to 93600s (26h) to survive cron scheduling drift
- Set health.js maxStaleMin to 2880 (daily cron × 2) for correct staleness detection
- Remove redundant seed-consumer-prices.mjs (publish.ts writes to Redis directly)
- Add src/cli/validate.ts DB health check script
- Fix z.record() to z.record(z.string(), z.unknown()) for Zod compat
* feat(consumer-prices): add basket price monitoring domain
Adds end-to-end consumer price tracking to enable inflation monitoring
across key markets, starting with UAE essentials basket.
- consumer-prices-core/: companion scraping service with pluggable
acquisition providers (Playwright, Exa, Firecrawl, Parallel P0),
config-driven retailer YAML, Postgres schema, Redis snapshots
- proto/worldmonitor/consumer_prices/v1/: 6-RPC service definition
- api/consumer-prices/v1/[rpc].ts: Vercel edge route
- server/worldmonitor/consumer-prices/v1/: Redis-backed RPC handlers
- src/services/consumer-prices/: circuit breakers + bootstrap hydration
- src/components/ConsumerPricesPanel.ts: 5-tab panel (overview /
categories / movers / spread / health)
- scripts/seed-consumer-prices.mjs: Railway cron seed script
- Wire into bootstrap, health, panels, gateway, cache-keys, locale
* fix(consumer-prices): resolve all code review findings
P0: populate topCategories — categoryResult was fetched but never used.
Added buildTopCategories() helper with grouped CTE query that extracts
current_index and week-over-week pct per category.
P1 (4 fixes):
- aggregate: replace N+1 getBaselinePrice loop with single batch query
getBaselinePrices(ids[], date) via ANY($1) — eliminates 119 DB roundtrips
per basket run
- aggregate/computeValueIndex: was dividing all category floors by the same
arbitrary first baseline; now uses per-item floor price with per-item
baseline (same methodology as fixed index but with cheapest price)
- basket-series endpoint now seeded: added buildBasketSeriesSnapshot() to
worldmonitor.ts, /basket-series route in companion API, publish.ts writes
7d/30d/90d series per basket, seed script fetches and writes all three ranges
- scrape: call teardownAll() after each retailer run to close Playwright
browser; without this the Chromium process leaked on Railway
P2 (4 fixes):
- db/client: remove rejectUnauthorized: false — was bypassing TLS cert
validation on all non-localhost connections
- publish: seed-meta now writes { fetchedAt, recordCount } matching the format
expected by _seed-utils.mjs writeExtraKeyWithMeta (was writing { fetchedAt, key })
- products: remove unused getMatchedProductsForBasket — exact duplicate of
getBasketRows in aggregate.ts; never imported by anything
Snapshot type overhaul:
- Flatten WMOverviewSnapshot to match proto GetConsumerPriceOverviewResponse
(was nested under overview:{}; handlers read flat)
- All asOf fields changed from number to string (int64 → string per proto JSON)
- freshnessMin/parseSuccessRate null -> 0 defaults
- lastRunAt changed from epoch number to ISO string
- Mover items now include currentPrice and currencyCode
- emptyOverview/Movers/Spread/Freshness in seed script use String(Date.now())
* feat(consumer-prices): wire Exa search engine as acquisition backend for UAE retailers
Ports the proven Exa+summary price extraction from PR #1904 (seed-grocery-basket.mjs)
into consumer-prices-core as ExaSearchAdapter, replacing unvalidated Playwright CSS
scraping for all three UAE retailers (Carrefour, Lulu, Noon).
- New ExaSearchAdapter: discovers targets from basket YAML config (one per item),
calls Exa API with contents.summary to get AI-extracted prices, uses matchPrice()
regex (ISO codes + symbol fallback + CURRENCY_MIN guards) to extract AED amounts
- New db/queries/matches.ts: upsertProductMatch() + getBasketItemId() for auto-linking
scraped Exa results to basket items without a separate matching step
- scrape.ts: selects ExaSearchAdapter when config.adapter === 'exa-search'; after
insertObservation(), auto-creates canonical product and product_match (status: 'auto')
so aggregate.ts can compute indices immediately without manual review
- All three UAE retailer YAMLs switched to adapter: exa-search and enabled: true;
CSS extraction blocks removed (not used by search adapter)
- config/types.ts: adds 'exa-search' to adapter enum
* fix(consumer-prices): use EXA_API_KEYS (with fallback to EXA_API_KEY) matching PR #1904 pattern
* fix(consumer-prices): wire ConsumerPricesPanel in layout + fix movers limit:0 bug
Addresses Codex P1 findings on PR #1901:
- panel-layout.ts: import and createPanel('consumer-prices') so the panel
actually renders in finance/commodity variants where it is enabled in config
- consumer-prices/index.ts: limit was hardcoded 0 causing slice(0,0) to always
return empty risers/fallers after bootstrap is consumed; fixed to 10
* fix(consumer-prices): add categories snapshot to close P2 gap
consumer-prices:categories:ae:* was in BOOTSTRAP_KEYS but had no producer,
so the Categories tab always showed upstreamUnavailable.
- buildCategoriesSnapshot() in worldmonitor.ts — wraps buildTopCategories()
and returns WMCategoriesSnapshot matching ListConsumerPriceCategoriesResponse
- /categories route in consumer-prices-core API
- publish.ts writes consumer-prices:categories:{market}:{range} for 7d/30d/90d
- seed-consumer-prices.mjs fetches all three ranges from consumer-prices-core
and writes them to Redis alongside the other snapshots
P1 issues (snapshot structure mismatch + limit:0 movers) were already fixed
in earlier commits on this branch.
* fix(types): add variants? to PANEL_CATEGORY_MAP type
* feat(economic): add ME grocery basket price index
Adds a grocery basket price comparison panel for 9 Middle East
countries (UAE, KSA, Qatar, Kuwait, Bahrain, Oman, Egypt, Jordan,
Lebanon) using EXA AI to discover prices from regional e-commerce
sites (Carrefour, Lulu, Noon, Amazon) and Yahoo Finance for FX rates.
- proto: ListGroceryBasketPrices RPC with CountryBasket/GroceryItemPrice messages
- seed: seed-grocery-basket.mjs, 90 EXA calls/run, 150ms delay, hardcoded
FX fallbacks for pegged GCC currencies, 6h TTL
- handler: seed-only RPC reading economic:grocery-basket:v1
- gateway: static cache tier for the new route
- bootstrap/health: groceryBasket key in SLOW tier, 720min stale threshold
- frontend: GroceryBasketPanel with scrollable table, cheapest/priciest
column highlighting, styles moved to panels.css
- panel disabled by default until seed is run on Railway
* fix(generated): restore @ts-nocheck in economic service codegen
* fix(grocery-basket): tighten seed health staleness and seed script robustness
- Set maxStaleMin to 360 (6h) matching CACHE_TTL so health alerts on first missed run
- Use ?? over || for FX fallback to handle 0-value rates correctly
- Add labeled regex patterns with bare-number warning in extractPrice
- Replace conditional delay logic with unconditional per-item sleep
* fix(grocery-basket): fix EXA API format and price extraction after live validation
- Use contents.summary format (not top-level summary) — previous format returned no data
- Support EXA_API_KEYS (comma-separated) in addition to EXA_API_KEY
- Extract price from plain-text summary string (EXA returns text, not JSON schema)
- Remove bare-number fallback — too noisy (matched "500" from "pasta 500g" as SAR 500)
- Fix LBP FX rate zero-guard: use fallback when Yahoo returns 0 for ultra-low-value currencies
Validated locally: 9 countries seeded, Redis write confirmed, ~111s runtime
* fix(grocery-basket): validate extracted currency against expected country currency
- matchPrice now returns the currency code alongside the price
- extractPrice rejects results where currency != expected country currency
(prevents AED prices from being treated as JOD prices on gcc.luluhypermarket.com)
- Tighten item queries (white granulated sugar, spaghetti pasta, etc.) to reduce
irrelevant product matches like Stevia on sugar queries
- Replace Jordan's gcc.luluhypermarket.com (GCC-only) with carrefour.jo + ounasdelivery.com
- Sync scripts/shared/grocery-basket.json
* feat(bigmac): add Big Mac Index seed + drop grocery basket includeDomains
Grocery basket:
- Remove includeDomains restriction — EXA neural search finds better sources
than hardcoded domain lists; currency validation prevents contamination
- Tighten query strings (supermarket retail price suffix)
Big Mac seed (scripts/seed-bigmac.mjs):
- Two-tier search: specialist sites (theburgerindex.com, eatmyindex.com) first,
fall back to open search for countries without per-country indexed pages
- Handle thousands-separator prices (480,000 LBP)
- Accept USD prices from cost-of-living index sites as fallback
- Exclude ranking/average pages (Numbeo country_price_rankings, Expatistan)
- Validated live: 7/9 countries with confirmed prices
UAE=19AED, KSA=19SAR, QAR=22QAR, KWD=1.4KWD, EGP=135EGP, JOD=3JOD, LBP=480kLBP
* feat(economic): expand grocery basket to 24 global countries, drop Big Mac tier-2 search
Grocery basket: extend coverage from 9 MENA to 24 countries across all
regions (US, UK, DE, FR, JP, CN, IN, AU, CA, BR, MX, ZA, TR, NG, KR,
SG, PK, AE, SA, EG, KE, AR, ID, PH). Add FX fallbacks and fxSymbols
for all 23 new currencies. CCY regex in seed script updated to match
all supported currency codes.
Big Mac: remove tier-2 open search (too noisy, non-specialist pages
report combo prices or global averages). Specialist sites only
(theburgerindex.com, eatmyindex.com) for clean per-country data.
* feat(bigmac): wire Big Mac Index RPC, proto, bootstrap, health
Add ListBigMacPrices RPC end-to-end:
- proto/list_bigmac_prices.proto: BigMacCountryPrice + request/response
- service.proto: register ListBigMacPrices endpoint (GET /list-bigmac-prices)
- buf generate: regenerate service_server.ts + all client stubs
- server/list-bigmac-prices.ts: seed-only handler reads economic:bigmac:v1
- handler.ts: wire listBigMacPrices into EconomicServiceHandler
- api/bootstrap.js: bigmac key in BOOTSTRAP_CACHE_KEYS + SLOW_KEYS
- api/health.js: bigmac key in BOOTSTRAP_KEYS + SEED_META (maxStaleMin: 1440)
- _bootstrap-cache-key-refs.ts: groceryBasket + bigmac refs
* feat(bigmac): add BigMacPanel + register in panel layout
BigMacPanel renders a country-by-country Big Mac price table sorted by
USD price (cheapest/most expensive highlighted). Wired into bootstrap
hydration, refresh scheduler, and panel registry. Registered in
panels.ts (enabled: false, to be flipped once seed data is verified).
Also updates grocery basket i18n from ME-specific to global wording.
* fix(bigmac): register bigmac in cache-keys and RPC_CACHE_TIER
Add bigmac to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in
server/_shared/cache-keys.ts, and to RPC_CACHE_TIER (static tier)
in gateway.ts. Both were caught by bootstrap and RPC tier parity tests.
* fix(generated): restore @ts-nocheck in all generated service files after buf regenerate
buf generate does not emit @ts-nocheck. Previous convention restores it
manually post-generate to suppress strict type errors in generated code.
* fix(grocery-basket): restore includeDomains per country, add userLocation, fix currency symbol parsing
Root cause of 0-item countries (UK, JP, IN, NG): removing includeDomains
caused EXA neural search to return USD-priced global comparison pages
(Numbeo, Tridge, Expatistan) which currency validation correctly rejected.
Fixes:
- Add per-country sites[] in grocery-basket.json (researched local
supermarket/retailer domains: tesco.com, kaufland.de, bigbasket.com, etc.)
- Pass includeDomains: country.sites to restrict EXA to local retailers
- Pass userLocation: country.code (ISO) to bias results to target country
- Add currency symbol fallback regex (£→GBP, €→EUR, ¥→JPY, ₹→INR,
₩→KRW, ₦→NGN, R$→BRL) — sites like BigBasket use ₹ not INR
- Summary query now explicitly requests ISO currency code
- Simplify item queries (drop country name — context from domains)
Smoke test results:
UK sugar → GBP 1.09 (tesco.com) ✓
IN rice → ₹66 (bigbasket.com) ✓
JP rice → JPY 500 (kakaku.com) ✓
* fix(grocery-basket): add Firecrawl fallback, parallel items, bulk caps, currency floors
- Add Firecrawl as JS-SPA fallback after EXA (handles noon.com, coupang, daraz, tokopedia, lazada)
- Parallelize item fetching per country with 200ms stagger: runtime 38min to ~4.5min
- Add CURRENCY_MIN floors (NGN:50, IDR:500, KRW:1000, etc.) to reject product codes as prices
- Add ITEM_USD_MAX bulk caps (sugar:8USD, salt:5USD, rice:6USD, etc.) applied to both EXA and Firecrawl
- Fix SA: use noon.com/saudi-en + carrefour.com.sa (removes luluhypermarket cross-country pollution)
- Fix EG: use carrefouregypt.com + spinneys.com.eg + seoudi.com (removes GCC luluhypermarket)
- Expand sites for DE, MX, ZA, TR, NG, KR, IN, PK, AR, ID, PH to improve coverage
- Sync scripts/shared/grocery-basket.json with shared/grocery-basket.json
* fix(grocery-basket): address PR review comments P1+P2
P1 - fix ranking with incomplete data:
only include countries with >=70% item coverage (>=7/10) in
cheapest/mostExpensive ranking — prevents a country with 4/10
items appearing cheapest due to missing data
P1 - fix regex false-match on pack sizes / weights:
try currency-first pattern (SAR 8.99) before number-first to
avoid matching pack counts; use matchAll and take last match
P2 - mark seed-miss responses as non-cacheable:
add upstream_unavailable to proto + return true on empty seed
so gateway sets Cache-Control: no-store on cold deploy
* fix(generated): update EconomicService OpenAPI docs for upstream_unavailable field
* feat(economic): add National Debt Clock panel with IMF + Treasury data
- Proto: GetNationalDebt RPC in EconomicService with NationalDebtEntry message
- Seed: seed-national-debt.mjs fetches IMF WEO (debt%, GDP, deficit%) + US Treasury FiscalData in parallel; filters aggregates/territories; sorts by total debt; 35-day TTL for monthly Railway cron
- Handler: get-national-debt.ts reads seeded Redis cache key economic:national-debt:v1
- Registry: nationalDebt key added to cache-keys.ts, bootstrap.js (SLOW tier), health.js (maxStaleMin=10080), gateway.ts (daily cache tier)
- Service: getNationalDebtData() in economic/index.ts with bootstrap hydration + RPC fallback
- Panel: NationalDebtPanel.ts with sort tabs (Total/Debt-GDP/1Y Growth), search, live ticking via direct DOM manipulation (avoids setContent debounce)
- Tests: 10 seed formula tests + 8 ticker math tests; all 2064 suite tests green
* fix(economic): address code review findings for national debt clock
* fix(economic): guard runSeed() call to prevent process.exit in test imports
seed-national-debt.mjs called runSeed() at module top-level. When imported
by tests (to access computeEntries), the seed ran, hit missing Redis creds
in CI, and called process.exit(1), failing the entire test suite.
Guard with isMain check so runSeed() only fires on direct execution.
Military detector forecasts (ADS-B flight tracking + theater posture API)
structurally score near zero on readiness metrics that require LLM-enriched
caseFile content (supporting evidence, news headlines, calibration, triggers).
This causes them to rank below the target count threshold every run despite
a valid elevated posture signal.
Add a domain guarantee post-pass after the 3 selection loops: if no military
forecast was selected and we have room below MAX_TARGET_PUBLISHED_FORECASTS,
inject the highest-scoring eligible military forecast. This does not displace
any already-selected forecast and respects all existing family/situation caps.
Diagnosis: Baltic theater at postureLevel='elevated' with 6 active flights
generates a military forecast (prob=0.41, confidence=0.30, score=0.136) but
gets buried behind 15+ well-grounded situation cluster forecasts at score 0.4+.
Tests: 3 new assertions in 'military domain guarantee in publish selection'.
* fix(forecast): block generic-actor cross-theater interactions + raise enrichment budget
Root cause: actor registry uses name:category as key (e.g. "Incumbent leadership:state"),
causing unrelated situations (Israel conflict, Taiwan political) to share the same actor
ID and fire sharedActor=true in pushInteraction. This propagated into the reportable
ledger and surfaced as junk effects like Israel→Taiwan at 80% confidence.
Two-pronged fix:
1. Specificity gate in pushInteraction: sharedActor now requires avgSpecificity >= 0.75.
Generic blueprint actors ("Incumbent leadership" ~0.68, "Civil protection authorities"
~0.73) no longer qualify as structural cross-situation links. Named domain-specific
actors ("Threat actors:adversarial" ~0.95) continue to qualify.
2. MACRO_REGION_MAP + isCrossTheaterPair + gate in buildCrossSituationEffects: for
cross-theater pairs (different macro-regions) with non-exempt channels, requires
sharedActor=true AND avgActorSpecificity >= 0.90. Exempt channels: cyber_disruption,
market_repricing (legitimately global). Same-macro-region pairs (Brazil/Mexico both
AMERICAS) are unaffected.
Verified against live run 1773983083084-bu6b1f:
BLOCKED: Israel→Taiwan (MENA/EAST_ASIA, spec 0.68)
BLOCKED: Israel→US political (MENA/AMERICAS, spec 0.68)
BLOCKED: Cuba→Iran (AMERICAS/MENA, spec 0.73)
BLOCKED: Brazil→Israel (AMERICAS/MENA, spec 0.85 < 0.90)
ALLOWED: China→US cyber_disruption (exempt channel)
ALLOWED: Brazil→Mexico (same AMERICAS)
Also raises ENRICHMENT_COMBINED_MAX from 3 to 5 (total budget 6→8),
targeting enrichedRate improvement from ~38% to ~60%.
* fix(plans): fix markdown lint errors in forecast semantic quality plan
* fix(plans): fix remaining markdown lint error in plan file
* fix(cors): use ACAO: * for bootstrap to fix CF cache origin pinning
CF ignores Vary: Origin and pins the first request's ACAO header on the
cached response. Preview deployments from *.vercel.app got ACAO: worldmonitor.app
from CF's cache, blocking CORS. Bootstrap data is fully public (world events,
market prices, seismic data) so ACAO: * is safe and allows CF to cache one
entry valid for all origins. isDisallowedOrigin() still gates non-cache paths.
* chore: finish security triage
* fix(aviation): update isArray callback signature for fast-xml-parser 5.5.x
fast-xml-parser bumped from 5.4.2 to 5.5.7 changed the isArray callback's
second parameter type from string to unknown. Guard with typeof check before
calling .test() to satisfy the new type contract.
* docs: fix MD032 blank lines around lists in tradingview-screener-integration
* fix(security): address code review findings from PR #1903
- api/_json-response.js: add recursion depth limit (20) to sanitizeJsonValue
and strip Error.cause chain alongside stack/stackTrace
- scripts/ais-relay.cjs: extract WORLD_BANK_COUNTRY_ALLOWLIST to module level
to eliminate duplicate; clamp years param to [1,30] to prevent unbounded
World Bank date ranges
- src-tauri/sidecar/local-api-server.mjs: use JSON.stringify for vq value
in inline JS, consistent with safeVideoId/safeOrigin handling
- src/services/story-share.ts: simplify sanitizeStoryType to use typed array
instead of repeated as-casts
* fix(desktop): use parent window origin for YouTube embed postMessage
Sidecar youtube-embed route was targeting the iframe's own localhost origin
for all window.parent.postMessage calls, so browsers dropped yt-ready/
yt-state/yt-error on Tauri builds where the parent is tauri://localhost or
asset://localhost. LiveNewsPanel and LiveWebcamsPanel already pass
parentOrigin=window.location.origin in the embed URL; the sidecar now reads,
validates, and uses it as the postMessage target for all player event
messages. The YT API playerVars origin/widget_referrer continue to use the
sidecar's own localhost origin which YouTube requires.
Also restore World Bank relay to a generic proxy: replace TECH_INDICATORS
membership check with a format-only regex so any valid indicator code
(NY.GDP.MKTP.CD etc.) is accepted, not just the 16 tech-sector codes.
CACHE_TTL = 1800 equaled the cron interval. Any missed run or Polymarket
timeout caused the key to expire, RPC returns [], circuit breaker returns
[], panel shows "Predictions temporarily unavailable".
Also raise health.js maxStaleMin: 30 → 90 to stop false-positive alerts
on the first slightly-delayed cron run.
* feat(market): add crypto sectors heatmap and token panels (DeFi, AI, Other) backend
- Add shared/crypto-sectors.json, defi-tokens.json, ai-tokens.json, other-tokens.json configs
- Add scripts/seed-crypto-sectors.mjs and seed-token-panels.mjs seed scripts
- Add proto messages for ListCryptoSectors, ListDefiTokens, ListAiTokens, ListOtherTokens
- Add change7d field (field 6) to CryptoQuote proto message
- Run buf generate to produce updated TypeScript bindings
- Add server handlers for all 4 new RPCs reading from seeded Redis cache
- Wire handlers into marketHandler and register cache keys with BOOTSTRAP_TIERS=slow
- Wire seedCryptoSectors and seedTokenPanels into ais-relay.cjs seedAllMarketData loop
* feat(panels): add crypto sectors heatmap and token panels (DeFi, AI, Other)
- Add TokenData interface to src/types/index.ts
- Wire ListCryptoSectorsResponse/ListDefiTokensResponse/ListAiTokensResponse/ListOtherTokensResponse into market service with circuit breakers and hydration fallbacks
- Add CryptoHeatmapPanel, TokenListPanel, DefiTokensPanel, AiTokensPanel, OtherTokensPanel to MarketPanel.ts
- Register 4 new panels in panels.ts FINANCE_PANELS and cryptoDigital category
- Instantiate new panels in panel-layout.ts
- Load data in data-loader.ts loadMarkets() alongside existing crypto fetch
* fix(crypto-panels): resolve test failures and type errors post-review
- Add @ts-nocheck to regenerated market service_server/client (matches repo convention)
- Add 4 new RPC routes to RPC_CACHE_TIER in gateway.ts (route-cache-tier test)
- Sync scripts/shared/ with shared/ for new token/sector JSON configs
- Restore non-market generated files to origin/main state (avoid buf version diff)
* fix(crypto-panels): address code review findings (P1-P3)
- ais-relay seedTokenPanels: add empty-guard before Redis write to
prevent overwriting cached data when all IDs are unresolvable
- server _feeds.ts: sync 4 missing crypto feeds (Wu Blockchain, Messari,
NFT News, Stablecoin Policy) with client-side feeds.ts
- data-loader: expose panel refs outside try block so catch can call
showRetrying(); log error instead of swallowing silently
- MarketPanel: replace hardcoded English error strings with t() calls
(failedSectorData / failedCryptoData) to honour user locale
- seed-token-panels.mjs: remove unused getRedisCredentials import
- cache-keys.ts: one BOOTSTRAP_TIERS entry per line for consistency
* fix(crypto-panels): three correctness fixes for RSS proxy, refresh, and Redis write visibility
- api/_rss-allowed-domains.js: add 7 new crypto domains (decrypt.co,
blockworks.co, thedefiant.io, bitcoinmagazine.com, www.dlnews.com,
cryptoslate.com, unchainedcrypto.com) so rss-proxy.js accepts the
new finance feeds instead of rejecting them as disallowed hosts
- src/App.ts: add crypto-heatmap/defi-tokens/ai-tokens/other-tokens to
the periodic markets refresh viewport condition so panels on screen
continue receiving live updates, not just the initial load
- ais-relay seedTokenPanels: capture upstashSet return values and log
PARTIAL if any Redis write fails, matching seedCryptoSectors pattern
Railway datacenter IPs are blocked by USNI's Cloudflare. Reuses the
existing ytFetchViaProxy/parseProxyUrl pattern with OREF_PROXY_AUTH.
Falls back to direct fetch when proxy is not configured (dev/local).
Replaces curl+proxy approach which failed because curl is not available
in Railway's Node.js container (ENOENT).
* fix(relay): load COMMODITY_SYMBOLS from shared/commodities.json
The relay had a hardcoded 6-symbol list (VIX, Gold, Oil, NatGas, Silver,
Copper) that overwrote market:commodities-bootstrap:v1 every ~40 minutes,
wiping Platinum, Palladium, Aluminum, Brent, Gasoline, Heating Oil,
Uranium, and Lithium seeded by seed-commodity-quotes.mjs.
Use requireShared('commodities.json') — same source as the seed script
and the frontend — so adding a commodity to the JSON file is the only
change needed. YAHOO_ONLY is auto-derived: futures (=F) and indices (^).
* ci: trigger typecheck
* fix(relay): restore ^GSPC/^DJI/^IXIC in YAHOO_ONLY and add URA/LIT
Deriving YAHOO_ONLY from COMMODITY_SYMBOLS alone dropped the three major
indices (^GSPC, ^DJI, ^IXIC) which live in MARKET_SYMBOLS, not COMMODITY_SYMBOLS.
With Finnhub API key present, seedMarketQuotes() would route them through
Finnhub instead of Yahoo, potentially caching wrong index values.
Explicit list: indices hardcoded + futures/^ from COMMODITY_SYMBOLS + ETFs.
* fix(relay): replace proxy+curl with native fetch for USNI seed
USNI WordPress API is a public endpoint (no auth required). The previous
implementation required RESIDENTIAL_PROXY_AUTH or OREF_PROXY_AUTH to be
set, silently skipping the seed when neither was available in Railway.
* fix(ci): remove paths-ignore from typecheck workflow
When a PR only touches scripts/, the typecheck workflow was skipped
entirely. GitHub posts no status, so the required 'typecheck' check
is never satisfied and merge is blocked.
paths-ignore is unsafe on required status checks.
* fix(cable-health): extend Redis TTL from 1h to 24h to survive relay downtime
CACHE_TTL was 3600s (1h). Relay pings every 30min. Two consecutive missed
pings expired the key, leaving health.js reading null and reporting EMPTY.
cachedFetchJson is cache-aside, not stale-while-revalidate: once the key
expires from Redis there is no stale fallback at the health.js layer.
24h TTL keeps the key alive through multi-hour relay outages; the 30min
warm-ping still keeps data fresh in normal operation.
* fix(relay): extend 2x-interval TTLs to 6x to survive relay downtime
Keys with TTL=2x their seed interval expire after just one missed ping.
Same root cause as cable-health (CACHE_TTL=3600 with 30min ping).
Changes:
- CHOKEPOINT_TRANSIT_TTL: 1200s → 3600s (10min interval)
- TRANSIT_SUMMARY_TTL: 1200s → 3600s (10min interval)
- WEATHER_CACHE_TTL: 1800s → 5400s (15min interval)
- AVIATION_SEED_TTL: 3600s → 10800s (30min interval, intl delays)
- NOTAM_SEED_TTL: 3600s → 10800s (30min interval)
All now at 6x their seed interval, matching the gold standard pattern.
* test: update TRANSIT_SUMMARY_TTL assertion to require 6x interval minimum
buildForecastTraceArtifacts was building worldState after tracedPredictions,
so simulation data was never available to buildForecastTraceRecord. Each
forecast's caseFile.worldState had situationId/familyId/simulationSummary
all undefined, making the 3-round MiroFish simulation invisible at the
forecast level.
Fix:
- Compute worldState before tracing (so simulationState is ready)
- Build forecastId → situationSimulation lookup from worldState.simulationState
- Pass lookup into buildForecastTraceRecord; inject situationId, familyId,
familyLabel, simulationSummary, simulationPosture, simulationPostureScore
into caseFile.worldState for each matched forecast
- Add regression assertion to forecast-trace-export tests
All 194 forecast tests pass.
* feat: self-hosted Docker stack with nginx, Redis REST proxy, and seeders
Multi-stage Docker build: esbuild TS handler compilation, vite frontend
build, nginx + Node.js API under supervisord. Upstash-compatible Redis
REST proxy with command allowlist for security. AIS relay WebSocket
sidecar. Seeder wrapper script with auto-sourced env vars from
docker-compose.override.yml. Self-hosting guide with architecture
diagram, API key setup, and troubleshooting.
Security: Redis proxy command allowlist (blocks FLUSHALL/CONFIG/EVAL),
nginx security headers (X-Content-Type-Options, X-Frame-Options,
Referrer-Policy), non-root container user.
* feat(docker): add Docker secrets support for API keys
Entrypoint reads /run/secrets/* files and exports as env vars at
startup. Secrets take priority over environment block values and
stay out of docker inspect / process metadata.
Both methods (env vars and secrets) work simultaneously.
* fix(docker): point supervisord at templated nginx config
The entrypoint runs envsubst on nginx.conf.template and writes
the result to /tmp/nginx.conf (with LOCAL_API_PORT substituted
and listening on port 8080 for non-root). But supervisord was
still launching nginx with /etc/nginx/nginx.conf — the default
Alpine config that listens on port 80, which fails with
"Permission denied" under the non-root appuser.
* fix(docker): remove KEYS from Redis allowlist, fix nginx header inheritance, add LLM vars to seeders
- Remove KEYS from redis-rest-proxy allowlist (O(N) blocking, Redis DoS risk)
- Move security headers into each nginx location block to prevent add_header
inheritance suppression
- Add LLM_API_URL / LLM_API_KEY / LLM_MODEL to run-seeders.sh grep filter
so LLM API keys set in docker-compose.override.yml are forwarded to seed scripts
* fix(docker): add path-based POST to Redis proxy, expand allowlist, add missing seeder secrets
- Add POST /{command}/{args...} handler to redis-rest-proxy so Upstash-style
path POSTs work (setCachedJson uses POST /set/<key>/<value>/EX/<ttl>)
- Expand allowlist: HLEN, LTRIM (seed-military-bases, seed-forecasts),
ZREVRANGE (premium-stock-store), ZRANDMEMBER (seed-military-bases)
- Add ACLED_EMAIL, ACLED_PASSWORD, OPENROUTER_API_KEY, OLLAMA_API_URL,
OLLAMA_MODEL to run-seeders.sh so override keys reach host-run seeders
---------
Co-authored-by: Elie Habib <elie.habib@gmail.com>
* feat(config): add NSE and BSE (India) market support (#1102)
* fix(india-markets): wire NSE/BSE symbols into stocks.json so seed fetches them
- Add 20 India symbols (^NSEI, ^BSESN, 18x .NS equities) to shared/stocks.json
- Mark all .NS symbols + indices as yahooOnly (Finnhub does not support NSE)
- Remove orphan src/config/india-markets.ts; stocks.json is the seed source of truth
* fix(india-markets): sync scripts/shared/stocks.json mirror
* fix(ci): exclude scripts/data/ and scripts/node_modules/ from unicode safety scan
---------
Co-authored-by: lspassos1 <lspassos@icloud.com>
* ci: skip typecheck for scripts-only PRs; fix vercel-ignore empty SHA
Typecheck workflow:
- Add paths-ignore for scripts/** and .github/** on pull_request and push.
Seed scripts are plain .mjs — not TypeScript — so typechecking adds ~2min
with zero coverage benefit for scripts-only changes.
vercel-ignore.sh:
- When VERCEL_GIT_PREVIOUS_SHA is empty or invalid (can happen on incremental
PR pushes), fall back to git merge-base HEAD origin/main instead of defaulting
to exit 1 (build). This was causing Vercel to deploy on scripts-only PRs even
though the ignore script correctly excludes scripts/ from web-relevant paths.
* fix(ci): remove .github/** from typecheck paths-ignore to unblock PR
* security: add unicode safety guard to hooks and CI
* fix(unicode-safety): drop FE0F, PUA; fix col tracking; scan .husky/
- Remove FE0F (emoji presentation selector) from suspicious set — it
false-positives on ASCII keycap sequences (#️⃣ etc.) in source strings
- Remove Private Use Area (E000–F8FF) check — not a parser attack vector
and legitimately used by icon font string literals
- Fix column tracking for astral-plane characters (cp > 0xFFFF): increment
by 2 to match UTF-16 editor column positions
- Remove now-unused prevCp variable
- Add .husky/ to SCAN_ROOTS and '' to INCLUDED_EXTENSIONS so extensionless
hook scripts (pre-commit, pre-push) are included in full-repo scans
---------
Co-authored-by: Elie Habib <elie.habib@gmail.com>
