* feat(commodities): expand tracking to cover agricultural and coal futures
Adds 9 new commodity symbols to cover the price rally visible in our
intelligence feeds: Newcastle Coal (MTF=F), Wheat (ZW=F), Corn (ZC=F),
Soybeans (ZS=F), Rough Rice (ZR=F), Coffee (KC=F), Sugar (SB=F),
Cocoa (CC=F), and Cotton (CT=F).
Also fixes ais-relay seeder to use display names from commodities.json
instead of raw symbols, so seeded data is self-consistent.
* fix(commodities): gold standard cache, 3-col grid, cleanup
- Add upstashExpire on zero-quotes failure path so bootstrap key TTL
extends during Yahoo outages (gold standard pattern)
- Remove unreachable fallback in retry loop (COMMODITY_META always has
the symbol since it mirrors COMMODITY_SYMBOLS)
- Switch commodities panel to 3-column grid (19 items → ~7 rows vs 10)
* fix(relay): add missing USNI regions and remove dead NZ safetravel feed
USNI fleet tracker warns on unknown regions "Tasman Sea" and "Eastern
Atlantic" — add coords to USNI_REGION_COORDS map.
safetravel.govt.nz/news/feed redirects to /404 on every request; remove
from advisory feeds and RSS allowed domains.
* fix: remove safetravel.govt.nz from edge function allowed domains copy
* fix: remove safetravel.govt.nz from shared allowed domains copy
* feat(economic): add ME grocery basket price index
Adds a grocery basket price comparison panel for 9 Middle East
countries (UAE, KSA, Qatar, Kuwait, Bahrain, Oman, Egypt, Jordan,
Lebanon) using EXA AI to discover prices from regional e-commerce
sites (Carrefour, Lulu, Noon, Amazon) and Yahoo Finance for FX rates.
- proto: ListGroceryBasketPrices RPC with CountryBasket/GroceryItemPrice messages
- seed: seed-grocery-basket.mjs, 90 EXA calls/run, 150ms delay, hardcoded
FX fallbacks for pegged GCC currencies, 6h TTL
- handler: seed-only RPC reading economic:grocery-basket:v1
- gateway: static cache tier for the new route
- bootstrap/health: groceryBasket key in SLOW tier, 720min stale threshold
- frontend: GroceryBasketPanel with scrollable table, cheapest/priciest
column highlighting, styles moved to panels.css
- panel disabled by default until seed is run on Railway
* fix(generated): restore @ts-nocheck in economic service codegen
* fix(grocery-basket): tighten seed health staleness and seed script robustness
- Set maxStaleMin to 360 (6h) matching CACHE_TTL so health alerts on first missed run
- Use ?? over || for FX fallback to handle 0-value rates correctly
- Add labeled regex patterns with bare-number warning in extractPrice
- Replace conditional delay logic with unconditional per-item sleep
* fix(grocery-basket): fix EXA API format and price extraction after live validation
- Use contents.summary format (not top-level summary) — previous format returned no data
- Support EXA_API_KEYS (comma-separated) in addition to EXA_API_KEY
- Extract price from plain-text summary string (EXA returns text, not JSON schema)
- Remove bare-number fallback — too noisy (matched "500" from "pasta 500g" as SAR 500)
- Fix LBP FX rate zero-guard: use fallback when Yahoo returns 0 for ultra-low-value currencies
Validated locally: 9 countries seeded, Redis write confirmed, ~111s runtime
* fix(grocery-basket): validate extracted currency against expected country currency
- matchPrice now returns the currency code alongside the price
- extractPrice rejects results where currency != expected country currency
(prevents AED prices from being treated as JOD prices on gcc.luluhypermarket.com)
- Tighten item queries (white granulated sugar, spaghetti pasta, etc.) to reduce
irrelevant product matches like Stevia on sugar queries
- Replace Jordan's gcc.luluhypermarket.com (GCC-only) with carrefour.jo + ounasdelivery.com
- Sync scripts/shared/grocery-basket.json
* feat(bigmac): add Big Mac Index seed + drop grocery basket includeDomains
Grocery basket:
- Remove includeDomains restriction — EXA neural search finds better sources
than hardcoded domain lists; currency validation prevents contamination
- Tighten query strings (supermarket retail price suffix)
Big Mac seed (scripts/seed-bigmac.mjs):
- Two-tier search: specialist sites (theburgerindex.com, eatmyindex.com) first,
fall back to open search for countries without per-country indexed pages
- Handle thousands-separator prices (480,000 LBP)
- Accept USD prices from cost-of-living index sites as fallback
- Exclude ranking/average pages (Numbeo country_price_rankings, Expatistan)
- Validated live: 7/9 countries with confirmed prices
UAE=19AED, KSA=19SAR, QAR=22QAR, KWD=1.4KWD, EGP=135EGP, JOD=3JOD, LBP=480kLBP
* feat(economic): expand grocery basket to 24 global countries, drop Big Mac tier-2 search
Grocery basket: extend coverage from 9 MENA to 24 countries across all
regions (US, UK, DE, FR, JP, CN, IN, AU, CA, BR, MX, ZA, TR, NG, KR,
SG, PK, AE, SA, EG, KE, AR, ID, PH). Add FX fallbacks and fxSymbols
for all 23 new currencies. CCY regex in seed script updated to match
all supported currency codes.
Big Mac: remove tier-2 open search (too noisy, non-specialist pages
report combo prices or global averages). Specialist sites only
(theburgerindex.com, eatmyindex.com) for clean per-country data.
* feat(bigmac): wire Big Mac Index RPC, proto, bootstrap, health
Add ListBigMacPrices RPC end-to-end:
- proto/list_bigmac_prices.proto: BigMacCountryPrice + request/response
- service.proto: register ListBigMacPrices endpoint (GET /list-bigmac-prices)
- buf generate: regenerate service_server.ts + all client stubs
- server/list-bigmac-prices.ts: seed-only handler reads economic:bigmac:v1
- handler.ts: wire listBigMacPrices into EconomicServiceHandler
- api/bootstrap.js: bigmac key in BOOTSTRAP_CACHE_KEYS + SLOW_KEYS
- api/health.js: bigmac key in BOOTSTRAP_KEYS + SEED_META (maxStaleMin: 1440)
- _bootstrap-cache-key-refs.ts: groceryBasket + bigmac refs
* feat(bigmac): add BigMacPanel + register in panel layout
BigMacPanel renders a country-by-country Big Mac price table sorted by
USD price (cheapest/most expensive highlighted). Wired into bootstrap
hydration, refresh scheduler, and panel registry. Registered in
panels.ts (enabled: false, to be flipped once seed data is verified).
Also updates grocery basket i18n from ME-specific to global wording.
* fix(bigmac): register bigmac in cache-keys and RPC_CACHE_TIER
Add bigmac to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in
server/_shared/cache-keys.ts, and to RPC_CACHE_TIER (static tier)
in gateway.ts. Both were caught by bootstrap and RPC tier parity tests.
* fix(generated): restore @ts-nocheck in all generated service files after buf regenerate
buf generate does not emit @ts-nocheck. Previous convention restores it
manually post-generate to suppress strict type errors in generated code.
* fix(grocery-basket): restore includeDomains per country, add userLocation, fix currency symbol parsing
Root cause of 0-item countries (UK, JP, IN, NG): removing includeDomains
caused EXA neural search to return USD-priced global comparison pages
(Numbeo, Tridge, Expatistan) which currency validation correctly rejected.
Fixes:
- Add per-country sites[] in grocery-basket.json (researched local
supermarket/retailer domains: tesco.com, kaufland.de, bigbasket.com, etc.)
- Pass includeDomains: country.sites to restrict EXA to local retailers
- Pass userLocation: country.code (ISO) to bias results to target country
- Add currency symbol fallback regex (£→GBP, €→EUR, ¥→JPY, ₹→INR,
₩→KRW, ₦→NGN, R$→BRL) — sites like BigBasket use ₹ not INR
- Summary query now explicitly requests ISO currency code
- Simplify item queries (drop country name — context from domains)
Smoke test results:
UK sugar → GBP 1.09 (tesco.com) ✓
IN rice → ₹66 (bigbasket.com) ✓
JP rice → JPY 500 (kakaku.com) ✓
* fix(grocery-basket): add Firecrawl fallback, parallel items, bulk caps, currency floors
- Add Firecrawl as JS-SPA fallback after EXA (handles noon.com, coupang, daraz, tokopedia, lazada)
- Parallelize item fetching per country with 200ms stagger: runtime 38min to ~4.5min
- Add CURRENCY_MIN floors (NGN:50, IDR:500, KRW:1000, etc.) to reject product codes as prices
- Add ITEM_USD_MAX bulk caps (sugar:8USD, salt:5USD, rice:6USD, etc.) applied to both EXA and Firecrawl
- Fix SA: use noon.com/saudi-en + carrefour.com.sa (removes luluhypermarket cross-country pollution)
- Fix EG: use carrefouregypt.com + spinneys.com.eg + seoudi.com (removes GCC luluhypermarket)
- Expand sites for DE, MX, ZA, TR, NG, KR, IN, PK, AR, ID, PH to improve coverage
- Sync scripts/shared/grocery-basket.json with shared/grocery-basket.json
* fix(grocery-basket): address PR review comments P1+P2
P1 - fix ranking with incomplete data:
only include countries with >=70% item coverage (>=7/10) in
cheapest/mostExpensive ranking — prevents a country with 4/10
items appearing cheapest due to missing data
P1 - fix regex false-match on pack sizes / weights:
try currency-first pattern (SAR 8.99) before number-first to
avoid matching pack counts; use matchAll and take last match
P2 - mark seed-miss responses as non-cacheable:
add upstream_unavailable to proto + return true on empty seed
so gateway sets Cache-Control: no-store on cold deploy
* fix(generated): update EconomicService OpenAPI docs for upstream_unavailable field
* feat(market): add crypto sectors heatmap and token panels (DeFi, AI, Other) backend
- Add shared/crypto-sectors.json, defi-tokens.json, ai-tokens.json, other-tokens.json configs
- Add scripts/seed-crypto-sectors.mjs and seed-token-panels.mjs seed scripts
- Add proto messages for ListCryptoSectors, ListDefiTokens, ListAiTokens, ListOtherTokens
- Add change7d field (field 6) to CryptoQuote proto message
- Run buf generate to produce updated TypeScript bindings
- Add server handlers for all 4 new RPCs reading from seeded Redis cache
- Wire handlers into marketHandler and register cache keys with BOOTSTRAP_TIERS=slow
- Wire seedCryptoSectors and seedTokenPanels into ais-relay.cjs seedAllMarketData loop
* feat(panels): add crypto sectors heatmap and token panels (DeFi, AI, Other)
- Add TokenData interface to src/types/index.ts
- Wire ListCryptoSectorsResponse/ListDefiTokensResponse/ListAiTokensResponse/ListOtherTokensResponse into market service with circuit breakers and hydration fallbacks
- Add CryptoHeatmapPanel, TokenListPanel, DefiTokensPanel, AiTokensPanel, OtherTokensPanel to MarketPanel.ts
- Register 4 new panels in panels.ts FINANCE_PANELS and cryptoDigital category
- Instantiate new panels in panel-layout.ts
- Load data in data-loader.ts loadMarkets() alongside existing crypto fetch
* fix(crypto-panels): resolve test failures and type errors post-review
- Add @ts-nocheck to regenerated market service_server/client (matches repo convention)
- Add 4 new RPC routes to RPC_CACHE_TIER in gateway.ts (route-cache-tier test)
- Sync scripts/shared/ with shared/ for new token/sector JSON configs
- Restore non-market generated files to origin/main state (avoid buf version diff)
* fix(crypto-panels): address code review findings (P1-P3)
- ais-relay seedTokenPanels: add empty-guard before Redis write to
prevent overwriting cached data when all IDs are unresolvable
- server _feeds.ts: sync 4 missing crypto feeds (Wu Blockchain, Messari,
NFT News, Stablecoin Policy) with client-side feeds.ts
- data-loader: expose panel refs outside try block so catch can call
showRetrying(); log error instead of swallowing silently
- MarketPanel: replace hardcoded English error strings with t() calls
(failedSectorData / failedCryptoData) to honour user locale
- seed-token-panels.mjs: remove unused getRedisCredentials import
- cache-keys.ts: one BOOTSTRAP_TIERS entry per line for consistency
* fix(crypto-panels): three correctness fixes for RSS proxy, refresh, and Redis write visibility
- api/_rss-allowed-domains.js: add 7 new crypto domains (decrypt.co,
blockworks.co, thedefiant.io, bitcoinmagazine.com, www.dlnews.com,
cryptoslate.com, unchainedcrypto.com) so rss-proxy.js accepts the
new finance feeds instead of rejecting them as disallowed hosts
- src/App.ts: add crypto-heatmap/defi-tokens/ai-tokens/other-tokens to
the periodic markets refresh viewport condition so panels on screen
continue receiving live updates, not just the initial load
- ais-relay seedTokenPanels: capture upstashSet return values and log
PARTIAL if any Redis write fails, matching seedCryptoSectors pattern
* feat(config): add NSE and BSE (India) market support (#1102)
* fix(india-markets): wire NSE/BSE symbols into stocks.json so seed fetches them
- Add 20 India symbols (^NSEI, ^BSESN, 18x .NS equities) to shared/stocks.json
- Mark all .NS symbols + indices as yahooOnly (Finnhub does not support NSE)
- Remove orphan src/config/india-markets.ts; stocks.json is the seed source of truth
* fix(india-markets): sync scripts/shared/stocks.json mirror
* fix(ci): exclude scripts/data/ and scripts/node_modules/ from unicode safety scan
---------
Co-authored-by: lspassos1 <lspassos@icloud.com>
* feat(advisories): gold standard migration for security advisories
Move security advisories from client-side RSS fetching (24 feeds per
page load) to Railway cron seed with Redis-read-only Vercel handler.
- Add seed script fetching via relay RSS proxy with domain allowlist
- Add ListSecurityAdvisories proto, handler, and RPC cache tier
- Add bootstrap hydration key for instant page load
- Rewrite client service: bootstrap -> RPC fallback, no browser RSS
- Wire health.js, seed-health.js, and dataSize tracking
* fix(advisories): empty RPC returns ok:true, use full country map
P1 fixes from Codex review:
- Return ok:true for empty-but-successful RPC responses so the panel
clears to empty instead of stuck loading on cold environments
- Replace 50-entry hardcoded country map with 251-entry shared config
generated from the project GeoJSON + aliases, matching coverage of
the old client-side nameToCountryCode matcher
* fix(advisories): add Cote d'Ivoire and other missing country aliases
Adds 14 missing aliases including "cote d ivoire" (US State Dept
title format), common article-prefixed names (the Bahamas, the
Gambia), and alternative official names (Czechia, Eswatini, Cabo
Verde, Timor-Leste).
* fix(proto): inject @ts-nocheck via Makefile generate target
buf generate does not emit @ts-nocheck, but tsc strict mode rejects
the generated code. Adding a post-generation sed step in the Makefile
ensures both CI proto-freshness (make generate + diff) and CI
typecheck (tsc --noEmit) pass consistently.
PR #1596 removed the feeds but left the domains in the allowlist.
The relay still accepted proxy requests for these 403-blocked domains
from clients with cached old bundles. Removed:
- breakingdefense.com (403)
- www.arabnews.com (403)
- www.aei.org (403)
- mymodernmet.com (403)
Updated all 3 copies: shared/, scripts/shared/, api/
* fix(acled): add OAuth token manager with automatic refresh
ACLED access tokens expire every 24 hours, but WorldMonitor stores a
static ACLED_ACCESS_TOKEN with no refresh logic — causing all ACLED
API calls to fail after the first day.
This commit adds `acled-auth.ts`, an OAuth token manager that:
- Exchanges ACLED_EMAIL + ACLED_PASSWORD for an access token (24h)
and refresh token (14d) via the official ACLED OAuth endpoint
- Caches tokens in memory and auto-refreshes before expiry
- Falls back to static ACLED_ACCESS_TOKEN for backward compatibility
- Deduplicates concurrent refresh attempts
- Degrades gracefully when no credentials are configured
The only change to the existing `acled.ts` is replacing the synchronous
`process.env.ACLED_ACCESS_TOKEN` read with an async call to the new
`getAcledAccessToken()` helper.
Fixes#1283
Relates to #290
* fix: address review feedback on ACLED OAuth PR
- Use Redis (Upstash) as L2 token cache to survive Vercel Edge cold starts
(in-memory cache retained as fast-path L1)
- Add CHROME_UA User-Agent header on OAuth token exchange and refresh
- Update seed script to use OAuth flow via getAcledToken() helper
instead of raw process.env.ACLED_ACCESS_TOKEN
- Add security comment to .env.example about plaintext password trade-offs
- Sidecar ACLED_ACCESS_TOKEN case is a validation probe (tests user-provided
value, not process.env) — data fetching delegates to handler modules
* feat(sidecar): add ACLED_EMAIL/ACLED_PASSWORD to env allowlist and validation
- Add ACLED_EMAIL and ACLED_PASSWORD to ALLOWED_ENV_KEYS set
- Add ACLED_EMAIL validation case (store-only, verified with password)
- Add ACLED_PASSWORD validation case with OAuth token exchange via
acleddata.com/api/acled/user/login
- On successful login, store obtained OAuth token in ACLED_ACCESS_TOKEN
- Follows existing validation patterns (Cloudflare challenge handling,
auth failure detection, User-Agent header)
* fix: address remaining review feedback (duplicate OAuth, em dashes, emoji)
- Extract shared ACLED OAuth helper into scripts/shared/acled-oauth.mjs
- Remove ~55 lines of duplicate OAuth logic from seed-unrest-events.mjs,
now imports getAcledToken from the shared helper
- Replace em dashes with ASCII dashes in acled-auth.ts section comments
- Replace em dash with parentheses in sidecar validation message
- Remove emoji from .env.example security note
Addresses koala73's second review: MEDIUM (duplicate OAuth), LOW (em
dashes), LOW (emoji).
* fix: align sidecar OAuth endpoint, fix L1/L2 cache, cleanup artifacts
- Sidecar: switch from /api/acled/user/login (JSON) to /oauth/token
(URL-encoded) to match server/_shared/acled-auth.ts exactly
- acled-auth.ts: check L2 Redis when L1 is expired, not only when L1
is null (fixes stale L1 skipping fresher L2 from another isolate)
- acled-oauth.mjs: remove stray backslash on line 9
- seed-unrest-events.mjs: remove extra blank line at line 13
---------
Co-authored-by: Elie Habib <elie.habib@gmail.com>
Co-authored-by: RepairYourTech <30200484+RepairYourTech@users.noreply.github.com>
Railway rootDirectory isolates build context — postinstall cp from
../shared/ fails because parent dirs aren't in the Nixpacks image.
Commit JSON/CJS configs directly into scripts/shared/.
- Remove useless postinstall from scripts/package.json
- Remove scripts/shared/ from .gitignore
- Commit all shared config files into scripts/shared/
- Add sync test to catch drift between shared/ and scripts/shared/
