mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
main
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
9b07fc8d8a |
feat(yahoo): _yahoo-fetch helper with curl-only Decodo proxy fallback + 4 seeder migrations (#3120)
* feat(_yahoo-fetch): curl-only Decodo proxy fallback helper Yahoo Finance throttles Railway egress IPs aggressively. 4 seeders (seed-commodity-quotes, seed-etf-flows, seed-gulf-quotes, seed-market-quotes) duplicated the same fetchYahooWithRetry block with no proxy fallback. This helper consolidates them and adds the proxy fallback. Yahoo-specific: CURL-ONLY proxy strategy. Probed 2026-04-16: query1.finance.yahoo.com via CONNECT (httpsProxyFetchRaw): HTTP 404 query1.finance.yahoo.com via curl (curlFetch): HTTP 200 Yahoo's edge blocks Decodo's CONNECT egress IPs but accepts the curl egress IPs. Helper deliberately omits the CONNECT leg — adding it would burn time on guaranteed-404 attempts. Production defaults expose ONLY curlProxyResolver + curlFetcher. All learnings from PR #3118 + #3119 reviews baked in: - lastDirectError accumulator across the loop, embedded in final throw + Error.cause chain - catch block uses break (NOT throw) so thrown errors also reach proxy - DI seams (_curlProxyResolver, _proxyCurlFetcher) for hermetic tests - _PROXY_DEFAULTS exported for production-default lock tests - Sync curlFetch wrapped with await Promise.resolve() to future-proof against an async refactor (Greptile P2 from #3119) Tests (tests/yahoo-fetch.test.mjs, 11 cases): - Production defaults: curl resolver/fetcher reference equality - Production defaults: NO CONNECT leg present (regression guard) - 200 OK passthrough, never touches proxy - 429 with no proxy → throws exhausted with HTTP 429 in message - Retry-After header parsed correctly - 429 + curl proxy succeeds → returns proxy data - Thrown fetch error on final retry → proxy fallback runs (P1 guard) - 429 + proxy ALSO fails → both errors visible in message + cause chain - Proxy malformed JSON → throws exhausted - Non-retryable 500 → no extra direct retry, falls to proxy - parseRetryAfterMs unit (exported sanity check) Verification: 11/11 helper tests pass. node --check clean. Phase 1 of 2 — seeder migrations follow. * feat(yahoo-seeders): migrate 4 seeders to _yahoo-fetch helper Removes the duplicated fetchYahooWithRetry function (4 byte-identical copies across seed-commodity-quotes, seed-etf-flows, seed-gulf-quotes, seed-market-quotes) and routes all Yahoo Finance fetches through the new scripts/_yahoo-fetch.mjs helper. Each seeder gains the curl-only Decodo proxy fallback baked into the helper. Per-seeder changes (mechanical): - import { fetchYahooJson } from './_yahoo-fetch.mjs' - delete the local fetchYahooWithRetry function - replace 'const resp = await fetchYahooWithRetry(url, label); if (!resp) return X; const json = await resp.json()' with 'let json; try { json = await fetchYahooJson(url, { label }); } catch { return X; }' - prune now-unused CHROME_UA/sleep imports where applicable Latent bugs fixed in passing: - seed-etf-flows.mjs:23 and seed-market-quotes.mjs:38 referenced CHROME_UA without importing it (would throw ReferenceError at runtime if the helper were called). Now the call site is gone in etf-flows; in market-quotes CHROME_UA is properly imported because Finnhub call still uses it. seed-commodity-quotes also has fetchYahooChart1y (separate non-retry function for gold history). Migrated to use fetchYahooJson under the hood — preserves return shape, adds proxy fallback automatically. Verification: - node --check clean on all 4 modified seeders - npm run typecheck:all clean - npm run test:data: 5374/5374 pass Phase 2 of 2. * fix(_yahoo-fetch): log success AFTER parse + add _sleep DI seam for honest Retry-After test Greptile P2: "[YAHOO] proxy (curl) succeeded" was logged BEFORE JSON.parse(text). On malformed proxy JSON, Railway logs would show: [YAHOO] proxy (curl) succeeded for AAPL throw: Yahoo retries exhausted ... Contradictory + breaks the post-deploy log-grep verification this PR relies on ("look for [YAHOO] proxy (curl) succeeded"). Fix: parse first; success log only fires when parse succeeds AND the value is about to be returned. Greptile P3: 'Retry-After header parsed correctly' test used header value '0', but parseRetryAfterMs() treats non-positive seconds as null → helper falls through to default linear backoff. So the test was exercising the wrong branch despite its name. Fix: added _sleep DI opt seam to the helper. New test injects a sleep spy and asserts the captured duration: Retry-After: '7' → captured sleep == [7000] (Retry-After branch) no Retry-After → captured sleep == [10] (default backoff = retryBaseMs * 1) Two paired tests lock both branches separately so a future regression that collapses them is caught. Also added a log-ordering regression test: malformed proxy JSON must NOT emit the 'succeeded' log. Captures console.log into an array and asserts no 'proxy (curl) succeeded' line appeared before the throw. Verification: - tests/yahoo-fetch.test.mjs: 13/13 (was 11, +2) - npm run test:data: 5376/5376 (+2) - npm run typecheck:all: clean Followup commits on PR #3120. |
||
|
Elie Habib
|
044598346e |
feat(seed-contract): PR 2a — runSeed envelope dual-write + 91 seeders migrated (#3097)
* feat(seed-contract): PR 2a — runSeed envelope dual-write + 91 seeders migrated
Opt-in contract path in runSeed: when opts.declareRecords is provided, write
{_seed, data} envelope to the canonical key alongside legacy seed-meta:*
(dual-write). State machine: OK / OK_ZERO / RETRY with zeroIsValid opt.
declareRecords throws or returns non-integer → hard fail (contract violation).
extraKeys[*] support per-key declareRecords; each extra key writes its own
envelope. Legacy seeders (no declareRecords) entirely unchanged.
Migrated all 91 scripts/seed-*.mjs to contract mode. Each exports
declareRecords returning the canonical record count, and passes
schemaVersion: 1 + maxStaleMin (matched to api/health.js SEED_META, or 2.5x
interval where no registry entry exists). Contract conformance reports 84/86
seeders with full descriptor (2 pre-existing warnings).
Legacy seed-meta keys still written so unmigrated readers keep working;
follow-up slices flip health.js + readers to envelope-first.
Tests: 61/61 PR 1 tests still pass.
Next slices for PR 2:
- api/health.js registry collapse + 15 seed-bundle-*.mjs canonicalKey wiring
- reader migration (mcp, resilience, aviation, displacement, regional-snapshot)
- direct writers — ais-relay.cjs, consumer-prices-core publish.ts
- public-boundary stripSeedEnvelope + test migration
Plan: docs/plans/2026-04-14-002-fix-runseed-zero-record-lockout-plan.md
* fix(seed-contract): unwrap envelopes in internal cross-seed readers
After PR 2a enveloped 91 canonical keys as {_seed, data}, every script-side
reader that returned the raw parsed JSON started silently handing callers the
envelope instead of the bare payload. WoW baselines (bigmac, grocery-basket,
fear-greed) saw undefined .countries / .composite; seed-climate-anomalies saw
undefined .normals from climate:zone-normals:v1; seed-thermal-escalation saw
undefined .fireDetections from wildfire:fires:v1; seed-forecasts' ~40-key
pipeline batch returned envelopes for every input.
Fix: route every script-side reader through unwrapEnvelope(...).data. Legacy
bare-shape values pass through unchanged (unwrapEnvelope returns
{_seed: null, data: raw} for any non-envelope shape).
Changed:
- scripts/_seed-utils.mjs: import unwrapEnvelope; redisGet, readSeedSnapshot,
verifySeedKey all unwrap. Exported new readCanonicalValue() helper for
cross-seed consumers.
- 18 seed-*.mjs scripts with local redisGet-style helpers or inline fetch
patched to unwrap via the envelope source module (subagent sweep).
- scripts/seed-forecasts.mjs pipeline batch: parse() unwraps each result.
- scripts/seed-energy-spine.mjs redisMget: unwraps each result.
Tests:
- tests/seed-utils-envelope-reads.test.mjs: 7 new cases covering envelope
+ legacy + null paths for readSeedSnapshot and verifySeedKey.
- Full seed suite: 67/67 pass (was 61, +6 new).
Addresses both of user's P1 findings on PR #3097.
* feat(seed-contract): envelope-aware reads in server + api helpers
Every RPC and public-boundary reader now automatically strips _seed from
contract-mode canonical keys. Legacy bare-shape values pass through unchanged
(unwrapEnvelope no-ops on non-envelope shapes).
Changed helpers (one-place fix — unblocks ~60 call sites):
- server/_shared/redis.ts: getRawJson, getCachedJson, getCachedJsonBatch
unwrap by default. cachedFetchJson inherits via getCachedJson.
- api/_upstash-json.js: readJsonFromUpstash unwraps (covers api/mcp.ts
tool responses + all its canonical-key reads).
- api/bootstrap.js: getCachedJsonBatch unwraps (public-boundary —
clients never see envelope metadata).
Left intentionally unchanged:
- api/health.js / api/seed-health.js: read only seed-meta:* keys which
remain bare-shape during dual-write. unwrapEnvelope already imported at
the meta-read boundary (PR 1) as a defensive no-op.
Tests: 67/67 seed tests pass. typecheck + typecheck:api clean.
This is the blast-radius fix the PR #3097 review called out — external
readers that would otherwise see {_seed, data} after the writer side
migrated.
* fix(test): strip export keyword in vm.runInContext'd seed source
cross-source-signals-regulatory.test.mjs loads scripts/seed-cross-source-signals.mjs
via vm.runInContext, which cannot parse ESM `export` syntax. PR 2a added
`export function declareRecords` to every seeder, which broke this test's
static-analysis approach.
Fix: strip the `export` keyword from the declareRecords line in the
preprocessed source string so the function body still evaluates as a plain
declaration.
Full test:data suite: 5307/5307 pass. typecheck + typecheck:api clean.
* feat(seed-contract): consumer-prices publish.ts writes envelopes
Wrap the 5 canonical keys written by consumer-prices-core/src/jobs/publish.ts
(overview, movers:7d/30d, freshness, categories:7d/30d/90d, retailer-spread,
basket-series) in {_seed, data} envelopes. Legacy seed-meta:<key> writes
preserved for dual-write.
Inlined a buildEnvelope helper (10 lines) rather than taking a cross-package
dependency — consumer-prices-core is a standalone npm package. Documented the
four-file parity contract (mjs source, ts mirror, js edge mirror, this copy).
Contract fields: sourceVersion='consumer-prices-core-publish-v1', schemaVersion=1,
state='OK' (recordCount>0) or 'OK_ZERO' (legitimate zero).
Typecheck: no new errors in publish.ts.
* fix(seed-contract): 3 more server-side readers unwrap envelopes
Found during final audit:
- server/worldmonitor/resilience/v1/_shared.ts: resilience score reader
parsed cached GetResilienceScoreResponse raw. Contract-mode seed-resilience-scores
now envelopes those keys.
- server/worldmonitor/resilience/v1/get-resilience-ranking.ts: p05/p95
interval lookup parsed raw from seed-resilience-scores' extra-key path.
- server/worldmonitor/infrastructure/v1/_shared.ts: mgetJson() used for
count-source keys (wildfire:fires:v1, news:insights:v1) which are both
contract-mode now.
All three now unwrap via server/_shared/seed-envelope. Legacy shapes pass
through unchanged.
Typecheck clean.
* feat(seed-contract): ais-relay.cjs direct writes produce envelopes
32 canonical-key write sites in scripts/ais-relay.cjs now produce {_seed, data}
envelopes. Inlined buildEnvelope() (CJS module can't require ESM source) +
envelopeWrite(key, data, ttlSeconds, meta) wrapper. Enveloped keys span market
bootstrap, aviation, cyber-threats, theater-posture, weather-alerts, economic
spending/fred/worldbank, tech-events, corridor-risk, usni-fleet, shipping-stress,
social:reddit, wsb-tickers, pizzint, product-catalog, chokepoint transits,
ucdp-events, satellites, oref.
Left bare (not seeded data keys): seed-meta:* (dual-write legacy),
classifyCacheKey LLM cache, notam:prev-closed-state internal state,
wm:notif:scan-dedup flags.
Updated tests/ucdp-seed-resilience.test.mjs regex to accept both upstashSet
(pre-contract) and envelopeWrite (post-contract) call patterns.
* feat(seed-contract): 15 bundle files add canonicalKey for envelope gate
54 bundle sections across 12 files now declare canonicalKey alongside the
existing seedMetaKey. _bundle-runner.mjs (from PR 1) prefers canonicalKey
when both are present — gates section runs on envelope._seed.fetchedAt
read directly from the data key, eliminating the meta-outlives-data class
of bugs.
Files touched:
- climate (5), derived-signals (2), ecb-eu (3), energy-sources (6),
health (2), imf-extended (4), macro (10), market-backup (9),
portwatch (4), relay-backup (2), resilience-recovery (5), static-ref (2)
Skipped (14 sections, 3 whole bundles): multi-key writers, dynamic
templated keys (displacement year-scoped), or non-runSeed orchestrators
(regional brief cron, resilience-scores' 222-country publish, validation/
benchmark scripts). These continue to use seedMetaKey or their own gate.
seedMetaKey preserved everywhere — dual-write. _bundle-runner.mjs falls
back to legacy when canonicalKey is absent.
All 15 bundles pass node --check. test:data: 5307/5307. typecheck:all: clean.
* fix(seed-contract): 4 PR #3097 review P1s — transform/declareRecords mismatches + envelope leaks
Addresses both P1 findings and the extra-key seed-meta leak surfaced in review:
1. runSeed helper-level invariant: seed-meta:* keys NEVER envelope.
scripts/_seed-utils.mjs exports shouldEnvelopeKey(key) — returns false for
any key starting with 'seed-meta:'. Both atomicPublish (canonical) and
writeExtraKey (extras) gate the envelope wrap through this helper. Fixes
seed-iea-oil-stocks' ANALYSIS_META_EXTRA_KEY silently getting enveloped,
which broke health.js parsing the value as bare {fetchedAt, recordCount}.
Also defends against any future manual writeExtraKey(..., envelopeMeta)
call that happens to target a seed-meta:* key.
2. seed-token-panels canonical + extras fixed.
publishTransform returns data.defi (the defi panel itself, shape {tokens}).
Old declareRecords counted data.defi.tokens + data.ai.tokens + data.other.tokens
on the transformed payload → 0 → RETRY path → canonical market:defi-tokens:v1
never wrote, and because runSeed returned before the extraKeys loop,
market:ai-tokens:v1 + market:other-tokens:v1 stayed stale too.
New: declareRecords counts data.tokens on the transformed shape. AI_KEY +
OTHER_KEY extras reuse the same function (transforms return structurally
identical panels). Added isMain guard so test imports don't fire runSeed.
3. api/product-catalog.js cached reader unwraps envelope.
ais-relay.cjs now envelopes product-catalog:v2 via envelopeWrite(). The
edge reader did raw JSON.parse(result) and returned {_seed, data} to
clients, breaking the cached path. Fix: import unwrapEnvelope from
./_seed-envelope.js, apply after JSON.parse. One site — :238-241 is
downstream of getFromCache(), so the single reader fix covers both.
4. Regression lock tests/seed-contract-transform-regressions.test.mjs (11 cases):
- shouldEnvelopeKey invariant: seed-meta:* false, canonical true
- Token-panels declareRecords works on transformed shape (canonical + both extras)
- Explicit repro of pre-fix buggy signature returning 0 — guards against revert
- resolveRecordCount accepts 0, rejects non-integer
- Product-catalog envelope unwrap returns bare shape; legacy passes through
Verification:
- npm run test:data → 5318/5318 pass (was 5307 — 11 new regressions)
- npm run typecheck:all → clean
- node --check on every modified script
iea-oil-stocks canonical declareRecords was NOT broken (user confirmed during
review — buildIndex preserves .members); only its ANALYSIS_META_EXTRA_KEY
was affected, now covered generically by commit 1's helper invariant.
* fix(seed-contract): seed-token-panels validateFn also runs on post-transform shape
Review finding: fixing declareRecords wasn't sufficient — atomicPublish() runs
validateFn(publishData) on the transformed payload too. seed-token-panels'
validate() checked data.defi/.ai/.other on the transformed {tokens} shape,
returned false, and runSeed took the early skipped-write branch (before even
reaching the declareRecords RETRY logic). Net effect: same as before the
declareRecords fix — canonical + both extras stayed stale.
Fix: validate() now checks the canonical defi panel directly (Array.isArray
(data?.tokens) && has at least one t.price > 0). AI/OTHER panels are validated
implicitly by their own extraKey declareRecords on write.
Audited the other 9 seeders with publishTransform (bls-series, bis-extended,
bis-data, gdelt-intel, trade-flows, iea-oil-stocks, jodi-gas, sanctions-pressure,
forecasts): all validateFn's correctly target the post-transform shape. Only
token-panels regressed.
Added 4 regression tests (tests/seed-contract-transform-regressions.test.mjs):
- validate accepts transformed panel with priced tokens
- validate rejects all-zero-price tokens
- validate rejects empty/missing tokens
- Explicit pre-fix repro (buggy old signature fails on transformed shape)
Verification:
- npm run test:data → 5322/5322 pass (was 5318; +4 new)
- npm run typecheck:all → clean
- node --check clean
* feat(seed-contract): add /api/seed-contract-probe validation endpoint
Single machine-readable gate for 'is PR #3097 working in production'.
Replaces the curl/jq ritual with one authenticated edge call that returns
HTTP 200 ok:true or 503 + failing check list.
What it validates:
- 8 canonical keys have {_seed, data} envelopes with required data fields
and minRecords floors (fsi-eu, zone-normals, 3 token panels + minRecords
guard against token-panels RETRY regression, product-catalog, wildfire,
earthquakes).
- 2 seed-meta:* keys remain BARE (shouldEnvelopeKey invariant; guards
against iea-oil-stocks ANALYSIS_META_EXTRA_KEY-class regressions).
- /api/product-catalog + /api/bootstrap responses contain no '_seed' leak.
Auth: x-probe-secret header must match RELAY_SHARED_SECRET (reuses existing
Vercel↔Railway internal trust boundary).
Probe logic is exported (checkProbe, checkPublicBoundary, DEFAULT_PROBES) for
hermetic testing. tests/seed-contract-probe.test.mjs covers every branch:
envelope pass/fail on field/records/shape, bare pass/fail on shape/field,
missing/malformed JSON, Redis non-2xx, boundary seed-leak detection,
DEFAULT_PROBES sanity (seed-meta invariant present, token-panels minRecords
guard present).
Usage:
curl -H "x-probe-secret: $RELAY_SHARED_SECRET" \
https://api.worldmonitor.app/api/seed-contract-probe
PR 3 will extend the probe with a stricter mode that asserts seed-meta:*
keys are GONE (not just bare) once legacy dual-write is removed.
Verification:
- tests/seed-contract-probe.test.mjs → 15/15 pass
- npm run test:data → 5338/5338 (was 5322; +16 new incl. conformance)
- npm run typecheck:all → clean
* fix(seed-contract): tighten probe — minRecords on AI/OTHER + cache-path source header
Review P2 findings: the probe's stated guards were weaker than advertised.
1. market:ai-tokens:v1 + market:other-tokens:v1 probes claimed to guard the
token-panels extra-key RETRY regression but only checked shape='envelope'
+ dataHas:['tokens']. If an extra-key declareRecords regressed to 0, both
probes would still pass because checkProbe() only inspects _seed.recordCount
when minRecords is set. Now both enforce minRecords: 1.
2. /api/product-catalog boundary check only asserted no '_seed' leak — which
is also true for the static fallback path. A broken cached reader
(getFromCache returning null or throwing) could serve fallback silently
and still pass this probe. Now:
- api/product-catalog.js emits X-Product-Catalog-Source: cache|dodo|fallback
on the response (the json() helper gained an optional source param wired
to each of the three branches).
- checkPublicBoundary declaratively requires that header's value match
'cache' for /api/product-catalog, so a fallback-serve fails the probe
with reason 'source:fallback!=cache' or 'source:missing!=cache'.
Test updates (tests/seed-contract-probe.test.mjs):
- Boundary check reworked to use a BOUNDARY_CHECKS config with optional
requireSourceHeader per endpoint.
- New cases: served-from-cache passes, served-from-fallback fails with source
mismatch, missing header fails, seed-leak still takes precedence, bad
status fails.
- Token-panels sanity test now asserts minRecords≥1 on all 3 panels.
Verification:
- tests/seed-contract-probe.test.mjs → 17/17 pass (was 15, +2 net)
- npm run test:data → 5340/5340
- npm run typecheck:all → clean
|
||
|
Elie Habib
|
1da80c002d |
feat(market): add Alpha Vantage as primary market data source (#2055) (#2097)
* feat(market): add Alpha Vantage as primary market data source Add fetchAlphaVantageQuotesBatch() and fetchAlphaVantagePhysicalCommodity() to server/worldmonitor/market/v1/_shared.ts. Update three Railway seed scripts to try ALPHA_VANTAGE_API_KEY first: - seed-market-quotes.mjs: AV REALTIME_BULK_QUOTES (up to 100 symbols/call) → Finnhub secondary → Yahoo fallback (Indian NSE + Yahoo-only symbols) - seed-commodity-quotes.mjs: AV physical functions (WTI/BRENT/NATURAL_GAS/ COPPER/ALUMINUM) + AV bulk for ETF proxies → Yahoo fallback for futures (GC=F, SI=F, PL=F, ^VIX etc.) not covered by AV physical endpoints - seed-etf-flows.mjs: AV REALTIME_BULK_QUOTES for all BTC-spot ETFs → Yahoo 5d chart fallback for uncovered tickers Yahoo Finance and Finnhub remain as fallbacks; Railway relay Yahoo proxy route is not yet removed (pending migration validation). Closes #2055 * fix(market): fix AV rate-limit detection, prevClose guard, and Yahoo sleep ordering - Detect AV `Information` rate-limit response in all four AV fetch paths (_shared.ts bulk + physical, seed-market-quotes, seed-commodity-quotes, seed-etf-flows) and break/return early instead of silently continuing - Add 500ms inter-batch delay in fetchAlphaVantageQuotesBatch (both _shared.ts and seed-market-quotes) to avoid bursting AV 150 req/min limit - Fix prevClose guard in _shared.ts: `prevClose && Number.isFinite(prevClose)` replaced with `Number.isFinite(prevClose) && prevClose > 0` for consistency with seed scripts - Fix Yahoo sleep-before-covered-check in seed-commodity-quotes and seed-etf-flows: delay now only fires for actual Yahoo requests, not for AV-already-covered symbols at i > 0 * fix(market): address AV code review findings - Extract shared _shared-av.mjs: fetchAvBulkQuotes + fetchAvPhysicalCommodity (was duplicated in 3 seed scripts and _shared.ts) - Add 1-retry with 1s backoff on all AV fetch calls (network transients) - Log count of dropped symbols when rate limit breaks a batch loop - Populate sparkline from daily data array in fetchAvPhysicalCommodity (last 7 closes, oldest→newest; was always []) - Fix ETF seed: avgVolume=0, volumeRatio=0 when AV covers ticker (REALTIME_BULK_QUOTES has no 5-day history; 1/volume was misleading) - Consolidate NSE filter in seed-market-quotes to use YAHOO_ONLY set (was three separate endsWith/startsWith checks) * fix(market): add GOLD and SILVER to AV physical commodity map GC=F (Gold) and SI=F (Silver) are supported AV physical commodity functions per AV docs and were explicitly listed in issue #2055. Both were missing from AV_PHYSICAL_MAP, falling back to Yahoo. PL=F, PA=F, RB=F, HO=F have no AV physical function — Yahoo fallback for those is correct. * feat(market): extend AV to gulf oil, gulf currencies, and FX rates seed-gulf-quotes.mjs: - Oil (CL=F, BZ=F): AV physical functions (WTI/BRENT) as primary - Currencies (SAR/AED/QAR/KWD/BHD/OMR): AV FX_DAILY as primary (gives daily close + change% + 7-point sparkline) - Indices (^TASI.SR, DFMGI.AE, etc.): Yahoo fallback (no AV equivalent) seed-fx-rates.mjs: - AV CURRENCY_EXCHANGE_RATE as primary for all ~50 currencies - 900ms between calls (67 req/min, safe under 75/min limit) - Yahoo fallback for any currencies AV does not cover _shared-av.mjs: - Add fetchAvCurrencyRate() — CURRENCY_EXCHANGE_RATE, single pair → rate - Add fetchAvFxDaily() — FX_DAILY, daily close + change% + sparkline * revert(market): keep seed-fx-rates on Yahoo (no AV bulk FX endpoint) AV CURRENCY_EXCHANGE_RATE is one-call-per-pair with no bulk equivalent, offers worse exotic-currency coverage than Yahoo (LBP, NGN, KES, VND), and a daily cron doesn't face Yahoo's IP rate-limit pressure. seed-gulf-quotes currencies (6 pairs, 10-min cron) keep AV FX_DAILY. |
||
|
Elie Habib
|
df29d59ff7 |
fix(health): enforce 1h+ TTL buffer across all seed jobs (#2072)
Full audit of seed TTL vs cron cadence. Rule: TTL >= cron_interval + 1h. CRITICAL (TTL = cron, 0 buffer): - seed-supply-chain-trade: tariffTrendsUs TRADE_TTL(6h) → TARIFF_TTL(8h) - seed-supply-chain-trade: customsRevenue TRADE_TTL(6h) → CUSTOMS_TTL(24h) - seed-sanctions-pressure: CACHE_TTL 12h → 15h (12h cron, 3h buffer) - seed-usa-spending: CACHE_TTL 1h → 2h (1h cron, 1h buffer) WARN (<1h buffer): - seed-security-advisories: TTL 2h → 3h (1h cron, now 2h buffer) - seed-token-panels: TTL 1h → 90min (30min cron, now 1h buffer) - seed-etf-flows: TTL 1h → 90min (15min cron, now 75min buffer) - seed-stablecoin-markets: TTL 1h → 90min (10min cron, now 80min buffer) - seed-gulf-quotes: TTL 1h → 90min (10min cron, now 80min buffer) - seed-crypto-quotes: TTL 1h → 2h (5min cron, now 115min buffer) - ais-relay CRYPTO_SEED_TTL: 1h → 2h - ais-relay STABLECOIN_SEED_TTL: 1h → 2h - ais-relay SECTORS_SEED_TTL: 1h → 2h |
||
|
Elie Habib
|
4008f56254 |
fix: log fetch error cause in seed retry/FATAL handlers (#1638)
* test: rewrite transit chart test as structural contract verification Replace fragile source-string extraction + new Function() compilation with structural pattern checks on the source code. Tests verify: - render() clears chart before content change - clearTransitChart() cancels timer, disconnects observer, destroys chart - MutationObserver setup for DOM readiness detection - Fallback timer for no-op renders (100-500ms range) - Both callbacks (observer + timer) clean up each other - Tab switch and collapse clear chart state - Mount function guards against missing element/data Replaces PR #1634's approach which was brittle (method body extraction, TypeScript cast stripping, sandboxed execution). * fix: log fetch error cause in seed retry and FATAL handlers Node 20 fetch() throws TypeError('fetch failed') with the real error hidden in err.cause (DNS, TLS, timeout). The current logging only shows 'fetch failed' which is useless for diagnosis. Now logs err.cause.message in both withRetry() retries and FATAL catch blocks. |
||
|
Elie Habib
|
364e497bd1 |
fix(scripts): resolve shared JSON configs for Railway rootDirectory (#1231)
Railway deploys seed services with rootDirectory=scripts/, placing files
at /app/ without the parent shared/ directory. The createRequire +
require('../shared/X.json') pattern resolves to /shared/ which doesn't
exist in the container.
- Add loadSharedConfig() to _seed-utils.mjs: tries ../shared/ (local)
then ./shared/ (Railway) with clear error on miss
- Add requireShared() to ais-relay.cjs with same dual-path fallback
- Add postinstall to scripts/package.json that copies ../shared/ into
./shared/ during Railway build
- Update all 6 seed scripts to use loadSharedConfig instead of
createRequire + require
- Add scripts/shared/ to .gitignore
Fixes crash introduced by #1212 (shared JSON consolidation).
|
||
|
Elie Habib
|
dd127447c0 |
refactor: consolidate duplicated market data lists into shared JSON configs (#1212)
Adding a new item (crypto, ETF, stablecoin, gulf symbol, etc.) previously required editing 2-4 files because the same list was hardcoded independently in seed scripts, RPC handlers, and frontend config. Following the proven shared/crypto.json pattern, extract 6 new shared JSON configs so each list has a single source of truth. New shared configs: - shared/stablecoins.json (ids + coinpaprika mappings) - shared/etfs.json (BTC spot ETF tickers + issuers) - shared/gulf.json (GCC indices, currencies, oil benchmarks) - shared/sectors.json (sector ETF symbols + names) - shared/commodities.json (VIX, gold, oil, gas, silver, copper) - shared/stocks.json (market symbols + yahoo-only set) All seed scripts, RPC handlers, and frontend config now import from these shared JSON files instead of maintaining independent copies. |
||
|
Elie Habib
|
2f2486cc8e |
fix: harden seed scripts with 429 rate-limit retry and relaxed validation (#1026)
* fix: allow zero fire detections in seed validation FIRMS NRT data has a rolling window — at certain hours, all 9 monitored regions can legitimately return 0 active fire detections. The strict length > 0 validation caused CRASHED status on Railway cron runs during these periods. Structure-only validation is sufficient. * fix: add rate-limit-aware retry for CoinGecko 429s The default withRetry (1s/2s/4s backoff) is too short for CoinGecko rate limits. New fetchWithRateLimitRetry uses 10s/20s/30s/40s/50s delays with up to 5 attempts specifically for 429 responses. * fix: add 429 rate-limit retry to all Yahoo and CoinGecko seed scripts Yahoo Finance and CoinGecko both return 429 when rate limited. The default withRetry (1s/2s/4s) is too fast for rate limits. Added per-request 429-specific retry with longer backoff: - Yahoo: 5s/10s/15s/20s (4 attempts per symbol) - CoinGecko: 10s/20s/30s/40s/50s (5 attempts) Scripts updated: seed-etf-flows, seed-gulf-quotes, seed-commodity-quotes, seed-market-quotes, seed-stablecoin-markets. |
||
|
Elie Habib
|
78a14306d9 |
feat: add seed-first pattern to 15 RPC handlers with Railway seed scripts (#989)
Migrate handlers from direct external API calls to seed-first pattern: Railway cron seeds Redis → handlers read from Redis → fallback to live fetch if seed stale and SEED_FALLBACK_* env enabled. Handlers updated: earthquakes, fire-detections, internet-outages, climate-anomalies, unrest-events, cyber-threats, market-quotes, commodity-quotes, crypto-quotes, etf-flows, gulf-quotes, stablecoin-markets, natural-events, displacement-summary, risk-scores. Also adds: - scripts/_seed-utils.mjs (shared seed framework with atomic publish, distributed locks, retry, freshness metadata) - 13 seed scripts for Railway cron - api/seed-health.js monitoring endpoint - scripts/validate-seed-migration.mjs post-deploy validation - Restored multi-source CII in get-risk-scores (8 sources: ACLED, UCDP, outages, climate, cyber, fires, GPS, Iran) |