feat(yahoo): _yahoo-fetch helper with curl-only Decodo proxy fallback + 4 seeder migrations (#3120)

* feat(_yahoo-fetch): curl-only Decodo proxy fallback helper

Yahoo Finance throttles Railway egress IPs aggressively. 4 seeders
(seed-commodity-quotes, seed-etf-flows, seed-gulf-quotes, seed-market-quotes)
duplicated the same fetchYahooWithRetry block with no proxy fallback.
This helper consolidates them and adds the proxy fallback.

Yahoo-specific: CURL-ONLY proxy strategy. Probed 2026-04-16:
  query1.finance.yahoo.com via CONNECT (httpsProxyFetchRaw): HTTP 404
  query1.finance.yahoo.com via curl    (curlFetch):          HTTP 200
Yahoo's edge blocks Decodo's CONNECT egress IPs but accepts the curl
egress IPs. Helper deliberately omits the CONNECT leg — adding it
would burn time on guaranteed-404 attempts. Production defaults expose
ONLY curlProxyResolver + curlFetcher.

All learnings from PR #3118 + #3119 reviews baked in:
- lastDirectError accumulator across the loop, embedded in final throw +
  Error.cause chain
- catch block uses break (NOT throw) so thrown errors also reach proxy
- DI seams (_curlProxyResolver, _proxyCurlFetcher) for hermetic tests
- _PROXY_DEFAULTS exported for production-default lock tests
- Sync curlFetch wrapped with await Promise.resolve() to future-proof
  against an async refactor (Greptile P2 from #3119)

Tests (tests/yahoo-fetch.test.mjs, 11 cases):
- Production defaults: curl resolver/fetcher reference equality
- Production defaults: NO CONNECT leg present (regression guard)
- 200 OK passthrough, never touches proxy
- 429 with no proxy → throws exhausted with HTTP 429 in message
- Retry-After header parsed correctly
- 429 + curl proxy succeeds → returns proxy data
- Thrown fetch error on final retry → proxy fallback runs (P1 guard)
- 429 + proxy ALSO fails → both errors visible in message + cause chain
- Proxy malformed JSON → throws exhausted
- Non-retryable 500 → no extra direct retry, falls to proxy
- parseRetryAfterMs unit (exported sanity check)

Verification: 11/11 helper tests pass. node --check clean.

Phase 1 of 2 — seeder migrations follow.

* feat(yahoo-seeders): migrate 4 seeders to _yahoo-fetch helper

Removes the duplicated fetchYahooWithRetry function (4 byte-identical
copies across seed-commodity-quotes, seed-etf-flows, seed-gulf-quotes,
seed-market-quotes) and routes all Yahoo Finance fetches through the
new scripts/_yahoo-fetch.mjs helper. Each seeder gains the curl-only
Decodo proxy fallback baked into the helper.

Per-seeder changes (mechanical):
- import { fetchYahooJson } from './_yahoo-fetch.mjs'
- delete the local fetchYahooWithRetry function
- replace 'const resp = await fetchYahooWithRetry(url, label); if (!resp)
  return X; const json = await resp.json()' with
  'let json; try { json = await fetchYahooJson(url, { label }); }
  catch { return X; }'
- prune now-unused CHROME_UA/sleep imports where applicable

Latent bugs fixed in passing:
- seed-etf-flows.mjs:23 and seed-market-quotes.mjs:38 referenced
  CHROME_UA without importing it (would throw ReferenceError at
  runtime if the helper were called). Now the call site is gone in
  etf-flows; in market-quotes CHROME_UA is properly imported because
  Finnhub call still uses it.

seed-commodity-quotes also has fetchYahooChart1y (separate non-retry
function for gold history). Migrated to use fetchYahooJson under the
hood — preserves return shape, adds proxy fallback automatically.

Verification:
- node --check clean on all 4 modified seeders
- npm run typecheck:all clean
- npm run test:data: 5374/5374 pass

Phase 2 of 2.

* fix(_yahoo-fetch): log success AFTER parse + add _sleep DI seam for honest Retry-After test

Greptile P2: "[YAHOO] proxy (curl) succeeded" was logged BEFORE
JSON.parse(text). On malformed proxy JSON, Railway logs would show:

  [YAHOO] proxy (curl) succeeded for AAPL
  throw: Yahoo retries exhausted ...

Contradictory + breaks the post-deploy log-grep verification this PR
relies on ("look for [YAHOO] proxy (curl) succeeded"). Fix: parse
first; success log only fires when parse succeeds AND the value is
about to be returned.

Greptile P3: 'Retry-After header parsed correctly' test used header
value '0', but parseRetryAfterMs() treats non-positive seconds as null
→ helper falls through to default linear backoff. So the test was
exercising the wrong branch despite its name.

Fix: added _sleep DI opt seam to the helper. New test injects a sleep
spy and asserts the captured duration:

  Retry-After: '7' → captured sleep == [7000]   (Retry-After branch)
  no Retry-After  → captured sleep == [10]      (default backoff = retryBaseMs * 1)

Two paired tests lock both branches separately so a future regression
that collapses them is caught.

Also added a log-ordering regression test: malformed proxy JSON must
NOT emit the 'succeeded' log. Captures console.log into an array and
asserts no 'proxy (curl) succeeded' line appeared before the throw.

Verification:
- tests/yahoo-fetch.test.mjs: 13/13 (was 11, +2)
- npm run test:data: 5376/5376 (+2)
- npm run typecheck:all: clean

Followup commits on PR #3120.
This commit is contained in:
Elie Habib
2026-04-16 09:25:06 +04:00
committed by GitHub
parent 57414e4762
commit 9b07fc8d8a
6 changed files with 524 additions and 123 deletions

View File

@@ -1,6 +1,7 @@
#!/usr/bin/env node
import { loadEnvFile, loadSharedConfig, runSeed } from './_seed-utils.mjs';
import { fetchYahooJson } from './_yahoo-fetch.mjs';
import { fetchAvBulkQuotes } from './_shared-av.mjs';
const etfConfig = loadSharedConfig('etfs.json');
@@ -17,28 +18,6 @@ function sleep(ms) {
return new Promise((r) => setTimeout(r, ms));
}
async function fetchYahooWithRetry(url, label, maxAttempts = 4) {
for (let i = 0; i < maxAttempts; i++) {
const resp = await fetch(url, {
headers: { 'User-Agent': CHROME_UA },
signal: AbortSignal.timeout(10_000),
});
if (resp.status === 429) {
const wait = 5000 * (i + 1);
console.warn(` [Yahoo] ${label} 429 — waiting ${wait / 1000}s (attempt ${i + 1}/${maxAttempts})`);
await sleep(wait);
continue;
}
if (!resp.ok) {
console.warn(` [Yahoo] ${label} HTTP ${resp.status}`);
return null;
}
return resp;
}
console.warn(` [Yahoo] ${label} rate limited after ${maxAttempts} attempts`);
return null;
}
function parseEtfChartData(chart, ticker, issuer) {
const result = chart?.chart?.result?.[0];
if (!result) return null;
@@ -112,9 +91,13 @@ async function fetchEtfFlows() {
try {
const url = `https://query1.finance.yahoo.com/v8/finance/chart/${ticker}?range=5d&interval=1d`;
const resp = await fetchYahooWithRetry(url, ticker);
if (!resp) { misses++; continue; }
const chart = await resp.json();
let chart;
try {
chart = await fetchYahooJson(url, { label: ticker });
} catch {
misses++;
continue;
}
const parsed = parseEtfChartData(chart, ticker, issuer);
if (parsed) {
etfs.push(parsed);