Files
worldmonitor/tests/sector-valuations.test.mjs
Elie Habib 0075af5a47 fix(sector-valuations): proxy Yahoo quoteSummary via Decodo curl egress (#3134)
* fix(sector-valuations): proxy Yahoo quoteSummary via Decodo curl egress

Yahoo's /v10/finance/quoteSummary returns HTTP 401 from Railway container
IPs. Railway logs 2026-04-16 show all 12 sector ETFs failing every 5-min
cron:

  [Sector] Yahoo quoteSummary XLK HTTP 401  (x12 per tick)
  [Market] Seeded 12/12 sectors, 0 valuations

Add a curl-based proxy fallback that matches scripts/_yahoo-fetch.mjs:
hit us.decodo.com (curl egress pool) NOT gate.decodo.com (CONNECT egress
pool). Per the 2026-04-16 probe documented in _yahoo-fetch.mjs header,
Yahoo blocks Decodo's CONNECT egress IPs but accepts the curl egress.
Reusing ytFetchViaProxy here would keep failing silently.

Shares the existing _yahooProxyFailCount / _yahooProxyCooldownUntil
cooldown state with fetchYahooChartDirect so both Yahoo paths pause
together if Decodo's curl pool also gets blocked.

No change to direct-path behavior when Yahoo is healthy.

* fix(sector-valuations): don't proxy on empty quoteSummary result (review)

Direct 200 with data.quoteSummary.result[0] absent is an app-level "no
data for this symbol" signal (e.g. delisted ETF). Proxy won't return
different data for a symbol Yahoo itself doesn't carry — falling back
would burn the 5-failure cooldown budget on structurally empty symbols
and mask a genuine proxy outage.

Resolve null on !result; keep JSON.parse catch going to proxy (garbage
body IS a transport-level signal — captive portal, Cloudflare challenge).

Review feedback from PR #3134.

* fix(sector-valuations): split cooldown per egress route, cover transport failures (review)

Review feedback on PR #3134, both P1.

P1 #1 — transport failures bypassed cooldown
  execFileSync timeouts, proxy-connect refusals, and JSON.parse on garbage
  bodies all went through the catch block and returned null without
  ticking _yahooProxyFailCount. In the exact failure mode this PR hardens
  against, the relay would have thrashed through 12 × 20s curl attempts
  per tick with no backoff. Extract a bumpCooldown() helper and call it
  from both the non-2xx branch and the catch block.

P1 #2 — two Decodo egress pools shared one cooldown budget
  fetchYahooChartDirect uses CONNECT via gate.decodo.com.
  _yahooQuoteSummaryProxyFallback uses curl via us.decodo.com.
  These are independent egress IP pools — per the 2026-04-16 probe,
  Yahoo blocks CONNECT but accepts curl. Sharing cooldown means 5
  CONNECT failures suppress the healthy curl path (and vice versa).
  Split into _yahooConnectProxy* (chart) and _yahooCurlProxy* (sector
  valuations).

Also: on proxy 200 with empty result, reset the curl counter. The route
is healthy even if this specific symbol has no data — don't pretend it's
a failure.

* fix(sector-valuations): non-blocking curl + settle guard (review round 3)

Review feedback on PR #3134, both P1.

P1 #1 - double proxy invocation on timeout/error race
  req.destroy() inside the timeout handler can still emit 'error', and
  both handlers eagerly called resolve(_yahooQuoteSummaryProxyFallback(...)).
  A single upstream timeout therefore launched two curl subprocesses,
  double-ticked the cooldown counter, and blocked twice. Add a settled
  flag; settle() exits early on the second handler before evaluating the
  fallback.

P1 #2 - execFileSync blocks the relay event loop
  The relay serves HTTP/WS traffic on the same thread that awaits
  seedSectorSummary's per-symbol Yahoo fetch. execFileSync for up to 20s
  per failure x 5 failures before cooldown = ~100s of frozen event loop.
  Switch to promisify(execFile). resolve(promise) chains the Promise
  through fetchYahooQuoteSummary's outer Promise, so the main-loop await
  yields while curl runs. Other traffic continues during the fetch.

tests/sector-valuations.test.mjs: bump the static-analysis window from
1500 to 2000 chars so the field-extraction markers (ytdReturn etc.)
stay inside the window after the settle guard was added.
2026-04-16 20:02:31 +04:00

163 lines
5.2 KiB
JavaScript

import { describe, it } from 'node:test';
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
const src = readFileSync('scripts/ais-relay.cjs', 'utf8');
const extractFn = (name) => {
const start = src.indexOf(`function ${name}(`);
if (start === -1) throw new Error(`Function ${name} not found`);
let depth = 0;
let i = src.indexOf('{', start);
const bodyStart = i;
for (; i < src.length; i++) {
if (src[i] === '{') depth++;
if (src[i] === '}') depth--;
if (depth === 0) break;
}
return src.slice(bodyStart, i + 1);
};
// eslint-disable-next-line no-new-func
const parseSectorValuation = new Function(
'raw',
extractFn('parseSectorValuation')
.replace(/^{/, '')
.replace(/}$/, ''),
);
describe('parseSectorValuation', () => {
it('returns null for null input', () => {
assert.equal(parseSectorValuation(null), null);
});
it('returns null for undefined input', () => {
assert.equal(parseSectorValuation(undefined), null);
});
it('returns null when both PE values are missing', () => {
assert.equal(parseSectorValuation({ beta: 1.2 }), null);
});
it('parses numeric values correctly', () => {
const result = parseSectorValuation({
trailingPE: 25.3,
forwardPE: 22.1,
beta: 1.05,
ytdReturn: 0.08,
threeYearReturn: 0.12,
fiveYearReturn: 0.10,
});
assert.equal(result.trailingPE, 25.3);
assert.equal(result.forwardPE, 22.1);
assert.equal(result.beta, 1.05);
assert.equal(result.ytdReturn, 0.08);
assert.equal(result.threeYearReturn, 0.12);
assert.equal(result.fiveYearReturn, 0.10);
});
it('handles string values via typeof guard (PizzINT pattern)', () => {
const result = parseSectorValuation({
trailingPE: '18.5',
forwardPE: '16.2',
beta: '0.95',
ytdReturn: '0.05',
});
assert.equal(result.trailingPE, 18.5);
assert.equal(result.forwardPE, 16.2);
assert.equal(result.beta, 0.95);
assert.equal(result.ytdReturn, 0.05);
});
it('returns null for NaN/Infinity values', () => {
const result = parseSectorValuation({
trailingPE: NaN,
forwardPE: Infinity,
});
assert.equal(result, null);
});
it('allows partial data (trailingPE only)', () => {
const result = parseSectorValuation({
trailingPE: 20,
});
assert.equal(result.trailingPE, 20);
assert.equal(result.forwardPE, null);
assert.equal(result.beta, null);
assert.equal(result.ytdReturn, null);
});
it('allows partial data (forwardPE only)', () => {
const result = parseSectorValuation({
forwardPE: 15,
});
assert.equal(result.trailingPE, null);
assert.equal(result.forwardPE, 15);
});
});
describe('fetchYahooQuoteSummary (static analysis)', () => {
const fnStart = src.indexOf('function fetchYahooQuoteSummary(');
// Window sized to cover the direct-fetch block (headers, timeout, field
// extraction). Grown to 2000 when proxy-fallback wiring (settled guard,
// curl helper reference) was added — field extraction must stay visible.
const fnChunk = src.slice(fnStart, fnStart + 2000);
it('exists in ais-relay.cjs', () => {
assert.ok(fnStart > -1, 'fetchYahooQuoteSummary function not found');
});
it('uses summaryDetail and defaultKeyStatistics modules', () => {
assert.match(fnChunk, /summaryDetail/, 'should request summaryDetail module');
assert.match(fnChunk, /defaultKeyStatistics/, 'should request defaultKeyStatistics module');
});
it('uses v10/finance/quoteSummary endpoint', () => {
assert.match(fnChunk, /v10\/finance\/quoteSummary/, 'should call Yahoo quoteSummary v10 API');
});
it('extracts trailingPE, forwardPE, and beta', () => {
assert.match(fnChunk, /trailingPE/, 'should extract trailingPE');
assert.match(fnChunk, /forwardPE/, 'should extract forwardPE');
assert.match(fnChunk, /beta/, 'should extract beta');
});
it('extracts return metrics from defaultKeyStatistics', () => {
assert.match(fnChunk, /ytdReturn/, 'should extract ytdReturn');
});
it('includes User-Agent header', () => {
assert.match(fnChunk, /User-Agent/, 'should include User-Agent for Yahoo requests');
});
it('has timeout configured', () => {
assert.match(fnChunk, /timeout:\s*\d+/, 'should have a timeout set');
});
});
describe('seedSectorSummary valuation integration (static analysis)', () => {
const fnStart = src.indexOf('async function seedSectorSummary()');
const fnEnd = src.indexOf('\n// Gulf Quotes');
const fnBody = src.slice(fnStart, fnEnd);
it('calls fetchYahooQuoteSummary for each sector', () => {
assert.match(fnBody, /fetchYahooQuoteSummary\(s\)/, 'should call fetchYahooQuoteSummary per sector');
});
it('calls parseSectorValuation on raw response', () => {
assert.match(fnBody, /parseSectorValuation\(raw\)/, 'should parse raw valuation data');
});
it('includes valuations in payload', () => {
assert.match(fnBody, /valuations/, 'payload should include valuations object');
});
it('sleeps between Yahoo requests (rate limit)', () => {
assert.match(fnBody, /await sleep\(150\)/, 'should sleep 150ms between Yahoo calls');
});
it('logs valuation count', () => {
assert.match(fnBody, /valCount/, 'should log how many valuations were fetched');
});
});