mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
* fix(sector-valuations): proxy Yahoo quoteSummary via Decodo curl egress Yahoo's /v10/finance/quoteSummary returns HTTP 401 from Railway container IPs. Railway logs 2026-04-16 show all 12 sector ETFs failing every 5-min cron: [Sector] Yahoo quoteSummary XLK HTTP 401 (x12 per tick) [Market] Seeded 12/12 sectors, 0 valuations Add a curl-based proxy fallback that matches scripts/_yahoo-fetch.mjs: hit us.decodo.com (curl egress pool) NOT gate.decodo.com (CONNECT egress pool). Per the 2026-04-16 probe documented in _yahoo-fetch.mjs header, Yahoo blocks Decodo's CONNECT egress IPs but accepts the curl egress. Reusing ytFetchViaProxy here would keep failing silently. Shares the existing _yahooProxyFailCount / _yahooProxyCooldownUntil cooldown state with fetchYahooChartDirect so both Yahoo paths pause together if Decodo's curl pool also gets blocked. No change to direct-path behavior when Yahoo is healthy. * fix(sector-valuations): don't proxy on empty quoteSummary result (review) Direct 200 with data.quoteSummary.result[0] absent is an app-level "no data for this symbol" signal (e.g. delisted ETF). Proxy won't return different data for a symbol Yahoo itself doesn't carry — falling back would burn the 5-failure cooldown budget on structurally empty symbols and mask a genuine proxy outage. Resolve null on !result; keep JSON.parse catch going to proxy (garbage body IS a transport-level signal — captive portal, Cloudflare challenge). Review feedback from PR #3134. * fix(sector-valuations): split cooldown per egress route, cover transport failures (review) Review feedback on PR #3134, both P1. P1 #1 — transport failures bypassed cooldown execFileSync timeouts, proxy-connect refusals, and JSON.parse on garbage bodies all went through the catch block and returned null without ticking _yahooProxyFailCount. In the exact failure mode this PR hardens against, the relay would have thrashed through 12 × 20s curl attempts per tick with no backoff. Extract a bumpCooldown() helper and call it from both the non-2xx branch and the catch block. P1 #2 — two Decodo egress pools shared one cooldown budget fetchYahooChartDirect uses CONNECT via gate.decodo.com. _yahooQuoteSummaryProxyFallback uses curl via us.decodo.com. These are independent egress IP pools — per the 2026-04-16 probe, Yahoo blocks CONNECT but accepts curl. Sharing cooldown means 5 CONNECT failures suppress the healthy curl path (and vice versa). Split into _yahooConnectProxy* (chart) and _yahooCurlProxy* (sector valuations). Also: on proxy 200 with empty result, reset the curl counter. The route is healthy even if this specific symbol has no data — don't pretend it's a failure. * fix(sector-valuations): non-blocking curl + settle guard (review round 3) Review feedback on PR #3134, both P1. P1 #1 - double proxy invocation on timeout/error race req.destroy() inside the timeout handler can still emit 'error', and both handlers eagerly called resolve(_yahooQuoteSummaryProxyFallback(...)). A single upstream timeout therefore launched two curl subprocesses, double-ticked the cooldown counter, and blocked twice. Add a settled flag; settle() exits early on the second handler before evaluating the fallback. P1 #2 - execFileSync blocks the relay event loop The relay serves HTTP/WS traffic on the same thread that awaits seedSectorSummary's per-symbol Yahoo fetch. execFileSync for up to 20s per failure x 5 failures before cooldown = ~100s of frozen event loop. Switch to promisify(execFile). resolve(promise) chains the Promise through fetchYahooQuoteSummary's outer Promise, so the main-loop await yields while curl runs. Other traffic continues during the fetch. tests/sector-valuations.test.mjs: bump the static-analysis window from 1500 to 2000 chars so the field-extraction markers (ytdReturn etc.) stay inside the window after the settle guard was added.
163 lines
5.2 KiB
JavaScript
163 lines
5.2 KiB
JavaScript
import { describe, it } from 'node:test';
|
|
import assert from 'node:assert/strict';
|
|
import { readFileSync } from 'node:fs';
|
|
|
|
const src = readFileSync('scripts/ais-relay.cjs', 'utf8');
|
|
|
|
const extractFn = (name) => {
|
|
const start = src.indexOf(`function ${name}(`);
|
|
if (start === -1) throw new Error(`Function ${name} not found`);
|
|
let depth = 0;
|
|
let i = src.indexOf('{', start);
|
|
const bodyStart = i;
|
|
for (; i < src.length; i++) {
|
|
if (src[i] === '{') depth++;
|
|
if (src[i] === '}') depth--;
|
|
if (depth === 0) break;
|
|
}
|
|
return src.slice(bodyStart, i + 1);
|
|
};
|
|
|
|
// eslint-disable-next-line no-new-func
|
|
const parseSectorValuation = new Function(
|
|
'raw',
|
|
extractFn('parseSectorValuation')
|
|
.replace(/^{/, '')
|
|
.replace(/}$/, ''),
|
|
);
|
|
|
|
describe('parseSectorValuation', () => {
|
|
it('returns null for null input', () => {
|
|
assert.equal(parseSectorValuation(null), null);
|
|
});
|
|
|
|
it('returns null for undefined input', () => {
|
|
assert.equal(parseSectorValuation(undefined), null);
|
|
});
|
|
|
|
it('returns null when both PE values are missing', () => {
|
|
assert.equal(parseSectorValuation({ beta: 1.2 }), null);
|
|
});
|
|
|
|
it('parses numeric values correctly', () => {
|
|
const result = parseSectorValuation({
|
|
trailingPE: 25.3,
|
|
forwardPE: 22.1,
|
|
beta: 1.05,
|
|
ytdReturn: 0.08,
|
|
threeYearReturn: 0.12,
|
|
fiveYearReturn: 0.10,
|
|
});
|
|
assert.equal(result.trailingPE, 25.3);
|
|
assert.equal(result.forwardPE, 22.1);
|
|
assert.equal(result.beta, 1.05);
|
|
assert.equal(result.ytdReturn, 0.08);
|
|
assert.equal(result.threeYearReturn, 0.12);
|
|
assert.equal(result.fiveYearReturn, 0.10);
|
|
});
|
|
|
|
it('handles string values via typeof guard (PizzINT pattern)', () => {
|
|
const result = parseSectorValuation({
|
|
trailingPE: '18.5',
|
|
forwardPE: '16.2',
|
|
beta: '0.95',
|
|
ytdReturn: '0.05',
|
|
});
|
|
assert.equal(result.trailingPE, 18.5);
|
|
assert.equal(result.forwardPE, 16.2);
|
|
assert.equal(result.beta, 0.95);
|
|
assert.equal(result.ytdReturn, 0.05);
|
|
});
|
|
|
|
it('returns null for NaN/Infinity values', () => {
|
|
const result = parseSectorValuation({
|
|
trailingPE: NaN,
|
|
forwardPE: Infinity,
|
|
});
|
|
assert.equal(result, null);
|
|
});
|
|
|
|
it('allows partial data (trailingPE only)', () => {
|
|
const result = parseSectorValuation({
|
|
trailingPE: 20,
|
|
});
|
|
assert.equal(result.trailingPE, 20);
|
|
assert.equal(result.forwardPE, null);
|
|
assert.equal(result.beta, null);
|
|
assert.equal(result.ytdReturn, null);
|
|
});
|
|
|
|
it('allows partial data (forwardPE only)', () => {
|
|
const result = parseSectorValuation({
|
|
forwardPE: 15,
|
|
});
|
|
assert.equal(result.trailingPE, null);
|
|
assert.equal(result.forwardPE, 15);
|
|
});
|
|
});
|
|
|
|
describe('fetchYahooQuoteSummary (static analysis)', () => {
|
|
const fnStart = src.indexOf('function fetchYahooQuoteSummary(');
|
|
// Window sized to cover the direct-fetch block (headers, timeout, field
|
|
// extraction). Grown to 2000 when proxy-fallback wiring (settled guard,
|
|
// curl helper reference) was added — field extraction must stay visible.
|
|
const fnChunk = src.slice(fnStart, fnStart + 2000);
|
|
|
|
it('exists in ais-relay.cjs', () => {
|
|
assert.ok(fnStart > -1, 'fetchYahooQuoteSummary function not found');
|
|
});
|
|
|
|
it('uses summaryDetail and defaultKeyStatistics modules', () => {
|
|
assert.match(fnChunk, /summaryDetail/, 'should request summaryDetail module');
|
|
assert.match(fnChunk, /defaultKeyStatistics/, 'should request defaultKeyStatistics module');
|
|
});
|
|
|
|
it('uses v10/finance/quoteSummary endpoint', () => {
|
|
assert.match(fnChunk, /v10\/finance\/quoteSummary/, 'should call Yahoo quoteSummary v10 API');
|
|
});
|
|
|
|
it('extracts trailingPE, forwardPE, and beta', () => {
|
|
assert.match(fnChunk, /trailingPE/, 'should extract trailingPE');
|
|
assert.match(fnChunk, /forwardPE/, 'should extract forwardPE');
|
|
assert.match(fnChunk, /beta/, 'should extract beta');
|
|
});
|
|
|
|
it('extracts return metrics from defaultKeyStatistics', () => {
|
|
assert.match(fnChunk, /ytdReturn/, 'should extract ytdReturn');
|
|
});
|
|
|
|
it('includes User-Agent header', () => {
|
|
assert.match(fnChunk, /User-Agent/, 'should include User-Agent for Yahoo requests');
|
|
});
|
|
|
|
it('has timeout configured', () => {
|
|
assert.match(fnChunk, /timeout:\s*\d+/, 'should have a timeout set');
|
|
});
|
|
});
|
|
|
|
describe('seedSectorSummary valuation integration (static analysis)', () => {
|
|
const fnStart = src.indexOf('async function seedSectorSummary()');
|
|
const fnEnd = src.indexOf('\n// Gulf Quotes');
|
|
const fnBody = src.slice(fnStart, fnEnd);
|
|
|
|
it('calls fetchYahooQuoteSummary for each sector', () => {
|
|
assert.match(fnBody, /fetchYahooQuoteSummary\(s\)/, 'should call fetchYahooQuoteSummary per sector');
|
|
});
|
|
|
|
it('calls parseSectorValuation on raw response', () => {
|
|
assert.match(fnBody, /parseSectorValuation\(raw\)/, 'should parse raw valuation data');
|
|
});
|
|
|
|
it('includes valuations in payload', () => {
|
|
assert.match(fnBody, /valuations/, 'payload should include valuations object');
|
|
});
|
|
|
|
it('sleeps between Yahoo requests (rate limit)', () => {
|
|
assert.match(fnBody, /await sleep\(150\)/, 'should sleep 150ms between Yahoo calls');
|
|
});
|
|
|
|
it('logs valuation count', () => {
|
|
assert.match(fnBody, /valCount/, 'should log how many valuations were fetched');
|
|
});
|
|
});
|