mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
main
17 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
044598346e |
feat(seed-contract): PR 2a — runSeed envelope dual-write + 91 seeders migrated (#3097)
* feat(seed-contract): PR 2a — runSeed envelope dual-write + 91 seeders migrated
Opt-in contract path in runSeed: when opts.declareRecords is provided, write
{_seed, data} envelope to the canonical key alongside legacy seed-meta:*
(dual-write). State machine: OK / OK_ZERO / RETRY with zeroIsValid opt.
declareRecords throws or returns non-integer → hard fail (contract violation).
extraKeys[*] support per-key declareRecords; each extra key writes its own
envelope. Legacy seeders (no declareRecords) entirely unchanged.
Migrated all 91 scripts/seed-*.mjs to contract mode. Each exports
declareRecords returning the canonical record count, and passes
schemaVersion: 1 + maxStaleMin (matched to api/health.js SEED_META, or 2.5x
interval where no registry entry exists). Contract conformance reports 84/86
seeders with full descriptor (2 pre-existing warnings).
Legacy seed-meta keys still written so unmigrated readers keep working;
follow-up slices flip health.js + readers to envelope-first.
Tests: 61/61 PR 1 tests still pass.
Next slices for PR 2:
- api/health.js registry collapse + 15 seed-bundle-*.mjs canonicalKey wiring
- reader migration (mcp, resilience, aviation, displacement, regional-snapshot)
- direct writers — ais-relay.cjs, consumer-prices-core publish.ts
- public-boundary stripSeedEnvelope + test migration
Plan: docs/plans/2026-04-14-002-fix-runseed-zero-record-lockout-plan.md
* fix(seed-contract): unwrap envelopes in internal cross-seed readers
After PR 2a enveloped 91 canonical keys as {_seed, data}, every script-side
reader that returned the raw parsed JSON started silently handing callers the
envelope instead of the bare payload. WoW baselines (bigmac, grocery-basket,
fear-greed) saw undefined .countries / .composite; seed-climate-anomalies saw
undefined .normals from climate:zone-normals:v1; seed-thermal-escalation saw
undefined .fireDetections from wildfire:fires:v1; seed-forecasts' ~40-key
pipeline batch returned envelopes for every input.
Fix: route every script-side reader through unwrapEnvelope(...).data. Legacy
bare-shape values pass through unchanged (unwrapEnvelope returns
{_seed: null, data: raw} for any non-envelope shape).
Changed:
- scripts/_seed-utils.mjs: import unwrapEnvelope; redisGet, readSeedSnapshot,
verifySeedKey all unwrap. Exported new readCanonicalValue() helper for
cross-seed consumers.
- 18 seed-*.mjs scripts with local redisGet-style helpers or inline fetch
patched to unwrap via the envelope source module (subagent sweep).
- scripts/seed-forecasts.mjs pipeline batch: parse() unwraps each result.
- scripts/seed-energy-spine.mjs redisMget: unwraps each result.
Tests:
- tests/seed-utils-envelope-reads.test.mjs: 7 new cases covering envelope
+ legacy + null paths for readSeedSnapshot and verifySeedKey.
- Full seed suite: 67/67 pass (was 61, +6 new).
Addresses both of user's P1 findings on PR #3097.
* feat(seed-contract): envelope-aware reads in server + api helpers
Every RPC and public-boundary reader now automatically strips _seed from
contract-mode canonical keys. Legacy bare-shape values pass through unchanged
(unwrapEnvelope no-ops on non-envelope shapes).
Changed helpers (one-place fix — unblocks ~60 call sites):
- server/_shared/redis.ts: getRawJson, getCachedJson, getCachedJsonBatch
unwrap by default. cachedFetchJson inherits via getCachedJson.
- api/_upstash-json.js: readJsonFromUpstash unwraps (covers api/mcp.ts
tool responses + all its canonical-key reads).
- api/bootstrap.js: getCachedJsonBatch unwraps (public-boundary —
clients never see envelope metadata).
Left intentionally unchanged:
- api/health.js / api/seed-health.js: read only seed-meta:* keys which
remain bare-shape during dual-write. unwrapEnvelope already imported at
the meta-read boundary (PR 1) as a defensive no-op.
Tests: 67/67 seed tests pass. typecheck + typecheck:api clean.
This is the blast-radius fix the PR #3097 review called out — external
readers that would otherwise see {_seed, data} after the writer side
migrated.
* fix(test): strip export keyword in vm.runInContext'd seed source
cross-source-signals-regulatory.test.mjs loads scripts/seed-cross-source-signals.mjs
via vm.runInContext, which cannot parse ESM `export` syntax. PR 2a added
`export function declareRecords` to every seeder, which broke this test's
static-analysis approach.
Fix: strip the `export` keyword from the declareRecords line in the
preprocessed source string so the function body still evaluates as a plain
declaration.
Full test:data suite: 5307/5307 pass. typecheck + typecheck:api clean.
* feat(seed-contract): consumer-prices publish.ts writes envelopes
Wrap the 5 canonical keys written by consumer-prices-core/src/jobs/publish.ts
(overview, movers:7d/30d, freshness, categories:7d/30d/90d, retailer-spread,
basket-series) in {_seed, data} envelopes. Legacy seed-meta:<key> writes
preserved for dual-write.
Inlined a buildEnvelope helper (10 lines) rather than taking a cross-package
dependency — consumer-prices-core is a standalone npm package. Documented the
four-file parity contract (mjs source, ts mirror, js edge mirror, this copy).
Contract fields: sourceVersion='consumer-prices-core-publish-v1', schemaVersion=1,
state='OK' (recordCount>0) or 'OK_ZERO' (legitimate zero).
Typecheck: no new errors in publish.ts.
* fix(seed-contract): 3 more server-side readers unwrap envelopes
Found during final audit:
- server/worldmonitor/resilience/v1/_shared.ts: resilience score reader
parsed cached GetResilienceScoreResponse raw. Contract-mode seed-resilience-scores
now envelopes those keys.
- server/worldmonitor/resilience/v1/get-resilience-ranking.ts: p05/p95
interval lookup parsed raw from seed-resilience-scores' extra-key path.
- server/worldmonitor/infrastructure/v1/_shared.ts: mgetJson() used for
count-source keys (wildfire:fires:v1, news:insights:v1) which are both
contract-mode now.
All three now unwrap via server/_shared/seed-envelope. Legacy shapes pass
through unchanged.
Typecheck clean.
* feat(seed-contract): ais-relay.cjs direct writes produce envelopes
32 canonical-key write sites in scripts/ais-relay.cjs now produce {_seed, data}
envelopes. Inlined buildEnvelope() (CJS module can't require ESM source) +
envelopeWrite(key, data, ttlSeconds, meta) wrapper. Enveloped keys span market
bootstrap, aviation, cyber-threats, theater-posture, weather-alerts, economic
spending/fred/worldbank, tech-events, corridor-risk, usni-fleet, shipping-stress,
social:reddit, wsb-tickers, pizzint, product-catalog, chokepoint transits,
ucdp-events, satellites, oref.
Left bare (not seeded data keys): seed-meta:* (dual-write legacy),
classifyCacheKey LLM cache, notam:prev-closed-state internal state,
wm:notif:scan-dedup flags.
Updated tests/ucdp-seed-resilience.test.mjs regex to accept both upstashSet
(pre-contract) and envelopeWrite (post-contract) call patterns.
* feat(seed-contract): 15 bundle files add canonicalKey for envelope gate
54 bundle sections across 12 files now declare canonicalKey alongside the
existing seedMetaKey. _bundle-runner.mjs (from PR 1) prefers canonicalKey
when both are present — gates section runs on envelope._seed.fetchedAt
read directly from the data key, eliminating the meta-outlives-data class
of bugs.
Files touched:
- climate (5), derived-signals (2), ecb-eu (3), energy-sources (6),
health (2), imf-extended (4), macro (10), market-backup (9),
portwatch (4), relay-backup (2), resilience-recovery (5), static-ref (2)
Skipped (14 sections, 3 whole bundles): multi-key writers, dynamic
templated keys (displacement year-scoped), or non-runSeed orchestrators
(regional brief cron, resilience-scores' 222-country publish, validation/
benchmark scripts). These continue to use seedMetaKey or their own gate.
seedMetaKey preserved everywhere — dual-write. _bundle-runner.mjs falls
back to legacy when canonicalKey is absent.
All 15 bundles pass node --check. test:data: 5307/5307. typecheck:all: clean.
* fix(seed-contract): 4 PR #3097 review P1s — transform/declareRecords mismatches + envelope leaks
Addresses both P1 findings and the extra-key seed-meta leak surfaced in review:
1. runSeed helper-level invariant: seed-meta:* keys NEVER envelope.
scripts/_seed-utils.mjs exports shouldEnvelopeKey(key) — returns false for
any key starting with 'seed-meta:'. Both atomicPublish (canonical) and
writeExtraKey (extras) gate the envelope wrap through this helper. Fixes
seed-iea-oil-stocks' ANALYSIS_META_EXTRA_KEY silently getting enveloped,
which broke health.js parsing the value as bare {fetchedAt, recordCount}.
Also defends against any future manual writeExtraKey(..., envelopeMeta)
call that happens to target a seed-meta:* key.
2. seed-token-panels canonical + extras fixed.
publishTransform returns data.defi (the defi panel itself, shape {tokens}).
Old declareRecords counted data.defi.tokens + data.ai.tokens + data.other.tokens
on the transformed payload → 0 → RETRY path → canonical market:defi-tokens:v1
never wrote, and because runSeed returned before the extraKeys loop,
market:ai-tokens:v1 + market:other-tokens:v1 stayed stale too.
New: declareRecords counts data.tokens on the transformed shape. AI_KEY +
OTHER_KEY extras reuse the same function (transforms return structurally
identical panels). Added isMain guard so test imports don't fire runSeed.
3. api/product-catalog.js cached reader unwraps envelope.
ais-relay.cjs now envelopes product-catalog:v2 via envelopeWrite(). The
edge reader did raw JSON.parse(result) and returned {_seed, data} to
clients, breaking the cached path. Fix: import unwrapEnvelope from
./_seed-envelope.js, apply after JSON.parse. One site — :238-241 is
downstream of getFromCache(), so the single reader fix covers both.
4. Regression lock tests/seed-contract-transform-regressions.test.mjs (11 cases):
- shouldEnvelopeKey invariant: seed-meta:* false, canonical true
- Token-panels declareRecords works on transformed shape (canonical + both extras)
- Explicit repro of pre-fix buggy signature returning 0 — guards against revert
- resolveRecordCount accepts 0, rejects non-integer
- Product-catalog envelope unwrap returns bare shape; legacy passes through
Verification:
- npm run test:data → 5318/5318 pass (was 5307 — 11 new regressions)
- npm run typecheck:all → clean
- node --check on every modified script
iea-oil-stocks canonical declareRecords was NOT broken (user confirmed during
review — buildIndex preserves .members); only its ANALYSIS_META_EXTRA_KEY
was affected, now covered generically by commit 1's helper invariant.
* fix(seed-contract): seed-token-panels validateFn also runs on post-transform shape
Review finding: fixing declareRecords wasn't sufficient — atomicPublish() runs
validateFn(publishData) on the transformed payload too. seed-token-panels'
validate() checked data.defi/.ai/.other on the transformed {tokens} shape,
returned false, and runSeed took the early skipped-write branch (before even
reaching the declareRecords RETRY logic). Net effect: same as before the
declareRecords fix — canonical + both extras stayed stale.
Fix: validate() now checks the canonical defi panel directly (Array.isArray
(data?.tokens) && has at least one t.price > 0). AI/OTHER panels are validated
implicitly by their own extraKey declareRecords on write.
Audited the other 9 seeders with publishTransform (bls-series, bis-extended,
bis-data, gdelt-intel, trade-flows, iea-oil-stocks, jodi-gas, sanctions-pressure,
forecasts): all validateFn's correctly target the post-transform shape. Only
token-panels regressed.
Added 4 regression tests (tests/seed-contract-transform-regressions.test.mjs):
- validate accepts transformed panel with priced tokens
- validate rejects all-zero-price tokens
- validate rejects empty/missing tokens
- Explicit pre-fix repro (buggy old signature fails on transformed shape)
Verification:
- npm run test:data → 5322/5322 pass (was 5318; +4 new)
- npm run typecheck:all → clean
- node --check clean
* feat(seed-contract): add /api/seed-contract-probe validation endpoint
Single machine-readable gate for 'is PR #3097 working in production'.
Replaces the curl/jq ritual with one authenticated edge call that returns
HTTP 200 ok:true or 503 + failing check list.
What it validates:
- 8 canonical keys have {_seed, data} envelopes with required data fields
and minRecords floors (fsi-eu, zone-normals, 3 token panels + minRecords
guard against token-panels RETRY regression, product-catalog, wildfire,
earthquakes).
- 2 seed-meta:* keys remain BARE (shouldEnvelopeKey invariant; guards
against iea-oil-stocks ANALYSIS_META_EXTRA_KEY-class regressions).
- /api/product-catalog + /api/bootstrap responses contain no '_seed' leak.
Auth: x-probe-secret header must match RELAY_SHARED_SECRET (reuses existing
Vercel↔Railway internal trust boundary).
Probe logic is exported (checkProbe, checkPublicBoundary, DEFAULT_PROBES) for
hermetic testing. tests/seed-contract-probe.test.mjs covers every branch:
envelope pass/fail on field/records/shape, bare pass/fail on shape/field,
missing/malformed JSON, Redis non-2xx, boundary seed-leak detection,
DEFAULT_PROBES sanity (seed-meta invariant present, token-panels minRecords
guard present).
Usage:
curl -H "x-probe-secret: $RELAY_SHARED_SECRET" \
https://api.worldmonitor.app/api/seed-contract-probe
PR 3 will extend the probe with a stricter mode that asserts seed-meta:*
keys are GONE (not just bare) once legacy dual-write is removed.
Verification:
- tests/seed-contract-probe.test.mjs → 15/15 pass
- npm run test:data → 5338/5338 (was 5322; +16 new incl. conformance)
- npm run typecheck:all → clean
* fix(seed-contract): tighten probe — minRecords on AI/OTHER + cache-path source header
Review P2 findings: the probe's stated guards were weaker than advertised.
1. market:ai-tokens:v1 + market:other-tokens:v1 probes claimed to guard the
token-panels extra-key RETRY regression but only checked shape='envelope'
+ dataHas:['tokens']. If an extra-key declareRecords regressed to 0, both
probes would still pass because checkProbe() only inspects _seed.recordCount
when minRecords is set. Now both enforce minRecords: 1.
2. /api/product-catalog boundary check only asserted no '_seed' leak — which
is also true for the static fallback path. A broken cached reader
(getFromCache returning null or throwing) could serve fallback silently
and still pass this probe. Now:
- api/product-catalog.js emits X-Product-Catalog-Source: cache|dodo|fallback
on the response (the json() helper gained an optional source param wired
to each of the three branches).
- checkPublicBoundary declaratively requires that header's value match
'cache' for /api/product-catalog, so a fallback-serve fails the probe
with reason 'source:fallback!=cache' or 'source:missing!=cache'.
Test updates (tests/seed-contract-probe.test.mjs):
- Boundary check reworked to use a BOUNDARY_CHECKS config with optional
requireSourceHeader per endpoint.
- New cases: served-from-cache passes, served-from-fallback fails with source
mismatch, missing header fails, seed-leak still takes precedence, bad
status fails.
- Token-panels sanity test now asserts minRecords≥1 on all 3 panels.
Verification:
- tests/seed-contract-probe.test.mjs → 17/17 pass (was 15, +2 net)
- npm run test:data → 5340/5340
- npm run typecheck:all → clean
|
||
|
Elie Habib
|
4fe3bf97e5 | fix(seed-fear-greed): declare _proxyAuth — resolves ReferenceError in fetchAll() (#2540) | ||
|
Elie Habib
|
6b4dadf48c |
fix(seeder): replace curlFetch with Node.js CONNECT tunnel in fredFetchJson (#2451)
* chore: redeploy to pick up WORLDMONITOR_VALID_KEYS fix * fix(seeder): replace curlFetch with Node.js CONNECT tunnel in fredFetchJson Seeder Railway containers use node:22-alpine (no curl). fredFetchJson was routing through curlFetch when PROXY_URL is set, causing spawnSync curl ENOENT on every FRED series — all 22 series failed silently. Fix: replace curlFetch call in fredFetchJson with a pure Node.js HTTPS-through-HTTP-proxy CONNECT tunnel using built-in http/tls/https modules. No new dependencies. curlFetch is kept for ais-relay.cjs callers (Dockerfile.relay installs curl via apk add). Root cause confirmed via logs: spawnSync curl ENOENT on all 22 FRED series. * fix(seeder): try direct first, proxy as fallback in fredFetchJson * chore(seeder): TODO to consolidate all curlFetch/proxy patterns into one helper * fix(seeders): remove curl dependency from disease-outbreaks and fear-greed Both seeders used curl as primary fetch path when PROXY_URL was set. The Decodo proxy was returning SSL_ERROR_SYSCALL causing fetch failures. Replace with native fetch() — same direct-first pattern as fredFetchJson after the FRED fix. No fallback needed: these feeds are publicly accessible, and partial failures are already handled by per-source try/catch. * fix(seeder): add User-Agent to proxy tunnel; destroy socket on CONNECT failure P1: httpsProxyFetchJson was missing User-Agent header — AGENTS.md requires it for all server-side fetches; FRED CDN/WAF may reject headless requests. P2: TCP socket left open on non-200 CONNECT response; call socket.destroy(). |
||
|
Elie Habib
|
f56e7c24ad |
refactor(proxy): extract shared _proxy-utils.cjs, support Decodo host:port:user:pass format (#2399)
Previously each seeder (ais-relay.cjs, _seed-utils.mjs, seed-fear-greed.mjs, seed-disease-outbreaks.mjs) had its own inline resolveProxy() with slightly different implementations. This caused USNI seeding to fail because parseProxyUrl() only handled URL format while PROXY_URL uses Decodo host:port:user:pass format. - Add scripts/_proxy-utils.cjs with parseProxyConfig(), resolveProxyConfig(), resolveProxyString() handling both http://user:pass@host:port and host:port:user:pass formats - ais-relay.cjs: require _proxy-utils.cjs, alias parseProxyUrl = parseProxyConfig - _seed-utils.mjs: import resolveProxyString via createRequire, delegate resolveProxy() - seed-fear-greed.mjs, seed-disease-outbreaks.mjs: remove inline resolveProxy(), import from _seed-utils.mjs instead |
||
|
Elie Habib
|
2939b1f4a1 |
feat(finance-panels): add 7 macro/market panels + Daily Brief context (issues #2245-#2253) (#2258)
* feat(fear-greed): add regime state label, action stance badge, divergence warnings Closes #2245 * feat(finance-panels): add 7 new finance panels + Daily Brief macro context Implements issues #2245 (F&G Regime), #2246 (Sector Heatmap bars), #2247 (MacroTiles), #2248 (FSI), #2249 (Yield Curve), #2250 (Earnings Calendar), #2251 (Economic Calendar), #2252 (COT Positioning), #2253 (Daily Brief prompt extension). New panels: - MacroTilesPanel: CPI YoY, Unemployment, GDP, Fed Rate tiles via FRED - FSIPanel: Financial Stress Indicator gauge (HYG/TLT/VIX/HY-spread) - YieldCurvePanel: SVG yield curve chart with inverted/normal badge - EarningsCalendarPanel: Finnhub earnings calendar with BMO/AMC/BEAT/MISS - EconomicCalendarPanel: FOMC/CPI/NFP events with impact badges - CotPositioningPanel: CFTC disaggregated COT positioning bars - MarketPanel: adds sorted bar chart view above sector heatmap grid New RPCs: - ListEarningsCalendar (market/v1) - GetCotPositioning (market/v1) - GetEconomicCalendar (economic/v1) Seed scripts: - seed-earnings-calendar.mjs (Finnhub, 14-day window, TTL 12h) - seed-economic-calendar.mjs (Finnhub, 30-day window, TTL 12h) - seed-cot.mjs (CFTC disaggregated text file, TTL 7d) - seed-economy.mjs: adds yield curve tenors DGS1MO/3MO/6MO/1/2/5/30 - seed-fear-greed.mjs: adds FSI computation + sector performance Daily Brief: extends buildDailyMarketBrief with optional regime, yield curve, and sector context fed to the LLM summarization prompt. All panels default enabled in FINANCE_PANELS, disabled in FULL_PANELS. 🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/claude-code) + Compound Engineering v2.40.0 Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> * fix(finance-panels): address code review P1/P2 findings P1 - Security/Correctness: - EconomicCalendarPanel: add escapeHtml on all 7 Finnhub-sourced fields - EconomicCalendarPanel: fix panel contract (public fetchData():boolean, remove constructor self-init, add retry callbacks to all showError calls) - YieldCurvePanel: fix NaN in xPos() when count <= 1 (divide-by-zero) - seed-earnings-calendar: move Finnhub API key from URL to X-Finnhub-Token header - seed-economic-calendar: move Finnhub API key from URL to X-Finnhub-Token header - seed-earnings-calendar: add isMain guard around runSeed() call - health.js + bootstrap.js: register earningsCalendar, econCalendar, cotPositioning keys - health.js dataSize(): add earnings + instruments to property name list P2 - Quality: - FSIPanel: change !resp.fsiValue → resp.fsiValue <= 0 (rejects valid zero) - data-loader: fix Promise.allSettled type inference via indexed destructure - seed-fear-greed: allowlist cnnLabel against known values before writing to Redis - seed-economic-calendar: remove unused sleep import - seed-earnings-calendar + econ-calendar: increase TTL 43200 → 129600 (36h = 3x interval) - YieldCurvePanel: use SERIES_IDS const in RPC call (single source of truth) * fix(bootstrap): remove on-demand panel keys from bootstrap.js earningsCalendar, econCalendar, cotPositioning panels fetch via RPC on demand — they have no getHydratedData consumer in src/ and must not be in api/bootstrap.js. They remain in api/health.js BOOTSTRAP_KEYS for staleness monitoring. * fix(compound-engineering): fix markdown lint error in local settings * fix(finance-panels): resolve all P3 code-review findings - 030: MacroTilesPanel: add `deltaFormat?` field to MacroTile interface, define per-tile delta formatters (CPI pp, GDP localeString+B), replace fragile tile.id switch in tileHtml with fmt = deltaFormat ?? format - 031: FSIPanel: check getHydratedData('fearGreedIndex') at top of fetchData(); extract fsi/vix/hySpread from headerMetrics and render synchronously; fall back to live RPC only when bootstrap absent - 032: All 6 finance panels: extract lazy module-level client singletons (EconomicServiceClient or MarketServiceClient) so the client is constructed at most once per panel module lifetime, not on every fetchData - 033: get-fred-series-batch: add BAMLC0A0CM and SOFR to ALLOWED_SERIES (both seeded by seed-economy.mjs but previously unreachable via RPC) * fix(finance-panels): health.js SEED_META, FSI calibration, seed-cot catch handler - health.js: add SEED_META entries for earningsCalendar (1440min), econCalendar (1440min), cotPositioning (14400min) — without these, stopped seeds only alarm CRIT:EMPTY after TTL expiry instead of earlier WARN:STALE_SEED - seed-cot.mjs: replace bare await with .catch() handler consistent with other seeds - seed-fear-greed.mjs: recalibrate FSI thresholds to match formula output range (Low>=1.5, Moderate>=0.8, Elevated>=0.3; old values >=0.08/0.05/0.03 were calibrated for [0,0.15] but formula yields ~1-2 in normal conditions) - FSIPanel.ts: fix gauge fillPct range to [0, 2.5] matching recalibrated thresholds - todos: fix MD022/MD032 markdown lint errors in P3 review files --------- Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> |
||
|
Elie Habib
|
81b8bc5bc6 |
fix(fear-greed): correct M2SL YoY window (12→52 weeks), WALCL WoW→MoM, VIX9D gate (#2236)
* fix(panels): show radar error state on fetch failure across 22 panels
Add showError() call in catch blocks for 21 data-loader loaders so all
panels display the red radar error state instead of staying on "Loading..."
when upstream fetches fail. Also wraps ConsumerPricesPanel.fetchData() in
try/catch with showError + retry since it self-fetches.
Panels covered: stock-analysis, stock-backtest, tech-events, weather,
infra-outages, cyber-threats, protests, webcams, flight-delays,
energy-complex, trade-policy, supply-chain, satellite-fires,
security-advisories, sanctions-pressure, radiation-watch,
thermal-escalation, displacement, climate, oref-sirens,
population-exposure, consumer-prices.
* fix(panels): correct wrong panel IDs and fix ConsumerPrices error state
data-loader.ts:
- tech-events → events (actual registered panel ID)
- infra-outages → internet-disruptions (actual registered panel ID)
- Remove callPanel showError for weather/cyber-threats/protests/webcams/
flight-delays — these are map layers with no panel registration, so
those calls were no-ops
ConsumerPricesPanel.ts:
- Check overview.upstreamUnavailable after Promise.all and call showError()
with retry, since service functions swallow transport failures and return
the emptyOverview default (upstreamUnavailable:true) rather than rejecting
— the try/catch alone was dead code for this failure mode
* fix(panels): remove events showError from map-only catch, fix all-markets path
- Remove callPanel('events', 'showError') from loadTechEvents catch —
that loader runs a map-specific query (conference/mappable/90d) and
TechEventsPanel manages its own independent data fetch; a map failure
must not blank a healthy panel
- Add upstreamUnavailable check in ConsumerPricesPanel fetchData()
all-markets branch — the default view (market='all') called
fetchAllMarketsOverview() without checking the result, so all users
on the default view still got "Pending data" rows instead of the
radar error state
* fix(panels): remove update([], 0) after showError in satellite-fires catch
update() always re-renders, so calling it after showError() was
immediately replacing the radar error state with the empty dataset UI.
* revert(panels): drop ConsumerPricesPanel error state changes
upstreamUnavailable:true is set by the server on both cache miss
(seed not yet populated) and transport failure — the two cases are
indistinguishable from the client. Treating it as a hard error would
show the radar error screen on fresh deploys before the seeder has run.
The existing seeding placeholder is accurate for both states.
ConsumerPricesPanel is out of scope for this PR.
* fix(panels): remove dead showError calls from cache-fallback paths; add retry callbacks
stock-analysis / stock-backtest catch blocks had callPanel('showError')
at the top, immediately followed by a cache-fallback fetch that calls
renderAnalyses/renderBacktests (setContent) on success — wiping the
error state. Remove the premature callPanel calls; the explicit
panel.showError() at the end of each catch already handles the
no-cache path correctly.
Add onRetry callbacks for panels with long refresh intervals so
transient failures show a retry button instead of a permanent error
state lasting up to 6 hours:
- energy-complex (6h interval): → () => this.loadOilAnalytics()
- trade-policy (~1h): → () => this.loadTradePolicy()
- supply-chain (~1h): → () => this.loadSupplyChain()
* fix(fear-greed): correct M2SL YoY window (12→52 weeks), WALCL MoM (1→4 weeks), VIX9D gate
M2SL converted to weekly in 2021. fredNMonthsAgo(m2Obs, 12) = 12 weeks ≈ 3
months, not 12 months. True YoY requires 52 weekly observations. m2Score
was systematically understating expansionary conditions.
WALCL (Fed balance sheet) is weekly. fredNMonthsAgo(walclObs, 1) = 1 week
(WoW noise from settlements), not MoM. Use 4 observations for a cleaner
~1-month signal.
VIX term structure condition gated on both vix9d and vix3m, but vix9d was
unused in the actual score. If ^VIX9D failed on Yahoo, the entire term
structure fell back to neutral 50 even when ^VIX3M was available. Now
gates on vix3m only.
|
||
|
Elie Habib
|
13549a3907 |
fix(fear-greed): expand sector coverage, remove bogus C:ISSU, fix WALCL cadence (#2237)
* fix(fear-greed): expand sector coverage, remove bogus C:ISSU, fix WALCL cadence - Add 7 missing S&P sectors to YAHOO_SYMBOLS and momentum sectorCloses: XLY, XLP, XLI, XLB, XLU, XLRE, XLC (was only 4: XLK, XLF, XLE, XLV) Bias toward tech+financials was understating sector RSI in defensive rallies - Remove C:ISSU advance/decline proxy — C: prefix is Yahoo crypto pair notation, not a real A/D symbol; silent corruption risk if Yahoo returns a stale price - WALCL momentum: fredNMonthsAgo(walclObs, 1→4) — 1 observation on weekly WALCL data is 1 week (noise), 4 observations = 1 month (meaningful trend) * fix(cmd-k): strip double Panel: prefix from new commands and economic-correlation resolveCommandLabel already prepends 'Panel: ' via i18n. Commands without a matching panels.* i18n key fall back to cmd.label, causing double prefix when cmd.label itself started with 'Panel: '. Affects all 23 new commands added in the previous commit plus the pre-existing panel:economic-correlation. Fix: label field now holds the bare panel name as fallback (no prefix). |
||
|
Elie Habib
|
e964f9d43b |
fix(fear-greed): fix SMA200 null (range=1y) and VIX neutral calibration (#2224)
Two more scoring bugs in addition to the Credit/Liquidity fixes in #2222. SMA200 always null (range=3mo → range=1y): Yahoo was fetched with range=3mo (~63 trading days). SMA200 requires 200 bars so it was ALWAYS null. The trend formula falls back to dist200=0 → score=25 (below-average-MA penalty capped). With 1y data, SPX is still above its 200dMA despite the current correction, so aboveCount rises from 0→1 and dist200 reflects the actual buffer above the long-term trend (score ~47-53 instead of 25). VIX neutral point shifted (range 12–40 → 12–35): Old range put neutral at VIX=26, but the historical long-run average is ~19-20. VIX=26.57 scored 48 (neutral). With the corrected 12–35 range, neutral is at ~23.5 and VIX=26.57 scores ~37 (mild fear) — more consistent with CNN F&G=15, AAII Bear=52%, and the VIX backwardation term structure observed simultaneously. |
||
|
Elie Habib
|
b947b25e0a |
fix(fear-greed): recalibrate Credit and Liquidity scoring formulas (#2222)
Three calibration bugs were producing inflated Credit (88) and Liquidity (69) scores even while VIX>26, CNN F&G=15, and AAII Bear=52% signaled market stress. Credit fixes: - HY OAS baseline was 3.0% (near all-time tights), causing scores ≈100 in normal conditions. Recalibrated to range 2.0%–10.0% (long-run avg ~5.0%). New score at 3.19%: 85 vs 96 previously. - IG OAS similarly recalibrated to range 0.4%–3.0% (long-run avg ~1.3%). - Trend now uses fredNTradingDaysAgo(obs, 20) (~1 calendar month for daily series) instead of fredNMonthsAgo(obs, 1) which was stepping back only 1 observation (= yesterday) on daily FRED data — comparing Friday→Monday noise, not a real 1-month trend. Liquidity fix: - M2 YoY multiplier reduced from 10x to 5x. With 10x, perfectly normal 5% annual M2 growth pegged the score at 100 (max greed), masking SOFR's restrictive signal. With 5x, 5% growth ≈ 75 (moderately accommodative). Net effect at current market conditions: - Credit: 88 → 68 (HY spreads still tight, but widening trend now reflected) - Liquidity: 69 → 59 (M2 growth acknowledged but not dominant) |
||
|
Elie Habib
|
c0e241cadd |
fix(fear-greed): fix AAII bear extraction — use table cell position not header label (#2211)
Bullish|Neutral|Bearish are column headers; data follows in the next row. Label-based regex /Bearish[^%]*?([\d.]+)%/ was catching the Bullish data cell (30.4%) because it is the first % sign after the Bearish header in DOM order. Fix: extract first 3 tableTxt percentage cells positionally. Columns: Bullish(0) | Neutral(1) | Bearish(2). Result: bear now correctly reads 52.0% instead of 30.4%. |
||
|
Elie Habib
|
f4ad19fa48 |
fix(seed): route Yahoo Finance through PROXY_URL (Decodo) when set (#2205)
Railway container IPs get blocked by Yahoo after restarts. Reads PROXY_URL="host:port:user:pass" (Decodo residential) first, falls back to OREF_PROXY_AUTH if absent. Also surfaces error cause in "fetch failed" log lines and appends proxy=yes/no to the sources summary line. |
||
|
Elie Habib
|
aa41418fd9 |
fix(fear-greed): replace broken CBOE CDN + CNN with working alternatives (#2201)
* fix(fear-greed): swap proxy country il→us for CBOE/CNN fetches OREF_PROXY_AUTH targets Israel (wifi;il;;;) — Froxy returns 422 CONNECT when routing Israeli exits to US financial CDNs (cdn.cboe.com, dataviz.cnn.io). resolveUsProxy() swaps %3Bil%3B→%3Bus%3B in OREF_PROXY_AUTH so CBOE/CNN requests exit via US residential IPs. FEAR_GREED_PROXY_AUTH env var can override explicitly. Also log proxy country in Sources line for diagnostics. * fix(fear-greed): replace broken CBOE CDN + CNN endpoints cdn.cboe.com returns 403 (Cloudflare Bot Management blocks all server-side requests regardless of proxy or UA). CNN /graphdata/:date returns 500 and the /current endpoint returns 418 with Windows UA. - CBOE: switch to Barchart $CPC for total put/call ratio (same scraping approach as existing $S5TH, confirmed 0.80 live) - CNN: switch to /current endpoint with Mac UA (confirmed working, score=15) - Remove curlGet(), execFileSync import, resolveUsProxy(), _proxyAuth (no longer needed -- all sources work without proxy) |
||
|
Elie Habib
|
f8b793dcc8 |
fix(fear-greed): switch CBOE+CNN proxy fetch from undici to curl (#2199)
Node.js TLS fingerprint (JA3) is blocked by CBOE CDN and CNN dataviz even when routed through a residential proxy. curl's fingerprint passes. Replaces undici ProxyAgent with execFileSync curl, same pattern as orefCurlFetch() in ais-relay.cjs. Removes undici import entirely. |
||
|
Elie Habib
|
db1211e2c9 |
fix(fear-greed): shrink header score 56px→36px + add per-source seed diagnostics (#2196)
* fix(fear-greed): reduce header score size + add per-source seed diagnostics - FearGreedPanel: score 56px→36px, label 16px→13px (matches Strategic Risk scale) - seed: add source status log line showing each data source value/null per run, so Railway container logs show whether CBOE/CNN/Barchart are returning data and whether the proxy is active * fix(docs): add blank lines around lists to pass MD032 lint |
||
|
Elie Habib
|
9b4a7f793f |
fix(fear-greed): route CBOE+CNN through residential proxy, update Chrome UA to 134 (#2191)
CBOE CDN (cdn.cboe.com) returns 403 and CNN dataviz returns 418 to Railway datacenter IPs — both block non-residential server traffic. Route fetchCBOE() and fetchCNN() through the OREF_PROXY_AUTH residential proxy (froxy.com) using undici ProxyAgent. Falls back to native fetch when OREF_PROXY_AUTH is unset (local dev). Also adds Referer headers and explicit HTTP status logging so failures are visible in seed logs. Updates CHROME_UA from Chrome/120 to 134. |
||
|
Elie Habib
|
1d1a82004f |
fix(fear-greed): mark breadth as degraded when ^MMTH unavailable (404) (#2187)
* fix(fear-greed): mark breadth as degraded when ^MMTH is unavailable
^MMTH (% S&P 500 stocks above 200d MA) returns HTTP 404 from Yahoo.
The breadth score already fell back to neutral 50 but silently —
no degraded flag was set, so the UI showed full confidence on an
input that was missing.
- scoreCategory('breadth'): return degraded: mmthPrice == null
- payload.categories.breadth: propagate degraded: cats.breadth.degraded ?? false
* fix(fear-greed): replace invalid ^MMTH with Barchart $S5TH + RSP/SPY proxy
^MMTH is a StockCharts/Barchart breadth symbol — not a Yahoo Finance
ticker. Yahoo's v8/finance/chart API serves price data for tradable
instruments only; ^MMTH, ^MMFI, ^BPSPX, ^SPXAH all return 404.
Replace with a two-tier approach:
- Primary: Barchart $S5TH (% of S&P 500 above 200d MA) — exact metric,
free, no key required, runs in parallel with existing data sources
- Fallback: RSP/SPY relative-to-mean ratio, normalized to 0-100 —
equal-weight vs cap-weight spread is a recognized breadth proxy
Remove ^MMTH from YAHOO_SYMBOLS (saves 150ms per run).
degraded flag on breadth now only fires when both sources fail.
* fix(fear-greed): address code review findings on Barchart breadth fetch
P1: NaN from parseFloat() bypassed proxy — use Number.isFinite() guard
P1: RSP/SPY double-counted in breadth score when proxy active — split
pctAbove200dScore (Barchart-only, fed into scoreCategory) from
pctAbove200dDisplay (Barchart or proxy, for header metric only)
P2: Narrow lastPrice regex to __NEXT_DATA__ script block only
P2: Widen proxy normalization band from ±10% to ±20% (denominator 0.4)
P2: Raise minimum closes guard from 5 to 20 for proxy computation
P3: Log warn when Barchart $S5TH returns null (observability)
* fix(fear-greed): pctAbove200d shows only real Barchart value, never proxy
RSP/SPY proxy was writing into headerMetrics.pctAbove200d which the panel
renders as the literal "% S&P 500 above 200d MA". A normalized RSP/SPY
ratio is not that metric — showing it as a percentage is a correctness bug.
- Remove proxy from pctAbove200dDisplay; collapse back to single pctAbove200d
- pctAbove200d null when Barchart unavailable → header shows N/A (honest)
- Breadth scoring already captures RSP/SPY signal via rspScore independently
- Add structured error logging to fetchBarchartS5TH (HTTP status + exception message)
|
||
|
Elie Habib
|
7013b2f9f1 |
feat(market): Fear & Greed Index 2.0 — 10-category composite sentiment panel (#2181)
* Add Fear & Greed Index 2.0 reverse engineering brief Analyzes the 10-category weighted composite (Sentiment, Volatility, Positioning, Trend, Breadth, Momentum, Liquidity, Credit, Macro, Cross-Asset) with scoring formulas, data source audit, and implementation plan for building it as a worldmonitor panel. https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Add seed script implementation plan to F&G brief Details exact endpoints, Yahoo symbols (17 calls), Redis key schema, computed metrics, FRED series to add (BAMLC0A0CM, SOFR), CNN/AAII sources, output JSON schema, and estimated runtime (~8s per seed run). https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Update brief: all sources are free, zero paid APIs needed - CBOE CDN CSVs for put/call ratios (totalpc.csv, equitypc.csv) - CNN dataviz API for Fear & Greed (production.dataviz.cnn.io) - Yahoo Finance for VIX9D/VIX3M/SKEW/RSP/NYA (standard symbols) - FRED for IG spread (BAMLC0A0CM) and SOFR (add to existing array) - AAII scrape for bull/bear survey (only medium-effort source) - Breadth via RSP/SPY divergence + NYSE composite (no scraping) https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Add verified Yahoo symbols for breadth + finalized source list New discoveries: - ^MMTH = % stocks above 200 DMA (direct Yahoo symbol!) - C:ISSU = NYSE advance/decline data - CNN endpoint accepts date param for historical data - CBOE CSVs have data back to 2003 - 33 total calls per seed run, ~6s runtime All 10 categories now have confirmed free sources. https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Rewrite F&G brief as forward-looking design doc Remove all reverse-engineering language, screenshot references, and discovery notes. Clean structure: goal, scoring model, data sources, formulas, seed script plan, implementation phases, MVP path. https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * docs: apply gold standard corrections to fear-greed-index-2.0 brief * feat(market): add Fear & Greed Index 2.0 — 10-category composite sentiment panel Composite 0-100 index from 10 weighted categories: sentiment (CNN F&G, AAII, crypto F&G), volatility (VIX, term structure), positioning (P/C ratio, SKEW), trend (SPX vs MAs), breadth (% >200d, RSP/SPY divergence), momentum (sector RSI, ROC), liquidity (M2, Fed BS, SOFR), credit (HY/IG spreads), macro (Fed rate, yield curve, unemployment), cross-asset (gold/bonds/DXY vs equities). Data layer: - seed-fear-greed.mjs: 19 Yahoo symbols (150ms gaps), CBOE P/C CSVs, CNN F&G API, AAII scrape (degraded-safe), FRED Redis reads. TTL 64800s. - seed-economy.mjs: add BAMLC0A0CM (IG spread) and SOFR to FRED_SERIES. - Bootstrap 4-file checklist: cache-keys, bootstrap.js, health.js, handler. Proto + RPC: - get_fear_greed_index.proto with FearGreedCategory message. - get-fear-greed-index.ts handler reads seeded Redis data. Frontend: - FearGreedPanel with gauge, 9-metric header grid, 10-category breakdown. - Self-loading via bootstrap hydration + RPC fallback. - Registered in panel-layout, App.ts (prime + refresh), panel config, Cmd-K commands, finance variant, i18n (en/ar/zh/es). * fix(market): add RPC_CACHE_TIER entry for get-fear-greed-index * fix(docs): escape bare angle bracket in fear-greed brief for MDX * fix(docs): fix markdown lint errors in fear-greed brief (blank lines around headings/lists) * fix(market): fix seed-fear-greed bugs from code review - fredLatest/fredNMonthsAgo: guard parseFloat with Number.isFinite to handle FRED's "." missing-data sentinel (was returning NaN which propagated through scoring as a truthy non-null value) - Remove 3 unused Yahoo symbols (^NYA, HYG, LQD) that were fetched but not referenced in any scoring category (saves ~450ms per run) - fedRateStr: display effective rate directly instead of deriving target range via (fedRate - 0.25) which was incorrect * fix(market): address P2/P3 review findings in Fear & Greed - FearGreedPanel: add mapSeedPayload() to correctly map raw seed JSON to proto-shaped FearGreedData; bootstrap hydration was always falling through to RPC because seed shape (composite.score) differs from proto shape (compositeScore) - FearGreedPanel: fix fmt() — remove === 0 guard and add explicit > 0 checks on VIX and P/C Ratio display to handle proto default zeros without masking genuine zero values (e.g. pctAbove200d) - seed-fear-greed: remove broken history write — each run overwrote the key with a single-entry array (no read-then-append), making the 90-day TTL meaningless; no consumer exists yet so defer to later - seed-fear-greed: extract hySpreadVal const to avoid double fredLatest call - seed-fear-greed: fix stale comment (19 symbols → 16 after prior cleanup) --------- Co-authored-by: Claude <noreply@anthropic.com> |