mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
2939b1f4a1cadfa67feacdec3767806f00d9117d
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
2939b1f4a1 |
feat(finance-panels): add 7 macro/market panels + Daily Brief context (issues #2245-#2253) (#2258)
* feat(fear-greed): add regime state label, action stance badge, divergence warnings Closes #2245 * feat(finance-panels): add 7 new finance panels + Daily Brief macro context Implements issues #2245 (F&G Regime), #2246 (Sector Heatmap bars), #2247 (MacroTiles), #2248 (FSI), #2249 (Yield Curve), #2250 (Earnings Calendar), #2251 (Economic Calendar), #2252 (COT Positioning), #2253 (Daily Brief prompt extension). New panels: - MacroTilesPanel: CPI YoY, Unemployment, GDP, Fed Rate tiles via FRED - FSIPanel: Financial Stress Indicator gauge (HYG/TLT/VIX/HY-spread) - YieldCurvePanel: SVG yield curve chart with inverted/normal badge - EarningsCalendarPanel: Finnhub earnings calendar with BMO/AMC/BEAT/MISS - EconomicCalendarPanel: FOMC/CPI/NFP events with impact badges - CotPositioningPanel: CFTC disaggregated COT positioning bars - MarketPanel: adds sorted bar chart view above sector heatmap grid New RPCs: - ListEarningsCalendar (market/v1) - GetCotPositioning (market/v1) - GetEconomicCalendar (economic/v1) Seed scripts: - seed-earnings-calendar.mjs (Finnhub, 14-day window, TTL 12h) - seed-economic-calendar.mjs (Finnhub, 30-day window, TTL 12h) - seed-cot.mjs (CFTC disaggregated text file, TTL 7d) - seed-economy.mjs: adds yield curve tenors DGS1MO/3MO/6MO/1/2/5/30 - seed-fear-greed.mjs: adds FSI computation + sector performance Daily Brief: extends buildDailyMarketBrief with optional regime, yield curve, and sector context fed to the LLM summarization prompt. All panels default enabled in FINANCE_PANELS, disabled in FULL_PANELS. 🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/claude-code) + Compound Engineering v2.40.0 Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> * fix(finance-panels): address code review P1/P2 findings P1 - Security/Correctness: - EconomicCalendarPanel: add escapeHtml on all 7 Finnhub-sourced fields - EconomicCalendarPanel: fix panel contract (public fetchData():boolean, remove constructor self-init, add retry callbacks to all showError calls) - YieldCurvePanel: fix NaN in xPos() when count <= 1 (divide-by-zero) - seed-earnings-calendar: move Finnhub API key from URL to X-Finnhub-Token header - seed-economic-calendar: move Finnhub API key from URL to X-Finnhub-Token header - seed-earnings-calendar: add isMain guard around runSeed() call - health.js + bootstrap.js: register earningsCalendar, econCalendar, cotPositioning keys - health.js dataSize(): add earnings + instruments to property name list P2 - Quality: - FSIPanel: change !resp.fsiValue → resp.fsiValue <= 0 (rejects valid zero) - data-loader: fix Promise.allSettled type inference via indexed destructure - seed-fear-greed: allowlist cnnLabel against known values before writing to Redis - seed-economic-calendar: remove unused sleep import - seed-earnings-calendar + econ-calendar: increase TTL 43200 → 129600 (36h = 3x interval) - YieldCurvePanel: use SERIES_IDS const in RPC call (single source of truth) * fix(bootstrap): remove on-demand panel keys from bootstrap.js earningsCalendar, econCalendar, cotPositioning panels fetch via RPC on demand — they have no getHydratedData consumer in src/ and must not be in api/bootstrap.js. They remain in api/health.js BOOTSTRAP_KEYS for staleness monitoring. * fix(compound-engineering): fix markdown lint error in local settings * fix(finance-panels): resolve all P3 code-review findings - 030: MacroTilesPanel: add `deltaFormat?` field to MacroTile interface, define per-tile delta formatters (CPI pp, GDP localeString+B), replace fragile tile.id switch in tileHtml with fmt = deltaFormat ?? format - 031: FSIPanel: check getHydratedData('fearGreedIndex') at top of fetchData(); extract fsi/vix/hySpread from headerMetrics and render synchronously; fall back to live RPC only when bootstrap absent - 032: All 6 finance panels: extract lazy module-level client singletons (EconomicServiceClient or MarketServiceClient) so the client is constructed at most once per panel module lifetime, not on every fetchData - 033: get-fred-series-batch: add BAMLC0A0CM and SOFR to ALLOWED_SERIES (both seeded by seed-economy.mjs but previously unreachable via RPC) * fix(finance-panels): health.js SEED_META, FSI calibration, seed-cot catch handler - health.js: add SEED_META entries for earningsCalendar (1440min), econCalendar (1440min), cotPositioning (14400min) — without these, stopped seeds only alarm CRIT:EMPTY after TTL expiry instead of earlier WARN:STALE_SEED - seed-cot.mjs: replace bare await with .catch() handler consistent with other seeds - seed-fear-greed.mjs: recalibrate FSI thresholds to match formula output range (Low>=1.5, Moderate>=0.8, Elevated>=0.3; old values >=0.08/0.05/0.03 were calibrated for [0,0.15] but formula yields ~1-2 in normal conditions) - FSIPanel.ts: fix gauge fillPct range to [0, 2.5] matching recalibrated thresholds - todos: fix MD022/MD032 markdown lint errors in P3 review files --------- Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> |
||
|
|
7013b2f9f1 |
feat(market): Fear & Greed Index 2.0 — 10-category composite sentiment panel (#2181)
* Add Fear & Greed Index 2.0 reverse engineering brief Analyzes the 10-category weighted composite (Sentiment, Volatility, Positioning, Trend, Breadth, Momentum, Liquidity, Credit, Macro, Cross-Asset) with scoring formulas, data source audit, and implementation plan for building it as a worldmonitor panel. https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Add seed script implementation plan to F&G brief Details exact endpoints, Yahoo symbols (17 calls), Redis key schema, computed metrics, FRED series to add (BAMLC0A0CM, SOFR), CNN/AAII sources, output JSON schema, and estimated runtime (~8s per seed run). https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Update brief: all sources are free, zero paid APIs needed - CBOE CDN CSVs for put/call ratios (totalpc.csv, equitypc.csv) - CNN dataviz API for Fear & Greed (production.dataviz.cnn.io) - Yahoo Finance for VIX9D/VIX3M/SKEW/RSP/NYA (standard symbols) - FRED for IG spread (BAMLC0A0CM) and SOFR (add to existing array) - AAII scrape for bull/bear survey (only medium-effort source) - Breadth via RSP/SPY divergence + NYSE composite (no scraping) https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Add verified Yahoo symbols for breadth + finalized source list New discoveries: - ^MMTH = % stocks above 200 DMA (direct Yahoo symbol!) - C:ISSU = NYSE advance/decline data - CNN endpoint accepts date param for historical data - CBOE CSVs have data back to 2003 - 33 total calls per seed run, ~6s runtime All 10 categories now have confirmed free sources. https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * Rewrite F&G brief as forward-looking design doc Remove all reverse-engineering language, screenshot references, and discovery notes. Clean structure: goal, scoring model, data sources, formulas, seed script plan, implementation phases, MVP path. https://claude.ai/code/session_01HR69u6oF1VCMwsC2PHFL8i * docs: apply gold standard corrections to fear-greed-index-2.0 brief * feat(market): add Fear & Greed Index 2.0 — 10-category composite sentiment panel Composite 0-100 index from 10 weighted categories: sentiment (CNN F&G, AAII, crypto F&G), volatility (VIX, term structure), positioning (P/C ratio, SKEW), trend (SPX vs MAs), breadth (% >200d, RSP/SPY divergence), momentum (sector RSI, ROC), liquidity (M2, Fed BS, SOFR), credit (HY/IG spreads), macro (Fed rate, yield curve, unemployment), cross-asset (gold/bonds/DXY vs equities). Data layer: - seed-fear-greed.mjs: 19 Yahoo symbols (150ms gaps), CBOE P/C CSVs, CNN F&G API, AAII scrape (degraded-safe), FRED Redis reads. TTL 64800s. - seed-economy.mjs: add BAMLC0A0CM (IG spread) and SOFR to FRED_SERIES. - Bootstrap 4-file checklist: cache-keys, bootstrap.js, health.js, handler. Proto + RPC: - get_fear_greed_index.proto with FearGreedCategory message. - get-fear-greed-index.ts handler reads seeded Redis data. Frontend: - FearGreedPanel with gauge, 9-metric header grid, 10-category breakdown. - Self-loading via bootstrap hydration + RPC fallback. - Registered in panel-layout, App.ts (prime + refresh), panel config, Cmd-K commands, finance variant, i18n (en/ar/zh/es). * fix(market): add RPC_CACHE_TIER entry for get-fear-greed-index * fix(docs): escape bare angle bracket in fear-greed brief for MDX * fix(docs): fix markdown lint errors in fear-greed brief (blank lines around headings/lists) * fix(market): fix seed-fear-greed bugs from code review - fredLatest/fredNMonthsAgo: guard parseFloat with Number.isFinite to handle FRED's "." missing-data sentinel (was returning NaN which propagated through scoring as a truthy non-null value) - Remove 3 unused Yahoo symbols (^NYA, HYG, LQD) that were fetched but not referenced in any scoring category (saves ~450ms per run) - fedRateStr: display effective rate directly instead of deriving target range via (fedRate - 0.25) which was incorrect * fix(market): address P2/P3 review findings in Fear & Greed - FearGreedPanel: add mapSeedPayload() to correctly map raw seed JSON to proto-shaped FearGreedData; bootstrap hydration was always falling through to RPC because seed shape (composite.score) differs from proto shape (compositeScore) - FearGreedPanel: fix fmt() — remove === 0 guard and add explicit > 0 checks on VIX and P/C Ratio display to handle proto default zeros without masking genuine zero values (e.g. pctAbove200d) - seed-fear-greed: remove broken history write — each run overwrote the key with a single-entry array (no read-then-append), making the 90-day TTL meaningless; no consumer exists yet so defer to later - seed-fear-greed: extract hySpreadVal const to avoid double fredLatest call - seed-fear-greed: fix stale comment (19 symbols → 16 after prior cleanup) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
45ad2d4132 | fix(fred): remove GSCPI series — does not exist on FRED API (#2110) | ||
|
|
a4cc6be9f4 |
fix(fred): bump FRED_TTL to 26h and health maxStaleMin to 1500min (#2109)
FRED_TTL was 1h but seed runs daily — key expires 23h before next run, causing persistent EMPTY CRIT. Same pattern as consumer-prices publish.ts (93600s TTL). maxStaleMin bumped from 90 to 1500 to match daily cadence. |
||
|
|
a1509c811b |
fix(seed-economy): add missing FRED series to seeder (#2066)
Adds BAMLH0A0HYM2, ICSA, MORTGAGE30US, GSCPI to FRED_SERIES array. These were in the server allowlist and frontend config but absent from the seeder, causing empty data in production on every request. Closes #2041 |
||
|
|
e0800f1a31 |
fix(resilience): extend TTLs and add Finnhub/FRED fallbacks for macro signals (#1982)
* fix(panels): hide finance-only panels on non-finance variants
stock-analysis, stock-backtest, and daily-market-brief are FINANCE_PANELS-only
but persisted panelSettings from a previous finance/full session caused them
to appear on tech variant. Guard creation with variantPanelKeys set derived
from VARIANT_DEFAULTS[SITE_VARIANT].
* fix(pro): remove duplicate Create with AI button, gate MCP connect as pro
With unified isProUser() both widget creation buttons showed simultaneously.
Remove the basic-tier 'Create with AI' block (duplicate of 'Create Interactive
Widget'). Gate 'Connect with MCP' with isProUser() — was previously ungated.
* feat(pro): gate export (⬇) and playback (⏪) toolbar buttons for pro users
Both setupExportPanel() and setupPlaybackControl() now return early
unless isProUser() — either wm-widget-key or wm-pro-key grants access.
* revert(panels): remove incorrect variant guard on stock/daily-market panels
The variantPanelKeys.has() guards were wrong — shouldCreatePanel() already
handles variant defaults via panelSettings. Users who manually enable these
panels on any variant should see them (locked if not pro, unlocked if pro).
The guard broke cross-variant user customization.
* fix(resilience): extend TTLs and add Finnhub/FRED fallbacks for macro signals
- MACRO_TTL 1800s → 21600s (6h) so stale data serves during Yahoo outages
- MARKET_SEED_TTL 1800s → 7200s (2h) in ais-relay for sectors/quotes
- Finnhub stock/candle fallback for QQQ and XLP when Yahoo returns null
- Finnhub crypto/candle fallback for BTC (BINANCE:BTCUSDT) when Yahoo fails
- FRED DEXJPUS fallback for JPY/USD historical prices (no new API key needed)
Prevents Macro Stress and Sector Heatmap panels from showing "unavailable"
during Yahoo Finance 429 rate-limit windows from Railway IPs.
|
||
|
|
dd56ab052c |
fix(market): prevent stale-data overwrite and add Yahoo retry for commodities/macro panels (#1854)
- seed-economy: skip Redis write for macro-signals when totalCount=0 so Yahoo failures don't overwrite previously cached good data - seed-economy: raise MACRO_TTL 900→1800s so valid data survives two missed 15-min cron cycles - health: tighten macroSignals maxStaleMin 60→20 to alert before TTL expires (was 4x longer than the data lifetime) - ais-relay: add one retry pass for commodity symbols that return null, with 3s cooldown, to recover from transient Yahoo 429 rate limits |
||
|
|
467608c2d7 |
chore: clear baseline lint debt (173 warnings → 49) (#1712)
Mechanical fixes across 13 files: - isNaN() → Number.isNaN() (all values already numeric from parseFloat/parseInt) - let → const where never reassigned - Math.pow() → ** operator - Unnecessary continue in for loop - Useless string escape in test description - Missing parseInt radix parameter - Remove unused private class member (write-only counter) - Prefix unused function parameter with _ Config: suppress noImportantStyles (CSS !important is intentional) and useLiteralKeys (bracket notation used for computed/dynamic keys) in biome.json. Remaining 49 warnings are all noExcessiveCognitiveComplexity (already configured as warn, safe to address incrementally). |
||
|
|
65543d71d5 |
fix(seeds): improve resilience and fix dead APIs across seed scripts (#1644)
* fix(seeds): improve resilience and fix dead APIs across seed scripts
- Fix wrong domain in seed-service-statuses (worldmonitor.app to api.worldmonitor.app)
- Fix Kalshi API domain migration (trading-api.kalshi.com to api.elections.kalshi.com)
- Replace dead trending APIs (gitterapp.com, herokuapp.com) with OSSInsight + GitHub Search
- Fix case-sensitive HTML detection in seed-usni-fleet (lowercase doctype not matched)
- Add Promise.allSettled rejection logging across 8 seed scripts
- Wrap fetch loops in try-catch (seed-supply-chain-trade, seed-economy) so a single
network error no longer kills the entire function
- Update list-trending-repos.ts RPC handler to match seed changes
* fix(seeds): correct OSSInsight response parsing and period-aware GitHub Search fallback
- OSSInsight returns {data: {rows: [...]}} not {data: [...]}, fix both seed and handler
- GitHub Search fallback now respects period parameter (daily=1d, weekly=7d, monthly=30d)
* fix(seeds): correct OSSInsight period values (past_week/past_month, not past_7_days/past_28_days)
|
||
|
|
249c088639 | fix: add fetch error cause logging to all remaining seed scripts (#1643) | ||
|
|
485d416065 |
feat(seeds): Railway seed scripts for all unseeded Vercel RPC endpoints (#1599)
* feat(seeds): add Railway seed scripts for economic and trade endpoints
Two new seed scripts to eliminate Vercel edge external API calls:
seed-economy.mjs:
- EIA energy prices (WTI, Brent) -> economic:energy:v1:all
- EIA energy capacity (Solar, Wind, Coal) -> economic:capacity:v1:COL,SUN,WND:20
- FRED series (10 series) -> economic:fred:v1:<id>:120
- Macro signals (Yahoo, Alternative.me, Mempool) -> economic:macro-signals:v1
seed-supply-chain-trade.mjs:
- Shipping rates (FRED) -> supply_chain:shipping:v2
- Trade barriers (WTO tariff gap) -> trade:barriers:v1:tariff-gap:50
- Trade restrictions (WTO MFN overview) -> trade:restrictions:v1:tariff-overview:50
- Trade flows (WTO, 15 major reporters) -> trade:flows:v1:<reporter>:000:10
- Tariff trends (WTO, 15 major reporters) -> trade:tariffs:v1:<reporter>:all:10
Cache keys match handler patterns exactly so cachedFetchJson finds
pre-seeded data and avoids live external API calls from Vercel edge.
* feat(seeds): add seed-aviation.mjs for airport ops and aviation news
Seeds 2 aviation endpoints with predictable default params:
- getAirportOpsSummary (AviationStack + NOTAM) -> aviation:ops-summary:v1:CDG,ESB,FRA,IST,LHR,SAW
- listAviationNews (9 RSS feeds, 24h window) -> aviation:news::24:v1
NOT seeded (inherently on-demand, user-specific inputs):
- getFlightStatus: specific flight number lookup
- trackAircraft: bounding-box or icao24 queries
- listAirportFlights: arbitrary airport+direction+limit combos
- getCarrierOps: depends on listAirportFlights with variable params
* feat(seeds): add seed-conflict-intel.mjs for ACLED, HAPI, and PizzINT
Seeds 3 conflict/intelligence endpoints with predictable default params:
- listAcledEvents (all countries, last 30 days) -> conflict:acled:v1:all:0:0
- getHumanitarianSummary (20 top conflict countries) -> conflict:humanitarian:v1:<CC>
- getPizzintStatus (base + GDELT variants) -> intel:pizzint:v1:base, intel:pizzint:v1:gdelt
NOT seeded (inherently on-demand, LLM or user-specific inputs):
- classifyEvent: per-headline LLM classification
- deductSituation: per-query LLM deduction
- getCountryIntelBrief: per-country LLM brief with context hash
- getCountryFacts: per-country REST Countries + Wikidata + Wikipedia
- searchGdeltDocuments: per-query GDELT search
Requires: ACLED_EMAIL, ACLED_KEY, UPSTASH_REDIS_REST_URL/TOKEN
* feat(seeds): add seed-research.mjs for arXiv, HN, tech events, trending repos
Seeds 4 research endpoints:
- listArxivPapers (cs.AI, cs.CL, cs.CR) -> research:arxiv:v1:<cat>::50
- listHackernewsItems (top, best feeds) -> research:hackernews:v1:<feed>:30
- listTechEvents (Techmeme ICS + dev.events RSS) -> research:tech-events:v1
- listTrendingRepos (python, javascript, typescript) -> research:trending:v1:<lang>:daily:50
Tech events key is also seeded by the relay, this script provides backup
hydration and ensures the key is warm even if relay hasn't run yet.
Requires: UPSTASH_REDIS_REST_URL/TOKEN
* feat(seeds): add seed-military-maritime-news.mjs for USNI and nav warnings
Seeds 2 endpoints with predictable default params:
- USNI Fleet Report (WordPress JSON API) -> usni-fleet:sebuf:v1 + stale backup
- Navigational Warnings (NGA broadcast, all areas) -> maritime:navwarnings:v1:all
NOT seeded (inherently on-demand):
- getAircraftDetails/batch: per-icao24 Wingbits lookup
- listMilitaryFlights: bounding-box query (quantized 1-degree grid)
- getVesselSnapshot: in-memory cache, reads from relay /ais-snapshot
- listFeedDigest: per-feed-URL RSS caching (hundreds of feeds, relay proxied)
- summarizeArticle: per-article LLM summarization
Requires: UPSTASH_REDIS_REST_URL/TOKEN
* feat(seeds): add seed-infra.mjs warm-ping for service statuses and cable health
Uses warm-ping pattern (calls Vercel RPC from Railway) because:
- list-service-statuses: 30 status page parsers with 8 custom formats
- get-cable-health: NGA text analysis with cable name matching + proximity
Replicating this logic in a standalone script is fragile and duplicative.
NOT seeded (on-demand):
- search-imagery: per-bbox/datetime STAC query
- get-giving-summary: hardcoded baselines, no external fetches
- get-webcam-image: per-webcamId Windy API lookup
* fix(seeds): move secondary key writes before process.exit, fix data shapes
Critical bugs found in code review:
1. runSeed() calls process.exit(0) after primary key write, so .then()
callbacks were dead code. All secondary keys (FRED, macro signals,
trade data, HAPI summaries, pizzint, HN, trending, etc.) were NEVER
written. Fix: move writeExtraKey calls inside fetchAll() before return.
2. FRED cache key used :120 suffix but handler default is :0 (req.limit||0).
Fixed to :0 so seed matches handler cache key for default requests.
3. USNI and nav warnings seed parsers produced wrong data shapes vs handler
(different field names, missing fields). Converted to warm-ping pattern
(like seed-infra.mjs) to avoid shape divergence.
* fix(seeds): reduce GDELT 429 rate limiting in seed-gdelt-intel
Problems from logs: every topic fetch hits 429, runs take 3-5min,
4th run failed fatally after 12min of cascading retries.
Fixes:
- Increase inter-topic delay: 12s -> 20s (GDELT needs longer cooldown)
- Increase initial backoff: 10s -> 20s, with 15s increments per retry
- Graceful degradation: exhausted retries return empty topic instead of
throwing (prevents withRetry from restarting ALL topics from scratch)
- Align TTL with health.js: 3600s -> 7200s (matches maxStaleMin:120)
- Validation allows partial success (3/6 topics minimum)
Cron interval should also be increased from 30min to 2h on Railway
to match the new 2h TTL.
* fix(seeds): 4 bugs from review - ACLED auth, NOTAM key, infra precedence, curated events
P1: ACLED auth used wrong endpoint (api/acled/token) and env vars (ACLED_KEY).
Fixed to match server/acled-auth.ts: ACLED_EMAIL+ACLED_PASSWORD via /oauth/token,
with ACLED_ACCESS_TOKEN static fallback.
P1: Aviation NOTAM key was aviation:notam-closures:v1, handler reads
aviation:notam:closures:v2. Fixed key to match _shared.ts.
P2: Infra warm-ping had operator precedence bug in nullish coalescing:
(a ?? b) ? c : d instead of a ?? (b ? c : d). Added parens.
P2: Research seed missed curated conferences that the handler appends
(CURATED_EVENTS in list-tech-events.ts). Added same curated events so
seeded data matches what the handler would produce.
* fix(seeds): add seed-meta freshness metadata for all secondary keys
Added writeExtraKeyWithMeta() to _seed-utils.mjs that writes both the
data key and a seed-meta:<key> freshness metadata entry. All secondary
key writes in seed scripts now use this helper so health.js can track
freshness for: energy capacity, FRED series, macro signals, trade
barriers/restrictions/flows/tariffs, aviation news, HAPI summaries,
PizzINT, arXiv categories, HN feeds, tech events, trending repos.
Previously only the primary key per script got seed-meta (via runSeed),
leaving secondary keys operationally invisible to health monitoring.
* fix(seeds): align seed-meta keys with health.js conventions
P1: writeExtraKeyWithMeta wrote seed-meta:<full-cache-key> (e.g.,
seed-meta:economic:macro-signals:v1), but health.js expects normalized
names without version suffixes (seed-meta:economic:macro-signals).
Fixed by stripping trailing :v\d+ from key. Added metaKeyOverride
param for cases needing explicit control.
P1: shipping seed used runSeed('supply-chain', 'shipping-trade', ...)
producing seed-meta:supply-chain:shipping-trade, but health.js expects
seed-meta:supply_chain:shipping. Fixed domain/resource to match.
* fix(seeds): only write seed-meta after successful data key write
writeExtraKey() now returns false on failure. writeExtraKeyWithMeta()
skips seed-meta write when the data write fails, preventing false-positive
health reports for keys like macro-signals and tech-events.
|