mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
cd5ed0d183
* feat(seeds): BIS DSR + property prices (2 of 7) Ships 2 of 7 BIS dataflows flagged as genuinely new signals in #3026 — the rest are redundant with IMF/WB or are low-fit global aggregates. New seeder: scripts/seed-bis-extended.mjs - WS_DSR household debt service ratio (% income, quarterly) - WS_SPP residential property prices (real index, quarterly) - WS_CPP commercial property prices (real index, quarterly) Gold-standard pattern: atomic publish + writeExtraKey for extras, retry on missing startPeriod, TTL = 3 days (3× 12h cron), runSeed drives seed-meta:economic:bis-extended. Series selection scores dimension matches (PP_VALUATION=R / UNIT_MEASURE=628 for property, DSR_BORROWERS=P / DSR_ADJUST=A for DSR), then falls back to observation count. Wired into: - bootstrap (slow tier) + cache-keys.ts - api/health.js (STANDALONE_KEYS + SEED_META, maxStaleMin = 24h) - api/mcp.ts get_economic_data tool (_cacheKeys + _freshnessChecks) - resilience macroFiscal: new householdDebtService sub-metric (weight 0.05, currentAccountPct rebalanced 0.3 → 0.25) - Housing Cycle tile on CountryDeepDivePanel (Economic Indicators card) with euro-area (XM) fallback for EU member states - seed-bundle-macro Railway cron (BIS-Extended, 12h interval) Tests: tests/bis-extended-seed.test.mjs covers CSV parsing, series selection, quarter math + YoY. Updated resilience golden-value tests for the macroFiscal weight rebalance. Closes #3026 https://claude.ai/code/session_01DDo39mPD9N2fNHtUntHDqN * fix(resilience): unblock PR #3048 on #3046 stack - rebase onto #3046; final macroFiscal weights: govRevenue 0.40, currentAccount 0.20, debtGrowth 0.20, unemployment 0.15, householdDebtService 0.05 (=1.00) - add updateHousingCycle? stub to CountryBriefPanel interface so country-intel dispatch typechecks - add HR to EURO_AREA fallback set (joined euro 2023-01-01) - seed-bis-extended: extend SPP/CPP TTLs when DSR fetch returns empty so the rejected publish does not silently expire the still-good property keys - update resilience goldens for the 5-sub-metric macroFiscal blend * fix(country-brief): housing tile renders em-dash for null change values The new Housing cycle tile used `?? 0` to default qoqChange/yoyChange/change when missing, fabricating a flat "0.0%" label (with positive-trend styling) for countries with no prior comparable period. Fetch path and builders correctly return null; the panel was coercing it. formatPctTrend now accepts null|undefined and returns an em-dash, matching how other cards surface unavailable metrics. Drop the `?? 0` fallbacks at the three housing call sites. * fix(seed-health): register economic:bis-extended seed-meta monitoring 12h Railway cron writes seed-meta:economic:bis-extended but it was missing from SEED_DOMAINS, so /api/seed-health never reported its freshness. intervalMin=720 matches maxStaleMin/2 (1440/2) from api/health.js. * fix(seed-bis-extended): decouple DSR/SPP/CPP so one fetch failure doesn't block the others Previously validate() required data.entries.length > 0 on the DSR slice after publishTransform pulled it out of the aggregate payload. If WS_DSR fetch failed but WS_SPP / WS_CPP succeeded, validate() rejected the publish → afterPublish() never ran → fresh SPP/CPP data was silently discarded and only the old snapshots got a TTL bump. This treats the three datasets as independent: - SPP and CPP are now published (or have their existing TTLs extended) as side-effects of fetchAll(), per-dataset. A failure in one never affects the others. - DSR continues to flow through runSeed's canonical-key path. When DSR is empty, publishTransform yields { entries: [] } so atomicPublish skips the canonical write (preserving the old DSR snapshot); runSeed's skipped branch extends its TTL and refreshes seed-meta. Shape B (one runSeed call, semantics changed) chosen over Shape A (three sequential runSeed calls) because runSeed owns the lock + process.exit lifecycle and can't be safely called three times in a row, and Shape B keeps the single aggregate seed-meta:economic:bis-extended key that health.js already monitors. Tests cover both failure modes: - DSR empty + SPP/CPP healthy → SPP/CPP written, DSR TTL extended - DSR healthy + SPP/CPP empty → DSR written, SPP/CPP TTLs extended * fix(health): per-dataset seed-meta for BIS DSR/SPP/CPP Health was pointing bisDsr / bisPropertyResidential / bisPropertyCommercial at the shared seed-meta:economic:bis-extended key, which runSeed refreshes on every run (including its validation-failed "skipped" branch). A DSR-only outage therefore left bisDsr reporting fresh in api/health.js while the resilience scorer consumed stale/missing economic:bis:dsr:v1 data. Write a dedicated seed-meta key per dataset ONLY when that dataset actually published fresh entries. The aggregate bis-extended key stays as a "seeder ran" signal in api/seed-health.js. * fix(seed-bis-extended): write DSR seed-meta only after atomicPublish succeeds Previously fetchAll() wrote seed-meta:economic:bis-dsr inline before runSeed/atomicPublish ran. If atomicPublish then failed (Redis hiccup, validate rejection, etc.), seed-meta was already bumped — health would report DSR fresh while the canonical key was stale. Move the DSR seed-meta write into a dsrAfterPublish callback passed to runSeed via the existing afterPublish hook, which fires only after a successful canonical publish. SPP/CPP paths already used this ordering inside publishDatasetIndependently; this brings DSR in line. Adds a regression test exercising dsrAfterPublish with mocked Upstash: populated DSR -> single SET on seed-meta key; null/empty DSR -> zero Redis calls. * fix(resilience): per-dataset BIS seed-meta keys in freshness overrides SOURCE_KEY_META_OVERRIDES previously collapsed economic:bis:dsr:v1 and both property-* sourceKeys onto the aggregate seed-meta:economic:bis-extended key. api/health.js (SEED_META) writes per-dataset keys (seed-meta:economic:bis-dsr / bis-property-residential / bis-property-commercial), so a DSR-only outage showed stale in /api/health but the resilience dimension freshness code still reported macroFiscal inputs as fresh. Map each BIS sourceKey to its dedicated seed-meta key to match health.js. The aggregate bis-extended key is still written by the seeder and read by api/seed-health.js as a "seeder ran" signal, so it is retained upstream. * fix(bis): prefer households in DSR + per-dataset freshness in MCP Greptile review catches on #3048: 1. buildDsr() was selecting DSR_BORROWERS='P' (private non-financial) while the UI labels it "Household DSR" and resilience scoring uses it as `householdDebtService`. Changed to 'H' (households). Countries without an H series now get dropped rather than silently mislabeled. 2. api/mcp.ts get_economic_data still read only the aggregate seed-meta:economic:bis-extended for freshness. If DSR goes stale while SPP/CPP keep publishing, MCP would report the BIS block as fresh even though one of its returned keys is stale. Swapped to the three per-dataset seed-meta keys (bis-dsr, bis-property-residential, bis-property-commercial), matching the fix already applied to /api/health and the resilience dimension-freshness pipeline. --------- Co-authored-by: Claude <noreply@anthropic.com>
16 lines
1.5 KiB
JavaScript
16 lines
1.5 KiB
JavaScript
#!/usr/bin/env node
|
|
import { runBundle, HOUR, DAY } from './_bundle-runner.mjs';
|
|
|
|
await runBundle('macro', [
|
|
{ label: 'BIS-Data', script: 'seed-bis-data.mjs', seedMetaKey: 'economic:bis', intervalMs: 12 * HOUR, timeoutMs: 300_000 },
|
|
{ label: 'BIS-Extended', script: 'seed-bis-extended.mjs', seedMetaKey: 'economic:bis-extended', intervalMs: 12 * HOUR, timeoutMs: 300_000 },
|
|
{ label: 'BLS-Series', script: 'seed-bls-series.mjs', seedMetaKey: 'economic:bls-series', intervalMs: DAY, timeoutMs: 120_000 },
|
|
{ label: 'Eurostat', script: 'seed-eurostat-country-data.mjs', seedMetaKey: 'economic:eurostat-country-data', intervalMs: DAY, timeoutMs: 300_000 },
|
|
{ label: 'Eurostat-HousePrices', script: 'seed-eurostat-house-prices.mjs', seedMetaKey: 'economic:eurostat-house-prices', intervalMs: 7 * DAY, timeoutMs: 300_000 },
|
|
{ label: 'Eurostat-GovDebtQ', script: 'seed-eurostat-gov-debt-q.mjs', seedMetaKey: 'economic:eurostat-gov-debt-q', intervalMs: 2 * DAY, timeoutMs: 300_000 },
|
|
{ label: 'Eurostat-IndProd', script: 'seed-eurostat-industrial-production.mjs', seedMetaKey: 'economic:eurostat-industrial-production', intervalMs: DAY, timeoutMs: 300_000 },
|
|
{ label: 'IMF-Macro', script: 'seed-imf-macro.mjs', seedMetaKey: 'economic:imf-macro', intervalMs: 30 * DAY, timeoutMs: 300_000 },
|
|
{ label: 'National-Debt', script: 'seed-national-debt.mjs', seedMetaKey: 'economic:national-debt', intervalMs: 30 * DAY, timeoutMs: 300_000 },
|
|
{ label: 'FAO-FFPI', script: 'seed-fao-food-price-index.mjs', seedMetaKey: 'economic:fao-ffpi', intervalMs: DAY, timeoutMs: 120_000 },
|
|
]);
|