2 Commits

Author	SHA1	Message	Date
Elie Habib	8cca8d19e3	feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read (#3385) ... * feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup Prereq for the re-export-share Comtrade seeder (plan 2026-04-24-003), usable by any cohort seeder whose consumer needs bundle-level freshness. Two coupled changes: 1. `_bundle-runner.mjs` injects `BUNDLE_RUN_STARTED_AT_MS` into every spawned child. All siblings in a single bundle run share one value (captured at `runBundle` start, not spawn time). Consumers use this to detect stale peer keys — if a peer's seed-meta predates the current bundle run, fall back to a hard default rather than read a cohort-peer's last-week output. 2. `_seed-utils.mjs::runSeed` registers a `process.once('SIGTERM')` handler that releases the acquired lock and extends existing-data TTL before exiting 143. `_bundle-runner.mjs` sends SIGTERM on section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without this handler the `finally` path never runs on SIGKILL, leaving the 30-min acquireLock reservation in place until its own TTL expires — the next cron tick silently skips the resource. Regression guard memory: `bundle-runner-sigkill-leaks-child-lock` (PR #3128 root cause). Tests added: - bundle-runner env injection (value within run bounds) - sibling sections share the same timestamp (critical for the consumer freshness guard) - runSeed SIGTERM path: exit 143 + cleanup log - process.once contract: second SIGTERM does not re-enter handler * fix(seed): address P1/P2 review findings on SIGTERM + bundle contracts Addresses PR #3384 review findings (todos 256, 257, 259, 260): #256 (P1) — SIGTERM handler narrowed to fetch phase only. Was installed at runSeed entry and armed through every `process.exit` path; could race `emptyDataIsFailure: true` strict-floor exits (IMF-External, WB-bulk) and extend seed-meta TTL when the contract forbids it — silently re-masking 30-day outages. Now the handler is attached immediately before `withRetry(fetchFn)` and removed in a try/finally that covers all fetch-phase exit branches. #257 (P1) — `BUNDLE_RUN_STARTED_AT_MS` now has a first-class helper. Exported `getBundleRunStartedAtMs()` from `_seed-utils.mjs` with JSDoc describing the bundle-freshness contract. Fleet-wide helper so the next consumer seeder imports instead of rediscovering the idiom. #259 (P2) — SIGTERM cleanup runs `Promise.allSettled` on disjoint-key ops (`releaseLock` + `extendExistingTtl`). Serialising compounded Upstash latency during the exact failure mode (Redis degraded) this handler exists to handle, risking breach of the 5s SIGKILL grace. #260 (P2) — `_bundle-runner.mjs` asserts topological order on optional `dependsOn` section field. Throws on unknown-label refs and on deps appearing at a later index. Fleet-wide contract replacing the previous prose-comment ordering guarantee. Tests added/updated: - New: SIGTERM handler removed after fetchFn completes (narrowed-scope contract — post-fetch SIGTERM must NOT trigger TTL extension) - New: dependsOn unknown-label + out-of-order + happy-path (3 tests) Full test suite: 6,866 tests pass (+4 net). * fix(seed): getBundleRunStartedAtMs returns null outside a bundle run Review follow-up: the earlier `Math.floor(Date.now()/1000)*1000` fallback regressed standalone (non-bundle) runs. A consumer seeder invoked manually just after its peer wrote `fetchedAt = (now - 5s)` would see `bundleStartMs = Date.now()`, reject the perfectly-fresh peer envelope as "stale", and fall back to defaults — defeating the point of the peer-read path outside the bundle. Returning null when `BUNDLE_RUN_STARTED_AT_MS` is unset/invalid keeps the freshness gate scoped to its real purpose (across-bundle-tick staleness) and lets standalone runs skip the gate entirely. Consumers check `bundleStartMs != null` before applying the comparison; see the companion `seed-sovereign-wealth.mjs` change on the stacked PR. * test(seed): SIGTERM cleanup test now verifies Redis DEL + EXPIRE calls Greptile review P2 on PR #3384: the existing test only asserted exit code + log line, not that the Redis ops were actually issued. The log claim was ahead of the test. Fixture now logs every Upstash fetch call's shape (EVAL / pipeline- EXPIRE / other) to stderr. Test asserts: - >=1 EVAL op was issued during SIGTERM cleanup (releaseLock Lua script on the lock key) - >=1 pipeline-EXPIRE op was issued (extendExistingTtl on canonical + seed-meta keys) - The EVAL body carries the runSeed-generated runId (proves it's THIS run's release, not a phantom op) - The EXPIRE pipeline touches both the canonicalKey AND the seed-meta key (proves the keys[] array was built correctly including the extraKeys merge path) Full test suite: 6,866 tests pass, typecheck clean. * feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md Motivating case. Before this PR, the SWF `rawMonths` denominator for the `sovereignFiscalBuffer` dimension used GROSS annual imports for every country. For re-export hubs (goods transiting without domestic settlement), this structurally under-reports resilience: UAE's 2023 $941B of imports include $334B of transit flow that never represents domestic consumption. Net imports = gross × (1 − reexport_share). The previous (PR 3A) design flattened a hand-curated YAML into Redis; the YAML shipped empty and never populated, so the correction never applied and the cohort audit showed no movement. Gap #2 (this PR). Two coupled changes to make the correction actually apply: 1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`). Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and `flowCode=M` (imports) per cohort member, compute share = RX/M at the latest co-populated year, clamp to [0.05, 0.95], publish the envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription key never reaches URL/logs/Redis. `maxRecords=250000` cap with truncation detection. Sequential + retry-on-429 with backoff. Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0): ['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200 with zero RX rows — Comtrade doesn't expose RX for those reporters. 2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`). Swaps `loadReexportShareByCountry()` (YAML) for `loadReexportShareFromRedis()` (Redis key written by #1). Guarded by bundle-run freshness: if the sibling Reexport-Share seeder's `seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback to gross imports rather than apply last-month's stale share. Health registries. Both new keys registered in BOTH `api/health.js` SEED_META (60-day alert threshold) and `api/seed-health.js` SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match. Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min worst-case). Ordering preserved: Reexport-Share before Sovereign- Wealth so the SWF seeder reads a freshly-written key in the same cron tick. Deletions. YAML + loader + 7 obsolete loader tests removed; single source of truth is now Comtrade → Redis. Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm) which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed SIGTERM cleanup. This PR's bundle-freshness guard depends on that env variable. Tests (19 new, 7 deleted, +12 net): - Pure math: parseComtradeFlowResponse, computeShareFromFlows, clampShare, declareRecords + credential-leak source scan (15) - Integration (Gap #2 regression guards): SWF seeder loadReexport ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5) - Health registry dual-registry drift guard — scoped to this PR's keys, respecting pre-existing asymmetry (4) - Bundle-ordering + timeout assertions (2) Phase 0 cohort validation committed to plan. Full test suite passes: 6,881 tests. * fix(resilience): address P1/P2 review findings — adopt shared helpers, pin freshness boundary Addresses PR #3385 review findings: #257 (P1) consumer — `seed-sovereign-wealth.mjs` imports the shared `getBundleRunStartedAtMs` helper from `_seed-utils.mjs` (added in the prereq commit) instead of its own `getBundleStartMs`. Single source of truth for the bundle-freshness contract. #258 (P2) — `seed-recovery-reexport-share.mjs` isMain guard uses the canonical `pathToFileURL(process.argv[1]).href === import.meta.url` form instead of basename-suffix matching. Handles symlinks, case- different paths on macOS HFS+, and Windows path separators without string munging. #260 (P2) consumer — Sovereign-Wealth declares `dependsOn: ['Reexport-Share']` in the bundle spec. `_bundle-runner.mjs` (prereq commit) now enforces topological order on load and throws on violation — replaces the previous prose-comment ordering contract. #261 (P2) — added a test to `tests/seed-sovereign-wealth-reads-redis- reexport-share.test.mts` pinning the inclusive-boundary semantic: `fetchedAtMs === bundleStartMs` must be treated as FRESH. Guards against a future refactor to `<=` that would silently reject peers writing at the very first millisecond of the bundle run. Rebased onto updated prereq. Full test suite: 6,886 tests pass (+5 net). * fix(resilience): freshness gate skipped in standalone mode; meta still required Review catch: the previous `bundleStartMs = Date.now()` fallback made standalone/manual `seed-sovereign-wealth.mjs` runs ALWAYS reject any previously-seeded re-export-share meta as "stale" — even when the operator ran the Reexport seeder milliseconds beforehand. Defeated the point of the peer-read path outside the bundle. With `getBundleRunStartedAtMs()` now returning null outside a bundle (companion commit on the prereq branch), the consumer only applies the freshness gate when `bundleStartMs != null`. Standalone runs accept any `fetchedAt` — the operator is responsible for ordering. Two guards survive the change: - Meta MUST exist (absence = peer-outage fail-safe, both modes) - In-bundle: meta MUST be at or after `BUNDLE_RUN_STARTED_AT_MS` Two new tests pin both modes: - standalone: accepts meta written 10 min before this process started - standalone: still rejects missing meta (peer-outage fail-safe survives gate bypass) Rebased onto updated prereq. Full test suite: 6,888 tests (+2 net). * fix(resilience): filter world-aggregate Comtrade rows + skip final-retry sleep Greptile review of PR #3385 flagged two P2s in the Comtrade seeder. Finding #3 (parseComtradeFlowResponse double-count risk): `cmdCode=TOTAL` without a partner filter currently returns only world-aggregate rows in practice — but `parseComtradeFlowResponse` summed every row unconditionally. A future refactor adding per- partner querying would silently double-count (world-aggregate row + partner-level rows for the same year), cutting the derived share in half with no test signal. Fix: explicit `partnerCode ∈ {'0', 0, null/undefined}` filter. Matches current empirical behavior (aggregate-only responses) and makes the construct robust to a future partner-level query. Finding #4 (wasted backoff on final retry): 429 and 5xx branches slept `backoffMs` before `continue`, but on `attempt === RETRY_MAX_ATTEMPTS` the loop condition fails immediately after — the sleep was pure waste. Added early-return (parallel to the existing pattern in the network-error catch branch) so the final attempt exits the retry loop at the first non-success response without extra latency. Tests: - 3 new `parseComtradeFlowResponse` variants: world-only filter, numeric-0 partnerCode shape, rows without partnerCode field - Existing tests updated: the double-count assertion replaced with a "per-partner rows must NOT sum into the world-aggregate total" assertion that pins the new contract Rebased onto updated prereq. Full test suite: 6,890 tests (+2 net).	2026-04-25 00:14:17 +04:00
Elie Habib	184e82cb40	feat(resilience): PR 3A — net-imports denominator for sovereignFiscalBuffer (#3380) ... PR 3A of cohort-audit plan 2026-04-24-002. Construct correction for re-export hubs: the SWF rawMonths denominator was gross imports, which double-counted flow-through trade that never represents domestic consumption. Net-imports fix: rawMonths = aum / (grossImports × (1 − reexportShareOfImports)) × 12 applied to any country in the re-export share manifest. Countries NOT in the manifest get gross imports unchanged (status-quo fallback). Plan acceptance gates — verified synthetically in this PR: Construct invariant. Two synthetic countries, same SWF, same gross imports. A re-exports 60%; B re-exports 0%. Post-fix, A's rawMonths is 2.5× B's (1/(1-0.6) = 2.5). Pinned in tests/resilience-net-imports-denominator.test.mts. SWF-heavy exporter invariant. Country with share ≤ 5%: rawMonths lift < 5% vs baseline (negligible). Pinned. What shipped 1. Re-export share manifest infrastructure. - scripts/shared/reexport-share-manifest.yaml (new, empty) — schema committed; entries populated in follow-up PRs with UNCTAD Handbook citations. - scripts/shared/reexport-share-loader.mjs (new) — loader + strict validator, mirrors swf-manifest-loader.mjs. - scripts/seed-recovery-reexport-share.mjs (new) — publishes resilience:recovery:reexport-share:v1 from manifest. Empty manifest = valid (no countries, no adjustment). 2. SWF seeder uses net-imports denominator. - scripts/seed-sovereign-wealth.mjs exports computeNetImports(gross, share) — pure helper, unit-tested. - Per-country loop: reads manifest, computes denominatorImports, applies to rawMonths math. - Payload records annualImports (gross, audit), denominatorImports (used in math), reexportShareOfImports (provenance). - Summary log reports which countries had a net-imports adjustment applied with source year. 3. Bundle wiring. - Reexport-Share runs BEFORE Sovereign-Wealth in the recovery bundle so the SWF seeder reads fresh re-export data in the same cron tick. - tests/seed-bundle-resilience-recovery.test.mjs expected-entries updated (6 → 7) with ordering preservation. 4. Cache-prefix bump (per cache-prefix-bump-propagation-scope skill). - RESILIENCE_SCORE_CACHE_PREFIX: v11 → v12 - RESILIENCE_RANKING_CACHE_KEY: v11 → v12 - RESILIENCE_HISTORY_KEY_PREFIX: v6 → v7 (history rotation prevents 30-day rolling window from mixing pre/post-fix scores and manufacturing false "falling" trends on deploy day). - Source of truth: server/worldmonitor/resilience/v1/_shared.ts - Mirrored in: scripts/seed-resilience-scores.mjs, scripts/validate-resilience-correlation.mjs, scripts/backtest-resilience-outcomes.mjs, scripts/validate-resilience-backtest.mjs, scripts/benchmark-resilience-external.mjs, api/health.js - Test literals bumped in 4 test files (26 line edits). - EXTENDED tests/resilience-cache-keys-health-sync.test.mts with a parity pass that reads every known mirror file and asserts both (a) canonical prefix present AND (b) no stale v<older> literals in non-comment code. Found one legacy log-line that still referenced v9 (scripts/seed-resilience-scores.mjs:342) and refactored it to use the RESILIENCE_RANKING_CACHE_KEY constant so future bumps self-update. Explicitly NOT in this PR - liquidReserveAdequacy denominator fix. The plan's PR 3A wording mentions both dims, but the RESERVES ratio (WB FI.RES.TOTL.MO) is a PRE-COMPUTED WB series; applying a post-hoc net-imports adjustment mixes WB's denominator year with our manifest-year, and the math change belongs in PR 3B (unified liquidity) where the α calibration is explicit. This PR stays scoped to sovereignFiscalBuffer. - Live re-export share entries. The manifest ships EMPTY in this PR; entries with UNCTAD citations are one-per-PR follow-ups so each figure is individually auditable. Verified - tests/resilience-net-imports-denominator.test.mts — 9 pass (construct contract: 2.5× ratio gate, monotonicity, boundary rejections, backward-compat on missing manifest entry, cohort-proportionality, SWF-heavy-exporter-unchanged) - tests/reexport-share-loader.test.mts — 7 pass (committed-manifest shape + 6 schema-violation rejections) - tests/resilience-cache-keys-health-sync.test.mts — 5 pass (existing 3 + 2 new parity checks across all mirror files) - tests/seed-bundle-resilience-recovery.test.mjs — 17 pass (expected entries bumped to 7) - npm run test:data — 6714 pass / 0 fail - npm run typecheck / typecheck:api — green - npm run lint / lint:md — clean Deployment notes Score + ranking + history cache prefixes all bump in the same deploy. Per established v10→v11 precedent (and the cache-prefix-bump- propagation-scope skill): - Score / ranking: 6h TTL — the new prefix populates via the Railway resilience-scores cron within one tick. - History: 30d ring — the v7 ring starts empty; the first 30 days post-deploy lack baseline points, so trend / change30d will read as "no change" until v7 accumulates a window. - Legacy v11 keys can be deleted from Redis at any time post-deploy (no reader references them). Leaving them in place costs storage but does no harm.	2026-04-24 18:14:04 +04:00