Files
worldmonitor/tests/seed-sovereign-wealth-reads-redis-reexport-share.test.mts
Elie Habib 8cca8d19e3 feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read (#3385)
* feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup

Prereq for the re-export-share Comtrade seeder (plan 2026-04-24-003),
usable by any cohort seeder whose consumer needs bundle-level freshness.

Two coupled changes:

1. `_bundle-runner.mjs` injects `BUNDLE_RUN_STARTED_AT_MS` into every
   spawned child. All siblings in a single bundle run share one value
   (captured at `runBundle` start, not spawn time). Consumers use this
   to detect stale peer keys — if a peer's seed-meta predates the
   current bundle run, fall back to a hard default rather than read
   a cohort-peer's last-week output.

2. `_seed-utils.mjs::runSeed` registers a `process.once('SIGTERM')`
   handler that releases the acquired lock and extends existing-data
   TTL before exiting 143. `_bundle-runner.mjs` sends SIGTERM on
   section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without
   this handler the `finally` path never runs on SIGKILL, leaving
   the 30-min acquireLock reservation in place until its own TTL
   expires — the next cron tick silently skips the resource.

Regression guard memory: `bundle-runner-sigkill-leaks-child-lock` (PR
#3128 root cause).

Tests added:
- bundle-runner env injection (value within run bounds)
- sibling sections share the same timestamp (critical for the
  consumer freshness guard)
- runSeed SIGTERM path: exit 143 + cleanup log
- process.once contract: second SIGTERM does not re-enter handler

* fix(seed): address P1/P2 review findings on SIGTERM + bundle contracts

Addresses PR #3384 review findings (todos 256, 257, 259, 260):

#256 (P1) — SIGTERM handler narrowed to fetch phase only. Was installed
at runSeed entry and armed through every `process.exit` path; could
race `emptyDataIsFailure: true` strict-floor exits (IMF-External,
WB-bulk) and extend seed-meta TTL when the contract forbids it —
silently re-masking 30-day outages. Now the handler is attached
immediately before `withRetry(fetchFn)` and removed in a try/finally
that covers all fetch-phase exit branches.

#257 (P1) — `BUNDLE_RUN_STARTED_AT_MS` now has a first-class helper.
Exported `getBundleRunStartedAtMs()` from `_seed-utils.mjs` with JSDoc
describing the bundle-freshness contract. Fleet-wide helper so the
next consumer seeder imports instead of rediscovering the idiom.

#259 (P2) — SIGTERM cleanup runs `Promise.allSettled` on disjoint-key
ops (`releaseLock` + `extendExistingTtl`). Serialising compounded
Upstash latency during the exact failure mode (Redis degraded) this
handler exists to handle, risking breach of the 5s SIGKILL grace.

#260 (P2) — `_bundle-runner.mjs` asserts topological order on
optional `dependsOn` section field. Throws on unknown-label refs and
on deps appearing at a later index. Fleet-wide contract replacing
the previous prose-comment ordering guarantee.

Tests added/updated:
- New: SIGTERM handler removed after fetchFn completes (narrowed-scope
  contract — post-fetch SIGTERM must NOT trigger TTL extension)
- New: dependsOn unknown-label + out-of-order + happy-path (3 tests)

Full test suite: 6,866 tests pass (+4 net).

* fix(seed): getBundleRunStartedAtMs returns null outside a bundle run

Review follow-up: the earlier `Math.floor(Date.now()/1000)*1000` fallback
regressed standalone (non-bundle) runs. A consumer seeder invoked
manually just after its peer wrote `fetchedAt = (now - 5s)` would see
`bundleStartMs = Date.now()`, reject the perfectly-fresh peer envelope
as "stale", and fall back to defaults — defeating the point of the
peer-read path outside the bundle.

Returning null when `BUNDLE_RUN_STARTED_AT_MS` is unset/invalid keeps
the freshness gate scoped to its real purpose (across-bundle-tick
staleness) and lets standalone runs skip the gate entirely. Consumers
check `bundleStartMs != null` before applying the comparison; see the
companion `seed-sovereign-wealth.mjs` change on the stacked PR.

* test(seed): SIGTERM cleanup test now verifies Redis DEL + EXPIRE calls

Greptile review P2 on PR #3384: the existing test only asserted exit
code + log line, not that the Redis ops were actually issued. The
log claim was ahead of the test.

Fixture now logs every Upstash fetch call's shape (EVAL / pipeline-
EXPIRE / other) to stderr. Test asserts:

- >=1 EVAL op was issued during SIGTERM cleanup (releaseLock Lua
  script on the lock key)
- >=1 pipeline-EXPIRE op was issued (extendExistingTtl on canonical
  + seed-meta keys)
- The EVAL body carries the runSeed-generated runId (proves it's
  THIS run's release, not a phantom op)
- The EXPIRE pipeline touches both the canonicalKey AND the
  seed-meta key (proves the keys[] array was built correctly
  including the extraKeys merge path)

Full test suite: 6,866 tests pass, typecheck clean.

* feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read

Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md

Motivating case. Before this PR, the SWF `rawMonths` denominator for
the `sovereignFiscalBuffer` dimension used GROSS annual imports for
every country. For re-export hubs (goods transiting without domestic
settlement), this structurally under-reports resilience: UAE's 2023
$941B of imports include $334B of transit flow that never represents
domestic consumption. Net imports = gross × (1 − reexport_share).

The previous (PR 3A) design flattened a hand-curated YAML into Redis;
the YAML shipped empty and never populated, so the correction never
applied and the cohort audit showed no movement.

Gap #2 (this PR). Two coupled changes to make the correction actually
apply:

1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`).
   Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and
   `flowCode=M` (imports) per cohort member, compute share = RX/M at
   the latest co-populated year, clamp to [0.05, 0.95], publish the
   envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription
   key never reaches URL/logs/Redis. `maxRecords=250000` cap with
   truncation detection. Sequential + retry-on-429 with backoff.

   Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0):
   ['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200
   with zero RX rows — Comtrade doesn't expose RX for those reporters.

2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`).
   Swaps `loadReexportShareByCountry()` (YAML) for
   `loadReexportShareFromRedis()` (Redis key written by #1). Guarded
   by bundle-run freshness: if the sibling Reexport-Share seeder's
   `seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the
   prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback
   to gross imports rather than apply last-month's stale share.

Health registries. Both new keys registered in BOTH `api/health.js`
SEED_META (60-day alert threshold) and `api/seed-health.js`
SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match.

Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share
timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min
worst-case). Ordering preserved: Reexport-Share before Sovereign-
Wealth so the SWF seeder reads a freshly-written key in the same
cron tick.

Deletions. YAML + loader + 7 obsolete loader tests removed; single
source of truth is now Comtrade → Redis.

Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm)
which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed
SIGTERM cleanup. This PR's bundle-freshness guard depends on
that env variable.

Tests (19 new, 7 deleted, +12 net):
- Pure math: parseComtradeFlowResponse, computeShareFromFlows,
  clampShare, declareRecords + credential-leak source scan (15)
- Integration (Gap #2 regression guards): SWF seeder loadReexport
  ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5)
- Health registry dual-registry drift guard — scoped to this PR's
  keys, respecting pre-existing asymmetry (4)
- Bundle-ordering + timeout assertions (2)

Phase 0 cohort validation committed to plan. Full test suite
passes: 6,881 tests.

* fix(resilience): address P1/P2 review findings — adopt shared helpers, pin freshness boundary

Addresses PR #3385 review findings:

#257 (P1) consumer — `seed-sovereign-wealth.mjs` imports the shared
`getBundleRunStartedAtMs` helper from `_seed-utils.mjs` (added in the
prereq commit) instead of its own `getBundleStartMs`. Single source of
truth for the bundle-freshness contract.

#258 (P2) — `seed-recovery-reexport-share.mjs` isMain guard uses the
canonical `pathToFileURL(process.argv[1]).href === import.meta.url`
form instead of basename-suffix matching. Handles symlinks, case-
different paths on macOS HFS+, and Windows path separators without
string munging.

#260 (P2) consumer — Sovereign-Wealth declares `dependsOn:
['Reexport-Share']` in the bundle spec. `_bundle-runner.mjs` (prereq
commit) now enforces topological order on load and throws on
violation — replaces the previous prose-comment ordering contract.

#261 (P2) — added a test to `tests/seed-sovereign-wealth-reads-redis-
reexport-share.test.mts` pinning the inclusive-boundary semantic:
`fetchedAtMs === bundleStartMs` must be treated as FRESH. Guards
against a future refactor to `<=` that would silently reject peers
writing at the very first millisecond of the bundle run.

Rebased onto updated prereq. Full test suite: 6,886 tests pass (+5 net).

* fix(resilience): freshness gate skipped in standalone mode; meta still required

Review catch: the previous `bundleStartMs = Date.now()` fallback made
standalone/manual `seed-sovereign-wealth.mjs` runs ALWAYS reject any
previously-seeded re-export-share meta as "stale" — even when the
operator ran the Reexport seeder milliseconds beforehand. Defeated
the point of the peer-read path outside the bundle.

With `getBundleRunStartedAtMs()` now returning null outside a bundle
(companion commit on the prereq branch), the consumer only applies
the freshness gate when `bundleStartMs != null`. Standalone runs
accept any `fetchedAt` — the operator is responsible for ordering.

Two guards survive the change:
- Meta MUST exist (absence = peer-outage fail-safe, both modes)
- In-bundle: meta MUST be at or after `BUNDLE_RUN_STARTED_AT_MS`

Two new tests pin both modes:
- standalone: accepts meta written 10 min before this process started
- standalone: still rejects missing meta (peer-outage fail-safe
  survives gate bypass)

Rebased onto updated prereq. Full test suite: 6,888 tests (+2 net).

* fix(resilience): filter world-aggregate Comtrade rows + skip final-retry sleep

Greptile review of PR #3385 flagged two P2s in the Comtrade seeder.

Finding #3 (parseComtradeFlowResponse double-count risk):
`cmdCode=TOTAL` without a partner filter currently returns only
world-aggregate rows in practice — but `parseComtradeFlowResponse`
summed every row unconditionally. A future refactor adding per-
partner querying would silently double-count (world-aggregate row +
partner-level rows for the same year), cutting the derived share in
half with no test signal.

Fix: explicit `partnerCode ∈ {'0', 0, null/undefined}` filter. Matches
current empirical behavior (aggregate-only responses) and makes the
construct robust to a future partner-level query.

Finding #4 (wasted backoff on final retry):
429 and 5xx branches slept `backoffMs` before `continue`, but on
`attempt === RETRY_MAX_ATTEMPTS` the loop condition fails immediately
after — the sleep was pure waste. Added early-return (parallel to the
existing pattern in the network-error catch branch) so the final
attempt exits the retry loop at the first non-success response
without extra latency.

Tests:
- 3 new `parseComtradeFlowResponse` variants: world-only filter,
  numeric-0 partnerCode shape, rows without partnerCode field
- Existing tests updated: the double-count assertion replaced with
  a "per-partner rows must NOT sum into the world-aggregate total"
  assertion that pins the new contract

Rebased onto updated prereq. Full test suite: 6,890 tests (+2 net).
2026-04-25 00:14:17 +04:00

206 lines
9.0 KiB
TypeScript

// Regression guards for Gap #2: the SWF seeder MUST read the re-export
// share map from Redis (populated by the Comtrade seeder that runs
// immediately before it in the resilience-recovery bundle), NOT from
// the static YAML that was deleted in this PR.
//
// These four tests defend against the exact failure mode that surfaced
// in the 2026-04-24 cohort audit: SWF scores didn't move after the
// Comtrade work shipped because the SWF seeder was still reading a
// (now-absent) YAML. See plan 2026-04-24-003 §Phase 3 tests 7-10.
import assert from 'node:assert/strict';
import { beforeEach, afterEach, describe, it } from 'node:test';
import { loadReexportShareFromRedis } from '../scripts/seed-sovereign-wealth.mjs';
const REEXPORT_SHARE_KEY = 'resilience:recovery:reexport-share:v1';
const REEXPORT_SHARE_META_KEY = 'seed-meta:resilience:recovery:reexport-share';
const ORIGINAL_FETCH = globalThis.fetch;
const ORIGINAL_ENV = {
UPSTASH_REDIS_REST_URL: process.env.UPSTASH_REDIS_REST_URL,
UPSTASH_REDIS_REST_TOKEN: process.env.UPSTASH_REDIS_REST_TOKEN,
BUNDLE_RUN_STARTED_AT_MS: process.env.BUNDLE_RUN_STARTED_AT_MS,
};
let keyStore: Record<string, unknown>;
beforeEach(() => {
process.env.UPSTASH_REDIS_REST_URL = 'https://fake-upstash.example.com';
process.env.UPSTASH_REDIS_REST_TOKEN = 'fake-token';
keyStore = {};
// readSeedSnapshot issues `GET /get/<encodeURIComponent(key)>`.
// Stub: look up keyStore, return `{ result: JSON.stringify(value) }`
// or `{ result: null }` for absent keys.
globalThis.fetch = async (url) => {
const s = String(url);
const match = s.match(/\/get\/(.+)$/);
if (!match) {
return new Response(JSON.stringify({ result: null }), { status: 200 });
}
const key = decodeURIComponent(match[1]);
const value = keyStore[key];
const body = value !== undefined
? JSON.stringify({ result: JSON.stringify(value) })
: JSON.stringify({ result: null });
return new Response(body, { status: 200 });
};
});
afterEach(() => {
globalThis.fetch = ORIGINAL_FETCH;
for (const k of Object.keys(ORIGINAL_ENV) as Array<keyof typeof ORIGINAL_ENV>) {
const v = ORIGINAL_ENV[k];
if (v == null) delete process.env[k];
else process.env[k] = v;
}
});
describe('loadReexportShareFromRedis — Gap #2 regression guards', () => {
it('reads the Redis key and returns a Map of ISO2 → share when bundle-fresh', async () => {
const bundleStart = 1_700_000_000_000;
process.env.BUNDLE_RUN_STARTED_AT_MS = String(bundleStart);
keyStore[REEXPORT_SHARE_KEY] = {
manifestVersion: 2,
countries: {
AE: { reexportShareOfImports: 0.4, year: 2023, sources: ['https://comtrade.example/AE'] },
PA: { reexportShareOfImports: 0.07, year: 2024, sources: ['https://comtrade.example/PA'] },
},
};
keyStore[REEXPORT_SHARE_META_KEY] = {
fetchedAt: bundleStart + 1000, // 1s AFTER bundle start — fresh
};
const map = await loadReexportShareFromRedis();
assert.equal(map.size, 2);
assert.equal(map.get('AE')?.reexportShareOfImports, 0.4);
assert.equal(map.get('PA')?.reexportShareOfImports, 0.07);
assert.equal(map.get('AE')?.year, 2023);
assert.deepEqual(map.get('AE')?.sources, ['https://comtrade.example/AE']);
});
it('absent canonical key → empty map (status-quo gross-imports fallback)', async () => {
process.env.BUNDLE_RUN_STARTED_AT_MS = String(1_700_000_000_000);
// keyStore is empty — readSeedSnapshot returns null.
const map = await loadReexportShareFromRedis();
assert.equal(map.size, 0);
});
it('malformed entry (share is string) → skip that country, others unaffected', async () => {
const bundleStart = 1_700_000_000_000;
process.env.BUNDLE_RUN_STARTED_AT_MS = String(bundleStart);
keyStore[REEXPORT_SHARE_KEY] = {
manifestVersion: 2,
countries: {
AE: { reexportShareOfImports: 0.4, year: 2023 },
XX: { reexportShareOfImports: 'not-a-number' }, // type-wrong
YY: { reexportShareOfImports: 1.5 }, // > 0.95 cap
ZZ: { reexportShareOfImports: -0.1 }, // negative
AA: { reexportShareOfImports: NaN }, // NaN
},
};
keyStore[REEXPORT_SHARE_META_KEY] = { fetchedAt: bundleStart + 1000 };
const map = await loadReexportShareFromRedis();
assert.equal(map.size, 1);
assert.equal(map.get('AE')?.reexportShareOfImports, 0.4);
assert.equal(map.has('XX'), false);
assert.equal(map.has('YY'), false);
assert.equal(map.has('ZZ'), false);
assert.equal(map.has('AA'), false);
});
it('stale seed-meta (fetchedAt < bundle start) → empty map (hard fail-safe)', async () => {
const bundleStart = 1_700_000_000_000;
process.env.BUNDLE_RUN_STARTED_AT_MS = String(bundleStart);
keyStore[REEXPORT_SHARE_KEY] = {
manifestVersion: 2,
countries: {
AE: { reexportShareOfImports: 0.4, year: 2023 },
},
};
// fetchedAt is 1 hour BEFORE bundle start — previous bundle tick.
// The SWF seeder MUST NOT apply last-month's share to this month's
// data. Hard fallback: return empty, everyone uses gross imports.
keyStore[REEXPORT_SHARE_META_KEY] = { fetchedAt: bundleStart - 3_600_000 };
const map = await loadReexportShareFromRedis();
assert.equal(map.size, 0,
'stale seed-meta must produce empty map, NOT pass stale shares through');
});
it('missing seed-meta key → empty map (outage fail-safe)', async () => {
const bundleStart = 1_700_000_000_000;
process.env.BUNDLE_RUN_STARTED_AT_MS = String(bundleStart);
keyStore[REEXPORT_SHARE_KEY] = {
manifestVersion: 2,
countries: { AE: { reexportShareOfImports: 0.4, year: 2023 } },
};
// Meta is absent — seeder produced a data envelope but seed-meta
// write failed or races. Safer to treat as "did not run this
// bundle" than to trust the data-key alone.
const map = await loadReexportShareFromRedis();
assert.equal(map.size, 0);
});
it('standalone mode (BUNDLE_RUN_STARTED_AT_MS unset) skips the freshness gate', async () => {
// Regression guard for the standalone-regression bug: when a seeder
// runs manually (operator invocation, not bundle-runner), the env
// var is absent. Earlier designs fell back to `Date.now()` which
// rejected any previously-seeded peer envelope as "stale" — even
// when the operator ran the Reexport seeder milliseconds beforehand.
// The fix: getBundleRunStartedAtMs() returns null outside a bundle;
// the consumer skips the freshness gate but still requires meta
// existence (peer outage still fails safely).
delete process.env.BUNDLE_RUN_STARTED_AT_MS;
keyStore[REEXPORT_SHARE_KEY] = {
manifestVersion: 2,
countries: { AE: { reexportShareOfImports: 0.35, year: 2023 } },
};
// Meta written 10 MINUTES ago — rejected under the old `Date.now()`
// fallback, accepted under the null-return + skip-gate fix.
keyStore[REEXPORT_SHARE_META_KEY] = { fetchedAt: Date.now() - 600_000 };
const map = await loadReexportShareFromRedis();
assert.equal(map.size, 1,
'standalone: operator-seeded peer data must be accepted even if written before this process started');
assert.equal(map.get('AE')?.reexportShareOfImports, 0.35);
});
it('standalone mode still rejects missing meta (peer outage still fails safely)', async () => {
// Even in standalone mode, meta absence means "peer never ran" —
// must fall back to gross imports, don't apply potentially stale
// shares from a data key that has no freshness signal.
delete process.env.BUNDLE_RUN_STARTED_AT_MS;
keyStore[REEXPORT_SHARE_KEY] = {
manifestVersion: 2,
countries: { AE: { reexportShareOfImports: 0.35, year: 2023 } },
};
// No meta key written — peer outage.
const map = await loadReexportShareFromRedis();
assert.equal(map.size, 0,
'standalone: absent meta must still fall back (peer-outage fail-safe survives gate bypass)');
});
it('fetchedAtMs === bundleStartMs passes (inclusive freshness boundary)', async () => {
// The freshness check uses strict-less-than: `fetchedAt < bundleStart`.
// Exact equality is treated as FRESH. This pins the inclusive-boundary
// semantic so a future refactor to `<=` fails this test loudly
// instead of silently rejecting a peer that wrote at the very first
// millisecond of the bundle run (theoretically possible on a fast
// host where t0 and the peer's fetchedAt align on the same ms).
const bundleStart = 1_700_000_000_000;
process.env.BUNDLE_RUN_STARTED_AT_MS = String(bundleStart);
keyStore[REEXPORT_SHARE_KEY] = {
manifestVersion: 2,
countries: { AE: { reexportShareOfImports: 0.4, year: 2023 } },
};
keyStore[REEXPORT_SHARE_META_KEY] = { fetchedAt: bundleStart }; // EXACT equality
const map = await loadReexportShareFromRedis();
assert.equal(map.size, 1,
'equality at the freshness boundary must be treated as FRESH');
assert.equal(map.get('AE')?.reexportShareOfImports, 0.4);
});
});