mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
fix(resilience): fail closed on missing v2 energy seeds + health CRIT on absent inputs (#3363)
* fix(resilience): fail closed on missing v2 energy seeds + health CRIT on absent inputs PR #3289 shipped the v2 energy construct behind RESILIENCE_ENERGY_V2_ENABLED (default false). Audit on 2026-04-24 after the user flagged "AE only moved 1.49 points — we added nuclear credit, we should see more" revealed two safety gaps that made a future flag flip unsafe: 1. scoreEnergyV2 silently fell back to IMPUTE when any of its three required Redis seeds (low-carbon-generation, fossil-electricity-share, power-losses) was null. A future operator flipping the flag with seeds absent would produce fabricated-looking numbers for every country with zero operator signal. 2. api/health.js had those three seed labels in BOTH SEED_META (CRIT on missing) AND ON_DEMAND_KEYS (which demotes CRIT to WARN). The demotion won. Health has been reporting WARNING on a scorer dependency that has been 100% missing since PR #3289 merged — no paging trail existed. Changes: server/worldmonitor/resilience/v1/_dimension-scorers.ts - Add ResilienceConfigurationError with missingKeys[] payload. - scoreEnergy: preflight the three v2 seeds when flag=true. Throw ResilienceConfigurationError listing the specific absent keys. - scoreAllDimensions: wrap per-dimension dispatch in try/catch so a thrown ResilienceConfigurationError routes to the source-failure shape (imputationClass='source-failure', coverage=0) for that ONE dimension — country keeps scoring other dims normally. Log once per country-dimension pair so the gap is audit-traceable. api/health.js - Remove lowCarbonGeneration / fossilElectricityShare / powerLosses from ON_DEMAND_KEYS. They stay in BOOTSTRAP_KEYS + SEED_META. - Replace the transitional comment with a hard "do NOT add these back" note pointing at the scorer's fail-closed gate. tests/resilience-energy-v2.test.mts - New test: flag on + ALL three seeds missing → throws ResilienceConfigurationError naming all three keys. - New test: flag on + only one seed missing → throws naming ONLY the missing key (operator-clarity guard). - New test: flag on + all seeds present → v2 runs normally. - Update the file-level invariant comment to reflect the new fail-closed contract (replacing the prior "degrade gracefully" wording that codified the silent-IMPUTE bug). - Note: fixture's `??` fallbacks coerce null-overrides into real data, so the preflight tests use a direct-reader helper. docs/methodology/country-resilience-index.mdx - New "Fail-closed semantics" paragraph in the v2 Energy section documenting the throw + source-failure + health-CRIT contract. Non-goals (intentional): - This PR does NOT flip RESILIENCE_ENERGY_V2_ENABLED. - This PR does NOT provision seed-bundle-resilience-energy-v2 on Railway. - This PR does NOT touch RESILIENCE_PILLAR_COMBINE_ENABLED. Operational effect post-merge: - /api/health flips from WARNING → CRITICAL on the three v2 seed-meta entries. That is the intended alarm; it reveals that the Railway bundle was never provisioned. - scoreEnergy behavior with flag=false is unchanged (legacy path). - scoreEnergy behavior with flag=true + seeds present is unchanged. - scoreEnergy behavior with flag=true + seeds absent changes from "silently IMPUTE all 217 countries" to "source-failure on the energy dim for every country, visible in widget + API response". Tests: 511/511 resilience-* pass. Biome clean. Lint:md clean. Related plan: docs/plans/2026-04-24-001-fix-resilience-v2-fail-closed-on-missing-seeds-plan.md * docs(resilience): scrub stale ON_DEMAND_KEYS references for v2 energy seeds Greptile P2 on PR #3363: four stale references implied the three v2 energy seeds were still gated as ON_DEMAND_KEYS (WARN-on-missing) even though this PR's api/health.js change removed them (now strict SEED_META = CRIT on missing). Scrubbing each: - api/health.js:196 (BOOTSTRAP_KEYS comment) — was "ON_DEMAND_KEYS until Railway cron provisions; see below." Updated to cite plan 2026-04-24-001 and the strict-SEED_META posture. - api/health.js:398 (SEED_META comment) — was "Listed in ON_DEMAND_KEYS below until Railway cron provisions..." Updated for same reason. - docs/methodology/country-resilience-index.mdx:635 — v2.1 changelog entry said seed keys were ON_DEMAND_KEYS until graduation. Replaced with the fail-closed contract description. - docs/methodology/energy-v2-flag-flip-runbook.md:25 — step 3 said "ON_DEMAND_KEYS graduation" was required at flag-flip time. Rewrote to explain no graduation step is needed because the posture was removed pre-activation. No code change. Tests still 14/14 on the energy-v2 suite, lint:md clean. * fix(docs): escape MDX-unsafe `<=` in energy-v2 runbook to unblock Mintlify Mintlify deploy on PR #3363 failed with `Unexpected character '=' (U+003D) before name` at `docs/methodology/energy-v2-flag-flip-runbook.md`. Two lines had `<=` in plain prose, which MDX tries to parse as a JSX-tag-start. Replaced both with `≤` (U+2264) — and promoted the two existing `>=` on adjacent lines to `≥` for consistency. Prose is clearer and MDX safe. Same pattern as `mdx-unsafe-patterns-in-md` skill; also adjacent to PR #3344's `(<137 countries)` fix.
This commit is contained in:
@@ -7,9 +7,14 @@
|
||||
// cross-contamination from the v2 code path into the default
|
||||
// branch is a merge-blocker.
|
||||
// 2. Flag on = v2 composite. Each new indicator must move the score
|
||||
// in the documented direction (monotonicity), and countries
|
||||
// missing a v2 input should degrade gracefully to null per
|
||||
// weighted-blend contract rather than throw.
|
||||
// in the documented direction (monotonicity). When any REQUIRED
|
||||
// v2 seed is absent, the dispatch throws
|
||||
// `ResilienceConfigurationError` (fail-closed) so the operator
|
||||
// sees the misconfiguration via the source-failure path instead
|
||||
// of IMPUTE numbers that look computed. Pre-2026-04-24 the
|
||||
// scorer silently degraded to IMPUTE; plan
|
||||
// `docs/plans/2026-04-24-001-fix-resilience-v2-fail-closed-on-missing-seeds-plan.md`
|
||||
// inverts that contract.
|
||||
//
|
||||
// The tests use stubbed readers instead of Redis so the suite stays
|
||||
// hermetic.
|
||||
@@ -17,7 +22,11 @@
|
||||
import test, { describe, it, before, after } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
import { scoreEnergy, type ResilienceSeedReader } from '../server/worldmonitor/resilience/v1/_dimension-scorers.ts';
|
||||
import {
|
||||
ResilienceConfigurationError,
|
||||
scoreEnergy,
|
||||
type ResilienceSeedReader,
|
||||
} from '../server/worldmonitor/resilience/v1/_dimension-scorers.ts';
|
||||
|
||||
const TEST_ISO2 = 'ZZ'; // fictional country so test coverage checks don't flag it
|
||||
|
||||
@@ -159,13 +168,63 @@ describe('scoreEnergy — RESILIENCE_ENERGY_V2_ENABLED=true', () => {
|
||||
assert.ok(deHigh.score > deLow.score, `DE storage 10→90 should raise score; got ${deLow.score} → ${deHigh.score}`);
|
||||
});
|
||||
|
||||
it('missing v2 seed inputs degrade gracefully (no throw, coverage < 1.0)', async () => {
|
||||
const allMissing = await scoreEnergy(TEST_ISO2, makeEnergyReader(TEST_ISO2, {
|
||||
fossilBulk: null, lowCarbonBulk: null, lossesBulk: null,
|
||||
}));
|
||||
// Score may be null/low but must NOT throw. Coverage should be
|
||||
// well below 1.0 because most inputs are absent.
|
||||
assert.ok(allMissing.coverage < 1.0, `all-missing coverage should be < 1.0, got ${allMissing.coverage}`);
|
||||
// Fail-closed contract (plan 2026-04-24-001). When flag=true but any
|
||||
// required v2 seed is absent, the dispatch throws. Silent IMPUTE
|
||||
// fallback would produce fabricated-looking numbers with zero operator
|
||||
// signal — the exact failure the user post-mortem-caught on 2026-04-24.
|
||||
// The shared makeEnergyReader fixture uses `??` fallbacks that coerce
|
||||
// null-overrides back into default data, so these fail-closed tests
|
||||
// need a direct reader that can authoritatively return null for a
|
||||
// specific key without the fixture's defaulting logic.
|
||||
const makeReaderWithMissingV2Seeds = (missing: Set<string>): ResilienceSeedReader => {
|
||||
const base = makeEnergyReader(TEST_ISO2);
|
||||
return async (key: string) => (missing.has(key) ? null : base(key));
|
||||
};
|
||||
|
||||
it('flag on + ALL three v2 seeds missing → throws ResilienceConfigurationError naming all three keys', async () => {
|
||||
const reader = makeReaderWithMissingV2Seeds(new Set([
|
||||
'resilience:fossil-electricity-share:v1',
|
||||
'resilience:low-carbon-generation:v1',
|
||||
'resilience:power-losses:v1',
|
||||
]));
|
||||
await assert.rejects(
|
||||
scoreEnergy(TEST_ISO2, reader),
|
||||
(err: unknown) => {
|
||||
assert.ok(err instanceof ResilienceConfigurationError, `expected ResilienceConfigurationError, got ${err}`);
|
||||
assert.ok(err.message.includes('resilience:fossil-electricity-share:v1'), 'error must name missing fossil-share key');
|
||||
assert.ok(err.message.includes('resilience:low-carbon-generation:v1'), 'error must name missing low-carbon key');
|
||||
assert.ok(err.message.includes('resilience:power-losses:v1'), 'error must name missing power-losses key');
|
||||
assert.equal(err.missingKeys.length, 3, 'missingKeys must enumerate every absent seed');
|
||||
return true;
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it('flag on + partial missing (only fossil-share absent) → throws naming ONLY the missing key', async () => {
|
||||
// Operator-clarity: the error must tell the operator WHICH seed is
|
||||
// broken so they can fix that specific seeder rather than chase
|
||||
// all three.
|
||||
const reader = makeReaderWithMissingV2Seeds(new Set([
|
||||
'resilience:fossil-electricity-share:v1',
|
||||
]));
|
||||
await assert.rejects(
|
||||
scoreEnergy(TEST_ISO2, reader),
|
||||
(err: unknown) => {
|
||||
assert.ok(err instanceof ResilienceConfigurationError);
|
||||
assert.deepEqual([...err.missingKeys].sort(), ['resilience:fossil-electricity-share:v1']);
|
||||
assert.ok(!err.message.includes('low-carbon-generation'), 'must not mention keys that ARE present');
|
||||
assert.ok(!err.message.includes('power-losses'), 'must not mention keys that ARE present');
|
||||
return true;
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it('flag on + ALL three v2 seeds present → v2 runs normally, no throw', async () => {
|
||||
// Regression guard for the happy path — ensuring the preflight
|
||||
// check does not block a correctly-configured v2 activation.
|
||||
const result = await scoreEnergy(TEST_ISO2, makeEnergyReader(TEST_ISO2));
|
||||
assert.equal(typeof result.score, 'number', 'happy-path v2 must produce a numeric score');
|
||||
assert.ok(result.coverage > 0, 'happy-path v2 must report positive coverage');
|
||||
});
|
||||
|
||||
it('reserveMarginPct is NOT read in v2 path (deferred per plan §3.1)', async () => {
|
||||
|
||||
Reference in New Issue
Block a user