mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
* docs(resilience): PR 5.3 — foodWater scorer audit (construct-deterministic GCC identity) PR 5.3 of cohort-audit plan 2026-04-24-002. Stacked on PR 5.2 (#3373) so the known-limitations.md section append is additive. Read-only static audit of scoreFoodWater. Findings 1. The observed GCC-all-score-53 is CONSTRUCT-DETERMINISTIC, not a regional-default leak. Pinned mathematically: - IPC/HDX doesn't publish active food-crisis data for food-secure states → scorer's fao-null branch imputes IMPUTE.ipcFood=88 (class='stable-absence', cov=0.7) at combined weight 0.6 - WB indicator ER.H2O.FWST.ZS (labelled 'water stress') for GCC is EXTREME (KW ~3200%, BH ~3400%, UAE ~2080%, QA ~770%) — all clamp to sub-score 0 under the scorer's lower-better 0..100 normaliser at weight 0.4 - Blended with peopleInCrisis=0 (fao block present with zero): (100 * 0.45 + 0 * 0.4) / (0.45 + 0.4) = 45 / 0.85 ≈ 53 Every GCC country has the same inputs → same outputs. That's construct math, not a regional lookup. 2. Indicator-keyword routing is code-correct. `'water stress'`, `'withdrawal'`, `'dependency'` route to lower-better; `'availability'`, `'renewable'`, `'access'` route to higher-better; unrecognized indicators fall through to a value-range heuristic with a WARN log. 3. No bug or methodology decision required. The 53-all-GCC output is a correct summary statement: "non-crisis food security + severe water-withdrawal stress." A future construct decision might split foodWater into separate food and water dims so one saturated sub-signal doesn't dominate the combined dim for desert economies — but that's a construct redesign, not a bug. Shipped - `docs/methodology/known-limitations.md` — extended with a new section documenting the foodWater audit findings, the exact blend math that yields ~53 for GCC, cohort-determinism vs regional-default, and a follow-up data-side spot-check list gated on API-key access. - `tests/resilience-foodwater-field-mapping.test.mts` — 8 new regression-guard tests: 1. indicator='water stress' routes to lower-better 2. GCC extreme-withdrawal anchor (value=2000 → blended score 53) 3. indicator='renewable water availability' routes to higher-better 4. fao=null with static record → imputes 88; imputationClass=null because observed AQUASTAT wins (weightedBlend T1.7 rule) 5. fully-imputed (fao=null + aquastat=null) surfaces imputationClass='stable-absence' 6. static-record absent entirely → coverage=0, NOT impute 7. Cohort determinism — identical inputs → identical scores 8. Different water-profile inputs → different scores (rules out regional-default hypothesis) Verified - `npx tsx --test tests/resilience-foodwater-field-mapping.test.mts` — 8 pass / 0 fail - `npm run test:data` — 6711 pass / 0 fail (PR 5.2's 9 + PR 5.3's 8 = 17 new stacked) - `npm run typecheck` / `typecheck:api` — green - `npm run lint` / `lint:md` — clean * fix(resilience): PR 5.3 review — pin IMPUTE branch for GCC anchor; fix comment math Addresses 3 P2 Greptile findings on #3374 — all variations of the same root cause: the test fixture + doc described two different code paths that coincidentally both produce ~53 for GCC inputs. Changes 1. GCC anchor test now drives the IMPUTE branch (`fao: null`), matching what the static seeder emits for GCC in production. The else branch (`fao: { peopleInCrisis: 0 }`) happens to converge on ~52.94 by coincidence but is NOT the live code path for GCC. 2. Doc finding #4 updated to show the IMPUTE-branch math `(88×0.6 + 0×0.4) / 1.0 = 52.8 → 53` and explicitly notes the else-branch convergence as a coincidence — not the construct's intent. 3. Comment math off-by-one fix at line 107: (88×0.6 + 80×0.4) / (0.6+0.4) = 52.8 + 32.0 = 84.8 → 85 (was incorrectly stated as 85.6 → 86) Test assertion `>= 80 && <= 90` still accepts 85 so behaviour is unchanged; this was a comment-only error that would have misled anyone reproducing the math by hand. Verified - `npx tsx --test tests/resilience-foodwater-field-mapping.test.mts` — 8 pass / 0 fail (IMPUTE-branch anchor test produces 53 as expected) - `npm run lint:md` — clean Also rebased onto updated #3373 (which landed a backtick-escape fix).
201 lines
10 KiB
TypeScript
201 lines
10 KiB
TypeScript
// Regression guard for scoreFoodWater — inputs, branching, and the
|
|
// "identical cohort scores are construct-deterministic, not regional-
|
|
// default leaks" invariant that PR 5.3 of plan 2026-04-24-002 sets
|
|
// out to establish.
|
|
//
|
|
// Context. The plan flagged that all GCC countries score ~53 on
|
|
// `foodWater` and asked: is that a mystery regional default or is it
|
|
// a genuine construct output? This test suite pins the answer:
|
|
// identical inputs produce identical outputs, and the inputs are
|
|
// themselves explicable — IPC does not monitor rich food-secure
|
|
// states (impute 88) and AQUASTAT/WB water-stress values for the
|
|
// GCC are EXTREME (freshwater withdrawal > 100% of renewable
|
|
// resources), which clamps the AQUASTAT sub-score to 0. The blend
|
|
// of IMPUTE.ipcFood=88 with AQUASTAT=0 under the documented weights
|
|
// produces a near-identical score across countries with the same
|
|
// water-stress profile. That's the construct working — not a bug.
|
|
//
|
|
// See docs/methodology/known-limitations.md for the full write-up.
|
|
|
|
import assert from 'node:assert/strict';
|
|
import { describe, it } from 'node:test';
|
|
|
|
import {
|
|
scoreFoodWater,
|
|
type ResilienceSeedReader,
|
|
} from '../server/worldmonitor/resilience/v1/_dimension-scorers.ts';
|
|
|
|
const TEST_ISO2 = 'XX';
|
|
|
|
function makeStaticReader(staticRecord: unknown): ResilienceSeedReader {
|
|
return async (key: string) => (key === `resilience:static:${TEST_ISO2}` ? staticRecord : null);
|
|
}
|
|
|
|
describe('scoreAquastatValue — indicator-keyword routing contract', () => {
|
|
it('indicator="water stress" routes to lower-better (higher withdrawal = worse score)', async () => {
|
|
// WB indicator ER.H2O.FWST.ZS: "Level of water stress: freshwater
|
|
// withdrawal as a proportion of available freshwater resources."
|
|
// Seeded as `indicator: 'water stress'`. Containing "stress" →
|
|
// `normalizeLowerBetter(value, 0, 100)` per _dimension-scorers.ts:899.
|
|
const low = makeStaticReader({
|
|
aquastat: { value: 10, indicator: 'water stress' },
|
|
fao: { peopleInCrisis: 0, phase: null },
|
|
});
|
|
const high = makeStaticReader({
|
|
aquastat: { value: 80, indicator: 'water stress' },
|
|
fao: { peopleInCrisis: 0, phase: null },
|
|
});
|
|
const [lowScore, highScore] = await Promise.all([
|
|
scoreFoodWater(TEST_ISO2, low),
|
|
scoreFoodWater(TEST_ISO2, high),
|
|
]);
|
|
assert.ok(lowScore.score > highScore.score,
|
|
`higher water-stress must LOWER the score; got low=${lowScore.score}, high=${highScore.score}`);
|
|
});
|
|
|
|
it('AQUASTAT value > 100 clamps to 0 (the GCC extreme-withdrawal case)', async () => {
|
|
// GCC freshwater withdrawal % of renewable resources is well over
|
|
// 100% (KW ~3200, BH ~3400, AE ~2080, QA ~770). The normaliser
|
|
// clamps anything past the "worst" anchor to 0. Test with 2000,
|
|
// which comfortably exceeds the 100 anchor.
|
|
//
|
|
// IMPORTANT: drive the IMPUTE branch (`fao: null`), not the else
|
|
// branch. In production the static seeder writes `fao: null` for
|
|
// GCC (IPC/HDX does not monitor food-secure states), so the live
|
|
// blend uses the impute path with weights 0.6 (IPC impute=88) +
|
|
// 0.4 (AQUASTAT). The else branch (fao-present, peopleInCrisis=0)
|
|
// happens to converge on a near-identical number at these inputs
|
|
// by coincidence, but testing the wrong branch would let a future
|
|
// impute-branch regression slip through.
|
|
const reader = makeStaticReader({
|
|
aquastat: { value: 2000, indicator: 'water stress' },
|
|
fao: null, // fao==null branch — matches the GCC production shape
|
|
});
|
|
const result = await scoreFoodWater(TEST_ISO2, reader);
|
|
// Blend under the IMPUTE branch:
|
|
// { score: 88, weight: 0.6, cov: 0.7, imputed } // IMPUTE.ipcFood
|
|
// { score: 0, weight: 0.4, cov: 1.0, observed } // AQUASTAT clamped
|
|
// weightedScore = (88*0.6 + 0*0.4) / (0.6+0.4) = 52.8 → 53.
|
|
// This is EXACTLY the observed GCC cohort score. Pinning it.
|
|
assert.equal(Math.round(result.score), 53,
|
|
`GCC water-stress profile must yield ~53 on the IMPUTE branch; got ${result.score}`);
|
|
});
|
|
|
|
it('indicator="renewable water availability" routes to higher-better (more = better)', async () => {
|
|
const scarce = makeStaticReader({
|
|
aquastat: { value: 500, indicator: 'renewable water availability' },
|
|
fao: { peopleInCrisis: 0, phase: null },
|
|
});
|
|
const abundant = makeStaticReader({
|
|
aquastat: { value: 4500, indicator: 'renewable water availability' },
|
|
fao: { peopleInCrisis: 0, phase: null },
|
|
});
|
|
const [scarceScore, abundantScore] = await Promise.all([
|
|
scoreFoodWater(TEST_ISO2, scarce),
|
|
scoreFoodWater(TEST_ISO2, abundant),
|
|
]);
|
|
assert.ok(abundantScore.score > scarceScore.score,
|
|
`more renewable water = higher score; got scarce=${scarceScore.score}, abundant=${abundantScore.score}`);
|
|
});
|
|
});
|
|
|
|
describe('scoreFoodWater — IPC absence path (stable-absence imputation)', () => {
|
|
it('country not in IPC/HDX (fao=null) imputes ipcFood=88 when static record present', async () => {
|
|
const reader = makeStaticReader({
|
|
aquastat: { value: 20, indicator: 'water stress' },
|
|
fao: null, // crisis_monitoring_absent — food-secure country, not a monitored crisis
|
|
});
|
|
const result = await scoreFoodWater(TEST_ISO2, reader);
|
|
// Blend: {score:88, weight:0.6, cov:0.7, imputed} + {score:80, weight:0.4, cov:1.0, observed}
|
|
// weightedScore = (88*0.6 + 80*0.4) / (0.6+0.4) = 52.8 + 32.0 = 84.8 → 85
|
|
// Pinning the blended range to catch a formula regression.
|
|
assert.ok(result.score >= 80 && result.score <= 90,
|
|
`IPC-absent + moderate aquastat must blend to 80-90; got ${result.score}`);
|
|
// Per weightedBlend's T1.7 rule (line 601 of _dimension-scorers.ts):
|
|
// `imputationClass` surfaces ONLY when observedWeight === 0. Here
|
|
// AQUASTAT is observed data (score=80), so it "wins" and the final
|
|
// imputationClass is null. The IPC-impute is still reflected in
|
|
// `imputedWeight` and the lower coverage (70% for IPC * 0.6 + 100%
|
|
// for AQUASTAT * 0.4 = 82% weighted).
|
|
assert.equal(result.imputationClass, null,
|
|
'mixed observed+imputed → imputationClass=null (observed wins); IPC impute reflected in imputedWeight');
|
|
assert.ok(result.imputedWeight > 0, 'IPC slot must be counted as imputed');
|
|
assert.ok(result.observedWeight > 0, 'AQUASTAT slot must be counted as observed');
|
|
});
|
|
|
|
it('country fully imputed (fao=null AND aquastat absent) surfaces imputationClass=stable-absence', async () => {
|
|
// This is the scenario where `imputationClass` actually surfaces:
|
|
// AQUASTAT missing (null score, no impute) → contributes no weight
|
|
// AT ALL to observedWeight or imputedWeight. The remaining IPC
|
|
// slot is fully imputed, so the dimension is fully imputed and
|
|
// weightedBlend picks the dominant (only) class.
|
|
const reader = makeStaticReader({
|
|
aquastat: null, // AQUASTAT data missing entirely
|
|
fao: null, // IPC not monitoring
|
|
});
|
|
const result = await scoreFoodWater(TEST_ISO2, reader);
|
|
assert.equal(result.imputationClass, 'stable-absence',
|
|
'fully-imputed dim must surface stable-absence class');
|
|
// Single imputed slot {score:88, weight:0.6, cov:0.7}:
|
|
// weightedScore = 88 * 0.6 / 0.6 = 88 (only this slot has a score)
|
|
// weightedCertainty = 0.7 * 0.6 / (0.6+0.4) = 0.42 total weighted / total
|
|
assert.ok(result.score >= 85 && result.score <= 92,
|
|
`fully-imputed must score ~88; got ${result.score}`);
|
|
});
|
|
|
|
it('static-record absent entirely (seeder never ran) does NOT impute — returns null branch', async () => {
|
|
// Per the scorer comment at line 1482 of _dimension-scorers.ts:
|
|
// "A missing resilience:static:{ISO2} key means the seeder never
|
|
// ran — not crisis-free." So this path returns weight-null for
|
|
// the IPC slot, not an IMPUTE. The AQUASTAT slot is also null
|
|
// because scoreAquastatValue(null)=null. Result: zero-signal.
|
|
const reader = makeStaticReader(null);
|
|
const result = await scoreFoodWater(TEST_ISO2, reader);
|
|
// weightedBlend with two null-score slots returns coverage=0.
|
|
assert.equal(result.coverage, 0,
|
|
'fully-absent static record must produce coverage=0 (no impute-as-if-safe)');
|
|
});
|
|
});
|
|
|
|
describe('scoreFoodWater — cohort determinism (not a regional-default leak)', () => {
|
|
it('two countries with identical inputs produce identical scores (construct-deterministic)', async () => {
|
|
// Two synthetic "GCC-shaped" countries: extreme water stress
|
|
// (WB value > 100 clamps AQUASTAT to 0) + IPC-absent. Same
|
|
// inputs → same output is the CORRECT behavior. An identical
|
|
// score across a cohort is a construct signal, NOT evidence of
|
|
// a hardcoded regional default.
|
|
const gccShape = {
|
|
aquastat: { value: 2500, indicator: 'water stress' },
|
|
fao: null,
|
|
};
|
|
const [a, b] = await Promise.all([
|
|
scoreFoodWater('AE', async (k) => (k === 'resilience:static:AE' ? gccShape : null)),
|
|
scoreFoodWater('KW', async (k) => (k === 'resilience:static:KW' ? gccShape : null)),
|
|
]);
|
|
assert.equal(a.score, b.score,
|
|
'identical inputs must produce identical scores — the observed GCC cohort identity is construct-deterministic');
|
|
assert.equal(a.coverage, b.coverage, 'and identical coverage');
|
|
});
|
|
|
|
it('a different water-profile input produces a different score (rules out the regional-default hypothesis)', async () => {
|
|
// Same IPC-absent status, but different AQUASTAT indicator /
|
|
// value: a high-renewable country. If foodWater were using a
|
|
// hardcoded regional default, this country would score the
|
|
// same as the water-stress case above. It must not.
|
|
const waterStressShape = {
|
|
aquastat: { value: 2500, indicator: 'water stress' },
|
|
fao: null,
|
|
};
|
|
const waterAbundantShape = {
|
|
aquastat: { value: 8000, indicator: 'renewable water availability' },
|
|
fao: null,
|
|
};
|
|
const [stressed, abundant] = await Promise.all([
|
|
scoreFoodWater('AE', async (k) => (k === 'resilience:static:AE' ? waterStressShape : null)),
|
|
scoreFoodWater('IS', async (k) => (k === 'resilience:static:IS' ? waterAbundantShape : null)),
|
|
]);
|
|
assert.ok(abundant.score > stressed.score,
|
|
`water-abundant country must outscore water-stressed; got stressed=${stressed.score}, abundant=${abundant.score} — a regional default would have tied these`);
|
|
});
|
|
});
|