3 Commits

Author	SHA1	Message	Date
Elie Habib	da0f26a3cf	feat(resilience): PR 0 diagnostic freeze + fairness-audit harness (no scoring changes) (#3284) ... * feat(resilience): PR 0 diagnostic freeze + fairness-audit harness Lands the before-state and measurement apparatus every subsequent resilience-scorer PR validates against. Zero scoring changes. Per the v3 plan at docs/plans/2026-04-22-001-fix-resilience-scorer-structural- bias-plan.md this is tranche 0 of five. What lands: - Construct contract published in the methodology doc: absolute resilience not development-adjusted, mechanism test for every indicator, peer-relative views published separately from the core. - Known construct limitations section: six construct errors scheduled for PR 1-3 repair with explicit mapping to plan tranches. - Indicator-source manifest at docs/methodology/indicator-sources.yaml with source, seriesId, seriesUrl, coveragePct, lastObservedYear, license, mechanismTestRationale, and a constructStatus classification. - Pre-repair ranking snapshot at docs/snapshots/resilience-ranking-live-pre-repair-2026-04-22.json (217 items + 5 greyedOut, captured 2026-04-22 08:38 UTC at commit 425507d15). - Cohort configuration at tests/helpers/resilience-cohorts.mts: six cohorts covering 87 countries (net-fuel-exporters, net-energy- importers-oecd, nuclear-heavy-generation, coal-heavy-domestic, small-island-importers, fragile-states). - Matched-pair sanity panel at tests/helpers/resilience-matched-pairs.mts: six pairs (FR/DE, NO/CA, UAE/BH, JP/KR, IN/ZA, SG/CH) with expected- direction rationale and minGap for acceptance gate 7. - scripts/compare-resilience-current-vs-proposed.mjs extended to emit cohortSummary and matchedPairSummary alongside the existing output shape (backward compatible). - tests/resilience-cohort-config.test.mts: 11 validations ensuring the cohort + matched-pair configs stay well-formed. Deferred to PR 0.5 (before PR 1 lands): - Monotonicity test harness for all 19 dimension scorers pinning the sign of every indicator. - Pearson-derivative variable-influence baseline inside the sensitivity script producing the nominal-weight-vs-effective-influence table that plan acceptance gate 8 requires. Verification: typecheck:all clean, 430/430 resilience tests pass, 11/11 new cohort-config tests pass, snapshot auto-discovered and validated by the existing snapshot-test harness. * feat(resilience): PR 0 follow-ups — monotonicity harness, variable-influence baseline, cross-consumer formula gate Completes the PR 0 scope per the v3 plan §5 deliverables. Three adds: 1. Monotonicity test harness tests/resilience-dimension-monotonicity.test.mts pins the direction of movement for 14 indicators across 7 dimensions (reserve adequacy, fiscal space 3x, external debt coverage, import concentration, governance WGI, food/water 2x, energy 5x). Each test builds two synthetic ResilienceSeedReader fixtures differing only in the target indicator and asserts the dimension score moves in the documented direction. The scoreEnergy tests explicitly flag three indicators (gasShare, coalShare, electricityConsumption) that PR 1 §3.1-3.2 overturns so future readers understand which directional claims the plan intentionally replaces. 2. Variable-influence baseline scripts/compare-resilience-current-vs-proposed.mjs now computes per-dimension Pearson correlation against the current overallScore scaled by the dimension's nominal domain weight (a Pearson-derivative approximation of Sobol indices). The output carries a variableInfluence[] array sorted by abs(effectiveInfluence) desc. Acceptance gate 8 from the plan compares post-change effective influence against assigned nominal weight; divergences flag a wealth-proxy or saturated-signal construct problem. 3. Cross-consumer formula gate Five external consumers of resilience:score:v10:* now filter stale- formula entries so a flag flip does not serve mixed-formula data downstream: - server/worldmonitor/supply-chain/v1/get-route-impact.ts — readResilienceScore() checks _formula via the new getCurrentCacheFormula export and returns 0 on mismatch. - scripts/validate-resilience-correlation.mjs, scripts/validate-resilience-backtest.mjs, scripts/backtest-resilience-outcomes.mjs, scripts/benchmark-resilience-external.mjs — each inlines a currentCacheFormulaLocal() helper that mirrors the server's formula derivation from env, skips parsed entries whose _formula disagrees, and logs the skip count so operators can notice a mismatch during the flip window. A mixed-formula cohort (some countries d6-tagged, others pc-tagged) would confound every correlation, AUC, and Spearman this repair plan depends on for its acceptance gates. These guards close that gap. Verification: typecheck:all clean, 444/444 resilience tests pass (+14 from the new monotonicity harness). * fix(resilience): PR 0 review follow-ups — sample-union + doc tense Two review-driven fixes on top of PR 0. 1. scripts/compare-resilience-current-vs-proposed.mjs — the cohort and matched-pair summaries were computed against the historical 52-country sensitivity seed, which silently excluded the small-island-importers cohort (zero members in the seed) and the sg-vs-ch matched pair (Singapore not in the seed). With the current script those acceptance gates are partially measured at best. SAMPLE now = union(historical 52 seed, every cohort member, every matched-pair endpoint). The imports for RESILIENCE_COHORTS and MATCHED_PAIRS moved from inside main() to module scope so the union can be computed before the script runs. Net sample size grows from 52 to ~95 countries. Still fast enough for an interactive pass; makes the acceptance gates honest. 2. docs/methodology/country-resilience-index.mdx — the construct contract wording read as present-tense compliance ("Every indicator in the scorer passes a single mechanism test"), which contradicted the immediately-following passage about indicators that currently fail the test. Reworded to "is being evaluated against" and added an explicit PR-0-does-not-change-scoring paragraph that names the known-failing indicators (electricityConsumption, gas/coal flat penalties, WHO per-capita health spend) and points at the repair plan for the replacement schedule. Verification: typecheck:all clean, 444/444 resilience tests pass. * fix(resilience): compare-script loads frozen baseline + emits per-indicator influence Addresses two P1 review findings on PR #3284: 1. Script previously compared current-6d vs proposed-pillar-combined from the SAME checkout; never loaded the frozen pre-PR-0 baseline, so acceptance gates 2/6/7 ("no country moved >15pts vs baseline", cohort median shift vs baseline, matched-pair gap change vs baseline) could not be enforced for later scorer PRs. Now auto-discovers the most recent resilience-ranking-live-pre-repair-<date>.json (or post-<pr>-<date>) in docs/snapshots/ and emits a baselineComparison block with: spearmanVsBaseline, maxCountryAbsDelta, biggestDriftsVsBaseline, cohortShiftVsBaseline, matchedPairGapChange. If no baseline is found, the block is emitted with status 'unavailable' so callers distinguish missing-baseline from passed-baseline. 2. variableInfluence was emitted only at the dimension level, which hid the exact sub-indicators the repair plan targets (electricityConsumption, gasShare, coalShare, etc.) inside their parent dimension. Added extractIndicatorValues() which pulls twelve construct-risk indicators per country from the shared memoized reader, then computes per-indicator Pearson correlation against the current overall score. Emitted as perIndicatorInfluence[], sorted by absolute effective influence. Acceptance gate 8 ("effective influence agrees in sign and rank-order with assigned nominal weights") is now computable at the indicator level, not only at the dimension level. No production code touched; diagnostic-harness only. * fix(resilience): baseline-snapshot selection by structured parse, not filename sort Addresses P1 review on compare-resilience-current-vs-proposed.mjs:118-130. Plain filename sort breaks the "immediate-prior state" contract two ways: 1. Lexical ordering: `pre-repair` sorts after `post-*` (`pr...` to 'r' > 'o'), so the PR-0 freeze would keep winning even after post-PR snapshots exist. Later scorer PRs would then report acceptance-gate deltas against the original pre-repair freeze instead of the immediately-prior post-PR-(N-1) snapshot — the gate would appear valid while measuring against the wrong baseline. 2. Lexical ordering: `pr10` < `pr9` (digit-by-digit), so PR-10 would lose the selection to PR-9. Fix: parseBaselineSnapshotMeta() extracts (kind, prNumber, date) from the filename. Sort keys are (kindRank desc, prNumber desc, date desc): - post always beats pre-repair (kindRank 1 vs 0) - among posts, prNumber compared numerically (10 beats 9) - date breaks ties (same-PR re-snapshots, later capture wins) - unlabeled post tags get prNumber 0 so they sort between pre-repair and any numbered PR snapshot Surfaced in output: baselineKind / baselinePrNumber / baselineDate alongside baselineFile so the operator can verify which snapshot was selected without having to reopen the file. Module now isMain-guarded per feedback_seed_isMain_guard memory so tests can import parseBaselineSnapshotMeta without firing the scoring run. Added tests/resilience-baseline-snapshot-ordering.test.mjs (9 tests) pinning the ordering contract for every known failure mode. Diagnostic-harness change only. No production code touched. * fix(resilience): full scorable universe + registry-driven per-indicator influence Addresses two fresh P1 review findings on the PR 0 compare harness. Finding 1 — acceptance math ran on a curated ~95-country sample, so plan gate 2 could miss large regressions in excluded countries. - Main scoring loop now iterates the FULL scorable universe (listScorableCountries()), not the 52-country seed + cohort union. - Removed SAMPLE / HISTORICAL_SENSITIVITY_SEED constants. - Added scorableUniverseSize + cohortMissingFromScorable to output so operators see universe size and any cohort/pair endpoint that listScorable refuses to score (fail-loud, not silent drop). Finding 3 — per-indicator influence was a hand-picked 12-indicator subset, hiding most registry indicators from the baseline that later scorer PRs need. - Extraction is now driven by INDICATOR_REGISTRY. Every Core + Enrichment indicator gets a row with explicit extractionStatus: implemented | not-implemented (with reason) | unregistered-in-harness - EXTRACTION_RULES covers 40/59 indicators across 11 shape families (static-path, static-wb-infrastructure, static-wgi, static-wgi-mean, static-who, energy-mix-field, gas-storage-field, recovery-country- field, imf-macro/labor-country-field, national-debt, sanctions-count). - Remaining 19 indicators need either a scorer trace hook (PR 0.5) or a safe aggregation duplicate; each carries a reason string. - extractionCoverage summary (totalIndicators / implemented / notImplemented / unregisteredInHarness / coreImplemented / coreTotal) exposed in output so PR 0.5 progress is measurable. Added tests/resilience-indicator-extraction-plan.test.mjs (11 tests) pinning: every registry entry has an extraction row; not-implemented rows carry a reason; all 12 plan-named construct-risk indicators stay extractable; Core-tier coverage floor of 45%; shape-family unit tests. Diagnostic-harness change only. No production code touched. * fix(resilience): wire event-aggregate per-indicator influence via exported scorer helpers Addresses P1 review on PR 0 compare harness. Previous commit marked 16 Core-tier indicators as 'not-implemented' because they needed scorer event-window/severity-weighting math; that left the gate-9 acceptance apparatus incomplete for a large part of the shipped score. Fix: export the scorer-internal aggregation helpers so the harness calls them directly. Zero aggregation math duplicated in the harness, harness and scorer cannot drift. Exported from _dimension-scorers.ts (purely additive): summarizeCyber, summarizeOutages, summarizeGps, summarizeUcdp, summarizeUnrest, summarizeSocialVelocity, getCountryDisplacement, getThreatSummaryScore, countTradeRestrictions, countTradeBarriers. 13 extraction rules moved from not-implemented to implemented: cyberThreats, internetOutages, infraOutages, gpsJamming, ucdpConflict, unrestEvents, socialVelocity, newsThreatScore, displacementTotal, displacementHosted, tradeRestrictions, tradeBarriers, recoveryConflictPressure, recoveryDisplacementVelocity. Coverage: 52/59 total (88%), 46/50 Core-tier (92%). Four Core indicators remain not-implemented for STRUCTURAL reasons, NOT missing code. Scorer inputs are genuinely global scalars with zero per-country variance, so Pearson(indicator, overall) is 0 or NaN by construction: shippingStress, transitDisruption, energyPriceStress — scorer reads a global scalar applied to every country; a per-country effective signal would need re-expression as (global x per-country exposure), which is a derived signal in a different entry. aquastatWaterAvailability — needs a distinct sub-indicator path resolver; enrichment follow-up. New test asserts the three no-per-country-variance indicators STAY not-implemented with a matching reason, so any future extraction that appears to cover them without fixing the underlying construct fails. Dispatcher split into STATIC / SIMPLE / AGGREGATE extractor tables to stay under biome complexity limit. Core-tier floor test raised from 45% to 80%. 89 resilience tests pass, typecheck clean, biome clean. No production behaviour changes. * fix(resilience): tag-gated AQUASTAT extractor closes the last fixable Core gap Reviewer flagged aquastatWaterAvailability as the only remaining Core indicator where the not-implemented status was structurally fixable rather than conceptually impossible. Both aquastatWaterStress and aquastatWaterAvailability share a single .aquastat.value field; the scorer's scoreAquastatValue splits them by the sibling .aquastat.indicator tag keyword (stress/withdrawal/ dependency to stress family; availability/renewable/access to availability family). The harness now mirrors this branching: - classifyAquastatFamily implements the scorer's priority order (stress-family match wins even if the tag also contains an availability keyword, matching the sequential if-check at _dimension-scorers.ts L770-776). - static-aquastat-stress / static-aquastat-availability extractors return the value only when the family matches, so stress-family readings never corrupt the availability Pearson and vice versa. Core-tier coverage: 46/50 to 47/50 (94%). The 3 remaining Core not-implemented indicators (shippingStress, transitDisruption, energyPriceStress) are all structural impossibilities: scorer inputs are global scalars with zero per-country variance. New contract test pins both directions of the tag gate plus the priority-order edge case (a tag containing both families' keywords routes to stress). 90 resilience tests pass, typecheck clean, biome clean.	2026-04-22 16:44:12 +04:00
Elie Habib	fbaf07e106	feat(resilience): flag-gated pillar-combined score activation (default off) (#3267) ... Wires the non-compensatory 3-pillar combined overall_score behind a RESILIENCE_PILLAR_COMBINE_ENABLED env flag. Default is false so this PR ships zero behavior change in production. When flipped true the top-level overall_score switches from the 6-domain weighted aggregate to penalizedPillarScore(pillars) with alpha 0.5 and pillar weights 0.40 / 0.35 / 0.25. Evidence from docs/snapshots/resilience-pillar-sensitivity-2026-04-21: - Spearman rank correlation current vs proposed 0.9935 - Mean score delta -13.44 points (every country drops, penalty is always at most 1) - Max top-50 rank swing 6 positions (Russia) - No ceiling or floor effects under plus/minus 20pct perturbation - Release gate PASS 0/19 Code change in server/worldmonitor/resilience/v1/_shared.ts: - New isPillarCombineEnabled() reads env dynamically so tests can flip state without reloading the module - overallScore branches on (isPillarCombineEnabled() AND RESILIENCE_SCHEMA_V2_ENABLED AND pillars.length > 0); otherwise falls through to the 6-domain aggregate (unchanged default path) - RESILIENCE_SCORE_CACHE_PREFIX bumped v9 to v10 - RESILIENCE_RANKING_CACHE_KEY bumped v9 to v10 Cache invalidation: the version bump forces both per-country score cache and ranking cache to recompute from the current code path on first read after a flag flip. Without the bump, 6-domain values cached under the flag-off path would continue to serve for up to 6-12 hours after the flip, producing a ragged mix of formulas. Ripple of v9 to v10: - api/health.js registry entry - scripts/seed-resilience-scores.mjs (both keys) - scripts/validate-resilience-correlation.mjs, scripts/backtest-resilience-outcomes.mjs, scripts/validate-resilience-backtest.mjs, scripts/benchmark-resilience-external.mjs - tests/resilience-ranking.test.mts 24 fixture usages - tests/resilience-handlers.test.mts - tests/resilience-scores-seed.test.mjs explicit pin - tests/resilience-pillar-aggregation.test.mts explicit pin - docs/methodology/country-resilience-index.mdx New tests/resilience-pillar-combine-activation.test.mts: 7 assertions exercising the flag-on path against the release fixtures with re-anchored bands (NO at least 60, YE/SO at most 40, NO greater than US preserved, elite greater than fragile). Regression guard verifies flipping the flag back off restores the 6-domain aggregate. tests/resilience-ranking-snapshot.test.mts: band thresholds now resolve from a METHODOLOGY_BANDS table keyed on snapshot.methodologyFormula. Backward compatible (missing formula defaults to domain-weighted-6d bands). Snapshots: - docs/snapshots/resilience-ranking-2026-04-21.json tagged methodologyFormula domain-weighted-6d - docs/snapshots/resilience-ranking-pillar-combined-projected-2026-04-21.json new: top/bottom/major-economies tables projected from the 52-country sensitivity sample. Explicitly tagged projected (NOT a full-universe live capture). When the flag is flipped in production, run scripts/freeze-resilience-ranking.mjs to capture the authoritative full-universe snapshot. Methodology doc: Pillar-combined score activation section rewritten to describe the flag-gated mechanism (activation is an env-var flip, no code deploy) and the rollback path. Verification: npm run typecheck:all clean, 397/397 resilience tests pass (up from 390, +7 activation tests). Activation plan: 1. Merge this PR with flag default false (zero behavior change) 2. Set RESILIENCE_PILLAR_COMBINE_ENABLED=true in Vercel and Railway env 3. Redeploy or wait for next cold start; v9 to v10 bump forces every country to be rescored on first read 4. Run scripts/freeze-resilience-ranking.mjs against the flag-on deployment and commit the resulting snapshot 5. Ship a v2.0 methodology-change note explaining the re-anchored scale so analysts understand the universal ~13 point score drop is a scale rebase, not a country-level regression Rollback: set RESILIENCE_PILLAR_COMBINE_ENABLED=false, flush resilience:score:v10:* and resilience:ranking:v10 keys (or wait for TTLs). The 6-domain formula stays alongside the pillar combine in _shared.ts and needs no code change to come back.	2026-04-22 06:52:07 +04:00
Elie Habib	502bd4472c	docs(resilience): sync methodology/proto/widget to 6-domain + 3-pillar reality (#3264) ... Brings every user-facing surface into alignment with the live resilience scorer. Zero behavior change: overall_score is still the 6-domain weighted aggregate, schemaVersion is still 2.0 default, and every existing test continues to pass. Surfaces touched: - proto + OpenAPI: rewrote the ResiliencePillar + schema_version descriptions. 2.0 is correctly documented as default; shaped-but-empty language removed. - Widget: added missing recovery: 'Recovery' label (was rendering literal lowercase recovery before), retitled footer data-version chip from Data to Seed date so it is clear the value reflects the static seed bundle not every live input, rewrote help tooltip for 6 domains and 3 pillars and called out the 0.25 recovery weight. - Methodology doc: domains-and-weights table now carries all 6 rows with actual code weights (0.17/0.15/0.11/0.19/0.13/0.25), Recovery section header weight corrected from 1.0 to 0.25, new Pillar-combined score activation (pending) section with the measured Spearman 0.9935, top-5 movers, and the activation checklist. - documentation.mdx + features.mdx: product blurbs updated from 5 domains and 13 dimensions to 6 domains and 19 dimensions grouped into 3 pillars. - Tests: recovery-label regression pin, Seed date label pin, clarified pillar-schema degenerate-input semantics. New scaffolding for defensibility: - docs/snapshots/resilience-ranking-2026-04-21.json frozen published tables artifact with methodology metadata and commit SHA. - docs/snapshots/resilience-pillar-sensitivity-2026-04-21.json live Redis capture (52-country sample) combining sensitivity stability with the current-vs-proposed Spearman comparison. - scripts/freeze-resilience-ranking.mjs refresh script. - scripts/compare-resilience-current-vs-proposed.mjs comparison script. - tests/resilience-ranking-snapshot.test.mts 13 assertions auto discovered from any resilience-ranking-YYYY-MM-DD.json in snapshots. Verification: npm run typecheck:all clean, 390/390 resilience tests pass. Follow-up: pillar-combined score activation. The sensitivity artifact shows rank-preservation Spearman 0.9935 and no ceiling effects, which clears the methodological bar. Blocker is messaging because every country drops ~13 points under the penalty, so activation PR ships with re-anchored release-gate bands, refreshed frozen ranking, and a v2.0 methodology note.	2026-04-21 22:37:27 +04:00