mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
8cca8d19e33ac84881a5739ea1675f7224d6e6de
51 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
8cca8d19e3 |
feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read (#3385)
* feat(seed): BUNDLE_RUN_STARTED_AT_MS env + runSeed SIGTERM cleanup
Prereq for the re-export-share Comtrade seeder (plan 2026-04-24-003),
usable by any cohort seeder whose consumer needs bundle-level freshness.
Two coupled changes:
1. `_bundle-runner.mjs` injects `BUNDLE_RUN_STARTED_AT_MS` into every
spawned child. All siblings in a single bundle run share one value
(captured at `runBundle` start, not spawn time). Consumers use this
to detect stale peer keys — if a peer's seed-meta predates the
current bundle run, fall back to a hard default rather than read
a cohort-peer's last-week output.
2. `_seed-utils.mjs::runSeed` registers a `process.once('SIGTERM')`
handler that releases the acquired lock and extends existing-data
TTL before exiting 143. `_bundle-runner.mjs` sends SIGTERM on
section timeout, then SIGKILL after KILL_GRACE_MS (5s). Without
this handler the `finally` path never runs on SIGKILL, leaving
the 30-min acquireLock reservation in place until its own TTL
expires — the next cron tick silently skips the resource.
Regression guard memory: `bundle-runner-sigkill-leaks-child-lock` (PR
#3128 root cause).
Tests added:
- bundle-runner env injection (value within run bounds)
- sibling sections share the same timestamp (critical for the
consumer freshness guard)
- runSeed SIGTERM path: exit 143 + cleanup log
- process.once contract: second SIGTERM does not re-enter handler
* fix(seed): address P1/P2 review findings on SIGTERM + bundle contracts
Addresses PR #3384 review findings (todos 256, 257, 259, 260):
#256 (P1) — SIGTERM handler narrowed to fetch phase only. Was installed
at runSeed entry and armed through every `process.exit` path; could
race `emptyDataIsFailure: true` strict-floor exits (IMF-External,
WB-bulk) and extend seed-meta TTL when the contract forbids it —
silently re-masking 30-day outages. Now the handler is attached
immediately before `withRetry(fetchFn)` and removed in a try/finally
that covers all fetch-phase exit branches.
#257 (P1) — `BUNDLE_RUN_STARTED_AT_MS` now has a first-class helper.
Exported `getBundleRunStartedAtMs()` from `_seed-utils.mjs` with JSDoc
describing the bundle-freshness contract. Fleet-wide helper so the
next consumer seeder imports instead of rediscovering the idiom.
#259 (P2) — SIGTERM cleanup runs `Promise.allSettled` on disjoint-key
ops (`releaseLock` + `extendExistingTtl`). Serialising compounded
Upstash latency during the exact failure mode (Redis degraded) this
handler exists to handle, risking breach of the 5s SIGKILL grace.
#260 (P2) — `_bundle-runner.mjs` asserts topological order on
optional `dependsOn` section field. Throws on unknown-label refs and
on deps appearing at a later index. Fleet-wide contract replacing
the previous prose-comment ordering guarantee.
Tests added/updated:
- New: SIGTERM handler removed after fetchFn completes (narrowed-scope
contract — post-fetch SIGTERM must NOT trigger TTL extension)
- New: dependsOn unknown-label + out-of-order + happy-path (3 tests)
Full test suite: 6,866 tests pass (+4 net).
* fix(seed): getBundleRunStartedAtMs returns null outside a bundle run
Review follow-up: the earlier `Math.floor(Date.now()/1000)*1000` fallback
regressed standalone (non-bundle) runs. A consumer seeder invoked
manually just after its peer wrote `fetchedAt = (now - 5s)` would see
`bundleStartMs = Date.now()`, reject the perfectly-fresh peer envelope
as "stale", and fall back to defaults — defeating the point of the
peer-read path outside the bundle.
Returning null when `BUNDLE_RUN_STARTED_AT_MS` is unset/invalid keeps
the freshness gate scoped to its real purpose (across-bundle-tick
staleness) and lets standalone runs skip the gate entirely. Consumers
check `bundleStartMs != null` before applying the comparison; see the
companion `seed-sovereign-wealth.mjs` change on the stacked PR.
* test(seed): SIGTERM cleanup test now verifies Redis DEL + EXPIRE calls
Greptile review P2 on PR #3384: the existing test only asserted exit
code + log line, not that the Redis ops were actually issued. The
log claim was ahead of the test.
Fixture now logs every Upstash fetch call's shape (EVAL / pipeline-
EXPIRE / other) to stderr. Test asserts:
- >=1 EVAL op was issued during SIGTERM cleanup (releaseLock Lua
script on the lock key)
- >=1 pipeline-EXPIRE op was issued (extendExistingTtl on canonical
+ seed-meta keys)
- The EVAL body carries the runSeed-generated runId (proves it's
THIS run's release, not a phantom op)
- The EXPIRE pipeline touches both the canonicalKey AND the
seed-meta key (proves the keys[] array was built correctly
including the extraKeys merge path)
Full test suite: 6,866 tests pass, typecheck clean.
* feat(resilience): Comtrade-backed re-export-share seeder + SWF Redis read
Plan ref: docs/plans/2026-04-24-003-feat-reexport-share-comtrade-seeder-plan.md
Motivating case. Before this PR, the SWF `rawMonths` denominator for
the `sovereignFiscalBuffer` dimension used GROSS annual imports for
every country. For re-export hubs (goods transiting without domestic
settlement), this structurally under-reports resilience: UAE's 2023
$941B of imports include $334B of transit flow that never represents
domestic consumption. Net imports = gross × (1 − reexport_share).
The previous (PR 3A) design flattened a hand-curated YAML into Redis;
the YAML shipped empty and never populated, so the correction never
applied and the cohort audit showed no movement.
Gap #2 (this PR). Two coupled changes to make the correction actually
apply:
1. Comtrade-backed seeder (`scripts/seed-recovery-reexport-share.mjs`).
Rewritten to fetch UN Comtrade `flowCode=RX` (re-exports) and
`flowCode=M` (imports) per cohort member, compute share = RX/M at
the latest co-populated year, clamp to [0.05, 0.95], publish the
envelope. Header auth (`Ocp-Apim-Subscription-Key`) — subscription
key never reaches URL/logs/Redis. `maxRecords=250000` cap with
truncation detection. Sequential + retry-on-429 with backoff.
Hub cohort resolved by Phase 0 empirical probe (plan §Phase 0):
['AE', 'PA']. Six candidates (SG/HK/NL/BE/MY/LT) return HTTP 200
with zero RX rows — Comtrade doesn't expose RX for those reporters.
2. SWF seeder reads from Redis (`scripts/seed-sovereign-wealth.mjs`).
Swaps `loadReexportShareByCountry()` (YAML) for
`loadReexportShareFromRedis()` (Redis key written by #1). Guarded
by bundle-run freshness: if the sibling Reexport-Share seeder's
`seed-meta` predates `BUNDLE_RUN_STARTED_AT_MS` (set by the
prereq PR's `_bundle-runner.mjs` env-injection), HARD fallback
to gross imports rather than apply last-month's stale share.
Health registries. Both new keys registered in BOTH `api/health.js`
SEED_META (60-day alert threshold) and `api/seed-health.js`
SEED_DOMAINS (43200min interval). feedback_two_health_endpoints_must_match.
Bundle wiring. `seed-bundle-resilience-recovery` Reexport-Share
timeout bumped 60s → 300s (Comtrade + retry can take 2-3 min
worst-case). Ordering preserved: Reexport-Share before Sovereign-
Wealth so the SWF seeder reads a freshly-written key in the same
cron tick.
Deletions. YAML + loader + 7 obsolete loader tests removed; single
source of truth is now Comtrade → Redis.
Prereq. Stacks on PR #3384 (feat/bundle-runner-env-sigterm)
which adds BUNDLE_RUN_STARTED_AT_MS env injection + runSeed
SIGTERM cleanup. This PR's bundle-freshness guard depends on
that env variable.
Tests (19 new, 7 deleted, +12 net):
- Pure math: parseComtradeFlowResponse, computeShareFromFlows,
clampShare, declareRecords + credential-leak source scan (15)
- Integration (Gap #2 regression guards): SWF seeder loadReexport
ShareFromRedis — fresh/absent/malformed/stale-meta/missing-meta (5)
- Health registry dual-registry drift guard — scoped to this PR's
keys, respecting pre-existing asymmetry (4)
- Bundle-ordering + timeout assertions (2)
Phase 0 cohort validation committed to plan. Full test suite
passes: 6,881 tests.
* fix(resilience): address P1/P2 review findings — adopt shared helpers, pin freshness boundary
Addresses PR #3385 review findings:
#257 (P1) consumer — `seed-sovereign-wealth.mjs` imports the shared
`getBundleRunStartedAtMs` helper from `_seed-utils.mjs` (added in the
prereq commit) instead of its own `getBundleStartMs`. Single source of
truth for the bundle-freshness contract.
#258 (P2) — `seed-recovery-reexport-share.mjs` isMain guard uses the
canonical `pathToFileURL(process.argv[1]).href === import.meta.url`
form instead of basename-suffix matching. Handles symlinks, case-
different paths on macOS HFS+, and Windows path separators without
string munging.
#260 (P2) consumer — Sovereign-Wealth declares `dependsOn:
['Reexport-Share']` in the bundle spec. `_bundle-runner.mjs` (prereq
commit) now enforces topological order on load and throws on
violation — replaces the previous prose-comment ordering contract.
#261 (P2) — added a test to `tests/seed-sovereign-wealth-reads-redis-
reexport-share.test.mts` pinning the inclusive-boundary semantic:
`fetchedAtMs === bundleStartMs` must be treated as FRESH. Guards
against a future refactor to `<=` that would silently reject peers
writing at the very first millisecond of the bundle run.
Rebased onto updated prereq. Full test suite: 6,886 tests pass (+5 net).
* fix(resilience): freshness gate skipped in standalone mode; meta still required
Review catch: the previous `bundleStartMs = Date.now()` fallback made
standalone/manual `seed-sovereign-wealth.mjs` runs ALWAYS reject any
previously-seeded re-export-share meta as "stale" — even when the
operator ran the Reexport seeder milliseconds beforehand. Defeated
the point of the peer-read path outside the bundle.
With `getBundleRunStartedAtMs()` now returning null outside a bundle
(companion commit on the prereq branch), the consumer only applies
the freshness gate when `bundleStartMs != null`. Standalone runs
accept any `fetchedAt` — the operator is responsible for ordering.
Two guards survive the change:
- Meta MUST exist (absence = peer-outage fail-safe, both modes)
- In-bundle: meta MUST be at or after `BUNDLE_RUN_STARTED_AT_MS`
Two new tests pin both modes:
- standalone: accepts meta written 10 min before this process started
- standalone: still rejects missing meta (peer-outage fail-safe
survives gate bypass)
Rebased onto updated prereq. Full test suite: 6,888 tests (+2 net).
* fix(resilience): filter world-aggregate Comtrade rows + skip final-retry sleep
Greptile review of PR #3385 flagged two P2s in the Comtrade seeder.
Finding #3 (parseComtradeFlowResponse double-count risk):
`cmdCode=TOTAL` without a partner filter currently returns only
world-aggregate rows in practice — but `parseComtradeFlowResponse`
summed every row unconditionally. A future refactor adding per-
partner querying would silently double-count (world-aggregate row +
partner-level rows for the same year), cutting the derived share in
half with no test signal.
Fix: explicit `partnerCode ∈ {'0', 0, null/undefined}` filter. Matches
current empirical behavior (aggregate-only responses) and makes the
construct robust to a future partner-level query.
Finding #4 (wasted backoff on final retry):
429 and 5xx branches slept `backoffMs` before `continue`, but on
`attempt === RETRY_MAX_ATTEMPTS` the loop condition fails immediately
after — the sleep was pure waste. Added early-return (parallel to the
existing pattern in the network-error catch branch) so the final
attempt exits the retry loop at the first non-success response
without extra latency.
Tests:
- 3 new `parseComtradeFlowResponse` variants: world-only filter,
numeric-0 partnerCode shape, rows without partnerCode field
- Existing tests updated: the double-count assertion replaced with
a "per-partner rows must NOT sum into the world-aggregate total"
assertion that pins the new contract
Rebased onto updated prereq. Full test suite: 6,890 tests (+2 net).
|
||
|
Elie Habib
|
84ee2beb3e |
feat(energy): Energy Atlas end-to-end — pipelines + storage + shortages + disruptions + country drill-down (#3294)
* feat(energy): pipeline registries (gas + oil) — evidence-based schema
Day 6 of the Energy Atlas Release 1 plan (Week 2). First curated asset
registry for the atlas — the real gap vs GEF.
## Curated data (critical assets only, not global completeness)
scripts/data/pipelines-gas.json — 12 critical gas lines:
Nord Stream 1/2 (offline; Swedish EEZ sabotage 2022; EU sanctions refs),
TurkStream, Yamal–Europe (offline; Polish counter-sanctions),
Brotherhood/Soyuz (offline; Ukraine transit expired 2024-12-31),
Power of Siberia, Dolphin, Medgaz, TAP, TANAP,
Central Asia–China, Langeled.
scripts/data/pipelines-oil.json — 12 critical oil lines:
Druzhba North/South (N offline per EU 2022/879; S under landlocked
derogation), CPC, ESPO (+ price-cap sanction ref), BTC, TAPS,
Habshan–Fujairah (Hormuz bypass), Keystone, Kirkuk–Ceyhan (offline
since 2023 ICC ruling), Baku–Supsa, Trans-Mountain (TMX expansion
May 2024), ESPO spur to Daqing.
Scope note: 75+ each is Week 2b work via GEM bulk import. Today's cut
is curated from first-hand operator disclosures + regulator filings so
I can stand behind every evidence field.
## Evidence-based schema (not conclusion labels)
Per docs/methodology/pipelines.mdx: no bare `sanctions_blocked` field.
Every pipeline carries an evidence bundle with `physicalState`,
`physicalStateSource`, `operatorStatement`, `commercialState`,
`sanctionRefs[]`, `lastEvidenceUpdate`, `classifierVersion`,
`classifierConfidence`. The public badge (`flowing|reduced|offline|
disputed`) is derived server-side from this bundle at read time.
## Seeder
scripts/seed-pipelines.mjs — single process publishes BOTH keys
(energy:pipelines:{gas,oil}:v1) via two runSeed() calls. Tiny datasets
(<20KB each) so co-location is cheap and guarantees classifierVersion
consistency.
Conventions followed (worldmonitor-bootstrap-registration skill):
- TTL 21d = 3× weekly cadence (gold-standard per
feedback_seeder_gold_standard.md)
- maxStaleMin 20_160 = 2× cadence (health-maxstalemin-write-cadence skill)
- sourceVersion + schemaVersion + recordCount + declareRecords wired
(seed-contract-foundation)
- Zero-case explicitly NOT allowed — MIN_PIPELINES_PER_REGISTRY=8 floor
## Health registration (dual, per feedback_two_health_endpoints_must_match)
- api/health.js: BOOTSTRAP_KEYS adds pipelinesGas + pipelinesOil;
SEED_META adds both with maxStaleMin=20_160.
- api/seed-health.js: mirror entries with intervalMin=10_080 (maxStaleMin/2).
## Bundle registration
scripts/seed-bundle-energy-sources.mjs adds a single Pipelines entry
(not two) because seed-pipelines.mjs publishes both keys in one run —
listing oil separately would double-execute. Monitoring of the oil key
staleness happens in api/health.js instead.
## Tests (tests/pipelines-registry.test.mts)
17 passing node:test assertions covering:
- Schema validation (both registries pass validateRegistry)
- Identity resolution (no id collisions, id matches object key)
- Country ISO2 normalization (from/to/transit all match /^[A-Z]{2}$/)
- Endpoint geometry within Earth bounds
- Evidence rigor: non-flowing badges require at least one supporting
evidence source (operator statement / sanctionRefs / ais-relay /
satellite / press)
- ClassifierConfidence in 0..1
- Commodity/capacity pairing (gas uses capacityBcmYr, oil uses
capacityMbd — mixing = test fail)
- validateRegistry rejects: empty object, null, no-evidence fixtures,
below-floor counts
Typecheck clean (both tsconfig.json and tsconfig.api.json).
Next: Day 7 will add list-pipelines / get-pipeline-detail RPCs in
supply-chain/v1. Day 8 ships PipelineStatusPanel with DeckGL PathLayer
consuming the registry.
* fix(energy): split seed-pipelines.mjs into two entry points — runSeed hard-exits
High finding from PR review. scripts/seed-pipelines.mjs called runSeed()
twice in one process and awaited Promise.all. But runSeed() in
scripts/_seed-utils.mjs hard-exits via process.exit on ~9 terminal paths
(lines 816, 820, 839, 888, 917, 989, plus fetch-retry 946, fatal 859,
skipped-lock 81). The first runSeed to reach any terminal path exits the
entire node process, so the second runSeed's resolve never fires — only
one of energy:pipelines:{gas,oil}:v1 would ever be written.
Since the bundle scheduled seed-pipelines.mjs exactly once, and both
api/health.js and api/seed-health.js expect both keys populated, the
other registry would stay permanently EMPTY/STALE after deploy.
Fix: split into two entry-point scripts around a shared utility.
- scripts/_pipeline-registry.mjs (NEW, was seed-pipelines.mjs) — shared
helpers ONLY. Exports GAS_CANONICAL_KEY, OIL_CANONICAL_KEY,
PIPELINES_TTL_SECONDS, MAX_STALE_MIN, buildGasPayload, buildOilPayload,
validateRegistry, recordCount, declareRecords. Underscore prefix marks
it as non-entry-point (matches _seed-utils.mjs / _seed-envelope-source.mjs
convention).
- scripts/seed-pipelines-gas.mjs (NEW) — imports from the shared module,
single runSeed('energy','pipelines-gas',…) call.
- scripts/seed-pipelines-oil.mjs (NEW) — same shape, oil.
- scripts/seed-bundle-energy-sources.mjs — register BOTH seeders (not one).
- scripts/seed-pipelines.mjs — deleted.
- tests/pipelines-registry.test.mts — update import path to the shared
module. All 17 tests still pass.
Typecheck clean (both configs). Tests pass. No other consumers import
from the deleted script.
* fix(energy): complete pipeline bootstrap registration per 4-file checklist
High finding from PR review. My earlier PR description claimed
worldmonitor-bootstrap-registration was complete, but I only touched two
of the four registries (api/health.js + api/seed-health.js). The bootstrap
hydration payload itself (api/bootstrap.js) and the shared cache-keys
registry (server/_shared/cache-keys.ts) still had no entry for either
pipeline key, so any consumer that reads bootstrap data would see
pipelinesGas/pipelinesOil as missing on first load.
Files updated this commit:
- api/bootstrap.js — KEYS map + SLOW_KEYS set both gain pipelinesGas +
pipelinesOil. Placed next to sprPolicies (same curated-registry cadence
and tier). Slow tier is correct: weekly cron, not needed on first paint.
- server/_shared/cache-keys.ts — PIPELINES_GAS_KEY + PIPELINES_OIL_KEY
exported constants (matches SPR_POLICIES_KEY pattern), BOOTSTRAP_KEYS map
entries, and BOOTSTRAP_TIERS entries (both 'slow').
Not touched (intentional):
- server/gateway.ts — pipeline data is free-tier per the Energy Atlas
plan; no PREMIUM_RPC_PATHS entry required. Energy Atlas monetization
hooks (scenario runner, MCP tools, subscriptions) are Release 2.
Full 4-file checklist now complete:
✅ server/_shared/cache-keys.ts (this commit)
✅ api/bootstrap.js (this commit)
✅ api/health.js (earlier in PR)
✅ api/seed-health.js (earlier in PR — dual-registry rule)
Typecheck clean (both configs).
* feat(energy): ListPipelines + GetPipelineDetail RPCs with evidence-derived badges
Day 7 of the Energy Atlas Release 1 plan (Week 2). Exposes the pipeline
registries (shipped in Day 6) via two supply-chain RPCs and ships the
evidence-to-badge derivation server-side.
## Proto
proto/worldmonitor/supply_chain/v1/list_pipelines.proto — new:
- ListPipelinesRequest { commodity_type?: 'gas' | 'oil' }
- ListPipelinesResponse { pipelines[], fetched_at, classifier_version, upstream_unavailable }
- GetPipelineDetailRequest { pipeline_id (required, query-param) }
- GetPipelineDetailResponse { pipeline?, revisions[], fetched_at, unavailable }
- PipelineEntry — wire shape mirroring scripts/data/pipelines-{gas,oil}.json
+ a server-derived public_badge field
- PipelineEvidence, OperatorStatement, SanctionRef, LatLon, PipelineRevisionEntry
service.proto adds both rpc methods with HTTP_METHOD_GET + path bindings:
/api/supply-chain/v1/list-pipelines
/api/supply-chain/v1/get-pipeline-detail
`make generate` regenerated src/generated/{client,server}/… + docs/api/
OpenAPI json/yaml.
## Evidence-derivation
server/worldmonitor/supply-chain/v1/_pipeline-evidence.ts — new.
derivePublicBadge(evidence) → 'flowing' | 'reduced' | 'offline' | 'disputed'
is deterministic + versioned (DERIVER_VERSION='badge-deriver-v1').
Rules (first match wins):
1. offline + sanctionRef OR expired/suspended commercial → offline
2. offline + operator statement → offline
3. offline + only press/ais/satellite → disputed (single-source negative claim)
4. reduced → reduced
5. flowing → flowing
6. unknown / malformed → disputed
Staleness guard: non-flowing badges on >14d-old evidence demote to
disputed. Flowing is the optimistic default — stale "still flowing" is
safer than stale "offline". Matches seed-pipelines-{gas,oil}.mjs maxStaleMin.
Tests (tests/pipeline-evidence-derivation.test.mts) — 15 passing cases
covering happy paths, disputed fallbacks, staleness guard, versioning.
## Handlers
server/worldmonitor/supply-chain/v1/list-pipelines.ts
- Reads energy:pipelines:{gas,oil}:v1 via getCachedJson.
- projectPipeline() narrows the Upstash `unknown` into PipelineEntry
shape + calls derivePublicBadge.
- Honors commodity_type filter (skip the opposite registry's Redis read
when the client pre-filters).
- Returns upstream_unavailable=true when BOTH registries miss.
server/worldmonitor/supply-chain/v1/get-pipeline-detail.ts
- Scans both registries by id (ids are globally unique per
tests/pipelines-registry.test.mts).
- Empty revisions[] for now; auto-revision log wires up in Week 3.
handler.ts registers both into supplyChainHandler.
## Gateway
server/gateway.ts adds 'static' cache-tier for both new RPC paths
(registry is slow-moving; 'static' matches the other read-mostly
supply-chain endpoints).
## Consumer wiring
Not in this commit — PipelineStatusPanel (Day 8) is what will call
listPipelines/getPipelineDetail via the generated client. pipelinesGas
+ pipelinesOil stay in PENDING_CONSUMERS until Day 8.
Typecheck clean (both configs). 15 new tests + 17 registry tests all pass.
* feat(energy): PipelineStatusPanel — evidence-backed status table + drawer
Day 8 of the Energy Atlas Release 1 plan. First consumer of the Day 6–7
registries + RPCs.
## What this PR adds
- src/components/PipelineStatusPanel.ts — new panel (id=pipeline-status).
* Bootstrap-hydrates from pipelinesGas + pipelinesOil for instant first
paint; falls through to listPipelines() RPC if bootstrap misses.
Background re-fetch runs on every render so a classifier-version bump
between bootstrap stamp and first view produces a visible update.
* Table rows sorted non-flowing-first (offline / reduced / disputed
before flowing) — what an atlas reader cares about.
* Click-to-expand drawer calls getPipelineDetail() lazily — operator
statements, sanction refs (with clickable source URLs), commercial
state, classifier version + confidence %, capacity + route metadata.
* publicBadge color-chip palette matches the methodology doc.
* Attribution footer with GEM (CC-BY 4.0) credit + classifier version.
- src/components/index.ts — barrel export.
- src/app/panel-layout.ts — import + createPanel('pipeline-status', …).
- src/config/panels.ts — ENERGY_PANELS adds 'pipeline-status' at priority 1.
## PENDING_CONSUMERS cleanup
tests/bootstrap.test.mjs — removes 'pipelinesGas' + 'pipelinesOil' from
the allowlist. The invariant "every bootstrap key has a getHydratedData
consumer" now enforces real wiring for these keys: the panel literally
calls getHydratedData('pipelinesGas') and getHydratedData('pipelinesOil').
Future regressions that remove the consumer will fail pre-push.
## Consumer contract verified
- 67 tests pass including bootstrap.test.mjs consumer coverage check.
- Typecheck clean.
- No DeckGL PathLayer in this commit — existing 'pipelines-layer' has a
separate data source, so modifying DeckGLMap.ts to overlay evidence-
derived badges on the map is a follow-up commit to avoid clobbering.
## Out of scope for Day 8 (next steps on same PR)
- DeckGL PathLayer integration (color pipelines on the main map by
publicBadge, click-to-open this drawer) — Day 8b commit.
- Storage facility registry + StorageFacilityMapPanel — Days 9-10.
* fix(energy): PipelineStatusPanel bootstrap path — client-side badge derivation
High finding from PR review. The Day-8 panel crashed on first paint
whenever bootstrap hydration succeeded, because:
- Bootstrap hydrates raw scripts/data/pipelines-{gas,oil}.json verbatim.
- That JSON does NOT include publicBadge — that field is only added by
the server handler's projectPipeline() in list-pipelines.ts.
- PipelineStatusPanel passed raw entries into badgeChip(), which called
badgeLabel(undefined).charAt(0) → TypeError.
The background RPC refresh that would have repaired the data never ran
because the panel threw before reaching it. So the exact bootstrap path
newly wired in commit
|
||
|
Elie Habib
|
64c906a406 |
feat(eia): gold-standard /api/eia/petroleum (Railway seed → Redis → Vercel reads only) (#3161)
* feat(eia): move /api/eia/petroleum to gold-standard (Railway seed → Redis → Vercel reads only)
Live api.eia.gov fetches from the Vercel edge function were causing
FUNCTION_INVOCATION_TIMEOUT 504s on /api/eia/petroleum (Sydney edge →
US origin with no timeout, no cache, no stale fallback — one EIA blip
blew the 25s budget).
- New seeder scripts/seed-eia-petroleum.mjs — fetches WTI/Brent/
production/inventory from api.eia.gov with per-fetch 15s timeouts,
writes energy:eia-petroleum:v1 with the {_seed, data} envelope.
Accepts 1-of-4 series; 0-of-4 routes to contract-mode RETRY so
seed-meta stays stale and the bundle retries on next cron.
- Bundled into seed-bundle-energy-sources.mjs (daily, 90s timeout) —
no new Railway service needed.
- Rewrote api/eia/[[...path]].js as a Redis-only reader via
readJsonFromUpstash. Same response shape for backward compat with
widgets/MCP/external callers. 503 + Retry-After on miss (never 504).
- Registered eiaPetroleum in api/health.js STANDALONE_KEYS + gated as
ON_DEMAND_KEYS for the deploy window; promote to SEED_META
(maxStaleMin: 4320) in a follow-up after ~7 days of clean cron.
- Tests: 14 seeder unit tests + 9 edge handler tests.
Audit result: /api/eia/petroleum was the only Vercel route fetching
dashboard data live. Every other fetch(https://…) in api/ is
auth/payments/notifications/user-initiated enrichment.
* fix(eia): close silent-stale window — add SEED_META + seed-health registration
Review finding on PR #3161: without a SEED_META entry, readSeedMeta
returns seedStale: null and classifyKey never reaches STALE_SEED.
That meant a broken Railway cron or missing EIA_API_KEY after the first
successful seed would keep /api/eia/petroleum serving stale data for
up to 7 days (TTL) while /api/health reported OK.
- api/health.js: add SEED_META.eiaPetroleum with maxStaleMin=4320
(72h = 3× daily bundle cadence). Keep eiaPetroleum in ON_DEMAND_KEYS
so the Vercel-instant / Railway-delayed deploy window doesn't CRIT
on first seed, but stale-after-seed now properly fires STALE_SEED.
- api/seed-health.js: register energy:eia-petroleum in SEED_DOMAINS
(intervalMin=1440) so the secondary health endpoint reports it too.
- Updated ON_DEMAND_KEYS comment to reflect freshness is now enforced.
|
||
|
Elie Habib
|
935417e390 |
chore(relay): socialVelocity + wsbTickers to hourly fetch (6x Reddit traffic reduction) (#3135)
* chore(relay): socialVelocity + wsbTickers to hourly fetch (was 10min) Reduce Reddit rate-limiting blast radius. Both seeders fetch 5 subreddits combined (2 for SV: worldnews, geopolitics; 3 for WSB: wallstreetbets, stocks, investing) with no proxy or OAuth. Reddit's behavioral heuristic for datacenter IPs consistently flags the Railway IP after ~50min of 10-min polling and returns HTTP 403 on every subsequent cycle until the container restarts with a new IP. Evidence (2026-04-16 ais-relay log): 13:32-14:22 UTC: 6 successful 10-min cycles for both seeders 16:06-16:16 UTC: 2 more successful cycles after a restart 16:26 UTC: BOTH subs flip to HTTP 403 simultaneously 16:36, 16:46, 16:56: every cycle, all 5 subreddits return 403 Dropping success-path frequency from 6/hour to 1/hour cuts the traffic Reddit's heuristic sees by 6x. On failure path the 20-min retry is kept as-is — during a block we've already been flagged, so extra retries don't make it worse. Changes: - SOCIAL_VELOCITY_INTERVAL_MS: 10min → 60min - SOCIAL_VELOCITY_TTL: 30min → 3h (3× new interval) - WSB_TICKERS_INTERVAL_MS: 10min → 60min - WSB_TICKERS_TTL: 30min → 3h (3× new interval) - api/health.js maxStaleMin: 30min → 180min for both (3× interval) - api/seed-health.js intervalMin: 15 → 90 for wsb-tickers (maxStaleMin / 2) Proper fix (proxy fallback or Reddit OAuth) deferred. * fix(seed-health): add socialVelocity parity entry — greptile P2 Review finding on PR #3135: wsbTickers was bumped from intervalMin=15 to 90 but socialVelocity had no seed-health.js entry at all. Both Reddit seeders now share the same 60-min cadence; adding the missing entry gives parity. P2-1 (malformed comment lines 5682-5683) is a false positive — verified the lines do start with '//' in the file. |
||
|
Elie Habib
|
e32d9b631c |
feat(market): Hyperliquid perp positioning flow as leading indicator (#3074)
* feat(market): Hyperliquid perp positioning flow as leading indicator Adds a 4-component composite (funding × volume × OI × basis) "positioning stress" score for ~14 perps spanning crypto (BTC/ETH/SOL), tokenized gold (PAXG), commodity perps (WTI, Brent, Gold, Silver, Pt, Pd, Cu, NatGas), and FX perps (EUR, JPY). Polls Hyperliquid /info every 5min via Railway cron; publishes a single self-contained snapshot with embedded sparkline arrays (60 samples = 5h history). Surfaces as a new "Perp Flow" tab in CommoditiesPanel with separate Commodities / FX sections. Why: existing CFTC COT is weekly + US-centric; market quotes are price-only. Hyperliquid xyz: perps give 24/7 global positioning data that has been shown to lead spot moves on commodities and FX by minutes-to-hours. Implementation: - scripts/seed-hyperliquid-flow.mjs — pure scoring math, symbol whitelist, content-type + schema validation, prior-state read via readSeedSnapshot(), warmup contract (first run / post-outage zeroes vol/OI deltas), missing-symbol carry-forward, $500k/24h min-notional guard to suppress thin xyz: noise. TTL 2700s (9× cadence). - proto/worldmonitor/market/v1/get_hyperliquid_flow.proto + service.proto registration; make generate regenerated client/server bindings. - server/worldmonitor/market/v1/get-hyperliquid-flow.ts — getCachedJson reader matching get-cot-positioning.ts seeded-handler pattern. - server/gateway.ts cache-tier entry (medium). - api/health.js: hyperliquidFlow registered with maxStaleMin:15 (3× cadence) + transitional ON_DEMAND_KEYS gate for the first ~7 days of bake-in. - api/seed-health.js mirror with intervalMin:5. - scripts/seed-bundle-market-backup.mjs entry (NIXPACKS auto-redeploy on scripts/** watch). - src/components/MarketPanel.ts: CommoditiesPanel grows a Perp Flow tab + fetchHyperliquidFlow() RPC method; OI Δ1h derived from sparkOi tail. - src/App.ts: prime via primeVisiblePanelData() + recurring refresh via refreshScheduler.scheduleRefresh() at 5min cadence (panel does NOT own setInterval; matches the App.ts:1251 lifecycle convention). - 28 unit tests covering scoring parity, warmup flag, min-notional guard, schema rejection, missing-symbol carry-forward, post-outage cold start, sparkline cap, alert threshold. Tests: test:data 5169/5169, hyperliquid-flow-seed 28/28, route-cache-tier 5/5, typecheck + typecheck:api green. One pre-existing test:sidecar failure (cloud-fallback origin headers) is unrelated and reproduces on origin/main. * fix(hyperliquid-flow): address review feedback — volume baseline window, warmup lifecycle, error logging Two real correctness bugs and four review nits from PR #3074 review pass. Correctness fixes: 1. Volume baseline was anchored to the OLDEST 12 samples, not the newest. sparkVol is newest-at-tail (shiftAndAppend), so slice(0, 12) pinned the rolling mean to the first hour of data forever once len >= 12. Volume scoring would drift further from current conditions each poll. Switched to slice(-VOLUME_BASELINE_MIN_SAMPLES) so the baseline tracks the most recent window. Regression test added. 2. Warmup flag flipped to false on the second successful poll while volume scoring still needed 12+ samples to activate. UI told users warmup lasted ~1h but the badge disappeared after 5 min. Tied per-asset warmup to real baseline readiness (coldStart OR vol samples < 12 OR prior OI missing). Snapshot-level warmup = any asset still warming. Three new tests cover the persist-through-baseline-build, clear-once-ready, and missing-OI paths. Review nits: - Handler: bare catch swallowed Redis/parse errors; now logs err.message. - Panel: bare catch in fetchHyperliquidFlow hid RPC 500s; now logs. - MarketPanel.ts: deleted hand-rolled RawHyperliquidAsset; mapHyperliquidFlowResponse now takes GetHyperliquidFlowResponse from the generated client so proto drift fails compilation instead of silently. - Seeder: added @ts-check + JSDoc on computeAsset per type-safety rule. - validateUpstream: MAX_UPSTREAM_UNIVERSE=2000 cap bounds memory. - buildSnapshot: logs unknown xyz: perps upstream (once per run) so ops sees when Hyperliquid adds markets we could whitelist. Tests: 37/37 green; typecheck + typecheck:api clean. * fix(hyperliquid-flow): wire bootstrap hydration per AGENTS.md mandate Greptile review caught that AGENTS.md:187 mandates new data sources be wired into bootstrap hydration. Plan had deferred this on "lazy deep-dive signal" grounds, but the project convention is binding. - server/_shared/cache-keys.ts: add hyperliquidFlow to BOOTSTRAP_CACHE_KEYS + BOOTSTRAP_TIERS ('slow' — non-blocking, page-load-parallel). - api/bootstrap.js: add to inlined BOOTSTRAP_CACHE_KEYS + SLOW_KEYS so bootstrap.test.mjs canonical-mirror assertions pass. - src/components/MarketPanel.ts: - Import getHydratedData from @/services/bootstrap. - New mapHyperliquidFlowSeed() normalizes the raw seed-JSON shape (numeric fields) into HyperliquidFlowView. The RPC mapper handles the proto shape (string-encoded numbers); bootstrap emits the raw blob. - fetchHyperliquidFlow now reads hydrated data first, renders immediately, then refreshes from RPC — mirrors FearGreedPanel pattern. Tests: 72/72 green (bootstrap + cache-tier + hyperliquid-flow-seed). |
||
|
Elie Habib
|
cd5ed0d183 |
feat(seeds): BIS DSR + property prices (2 of 7) (#3048)
* feat(seeds): BIS DSR + property prices (2 of 7) Ships 2 of 7 BIS dataflows flagged as genuinely new signals in #3026 — the rest are redundant with IMF/WB or are low-fit global aggregates. New seeder: scripts/seed-bis-extended.mjs - WS_DSR household debt service ratio (% income, quarterly) - WS_SPP residential property prices (real index, quarterly) - WS_CPP commercial property prices (real index, quarterly) Gold-standard pattern: atomic publish + writeExtraKey for extras, retry on missing startPeriod, TTL = 3 days (3× 12h cron), runSeed drives seed-meta:economic:bis-extended. Series selection scores dimension matches (PP_VALUATION=R / UNIT_MEASURE=628 for property, DSR_BORROWERS=P / DSR_ADJUST=A for DSR), then falls back to observation count. Wired into: - bootstrap (slow tier) + cache-keys.ts - api/health.js (STANDALONE_KEYS + SEED_META, maxStaleMin = 24h) - api/mcp.ts get_economic_data tool (_cacheKeys + _freshnessChecks) - resilience macroFiscal: new householdDebtService sub-metric (weight 0.05, currentAccountPct rebalanced 0.3 → 0.25) - Housing Cycle tile on CountryDeepDivePanel (Economic Indicators card) with euro-area (XM) fallback for EU member states - seed-bundle-macro Railway cron (BIS-Extended, 12h interval) Tests: tests/bis-extended-seed.test.mjs covers CSV parsing, series selection, quarter math + YoY. Updated resilience golden-value tests for the macroFiscal weight rebalance. Closes #3026 https://claude.ai/code/session_01DDo39mPD9N2fNHtUntHDqN * fix(resilience): unblock PR #3048 on #3046 stack - rebase onto #3046; final macroFiscal weights: govRevenue 0.40, currentAccount 0.20, debtGrowth 0.20, unemployment 0.15, householdDebtService 0.05 (=1.00) - add updateHousingCycle? stub to CountryBriefPanel interface so country-intel dispatch typechecks - add HR to EURO_AREA fallback set (joined euro 2023-01-01) - seed-bis-extended: extend SPP/CPP TTLs when DSR fetch returns empty so the rejected publish does not silently expire the still-good property keys - update resilience goldens for the 5-sub-metric macroFiscal blend * fix(country-brief): housing tile renders em-dash for null change values The new Housing cycle tile used `?? 0` to default qoqChange/yoyChange/change when missing, fabricating a flat "0.0%" label (with positive-trend styling) for countries with no prior comparable period. Fetch path and builders correctly return null; the panel was coercing it. formatPctTrend now accepts null|undefined and returns an em-dash, matching how other cards surface unavailable metrics. Drop the `?? 0` fallbacks at the three housing call sites. * fix(seed-health): register economic:bis-extended seed-meta monitoring 12h Railway cron writes seed-meta:economic:bis-extended but it was missing from SEED_DOMAINS, so /api/seed-health never reported its freshness. intervalMin=720 matches maxStaleMin/2 (1440/2) from api/health.js. * fix(seed-bis-extended): decouple DSR/SPP/CPP so one fetch failure doesn't block the others Previously validate() required data.entries.length > 0 on the DSR slice after publishTransform pulled it out of the aggregate payload. If WS_DSR fetch failed but WS_SPP / WS_CPP succeeded, validate() rejected the publish → afterPublish() never ran → fresh SPP/CPP data was silently discarded and only the old snapshots got a TTL bump. This treats the three datasets as independent: - SPP and CPP are now published (or have their existing TTLs extended) as side-effects of fetchAll(), per-dataset. A failure in one never affects the others. - DSR continues to flow through runSeed's canonical-key path. When DSR is empty, publishTransform yields { entries: [] } so atomicPublish skips the canonical write (preserving the old DSR snapshot); runSeed's skipped branch extends its TTL and refreshes seed-meta. Shape B (one runSeed call, semantics changed) chosen over Shape A (three sequential runSeed calls) because runSeed owns the lock + process.exit lifecycle and can't be safely called three times in a row, and Shape B keeps the single aggregate seed-meta:economic:bis-extended key that health.js already monitors. Tests cover both failure modes: - DSR empty + SPP/CPP healthy → SPP/CPP written, DSR TTL extended - DSR healthy + SPP/CPP empty → DSR written, SPP/CPP TTLs extended * fix(health): per-dataset seed-meta for BIS DSR/SPP/CPP Health was pointing bisDsr / bisPropertyResidential / bisPropertyCommercial at the shared seed-meta:economic:bis-extended key, which runSeed refreshes on every run (including its validation-failed "skipped" branch). A DSR-only outage therefore left bisDsr reporting fresh in api/health.js while the resilience scorer consumed stale/missing economic:bis:dsr:v1 data. Write a dedicated seed-meta key per dataset ONLY when that dataset actually published fresh entries. The aggregate bis-extended key stays as a "seeder ran" signal in api/seed-health.js. * fix(seed-bis-extended): write DSR seed-meta only after atomicPublish succeeds Previously fetchAll() wrote seed-meta:economic:bis-dsr inline before runSeed/atomicPublish ran. If atomicPublish then failed (Redis hiccup, validate rejection, etc.), seed-meta was already bumped — health would report DSR fresh while the canonical key was stale. Move the DSR seed-meta write into a dsrAfterPublish callback passed to runSeed via the existing afterPublish hook, which fires only after a successful canonical publish. SPP/CPP paths already used this ordering inside publishDatasetIndependently; this brings DSR in line. Adds a regression test exercising dsrAfterPublish with mocked Upstash: populated DSR -> single SET on seed-meta key; null/empty DSR -> zero Redis calls. * fix(resilience): per-dataset BIS seed-meta keys in freshness overrides SOURCE_KEY_META_OVERRIDES previously collapsed economic:bis:dsr:v1 and both property-* sourceKeys onto the aggregate seed-meta:economic:bis-extended key. api/health.js (SEED_META) writes per-dataset keys (seed-meta:economic:bis-dsr / bis-property-residential / bis-property-commercial), so a DSR-only outage showed stale in /api/health but the resilience dimension freshness code still reported macroFiscal inputs as fresh. Map each BIS sourceKey to its dedicated seed-meta key to match health.js. The aggregate bis-extended key is still written by the seeder and read by api/seed-health.js as a "seeder ran" signal, so it is retained upstream. * fix(bis): prefer households in DSR + per-dataset freshness in MCP Greptile review catches on #3048: 1. buildDsr() was selecting DSR_BORROWERS='P' (private non-financial) while the UI labels it "Household DSR" and resilience scoring uses it as `householdDebtService`. Changed to 'H' (households). Countries without an H series now get dropped rather than silently mislabeled. 2. api/mcp.ts get_economic_data still read only the aggregate seed-meta:economic:bis-extended for freshness. If DSR goes stale while SPP/CPP keep publishing, MCP would report the BIS block as fresh even though one of its returned keys is stale. Swapped to the three per-dataset seed-meta keys (bis-dsr, bis-property-residential, bis-property-commercial), matching the fix already applied to /api/health and the resilience dimension-freshness pipeline. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Elie Habib
|
e7ef14aa02 |
fix(health): heal portwatch-disruptions + three stale-registry false alarms (#3051)
* fix(health): heal portwatch-disruptions + three stale-registry false alarms * fix(resilience): log Upstash non-2xx when writing ranking seed-meta fetch() doesn't throw on HTTP errors, so a 401/429/500 from Upstash would be treated as success — the new meta write would fail silently and /api/health would keep alerting with no diagnostic log. Check resp.ok explicitly and log status + body snippet on failure. Greptile review catch on #3051. * fix(health): sync seed-health.js portwatch cadence with api/health.js (WEEK) Companion fix to the same logical bug on api/health.js: api/seed-health.js still read 'portwatch:chokepoints-ref' as a daily cron (intervalMin 1440), so its stale threshold (intervalMin*2 = 48h) would still flag a false stale even though api/health.js was updated to 14d. Both endpoints now agree at 14d for a WEEK-cadence seeder. Greptile review catch on #3051. |
||
|
Elie Habib
|
f5d8ff9458 |
feat(seeds): Eurostat house prices + quarterly debt + industrial production (#3047)
* feat(seeds): Eurostat house prices + quarterly debt + industrial production Adds three new Eurostat overlay seeders covering all 27 EU members plus EA20 and EU27_2020 aggregates (issue #3028): - prc_hpi_a (annual house price index, 10y sparkline, TTL 35d) key: economic:eurostat:house-prices:v1 complements BIS WS_SPP (#3026) for the Housing cycle tile - gov_10q_ggdebt (quarterly gov debt %GDP, 8q sparkline, TTL 14d) key: economic:eurostat:gov-debt-q:v1 upgrades National Debt card cadence from annual IMF to quarterly for EU - sts_inpr_m (monthly industrial production, 12m sparkline, TTL 5d) key: economic:eurostat:industrial-production:v1 feeds "Real economy pulse" sparkline on Economic Indicators card Shared JSON-stat parser in scripts/_eurostat-utils.mjs handles the EL/GR and EA20 geo quirks and returns full time series for sparklines. Wires each seeder into bootstrap (SLOW_KEYS), health registries (keys + seed-meta thresholds matched to cadence), macro seed bundle, cache-keys shared module, and the MCP tool registry (get_eu_housing_cycle, get_eu_quarterly_gov_debt, get_eu_industrial_production). MCP tool count updated to 31. Tests cover JSON-stat parsing, sparkline ordering, EU-only coverage gating (non-EU geos return null so panels never render blank tiles), validator thresholds, and registry wiring across all surfaces. https://claude.ai/code/session_01Tgm6gG5yUMRoc2LRAKvmza * fix(bootstrap): register new Eurostat keys in tiers, defer consumers Adds eurostatHousePrices/GovDebtQ/IndProd to BOOTSTRAP_TIERS ('slow') to match SLOW_KEYS in api/bootstrap.js, and lists them as PENDING_CONSUMERS in the hydration coverage test (panel wiring lands in follow-up). * fix(eurostat): raise seeder coverage thresholds to catch partial publishes The three Eurostat overlay seeders (house prices, quarterly gov debt, monthly industrial production) all validated with makeValidator(10) against a fixed 29-geo universe (EU27 + EA20 + EU27_2020). A bad run returning only 10-15 geos would pass validation and silently publish a snapshot missing most of the EU. Raise thresholds to near-complete coverage, with a small margin for geos with patchy reporting: - house prices (annual): 10 -> 24 - gov debt (quarterly): 10 -> 24 - industrial prod (monthly): 10 -> 22 (monthly is slightly patchier) Add a guard test that asserts every overlay seeder keeps its threshold >=22 so this regression can't reappear. * fix(seed-health): register 3 Eurostat seed-meta entries house-prices, gov-debt-q, industrial-production were wired in api/health.js SEED_META but missing from api/seed-health.js SEED_DOMAINS, so /api/seed-health would not surface their freshness. intervalMin = health.js maxStaleMin / 2 per convention. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Elie Habib
|
71a6309503 |
feat(seeds): expand IMF WEO coverage — growth, labor, external themes (#3027) (#3046)
* feat(seeds): expand IMF WEO coverage — growth, labor, external themes (#3027) Adds three new SDMX-3.0 seeders alongside the existing imf-macro seeder to surface 15+ additional WEO indicators across ~210 countries at zero incremental API cost. Bundled into seed-bundle-imf-extended.mjs on the same monthly Railway cron cadence. Seeders + Redis keys: - seed-imf-growth.mjs → economic:imf:growth:v1 NGDP_RPCH, NGDPDPC, NGDP_R, PPPPC, PPPGDP, NID_NGDP, NGSD_NGDP - seed-imf-labor.mjs → economic:imf:labor:v1 LUR (unemployment), LP (population) - seed-imf-external.mjs → economic:imf:external:v1 BX, BM, BCA, TM_RPCH, TX_RPCH (+ derived trade balance) - seed-imf-macro.mjs extended with PCPI, PCPIEPCH, GGX_NGDP, GGXONLB_NGDP All four seeders share the 35-day TTL (monthly WEO release) and ~210 country coverage via the same imfSdmxFetchIndicator helper. Wiring: - api/bootstrap.js, api/health.js, server/_shared/cache-keys.ts — register new keys, mark them slow-tier, add SEED_META freshness thresholds matching the imfMacro entry (70d = 2× monthly cadence) - server/worldmonitor/resilience/v1/_dimension-freshness.ts — override entries for the dash-vs-colon seed-meta key shape - _indicator-registry.ts — add LUR as a 4th macroFiscal sub-metric (enrichment tier, weight 0.15); rebalance govRevenuePct (0.5→0.4) and currentAccountPct (0.3→0.25) so weights still sum to 1.0 - _dimension-scorers.ts — read economic:imf:labor:v1 in scoreMacroFiscal, normalize LUR with goalposts 3% (best) → 25% (worst); null-tolerant so weightedBlend redistributes when labor data is unavailable - api/mcp.ts — new get_country_macro tool bundling all four IMF keys with a single freshness check; describes per-country fields including growth/inflation/labor/BOP for LLM-driven country screening - src/services/imf-country-data.ts — bootstrap-cached client + pure buildImfEconomicIndicators helper - src/app/country-intel.ts — async-fetch the IMF bundle on country selection and merge real GDP growth, CPI inflation, unemployment, and GDP/capita rows into the Economic Indicators card; bumps card cap from 3 → 6 rows to fit live signals + IMF context Tests: - tests/seed-imf-extended.test.mjs — 13 unit tests across the three new seeders' pure helpers (canonical keys, ISO3→ISO2 mapping, aggregate filtering, derived savings-investment gap & trade balance, validate thresholds) - tests/imf-country-data.test.mts — 6 tests for the panel rendering helper, including stagflation flag and high-unemployment trend - tests/resilience-dimension-scorers.test.mts — new LUR sub-metric test (tight vs slack labor); existing scoreMacroFiscal coverage assertions updated for the new 4-metric weight split - tests/helpers/resilience-fixtures.mts — labor fixture for NO/US/YE so the existing macroFiscal ordering test still resolves the LUR weight - tests/bootstrap.test.mjs — register imfGrowth/imfLabor/imfExternal as pending consumers (matching imfMacro) - tests/mcp.test.mjs — bump tools/list count 28 → 29 https://claude.ai/code/session_018enRzZuRqaMudKsLD5RLZv * fix(resilience): update macroFiscal goldens for LUR weight rebalance Recompute pinned fixture values after adding labor-unemployment as 4th macroFiscal sub-metric (weight rebalance in _indicator-registry). Also align seed-imf-external tradeBalance to a single reference year to avoid mixing ex/im values from different WEO vintages. * fix(seeds): tighten IMF coverage gates to reject partial snapshots IMF WEO growth/labor/external indicators report ~210 countries for healthy runs. Previous thresholds (150/100/150) let a bad IMF run overwrite a good snapshot with dozens of missing countries and still pass validation. Raise all three to >=190, matching the pattern of sibling seeders and leaving a ~20-country margin for indicators with slightly narrower reporting. Labor validator unions LUR + population (LP), so healthy coverage tracks LP (~210), not LUR (~100) — the old 100 threshold was based on a misread of the union logic. * fix(seed-health): register imf-growth/labor/external seed-meta keys Missing SEED_DOMAINS entries meant the 3 new IMF WEO seeders (growth, labor, external) had no /api/seed-health visibility. intervalMin=50400 matches health.js maxStaleMin/2 (100800/2) — same monthly WEO cadence as imf-macro. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Elie Habib
|
0cdfddc885 |
feat(gold): central-bank reserves via IMF IFS (PR C) (#3038)
* feat(gold): central-bank gold reserves via IMF IFS (PR C) * fix(gold): prefer ounces indicator over USD in IMF IFS candidate list * fix(gold): align seed-health interval with monthly IMF cadence + drop ALG dup Review findings on PR #3038: - api/seed-health.js: intervalMin was 1440 (1 day), which flags stale at 2880min (48h) — contradicted health.js maxStaleMin=44640 (~31 days) and would false-alarm within 2 days on a monthly data source. Bumped to 22320 so both endpoints agree at ~31 days. - seed-gold-cb-reserves ISO3_NAMES: dropped duplicate ALG entry (World Bank variant); DZA is canonical ISO 3166-1 alpha-3 and stays. |
||
|
Elie Habib
|
a8b85e52c8 |
feat(gold): SPDR GLD physical holdings flows (PR B) (#3037)
* feat(gold): SPDR GLD physical holdings flows (PR B) * fix(gold): strip UTF-8 BOM from SPDR CSV header (greptile P2 #3037) |
||
|
Elie Habib
|
ee66b6b5c2 |
feat(gold): Gold Intelligence v2 — positioning depth, returns, drivers (#3034)
* feat(gold): richer Gold Intelligence panel with positioning, returns, drivers * fix(gold): restore leveragedFunds fields and derive P/S netPct in legacy fallback Review catch on PR #3034: 1. seed-cot.mjs stopped emitting leveragedFundsLong/Short during the v2 refactor, which would zero out the Leveraged Funds bars in the existing CotPositioningPanel on the next seed run. Re-read lev_money_* from the TFF rows and keep the fields on the output (commodity rows don't have this breakdown, stay at 0). 2. get-gold-intelligence legacy fallback hardcoded producerSwap.netPct to 0, meaning a pre-v2 COT snapshot rendered a neutral 0% Producer/Swap bar on deploy until seed-cot reran. Derive netPct from dealerLong/dealerShort (same formula as the v2 seeder). OI share stays 0 because open_interest wasn't captured pre-migration; clearly documented now. Tests: added two regression guards (leveragedFunds preserved for TFF, commodity rows emit 0 for those fields). * fix(gold): make enrichment layer monitored and honest about freshness Review catch on PR #3034: - seed-commodity-quotes now writes seed-meta:market:gold-extended via writeExtraKeyWithMeta on every successful run. Partial / failed fetches skip BOTH the data write and the meta bump, so health correctly reports STALE_SEED instead of masking a broken Yahoo fetch with a green check. - Require both gold (core) AND at least one driver/silver before writing, so a half-successful run doesn't overwrite healthy prior data with a degraded payload. - Handler no longer stamps updatedAt with new Date() when the enrichment key is missing. Emits empty string so the panel's freshness indicator shows "Updated —" with a dim dot, matching reality — enrichment is missing, not fresh. - Health: goldExtended entry in STANDALONE_KEYS + SEED_META (maxStaleMin 30, matching commodity quotes), and seed-health.js advertises the domain so upstream monitors pick it up. The panel already gates session/returns/drivers sections on presence, so legacy panels without the enrichment layer stay fully functional. |
||
|
Elie Habib
|
e070a97c3d |
Phase 3 PR2: Weekly regional briefs (LLM seeder + RPC) (#2989)
* feat(intelligence): weekly regional briefs (Phase 3 PR2) Phase 3 PR2 of the Regional Intelligence Model. Adds LLM-powered weekly intelligence briefs per region, completing the core feature set. ## New seeder: scripts/seed-regional-briefs.mjs Standalone weekly cron script (not part of the 6h derived-signals bundle). For each non-global region: 1. Read the latest snapshot via two-hop Redis read 2. Read recent regime transitions from the history log (#2981) 3. Call the LLM once per region with regime trajectory + balance + triggers + narrative context 4. Write structured brief to intelligence:regional-briefs:v1:weekly:{region} with 8-day TTL (survives one missed weekly run) Reuses the same injectable-callLlm + parse-validation + provider-chain pattern from narrative.mjs and weekly-brief.mjs. ## New module: scripts/regional-snapshot/weekly-brief.mjs generateWeeklyBrief(region, snapshot, transitions, opts?) -> { region_id, generated_at, period_start, period_end, situation_recap, regime_trajectory, key_developments[], risk_outlook, provider, model } buildBriefPrompt() — pure prompt builder parseBriefJson() — JSON parser with prose-extraction fallback emptyBrief() — canonical empty shape Global region is skipped. Provider chain: Groq -> OpenRouter. Validate callback ensures only parseable responses pass (narrative.mjs PR #2960 review fix pattern). ## Proto + RPC: GetRegionalBrief proto/worldmonitor/intelligence/v1/get_regional_brief.proto - GetRegionalBriefRequest { region_id } - GetRegionalBriefResponse { brief: RegionalBrief } - RegionalBrief { region_id, generated_at, period_start, period_end, situation_recap, regime_trajectory, key_developments[], risk_outlook, provider, model } ## Server handler server/worldmonitor/intelligence/v1/get-regional-brief.ts Simple getCachedJson read + adaptBrief snake->camel adapter. Returns upstreamUnavailable: true on Redis failure so the gateway skips caching (matching the get-regime-history pattern from #2981). ## Premium gating + cache tier src/shared/premium-paths.ts + server/gateway.ts RPC_CACHE_TIER ## Tests — 27 new unit tests buildBriefPrompt (5): region/balance/transitions/narrative rendered, empty transitions handled, missing fields tolerated parseBriefJson (5): valid JSON, garbage, all-empty, cap at 5, prose extraction generateWeeklyBrief (6): success, global skip, LLM fail, garbage, exception, period_start/end delta emptyBrief (2): region_id + empty fields handler (4): key prefix, adapter export, upstreamUnavailable, registration security (2): premium path + cache tier proto (3): RPC declared, import wired, RegionalBrief fields ## Verification - npm run test:data: 4651/4651 pass - npm run typecheck + typecheck:api: clean - biome lint: clean * fix(intelligence): address 3 review findings on #2989 P2 #1 — no consumer surface for GetRegionalBrief Acknowledged. The consumer is the RegionalIntelligenceBoard panel, which will call GetRegionalBrief and render a weekly brief block. This wiring is Phase 3 PR3 (UI) scope — the RPC + Redis key are the delivery mechanism, not the end surface. No code change in this commit; the RPC is ready for the panel to consume. P2 #2 — readRecentTransitions collapses failure to [] readRecentTransitions returned [] on Redis/network failure, which is indistinguishable from a genuinely quiet week. The LLM then generates a brief claiming "no regime transitions" when in reality the upstream is down — fabricating false input. Fix: return null on failure. The seeder skips the region with a clear log message when transitions is null, so the brief is never written with unreliable input. Empty array [] now only means genuinely no transitions in the 7-day window. P2 #3 — parseBriefJson accepts briefs the seeder rejects parseBriefJson treated non-empty key_developments as valid even if situation_recap was empty. The seeder gate only writes when brief.situation_recap is truthy. That mismatch means the validator pass + provider-fallback logic could accept a response that the seeder then silently drops. Fix: require situation_recap in parseBriefJson for valid=true, matching the seeder gate. Now both checks agree on what constitutes a usable brief, and the provider-fallback chain correctly falls through when a provider returns a brief with developments but no recap. * fix(intelligence): TTL path-segment fix + seed-meta always-write (Greptile P1+P2 on #2989) P1 — TTL silently not applied (briefs never expire) Upstash REST ignores query-string SET options (?EX=N). The correct form is path-segment: /set/{key}/{value}/EX/{seconds}. Without this fix every brief persists indefinitely and Redis storage grows unboundedly across weekly runs. P2 — seed-meta not written when all regions skipped writeExtraKeyWithMeta was gated on generated > 0. If every region was skipped (no snapshot yet, or LLM failed), seed-meta was never written, making the seeder indistinguishable from "never ran" in health tooling. Now writes seed-meta whenever failed === 0, carrying regionsSkipped count. P2 #3 (validate gate) — already fixed in previous commit (parseBriefJson now requires situation_recap for valid=true). * fix(intelligence): register regional-briefs in health.js SEED_META + STANDALONE_KEYS (review P2 on #2989) * fix(intelligence): register regional-briefs in api/seed-health.js (review P2 on #2989) * fix(intelligence): raise brief TTL to 15 days to cover missed weekly cycle (review P2 on #2989) * fix(intelligence): distinguish missing-key from Redis-error + coverage-gated health (review P2s on #2989) P2 #1 — false upstreamUnavailable before first seed getCachedJson returns null for both "key missing" and "Redis failed", so the handler was advertising an outage for every region before the first weekly seed ran. Switched to getRawJson (throws on Redis errors) so null = genuinely missing key → clean empty 200, and thrown error = upstream failure → upstreamUnavailable: true for gateway no-store. P2 #2 — partial run hides coverage loss in health The seed-meta was written with generated count even if only 1 of 7 regions produced a brief. /api/health treats any positive recordCount as healthy, so broad regional failure was invisible to operators. Fix: recordCount is set to 0 when generated < ceil(expectedRegions/2). This makes /api/health report EMPTY_DATA for severely partial runs while still writing seed-meta (so the seeder is confirmed to have run). coverageOk flag in the summary payload lets operators drill into the exact coverage state. * fix(intelligence): tighten coverage gate to expectedRegions-1 (review P2 on #2989) |
||
|
Elie Habib
|
d3836ba49b |
feat(sentiment): add AAII investor sentiment survey (#2930)
* feat(sentiment): add AAII investor sentiment survey Weekly bull/bear/neutral sentiment from AAII (1987-present). Shows current reading, bull-bear spread, and 52-week historical chart. Seeder fetches from AAII CSV, stores last 52 weeks in Redis. * fix(aaii): wire panel loading + mark fallback data explicitly * fix(aaii): keep panel live across refreshes + surface in health monitoring - fetchData now falls back to /api/bootstrap?keys=aaiiSentiment on refresh (getHydratedData is one-shot and returns undefined after the first read, causing a permanent spinner on hourly refresh) - Shows an error state with auto-retry when both hydrated and bootstrap-fetch miss, matching the WsbTickerScannerPanel pattern - Registered aaiiSentiment in api/health.js BOOTSTRAP_KEYS and api/seed-health.js SEED_DOMAINS so rollout failures and fallback-only operation are observable in the monitoring dashboards * fix(sentiment): handle BIFF8 SST trailing bytes and use UTC for AAII Thursday calc Two P2 greptile fixes from PR #2930 review: 1. BIFF8 SST parser was reading the rich-text run count (cRun, flags & 0x08) and extended-string size (cbExtRst, flags & 0x04) to advance past those header fields, but never skipped the trailing bytes AFTER the char data: 4 * cRun formatting-run bytes and cbExtRst ext-rst bytes. If any string before the column header was rich-text formatted, every subsequent SST entry parsed from the wrong offset, silently breaking XLS extraction and falling back to HTML scraping. 2. parseHtmlSentiment() computed last-Thursday via today.getDay() + setDate(today.getDate() - daysToThursday), both local-TZ-dependent. On Railway (non-UTC TZ) the inferred Thursday could drift by a day, causing the HTML-derived row to mismatch the XLS historical rows. Switched to getUTCDay() + Date.UTC() for TZ-stable arithmetic. |
||
|
Elie Habib
|
2decda6508 |
feat(wsb): add Reddit/WSB ticker scanner seeder + bootstrap registration (#2916)
* feat(wsb): add Reddit/WSB ticker scanner seeder + bootstrap registration Seeder in ais-relay.cjs fetches from r/wallstreetbets, r/stocks, r/investing every 10min. Extracts ticker mentions, validates against known ticker set, aggregates by frequency and engagement, writes top 50 to intelligence:wsb-tickers:v1. 4-file bootstrap registration: cache-keys.ts, bootstrap.js, health.js with SEED_META maxStaleMin=30. * fix(wsb): remove duplicate CEO + fix avgUpvoteRatio divisor * fix(wsb): require ticker validation set + condition seed-meta on write + add seed-health 1. Skip seed when ticker validation set is empty (cold start/bootstrap miss) 2. Only write seed-meta after successful canonical write 3. Register in api/seed-health.js for dedicated monitoring * fix(wsb): case-insensitive $ticker matching + BRK.B dotted symbol support * fix(wsb): split $-prefixed vs bare ticker extraction + BRK.B→BRK-B normalization 1. $-prefixed tickers ($nvda, $BRK.B) skip whitelist validation (strong signal) — catches GME, AMC, PLTR etc. not in the narrow market watchlist 2. Bare uppercase tokens still validated against known set (high false-positive) 3. BRK.B normalized to BRK-B before validation (dot→dash) 4. Empty known set no longer skips seed — $-prefixed tickers still extracted * fix(wsb): skip bare-uppercase branch entirely when ticker set unavailable |
||
|
Elie Habib
|
1af73975b9 |
feat(energy): SPR policy classification layer (#2881)
* feat(energy): add SPR policy classification layer with 66-country registry Static JSON registry classifying strategic petroleum reserve regimes for 66 countries (all IEA members + major producers/consumers). Integrates into energy profile handler, shock model limitations, analyst context, spine seeder, and CDP UI. - scripts/data/spr-policies.json: 66-entry registry with regime, source, asOf - scripts/seed-spr-policies.mjs: seeder following chokepoint-baselines pattern - Proto fields 51-59 on GetCountryEnergyProfileResponse - Handler reads SPR registry from Redis, populates proto fields - Shock model adds fuel-mode-gated SPR limitations for non-IEA gov SPR - Analyst context refactored to accumulator pattern (IEA + SPR parts) - CDP UI: SPR badge for non-IEA government_spr, muted text for spare_capacity - Spine integration: SPR fields in shockInputs + hasSprPolicy coverage flag - Cache keys, health, bootstrap, seed-health registrations - Tests: registry shape, ISO2, regime enum, required entries, no estimatedFillPct * fix(energy): remove SPR from bootstrap (server-only); narrow SPR hasAny gate to renderable regimes * feat(energy): render "no known SPR" risk note for countries with regime=none * fix(energy): human-readable SPR regime labels; parallelize spine+registry reads in analyst |
||
|
Elie Habib
|
0a1b74a9b2 |
feat(resilience): add score confidence intervals via batch Monte Carlo (#2877)
* feat(resilience): add score confidence intervals via batch Monte Carlo
Weekly cron perturbs domain weights ±10% across 100 draws per country,
stores p05/p95 in Redis. Score handler reads intervals and includes
them in the API response as ScoreInterval { p05, p95 }.
Proto field 14 (score_interval) added to GetResilienceScoreResponse.
* chore: regenerate proto types and OpenAPI docs for ScoreInterval
* fix(resilience): add seed-meta + lock + fix interval cache + percentile formula
1. Write seed-meta:resilience:intervals for health monitoring
2. Add distributed lock to prevent concurrent cron overlap
3. Move scoreInterval read outside 6h score cache boundary
4. Fix percentile index from floor to ceil-1 (nearest-rank)
* fix(health): add resilience:intervals to health + seed-health registries
* fix(seed): skip seed-meta on no-op runs + register intervals in health check
|
||
|
Elie Habib
|
b8924eb90f |
feat(energy): Ember monthly electricity seed (V5-6a) (#2815)
* feat(energy): Ember monthly electricity seed — V5-6a New seed-ember-electricity.mjs writes energy:ember:v1:<ISO2> and energy:ember:v1:_all from Ember Climate's monthly generation CSV (CC BY 4.0). Daily cron at 08:00 UTC, TTL 72h (3x interval), >=60 country coverage guard. Registers in api/health.js, api/seed-health.js, cache-keys.ts, and ais-relay.cjs. Dockerfile.relay COPY added. 🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/code) + Compound Engineering v2.49.0 Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> * fix(energy): add _country-resolver.mjs to Dockerfile.relay; correct Ember intervalMin (V5-6a) Two bugs in the Ember seed PR: 1. Dockerfile.relay was missing COPY for _country-resolver.mjs, which seed-ember-electricity.mjs imports. Would have crashed with ERR_MODULE_NOT_FOUND on first run in production. 2. api/seed-health.js had intervalMin:720 (12h) for a daily (24h) cron. With stale threshold = intervalMin*2, this gave only 24h grace -- the seed would flap stale during the CSV download window. Corrected to intervalMin:1440 so stale threshold = 48h (2x interval). * fix(energy): wire energy:ember:v1:_all into bootstrap hydration (V5-6a) Greptile P1: api/bootstrap.js was missing the emberElectricity slow-key entry, violating the AGENTS.md requirement that new data sources be registered for bootstrap hydration. energy:ember:v1:_all is a ~60-country bulk map (monthly cadence) - added to SLOW_KEYS consistent with faoFoodPriceIndex and other monthly-release bulk keys. Also updates server/_shared/cache-keys.ts BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS to keep the bootstrap test coverage green (bootstrap test validates that SLOW_KEYS and BOOTSTRAP_TIERS are in sync). * fix(energy): 3 review fixes for Ember seed (V5-6a) 1. Ember URL: updated to correct current download URL (old path returned HTTP 404, seeder could never run). 2. Count-drop guard after failure: failure path now preserves the previous recordCount in seed-meta instead of writing 0, so the 75% drop guard stays active after a failed run. 3. api/seed-health.js: status:error now marks seed as stale/error immediately instead of only checking age; prevents /api/seed-health showing ok for 48h while the seeder is failing. * fix(energy): correct Ember CSV column names + fix skipped-path meta (V5-6a) 1. CSV schema: parser was using country_code/series/unit/value/date but the real Ember CSV headers are "ISO 3 code"/"Variable"/"Unit"/ "Value"/"Date". Added COLS constants and updated all row field accesses. The schema sentinel (hasFossil check) was always firing because r.series was always undefined, causing every seeder run to abort. Updated test fixtures to use real column names. 2. Skipped-path meta: lock.skipped branch now reads existing meta and preserves recordCount and status while refreshing fetchedAt. Previously writing recordCount:0 disabled the count-drop guard after any skipped run and made health endpoints see false-ok with zero count. * fix(energy): remove skipped-path meta write + revert premature bootstrap (V5-6a) 1. lock.skipped: removed seed-meta write from the skipped path. The running instance writes correct meta on completion; refreshing fetchedAt on skip masked relay/lock failures from health endpoints. 2. Bootstrap: removed emberElectricity from BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS — no consumer exists in src/ yet. Per energyv5.md, bootstrap registration is deferred to PR7 when consumers land. * fix(energy): split ember pipeline writes; fix health.js recordCount lookup - api/health.js: add recordCount fallback in both seed-meta count reads so the Ember domain shows correct record count instead of always 1 - scripts/seed-ember-electricity.mjs: split single pipeline into Phase A (per-country + _all data) and Phase B (seed-meta only after Phase A succeeds) to prevent preservePreviousSnapshot reading a partial _all key * fix(energy): split ember pipeline writes; align SEED_ERROR in health.js; add tests * fix(energy): atomic rollback on partial pipeline failure; seedError priority in health cascade * fix(energy): DEL obsolete per-country keys on publish, rollback, and restore * fix(energy): MULTI/EXEC atomic pipeline; null recordCount on read-miss; dataWritten guard --------- Co-authored-by: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com> |
||
|
Elie Habib
|
d96259048d |
feat(energy): canonical energy spine — V5-1 (#2798)
* feat(energy): canonical energy spine seeder + handler read-through (V5-1) - Add scripts/seed-energy-spine.mjs: daily seeder that assembles one energy:spine:v1:<ISO2> key per country from 6 domain keys (OWID mix, JODI oil, JODI gas, IEA stocks, ENTSO-E electricity, GIE gas storage) - TTL 172800s (48h), count-drop guard at 80%, schema sentinel for OWID mix, lock pattern + Redis pipeline batch write mirroring seed-owid-energy-mix.mjs - Update get-country-energy-profile.ts to read from spine first; fall back to existing 6-key Promise.allSettled join on spine miss - Update chat-analyst-context.ts buildProductSupply/buildGasFlows/buildOilStocksCover to prefer spine key; fall through to direct domain key reads on miss - Update get-country-intel-brief.ts to read energy mix from spine.mix + sources.mixYear before falling back to energy:mix:v1: direct key - Add ENERGY_SPINE_KEY_PREFIX and ENERGY_SPINE_COUNTRIES_KEY to cache-keys.ts - Add energySpineCountries to api/health.js STANDALONE_KEYS and SEED_META - Add Railway cron comment (0 6 * * *) in ais-relay.cjs - Add tests/energy-spine-seed.test.mjs: 26 tests covering spine build logic, IEA anomaly guard, JODI oil fallback, schema sentinel, count-drop math * fix(energy): add cache-keys module replacement in redis-caching test The new ENERGY_SPINE_KEY_PREFIX import in get-country-intel-brief.ts was not patched in the importPatchedTsModule call used by redis-caching tests. Add cache-keys to the replacement map to resolve the module. * fix(energy): add missing fields to spine entry and read-through - buildOilFields: add product importsKbd (gasoline/diesel/jet/lpg) and belowObligation - buildMixFields: add windShare, solarShare, hydroShare - buildGasStorageFields: new helper storing fillPct, fillPctChange1d, trend - buildSpineEntry: add gasStorage section using new helper - EnergySpine interface: extend oil/mix/gasStorage to match seeder output - buildResponseFromSpine: read all new fields instead of hard-coding 0/false * fix(energy): exclude electricity/gas-storage from spine; add seed-health entry Spine-first path was returning stale gas-storage and electricity data for up to 8h after seeding (spine runs 06:00 UTC, gas storage updates 10:30 UTC, electricity updates 14:00 UTC). Fix: handler now reads gas-storage and electricity directly in parallel with the spine read (3-key allSettled). Fallback path drops from 6 to 4 keys since gas-storage and electricity are already fetched in the hot path. Also registers energy:spine in api/seed-health.js (daily cron, maxStaleMin inferred as 2×intervalMin = 2880 min). Seeder (seed-energy-spine.mjs) and its tests updated to reflect the narrowed spine schema — electricity and gasStorage fields removed from buildSpineEntry. * fix(energy): address Greptile P2 review findings - chat-analyst-context: return undefined when gas imports are 0 rather than falling back to totalDemandTj with an "imports" label — avoids mislabeling domestic gas demand as imports for net exporters (RU, QA, etc.) - seed-energy-spine: add status:'ok' to success-path seed-meta write so all seed-meta records have a consistent status field regardless of path |
||
|
Elie Habib
|
47af642d24 |
feat(energy): live chokepoint flow calibration from PortWatch DWT — V5-2 (#2797)
* feat(energy): chokepoint flow calibration seeder — V5-2 (Phase 4 PR A)
- Add CHOKEPOINT_FLOWS_KEY to server/_shared/cache-keys.ts
- Add energy:chokepoint-flows:v1 to health.js monitoring (maxStaleMin: 720)
- Add 6h chokepoint flow seed loop to ais-relay.cjs (seed-chokepoint-flows.mjs)
- Fix seeder to use degraded mode instead of throwing when PortWatch absent
- Add degraded-mode and ID-mapping tests to chokepoint-flows-seed.test.mjs
* fix(energy): restore throw for PortWatch absent + register seed-health
- seed-chokepoint-flows: revert degraded-path from warn+return{} back to
throw; PortWatch absent is an upstream-not-ready error, not a data-quality
issue — must throw so startChokepointFlowsSeedLoop schedules 20-min retry
- api/seed-health.js: add energy:chokepoint-flows to SEED_DOMAINS so
/api/seed-health surfaces missing/stale signal (intervalMin: 360 = 6h cron)
- tests: update degraded-mode assertions to match restored throw behavior
|
||
|
Elie Habib
|
aa794e1369 |
feat(portwatch): seed per-country port activity (Endpoints 3+4) (#2786)
* feat(portwatch): seed per-country port activity (Endpoints 3+4) * fix(portwatch): register portwatch-ports seed-meta in api/seed-health.js * fix(portwatch): correct Endpoint 3 field names and move Redis writes out of fetchAll() * fix(portwatch): add Endpoint 4 pagination loop and fix anomalySignal divisor symmetry * fix(portwatch): stable pagination order + add portwatchPortActivity to PENDING_CONSUMERS * fix(portwatch): degradation guard + hoist prevCountryKeys for correct catch-block TTL extension |
||
|
Elie Habib
|
cf27ffbfde |
feat(portwatch): seed chokepoints reference (Endpoint 2) (#2785)
* feat(portwatch): seed chokepoints reference data from Endpoint 2 * test(bootstrap): exempt portwatchChokepointsRef from consumer check (UI consumer in future PR) * fix(portwatch): register chokepoints-ref seed-meta in api/seed-health.js * fix(portwatch): add returnGeometry=false to reduce ArcGIS response size * fix(portwatch): raise validateFn threshold to 27 (guards against partial ArcGIS responses) * fix(portwatch): validateFn requires exactly 28 chokepoints (reject partial ArcGIS responses) |
||
|
Elie Habib
|
5d549a9b2c |
fix(seed-health): add FAO food price and product catalog to SEED_DOMAINS (#2707)
Both are in health.js SEED_META but missing from seed-health.js, so the seed-health dashboard doesn't track them. |
||
|
Elie Habib
|
e6d6e41ab0 | fix(seed-health): add missing OWID energy mix to SEED_DOMAINS (#2706) | ||
|
Elie Habib
|
f210c5511a |
feat(regulatory): add tier classification and Redis publish (#2691)
* feat(regulatory): add tier classification and Redis publish Builds on the fetch/parse layer from #2564. Adds keyword-based tier classification (high/medium/low/unknown) and publishes to Redis via runSeed with 6h TTL. - HIGH: enforcement, fraud, penalty, injunction, etc. - MEDIUM: rulemaking, guidance, investigation, etc. - LOW: routine notices matching title patterns - Register REGULATORY_ACTIONS_KEY in cache-keys.ts Closes #2493 Co-authored-by: Lucas Passos <lspassos1@users.noreply.github.com> * fix(regulatory): reject empty payloads, add health monitoring - validateFn now requires actions.length > 0 to prevent overwriting a healthy snapshot with an empty one on parser regression - Register regulatory:actions:v1 in STANDALONE_KEYS (api/health.js) - Add seed-meta:regulatory:actions to SEED_META (maxStaleMin: 360, 3x the 2h cron interval) - Add seed-health.js monitoring (intervalMin: 120) --------- Co-authored-by: Lucas Passos <lspassos1@users.noreply.github.com> |
||
|
Fayez Bast
|
8609ad1384 |
feat: " climate disasters alerts seeders " (#2550)
* Revert "Revert "feat(climate): add climate disasters seed + ListClimateDisast…"
This reverts commit
|
||
|
Lucas Passos
|
f36e337692 |
feat(resilience): add static country seeder (#2658)
* feat(resilience): add static country seeder Root cause: the resilience work needed a canonical per-country snapshot with health visibility and failure-safe Redis behavior, but the repo had no annual seed for multi-source country attributes. Changes: - add scripts/seed-resilience-static.mjs with per-country keys, manifest/meta writes, partial dataset failure handling, and prior-snapshot preservation on total failure - register the manifest/meta in api/health.js and api/seed-health.js without expanding bootstrap scope - extend scripts/railway-set-watch-paths.mjs with a dedicated seed-resilience-static service config and cron support - add focused tests for parser/shape contracts and Railway config wiring Validation: - node --test tests/resilience-static-seed.test.mjs tests/railway-set-watch-paths.test.mjs tests/bootstrap.test.mjs tests/edge-functions.test.mjs - npm run typecheck:api (fails on upstream baseline: missing vitest in server/__tests__/entitlement-check.test.ts) - smoke checks for fetchWhoDataset/fetchEnergyDependencyDataset/fetchRsfDataset against live sources * refactor(resilience): extract country resolver, wire real data sources - Extract country resolver (COUNTRY_ALIAS_MAP, normalizeCountryToken, isIso2, isIso3, createCountryResolvers, resolveIso2) into reusable scripts/_country-resolver.mjs for sharing with scoring layer - Replace env-gated GPI/FSIN/AQUASTAT stubs with real endpoints: - GPI: Vision of Humanity CSV (dynamic year URL with fallback) - FSIN: HDX IPC wide-format CSV (stable download URL) - AQUASTAT: FAO BigQuery API CSV (water stress + dependency + per capita) - Remove dead code: fetchBinary, parseTabularPayload, pickField, fetchOptionalTabularRows (no longer needed with known CSV formats) - Harden RSF parser: reject if < 100 countries (was === 0) 993 → 829 lines in seed script + 113 lines in shared resolver * fix(resilience): add _country-resolver to watch paths, catch Eurostat parse errors - Add scripts/_country-resolver.mjs to Railway watch patterns so resolver changes trigger a redeploy - Wrap parseEurostatEnergyDataset in try-catch so a malformed 200 response falls through to World Bank fallback instead of aborting * fix(resilience): cap pagination loops, check pipeline results - World Bank: cap at 100 pages to prevent runaway from malformed totalPages response - WHO GHO: cap at 50 pages and throw if pagination link persists (prevents infinite loop from cyclic nextLink) - publishSuccess: inspect per-command pipeline results and throw on partial failures to prevent status:ok with missing country keys (which would lock out same-year retries via shouldSkipSeedYear) --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
02f55dc584 |
feat(climate): add ocean ice indicators seed and RPC (#2652)
* feat(climate): add ocean ice indicators seed and RPC * fix(review): restore MCP maxStaleMin, widen health threshold, harden sea level parser, type globe.gl shim - Restore get_climate_data _maxStaleMin to 2880 (was accidentally lowered to 1440) - Bump oceanIce SEED_META maxStaleMin from 1440 to 2880 (2× daily interval, tolerates one missed run) - Add fallback regex patterns for NASA sea level overlay HTML parsing - Replace globe.gl GlobeInstance `any` with typed interface (index sig stays `any` for Three.js compat) * fix(review): merge prior cache on partial failures, fix fallback regex, omit trend without baseline - P1: fetchOceanIceData() now reads prior cache and merges last-known-good indicators when any upstream source fails, preventing partial overwrites from erasing previously healthy data - P1: sea level fallback regex now requires "current" context to avoid matching the historical 1993 baseline rate instead of the current rate - P2: classifyArcticTrend() returns null (omitted from payload) when no climatology baseline exists, instead of misleadingly labeling as "average" - Added tests for all three fixes * fix(review): merge prior cache by source field group, not whole object Prior-cache merge was too coarse: Object.assign(payload, priorCache) reintroduced stale arctic_extent_anomaly_mkm2 and arctic_trend from prior cache when sea-ice succeeded but intentionally omitted those fields (no climatology baseline), and an unrelated source like OHC or sea level failed in the same run. Fix: define per-source field groups (seaIce, seaLevel, ohc, sst). Only fall back to prior cache fields for groups whose source failed entirely. When a source succeeds, only its returned fields appear in the payload, even if it omits fields it previously provided. Added test covering the exact combined case: sea-ice climatology unavailable + unrelated source failure + prior-cache merge enabled. --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
9d94ad36aa |
feat(climate+health):add shared air quality seed and mirrored health (#2634)
* feat(climate+health):add shared air quality seed and mirrored health/climate RPCs * feat(climate+health):add shared air quality seed and mirrored health/climate RPCs * fix(air-quality): address review findings — TTL, seed-health, FAST_KEYS, shared meta - Raise CACHE_TTL from 3600 to 10800 (3× the 1h cron cadence; gold standard) - Add health:air-quality to api/seed-health.js SEED_DOMAINS so monitoring dashboard tracks freshness - Remove climateAirQuality and healthAirQuality from FAST_KEYS (large station payloads; load in slow batch) - Point climateAirQuality SEED_META to same meta key as healthAirQuality (same seeder run, one source of truth) * fix(bootstrap): move air quality keys to SLOW tier — large station payloads avoid critical-path batch * fix(air-quality): fix malformed OpenAQ URL and remove from bootstrap until panel exists - Drop deprecated first URL attempt (parameters=pm25, order_by=lastUpdated, sort=desc); use correct v3 params (parameters_id=2, sort_order=desc) directly — eliminates guaranteed 4xx retry cycle per page on 20-page crawl - Remove climateAirQuality and healthAirQuality from BOOTSTRAP_CACHE_KEYS, SLOW_KEYS, and BOOTSTRAP_TIERS — no panel consumes these yet; adding thousands of station records to every startup bootstrap is pure payload bloat - Remove normalizeAirQualityPayload helpers from bootstrap.js (no longer called) - Update service wrappers to fetch via RPC directly; re-add bootstrap hydration when a panel actually needs it * fix(air-quality): raise lock TTL to 3600s to cover 20-page crawl worst case 2 OpenAQ calls × 20 pages × (30s timeout × 3 attempts) = 3600s max runtime. Previous 600s TTL allowed concurrent cron runs on any degraded upstream. --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Elie Habib
|
ca76c13bb0 |
fix(climate): correct health interval values for anomalies and co2 seeders (#2613)
* fix(climate): correct health interval values for anomalies and co2 seeders climate:anomalies intervalMin was 60 but Railway cron runs every 2h (120). climate:co2-monitoring intervalMin was 2160 (36h) but daily cron = 1440min. climateAnomalies maxStaleMin was 120 (1x interval, no tolerance) — bumped to 240 to match the 2x gold standard and tolerate one missed cron run. * fix(climate): clarify co2-monitoring intervalMin vs maxStaleMin divergence intervalMin (1440) is the actual daily cron cadence; health.js maxStaleMin (4320 = 72h) is the alarm threshold set at 3x for tolerance. These serve different purposes and are intentionally different values. |
||
|
Fayez Bast
|
b2bae30bd8 |
Add climate news seed and ListClimateNews RPC (#2532)
* Add climate news seed and ListClimateNews RPC * Wire climate news into bootstrap and fix generated climate stubs * fix(climate): align seed health interval and parse Atom entries per feed * fix(climate-news): TTL 90min, retry timer on failure, named cache key constant - CACHE_TTL: 1800 to 5400 (90min = 3x 30-min relay interval, gold standard) - ais-relay: add 20-min retry timer on subprocess failure; clear on success - cache-keys.ts: export CLIMATE_NEWS_KEY named constant - list-climate-news.ts: import CLIMATE_NEWS_KEY instead of hard-coding string --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> |
||
|
Fayez Bast
|
bb4f8dcb12 |
feat(climate): add WMO normals seeding and CO2 monitoring (#2531)
* feat(climate): add WMO normals seeding and CO2 monitoring * fix(climate): skip missing normals per-zone and align anomaly tooltip copy * fix(climate): remove normals from bootstrap and harden health/cache key wiring * feat(climate): version anomaly cache to v2, harden seed freshness, and align CO2/normal baselines |
||
|
Elie Habib
|
ae4010a795 |
Revert "feat(climate): add climate disasters seed + ListClimateDisasters RPC with snake_case Redis payload (#2535)" (#2544)
This reverts commit
|
||
|
Fayez Bast
|
e2dea9440d | feat(climate): add climate disasters seed + ListClimateDisasters RPC with snake_case Redis payload (#2535) | ||
|
Elie Habib
|
ee8ca345cb |
refactor: consolidate Upstash helpers and extract DeckGL color config (#2465)
* refactor: consolidate Upstash helpers and extract DeckGL color config Part 1 — api/_upstash-json.js: add getRedisCredentials, redisPipeline, setCachedData exports. Migrate _oauth-token.js, reverse-geocode.js, health.js, bootstrap.js, seed-health.js, and cache-purge.js off their inline credential/pipeline boilerplate. Part 2 — DeckGLMap.ts: extract getBaseColor, mineralColor, windColor, TC_WIND_COLORS, CII_LEVEL_COLORS into src/config/ files so panels and tests can reuse them without importing DeckGLMap. Surfaced by reviewing nichm/worldmonitor-private fork. * fix(mcp): restore throw-on-Redis-error in fetchOAuthToken; fix health error message _oauth-token.js: readJsonFromUpstash returns null for HTTP errors, but mcp.ts:702 relies on a throw to return 503 (retryable) vs null→401 (re-authenticate). Restore the explicit fetch that throws on !resp.ok, using getRedisCredentials() for credential extraction. health.js: the single null guard produced "Redis not configured" for both missing creds and HTTP failures. Split into two checks so the 503 body correctly distinguishes env config problems from service outages. * fix(upstash): remove dead try/catch in reverse-geocode; atomic SET EX in setCachedData |
||
|
Elie Habib
|
2e16159bb6 |
feat(economic): WoW price tracking + weekly cadence for BigMac & Grocery panels (#1974)
* feat(economic): add WoW tracking and fix plumbing for bigmac/grocery-basket panels Phase 1 — Fix Plumbing: - Adjust CACHE_TTL to 10 days (864000s) for bigmac and grocery-basket seeds - Align health.js SEED_META maxStaleMin to 10080 (7 days) for both - Add grocery-basket and bigmac to seed-health.js SEED_DOMAINS with intervalMin: 5040 - Refactor publish.ts writeSnapshot to accept advanceSeedMeta param; only advance seed-meta when fresh data exists (overallFreshnessMin < 120) - Add manual-fallback-only comment to seed-consumer-prices.mjs Phase 2 — Week-over-Week Tracking: - Add wow_pct field to BigMacCountryPrice and CountryBasket proto messages - Add wow_avg_pct, wow_available, prev_fetched_at to both response protos - Regenerate client/server TypeScript from updated protos - Add readCurrentSnapshot() helper + WoW computation to seed-bigmac.mjs and seed-grocery-basket.mjs; write :prev key via extraKeys - Update BigMacPanel.ts to show per-country WoW column and global avg summary - Update GroceryBasketPanel.ts to show WoW badge on total row and basket avg summary - Add .bm-wow-up, .bm-wow-down, .bm-wow-summary, .gb-wow CSS classes - Fix server handlers to include new WoW fields in fallback responses * fix(economic): guard :prev extraKey against null on first seed run; eliminate double freshness query in publish.ts * refactor(economic): address code review findings from PR #1974 - Extract readSeedSnapshot() into _seed-utils.mjs (DRY: was duplicated verbatim in seed-bigmac and seed-grocery-basket) - Add FRESH_DATA_THRESHOLD_MIN constant in publish.ts (replace magic 120) - Fix seed-consumer-prices.mjs contradictory JSDoc (remove stale "Deployed as: Railway cron service" line that contradicted manual-only warning) - Add i18n keys panels.bigmacWow / panels.bigmacCountry to en.json - Replace hardcoded "WoW" / "Country" with t() calls in BigMacPanel - Replace IIFE-in-ternary pattern with plain if blocks in BigMacPanel and GroceryBasketPanel (P2/P3 from code review) * fix(publish): gate advanceSeedMeta on any-retailer freshness, not average overallFreshnessMin is the arithmetic mean across all retailers, so with 1 fresh + 2 stale retailers the average can exceed 120 min and suppress seed-meta advancement even while fresh data is being published. Use retailers.some(r => r.freshnessMin < 120) to correctly implement "at least one retailer scraped within the last 2 hours." |
||
|
Elie Habib
|
b52916b7e3 |
fix(health): adjust gdeltIntel maxStaleMin for 6h cron; warn on expired-key EXPIRE no-op (#1853)
* fix(health): adjust gdeltIntel maxStaleMin for 6h cron; fix silent EXPIRE no-op on expired keys - gdeltIntel maxStaleMin: 150 → 420 (6h cron + 1h grace). The 150 threshold was calibrated for the old 2h cron — with 6h intervals it fires STALE throughout most of each cycle, masking the signal entirely. - _seed-utils extendExistingTtl: EXPIRE returns 0 (no-op) on expired/missing keys, but the log always said "Extended TTL on N key(s)" regardless. Added per-result checking: keys that returned 0 now emit a WARNING so the death-spiral condition (validate fails + key expired + EXPIRE is silently a no-op) is visible in logs rather than silently passing as if TTL was extended. * fix(seed-health): align gdelt-intel intervalMin to 210 (420min maxStaleMin / 2) Codex flagged mismatch: health.js allows 420min before flagging gdelt-intel stale, but seed-health.js still used intervalMin: 150 (flags after 300min). Ops tooling monitoring seed-health would generate spurious alerts for most of each 6h cron cycle. Align to 210min per the maxStaleMin/2 convention. |
||
|
Elie Habib
|
11c444fcc9 |
fix(gdelt): reduce topics 6→4 to cut 429 rate-limit pressure (#1817)
* fix(gdelt): reduce topics 6→4 to cut 429 rate-limit pressure Drops sanctions and intelligence topics (covered by other data sources). Keeps military, cyber, nuclear, maritime as core high-signal topics. Happy-path runtime drops from ~2.5min to ~1.5min. Worst-case retry storm is now 3 gaps instead of 5, significantly reducing total backoff duration per run on the 2h cron cycle. Lowers validation threshold from ≥3 to ≥2 of 4 topics. * fix(gdelt): reduce cron to 4h and extend TTL to 6h Data from GDELT's 24h window doesn't turn over fast enough to justify 2h polling. Switching to 4h halves Railway runs (12/day → 6/day) and doubles the cooldown between IP hits, reducing 429 pressure. TTL bumped 4h→6h so cached data outlives the 4h cron gap. health.js maxStaleMin 200→300 (5h, warning window before 6h expiry). seed-health.js intervalMin 100→150 (150×2=300 = maxStaleMin). Railway cron schedule needs updating to: 0 */4 * * * * fix(gdelt): sync UI topic list with seed and require all 4 topics for write P1: Remove sanctions and intelligence from INTEL_TOPICS in src/services/gdelt-intel.ts so the panel no longer renders tabs that can never be hydrated from the 4-topic Redis payload. Tabs now match the seed: military, cyber, nuclear, maritime. P2: Raise validation threshold back to >=4 (all topics required). With >=2 a partial run would overwrite the complete snapshot with incomplete data, making missing tabs show blank panels until the next full run. Requiring all 4 means a partial run extends the existing TTL instead of replacing good data with bad. |
||
|
Elie Habib
|
12042641a7 |
fix(health): close GDELT warning window and sync seed-health intervalMin (#1815)
P1: Lower gdeltIntel maxStaleMin 240→200 so there is a 40-minute STALE warning window before the 4h Redis key expires and health jumps straight to EMPTY/critical. P2: Align seed-health.js intervalMin with relaxed health.js thresholds (intervalMin × 2 = maxStaleMin): - research:tech-events 210→240 (480/2) - intelligence:gdelt-intel 60→100 (200/2) - intelligence:advisories 45→60 (120/2) |
||
|
Elie Habib
|
cf52aee45b |
fix(seed-health): add radiation + sanctions to cron monitoring (#1772)
* fix(seed-health): add radiation and sanctions to cron monitoring Both seeds are running on Railway but were missing from seed-health.js, so /api/seed-health couldn't report their freshness. - radiation:observations: intervalMin=60 (cron runs every 15min) - sanctions:pressure: intervalMin=360 (cron runs every 6h) * fix(seed-health): align radiation intervalMin with 15min cron cadence intervalMin was 60 but cron runs every 15min. seed-health would only flag stale after 120min (2x intervalMin), inconsistent with health.js maxStaleMin: 30. Now intervalMin=15 so both endpoints agree. |
||
|
Elie Habib
|
3702463321 |
Add thermal escalation seeded service (#1747)
* feat(thermal): add thermal escalation seeded service Cherry-picked from codex/thermal-escalation-phase1 and retargeted to main. Includes thermal escalation seed script, RPC handler, proto definitions, bootstrap/health/seed-health wiring, gateway cache tier, client service, and tests. * fix(thermal): wire data-loader, fix typing, recalculate summary Wire fetchThermalEscalations into data-loader.ts with panel forwarding, freshness tracking, and variant gating. Fix seed-health intervalMin from 90 to 180 to match 3h TTL. Replace 8 as-any casts with typed interface. Recalculate summary counts after maxItems slice. * fix(thermal): enforce maxItems on hydrated data + fix bootstrap keys Codex P2: hydration branch now slices clusters to maxItems before mapping, matching the RPC fallback behavior. Also add thermalEscalation to bootstrap.js BOOTSTRAP_CACHE_KEYS and SLOW_KEYS (was lost during conflict resolution). * fix(thermal): recalculate summary on sliced hydrated clusters When maxItems truncates the cluster array from bootstrap hydration, the summary was still using the original full-set counts. Now recalculates clusterCount, elevatedCount, spikeCount, etc. on the sliced array, matching the handler's behavior. |
||
|
Elie Habib
|
bdd8743a26 |
refactor: dedupe edge api json response assembly (#1702)
* refactor: dedupe edge api json response assembly * refactor: expand jsonResponse helper to all edge functions Roll out jsonResponse() from _json-response.js to 16 files (14 handlers + 2 shared helpers), eliminating 55 instances of the new Response(JSON.stringify(...)) boilerplate. Only exception: health.js uses JSON.stringify(body, null, indent) for pretty-print mode, which is incompatible with the helper signature. Replaced local jsonResponse/json() definitions in contact.js, register-interest.js, and cache-purge.js with the shared import. |
||
|
Elie Habib
|
a4914607bb | fix(forecast): bundle military surge inputs (#1706) | ||
|
Elie Habib
|
4c11b46be3 |
feat(trade): add US Treasury customs revenue to Trade Policy panel (#1663)
* feat(trade): add US Treasury customs revenue to Trade Policy panel US customs duties revenue spiked 4-5x under Trump tariffs (from $7B/month to $27-31B/month) but the WTO tariff data only goes to 2024. Adds Treasury MTS data showing monthly customs revenue. - Add GetCustomsRevenue RPC (proto, handler, cache tier) - Add Treasury fetch to seed-supply-chain-trade.mjs (free API, no key) - Add Revenue tab to TradePolicyPanel with FYTD YoY comparison - Fix WTO gate: per-tab gating so Revenue works without WTO key - Wire bootstrap hydration, health, seed-health tracking * test(trade): add customs revenue feature tests 22 structural tests covering: - Handler: raw key mode, empty-cache behavior, correct Redis key - Seed: Treasury API URL, classification filter, timeout, row validation, amount conversion, sort order, seed-meta naming - Panel: WTO gate fix (per-tab not panel-wide), revenue tab defaults when WTO key missing, dynamic FYTD comparison - Client: no WTO feature gate, bootstrap hydration, type exports * fix(trade): align FYTD comparison by fiscal month count Prior FY comparison was filtering by calendar month, which compared 5 months of FY2026 (Oct-Feb) against only 2 months of FY2025 (Jan-Feb), inflating the YoY percentage. Now takes the first N months of the prior FY matching the current FY month count. * fix(trade): register treasury_revenue DataSourceId and localize revenue tab - Add treasury_revenue to DataSourceId union type so freshness tracking actually works (was silently ignored) - Register in data-freshness.ts source config + gap messages - Add i18n keys: revenue tab label, empty state, unavailable banner - Update infoTooltip to include Revenue tab description * fix(trade): complete revenue tab localization Use t() for all remaining hardcoded strings: footer source labels, FYTD summary headline, prior-year comparison, and table column headers. Wire the fytdLabel/vsPriorFy keys that were added but not used. * fix(test): update revenue source assertion for localized string |
||
|
Elie Habib
|
f336418c17 |
feat(advisories): gold standard migration for security advisories (#1637)
* feat(advisories): gold standard migration for security advisories Move security advisories from client-side RSS fetching (24 feeds per page load) to Railway cron seed with Redis-read-only Vercel handler. - Add seed script fetching via relay RSS proxy with domain allowlist - Add ListSecurityAdvisories proto, handler, and RPC cache tier - Add bootstrap hydration key for instant page load - Rewrite client service: bootstrap -> RPC fallback, no browser RSS - Wire health.js, seed-health.js, and dataSize tracking * fix(advisories): empty RPC returns ok:true, use full country map P1 fixes from Codex review: - Return ok:true for empty-but-successful RPC responses so the panel clears to empty instead of stuck loading on cold environments - Replace 50-entry hardcoded country map with 251-entry shared config generated from the project GeoJSON + aliases, matching coverage of the old client-side nameToCountryCode matcher * fix(advisories): add Cote d'Ivoire and other missing country aliases Adds 14 missing aliases including "cote d ivoire" (US State Dept title format), common article-prefixed names (the Bahamas, the Gambia), and alternative official names (Czechia, Eswatini, Cabo Verde, Timor-Leste). * fix(proto): inject @ts-nocheck via Makefile generate target buf generate does not emit @ts-nocheck, but tsc strict mode rejects the generated code. Adding a post-generation sed step in the Makefile ensures both CI proto-freshness (make generate + diff) and CI typecheck (tsc --noEmit) pass consistently. |
||
|
Elie Habib
|
db6a4a2763 |
feat(correlation): server-side correlation engine seed + bootstrap hydration (#1571)
* feat(correlation): server-side correlation engine seed + bootstrap hydration Move correlation card computation from client-side (per-browser, 10-30s delay) to server-side (Railway cron, instant via bootstrap). Seed script reads 8 Redis keys, runs 4 adapter signal collectors (military, escalation, economic, disaster), clusters/scores/generates cards, writes to Redis with 10min TTL. - New: scripts/seed-correlation.mjs (pure JS port of correlation engine) - bootstrap.js: add correlationCards to FAST_KEYS tier - health.js + seed-health.js: register for monitoring (maxStaleMin: 15) - CorrelationPanel: consume bootstrap on construction, show "Analyzing..." only after live engine has run (not for bootstrap-only cards) - _seed-utils.mjs: support opts.recordCount override (function or number) * fix(correlation): stale timestamp fallback + coordinate-based country resolution P1: news stories lacked per-story pubDate, causing Date.now() fallback on every seed run. Now _clustering.mjs propagates pubDate through to enrichedStories, and seed-correlation reads s.pubDate then generatedAt. P2: normalizeToCode dropped signals with unparseable country names. Added centroid-based coordinate fallback (haversine nearest-match within 800km) matching the live engine's getCountryAtCoordinates behavior. * fix(correlation): add 11 missing country centroids to coordinate fallback CI, CR, CV, CY, GA, IS, LA, SZ, TL, TT, XK were in the normalization maps but missing from COUNTRY_CENTROIDS, causing coordinate-only signals in those countries to be misclassified or dropped during bootstrap. * fix(correlation): align protest/outage field names with actual Redis schema Codex review P1 findings: seed-correlation read wrong field names from Redis data. Protests (unrest:events:v1): p.time -> p.occurredAt, p.lat/lon -> p.location.latitude/longitude, severity enum SEVERITY_LEVEL_* mapping. Outages (infra:outages:v1): o.pubDate -> o.detectedAt, o.lat/lon -> o.location.latitude/longitude, severity enum OUTAGE_SEVERITY_* mapping. Both escalation and disaster adapters updated. Old field names kept as fallbacks for data shape compatibility. |
||
|
Elie Habib
|
e0bf4f9bd2 |
feat: seed GDELT intelligence topics to Redis (#1556)
* feat: seed GDELT intelligence topics to Redis with bootstrap hydration Add standalone seed script that pre-populates all 6 Live Intelligence topics (military, cyber, nuclear, sanctions, intelligence, maritime) from the GDELT Doc API into Redis. Frontend consumes bootstrap data lazily via the service layer, falling back to RPC if unavailable. - scripts/seed-gdelt-intel.mjs: new seed script with per-topic 429 retry - api/bootstrap.js: register gdeltIntel in FAST_KEYS - api/health.js: register in BOOTSTRAP_KEYS + SEED_META + dataSize - api/seed-health.js: register in SEED_DOMAINS - scripts/_seed-utils.mjs: add topics to recordCount detection - src/services/gdelt-intel.ts: lazy bootstrap consumption in service layer * fix(seed): align staleness thresholds and strengthen GDELT validation - seed-health intervalMin 30→60 so staleness (120min) matches health.js maxStaleMin - validate requires ≥3/6 topics populated (not just military) - recordCount sums articles across topics instead of reporting topic count |
||
|
Elie Habib
|
41380b8e23 |
fix(health): close monitoring gaps in health and seed-health endpoints (#1531)
Add missing seed-meta write for intlDelays in ais-relay, add untracked SEED_META entries (intlDelays, faaDelays, theaterPosture) to health.js, add 6 missing domains to seed-health.js, and return 503 when degraded. |
||
|
Elie Habib
|
601a1028a4 |
fix(health): fix riskScores seeding gap and seed-meta key mismatch (#1366)
* fix(health): fix riskScores seeding gap and seed-meta key mismatch - Switch RPC handler to cachedFetchJsonWithMeta so stale key is refreshed on every successful response (cache hit or miss), not just cache misses - Fix seed-meta key mismatch: health.js and seed-health.js now check seed-meta:risk:scores:sebuf (matching what cachedFetchJson writes) - Add warm-ping loop in relay (8min interval) to keep RPC cache fresh - Remove dead startCiiSeedLoop and 345 lines of unused CII seed code * fix(scoring): await stale key write to prevent edge runtime drop Edge/serverless runtimes may terminate the isolate before a fire-and-forget Redis write completes. Await the setCachedJson call so the stale key TTL is guaranteed to be extended. |
||
|
Elie Habib
|
4721e3504a |
fix(health): separate status severity, expand seed-health, harden Redis errors (#1358)
* fix(health): separate status severity, expand seed-health domains, harden Redis errors - Distinguish WARNING (warns only, 200) from DEGRADED (few crits, 503) - Exclude OK_CASCADE entries from compact mode problems list - Add missing dataSize properties (sectors, statuses, scores) - Remove redisKey from public /api/health responses (info disclosure) - Expand seed-health.js from 12 to 40 domains aligned with health.js SEED_META - Return 503 for stale seeds, 200 for missing (cold start) in seed-health - Throw on Redis config/HTTP errors instead of masking as empty results * fix(seed-health): align severity with health.js, remove RPC-only domains - Revert severity order: missing = degraded/503, stale = warning/200 (matches health.js where empty/missing is higher severity than stale) - Remove RPC-populated domains (BIS, minerals, giving, worldbank, macro-signals) whose seed-meta is only written on-demand by cachedFetchJson, not by scheduled seeders * fix(seed-health): keep 200 for all states to avoid breaking migration validator scripts/validate-seed-migration.mjs (L203) treats non-200 as hard failure and skips body parsing. Returning 503 for degraded would break that flow. Keep 200 and let callers interpret the overall field instead. |