mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
ce797da3a42e4653bbbf551dbc24e10fae731ff7
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
d46c170012 |
feat(brief): hosted magazine edge route + latest-brief preview RPC (Phase 2) (#3153)
* feat(brief): hosted magazine edge route + latest-brief preview RPC (Phase 2)
Phase 2 of the WorldMonitor Brief plan (docs/plans/2026-04-17-003).
Adds the read path that every downstream delivery channel binds to.
Phase 1 shipped the renderer + envelope contract; Phase 3 (composer)
will write brief:{userId}:{issueDate} to Redis. Until then the route
returns a neutral "expired" page for every request — intentional, safe
to deploy ahead of the composer.
Files:
- server/_shared/brief-url.ts
HMAC-SHA256 sign + verify helpers using Web Crypto (edge-compatible,
no node:crypto). Rotation supported via optional prevSecret in
verifyBriefToken. Constant-time comparison. Strict userId and
YYYY-MM-DD shape validation before any crypto operation. Typed
BriefUrlError with code enum for caller branches.
- api/brief/[userId]/[issueDate].ts (Vercel edge)
Auth model: the HMAC-signed ?t= token IS the credential. No Clerk
session required — URLs are delivered over already-authenticated
channels (push, email, dashboard panel). Flow: verify token →
Redis GET brief:{userId}:{issueDate} → renderBriefMagazine → HTML.
403 on bad token, 404 on Redis miss, 503 on missing secret. Never
echoes the userId in error pages.
- api/latest-brief.ts (Clerk JWT + PRO gated, Vercel edge)
Returns { issueDate, dateLong, greeting, threadCount, magazineUrl }
or { status: 'composing', issueDate } when Redis is cold. Mirrors
the auth + entitlement pattern in api/notify.ts. magazineUrl is
freshly signed per request against the current BRIEF_URL_SIGNING_
SECRET so rotation takes effect immediately.
- tests/brief-url.test.mjs
20 tests: round-trip, tampering, wrong user/date/secret, malformed
token/inputs, rotation (accept prev secret + reject after removal),
URL composition (trailing-slash trim, path encoding).
Quality gates: typecheck + typecheck:api clean, biome lint clean,
render tests still 30/30, new HMAC tests 20/20.
PRE-MERGE: BRIEF_URL_SIGNING_SECRET must be set in Vercel before this
deploys. The route returns 503 (not 500) with a server-side log when
the secret is missing, so a mis-configured deploy is safe to roll.
* fix(brief): address Phase 2 review — broken import + HEAD body + host reflection
Addresses todos 212–217 from the /ce-review on PR #3153.
P0 / P1 blockers:
- todo 212 (P0): renderer import path was broken — pointed at
shared/render-brief-magazine.js which no longer exists (Phase 1
review moved it to server/_shared/brief-render.js). Route would
have 500'd on every successful token verification; @ts-expect-error
silenced the error at compile time. Fixed the path and removed the
now-unnecessary @ts-expect-error since brief-render.d.ts exists.
New tests/brief-edge-route-smoke.test.mjs imports the handler so a
future broken path cannot pass CI.
- todo 213 (P1): HEAD returned the full HTML body (RFC 7231
violation). htmlResponse() now takes the request and emits null
body when method is HEAD.
- todo 214 (P1): publicBaseUrl reflected x-forwarded-host, allowing a
signed magazineUrl to point at a non-canonical origin (preview
deploy, forwarded-host spoof). Now pinned to
WORLDMONITOR_PUBLIC_BASE_URL env var, falling back to the request
origin only in dev. This env var joins BRIEF_URL_SIGNING_SECRET on
the pre-merge Vercel checklist.
P2 cleanups:
- todo 215: dropped dead 'invalid_token_shape' enum member from
BriefUrlError — never thrown (verifyBriefToken returns false on
shape failure by design).
- todo 216: consolidated EXPIRED_PAGE + FORBIDDEN_PAGE (and added
UNAVAILABLE_PAGE) behind a single renderErrorPage(heading, body)
helper with shared styles.
- todo 217: extracted readRawJsonFromUpstash() into api/_upstash-json.js
so api/brief and api/latest-brief share one 3-second-timeout raw-GET
implementation. Dodges the readJsonFromUpstash unwrap that strips
seed envelopes (our brief envelope is flat, not seed-framed).
Also:
- brief-url.ts header now documents the rotation runbook + the
emergency kill-switch path (rotate SECRET without setting PREV).
- distinguishable log tag for malformed-envelope vs Redis-miss
(composer-bug grep vs expired-key grep).
Deferred (todo 218, P2): token-in-URL log leak is a design-level
issue that wants a cookie-based first-visit exchange or iat/exp in
the signed payload. Out of scope for this fix commit.
Tests: 55/55 (20 HMAC + 30 render + 5 edge-route smoke). Typecheck
clean, biome lint clean.
* fix(brief): distinguish infra failure from miss + walk tz boundary
Addresses two additional P1 review findings on PR #3153.
1. readRawJsonFromUpstash collapsed every failure mode (missing creds,
HTTP non-2xx, timeout, JSON parse failure, genuine miss) into a
single null return. Both edge routes then treated null as empty
state: api/brief → 404 "expired", api/latest-brief → 200
"composing". During any Upstash outage a user with a valid brief
would see misleading empty-state UX.
Helper now throws on every infrastructure/parse failure and
returns null ONLY on a genuine miss (Upstash replied 200 with
result: null). Callers:
- api/brief catches → 503 UNAVAILABLE_PAGE
- api/latest-brief catches → 503 { error: service_unavailable }
2. api/latest-brief probed today UTC only. For users ahead of or
behind UTC, a ready brief could be invisible for 12-24h around
the date boundary.
Route now accepts an optional ?date=YYYY-MM-DD query param so the
dashboard panel (or any client) can send its local date directly.
When ?date= is absent, we walk today UTC → yesterday UTC and
return the most recent hit; composing is only reported when both
slots miss. Malformed ?date= returns 400.
Tests added:
- helper-level: missing creds → throws, HTTP error → throws,
genuine miss → null
- route-level: api/brief returns 503 (not 404) on Upstash HTTP error
Final suite: 60/60 (20 HMAC + 30 render + 10 smoke). Typecheck +
lint clean.
* fix(brief): tomorrow-UTC probe + unified envelope validation
Addresses two third-round review findings on PR #3153.
1. Walk-back missed the tomorrow-UTC slot. Users at positive tz
offsets (UTC+1..UTC+14) store today's brief under tomorrow UTC;
the previous [today, yesterday] probe never saw them. Fixed to
walk [tomorrow, today, yesterday] UTC — covers the full tz range
and naturally prefers the most recently composed slot. Also
correctly echoes the caller's ?date= in composing responses
instead of always echoing today UTC.
2. readBriefPreview validated only dateLong + digest.greeting +
stories (array). The renderer's assertBriefEnvelope is much
stricter (closed-key contract, version guard, cross-field
surfaced === stories.length, per-story 7 required fields, etc.).
A partial envelope could be reported "ready" by the preview and
404-expired on click. Exported assertBriefEnvelope from the
renderer module, called it in the preview reader. An envelope
that would render successfully is exactly an envelope that the
preview reports as ready; any divergence is logged as a composer
bug and surfaced as "composing".
Tests: 62/62 (20 HMAC + 30 render + 12 smoke). New cases cover the
shared-validator export and catch the "partial envelope slips past
weak preview" regression.
Typecheck + lint clean.
|
||
|
Elie Habib
|
044598346e |
feat(seed-contract): PR 2a — runSeed envelope dual-write + 91 seeders migrated (#3097)
* feat(seed-contract): PR 2a — runSeed envelope dual-write + 91 seeders migrated
Opt-in contract path in runSeed: when opts.declareRecords is provided, write
{_seed, data} envelope to the canonical key alongside legacy seed-meta:*
(dual-write). State machine: OK / OK_ZERO / RETRY with zeroIsValid opt.
declareRecords throws or returns non-integer → hard fail (contract violation).
extraKeys[*] support per-key declareRecords; each extra key writes its own
envelope. Legacy seeders (no declareRecords) entirely unchanged.
Migrated all 91 scripts/seed-*.mjs to contract mode. Each exports
declareRecords returning the canonical record count, and passes
schemaVersion: 1 + maxStaleMin (matched to api/health.js SEED_META, or 2.5x
interval where no registry entry exists). Contract conformance reports 84/86
seeders with full descriptor (2 pre-existing warnings).
Legacy seed-meta keys still written so unmigrated readers keep working;
follow-up slices flip health.js + readers to envelope-first.
Tests: 61/61 PR 1 tests still pass.
Next slices for PR 2:
- api/health.js registry collapse + 15 seed-bundle-*.mjs canonicalKey wiring
- reader migration (mcp, resilience, aviation, displacement, regional-snapshot)
- direct writers — ais-relay.cjs, consumer-prices-core publish.ts
- public-boundary stripSeedEnvelope + test migration
Plan: docs/plans/2026-04-14-002-fix-runseed-zero-record-lockout-plan.md
* fix(seed-contract): unwrap envelopes in internal cross-seed readers
After PR 2a enveloped 91 canonical keys as {_seed, data}, every script-side
reader that returned the raw parsed JSON started silently handing callers the
envelope instead of the bare payload. WoW baselines (bigmac, grocery-basket,
fear-greed) saw undefined .countries / .composite; seed-climate-anomalies saw
undefined .normals from climate:zone-normals:v1; seed-thermal-escalation saw
undefined .fireDetections from wildfire:fires:v1; seed-forecasts' ~40-key
pipeline batch returned envelopes for every input.
Fix: route every script-side reader through unwrapEnvelope(...).data. Legacy
bare-shape values pass through unchanged (unwrapEnvelope returns
{_seed: null, data: raw} for any non-envelope shape).
Changed:
- scripts/_seed-utils.mjs: import unwrapEnvelope; redisGet, readSeedSnapshot,
verifySeedKey all unwrap. Exported new readCanonicalValue() helper for
cross-seed consumers.
- 18 seed-*.mjs scripts with local redisGet-style helpers or inline fetch
patched to unwrap via the envelope source module (subagent sweep).
- scripts/seed-forecasts.mjs pipeline batch: parse() unwraps each result.
- scripts/seed-energy-spine.mjs redisMget: unwraps each result.
Tests:
- tests/seed-utils-envelope-reads.test.mjs: 7 new cases covering envelope
+ legacy + null paths for readSeedSnapshot and verifySeedKey.
- Full seed suite: 67/67 pass (was 61, +6 new).
Addresses both of user's P1 findings on PR #3097.
* feat(seed-contract): envelope-aware reads in server + api helpers
Every RPC and public-boundary reader now automatically strips _seed from
contract-mode canonical keys. Legacy bare-shape values pass through unchanged
(unwrapEnvelope no-ops on non-envelope shapes).
Changed helpers (one-place fix — unblocks ~60 call sites):
- server/_shared/redis.ts: getRawJson, getCachedJson, getCachedJsonBatch
unwrap by default. cachedFetchJson inherits via getCachedJson.
- api/_upstash-json.js: readJsonFromUpstash unwraps (covers api/mcp.ts
tool responses + all its canonical-key reads).
- api/bootstrap.js: getCachedJsonBatch unwraps (public-boundary —
clients never see envelope metadata).
Left intentionally unchanged:
- api/health.js / api/seed-health.js: read only seed-meta:* keys which
remain bare-shape during dual-write. unwrapEnvelope already imported at
the meta-read boundary (PR 1) as a defensive no-op.
Tests: 67/67 seed tests pass. typecheck + typecheck:api clean.
This is the blast-radius fix the PR #3097 review called out — external
readers that would otherwise see {_seed, data} after the writer side
migrated.
* fix(test): strip export keyword in vm.runInContext'd seed source
cross-source-signals-regulatory.test.mjs loads scripts/seed-cross-source-signals.mjs
via vm.runInContext, which cannot parse ESM `export` syntax. PR 2a added
`export function declareRecords` to every seeder, which broke this test's
static-analysis approach.
Fix: strip the `export` keyword from the declareRecords line in the
preprocessed source string so the function body still evaluates as a plain
declaration.
Full test:data suite: 5307/5307 pass. typecheck + typecheck:api clean.
* feat(seed-contract): consumer-prices publish.ts writes envelopes
Wrap the 5 canonical keys written by consumer-prices-core/src/jobs/publish.ts
(overview, movers:7d/30d, freshness, categories:7d/30d/90d, retailer-spread,
basket-series) in {_seed, data} envelopes. Legacy seed-meta:<key> writes
preserved for dual-write.
Inlined a buildEnvelope helper (10 lines) rather than taking a cross-package
dependency — consumer-prices-core is a standalone npm package. Documented the
four-file parity contract (mjs source, ts mirror, js edge mirror, this copy).
Contract fields: sourceVersion='consumer-prices-core-publish-v1', schemaVersion=1,
state='OK' (recordCount>0) or 'OK_ZERO' (legitimate zero).
Typecheck: no new errors in publish.ts.
* fix(seed-contract): 3 more server-side readers unwrap envelopes
Found during final audit:
- server/worldmonitor/resilience/v1/_shared.ts: resilience score reader
parsed cached GetResilienceScoreResponse raw. Contract-mode seed-resilience-scores
now envelopes those keys.
- server/worldmonitor/resilience/v1/get-resilience-ranking.ts: p05/p95
interval lookup parsed raw from seed-resilience-scores' extra-key path.
- server/worldmonitor/infrastructure/v1/_shared.ts: mgetJson() used for
count-source keys (wildfire:fires:v1, news:insights:v1) which are both
contract-mode now.
All three now unwrap via server/_shared/seed-envelope. Legacy shapes pass
through unchanged.
Typecheck clean.
* feat(seed-contract): ais-relay.cjs direct writes produce envelopes
32 canonical-key write sites in scripts/ais-relay.cjs now produce {_seed, data}
envelopes. Inlined buildEnvelope() (CJS module can't require ESM source) +
envelopeWrite(key, data, ttlSeconds, meta) wrapper. Enveloped keys span market
bootstrap, aviation, cyber-threats, theater-posture, weather-alerts, economic
spending/fred/worldbank, tech-events, corridor-risk, usni-fleet, shipping-stress,
social:reddit, wsb-tickers, pizzint, product-catalog, chokepoint transits,
ucdp-events, satellites, oref.
Left bare (not seeded data keys): seed-meta:* (dual-write legacy),
classifyCacheKey LLM cache, notam:prev-closed-state internal state,
wm:notif:scan-dedup flags.
Updated tests/ucdp-seed-resilience.test.mjs regex to accept both upstashSet
(pre-contract) and envelopeWrite (post-contract) call patterns.
* feat(seed-contract): 15 bundle files add canonicalKey for envelope gate
54 bundle sections across 12 files now declare canonicalKey alongside the
existing seedMetaKey. _bundle-runner.mjs (from PR 1) prefers canonicalKey
when both are present — gates section runs on envelope._seed.fetchedAt
read directly from the data key, eliminating the meta-outlives-data class
of bugs.
Files touched:
- climate (5), derived-signals (2), ecb-eu (3), energy-sources (6),
health (2), imf-extended (4), macro (10), market-backup (9),
portwatch (4), relay-backup (2), resilience-recovery (5), static-ref (2)
Skipped (14 sections, 3 whole bundles): multi-key writers, dynamic
templated keys (displacement year-scoped), or non-runSeed orchestrators
(regional brief cron, resilience-scores' 222-country publish, validation/
benchmark scripts). These continue to use seedMetaKey or their own gate.
seedMetaKey preserved everywhere — dual-write. _bundle-runner.mjs falls
back to legacy when canonicalKey is absent.
All 15 bundles pass node --check. test:data: 5307/5307. typecheck:all: clean.
* fix(seed-contract): 4 PR #3097 review P1s — transform/declareRecords mismatches + envelope leaks
Addresses both P1 findings and the extra-key seed-meta leak surfaced in review:
1. runSeed helper-level invariant: seed-meta:* keys NEVER envelope.
scripts/_seed-utils.mjs exports shouldEnvelopeKey(key) — returns false for
any key starting with 'seed-meta:'. Both atomicPublish (canonical) and
writeExtraKey (extras) gate the envelope wrap through this helper. Fixes
seed-iea-oil-stocks' ANALYSIS_META_EXTRA_KEY silently getting enveloped,
which broke health.js parsing the value as bare {fetchedAt, recordCount}.
Also defends against any future manual writeExtraKey(..., envelopeMeta)
call that happens to target a seed-meta:* key.
2. seed-token-panels canonical + extras fixed.
publishTransform returns data.defi (the defi panel itself, shape {tokens}).
Old declareRecords counted data.defi.tokens + data.ai.tokens + data.other.tokens
on the transformed payload → 0 → RETRY path → canonical market:defi-tokens:v1
never wrote, and because runSeed returned before the extraKeys loop,
market:ai-tokens:v1 + market:other-tokens:v1 stayed stale too.
New: declareRecords counts data.tokens on the transformed shape. AI_KEY +
OTHER_KEY extras reuse the same function (transforms return structurally
identical panels). Added isMain guard so test imports don't fire runSeed.
3. api/product-catalog.js cached reader unwraps envelope.
ais-relay.cjs now envelopes product-catalog:v2 via envelopeWrite(). The
edge reader did raw JSON.parse(result) and returned {_seed, data} to
clients, breaking the cached path. Fix: import unwrapEnvelope from
./_seed-envelope.js, apply after JSON.parse. One site — :238-241 is
downstream of getFromCache(), so the single reader fix covers both.
4. Regression lock tests/seed-contract-transform-regressions.test.mjs (11 cases):
- shouldEnvelopeKey invariant: seed-meta:* false, canonical true
- Token-panels declareRecords works on transformed shape (canonical + both extras)
- Explicit repro of pre-fix buggy signature returning 0 — guards against revert
- resolveRecordCount accepts 0, rejects non-integer
- Product-catalog envelope unwrap returns bare shape; legacy passes through
Verification:
- npm run test:data → 5318/5318 pass (was 5307 — 11 new regressions)
- npm run typecheck:all → clean
- node --check on every modified script
iea-oil-stocks canonical declareRecords was NOT broken (user confirmed during
review — buildIndex preserves .members); only its ANALYSIS_META_EXTRA_KEY
was affected, now covered generically by commit 1's helper invariant.
* fix(seed-contract): seed-token-panels validateFn also runs on post-transform shape
Review finding: fixing declareRecords wasn't sufficient — atomicPublish() runs
validateFn(publishData) on the transformed payload too. seed-token-panels'
validate() checked data.defi/.ai/.other on the transformed {tokens} shape,
returned false, and runSeed took the early skipped-write branch (before even
reaching the declareRecords RETRY logic). Net effect: same as before the
declareRecords fix — canonical + both extras stayed stale.
Fix: validate() now checks the canonical defi panel directly (Array.isArray
(data?.tokens) && has at least one t.price > 0). AI/OTHER panels are validated
implicitly by their own extraKey declareRecords on write.
Audited the other 9 seeders with publishTransform (bls-series, bis-extended,
bis-data, gdelt-intel, trade-flows, iea-oil-stocks, jodi-gas, sanctions-pressure,
forecasts): all validateFn's correctly target the post-transform shape. Only
token-panels regressed.
Added 4 regression tests (tests/seed-contract-transform-regressions.test.mjs):
- validate accepts transformed panel with priced tokens
- validate rejects all-zero-price tokens
- validate rejects empty/missing tokens
- Explicit pre-fix repro (buggy old signature fails on transformed shape)
Verification:
- npm run test:data → 5322/5322 pass (was 5318; +4 new)
- npm run typecheck:all → clean
- node --check clean
* feat(seed-contract): add /api/seed-contract-probe validation endpoint
Single machine-readable gate for 'is PR #3097 working in production'.
Replaces the curl/jq ritual with one authenticated edge call that returns
HTTP 200 ok:true or 503 + failing check list.
What it validates:
- 8 canonical keys have {_seed, data} envelopes with required data fields
and minRecords floors (fsi-eu, zone-normals, 3 token panels + minRecords
guard against token-panels RETRY regression, product-catalog, wildfire,
earthquakes).
- 2 seed-meta:* keys remain BARE (shouldEnvelopeKey invariant; guards
against iea-oil-stocks ANALYSIS_META_EXTRA_KEY-class regressions).
- /api/product-catalog + /api/bootstrap responses contain no '_seed' leak.
Auth: x-probe-secret header must match RELAY_SHARED_SECRET (reuses existing
Vercel↔Railway internal trust boundary).
Probe logic is exported (checkProbe, checkPublicBoundary, DEFAULT_PROBES) for
hermetic testing. tests/seed-contract-probe.test.mjs covers every branch:
envelope pass/fail on field/records/shape, bare pass/fail on shape/field,
missing/malformed JSON, Redis non-2xx, boundary seed-leak detection,
DEFAULT_PROBES sanity (seed-meta invariant present, token-panels minRecords
guard present).
Usage:
curl -H "x-probe-secret: $RELAY_SHARED_SECRET" \
https://api.worldmonitor.app/api/seed-contract-probe
PR 3 will extend the probe with a stricter mode that asserts seed-meta:*
keys are GONE (not just bare) once legacy dual-write is removed.
Verification:
- tests/seed-contract-probe.test.mjs → 15/15 pass
- npm run test:data → 5338/5338 (was 5322; +16 new incl. conformance)
- npm run typecheck:all → clean
* fix(seed-contract): tighten probe — minRecords on AI/OTHER + cache-path source header
Review P2 findings: the probe's stated guards were weaker than advertised.
1. market:ai-tokens:v1 + market:other-tokens:v1 probes claimed to guard the
token-panels extra-key RETRY regression but only checked shape='envelope'
+ dataHas:['tokens']. If an extra-key declareRecords regressed to 0, both
probes would still pass because checkProbe() only inspects _seed.recordCount
when minRecords is set. Now both enforce minRecords: 1.
2. /api/product-catalog boundary check only asserted no '_seed' leak — which
is also true for the static fallback path. A broken cached reader
(getFromCache returning null or throwing) could serve fallback silently
and still pass this probe. Now:
- api/product-catalog.js emits X-Product-Catalog-Source: cache|dodo|fallback
on the response (the json() helper gained an optional source param wired
to each of the three branches).
- checkPublicBoundary declaratively requires that header's value match
'cache' for /api/product-catalog, so a fallback-serve fails the probe
with reason 'source:fallback!=cache' or 'source:missing!=cache'.
Test updates (tests/seed-contract-probe.test.mjs):
- Boundary check reworked to use a BOUNDARY_CHECKS config with optional
requireSourceHeader per endpoint.
- New cases: served-from-cache passes, served-from-fallback fails with source
mismatch, missing header fails, seed-leak still takes precedence, bad
status fails.
- Token-panels sanity test now asserts minRecords≥1 on all 3 panels.
Verification:
- tests/seed-contract-probe.test.mjs → 17/17 pass (was 15, +2 net)
- npm run test:data → 5340/5340
- npm run typecheck:all → clean
|
||
|
Elie Habib
|
ee8ca345cb |
refactor: consolidate Upstash helpers and extract DeckGL color config (#2465)
* refactor: consolidate Upstash helpers and extract DeckGL color config Part 1 — api/_upstash-json.js: add getRedisCredentials, redisPipeline, setCachedData exports. Migrate _oauth-token.js, reverse-geocode.js, health.js, bootstrap.js, seed-health.js, and cache-purge.js off their inline credential/pipeline boilerplate. Part 2 — DeckGLMap.ts: extract getBaseColor, mineralColor, windColor, TC_WIND_COLORS, CII_LEVEL_COLORS into src/config/ files so panels and tests can reuse them without importing DeckGLMap. Surfaced by reviewing nichm/worldmonitor-private fork. * fix(mcp): restore throw-on-Redis-error in fetchOAuthToken; fix health error message _oauth-token.js: readJsonFromUpstash returns null for HTTP errors, but mcp.ts:702 relies on a throw to return 503 (retryable) vs null→401 (re-authenticate). Restore the explicit fetch that throws on !resp.ok, using getRedisCredentials() for credential extraction. health.js: the single null guard produced "Redis not configured" for both missing creds and HTTP failures. Split into two checks so the 503 body correctly distinguishes env config problems from service outages. * fix(upstash): remove dead try/catch in reverse-geocode; atomic SET EX in setCachedData |
||
|
Elie Habib
|
8466779c9a | refactor: dedupe upstash json reads across edge endpoints (#1708) |