mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
2f19d96357
* feat(brief): route whyMatters through internal analyst-context endpoint
The brief's "why this is important" callout currently calls Gemini on
only {headline, source, threatLevel, category, country} with no live
state. The LLM can't know whether a ceasefire is on day 2 or day 50,
that IMF flagged >90% gas dependency in UAE/Qatar/Bahrain, or what
today's forecasts look like. Output is generic prose instead of the
situational analysis WMAnalyst produces when given live context.
This PR adds an internal Vercel edge endpoint that reuses a trimmed
variant of the analyst context (country-brief, risk scores, top-3
forecasts, macro signals, market data — no GDELT, no digest-search)
and ships it through a one-sentence LLM call with the existing
WHY_MATTERS_SYSTEM prompt. The endpoint owns its own Upstash cache
(v3 prefix, 6h TTL), supports a shadow mode that runs both paths in
parallel for offline diffing, and is auth'd via RELAY_SHARED_SECRET.
Three-layer graceful degradation (endpoint → legacy Gemini-direct →
stub) keeps the brief shipping on any failure.
Env knobs:
- BRIEF_WHY_MATTERS_PRIMARY=analyst|gemini (default: analyst; typo → gemini)
- BRIEF_WHY_MATTERS_SHADOW=0|1 (default: 1; only '0' disables)
- BRIEF_WHY_MATTERS_SHADOW_SAMPLE_PCT=0..100 (default: 100)
- BRIEF_WHY_MATTERS_ENDPOINT_URL (Railway, optional override)
Cache keys:
- brief:llm:whymatters:v3:{hash16} — envelope {whyMatters, producedBy,
at}, 6h TTL. Endpoint-owned.
- brief:llm:whymatters:shadow:v1:{hash16} — {analyst, gemini, chosen,
at}, 7d TTL. Fire-and-forget.
- brief:llm:whymatters:v2:{hash16} — legacy. Cron's fallback path
still reads/writes during the rollout window; expires in ≤24h.
Tests: 6022 pass (existing 5915 + 12 core + 36 endpoint + misc).
typecheck + typecheck:api + biome on changed files clean.
Plan (Codex-approved after 4 rounds):
docs/plans/2026-04-21-001-feat-brief-why-matters-analyst-endpoint-plan.md
* fix(brief): address /ce:review round 1 findings on PR #3248
Fixes 5 findings from multi-agent review, 2 of them P1:
- #241 P1: `.gitignore !api/internal/**` was too broad — it re-included
`.env`, `.env.local`, and any future secret file dropped into that
directory. Narrowed to explicit source extensions (`*.ts`, `*.js`,
`*.mjs`) so parent `.env` / secrets rules stay in effect inside
api/internal/.
- #242 P1: `Dockerfile.digest-notifications` did not COPY
`shared/brief-llm-core.js` + `.d.ts`. Cron would have crashed at
container start with ERR_MODULE_NOT_FOUND. Added alongside
brief-envelope + brief-filter COPY lines.
- #243 P2: Cron dropped the endpoint's source/producedBy ground-truth
signal, violating PR #3247's own round-3 memory
(feedback_gate_on_ground_truth_not_configured_state.md). Added
structured log at the call site: `[brief-llm] whyMatters source=<src>
producedBy=<pb> hash=<h>`. Endpoint response now includes `hash` so
log + shadow-record pairs can be cross-referenced.
- #244 P2: Defense-in-depth prompt-injection hardening. Story fields
flowed verbatim into both LLM prompts, bypassing the repo's
sanitizeForPrompt convention. Added sanitizeStoryFields helper and
applied in both analyst and gemini paths.
- #245 P2: Removed redundant `validate` option from callLlmReasoning.
With only openrouter configured in prod, a parse-reject walked the
provider chain, then fell through to the other path (same provider),
then the cron's own fallback (same model) — 3x billing on one reject.
Post-call parseWhyMatters check already handles rejection cleanly.
Deferred to P3 follow-ups (todos 246-248): singleflight, v2 sunset,
misc polish (country-normalize LOC, JSDoc pruning, shadow waitUntil,
auto-sync mirror, context-assembly caching).
Tests: 6022 pass. typecheck + typecheck:api clean.
* fix(brief-why-matters): ctx.waitUntil for shadow write + sanitize legacy fallback
Two P2 findings on PR #3248:
1. Shadow record was fire-and-forget without ctx.waitUntil on an Edge
function. Vercel can terminate the isolate after response return,
so the background redisPipeline write completes unreliably — i.e.
the rollout-validation signal the shadow keys were supposed to
provide was flaky in production.
Fix: accept an optional EdgeContext 2nd arg. Build the shadow
promise up front (so it starts executing immediately) then register
it with ctx.waitUntil when present. Falls back to plain unawaited
execution when ctx is absent (local harness / tests).
2. scripts/lib/brief-llm.mjs legacy fallback path called
buildWhyMattersPrompt(story) on raw fields with no sanitization.
The analyst endpoint sanitizes before its own prompt build, but
the fallback is exactly what runs when the endpoint misses /
errors — so hostile headlines / sources reached the LLM verbatim
on that path.
Fix: local sanitizeStoryForPrompt wrapper imports sanitizeForPrompt
from server/_shared/llm-sanitize.js (existing pattern — see
scripts/seed-digest-notifications.mjs:41). Wraps story fields
before buildWhyMattersPrompt. Cache key unchanged (hash is over raw
story), so cache parity with the analyst endpoint's v3 entries is
preserved.
Regression guard: new test asserts the fallback prompt strips
"ignore previous instructions", "### Assistant:" line prefixes, and
`<|im_start|>` tokens when injection-crafted fields arrive.
Typecheck + typecheck:api clean. 6023 / 6023 data tests pass.
* fix(digest-cron): COPY server/_shared/llm-sanitize into digest-notifications image
Reviewer P1 on PR #3248: my previous commit (4eee22083) added
`import sanitizeForPrompt from server/_shared/llm-sanitize.js` to
scripts/lib/brief-llm.mjs, but Dockerfile.digest-notifications cherry-
picks server/_shared/* files and doesn't copy llm-sanitize. Import is
top-level/static — the container would crash at module load with
ERR_MODULE_NOT_FOUND the moment seed-digest-notifications.mjs pulls in
scripts/lib/brief-llm.mjs. Not just on fallback — every startup.
Fix: add `COPY server/_shared/llm-sanitize.js server/_shared/llm-sanitize.d.ts`
next to the existing brief-render COPY line. Module is pure string
manipulation with zero transitive imports — nothing else needs to land.
Cites feedback_validation_docker_ship_full_scripts_dir.md in the comment
next to the COPY; the cherry-pick convention keeps biting when new
cross-dir imports land in scripts/lib/ or scripts/shared/.
Can't regression-test at build time from this branch without a
docker-build CI job, but the symptom is deterministic — local runs
remain green (they resolve against the live filesystem); only the
container crashes. Post-merge, Railway redeploy of seed-digest-
notifications should show a clean `Starting Container` log line
instead of the MODULE_NOT_FOUND crash my prior commit would have caused.
75 lines
3.7 KiB
Docker
75 lines
3.7 KiB
Docker
# =============================================================================
|
|
# Digest notifications cron (consolidated: digest + brief compose + channel send)
|
|
# =============================================================================
|
|
# Runs scripts/seed-digest-notifications.mjs as a Railway cron (every 30 min).
|
|
# The script now also owns the brief envelope write path — per-user
|
|
# brief:{userId}:{issueDate} keys are produced here, and every channel
|
|
# dispatch (email/telegram/slack/discord) gets a signed magazine URL CTA.
|
|
#
|
|
# Historical context: before 2026-04-18 this service built from the
|
|
# scripts/ root with plain `npm ci`. The consolidation PR introduced
|
|
# cross-directory imports (shared/*, server/_shared/*, api/*) so the
|
|
# service now needs repo-root as build context with the specific
|
|
# modules COPY'd in. The retired seed-brief-composer Dockerfile had
|
|
# the same pattern.
|
|
#
|
|
# Required env (Railway service vars):
|
|
# UPSTASH_REDIS_REST_URL
|
|
# UPSTASH_REDIS_REST_TOKEN
|
|
# CONVEX_URL (or CONVEX_SITE_URL)
|
|
# RELAY_SHARED_SECRET
|
|
# RESEND_API_KEY
|
|
# TELEGRAM_BOT_TOKEN
|
|
# BRIEF_URL_SIGNING_SECRET (brief compose disabled without this)
|
|
# WORLDMONITOR_PUBLIC_BASE_URL (defaults to https://worldmonitor.app)
|
|
# Optional:
|
|
# DIGEST_CRON_ENABLED=0 (kill switch for the whole cron)
|
|
# BRIEF_COMPOSE_ENABLED=0 (kill switch for just brief compose)
|
|
# AI_DIGEST_ENABLED=0 (kill switch for AI summary LLM call)
|
|
# =============================================================================
|
|
|
|
FROM node:22-alpine
|
|
|
|
WORKDIR /app
|
|
|
|
# The repo-root package.json has "type":"module" which tells Node to
|
|
# parse .js files under shared/, server/_shared/, api/ as ESM. Inside
|
|
# this image we don't ship the full root package.json (it would pull
|
|
# in dev-deps metadata we don't need), but the .js files we DO ship
|
|
# still need the nearest-pjson walk to resolve to an ESM declaration.
|
|
# A minimal /app/package.json avoids "SyntaxError: Unexpected token
|
|
# 'export'" at container startup.
|
|
RUN printf '{"type":"module","private":true}\n' > /app/package.json
|
|
|
|
# Install scripts/ runtime dependencies (resend, convex, etc.).
|
|
COPY scripts/package.json scripts/package-lock.json ./scripts/
|
|
RUN npm ci --prefix scripts --omit=dev
|
|
|
|
# Digest cron + shared script helpers it imports via createRequire.
|
|
COPY scripts/seed-digest-notifications.mjs ./scripts/
|
|
COPY scripts/_digest-markdown.mjs ./scripts/
|
|
COPY scripts/lib/ ./scripts/lib/
|
|
|
|
# Brief envelope contract + filter + renderer assertion. These live
|
|
# under shared/ and server/_shared/ in the repo and are imported from
|
|
# scripts/lib/brief-compose.mjs. Keep this COPY list tight — adding
|
|
# unrelated shared/* files expands the rebuild watch surface.
|
|
COPY shared/brief-envelope.js shared/brief-envelope.d.ts ./shared/
|
|
COPY shared/brief-filter.js shared/brief-filter.d.ts ./shared/
|
|
COPY shared/brief-llm-core.js shared/brief-llm-core.d.ts ./shared/
|
|
COPY server/_shared/brief-render.js server/_shared/brief-render.d.ts ./server/_shared/
|
|
# llm-sanitize is imported by scripts/lib/brief-llm.mjs on the fallback
|
|
# path (legacy whyMatters generator) to strip prompt-injection patterns
|
|
# from story fields before they reach the LLM. Without this COPY, the
|
|
# digest cron crashes at import with ERR_MODULE_NOT_FOUND once the cron
|
|
# hits any story whose analyst endpoint call falls through to the
|
|
# fallback. See feedback_validation_docker_ship_full_scripts_dir.md —
|
|
# the cherry-pick pattern keeps biting when new cross-dir imports land.
|
|
COPY server/_shared/llm-sanitize.js server/_shared/llm-sanitize.d.ts ./server/_shared/
|
|
|
|
# Upstash REST helper (brief compose uses redisPipeline + readRawJson).
|
|
COPY api/_upstash-json.js ./api/
|
|
COPY api/_seed-envelope.js ./api/
|
|
|
|
CMD ["node", "scripts/seed-digest-notifications.mjs"]
|