feat(brief): make MAX_STORIES_PER_USER env-tunable (default 12, evidence kept it at 12) (#3389)

* fix(brief): bump MAX_STORIES_PER_USER 12 → 16

Production telemetry from PR #3387 surfaced cap-truncation as the
dominant filter loss: 73% of `sensitivity=all` users had `dropped_cap=18`
per tick (30 qualified stories truncated to 12). Multi-member topics
straddling the position-12 boundary lost members.

Bumping the cap to 16 lets larger leading topics fit fully without
affecting `sensitivity=critical` users (their pools cap at 7-10 stories
— well below either threshold). Reduces dropped_cap from ~18 to ~14
per tick.

Validation signal: watch the `[digest] brief filter drops` log line on
Railway after deploy — `dropped_cap=` should drop by ~4 per tick.

Side effect: this addresses the dominant production signal that
Solution 3 (post-filter regroup, originally planned in
docs/plans/2026-04-24-004-fix-brief-topic-adjacency-defects-plan.md)
was supposed to handle. Production evidence killed Sol-3's premise
(0 non-cap drops in 70 samples), so this is a simpler, evidence-backed
alternative.

* revise(brief): keep MAX_STORIES_PER_USER default at 12, add env-tunability

Reviewer asked "why 16?" and the honest answer turned out to be: the
data doesn't support it. After landing PR #3390's sweep harness with
visible-window metrics, re-ran against 2026-04-24 production replay:

  threshold=0.45 cap=12 -> visible_quality 0.916 (best at this cap)
  threshold=0.45 cap=16 -> visible_quality 0.716 (cap bump HURTS)
  threshold=0.42 cap=12 -> visible_quality 0.845
  threshold=0.42 cap=16 -> visible_quality 0.845 (neutral)

At the current 0.45 threshold, positions 13-16 are mostly singletons
or members of "should-separate" clusters — they dilute the brief
without helping topic adjacency. Bumping the cap default to 16 was a
wrong inference from the dropped_cap=18 signal alone.

Revised approach:

- Default MAX_STORIES_PER_USER stays at 12 (matches historical prod).
- Constant becomes env-tunable via DIGEST_MAX_STORIES_PER_USER so any
  future sweep result can be acted on with a Railway env flip without
  a redeploy.

The actual evidence-backed adjacency fix from the sweep is to lower
DIGEST_DEDUP_TOPIC_THRESHOLD from 0.45 -> 0.42 (env flip; see PR #3390).

* fix(brief-llm): tie buildDigestPrompt + hashDigestInput slice to MAX_STORIES_PER_USER

Greptile P1 on PR #3389: with MAX_STORIES_PER_USER now env-tunable,
hard-coded stories.slice(0, 12) in buildDigestPrompt and hashDigestInput
would mean the LLM prose only references the first 12 stories when
the brief carries more. Stories 13+ would appear as visible cards
but be invisible to the AI summary — a quiet mismatch between reader
narrative and brief content.

Cache key MUST stay aligned with the prompt slice or it drifts from
the prompt content; same constant fixes both sites.

Exports MAX_STORIES_PER_USER from brief-compose.mjs (single source
of truth) and imports it in brief-llm.mjs. No behaviour change at
the default cap of 12.
This commit is contained in:
Elie Habib
2026-04-25 12:07:48 +04:00
committed by GitHub
parent abdcdb581f
commit 3373b542e9
3 changed files with 46 additions and 5 deletions

View File

@@ -192,7 +192,14 @@ describe('composeBriefFromDigestStories — continued', () => {
assert.deepEqual(env.data.stories.map((s) => s.headline), ['A', 'B']);
});
it('caps at 12 stories per brief', () => {
it('caps at 12 stories per brief by default (env-tunable via DIGEST_MAX_STORIES_PER_USER)', () => {
// Default kept at 12. Offline sweep harness against 2026-04-24
// production replay showed cap=16 dropped visible_quality from
// 0.916 → 0.716 at the active 0.45 threshold (positions 13-16
// are mostly singletons or "should-separate" members at this
// threshold, so they dilute without helping adjacency). The
// constant is env-tunable so a Railway flip can experiment with
// cap values once new sweep evidence justifies them.
const many = Array.from({ length: 30 }, (_, i) =>
digestStory({ hash: `h${i}`, title: `Story ${i}` }),
);