fix(cache): digest TTL alignment + slow-browser tier + feedStatuses trim (#1798)

* fix(cache): align Redis digest + RSS feed TTLs to CF CDN TTL

RSS feed TTL 600s → 3600s; digest TTL 900s → 3600s.
CF CDN caches at 3600s, so Redis expiring earlier caused every hourly
CF revalidation to hit a cold origin and run the full buildDigest()
pipeline (75 feeds, up to 25s). Aligning both to 3600s ensures CF
revalidation gets a warm Redis hit and returns immediately.

* fix(cache): emit only non-ok feedStatuses; update proto comment + make generate

Digest was emitting 'ok' for every successful feed (~50 entries, ~1-2KB
per response). No in-repo client reads feedStatuses values. Changed to
only emit 'empty' and 'timeout'; absent key implies ok.

Updated proto comment to document the absence-implies-ok contract and
ran make generate to regenerate docs/api/ OpenAPI files.

* fix(cache): add slow-browser tier; move digest route to it

New 'slow-browser' tier is identical to 'slow' but adds max-age=300,
letting browsers skip the network for 5 minutes. Without max-age,
browsers ignore s-maxage and send conditional If-None-Match on every
20-min poll — each costing 1 billable edge request even for 304s.

Scoped only to list-feed-digest (a safe polling endpoint). Premium
user-triggered endpoints (analyze-stock, backtest-stock) stay on 'slow'
where browser caching is inappropriate.

* test: regression tests for feedStatuses and slow-browser tier

- digest-no-reclassify: assert buildDigest does not write 'ok' to feedStatuses
- route-cache-tier: include slow-browser in tier regex; assert slow-browser
  has max-age and slow tier does not

* fix(cache): add variant to per-feed RSS cache key

rss:feed:v1:${url} was shared across variants even though classifyByKeyword()
bakes variant-specific threat/category labels into the cached ParsedItem[].
Feeds shared between full and tech variants (Verge, Ars, HN, etc.) had
whichever variant populated the cache first control the other variant's
classifications for the full 3600s TTL — turning a pre-existing 10-minute
bleed-through into a 1-hour accuracy bug for the tech dashboard.

Fix: key is now rss:feed:v1:${variant}:${url}.

* fix(cache): bypass browser HTTP cache on digest fetch

max-age=300 on the slow-browser tier lets browsers serve the digest
from their HTTP cache for up to 5 minutes, including on explicit
in-app refresh (window.location.reload) or page reload after a
breaking event. Users would see stale data until the TTL expired.

Add cache: 'no-cache' to tryFetchDigest() so every fetch revalidates
against CF edge. CF returns 304 (minimal cost) when data is unchanged,
or 200 with the current digest. s-maxage and CF-level caching are
unaffected; max-age still benefits browser back/forward cache.

* fix(cache): 15-min consistent TTL + degrade guard for digest

Issue 1 — TTL alignment: Redis digest TTL reverted to 900s (from 3600).
slow-browser tier reduced from s-maxage=1800/CDN=3600 to s-maxage=900 on
both sides, matching the Redis TTL. The freshness window is now consistently
15 minutes across Redis, Vercel edge, and CF CDN. max-age=300 (browser
local) is kept to avoid unnecessary revalidations on tab switch.

Issue 2 — Cache poisoning: replaced cachedFetchJson in listFeedDigest with
explicit getCachedJson/setCachedJson. After buildDigest(), if total items
across all categories is 0 the response is treated as degraded: Redis write
is skipped and markNoCacheResponse(ctx.request) is called so the gateway
sets Cache-Control: no-store instead of the normal tier headers. This
prevents a transient bad run from poisoning Redis and browser/CDN for the
full TTL. Error paths also call markNoCacheResponse.

docs/api/NewsService.openapi.yaml

@@ -235,7 +235,9 @@ components:
                     type: object
                     additionalProperties:
                         type: string
-                    description: Per-feed status (ok/error/timeout)
+                    description: |-
+                        Per-feed status — only non-ok states emitted; absent key implies ok.
+                         Values: empty (feed returned 0 items), timeout (timed out during fetch).
                 generatedAt:
                     type: string
                     description: ISO 8601 timestamp of when this digest was generated

proto/worldmonitor/news/v1/list_feed_digest.proto

@@ -15,7 +15,8 @@ message ListFeedDigestRequest {
 message ListFeedDigestResponse {
   // Per-category buckets — keys match category names from feed config
   map<string, CategoryBucket> categories = 1;
-  // Per-feed status (ok/error/timeout)
+  // Per-feed status — only non-ok states emitted; absent key implies ok.
+  // Values: empty (feed returned 0 items), timeout (timed out during fetch).
   map<string, string> feed_statuses = 2;
   // ISO 8601 timestamp of when this digest was generated
   string generated_at = 3;

server/gateway.ts

@@ -24,12 +24,13 @@ export const serverOptions: ServerOptions = { onError: mapErrorToResponse };
 // NOTE: This map is shared across all domain bundles (~3KB). Kept centralised for
 // single-source-of-truth maintainability; the size is negligible vs handler code.
 
-type CacheTier = 'fast' | 'medium' | 'slow' | 'static' | 'daily' | 'no-store';
+type CacheTier = 'fast' | 'medium' | 'slow' | 'slow-browser' | 'static' | 'daily' | 'no-store';
 
 const TIER_HEADERS: Record<CacheTier, string> = {
   fast: 'public, s-maxage=300, stale-while-revalidate=60, stale-if-error=600',
   medium: 'public, s-maxage=600, stale-while-revalidate=120, stale-if-error=900',
   slow: 'public, s-maxage=1800, stale-while-revalidate=300, stale-if-error=3600',
+  'slow-browser': 'public, max-age=300, s-maxage=900, stale-while-revalidate=60, stale-if-error=1800',
   static: 'public, s-maxage=7200, stale-while-revalidate=600, stale-if-error=14400',
   daily: 'public, s-maxage=86400, stale-while-revalidate=7200, stale-if-error=172800',
   'no-store': 'no-store',
@@ -41,6 +42,7 @@ const TIER_CDN_CACHE: Record<CacheTier, string | null> = {
   fast: 'public, s-maxage=600, stale-while-revalidate=300, stale-if-error=1200',
   medium: 'public, s-maxage=1200, stale-while-revalidate=600, stale-if-error=1800',
   slow: 'public, s-maxage=3600, stale-while-revalidate=900, stale-if-error=7200',
+  'slow-browser': 'public, s-maxage=900, stale-while-revalidate=60, stale-if-error=1800',
   static: 'public, s-maxage=14400, stale-while-revalidate=3600, stale-if-error=28800',
   daily: 'public, s-maxage=86400, stale-while-revalidate=14400, stale-if-error=172800',
   'no-store': null,
@@ -127,7 +129,7 @@ const RPC_CACHE_TIER: Record<string, CacheTier> = {
   '/api/prediction/v1/list-prediction-markets': 'medium',
   '/api/forecast/v1/get-forecasts': 'medium',
   '/api/supply-chain/v1/get-chokepoint-status': 'medium',
-  '/api/news/v1/list-feed-digest': 'slow',
+  '/api/news/v1/list-feed-digest': 'slow-browser',
   '/api/intelligence/v1/classify-event': 'static',
   '/api/intelligence/v1/get-country-facts': 'daily',
   '/api/intelligence/v1/list-security-advisories': 'slow',

server/worldmonitor/news/v1/list-feed-digest.ts

@@ -6,7 +6,8 @@ import type {
   NewsItem as ProtoNewsItem,
   ThreatLevel as ProtoThreatLevel,
 } from '../../../../src/generated/server/worldmonitor/news/v1/service_server';
-import { cachedFetchJson, getCachedJsonBatch } from '../../../_shared/redis';
+import { cachedFetchJson, getCachedJson, setCachedJson, getCachedJsonBatch } from '../../../_shared/redis';
+import { markNoCacheResponse } from '../../../_shared/response-headers';
 import { sha256Hex } from '../../../_shared/hash';
 import { CHROME_UA } from '../../../_shared/constants';
 import { VARIANT_FEEDS, INTEL_SOURCES, type ServerFeed } from './_feeds';
@@ -106,10 +107,10 @@ async function fetchAndParseRss(
   variant: string,
   signal: AbortSignal,
 ): Promise<ParsedItem[]> {
-  const cacheKey = `rss:feed:v1:${feed.url}`;
+  const cacheKey = `rss:feed:v1:${variant}:${feed.url}`;
 
   try {
-    const cached = await cachedFetchJson<ParsedItem[]>(cacheKey, 600, async () => {
+    const cached = await cachedFetchJson<ParsedItem[]>(cacheKey, 3600, async () => {
       // Try direct fetch first
       let text = await fetchRssText(feed.url, signal).catch(() => null);
 
@@ -271,26 +272,44 @@ function toProtoItem(item: ParsedItem): ProtoNewsItem {
 }
 
 export async function listFeedDigest(
-  _ctx: ServerContext,
+  ctx: ServerContext,
   req: ListFeedDigestRequest,
 ): Promise<ListFeedDigestResponse> {
   const variant = VALID_VARIANTS.has(req.variant) ? req.variant : 'full';
   const lang = req.lang || 'en';
 
   const digestCacheKey = `news:digest:v1:${variant}:${lang}`;
-
   const fallbackKey = `${variant}:${lang}`;
+
+  const empty = (): ListFeedDigestResponse => ({ categories: {}, feedStatuses: {}, generatedAt: new Date().toISOString() });
+
   try {
-    const cached = await cachedFetchJson<ListFeedDigestResponse>(digestCacheKey, 900, async () => {
-      return buildDigest(variant, lang);
-    });
+    // Check Redis first — warm path returns immediately
+    const cached = await getCachedJson(digestCacheKey) as ListFeedDigestResponse | null;
     if (cached) {
       if (fallbackDigestCache.size > 50) fallbackDigestCache.clear();
       fallbackDigestCache.set(fallbackKey, { data: cached, ts: Date.now() });
+      return cached;
     }
-    return cached ?? fallbackDigestCache.get(fallbackKey)?.data ?? { categories: {}, feedStatuses: {}, generatedAt: new Date().toISOString() };
+
+    // Cold path — build fresh digest
+    const fresh = await buildDigest(variant, lang);
+    const totalItems = Object.values(fresh.categories).reduce((sum, b) => sum + b.items.length, 0);
+
+    if (totalItems > 0) {
+      // Good response: cache in Redis and update in-memory fallback
+      await setCachedJson(digestCacheKey, fresh, 900);
+      if (fallbackDigestCache.size > 50) fallbackDigestCache.clear();
+      fallbackDigestCache.set(fallbackKey, { data: fresh, ts: Date.now() });
+    } else {
+      // Degraded response: skip Redis write; prevent gateway/CDN from caching
+      markNoCacheResponse(ctx.request);
+    }
+
+    return fresh;
   } catch {
-    return fallbackDigestCache.get(fallbackKey)?.data ?? { categories: {}, feedStatuses: {}, generatedAt: new Date().toISOString() };
+    markNoCacheResponse(ctx.request);
+    return fallbackDigestCache.get(fallbackKey)?.data ?? empty();
   }
 }
 
@@ -328,7 +347,7 @@ async function buildDigest(variant: string, lang: string): Promise<ListFeedDiges
       const settled = await Promise.allSettled(
         batch.map(async ({ category, feed }) => {
           const items = await fetchAndParseRss(feed, variant, deadlineController.signal);
-          feedStatuses[feed.name] = items.length > 0 ? 'ok' : 'empty';
+          if (items.length === 0) feedStatuses[feed.name] = 'empty';
           return { category, items };
         }),
       );

src/app/data-loader.ts

@@ -275,7 +275,7 @@ export class DataLoaderManager implements AppModule {
     try {
       const resp = await fetch(
         toApiUrl(`/api/news/v1/list-feed-digest?variant=${SITE_VARIANT}&lang=${getCurrentLanguage()}`),
-        { signal: AbortSignal.timeout(this.digestRequestTimeoutMs) },
+        { cache: 'no-cache', signal: AbortSignal.timeout(this.digestRequestTimeoutMs) },
       );
       if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
       const data = await resp.json() as ListFeedDigestResponse;

tests/digest-no-reclassify.test.mjs

@@ -14,6 +14,10 @@ import { fileURLToPath } from 'node:url';
 
 const __dirname = dirname(fileURLToPath(import.meta.url));
 const src = readFileSync(resolve(__dirname, '..', 'src', 'app', 'data-loader.ts'), 'utf-8');
+const serverSrc = readFileSync(
+  resolve(__dirname, '..', 'server', 'worldmonitor', 'news', 'v1', 'list-feed-digest.ts'),
+  'utf-8',
+);
 
 describe('Digest branch must not reclassify with AI', () => {
   const digestBranchStart = src.indexOf("// Digest branch: server already aggregated feeds");
@@ -50,3 +54,12 @@ describe('Digest branch must not reclassify with AI', () => {
       'canQueueAiClassification should not be imported (no call sites remain)');
   });
 });
+
+describe('feedStatuses must not emit ok entries', () => {
+  it('buildDigest does not write ok to feedStatuses', () => {
+    assert.ok(
+      !serverSrc.includes("feedStatuses[feed.name] = items.length > 0 ? 'ok' : 'empty'"),
+      "feedStatuses must not write 'ok' entries — wastes payload on every response",
+    );
+  });
+});

tests/route-cache-tier.test.mjs

@@ -40,7 +40,7 @@ function extractGetRoutes() {
 function extractCacheTierKeys() {
   const gatewayPath = join(root, 'server', 'gateway.ts');
   const src = readFileSync(gatewayPath, 'utf-8');
-  const re = /'\/(api\/[^']+)':\s*'(fast|medium|slow|static|daily|no-store)'/g;
+  const re = /'\/(api\/[^']+)':\s*'(fast|medium|slow|slow-browser|static|daily|no-store)'/g;
   const entries = {};
   let m;
   while ((m = re.exec(src)) !== null) {
@@ -84,4 +84,11 @@ describe('RPC_CACHE_TIER route parity', () => {
       'Gateway still has medium default fallback — ensure all routes are explicit',
     );
   });
+
+  it('slow-browser tier includes max-age, slow tier does not', () => {
+    const gatewaySrc = readFileSync(join(root, 'server', 'gateway.ts'), 'utf-8');
+    assert.match(gatewaySrc, /slow-browser.*max-age/s, 'slow-browser tier must include max-age');
+    const slowLine = gatewaySrc.match(/^\s+slow: 'public.*'/m)?.[0] ?? '';
+    assert.ok(!slowLine.includes('max-age'), 'slow tier must NOT include max-age');
+  });
 });