Files
worldmonitor/scripts
Elie Habib 439b8f1c7d fix(gdelt): relax validate threshold 4→2 to prevent recurring STALE_SEED (#1842)
* fix(gdelt): relax validate threshold from 4 to 2 topics to prevent STALE on partial 429s

When GDELT rate-limits any single topic, fetchWithRetry returns { articles: [] }.
With validate requiring all 4 topics populated, a single 429 causes atomicPublish
to skip the write entirely — seed-meta TTL is NOT extended. After 2-3 consecutive
cron runs that fail validation, seedAge exceeds maxStaleMin:300 and health reports
STALE_SEED.

Fix: require only 2/4 topics to have articles. A partial snapshot is far better
than no write at all — it keeps seed-meta fresh and prevents recurring STALE alerts.

Also fix stale comment claiming cron runs every 4h (actual cron is 2h).

* fix(gdelt): merge previous snapshot for 429-failed topics instead of overwriting with empty

The earlier validate relaxation (>= 2) still overwrote previously cached articles
for rate-limited topics with empty arrays, causing blank tabs and empty RPC responses
for up to the 6h TTL even though good data was available.

Fix: after fetching all topics, read the existing canonical key from Redis. For any
topic that returned 0 articles due to 429 exhaustion, restore the previous snapshot's
articles for that topic before publishing. The validate >= 2 threshold remains as a
safety net for first-run or total GDELT outages.
2026-03-19 02:29:44 +04:00
..