Node 20's fetch() (undici) tries IPv6 first. Railway containers don't
support IPv6 (IPV6_NDISC failures in network trace), causing all seed
services to crash.
Fix: set NODE_OPTIONS=--dns-result-order=ipv4first via nixpacks.toml
so all Railway services prefer IPv4. Keeps Node 20 for import attributes.
* test: rewrite transit chart test as structural contract verification
Replace fragile source-string extraction + new Function() compilation
with structural pattern checks on the source code. Tests verify:
- render() clears chart before content change
- clearTransitChart() cancels timer, disconnects observer, destroys chart
- MutationObserver setup for DOM readiness detection
- Fallback timer for no-op renders (100-500ms range)
- Both callbacks (observer + timer) clean up each other
- Tab switch and collapse clear chart state
- Mount function guards against missing element/data
Replaces PR #1634's approach which was brittle (method body extraction,
TypeScript cast stripping, sandboxed execution).
* fix: log fetch error cause in seed retry and FATAL handlers
Node 20 fetch() throws TypeError('fetch failed') with the real error
hidden in err.cause (DNS, TLS, timeout). The current logging only shows
'fetch failed' which is useless for diagnosis. Now logs err.cause.message
in both withRetry() retries and FATAL catch blocks.
* fix(usni-fleet): add Node.js HTTP CONNECT proxy fallback, detect Cloudflare HTML
curl is not available in Railway's Railpack v0.18.0 containers. The seed
was failing with ENOENT on curl, then getting Cloudflare-blocked on
Node.js direct.
- Add fetchViaHttpProxy: Node.js HTTP CONNECT tunnel through residential
proxy (no curl dependency). Uses the same RESIDENTIAL_PROXY_AUTH env.
- Add Cloudflare HTML detection: reject early when response starts with
<!DOCTYPE instead of passing HTML to JSON.parse.
- Fallback chain: curl direct -> curl+proxy -> Node.js+proxy -> Node.js direct
- Add nixpacks.toml with curl for future Railpack builds
* fix: use ESM import for node:http (require breaks in .mjs)
Previously, every merge to main triggered a Vercel build even for
scripts-only changes (seed scripts, relay updates). Now checks if
any web-relevant files changed on main too, skipping the build when
only scripts/, docs/, .github/, etc. are modified.
* refactor(seeds): extract USNI fleet seed from relay with proxy support
USNI's Cloudflare blocks Railway IPs (JA3 fingerprinting), causing
the relay's USNI seed to fail with HTML response instead of JSON.
Changes:
- New seed-usni-fleet.mjs: standalone script with HTTP CONNECT proxy
support via RESIDENTIAL_PROXY_AUTH (falls back to direct fetch)
- Removed ~240 lines of USNI code from ais-relay.cjs
- TTL bumped from 21600s (6h) to 25200s (7h) for cold start buffer
- Exact same parsing logic (hull types, region coords, vessel extraction,
strike group detection, battle force summary)
- runSeed pattern with lock, validation, seed-meta, verification
Deploy: Railway cron service, 6h interval, needs RESIDENTIAL_PROXY_AUTH
* fix(seeds): only use RESIDENTIAL_PROXY_AUTH for USNI proxy
* fix(seeds): use OREF_PROXY_AUTH for USNI proxy (the one proxy available)
PR #1596 removed the feeds but left the domains in the allowlist.
The relay still accepted proxy requests for these 403-blocked domains
from clients with cached old bundles. Removed:
- breakingdefense.com (403)
- www.arabnews.com (403)
- www.aei.org (403)
- mymodernmet.com (403)
Updated all 3 copies: shared/, scripts/shared/, api/
TTL (1h) equaled cron interval (1h), leaving zero buffer for container
cold starts. Health saw EMPTY records=0 during the gap between key
expiry and next seed run. 70min TTL covers the cold start window.
- Log which providers are available before calling
- Log HTTP error response body (shows Groq 429 reason)
- Log when response is empty/short
- Log success with response length
- Log explicit "All providers failed" at end of loop
Warm-ping requests from Railway IPs were getting Cloudflare bot
challenge HTML instead of JSON. The relay's service-statuses warm-ping
works because it sends Origin: 'https://worldmonitor.app'. Added the
same header to seed-infra.mjs and seed-military-maritime-news.mjs.
Fixes: usniFleet STALE_SEED, cableHealth EMPTY_ON_DEMAND
- Supply chain: add severity floor (critical=0.55, high=0.35) and raise
multiplier from 0.7 to 0.9. Hormuz at 80/100 RED now produces ~60-75%
instead of 47%.
- GPS jamming: raise normalization range from 30 to 60 hexes, multiplier
from 0.5 to 0.7, add 10% bonus for 20+ hexes. 53 hexes now produces
~72% instead of 49%.
- LLM: log when no API keys configured (was silently skipping all providers)
- Logging: add per-domain breakdown and top-5 predictions to pipeline output
for Railway log diagnosis
Railway's Railpack builder sets NODE_OPTIONS=--expose-gc which Node 18
rejects during mise install. Node 20+ accepts this flag. All Railway
seed services using scripts/package.json will now build with Node 20.
The CorridorRisk API provides rich intelligence that we were storing
but not displaying. Now surfaced in the panel:
- risk_summary: live intelligence narrative shown in the description
area (e.g. "Armed confrontations are active across the Persian Gulf
with 52% of events classified as armed clashes")
- risk_report.action: routing recommendation shown when card is
expanded (e.g. "Recommend REROUTING via Cape of Good Hope for all
non-essential Gulf cargo")
Changes:
- Proto: add risk_summary and risk_report_action to TransitSummary
- Relay: extract risk_report.action in seedCorridorRisk, pass both
fields through seedTransitSummaries
- Handler: pass through to API response + include in description
- UI: riskSummary in risk row, riskReportAction in expanded view
corridorrisk: EMPTY in health despite relay running new code. The seed
produces zero log output, making it impossible to diagnose. Added:
- Log fetch start ("Fetching...") and in-flight skip
- Log HTTP error with response body and content-type
- Detect HTML responses (Cloudflare challenge) before JSON.parse
- Increase timeout from 10s to 15s for slow Railway regions
* feat(forecast): add AI Forecasts prediction module (Pro-tier)
MiroFish-inspired prediction engine that generates structured forecasts
across 6 domains (conflict, market, supply chain, political, military,
infrastructure) using existing WorldMonitor data streams.
- Proto definitions for ForecastService with GetForecasts RPC
- Dedicated seed script (seed-forecasts.mjs) with 6 domain detectors,
cross-domain cascade resolver, prediction market calibration, and
trend detection via prior snapshot comparison
- Premium-gated RPC handler (PREMIUM_RPC_PATHS enforcement)
- Lazy-loaded ForecastPanel with domain filters, probability bars,
trend arrows, signal evidence, and cascade links
- Health monitoring integration (seed-meta freshness tracking)
- Refresh scheduler with API key guard
* test(forecast): add 47 unit tests for forecast detectors and utilities
Covers forecastId, normalize, resolveCascades, calibrateWithMarkets,
computeTrends, and smoke tests for all 6 domain detectors. Exports
testable functions from seed script with direct-run guard.
* fix(forecast): domain mismatch 'infra' vs 'infrastructure', add panel category
- Seed script used 'infra' but ForecastPanel filtered on 'infrastructure',
causing Infra tab to show zero results
- Added 'forecast' to intelligence category in PANEL_CATEGORY_MAP
* fix(forecast): move CSS to one-time injection, improve type safety
- P2: Move style block from setContent to one-time document.head injection
to prevent CSS accumulation on repeated renders
- P3: Replace +toFixed(3) with Math.round for readability in seed script
- P3: Use Forecast type instead of any[] in RPC handler filter
* fix(forecast): handle sebuf proto data shapes from Redis
Detectors now normalize CII scores from server-side proto format
(combinedScore, TREND_DIRECTION_RISING, region) to uniform shape.
Outage severity handles proto enum format (SEVERITY_LEVEL_HIGH).
Added confidence floor of 0.3 for single-source predictions.
Verified against live Redis: 2 predictions generated (Iran infra
shutdown, IL political instability).
* feat(forecast): unlock AI Forecasts on web, lock desktop only (trial)
- Remove forecast RPC from PREMIUM_RPC_PATHS (web access is free)
- Panel locked on desktop only (same as oref-sirens/telegram-intel)
- Remove API key guards from data-loader and refresh scheduler
- Web users get full access during trial period
* chore: regenerate proto types with make generate
Re-ran make generate after rebasing on main. Plugin v0.7.0 dropped
@ts-nocheck from output, added it back to all 50 generated files.
Fixed 4 type errors from proto codegen changes:
- MarketSource enum -> string union type
- TemporalAnomalyProto -> TemporalAnomaly rename
- webcam lastUpdated number -> string
* fix(forecast): use chokepoints v4 key, include ciiContribution in unrest
- P1: Switch chokepoints input from stale v2 to active v4 Redis key,
matching bootstrap.js and cache-keys.ts
- P2: Add ciiContribution to unrest component fallback chain in
normalizeCiiEntry so political detector reads the correct sebuf field
* feat(forecast): Phase 2 LLM scenario enrichment + confidence model
MiroFish-inspired enhancements:
- LLM scenario narratives via Groq/OpenRouter (narrative-only, no numeric
adjustment). Evidence-grounded prompts with mandatory signal citation
and few-shot examples from MiroFish's SECTION_SYSTEM_PROMPT_TEMPLATE.
- Top-4 predictions batched into single LLM call for cost efficiency.
- News context from newsInsights attached to all predictions for LLM
prompt grounding (NOT in signals, cannot affect confidence).
- Deterministic confidence model: source diversity via SIGNAL_TO_SOURCE
mapping (deduplicates cii+cii_delta, theater+indicators) + calibration
agreement from prediction market drift. Floor 0.2, ceiling 1.0.
- Output validation: rejects scenarios without signal references.
- Truncated JSON repair for small model output.
- Structured JSON logging for LLM calls.
- Redis cache for LLM scenarios (1h TTL).
- 23 new tests (70 total), all passing.
- Live-tested: OpenRouter gemini-2.5-flash produces evidence-grounded
scenario narratives from real WorldMonitor data.
* feat(forecast): Phase 3 multi-perspective scenarios, projections, data-driven cascades
MiroFish-inspired enhancements:
- Multi-perspective LLM analysis: top-2 predictions get strategic,
regional, and contrarian viewpoints via combined LLM call
- Probability projections: domain-specific decay curves (h24/d7/d30)
anchored to timeHorizon so probability equals projections[timeHorizon]
- Data-driven cascade rules: moved from hardcoded array to JSON config
(scripts/data/cascade-rules.json) with schema validation, named
predicate evaluators, unknown key rejection, and fallback to defaults
- 4 new cascade paths: infrastructure->supply_chain, infrastructure->market
(both requiresSeverity:total), conflict->political, political->market
- Proto: added Perspectives and Projections messages to Forecast
- ForecastPanel: renders projections row and conditional perspectives toggle
- 89 tests (19 new), all passing
- Live-tested: OpenRouter produces perspectives from real data
* feat(forecast): Phase 4 data utilization + entity graph
Fixes data gaps that prevented 4 of 6 detectors from firing:
- Input normalizers: chokepoint v4 shape + GPS hexes-to-zones mapping
- Chokepoint warm-ping (production-only, requires WM_API_BASE_URL)
- Lowered CII conflict threshold from 70 to 60, gated on level=high|critical
4 new standalone detectors:
- UCDP conflict zones (10+ events per country)
- Cyber threat concentration (5+ threats per country)
- GPS jamming in maritime shipping zones (5 regions)
- Prediction markets as signals (60-90% probability markets)
Entity-relationship graph (file-based, 38 nodes):
- Countries, theaters, commodities, chokepoints, alliances
- Alias table resolves both ISO codes and display names
- Graph cascade discovery links predictions across entities
Result: 51 predictions (up from 1-2), spanning conflict, infrastructure,
and supply chain domains. 112 tests, all passing.
* fix(forecast): redis cache format, signal source mapping, type safety
Fresh-eyes audit fixes:
- BUG: redisSet used wrong Upstash API format (POST body with {value,ex}
instead of command array ['SET',key,value,'EX',ttl]). LLM cache writes
were silently failing, causing fresh LLM calls every run.
- BUG: prediction_market signal type missing from SIGNAL_TO_SOURCE,
inflating confidence for market-derived predictions.
- CLEANUP: Remove unnecessary (f as any) casts in ForecastPanel since
generated Forecast type already has projections/perspectives fields.
- CLEANUP: Bump health maxStaleMin from 60 to 90 to avoid false STALE
alerts when LLM calls add latency to seed runs.
* feat(forecast): headline-entity matching with news corroboration signals
Uses entity graph aliases to match headlines to predictions by
country/theater (excludes commodity/infrastructure nodes to prevent
false positives). Predictions with matching headlines get a
news_corroboration signal visible in the panel.
Also fixes buildUserPrompt to merge unique headlines from ALL
predictions in the LLM batch (was only reading preds[0].newsContext).
Live-tested: 13 of 51 predictions now have corroborating headlines
(Iran, Israel, Syria, Ukraine, etc). 116 tests, all passing.
* feat(forecast): add country-codes.json for headline-entity matching
56 countries with ISO codes, full names, and scoring keywords (extracted
from src/config/countries.ts + UCDP-relevant additions). Used by
attachNewsContext for richer headline matching via getSearchTermsForRegion
which combines country-codes + entity graph + keyword aliases.
14/57 predictions now have news corroboration (limited by headline
coverage, not matching quality: only 8 headlines currently available).
* feat(forecast): read 300 headlines from news digest instead of 8
Read news:digest:v1:full:en (300 headlines across 16 categories) instead
of just news:insights:v1 topStories (8 headlines). Fallback to topStories
if digest is unavailable.
Result: news corroboration jumped from 25% to 64% (38/59 predictions).
* fix(forecast): handle parenthetical country names in headline matching
Strip suffixes like '(Zaire)', '(Burma)', '(Soviet Union)' from UCDP
region names before matching against country-codes.json. Also use
includes() for reverse name lookup to catch partial matches.
Corroboration: 64% -> 69% (41/59). Remaining 18 unmatched are countries
with no current English-language news coverage.
* fix(forecast): cache validated LLM output, add digest test, log cache errors
Fresh-eyes audit fixes:
- Combined LLM cache now stores only validated items (was caching raw
unvalidated output, serving potentially invalid scenarios on cache hit)
- redisSet logs warnings on failure (was silently swallowing all errors)
- Added digest-based test for attachNewsContext (primary path was untested)
- Fixed test arity: attachNewsContext(preds, news, digest) with 3 params
* fix(forecast): remove dead confidenceFromSources, reduce warm-ping timeout
- P2: Remove confidenceFromSources (dead code, computeConfidence overwrites
all initial confidence values). Inline the formula in original detectors.
- P3: Reduce warm-ping timeout from 30s to 15s (non-critical step)
- P3: Add trial status comment on forecast panel config
* fix(forecast): resolve ISO codes to country names, fix market detector, safe pre-push
P1 fixes from code review:
- CII ISO codes (IL, IR) now resolved to full country names (Israel, Iran)
via country-codes.json. Prevents substring false positives (IL matching
Chile) in event correlation. Uses word-boundary regex for matching.
- Market detector CII-to-theater mapping now uses entity graph traversal
instead of broken theater-name substring matching. Iran correctly maps
to Middle East theater via graph links.
- Pre-push hook no longer runs destructive git checkout on proto freshness
failure. Reports mismatch and exits without modifying worktree.
* fix(seeds): rethrow non-fetch failures in runSeed()
Split runSeed() into two phases so only upstream fetch errors get
the graceful TTL-extension path. Redis publish, seed-meta, and
verification failures now rethrow (exit 1) so monitoring catches them.
* fix(seeds): separate fetch from publish errors in standalone scripts
Split seed-airport-delays, seed-military-flights, and
seed-service-statuses into two phases matching runSeed() pattern:
- Phase 1: upstream fetch errors are graceful (extend TTL, exit 0)
- Phase 2: Redis publish/verify errors propagate (exit 1)
* fix(seeds): make Redis SET throw on failure so publish errors propagate
Local redisSet() returned false instead of throwing, silently masking
Redis write failures. writeExtraKey() also warned instead of throwing.
Both now throw on non-OK responses, ensuring Phase 2 catch fires.
* fix(seed): treat empty Redis key after successful RPC as publish failure
When cachedFetchJson() silently swallows a Redis write failure, the
warm-ping script now throws instead of warning, reaching the outer
catch handler (exit 1) so monitoring detects the issue.
* perf: reduce uncached API calls via client-side circuit breaker caches
Add client-side circuit breaker caches with IndexedDB persistence to the
top 3 uncached API endpoints (CF analytics: 10.5M uncached requests/day):
- classify-events (5.37M/day): 6hr cache per normalized title, shouldCache
guards against caching null/transient failures
- get-population-exposure (3.45M/day): 6hr cache per coordinate key
(toFixed(4) for ~11m precision), 64-entry LRU
- summarize-article (1.68M/day): 2hr cache per headline-set hash via
buildSummaryCacheKey, eliminates both cache-check and summarize RPCs
Fix workbox-*.js getting no-cache headers (3.62M/day): exclude from SPA
catch-all regex in vercel.json, add explicit immutable cache rule for
content-hashed workbox files.
Migrate USNI fleet fetch from Vercel edge to Railway relay (gold standard):
- Add seedUSNIFleet() loop to ais-relay.cjs (6hr interval, gzip support)
- Make server handler Redis-read-only (435 lines reduced to 38)
- Move usniFleet from ON_DEMAND to BOOTSTRAP_KEYS in health.js
- Add persistCache + shouldCache to client breaker
Estimated reduction: ~14.3M uncached requests/day.
* fix: address code review findings (P1 + P2)
P1: Include SummarizeOptions in summary cache key to prevent cross-option
cache pollution (e.g. cloud summary replayed after user disables cloud LLMs).
P2: Document that forceRefresh is intentionally ignored now that USNI
fetching moved to Railway relay (Vercel is Redis-read-only).
* fix: reject forceRefresh explicitly instead of silently ignoring it
Return an error response with explanation when forceRefresh=true is sent,
rather than silently returning cached data. Makes the behavior regression
visible to any caller instead of masking it.
* fix(build): set worker.format to 'es' for Vite 6 compatibility
Vite 6 defaults worker.format to 'iife' which fails with code-splitting
workers (analysis.worker.ts uses dynamic imports). Setting 'es' fixes
the Vercel production build.
* fix(test): update deploy-config test for workbox regex exclusion
The SPA catch-all regex test hard-coded the old pattern without the
workbox exclusion. Update to match the new vercel.json source pattern.
* feat(seeds): add Railway seed scripts for economic and trade endpoints
Two new seed scripts to eliminate Vercel edge external API calls:
seed-economy.mjs:
- EIA energy prices (WTI, Brent) -> economic:energy:v1:all
- EIA energy capacity (Solar, Wind, Coal) -> economic:capacity:v1:COL,SUN,WND:20
- FRED series (10 series) -> economic:fred:v1:<id>:120
- Macro signals (Yahoo, Alternative.me, Mempool) -> economic:macro-signals:v1
seed-supply-chain-trade.mjs:
- Shipping rates (FRED) -> supply_chain:shipping:v2
- Trade barriers (WTO tariff gap) -> trade:barriers:v1:tariff-gap:50
- Trade restrictions (WTO MFN overview) -> trade:restrictions:v1:tariff-overview:50
- Trade flows (WTO, 15 major reporters) -> trade:flows:v1:<reporter>:000:10
- Tariff trends (WTO, 15 major reporters) -> trade:tariffs:v1:<reporter>:all:10
Cache keys match handler patterns exactly so cachedFetchJson finds
pre-seeded data and avoids live external API calls from Vercel edge.
* feat(seeds): add seed-aviation.mjs for airport ops and aviation news
Seeds 2 aviation endpoints with predictable default params:
- getAirportOpsSummary (AviationStack + NOTAM) -> aviation:ops-summary:v1:CDG,ESB,FRA,IST,LHR,SAW
- listAviationNews (9 RSS feeds, 24h window) -> aviation:news::24:v1
NOT seeded (inherently on-demand, user-specific inputs):
- getFlightStatus: specific flight number lookup
- trackAircraft: bounding-box or icao24 queries
- listAirportFlights: arbitrary airport+direction+limit combos
- getCarrierOps: depends on listAirportFlights with variable params
* feat(seeds): add seed-conflict-intel.mjs for ACLED, HAPI, and PizzINT
Seeds 3 conflict/intelligence endpoints with predictable default params:
- listAcledEvents (all countries, last 30 days) -> conflict:acled:v1:all:0:0
- getHumanitarianSummary (20 top conflict countries) -> conflict:humanitarian:v1:<CC>
- getPizzintStatus (base + GDELT variants) -> intel:pizzint:v1:base, intel:pizzint:v1:gdelt
NOT seeded (inherently on-demand, LLM or user-specific inputs):
- classifyEvent: per-headline LLM classification
- deductSituation: per-query LLM deduction
- getCountryIntelBrief: per-country LLM brief with context hash
- getCountryFacts: per-country REST Countries + Wikidata + Wikipedia
- searchGdeltDocuments: per-query GDELT search
Requires: ACLED_EMAIL, ACLED_KEY, UPSTASH_REDIS_REST_URL/TOKEN
* feat(seeds): add seed-research.mjs for arXiv, HN, tech events, trending repos
Seeds 4 research endpoints:
- listArxivPapers (cs.AI, cs.CL, cs.CR) -> research:arxiv:v1:<cat>::50
- listHackernewsItems (top, best feeds) -> research:hackernews:v1:<feed>:30
- listTechEvents (Techmeme ICS + dev.events RSS) -> research:tech-events:v1
- listTrendingRepos (python, javascript, typescript) -> research:trending:v1:<lang>:daily:50
Tech events key is also seeded by the relay, this script provides backup
hydration and ensures the key is warm even if relay hasn't run yet.
Requires: UPSTASH_REDIS_REST_URL/TOKEN
* feat(seeds): add seed-military-maritime-news.mjs for USNI and nav warnings
Seeds 2 endpoints with predictable default params:
- USNI Fleet Report (WordPress JSON API) -> usni-fleet:sebuf:v1 + stale backup
- Navigational Warnings (NGA broadcast, all areas) -> maritime:navwarnings:v1:all
NOT seeded (inherently on-demand):
- getAircraftDetails/batch: per-icao24 Wingbits lookup
- listMilitaryFlights: bounding-box query (quantized 1-degree grid)
- getVesselSnapshot: in-memory cache, reads from relay /ais-snapshot
- listFeedDigest: per-feed-URL RSS caching (hundreds of feeds, relay proxied)
- summarizeArticle: per-article LLM summarization
Requires: UPSTASH_REDIS_REST_URL/TOKEN
* feat(seeds): add seed-infra.mjs warm-ping for service statuses and cable health
Uses warm-ping pattern (calls Vercel RPC from Railway) because:
- list-service-statuses: 30 status page parsers with 8 custom formats
- get-cable-health: NGA text analysis with cable name matching + proximity
Replicating this logic in a standalone script is fragile and duplicative.
NOT seeded (on-demand):
- search-imagery: per-bbox/datetime STAC query
- get-giving-summary: hardcoded baselines, no external fetches
- get-webcam-image: per-webcamId Windy API lookup
* fix(seeds): move secondary key writes before process.exit, fix data shapes
Critical bugs found in code review:
1. runSeed() calls process.exit(0) after primary key write, so .then()
callbacks were dead code. All secondary keys (FRED, macro signals,
trade data, HAPI summaries, pizzint, HN, trending, etc.) were NEVER
written. Fix: move writeExtraKey calls inside fetchAll() before return.
2. FRED cache key used :120 suffix but handler default is :0 (req.limit||0).
Fixed to :0 so seed matches handler cache key for default requests.
3. USNI and nav warnings seed parsers produced wrong data shapes vs handler
(different field names, missing fields). Converted to warm-ping pattern
(like seed-infra.mjs) to avoid shape divergence.
* fix(seeds): reduce GDELT 429 rate limiting in seed-gdelt-intel
Problems from logs: every topic fetch hits 429, runs take 3-5min,
4th run failed fatally after 12min of cascading retries.
Fixes:
- Increase inter-topic delay: 12s -> 20s (GDELT needs longer cooldown)
- Increase initial backoff: 10s -> 20s, with 15s increments per retry
- Graceful degradation: exhausted retries return empty topic instead of
throwing (prevents withRetry from restarting ALL topics from scratch)
- Align TTL with health.js: 3600s -> 7200s (matches maxStaleMin:120)
- Validation allows partial success (3/6 topics minimum)
Cron interval should also be increased from 30min to 2h on Railway
to match the new 2h TTL.
* fix(seeds): 4 bugs from review - ACLED auth, NOTAM key, infra precedence, curated events
P1: ACLED auth used wrong endpoint (api/acled/token) and env vars (ACLED_KEY).
Fixed to match server/acled-auth.ts: ACLED_EMAIL+ACLED_PASSWORD via /oauth/token,
with ACLED_ACCESS_TOKEN static fallback.
P1: Aviation NOTAM key was aviation:notam-closures:v1, handler reads
aviation:notam:closures:v2. Fixed key to match _shared.ts.
P2: Infra warm-ping had operator precedence bug in nullish coalescing:
(a ?? b) ? c : d instead of a ?? (b ? c : d). Added parens.
P2: Research seed missed curated conferences that the handler appends
(CURATED_EVENTS in list-tech-events.ts). Added same curated events so
seeded data matches what the handler would produce.
* fix(seeds): add seed-meta freshness metadata for all secondary keys
Added writeExtraKeyWithMeta() to _seed-utils.mjs that writes both the
data key and a seed-meta:<key> freshness metadata entry. All secondary
key writes in seed scripts now use this helper so health.js can track
freshness for: energy capacity, FRED series, macro signals, trade
barriers/restrictions/flows/tariffs, aviation news, HAPI summaries,
PizzINT, arXiv categories, HN feeds, tech events, trending repos.
Previously only the primary key per script got seed-meta (via runSeed),
leaving secondary keys operationally invisible to health monitoring.
* fix(seeds): align seed-meta keys with health.js conventions
P1: writeExtraKeyWithMeta wrote seed-meta:<full-cache-key> (e.g.,
seed-meta:economic:macro-signals:v1), but health.js expects normalized
names without version suffixes (seed-meta:economic:macro-signals).
Fixed by stripping trailing :v\d+ from key. Added metaKeyOverride
param for cases needing explicit control.
P1: shipping seed used runSeed('supply-chain', 'shipping-trade', ...)
producing seed-meta:supply-chain:shipping-trade, but health.js expects
seed-meta:supply_chain:shipping. Fixed domain/resource to match.
* fix(seeds): only write seed-meta after successful data key write
writeExtraKey() now returns false on failure. writeExtraKeyWithMeta()
skips seed-meta write when the data write fails, preventing false-positive
health reports for keys like macro-signals and tech-events.
* fix(seeds): extend TTL on stale data instead of crashing on fetch errors
Seed scripts crashed with process.exit(1) when upstream APIs returned
errors (e.g., Wingbits 401), causing Redis keys to expire and panels
to lose data. Now all seeds gracefully extend TTL on existing keys and
exit 0, keeping stale data alive until the API recovers.
- Add shared extendExistingTtl() helper to _seed-utils.mjs
- Update runSeed() catch block (fixes 24 scripts using it)
- Fix fetch-gpsjam.mjs, seed-airport-delays.mjs,
seed-military-flights.mjs, seed-service-statuses.mjs
* fix(seeds): preserve per-key TTLs when extending stale military data
THEATER_POSTURE_BACKUP_KEY has a 7-day TTL (604800s) but was being
extended with STALE_TTL (86400s), shortening it from 7 days to 1 day
during upstream outages. Now each key group gets its original TTL.
* fix(supply-chain): increase Redis timeout for PortWatch and remove content height cap
Root cause: getCachedJson has a 1500ms timeout, but the PortWatch
payload (~149KB for 13 chokepoints x 175 days) exceeds this on
high-latency Edge regions. The fetch silently times out and returns
null, so the handler builds responses with empty transit summaries.
Fix: add optional timeoutMs param to getCachedJson, use 5000ms for
the PortWatch fetch. Also remove the 300px max-height on
.economic-content so the Supply Chain panel fills available height.
* refactor(supply-chain): move transit summary assembly to Railway relay
Vercel Edge was reading 3 large Redis keys (PortWatch 149KB, transit
counts, CorridorRisk) and assembling transit summaries on every request.
The 1500ms Redis timeout caused the 149KB PortWatch fetch to silently
fail on high-latency Edge regions (Mumbai bom1), leaving all transit
data empty.
Now Railway builds the pre-assembled transit summaries (including
anomaly detection) and writes them to a single key. Vercel reads
ONE small pre-built key instead of 3 raw keys.
Flow: Railway seeds PortWatch + transit counts -> builds summaries ->
writes supply_chain:transit-summaries:v1 -> Vercel reads it.
This follows the gold standard: "Vercel reads Redis ONLY; Railway
makes ALL external API calls and data assembly."
* test(supply-chain): add sync tests for relay threat levels and name mappings
detectTrafficAnomalyRelay and CHOKEPOINT_THREAT_LEVELS in the relay are
duplicated from _scoring.mjs and get-chokepoint-status.ts because
ais-relay.cjs is CJS. Added sync tests that validate:
- Every canonical chokepoint has a relay threat level
- Relay threat levels match handler CHOKEPOINTS config
- RELAY_NAME_TO_ID covers all canonical chokepoints
This catches drift between the two source-of-truth files.
* fix(ui): restore bounded scroll on economic-content with flex layout
The previous fix replaced max-height: 300px with flex: 1 1 auto, but
.panel-content was not a flex container so the flex rule was ignored.
This caused tabs to scroll away with the content.
Fix: use :has(.economic-content) to make .panel-content a flex column
only for panels with tabbed economic content. Tabs stay pinned, content
area scrolls independently.
* feat(supply-chain): fix CorridorRisk API integration (open beta, no key needed)
The CorridorRisk API is in open beta at corridorrisk.io/api/corridors
(not api.corridorrisk.io/v1/corridors). No API key required during beta.
Changes:
- Fix URL to corridorrisk.io/api/corridors
- Remove API key requirement (open beta)
- Update name matching for actual API names (e.g. "Persian Gulf &
Strait of Hormuz" -> hormuz_strait)
- Derive riskLevel from score (>=70 critical, >=50 high, etc.)
- Store riskScore, vesselCount, eventCount7d, riskSummary
- Feed CorridorRisk data into transit summaries
* test(supply-chain): comprehensive transit summary integration tests
75 tests across 10 suites covering:
- Relay seedTransitSummaries assembly (Redis key, fields, triggers)
- CorridorRisk name mapping and risk level derivation from score
- Handler reads pre-built summaries (not raw upstream keys)
- Handler isolation: no PortWatchData/CorridorRiskData/CANONICAL_CHOKEPOINTS imports
- detectTrafficAnomalyRelay sync with _scoring.mjs (side-by-side execution)
- detectTrafficAnomaly edge cases (boundaries, threat levels, unsorted history)
- CHOKEPOINT_THREAT_LEVELS relay-handler sync validation
* fix(supply-chain): hydrate transit summaries from Redis on relay restart
After relay restart, latestPortwatchData and latestCorridorRiskData are
null. The initial seedTransitSummaries call (35s after boot) would
return early with no data, leaving the transit-summaries:v1 key stale
until the next PortWatch seed completes (6+ seconds later).
Fix: seedTransitSummaries now reads persisted PortWatch and CorridorRisk
data from Redis when in-memory state is empty. This covers the cold-start
gap so Vercel always has fresh transit summaries.
Also adds 5 tests validating the hydration path order and assignment.
* fix(supply-chain): add fallback to raw Redis keys when pre-built summaries are empty
P1: If supply_chain:transit-summaries:v1 is absent (relay not deployed,
restart in progress, or transient PortWatch failure), the handler now
falls back to reading the raw portwatch, corridorrisk, and transit count
keys directly and assembling summaries on the fly.
This ensures corridor risk data (riskLevel, incidentCount7d, disruptionPct)
is never silently zeroed out, and users keep history/counts even during
the 6-hour PortWatch re-seed window.
Strategy: pre-built summaries (fast path) -> raw keys fallback (slow path)
-> all-zero defaults (last resort).
* feat: harness engineering P0 - linting, testing, architecture docs
Add foundational infrastructure for agent-first development:
- AGENTS.md: agent entry point with progressive disclosure to deeper docs
- ARCHITECTURE.md: 12-section system reference with source-file refs and ownership rule
- Biome 2.4.7 linter with project-tuned rules, CI workflow (lint-code.yml)
- Architectural boundary lint enforcing forward-only dependency direction (lint-boundaries.mjs)
- Unit test CI workflow (test.yml), all 1083 tests passing
- Fixed 9 pre-existing test failures (bootstrap sync, deploy-config headers, globe parity, redis mocks, geometry URL, import.meta.env null safety)
- Fixed 12 architectural boundary violations (types moved to proper layers)
- Added 3 missing cache tier entries in gateway.ts
- Synced cache-keys.ts with bootstrap.js
- Renamed docs/architecture.mdx to "Design Philosophy" with cross-references
- Deprecated legacy docs/Docs_To_Review/ARCHITECTURE.md
- Harness engineering roadmap tracking doc
* fix: address PR review feedback on harness-engineering-p0
- countries-geojson.test.mjs: skip gracefully when CDN unreachable
instead of failing CI on network issues
- country-geometry-overrides.test.mts: relax timing assertion
(250ms -> 2000ms) for constrained CI environments
- lint-boundaries.mjs: implement the documented api/ boundary check
(was documented but missing, causing false green)
* fix(lint): scan api/ .ts files in boundary check
The api/ boundary check only scanned .js/.mjs files, missing the 25
sebuf RPC .ts edge functions. Now scans .ts files with correct rules:
- Legacy .js: fully self-contained (no server/ or src/ imports)
- RPC .ts: may import server/ and src/generated/ (bundled at deploy),
but blocks imports from src/ application code
* fix(lint): detect import() type expressions in boundary lint
- Move AppContext back to app/app-context.ts (aggregate type that
references components/services/utils belongs at the top, not types/)
- Move HappyContentCategory and TechHQ to types/ (simple enums/interfaces)
- Boundary lint now catches import('@/layer') expressions, not just
from '@/layer' imports
- correlation-engine imports of AppContext marked boundary-ignore
(type-only imports of top-level aggregate)
* fix(supply-chain): use ArcGIS timestamp syntax for PortWatch date filter
The PortWatch seed loop was silently failing on every cycle because
`date >= <epoch_ms>` is not valid ArcGIS Feature Service syntax.
ArcGIS requires `date >= timestamp 'YYYY-MM-DD HH:MM:SS'`.
This caused: no chart history, no transit counts, no WoW% in the
Supply Chain panel chokepoints tab (all added in PRs #1560/#1572/#1577).
Verified: 13/13 chokepoints return 175 days of history with the fix.
Also adds chokepointTransits to health.js STANDALONE_KEYS and SEED_META
so the transit counter seed freshness is monitored.
* fix(supply-chain): preserve full UTC time in PortWatch timestamp filter
pwEpochToTimestamp() was hardcoding 00:00:00, expanding the 180-day
window to the start of that UTC date. Now preserves HH:MM:SS from the
original epoch to match the intended query boundary exactly.
* perf(relay): add upstream compression, keep-alive, brotli, and OREF pre-serialization
1. OpenSky upstream: request gzip/br/deflate via Accept-Encoding header
and decompress responses, reducing inbound transfer on cache misses
2. HTTPS keep-alive agent: reuse TCP+TLS connections to OpenSky,
eliminating ~2-5KB handshake overhead per request
3. Brotli support: sendCompressed and sendPreGzipped now prefer br
encoding (~15-20% smaller than gzip on JSON). Pre-computed brotli
buffers stored alongside gzip in all caches (OpenSky, AIS snapshot)
4. OREF pre-serialization: pre-compute alerts and history JSON + gzip +
brotli buffers on each poll, switching handlers from on-the-fly
sendCompressed to zero-CPU sendPreGzipped
All changes reduce proxy egress volume (charged by volume).
* fix(relay): re-serialize OREF cache on poll failure
Ensures _alertsCache reflects lastError when a poll fails,
so /oref/alerts clients see the error instead of stale
success data. Addresses Codex P1 review comment.
* fix(relay): reject on decompression failure and honor Accept-Encoding q-values
P1: _collectDecompressed now rejects on stream error instead of
resolving with partial data. Callers in _openskyRawFetch catch the
rejection and return { status: 0, error }, preventing malformed
responses from being cached as valid 200s.
P2: Replace naive includes('br')/includes('gzip') with
_acceptsEncoding() that parses q-values. A client sending
br;q=0 will no longer receive Brotli-encoded responses.
* fix(supply-chain): correct PortWatch ArcGIS service URL, field names, and chokepoint mappings
The PortWatch seed was returning no data because the ArcGIS service name,
WHERE clause fields, date field, and chokepoint names were all wrong.
Verified all 12 chokepoints return 175 days of data against the live API.
Added error logging to pwFetchAllPages for future debugging.
* fix(supply-chain): sync geofence names with relayName renames
CHOKEPOINT_GEOFENCES in ais-relay.cjs still used old names
('Strait of Malacca', 'Bab el-Mandeb', 'Strait of Gibraltar')
while _chokepoint-ids.ts relayName was updated. buildRelayLookup
does exact string match, so these 3 chokepoints had zero transit
counts despite relay data being present.
Rename geofence entries to match the new relayName values and
update corresponding test assertions.
The Wingbits function silently returned null on missing API key,
HTTP errors, and caught exceptions. Now logs the specific failure
reason so issues are visible in Railway logs.
* fix(supply-chain): correct PortWatch ArcGIS URL, field names, and chokepoint mappings
The PortWatch seed was failing silently because:
1. Wrong service name: portal_chokepoint_daily -> Daily_Chokepoints_Data
2. Wrong query fields: chokepoint/observation_date -> portname/date (epoch)
3. Wrong data model: expected one row per vessel type, actual schema has
all counts as columns (n_tanker, n_cargo, n_total) per row
4. Wrong chokepoint names: e.g. "Strait of Malacca" -> "Malacca Strait",
"Bab el-Mandeb" -> "Bab el-Mandeb Strait", "Bosphorus" -> "Bosporus Strait"
5. Removed Dardanelles (not in PortWatch dataset)
Discovered via IMF PortWatch ArcGIS service directory and returnDistinctValues
query on the portname field.
* feat(supply-chain): add Korea, Dover, Kerch, Lombok chokepoints
Extend from 10 to 14 monitored chokepoints using PortWatch data
availability. All 4 new straits have IMF PortWatch coverage.
- Korea Strait: Japan-Korea trade, busiest East Asia corridor
- Dover Strait: world's busiest shipping lane
- Kerch Strait: war_zone (Russia controls, Ukraine grain restricted)
- Lombok Strait: Malacca bypass for VLCCs
Added to: handler config, canonical ID map, PortWatch seed names,
AIS relay transit counter, tests.
* docs: update maritime docs and changelog for 14 chokepoints + transit intelligence
- maritime-intelligence.mdx: 9 -> 14 chokepoints, add data source descriptions,
add chart rendering note
- changelog.mdx + CHANGELOG.md: add [Unreleased] section for #1560 and #1572
* fix(tests): update portwatch test for pre-aggregated column model
pwClassifyVesselType was removed when switching to pre-aggregated
n_tanker/n_cargo/n_total columns. Update test to verify the new
field names instead.
* fix(supply-chain): sync canonical PortWatch names with actual ArcGIS feed
P1: Dardanelles has no PortWatch data (0 rows). Set portwatchName to empty
string so it won't attempt fetch or show phantom zero history.
P2: portwatchNameToId() returned undefined for Malacca Strait, Bab el-Mandeb
Strait, Gibraltar Strait, Bosporus Strait because canonical map used
old names instead of actual ArcGIS portname values.
Fixed mappings:
Strait of Malacca -> Malacca Strait
Bab el-Mandeb -> Bab el-Mandeb Strait
Strait of Gibraltar -> Gibraltar Strait
Bosphorus -> Bosporus Strait
Dardanelles -> '' (not in PortWatch)
* refactor(supply-chain): merge Dardanelles into Turkish Straits
IMF PortWatch tracks Bosphorus+Dardanelles as a single corridor
(Bosporus Strait). Keeping them separate caused double-counting in
AIS transit data and left Dardanelles with permanently empty history.
- Merge into single "Turkish Straits" entry (id stays 'bosphorus')
- Absorb all Dardanelles keywords (canakkale, gallipoli, aegean)
- Single wider AIS geofence (lat 40.70, lon 28.0, radius 1.5)
- 14 -> 13 chokepoints
- Update docs, changelog, tests
* fix: rename Turkish Straits to Bosporus Strait (match PortWatch naming)
* feat(correlation): server-side correlation engine seed + bootstrap hydration
Move correlation card computation from client-side (per-browser, 10-30s delay)
to server-side (Railway cron, instant via bootstrap). Seed script reads 8 Redis
keys, runs 4 adapter signal collectors (military, escalation, economic, disaster),
clusters/scores/generates cards, writes to Redis with 10min TTL.
- New: scripts/seed-correlation.mjs (pure JS port of correlation engine)
- bootstrap.js: add correlationCards to FAST_KEYS tier
- health.js + seed-health.js: register for monitoring (maxStaleMin: 15)
- CorrelationPanel: consume bootstrap on construction, show "Analyzing..." only
after live engine has run (not for bootstrap-only cards)
- _seed-utils.mjs: support opts.recordCount override (function or number)
* fix(correlation): stale timestamp fallback + coordinate-based country resolution
P1: news stories lacked per-story pubDate, causing Date.now() fallback on
every seed run. Now _clustering.mjs propagates pubDate through to
enrichedStories, and seed-correlation reads s.pubDate then generatedAt.
P2: normalizeToCode dropped signals with unparseable country names.
Added centroid-based coordinate fallback (haversine nearest-match within
800km) matching the live engine's getCountryAtCoordinates behavior.
* fix(correlation): add 11 missing country centroids to coordinate fallback
CI, CR, CV, CY, GA, IS, LA, SZ, TL, TT, XK were in the normalization
maps but missing from COUNTRY_CENTROIDS, causing coordinate-only signals
in those countries to be misclassified or dropped during bootstrap.
* fix(correlation): align protest/outage field names with actual Redis schema
Codex review P1 findings: seed-correlation read wrong field names from
Redis data.
Protests (unrest:events:v1): p.time -> p.occurredAt, p.lat/lon ->
p.location.latitude/longitude, severity enum SEVERITY_LEVEL_* mapping.
Outages (infra:outages:v1): o.pubDate -> o.detectedAt, o.lat/lon ->
o.location.latitude/longitude, severity enum OUTAGE_SEVERITY_* mapping.
Both escalation and disaster adapters updated. Old field names kept as
fallbacks for data shape compatibility.
* feat(supply-chain): replace S&P Global with 3 free maritime data sources
Replace expensive S&P Global Maritime API with IMF PortWatch (vessel transit
counts), CorridorRisk (risk intelligence), and AISStream chokepoint crossing
counter. All external API calls run on Railway relay, Vercel reads Redis only.
- Add 4 new chokepoints (10 total): Cape of Good Hope, Gibraltar, Bosphorus, Dardanelles
- Add TransitSummary proto (field 14) with today counts, WoW%, 180d history, risk context
- Add D3 multi-line chart (tanker vs cargo) with expandable chokepoint cards
- Add crossing detection with enter+dwell+exit semantics, 30min cooldown, 5min min dwell
- Add PortWatch seed loop (6h), CorridorRisk seed loop (1h), transit seed loop (10min)
- Add canonical chokepoint ID map for cross-source name resolution
- 177 tests passing across 6 test files
* fix(supply-chain): address P2 review findings
- Discard partial PortWatch pagination results on mid-page failure (prevents
truncated history with wrong WoW numbers cached for 6h)
- Rename "Transit today" to "24h" label (rolling 24h window, not calendar day)
- Fix chart label from "30d" to "180d" (matches actual PortWatch query range)
- Add 30s initial seed for chokepoint transits on relay cold start (prevents
10min gap of zero transit data)
* feat(supply-chain): swap D3 chart for TradingView lightweight-charts
Replace hand-rolled D3 SVG transit chart with lightweight-charts v5 canvas
rendering for Bloomberg-quality time-series visualization.
- Add TransitChart helper class with mount/destroy lifecycle, theme listener,
and autoSize support
- Use MutationObserver (not rAF) to mount chart after setContent debounce
- Clean up chart on tab switch, collapse, and re-render (no orphaned canvases)
- Respond to theme-changed events via chart.applyOptions()
- D3 stays for other 5 components (ProgressCharts, RenewableEnergy, etc.)
* feat(supply-chain): add geo coords and trade routes for 4 new chokepoints
Cherry-pick from PR #1511: Cape of Good Hope, Gibraltar, Bosphorus, and
Dardanelles map-layer coordinates and trade route definitions.
* fix(supply-chain): health.js v2->v4 key + double cache TTLs for missed seeds
- health.js chokepoints key was still v2, now v4 (matches handler + bootstrap)
- PortWatch TTL: 21600s (6h) -> 43200s (12h), seed interval stays 6h
- CorridorRisk TTL: 3600s (1h) -> 7200s (2h), seed interval stays 1h
- Ensures one missed seed run doesn't expire the key and cause empty data
Windy API returns 400 at ~1050 offset globally regardless of bounding
box size. The quadrant-splitting on 400 caused infinite recursion since
every sub-region hit the same limit.
On 400: keep cameras already fetched, stop paginating. The 10K cap
split is retained with MAX_SPLIT_DEPTH=3 to prevent runaway recursion.
Mixing || and ?? in the same expression without explicit grouping is
a JS syntax error. This broke ALL Railway seed scripts after #1556.
Refactored to use ?? throughout with explicit Array.isArray guard so
non-topic seeds correctly fall through to their own length checks.
- Rename `const seedMeta` to `seedMetaVal` to avoid shadowing the
`seedMeta()` function, which caused "Cannot access before initialization"
- Auto-split into quadrants when Windy API returns 400 (offset limit ~1000),
instead of only splitting at the 10K safety cap
* feat: seed GDELT intelligence topics to Redis with bootstrap hydration
Add standalone seed script that pre-populates all 6 Live Intelligence
topics (military, cyber, nuclear, sanctions, intelligence, maritime)
from the GDELT Doc API into Redis. Frontend consumes bootstrap data
lazily via the service layer, falling back to RPC if unavailable.
- scripts/seed-gdelt-intel.mjs: new seed script with per-topic 429 retry
- api/bootstrap.js: register gdeltIntel in FAST_KEYS
- api/health.js: register in BOOTSTRAP_KEYS + SEED_META + dataSize
- api/seed-health.js: register in SEED_DOMAINS
- scripts/_seed-utils.mjs: add topics to recordCount detection
- src/services/gdelt-intel.ts: lazy bootstrap consumption in service layer
* fix(seed): align staleness thresholds and strengthen GDELT validation
- seed-health intervalMin 30→60 so staleness (120min) matches health.js maxStaleMin
- validate requires ≥3/6 topics populated (not just military)
- recordCount sums articles across topics instead of reporting topic count
Gate LlmStatusIndicator behind isDesktopRuntime() so Vercel web
never fires a wasted 404 fetch to /api/llm-health (sidecar-only).
Add Kharg Island, Qom, Andisheh, and Ankara to the Iran events
geolocation map for accurate event placement.
Three issues caused intermittent DEGRADED health:
1. When ICAO API returned empty (timeout, challenge page), the relay
seed updated seed-meta but did not refresh the data key TTL. After
1h the data key expired, health saw EMPTY, reported CRIT.
Fix: call EXPIRE on the data key to extend TTL on empty response.
2. health.js dataSize() did not recognize the closedIcaos array field,
falling back to Object.keys count (always 2). Now properly counts
the closure array length.
3. 0 airport closures is the normal healthy state, but health treated
it as EMPTY_DATA (CRIT). Added EMPTY_DATA_OK_KEYS set so NOTAM
closures with 0 records reports OK when the key exists or seed-meta
is fresh.
Add missing seed-meta write for intlDelays in ais-relay, add untracked
SEED_META entries (intlDelays, faaDelays, theaterPosture) to health.js,
add 6 missing domains to seed-health.js, and return 503 when degraded.
ICAO API rejects %2C-encoded commas with 403. The manual seed script
passes literal commas in the locations query param, which works.
Match that behavior.
Railway deploys with rootDirectory=scripts/, so ../shared/ resolves to
/shared/ which doesn't exist. Move the canonical file to scripts/data/
and update all four consumers.
NOTAM closures data (aviation:notam:closures:v2) had no scheduled seeder,
causing health endpoint to report CRIT/EMPTY. The manual seed script
(seed-airport-delays.mjs) was never deployed as a cron, and the RPC
handler only populates on-demand with a 30min TTL.
Add a 30-minute seed loop to the AIS relay that fetches from ICAO API
and writes closures to Redis, matching the pattern of all other relay
seed loops. Also add seed-meta tracking in health.js (maxStaleMin: 90).
- Move GEOPOLITICAL_TAGS, TECH_TAGS, FINANCE_TAGS, and EXCLUDE_KEYWORDS
to shared/prediction-tags.json so seed, RPC handler, and client all
reference a single source of truth
- Remove open_interest proto field (always 0 for Polymarket, never
displayed in UI) and corresponding openInterest assignments
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Elie Habib <elie.habib@gmail.com>
Kalshi trading API returns 401 without authentication. Disable all
Kalshi fetches when KALSHI_API_KEY is not set, and pass it as a
Bearer token when present. Seed logs "disabled" instead of spamming
401 errors on every run.
* feat(predictions): add Kalshi as prediction market data source
* fix(predictions): address Kalshi integration review feedback
- Gate Kalshi fetch behind category check to avoid wasted calls on tech-scoped requests
- Replace fragile double-cast bootstrap typing with BootstrapMarket interface
- Fix zero-price falsy bug in seed script using Number.isFinite guard
- Align RPC market selection with seed script (highest-volume via single-pass loop)
- Raise Kalshi volume threshold to 5000 for signal quality parity
- Add missing .prediction-source badge CSS with per-source color variants
* fix(predictions): address P1/P2 review items for Kalshi integration
- Apply isExcluded() filter and volume threshold (5000) to live Kalshi
RPC path so cache-miss results match seed curation quality
- Include FINANCE_TAGS in seed allTags so 'markets' tag is fetched
- Align Kalshi title mapping (market.title || event.title) between
seed and RPC handler
- Remove silent geopolitical fallback for finance variant so missing
finance bootstrap falls through to RPC fetch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(predictions): prefer yes_sub_title for Kalshi multi-contract events
For multi-contract Kalshi events (e.g. papal election candidates),
market.title is the generic event question while yes_sub_title
identifies the specific contract. Use yes_sub_title when present
in both seed and RPC paths so titles are accurate and consistent.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(predictions): use general Kalshi trading API subdomain
Switch from api.elections.kalshi.com (elections-only) to
trading-api.kalshi.com so economy, crypto, and other non-election
markets are included in the finance variant.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Elie Habib <elie.habib@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(acled): add OAuth token manager with automatic refresh
ACLED access tokens expire every 24 hours, but WorldMonitor stores a
static ACLED_ACCESS_TOKEN with no refresh logic — causing all ACLED
API calls to fail after the first day.
This commit adds `acled-auth.ts`, an OAuth token manager that:
- Exchanges ACLED_EMAIL + ACLED_PASSWORD for an access token (24h)
and refresh token (14d) via the official ACLED OAuth endpoint
- Caches tokens in memory and auto-refreshes before expiry
- Falls back to static ACLED_ACCESS_TOKEN for backward compatibility
- Deduplicates concurrent refresh attempts
- Degrades gracefully when no credentials are configured
The only change to the existing `acled.ts` is replacing the synchronous
`process.env.ACLED_ACCESS_TOKEN` read with an async call to the new
`getAcledAccessToken()` helper.
Fixes#1283
Relates to #290
* fix: address review feedback on ACLED OAuth PR
- Use Redis (Upstash) as L2 token cache to survive Vercel Edge cold starts
(in-memory cache retained as fast-path L1)
- Add CHROME_UA User-Agent header on OAuth token exchange and refresh
- Update seed script to use OAuth flow via getAcledToken() helper
instead of raw process.env.ACLED_ACCESS_TOKEN
- Add security comment to .env.example about plaintext password trade-offs
- Sidecar ACLED_ACCESS_TOKEN case is a validation probe (tests user-provided
value, not process.env) — data fetching delegates to handler modules
* feat(sidecar): add ACLED_EMAIL/ACLED_PASSWORD to env allowlist and validation
- Add ACLED_EMAIL and ACLED_PASSWORD to ALLOWED_ENV_KEYS set
- Add ACLED_EMAIL validation case (store-only, verified with password)
- Add ACLED_PASSWORD validation case with OAuth token exchange via
acleddata.com/api/acled/user/login
- On successful login, store obtained OAuth token in ACLED_ACCESS_TOKEN
- Follows existing validation patterns (Cloudflare challenge handling,
auth failure detection, User-Agent header)
* fix: address remaining review feedback (duplicate OAuth, em dashes, emoji)
- Extract shared ACLED OAuth helper into scripts/shared/acled-oauth.mjs
- Remove ~55 lines of duplicate OAuth logic from seed-unrest-events.mjs,
now imports getAcledToken from the shared helper
- Replace em dashes with ASCII dashes in acled-auth.ts section comments
- Replace em dash with parentheses in sidecar validation message
- Remove emoji from .env.example security note
Addresses koala73's second review: MEDIUM (duplicate OAuth), LOW (em
dashes), LOW (emoji).
* fix: align sidecar OAuth endpoint, fix L1/L2 cache, cleanup artifacts
- Sidecar: switch from /api/acled/user/login (JSON) to /oauth/token
(URL-encoded) to match server/_shared/acled-auth.ts exactly
- acled-auth.ts: check L2 Redis when L1 is expired, not only when L1
is null (fixes stale L1 skipping fresher L2 from another isolate)
- acled-oauth.mjs: remove stray backslash on line 9
- seed-unrest-events.mjs: remove extra blank line at line 13
---------
Co-authored-by: Elie Habib <elie.habib@gmail.com>
Co-authored-by: RepairYourTech <30200484+RepairYourTech@users.noreply.github.com>
* fix(predictions): replace volume-only sort with composite scoring, add finance variant and region ranking
The prediction panel was surfacing irrelevant near-certain markets (1%/99% meme
markets like celebrity presidential bids) because the discrepancy filter was
inverted and sorting was by volume alone.
- Replace broken discrepancy filter with composite scoring (60% uncertainty +
40% log-scaled volume) in seed script
- Add meme candidate detection and sports/entertainment keyword exclusion
- Add finance variant with dedicated tags for economy/trade/rates topics
- Add region-aware soft ranking outside circuit breaker cache
- Add input validation (category max 50, query max 100) in RPC handler
- Skip events without markets instead of defaulting to yesPrice=50
- Per-bucket relaxation safety valve when <15 markets pass strict filters
* fix(predictions): apply region sort before truncation, add RPC fallback scoring, validate finance seed
- Keep 25 candidates from bootstrap/RPC, apply region sort, then slice to 15
(previously sliced to 15 first, making region boost ineffective for markets
ranked 16-25)
- Add client-side uncertainty scoring + near-certain filter (10-90%) for RPC
fallback path (previously fell back to Gamma's volume-only ordering)
- Include finance array in seed validation (previously only checked
geopolitical/tech, allowing broken finance data to ship silently)
* test(predictions): add 54 unit tests for scoring, filtering, and region tagging
Extract pure prediction scoring functions into shared module
(_prediction-scoring.mjs) for testability. Tests cover parseYesPrice,
isExcluded, isMemeCandidate, tagRegions, shouldInclude, scoreMarket,
filterAndScore, isExpired, plus regression tests for the meme market
surfacing bug that motivated this fix.
