mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
feat(forecast): add structured scenario pipeline and trace export (#1646)
* feat(forecast): add AI Forecasts prediction module (Pro-tier)
MiroFish-inspired prediction engine that generates structured forecasts
across 6 domains (conflict, market, supply chain, political, military,
infrastructure) using existing WorldMonitor data streams.
- Proto definitions for ForecastService with GetForecasts RPC
- Dedicated seed script (seed-forecasts.mjs) with 6 domain detectors,
cross-domain cascade resolver, prediction market calibration, and
trend detection via prior snapshot comparison
- Premium-gated RPC handler (PREMIUM_RPC_PATHS enforcement)
- Lazy-loaded ForecastPanel with domain filters, probability bars,
trend arrows, signal evidence, and cascade links
- Health monitoring integration (seed-meta freshness tracking)
- Refresh scheduler with API key guard
* test(forecast): add 47 unit tests for forecast detectors and utilities
Covers forecastId, normalize, resolveCascades, calibrateWithMarkets,
computeTrends, and smoke tests for all 6 domain detectors. Exports
testable functions from seed script with direct-run guard.
* fix(forecast): domain mismatch 'infra' vs 'infrastructure', add panel category
- Seed script used 'infra' but ForecastPanel filtered on 'infrastructure',
causing Infra tab to show zero results
- Added 'forecast' to intelligence category in PANEL_CATEGORY_MAP
* fix(forecast): move CSS to one-time injection, improve type safety
- P2: Move style block from setContent to one-time document.head injection
to prevent CSS accumulation on repeated renders
- P3: Replace +toFixed(3) with Math.round for readability in seed script
- P3: Use Forecast type instead of any[] in RPC handler filter
* fix(forecast): handle sebuf proto data shapes from Redis
Detectors now normalize CII scores from server-side proto format
(combinedScore, TREND_DIRECTION_RISING, region) to uniform shape.
Outage severity handles proto enum format (SEVERITY_LEVEL_HIGH).
Added confidence floor of 0.3 for single-source predictions.
Verified against live Redis: 2 predictions generated (Iran infra
shutdown, IL political instability).
* feat(forecast): unlock AI Forecasts on web, lock desktop only (trial)
- Remove forecast RPC from PREMIUM_RPC_PATHS (web access is free)
- Panel locked on desktop only (same as oref-sirens/telegram-intel)
- Remove API key guards from data-loader and refresh scheduler
- Web users get full access during trial period
* chore: regenerate proto types with make generate
Re-ran make generate after rebasing on main. Plugin v0.7.0 dropped
@ts-nocheck from output, added it back to all 50 generated files.
Fixed 4 type errors from proto codegen changes:
- MarketSource enum -> string union type
- TemporalAnomalyProto -> TemporalAnomaly rename
- webcam lastUpdated number -> string
* chore: add proto freshness check to pre-push hook
Runs make generate before push and compares checksums of generated files.
If proto types are stale, blocks push with instructions to regenerate.
Skips gracefully if buf CLI is not installed.
* fix(forecast): use chokepoints v4 key, include ciiContribution in unrest
- P1: Switch chokepoints input from stale v2 to active v4 Redis key,
matching bootstrap.js and cache-keys.ts
- P2: Add ciiContribution to unrest component fallback chain in
normalizeCiiEntry so political detector reads the correct sebuf field
* feat(forecast): Phase 2 LLM scenario enrichment + confidence model
MiroFish-inspired enhancements:
- LLM scenario narratives via Groq/OpenRouter (narrative-only, no numeric
adjustment). Evidence-grounded prompts with mandatory signal citation
and few-shot examples from MiroFish's SECTION_SYSTEM_PROMPT_TEMPLATE.
- Top-4 predictions batched into single LLM call for cost efficiency.
- News context from newsInsights attached to all predictions for LLM
prompt grounding (NOT in signals, cannot affect confidence).
- Deterministic confidence model: source diversity via SIGNAL_TO_SOURCE
mapping (deduplicates cii+cii_delta, theater+indicators) + calibration
agreement from prediction market drift. Floor 0.2, ceiling 1.0.
- Output validation: rejects scenarios without signal references.
- Truncated JSON repair for small model output.
- Structured JSON logging for LLM calls.
- Redis cache for LLM scenarios (1h TTL).
- 23 new tests (70 total), all passing.
- Live-tested: OpenRouter gemini-2.5-flash produces evidence-grounded
scenario narratives from real WorldMonitor data.
* feat(forecast): Phase 3 multi-perspective scenarios, projections, data-driven cascades
MiroFish-inspired enhancements:
- Multi-perspective LLM analysis: top-2 predictions get strategic,
regional, and contrarian viewpoints via combined LLM call
- Probability projections: domain-specific decay curves (h24/d7/d30)
anchored to timeHorizon so probability equals projections[timeHorizon]
- Data-driven cascade rules: moved from hardcoded array to JSON config
(scripts/data/cascade-rules.json) with schema validation, named
predicate evaluators, unknown key rejection, and fallback to defaults
- 4 new cascade paths: infrastructure->supply_chain, infrastructure->market
(both requiresSeverity:total), conflict->political, political->market
- Proto: added Perspectives and Projections messages to Forecast
- ForecastPanel: renders projections row and conditional perspectives toggle
- 89 tests (19 new), all passing
- Live-tested: OpenRouter produces perspectives from real data
* feat(forecast): Phase 4 data utilization + entity graph
Fixes data gaps that prevented 4 of 6 detectors from firing:
- Input normalizers: chokepoint v4 shape + GPS hexes-to-zones mapping
- Chokepoint warm-ping (production-only, requires WM_API_BASE_URL)
- Lowered CII conflict threshold from 70 to 60, gated on level=high|critical
4 new standalone detectors:
- UCDP conflict zones (10+ events per country)
- Cyber threat concentration (5+ threats per country)
- GPS jamming in maritime shipping zones (5 regions)
- Prediction markets as signals (60-90% probability markets)
Entity-relationship graph (file-based, 38 nodes):
- Countries, theaters, commodities, chokepoints, alliances
- Alias table resolves both ISO codes and display names
- Graph cascade discovery links predictions across entities
Result: 51 predictions (up from 1-2), spanning conflict, infrastructure,
and supply chain domains. 112 tests, all passing.
* fix(forecast): redis cache format, signal source mapping, type safety
Fresh-eyes audit fixes:
- BUG: redisSet used wrong Upstash API format (POST body with {value,ex}
instead of command array ['SET',key,value,'EX',ttl]). LLM cache writes
were silently failing, causing fresh LLM calls every run.
- BUG: prediction_market signal type missing from SIGNAL_TO_SOURCE,
inflating confidence for market-derived predictions.
- CLEANUP: Remove unnecessary (f as any) casts in ForecastPanel since
generated Forecast type already has projections/perspectives fields.
- CLEANUP: Bump health maxStaleMin from 60 to 90 to avoid false STALE
alerts when LLM calls add latency to seed runs.
* feat(forecast): headline-entity matching with news corroboration signals
Uses entity graph aliases to match headlines to predictions by
country/theater (excludes commodity/infrastructure nodes to prevent
false positives). Predictions with matching headlines get a
news_corroboration signal visible in the panel.
Also fixes buildUserPrompt to merge unique headlines from ALL
predictions in the LLM batch (was only reading preds[0].newsContext).
Live-tested: 13 of 51 predictions now have corroborating headlines
(Iran, Israel, Syria, Ukraine, etc). 116 tests, all passing.
* feat(forecast): add country-codes.json for headline-entity matching
56 countries with ISO codes, full names, and scoring keywords (extracted
from src/config/countries.ts + UCDP-relevant additions). Used by
attachNewsContext for richer headline matching via getSearchTermsForRegion
which combines country-codes + entity graph + keyword aliases.
14/57 predictions now have news corroboration (limited by headline
coverage, not matching quality: only 8 headlines currently available).
* feat(forecast): read 300 headlines from news digest instead of 8
Read news:digest:v1:full:en (300 headlines across 16 categories) instead
of just news:insights:v1 topStories (8 headlines). Fallback to topStories
if digest is unavailable.
Result: news corroboration jumped from 25% to 64% (38/59 predictions).
* fix(forecast): handle parenthetical country names in headline matching
Strip suffixes like '(Zaire)', '(Burma)', '(Soviet Union)' from UCDP
region names before matching against country-codes.json. Also use
includes() for reverse name lookup to catch partial matches.
Corroboration: 64% -> 69% (41/59). Remaining 18 unmatched are countries
with no current English-language news coverage.
* fix(forecast): cache validated LLM output, add digest test, log cache errors
Fresh-eyes audit fixes:
- Combined LLM cache now stores only validated items (was caching raw
unvalidated output, serving potentially invalid scenarios on cache hit)
- redisSet logs warnings on failure (was silently swallowing all errors)
- Added digest-based test for attachNewsContext (primary path was untested)
- Fixed test arity: attachNewsContext(preds, news, digest) with 3 params
* fix(forecast): remove dead confidenceFromSources, reduce warm-ping timeout
- P2: Remove confidenceFromSources (dead code, computeConfidence overwrites
all initial confidence values). Inline the formula in original detectors.
- P3: Reduce warm-ping timeout from 30s to 15s (non-critical step)
- P3: Add trial status comment on forecast panel config
* fix(forecast): resolve ISO codes to country names, fix market detector, safe pre-push
P1 fixes from code review:
- CII ISO codes (IL, IR) now resolved to full country names (Israel, Iran)
via country-codes.json. Prevents substring false positives (IL matching
Chile) in event correlation. Uses word-boundary regex for matching.
- Market detector CII-to-theater mapping now uses entity graph traversal
instead of broken theater-name substring matching. Iran correctly maps
to Middle East theater via graph links.
- Pre-push hook no longer runs destructive git checkout on proto freshness
failure. Reports mismatch and exits without modifying worktree.
* feat(forecast): add structured scenario pipeline and trace export
* fix(forecast): hydrate bootstrap and trim generated drift
* fix(forecast): keep required supply-chain contract updates
* fix(ci): add forecasts to cache-keys registry and regenerate proto
Add forecasts entry to BOOTSTRAP_CACHE_KEYS and BOOTSTRAP_TIERS in
cache-keys.ts to match api/bootstrap.js. Regenerate SupplyChain proto
to fix duplicate TransitDayCount and add riskSummary/riskReportAction.
This commit is contained in:
Elie Habib
12
.env.example
12
.env.example
@@ -106,6 +106,18 @@ UCDP_ACCESS_TOKEN=
|
||||
# Cloudflare Radar API (requires free Cloudflare account with Radar access)
|
||||
CLOUDFLARE_API_TOKEN=
|
||||
|
||||
# Cloudflare R2 account id for seed scripts that read or write R2 objects
|
||||
CLOUDFLARE_R2_ACCOUNT_ID=
|
||||
|
||||
# Cloudflare R2 trace storage for forecast seed review artifacts
|
||||
# Create R2 access keys in Cloudflare and target the bucket you want to use for forecast traces.
|
||||
CLOUDFLARE_R2_BUCKET=
|
||||
CLOUDFLARE_R2_TRACE_BUCKET=
|
||||
CLOUDFLARE_R2_ACCESS_KEY_ID=
|
||||
CLOUDFLARE_R2_SECRET_ACCESS_KEY=
|
||||
CLOUDFLARE_R2_REGION=auto
|
||||
CLOUDFLARE_R2_TRACE_PREFIX=seed-data/forecast-traces
|
||||
|
||||
|
||||
# ------ Satellite Fire Detection (Vercel) ------
|
||||
|
||||
|
||||
@@ -36,6 +36,9 @@ node --test tests/mdx-lint.test.mjs || exit 1
|
||||
|
||||
echo "Running proto freshness check..."
|
||||
if git diff --name-only origin/main -- proto/ src/generated/ docs/api/ Makefile | grep -q .; then
|
||||
if command -v buf >/dev/null 2>&1 || [ -x "$HOME/go/bin/buf" ]; then
|
||||
export PATH="$HOME/go/bin:$PATH"
|
||||
fi
|
||||
if command -v buf &>/dev/null && command -v protoc-gen-ts-client &>/dev/null; then
|
||||
make generate
|
||||
if ! git diff --exit-code src/generated/ docs/api/; then
|
||||
|
||||
3
api/bootstrap.js
vendored
3
api/bootstrap.js
vendored
@@ -45,6 +45,7 @@ const BOOTSTRAP_CACHE_KEYS = {
|
||||
techEvents: 'research:tech-events-bootstrap:v1',
|
||||
gdeltIntel: 'intelligence:gdelt-intel:v1',
|
||||
correlationCards: 'correlation:cards-bootstrap:v1',
|
||||
forecasts: 'forecast:predictions:v2',
|
||||
securityAdvisories: 'intelligence:advisories-bootstrap:v1',
|
||||
};
|
||||
|
||||
@@ -61,7 +62,7 @@ const FAST_KEYS = new Set([
|
||||
'earthquakes', 'outages', 'serviceStatuses', 'macroSignals', 'chokepoints', 'chokepointTransits',
|
||||
'marketQuotes', 'commodityQuotes', 'positiveGeoEvents', 'riskScores', 'flightDelays','insights', 'predictions',
|
||||
'iranEvents', 'temporalAnomalies', 'weatherAlerts', 'spending', 'theaterPosture', 'gdeltIntel',
|
||||
'correlationCards',
|
||||
'correlationCards', 'forecasts',
|
||||
]);
|
||||
|
||||
const TIER_CACHE = {
|
||||
|
||||
@@ -30,7 +30,7 @@ const BOOTSTRAP_KEYS = {
|
||||
techEvents: 'research:tech-events-bootstrap:v1',
|
||||
gdeltIntel: 'intelligence:gdelt-intel:v1',
|
||||
correlationCards: 'correlation:cards-bootstrap:v1',
|
||||
forecasts: 'forecast:predictions:v1',
|
||||
forecasts: 'forecast:predictions:v2',
|
||||
securityAdvisories: 'intelligence:advisories-bootstrap:v1',
|
||||
};
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -103,6 +103,8 @@ components:
|
||||
type: string
|
||||
scenario:
|
||||
type: string
|
||||
feedSummary:
|
||||
type: string
|
||||
probability:
|
||||
type: number
|
||||
format: double
|
||||
@@ -138,6 +140,8 @@ components:
|
||||
$ref: '#/components/schemas/Perspectives'
|
||||
projections:
|
||||
$ref: '#/components/schemas/Projections'
|
||||
caseFile:
|
||||
$ref: '#/components/schemas/ForecastCase'
|
||||
ForecastSignal:
|
||||
type: object
|
||||
properties:
|
||||
@@ -192,3 +196,134 @@ components:
|
||||
d30:
|
||||
type: number
|
||||
format: double
|
||||
ForecastCase:
|
||||
type: object
|
||||
properties:
|
||||
supportingEvidence:
|
||||
type: array
|
||||
items:
|
||||
$ref: '#/components/schemas/ForecastCaseEvidence'
|
||||
counterEvidence:
|
||||
type: array
|
||||
items:
|
||||
$ref: '#/components/schemas/ForecastCaseEvidence'
|
||||
triggers:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
actorLenses:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
baseCase:
|
||||
type: string
|
||||
escalatoryCase:
|
||||
type: string
|
||||
contrarianCase:
|
||||
type: string
|
||||
changeSummary:
|
||||
type: string
|
||||
changeItems:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
actors:
|
||||
type: array
|
||||
items:
|
||||
$ref: '#/components/schemas/ForecastActor'
|
||||
worldState:
|
||||
$ref: '#/components/schemas/ForecastWorldState'
|
||||
branches:
|
||||
type: array
|
||||
items:
|
||||
$ref: '#/components/schemas/ForecastBranch'
|
||||
ForecastCaseEvidence:
|
||||
type: object
|
||||
properties:
|
||||
type:
|
||||
type: string
|
||||
summary:
|
||||
type: string
|
||||
weight:
|
||||
type: number
|
||||
format: double
|
||||
ForecastActor:
|
||||
type: object
|
||||
properties:
|
||||
id:
|
||||
type: string
|
||||
name:
|
||||
type: string
|
||||
category:
|
||||
type: string
|
||||
role:
|
||||
type: string
|
||||
objectives:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
constraints:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
likelyActions:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
influenceScore:
|
||||
type: number
|
||||
format: double
|
||||
ForecastWorldState:
|
||||
type: object
|
||||
properties:
|
||||
summary:
|
||||
type: string
|
||||
activePressures:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
stabilizers:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
keyUnknowns:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
ForecastBranch:
|
||||
type: object
|
||||
properties:
|
||||
kind:
|
||||
type: string
|
||||
title:
|
||||
type: string
|
||||
summary:
|
||||
type: string
|
||||
outcome:
|
||||
type: string
|
||||
projectedProbability:
|
||||
type: number
|
||||
format: double
|
||||
rounds:
|
||||
type: array
|
||||
items:
|
||||
$ref: '#/components/schemas/ForecastBranchRound'
|
||||
ForecastBranchRound:
|
||||
type: object
|
||||
properties:
|
||||
round:
|
||||
type: integer
|
||||
format: int32
|
||||
focus:
|
||||
type: string
|
||||
developments:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
actorMoves:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
probabilityShift:
|
||||
type: number
|
||||
format: double
|
||||
|
||||
1591
package-lock.json
generated
1591
package-lock.json
generated
File diff suppressed because it is too large
Load Diff
@@ -86,6 +86,7 @@
|
||||
"vite-plugin-pwa": "^1.2.0"
|
||||
},
|
||||
"dependencies": {
|
||||
"@aws-sdk/client-s3": "^3.1009.0",
|
||||
"@deck.gl/aggregation-layers": "^9.2.6",
|
||||
"@deck.gl/core": "^9.2.6",
|
||||
"@deck.gl/geo-layers": "^9.2.6",
|
||||
|
||||
@@ -35,12 +35,69 @@ message Projections {
|
||||
double d30 = 3;
|
||||
}
|
||||
|
||||
message ForecastCaseEvidence {
|
||||
string type = 1;
|
||||
string summary = 2;
|
||||
double weight = 3;
|
||||
}
|
||||
|
||||
message ForecastActor {
|
||||
string id = 1;
|
||||
string name = 2;
|
||||
string category = 3;
|
||||
string role = 4;
|
||||
repeated string objectives = 5;
|
||||
repeated string constraints = 6;
|
||||
repeated string likely_actions = 7;
|
||||
double influence_score = 8;
|
||||
}
|
||||
|
||||
message ForecastWorldState {
|
||||
string summary = 1;
|
||||
repeated string active_pressures = 2;
|
||||
repeated string stabilizers = 3;
|
||||
repeated string key_unknowns = 4;
|
||||
}
|
||||
|
||||
message ForecastBranchRound {
|
||||
int32 round = 1;
|
||||
string focus = 2;
|
||||
repeated string developments = 3;
|
||||
repeated string actor_moves = 4;
|
||||
double probability_shift = 5;
|
||||
}
|
||||
|
||||
message ForecastBranch {
|
||||
string kind = 1;
|
||||
string title = 2;
|
||||
string summary = 3;
|
||||
string outcome = 4;
|
||||
double projected_probability = 5;
|
||||
repeated ForecastBranchRound rounds = 6;
|
||||
}
|
||||
|
||||
message ForecastCase {
|
||||
repeated ForecastCaseEvidence supporting_evidence = 1;
|
||||
repeated ForecastCaseEvidence counter_evidence = 2;
|
||||
repeated string triggers = 3;
|
||||
repeated string actor_lenses = 4;
|
||||
string base_case = 5;
|
||||
string escalatory_case = 6;
|
||||
string contrarian_case = 7;
|
||||
string change_summary = 8;
|
||||
repeated string change_items = 9;
|
||||
repeated ForecastActor actors = 10;
|
||||
ForecastWorldState world_state = 11;
|
||||
repeated ForecastBranch branches = 12;
|
||||
}
|
||||
|
||||
message Forecast {
|
||||
string id = 1;
|
||||
string domain = 2;
|
||||
string region = 3;
|
||||
string title = 4;
|
||||
string scenario = 5;
|
||||
string feed_summary = 19;
|
||||
double probability = 6;
|
||||
double confidence = 7;
|
||||
string time_horizon = 8;
|
||||
@@ -53,4 +110,5 @@ message Forecast {
|
||||
int64 updated_at = 15 [(sebuf.http.int64_encoding) = INT64_ENCODING_NUMBER];
|
||||
Perspectives perspectives = 16;
|
||||
Projections projections = 17;
|
||||
ForecastCase case_file = 18;
|
||||
}
|
||||
|
||||
122
scripts/_r2-storage.mjs
Normal file
122
scripts/_r2-storage.mjs
Normal file
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
|
||||
|
||||
function getEnvValue(env, keys) {
|
||||
for (const key of keys) {
|
||||
if (env[key]) return env[key];
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function parseBoolean(value, fallback) {
|
||||
if (value == null || value === '') return fallback;
|
||||
const normalized = String(value).trim().toLowerCase();
|
||||
if (['1', 'true', 'yes', 'on'].includes(normalized)) return true;
|
||||
if (['0', 'false', 'no', 'off'].includes(normalized)) return false;
|
||||
return fallback;
|
||||
}
|
||||
|
||||
function resolveR2StorageConfig(env = process.env, options = {}) {
|
||||
const accountId = getEnvValue(env, ['CLOUDFLARE_R2_ACCOUNT_ID']);
|
||||
const bucket = getEnvValue(env, [options.bucketEnv || 'CLOUDFLARE_R2_TRACE_BUCKET', 'CLOUDFLARE_R2_BUCKET']);
|
||||
const accessKeyId = getEnvValue(env, ['CLOUDFLARE_R2_ACCESS_KEY_ID']);
|
||||
const secretAccessKey = getEnvValue(env, ['CLOUDFLARE_R2_SECRET_ACCESS_KEY']);
|
||||
const apiToken = getEnvValue(env, ['CLOUDFLARE_R2_TOKEN', 'CLOUDFLARE_API_TOKEN']);
|
||||
const endpoint = getEnvValue(env, ['CLOUDFLARE_R2_ENDPOINT']) || (accountId ? `https://${accountId}.r2.cloudflarestorage.com` : '');
|
||||
const apiBaseUrl = getEnvValue(env, ['CLOUDFLARE_API_BASE_URL']) || 'https://api.cloudflare.com/client/v4';
|
||||
const region = getEnvValue(env, ['CLOUDFLARE_R2_REGION']) || 'auto';
|
||||
const basePrefix = (getEnvValue(env, [options.prefixEnv || 'CLOUDFLARE_R2_TRACE_PREFIX']) || 'seed-data/forecast-traces')
|
||||
.replace(/^\/+|\/+$/g, '');
|
||||
const forcePathStyle = parseBoolean(getEnvValue(env, ['CLOUDFLARE_R2_FORCE_PATH_STYLE']), true);
|
||||
|
||||
if (!bucket || !accountId) return null;
|
||||
|
||||
if (endpoint && accessKeyId && secretAccessKey) {
|
||||
return {
|
||||
mode: 's3',
|
||||
accountId,
|
||||
bucket,
|
||||
endpoint,
|
||||
region,
|
||||
credentials: { accessKeyId, secretAccessKey },
|
||||
forcePathStyle,
|
||||
basePrefix,
|
||||
};
|
||||
}
|
||||
|
||||
if (apiToken) {
|
||||
return {
|
||||
mode: 'api',
|
||||
accountId,
|
||||
bucket,
|
||||
apiToken,
|
||||
apiBaseUrl,
|
||||
basePrefix,
|
||||
};
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
const CLIENT_CACHE = new Map();
|
||||
|
||||
function getR2StorageClient(config) {
|
||||
const cacheKey = JSON.stringify({
|
||||
endpoint: config.endpoint,
|
||||
region: config.region,
|
||||
bucket: config.bucket,
|
||||
accessKeyId: config.credentials.accessKeyId,
|
||||
forcePathStyle: config.forcePathStyle,
|
||||
});
|
||||
let client = CLIENT_CACHE.get(cacheKey);
|
||||
if (!client) {
|
||||
client = new S3Client({
|
||||
endpoint: config.endpoint,
|
||||
region: config.region,
|
||||
credentials: config.credentials,
|
||||
forcePathStyle: config.forcePathStyle,
|
||||
});
|
||||
CLIENT_CACHE.set(cacheKey, client);
|
||||
}
|
||||
return client;
|
||||
}
|
||||
|
||||
async function putR2JsonObject(config, key, payload, metadata = {}) {
|
||||
const body = `${JSON.stringify(payload, null, 2)}\n`;
|
||||
|
||||
if (config.mode === 'api') {
|
||||
const encodedKey = key.split('/').map(part => encodeURIComponent(part)).join('/');
|
||||
const resp = await fetch(`${config.apiBaseUrl}/accounts/${config.accountId}/r2/buckets/${config.bucket}/objects/${encodedKey}`, {
|
||||
method: 'PUT',
|
||||
headers: {
|
||||
Authorization: `Bearer ${config.apiToken}`,
|
||||
'Content-Type': 'application/json; charset=utf-8',
|
||||
},
|
||||
body,
|
||||
signal: AbortSignal.timeout(30_000),
|
||||
});
|
||||
if (!resp.ok) {
|
||||
const text = await resp.text().catch(() => '');
|
||||
throw new Error(`Cloudflare R2 API upload failed: HTTP ${resp.status} — ${text.slice(0, 200)}`);
|
||||
}
|
||||
return { bucket: config.bucket, key, bytes: Buffer.byteLength(body, 'utf8') };
|
||||
}
|
||||
|
||||
const client = getR2StorageClient(config);
|
||||
await client.send(new PutObjectCommand({
|
||||
Bucket: config.bucket,
|
||||
Key: key,
|
||||
Body: body,
|
||||
ContentType: 'application/json; charset=utf-8',
|
||||
CacheControl: 'no-store',
|
||||
Metadata: metadata,
|
||||
}));
|
||||
return { bucket: config.bucket, key, bytes: Buffer.byteLength(body, 'utf8') };
|
||||
}
|
||||
|
||||
export {
|
||||
resolveR2StorageConfig,
|
||||
getR2StorageClient,
|
||||
putR2JsonObject,
|
||||
};
|
||||
@@ -266,7 +266,7 @@ export function parseYahooChart(data, symbol) {
|
||||
}
|
||||
|
||||
export async function runSeed(domain, resource, canonicalKey, fetchFn, opts = {}) {
|
||||
const { validateFn, ttlSeconds, lockTtlMs = 120_000, extraKeys } = opts;
|
||||
const { validateFn, ttlSeconds, lockTtlMs = 120_000, extraKeys, afterPublish } = opts;
|
||||
const runId = `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
|
||||
const startMs = Date.now();
|
||||
|
||||
@@ -318,6 +318,7 @@ export async function runSeed(domain, resource, canonicalKey, fetchFn, opts = {}
|
||||
? (typeof opts.recordCount === 'function' ? opts.recordCount(data) : opts.recordCount)
|
||||
: Array.isArray(data) ? data.length
|
||||
: (topicArticleCount
|
||||
?? data?.predictions?.length
|
||||
?? data?.events?.length ?? data?.earthquakes?.length ?? data?.outages?.length
|
||||
?? data?.fireDetections?.length ?? data?.anomalies?.length ?? data?.threats?.length
|
||||
?? data?.quotes?.length ?? data?.stablecoins?.length
|
||||
@@ -330,6 +331,10 @@ export async function runSeed(domain, resource, canonicalKey, fetchFn, opts = {}
|
||||
}
|
||||
}
|
||||
|
||||
if (afterPublish) {
|
||||
await afterPublish(data, { canonicalKey, ttlSeconds, recordCount, runId });
|
||||
}
|
||||
|
||||
const meta = await writeFreshnessMetadata(domain, resource, recordCount, opts.sourceVersion);
|
||||
|
||||
const durationMs = Date.now() - startMs;
|
||||
|
||||
89
scripts/data/forecast-evaluation-benchmark.json
Normal file
89
scripts/data/forecast-evaluation-benchmark.json
Normal file
@@ -0,0 +1,89 @@
|
||||
[
|
||||
{
|
||||
"name": "well_grounded_conflict",
|
||||
"forecast": {
|
||||
"domain": "conflict",
|
||||
"region": "Iran",
|
||||
"title": "Escalation risk: Iran",
|
||||
"probability": 0.71,
|
||||
"confidence": 0.62,
|
||||
"timeHorizon": "7d",
|
||||
"trend": "rising",
|
||||
"signals": [
|
||||
{ "type": "cii", "value": "Iran CII 87 (critical)", "weight": 0.4 },
|
||||
{ "type": "ucdp", "value": "3 UCDP conflict events", "weight": 0.3 },
|
||||
{ "type": "theater", "value": "Middle East theater posture elevated", "weight": 0.2 }
|
||||
],
|
||||
"newsContext": [
|
||||
"Iran military drills intensify after border incident",
|
||||
"Regional officials warn of retaliation risk"
|
||||
],
|
||||
"calibration": {
|
||||
"marketTitle": "Will Iran conflict escalate before July?",
|
||||
"marketPrice": 0.58,
|
||||
"drift": 0.04,
|
||||
"source": "polymarket"
|
||||
},
|
||||
"cascades": [
|
||||
{ "domain": "market", "effect": "commodity price shock", "probability": 0.41 }
|
||||
]
|
||||
},
|
||||
"thresholds": {
|
||||
"overallMin": 0.7,
|
||||
"groundingMin": 0.6
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "well_grounded_supply_chain",
|
||||
"forecast": {
|
||||
"domain": "supply_chain",
|
||||
"region": "Red Sea",
|
||||
"title": "Shipping disruption: Red Sea",
|
||||
"probability": 0.64,
|
||||
"confidence": 0.57,
|
||||
"timeHorizon": "7d",
|
||||
"trend": "rising",
|
||||
"signals": [
|
||||
{ "type": "chokepoint", "value": "Red Sea disruption detected", "weight": 0.5 },
|
||||
{ "type": "gps_jamming", "value": "GPS interference near Red Sea", "weight": 0.2 }
|
||||
],
|
||||
"newsContext": [
|
||||
"Red Sea shipping disruption worsens after new attacks",
|
||||
"Freight rates react to Red Sea rerouting"
|
||||
],
|
||||
"calibration": {
|
||||
"marketTitle": "Will oil close above $90?",
|
||||
"marketPrice": 0.62,
|
||||
"drift": 0.03,
|
||||
"source": "polymarket"
|
||||
},
|
||||
"cascades": [
|
||||
{ "domain": "market", "effect": "supply shortage pricing", "probability": 0.38 }
|
||||
]
|
||||
},
|
||||
"thresholds": {
|
||||
"overallMin": 0.66,
|
||||
"groundingMin": 0.6
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "thin_generic_market",
|
||||
"forecast": {
|
||||
"domain": "market",
|
||||
"region": "Europe",
|
||||
"title": "Energy stress: Europe",
|
||||
"probability": 0.69,
|
||||
"confidence": 0.58,
|
||||
"timeHorizon": "30d",
|
||||
"trend": "stable",
|
||||
"signals": [
|
||||
{ "type": "prediction_market", "value": "Broad market stress chatter", "weight": 0.2 }
|
||||
],
|
||||
"newsContext": [],
|
||||
"cascades": []
|
||||
},
|
||||
"thresholds": {
|
||||
"overallMax": 0.55
|
||||
}
|
||||
}
|
||||
]
|
||||
176
scripts/data/forecast-historical-benchmark.json
Normal file
176
scripts/data/forecast-historical-benchmark.json
Normal file
@@ -0,0 +1,176 @@
|
||||
[
|
||||
{
|
||||
"name": "red_sea_shipping_disruption_2024_01_15",
|
||||
"eventDate": "2024-01-15",
|
||||
"description": "Red Sea disruption risk hardens after rerouting and interference signals broaden.",
|
||||
"priorForecast": {
|
||||
"domain": "supply_chain",
|
||||
"region": "Red Sea",
|
||||
"title": "Shipping disruption: Red Sea",
|
||||
"probability": 0.52,
|
||||
"confidence": 0.54,
|
||||
"timeHorizon": "7d",
|
||||
"signals": [
|
||||
{ "type": "chokepoint", "value": "Red Sea disruption detected", "weight": 0.5 }
|
||||
],
|
||||
"newsContext": [
|
||||
"Shipping firms monitor Red Sea route risk"
|
||||
],
|
||||
"calibration": {
|
||||
"marketTitle": "Will oil close above $90?",
|
||||
"marketPrice": 0.54,
|
||||
"drift": 0.02,
|
||||
"source": "polymarket"
|
||||
}
|
||||
},
|
||||
"forecast": {
|
||||
"domain": "supply_chain",
|
||||
"region": "Red Sea",
|
||||
"title": "Shipping disruption: Red Sea",
|
||||
"probability": 0.68,
|
||||
"confidence": 0.59,
|
||||
"timeHorizon": "7d",
|
||||
"signals": [
|
||||
{ "type": "chokepoint", "value": "Red Sea disruption detected", "weight": 0.5 },
|
||||
{ "type": "gps_jamming", "value": "GPS interference near Red Sea", "weight": 0.2 }
|
||||
],
|
||||
"newsContext": [
|
||||
"Shipping firms monitor Red Sea route risk",
|
||||
"Freight rates react to Red Sea rerouting"
|
||||
],
|
||||
"calibration": {
|
||||
"marketTitle": "Will oil close above $90?",
|
||||
"marketPrice": 0.67,
|
||||
"drift": 0.01,
|
||||
"source": "polymarket"
|
||||
},
|
||||
"cascades": [
|
||||
{ "domain": "market", "effect": "supply shortage pricing", "probability": 0.38 }
|
||||
]
|
||||
},
|
||||
"thresholds": {
|
||||
"overallMin": 0.72,
|
||||
"groundingMin": 0.65,
|
||||
"trend": "rising",
|
||||
"changeSummaryIncludes": ["rose from 52% to 68%"],
|
||||
"changeItemsInclude": [
|
||||
"New signal: GPS interference near Red Sea",
|
||||
"New reporting: Freight rates react to Red Sea rerouting",
|
||||
"Market moved from 54% to 67%"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "iran_exchange_2024_04_14",
|
||||
"eventDate": "2024-04-14",
|
||||
"description": "Iran escalation risk jumps as conflict-event and theater signals stack on top of already-high instability.",
|
||||
"priorForecast": {
|
||||
"domain": "conflict",
|
||||
"region": "Iran",
|
||||
"title": "Escalation risk: Iran",
|
||||
"probability": 0.46,
|
||||
"confidence": 0.55,
|
||||
"timeHorizon": "7d",
|
||||
"signals": [
|
||||
{ "type": "cii", "value": "Iran CII 79 (high)", "weight": 0.4 }
|
||||
],
|
||||
"newsContext": [
|
||||
"Iran military drills intensify after border incident"
|
||||
],
|
||||
"calibration": {
|
||||
"marketTitle": "Will Iran conflict escalate before July?",
|
||||
"marketPrice": 0.45,
|
||||
"drift": 0.03,
|
||||
"source": "polymarket"
|
||||
}
|
||||
},
|
||||
"forecast": {
|
||||
"domain": "conflict",
|
||||
"region": "Iran",
|
||||
"title": "Escalation risk: Iran",
|
||||
"probability": 0.74,
|
||||
"confidence": 0.64,
|
||||
"timeHorizon": "7d",
|
||||
"signals": [
|
||||
{ "type": "cii", "value": "Iran CII 79 (high)", "weight": 0.4 },
|
||||
{ "type": "ucdp", "value": "3 UCDP conflict events", "weight": 0.3 },
|
||||
{ "type": "theater", "value": "Middle East theater posture elevated", "weight": 0.2 }
|
||||
],
|
||||
"newsContext": [
|
||||
"Iran military drills intensify after border incident",
|
||||
"Regional officials warn of retaliation risk"
|
||||
],
|
||||
"calibration": {
|
||||
"marketTitle": "Will Iran conflict escalate before July?",
|
||||
"marketPrice": 0.71,
|
||||
"drift": 0.03,
|
||||
"source": "polymarket"
|
||||
},
|
||||
"cascades": [
|
||||
{ "domain": "market", "effect": "commodity price shock", "probability": 0.41 }
|
||||
]
|
||||
},
|
||||
"thresholds": {
|
||||
"overallMin": 0.78,
|
||||
"groundingMin": 0.65,
|
||||
"trend": "rising",
|
||||
"changeSummaryIncludes": ["rose from 46% to 74%"],
|
||||
"changeItemsInclude": [
|
||||
"New signal: 3 UCDP conflict events",
|
||||
"New reporting: Regional officials warn of retaliation risk",
|
||||
"Market moved from 45% to 71%"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "europe_energy_stress_eases_2025_02_01",
|
||||
"eventDate": "2025-02-01",
|
||||
"description": "A softer market path and thinner corroboration pull a European energy stress forecast down.",
|
||||
"priorForecast": {
|
||||
"domain": "market",
|
||||
"region": "Europe",
|
||||
"title": "Energy stress: Europe",
|
||||
"probability": 0.64,
|
||||
"confidence": 0.58,
|
||||
"timeHorizon": "30d",
|
||||
"signals": [
|
||||
{ "type": "prediction_market", "value": "EU gas price stress remains elevated", "weight": 0.25 }
|
||||
],
|
||||
"newsContext": [
|
||||
"European gas storage draw accelerates"
|
||||
],
|
||||
"calibration": {
|
||||
"marketTitle": "Will EU gas prices spike this month?",
|
||||
"marketPrice": 0.59,
|
||||
"drift": 0.02,
|
||||
"source": "polymarket"
|
||||
}
|
||||
},
|
||||
"forecast": {
|
||||
"domain": "market",
|
||||
"region": "Europe",
|
||||
"title": "Energy stress: Europe",
|
||||
"probability": 0.49,
|
||||
"confidence": 0.54,
|
||||
"timeHorizon": "30d",
|
||||
"signals": [
|
||||
{ "type": "prediction_market", "value": "EU gas price stress remains elevated", "weight": 0.25 }
|
||||
],
|
||||
"newsContext": [],
|
||||
"calibration": {
|
||||
"marketTitle": "Will EU gas prices spike this month?",
|
||||
"marketPrice": 0.44,
|
||||
"drift": 0.05,
|
||||
"source": "polymarket"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"overallMax": 0.58,
|
||||
"trend": "falling",
|
||||
"changeSummaryIncludes": ["fell from 64% to 49%"],
|
||||
"changeItemsInclude": [
|
||||
"Market moved from 59% to 44%"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
123
scripts/evaluate-forecast-benchmark.mjs
Normal file
123
scripts/evaluate-forecast-benchmark.mjs
Normal file
@@ -0,0 +1,123 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
import {
|
||||
makePrediction,
|
||||
computeTrends,
|
||||
buildForecastCase,
|
||||
buildPriorForecastSnapshot,
|
||||
annotateForecastChanges,
|
||||
scoreForecastReadiness,
|
||||
computeAnalysisPriority,
|
||||
} from './seed-forecasts.mjs';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const benchmarkPaths = [
|
||||
join(__dirname, 'data', 'forecast-evaluation-benchmark.json'),
|
||||
join(__dirname, 'data', 'forecast-historical-benchmark.json'),
|
||||
];
|
||||
|
||||
function materializeForecast(input) {
|
||||
const pred = makePrediction(
|
||||
input.domain,
|
||||
input.region,
|
||||
input.title,
|
||||
input.probability,
|
||||
input.confidence,
|
||||
input.timeHorizon,
|
||||
input.signals || [],
|
||||
);
|
||||
pred.trend = input.trend || pred.trend;
|
||||
pred.newsContext = input.newsContext || [];
|
||||
pred.calibration = input.calibration || null;
|
||||
pred.cascades = input.cascades || [];
|
||||
buildForecastCase(pred);
|
||||
return pred;
|
||||
}
|
||||
|
||||
function evaluateEntry(entry) {
|
||||
const pred = materializeForecast(entry.forecast);
|
||||
let priorPred = null;
|
||||
let prior = null;
|
||||
|
||||
if (entry.priorForecast) {
|
||||
priorPred = materializeForecast(entry.priorForecast);
|
||||
prior = { predictions: [buildPriorForecastSnapshot(priorPred)] };
|
||||
computeTrends([pred], prior);
|
||||
buildForecastCase(pred);
|
||||
annotateForecastChanges([pred], prior);
|
||||
}
|
||||
|
||||
const readiness = scoreForecastReadiness(pred);
|
||||
const priority = computeAnalysisPriority(pred);
|
||||
const failures = [];
|
||||
const thresholds = entry.thresholds || {};
|
||||
|
||||
if (typeof thresholds.overallMin === 'number' && readiness.overall < thresholds.overallMin) {
|
||||
failures.push(`overall ${readiness.overall} < ${thresholds.overallMin}`);
|
||||
}
|
||||
if (typeof thresholds.overallMax === 'number' && readiness.overall > thresholds.overallMax) {
|
||||
failures.push(`overall ${readiness.overall} > ${thresholds.overallMax}`);
|
||||
}
|
||||
if (typeof thresholds.groundingMin === 'number' && readiness.groundingScore < thresholds.groundingMin) {
|
||||
failures.push(`grounding ${readiness.groundingScore} < ${thresholds.groundingMin}`);
|
||||
}
|
||||
if (typeof thresholds.priorityMin === 'number' && priority < thresholds.priorityMin) {
|
||||
failures.push(`priority ${priority} < ${thresholds.priorityMin}`);
|
||||
}
|
||||
if (typeof thresholds.priorityMax === 'number' && priority > thresholds.priorityMax) {
|
||||
failures.push(`priority ${priority} > ${thresholds.priorityMax}`);
|
||||
}
|
||||
if (typeof thresholds.trend === 'string' && pred.trend !== thresholds.trend) {
|
||||
failures.push(`trend ${pred.trend} !== ${thresholds.trend}`);
|
||||
}
|
||||
for (const fragment of thresholds.changeSummaryIncludes || []) {
|
||||
if (!pred.caseFile?.changeSummary?.includes(fragment)) {
|
||||
failures.push(`changeSummary missing "${fragment}"`);
|
||||
}
|
||||
}
|
||||
for (const fragment of thresholds.changeItemsInclude || []) {
|
||||
const found = (pred.caseFile?.changeItems || []).some(item => item.includes(fragment));
|
||||
if (!found) failures.push(`changeItems missing "${fragment}"`);
|
||||
}
|
||||
|
||||
return {
|
||||
name: entry.name,
|
||||
eventDate: entry.eventDate || null,
|
||||
description: entry.description || '',
|
||||
readiness,
|
||||
priority,
|
||||
trend: pred.trend,
|
||||
changeSummary: pred.caseFile?.changeSummary || '',
|
||||
changeItems: pred.caseFile?.changeItems || [],
|
||||
pass: failures.length === 0,
|
||||
failures,
|
||||
};
|
||||
}
|
||||
|
||||
const suites = benchmarkPaths.map(benchmarkPath => {
|
||||
const benchmark = JSON.parse(readFileSync(benchmarkPath, 'utf8'));
|
||||
const results = benchmark.map(evaluateEntry);
|
||||
const passed = results.filter(result => result.pass).length;
|
||||
return {
|
||||
benchmark: benchmarkPath,
|
||||
cases: results.length,
|
||||
passed,
|
||||
failed: results.length - passed,
|
||||
results,
|
||||
};
|
||||
});
|
||||
|
||||
const summary = {
|
||||
cases: suites.reduce((sum, suite) => sum + suite.cases, 0),
|
||||
passed: suites.reduce((sum, suite) => sum + suite.passed, 0),
|
||||
failed: suites.reduce((sum, suite) => sum + suite.failed, 0),
|
||||
suites,
|
||||
};
|
||||
|
||||
console.log(JSON.stringify(summary, null, 2));
|
||||
|
||||
if (summary.failed > 0) process.exit(1);
|
||||
153
scripts/extract-forecast-benchmark-candidates.mjs
Normal file
153
scripts/extract-forecast-benchmark-candidates.mjs
Normal file
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { loadEnvFile } from './_seed-utils.mjs';
|
||||
import { HISTORY_KEY } from './seed-forecasts.mjs';
|
||||
|
||||
const _isDirectRun = process.argv[1] && import.meta.url.endsWith(process.argv[1].replace(/\\/g, '/'));
|
||||
if (_isDirectRun) loadEnvFile(import.meta.url);
|
||||
|
||||
const NOISE_SIGNAL_TYPES = new Set(['news_corroboration']);
|
||||
|
||||
function slugify(value) {
|
||||
return (value || '')
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, '_')
|
||||
.replace(/^_+|_+$/g, '')
|
||||
.slice(0, 64);
|
||||
}
|
||||
|
||||
function toBenchmarkForecast(entry) {
|
||||
return {
|
||||
domain: entry.domain,
|
||||
region: entry.region,
|
||||
title: entry.title,
|
||||
probability: entry.probability,
|
||||
confidence: entry.confidence,
|
||||
timeHorizon: entry.timeHorizon,
|
||||
trend: entry.trend,
|
||||
signals: entry.signals || [],
|
||||
newsContext: entry.newsContext || [],
|
||||
calibration: entry.calibration || null,
|
||||
cascades: entry.cascades || [],
|
||||
};
|
||||
}
|
||||
|
||||
function summarizeObservedChange(current, prior) {
|
||||
const currentSignals = new Set((current.signals || [])
|
||||
.filter(signal => !NOISE_SIGNAL_TYPES.has(signal.type))
|
||||
.map(signal => signal.value));
|
||||
const priorSignals = new Set((prior.signals || [])
|
||||
.filter(signal => !NOISE_SIGNAL_TYPES.has(signal.type))
|
||||
.map(signal => signal.value));
|
||||
const currentHeadlines = new Set(current.newsContext || []);
|
||||
const priorHeadlines = new Set(prior.newsContext || []);
|
||||
const deltaProbability = +(current.probability - prior.probability).toFixed(3);
|
||||
const newSignals = [...currentSignals].filter(value => !priorSignals.has(value));
|
||||
const newHeadlines = [...currentHeadlines].filter(value => !priorHeadlines.has(value));
|
||||
const marketMove = current.calibration && prior.calibration
|
||||
&& current.calibration.marketTitle === prior.calibration.marketTitle
|
||||
? +((current.calibration.marketPrice || 0) - (prior.calibration.marketPrice || 0)).toFixed(3)
|
||||
: null;
|
||||
|
||||
return {
|
||||
deltaProbability,
|
||||
trend: current.trend,
|
||||
newSignals,
|
||||
newHeadlines,
|
||||
marketMove,
|
||||
};
|
||||
}
|
||||
|
||||
function buildBenchmarkCandidate(current, prior, snapshotAt) {
|
||||
const eventDate = new Date(snapshotAt).toISOString().slice(0, 10);
|
||||
const observedChange = summarizeObservedChange(current, prior);
|
||||
return {
|
||||
name: `${slugify(current.title)}_${eventDate.replace(/-/g, '_')}`,
|
||||
eventDate,
|
||||
description: `${current.title} moved from ${Math.round(prior.probability * 100)}% to ${Math.round(current.probability * 100)}% between consecutive forecast snapshots.`,
|
||||
priorForecast: toBenchmarkForecast(prior),
|
||||
forecast: toBenchmarkForecast(current),
|
||||
observedChange,
|
||||
};
|
||||
}
|
||||
|
||||
function scoreCandidate(candidate) {
|
||||
const absDelta = Math.abs(candidate.observedChange.deltaProbability || 0);
|
||||
const signalBonus = Math.min(0.15, (candidate.observedChange.newSignals?.length || 0) * 0.05);
|
||||
const marketBonus = Math.min(0.15, Math.abs(candidate.observedChange.marketMove || 0) * 0.7);
|
||||
const hasStructuredChange = absDelta >= 0.03
|
||||
|| (candidate.observedChange.newSignals?.length || 0) > 0
|
||||
|| Math.abs(candidate.observedChange.marketMove || 0) >= 0.03;
|
||||
const headlineBonus = hasStructuredChange
|
||||
? Math.min(0.04, (candidate.observedChange.newHeadlines?.length || 0) * 0.02)
|
||||
: 0;
|
||||
return +(absDelta + signalBonus + headlineBonus + marketBonus).toFixed(3);
|
||||
}
|
||||
|
||||
function selectBenchmarkCandidates(historySnapshots, options = {}) {
|
||||
const minDelta = options.minDelta ?? 0.08;
|
||||
const minMarketMove = options.minMarketMove ?? 0.08;
|
||||
const maxCandidates = options.maxCandidates ?? 10;
|
||||
const minInterestingness = options.minInterestingness ?? 0.12;
|
||||
const candidates = [];
|
||||
|
||||
for (let i = 0; i < historySnapshots.length - 1; i++) {
|
||||
const currentSnapshot = historySnapshots[i];
|
||||
const priorSnapshot = historySnapshots[i + 1];
|
||||
const priorMap = new Map((priorSnapshot?.predictions || []).map(pred => [pred.id, pred]));
|
||||
|
||||
for (const current of currentSnapshot?.predictions || []) {
|
||||
const prior = priorMap.get(current.id);
|
||||
if (!prior) continue;
|
||||
const candidate = buildBenchmarkCandidate(current, prior, currentSnapshot.generatedAt);
|
||||
const interestingness = scoreCandidate(candidate);
|
||||
const hasMeaningfulStateChange =
|
||||
Math.abs(candidate.observedChange.deltaProbability) >= minDelta
|
||||
|| Math.abs(candidate.observedChange.marketMove || 0) >= minMarketMove
|
||||
|| (candidate.observedChange.newSignals?.length || 0) > 0;
|
||||
if (!hasMeaningfulStateChange && interestingness < minInterestingness) continue;
|
||||
if (!hasMeaningfulStateChange) continue;
|
||||
candidates.push({ ...candidate, interestingness });
|
||||
}
|
||||
}
|
||||
|
||||
return candidates
|
||||
.sort((a, b) => b.interestingness - a.interestingness || b.eventDate.localeCompare(a.eventDate))
|
||||
.slice(0, maxCandidates);
|
||||
}
|
||||
|
||||
async function readForecastHistory(key = HISTORY_KEY, limit = 60) {
|
||||
const url = process.env.UPSTASH_REDIS_REST_URL;
|
||||
const token = process.env.UPSTASH_REDIS_REST_TOKEN;
|
||||
if (!url || !token) throw new Error('Missing UPSTASH_REDIS_REST_URL or UPSTASH_REDIS_REST_TOKEN');
|
||||
|
||||
const resp = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(['LRANGE', key, 0, Math.max(0, limit - 1)]),
|
||||
signal: AbortSignal.timeout(10_000),
|
||||
});
|
||||
if (!resp.ok) throw new Error(`Redis LRANGE failed: HTTP ${resp.status}`);
|
||||
const payload = await resp.json();
|
||||
const rows = Array.isArray(payload?.result) ? payload.result : [];
|
||||
return rows.map(row => {
|
||||
try { return JSON.parse(row); } catch { return null; }
|
||||
}).filter(Boolean);
|
||||
}
|
||||
|
||||
if (_isDirectRun) {
|
||||
const limitArg = Number(process.argv.find(arg => arg.startsWith('--limit='))?.split('=')[1] || 60);
|
||||
const maxArg = Number(process.argv.find(arg => arg.startsWith('--max-candidates='))?.split('=')[1] || 10);
|
||||
const history = await readForecastHistory(HISTORY_KEY, limitArg);
|
||||
const candidates = selectBenchmarkCandidates(history, { maxCandidates: maxArg });
|
||||
console.log(JSON.stringify({ key: HISTORY_KEY, snapshots: history.length, candidates }, null, 2));
|
||||
}
|
||||
|
||||
export {
|
||||
toBenchmarkForecast,
|
||||
summarizeObservedChange,
|
||||
buildBenchmarkCandidate,
|
||||
scoreCandidate,
|
||||
selectBenchmarkCandidates,
|
||||
readForecastHistory,
|
||||
};
|
||||
292
scripts/promote-forecast-benchmark-candidate.mjs
Normal file
292
scripts/promote-forecast-benchmark-candidate.mjs
Normal file
@@ -0,0 +1,292 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { basename, dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
import { loadEnvFile } from './_seed-utils.mjs';
|
||||
import {
|
||||
readForecastHistory,
|
||||
selectBenchmarkCandidates,
|
||||
} from './extract-forecast-benchmark-candidates.mjs';
|
||||
import {
|
||||
HISTORY_KEY,
|
||||
makePrediction,
|
||||
computeTrends,
|
||||
buildForecastCase,
|
||||
buildPriorForecastSnapshot,
|
||||
annotateForecastChanges,
|
||||
scoreForecastReadiness,
|
||||
computeAnalysisPriority,
|
||||
} from './seed-forecasts.mjs';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const DEFAULT_OUTPUT_PATH = join(__dirname, 'data', 'forecast-historical-benchmark.json');
|
||||
const _isDirectRun = process.argv[1] && import.meta.url.endsWith(process.argv[1].replace(/\\/g, '/'));
|
||||
if (_isDirectRun) loadEnvFile(import.meta.url);
|
||||
|
||||
function roundPct(value) {
|
||||
return `${Math.round((value || 0) * 100)}%`;
|
||||
}
|
||||
|
||||
function materializeForecast(input) {
|
||||
const pred = makePrediction(
|
||||
input.domain,
|
||||
input.region,
|
||||
input.title,
|
||||
input.probability,
|
||||
input.confidence,
|
||||
input.timeHorizon,
|
||||
input.signals || [],
|
||||
);
|
||||
pred.trend = input.trend || pred.trend;
|
||||
pred.newsContext = input.newsContext || [];
|
||||
pred.calibration = input.calibration || null;
|
||||
pred.cascades = input.cascades || [];
|
||||
buildForecastCase(pred);
|
||||
return pred;
|
||||
}
|
||||
|
||||
function buildSummaryExpectation(pred, priorForecast) {
|
||||
if (!priorForecast) return `new in the current run, entering at ${roundPct(pred.probability)}`;
|
||||
|
||||
const delta = pred.probability - priorForecast.probability;
|
||||
if (Math.abs(delta) >= 0.05) {
|
||||
return `${delta > 0 ? 'rose' : 'fell'} from ${roundPct(priorForecast.probability)} to ${roundPct(pred.probability)}`;
|
||||
}
|
||||
return `holding near ${roundPct(pred.probability)} versus ${roundPct(priorForecast.probability)}`;
|
||||
}
|
||||
|
||||
function buildItemExpectations(pred) {
|
||||
return (pred.caseFile?.changeItems || [])
|
||||
.filter(item => item && !item.startsWith('Evidence mix is broadly unchanged'))
|
||||
.slice(0, 3);
|
||||
}
|
||||
|
||||
function deriveThresholds(candidate, options = {}) {
|
||||
const readinessSlack = options.readinessSlack ?? 0.06;
|
||||
const prioritySlack = options.prioritySlack ?? 0.08;
|
||||
const pred = materializeForecast(candidate.forecast);
|
||||
let prior = null;
|
||||
|
||||
if (candidate.priorForecast) {
|
||||
const priorPred = materializeForecast(candidate.priorForecast);
|
||||
prior = { predictions: [buildPriorForecastSnapshot(priorPred)] };
|
||||
computeTrends([pred], prior);
|
||||
buildForecastCase(pred);
|
||||
annotateForecastChanges([pred], prior);
|
||||
}
|
||||
|
||||
const readiness = scoreForecastReadiness(pred);
|
||||
const priority = computeAnalysisPriority(pred);
|
||||
const thresholds = {
|
||||
overallMin: +Math.max(0, readiness.overall - readinessSlack).toFixed(3),
|
||||
overallMax: +Math.min(1, readiness.overall + readinessSlack).toFixed(3),
|
||||
groundingMin: +Math.max(0, readiness.groundingScore - readinessSlack).toFixed(3),
|
||||
priorityMin: +Math.max(0, priority - prioritySlack).toFixed(3),
|
||||
priorityMax: +Math.min(1, priority + prioritySlack).toFixed(3),
|
||||
trend: pred.trend,
|
||||
changeSummaryIncludes: [buildSummaryExpectation(pred, candidate.priorForecast || null)],
|
||||
};
|
||||
|
||||
const itemExpectations = buildItemExpectations(pred);
|
||||
if (itemExpectations.length > 0) thresholds.changeItemsInclude = itemExpectations;
|
||||
|
||||
return thresholds;
|
||||
}
|
||||
|
||||
function toHistoricalBenchmarkEntry(candidate, options = {}) {
|
||||
return {
|
||||
name: candidate.name,
|
||||
eventDate: candidate.eventDate,
|
||||
description: candidate.description,
|
||||
priorForecast: candidate.priorForecast,
|
||||
forecast: candidate.forecast,
|
||||
thresholds: deriveThresholds(candidate, options),
|
||||
};
|
||||
}
|
||||
|
||||
function mergeHistoricalBenchmarks(existingEntries, nextEntry, options = {}) {
|
||||
const replace = options.replace ?? false;
|
||||
const index = existingEntries.findIndex(entry => entry.name === nextEntry.name);
|
||||
|
||||
if (index >= 0 && !replace) {
|
||||
throw new Error(`Benchmark entry "${nextEntry.name}" already exists. Re-run with --replace to overwrite it.`);
|
||||
}
|
||||
|
||||
const merged = [...existingEntries];
|
||||
if (index >= 0) {
|
||||
merged[index] = nextEntry;
|
||||
} else {
|
||||
merged.push(nextEntry);
|
||||
}
|
||||
|
||||
merged.sort((a, b) => {
|
||||
const left = a.eventDate || '';
|
||||
const right = b.eventDate || '';
|
||||
return left.localeCompare(right) || a.name.localeCompare(b.name);
|
||||
});
|
||||
return merged;
|
||||
}
|
||||
|
||||
function createJsonPatch(existingEntries, nextEntry, options = {}) {
|
||||
const index = existingEntries.findIndex(entry => entry.name === nextEntry.name);
|
||||
if (index >= 0) {
|
||||
if (!(options.replace ?? false)) {
|
||||
throw new Error(`Benchmark entry "${nextEntry.name}" already exists. Re-run with --replace to overwrite it.`);
|
||||
}
|
||||
return [{ op: 'replace', path: `/${index}`, value: nextEntry }];
|
||||
}
|
||||
return [{ op: 'add', path: `/${existingEntries.length}`, value: nextEntry }];
|
||||
}
|
||||
|
||||
function renderUnifiedDiff(currentEntries, nextEntries, outputPath) {
|
||||
const tempDir = mkdtempSync(join(tmpdir(), 'forecast-benchmark-'));
|
||||
const currentPath = join(tempDir, `before-${basename(outputPath)}`);
|
||||
const nextPath = join(tempDir, `after-${basename(outputPath)}`);
|
||||
const currentText = `${JSON.stringify(currentEntries, null, 2)}\n`;
|
||||
const nextText = `${JSON.stringify(nextEntries, null, 2)}\n`;
|
||||
|
||||
writeFileSync(currentPath, currentText, 'utf8');
|
||||
writeFileSync(nextPath, nextText, 'utf8');
|
||||
|
||||
try {
|
||||
try {
|
||||
const rawDiff = execFileSync('git', ['diff', '--no-index', '--', currentPath, nextPath], { encoding: 'utf8' });
|
||||
return rawDiff
|
||||
.replaceAll(currentPath, `a/${outputPath}`)
|
||||
.replaceAll(nextPath, `b/${outputPath}`);
|
||||
} catch (error) {
|
||||
const output = `${error.stdout || ''}${error.stderr || ''}`.trim()
|
||||
.replaceAll(currentPath, `a/${outputPath}`)
|
||||
.replaceAll(nextPath, `b/${outputPath}`);
|
||||
if (output) return output;
|
||||
throw error;
|
||||
}
|
||||
} finally {
|
||||
rmSync(tempDir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = {
|
||||
limit: 60,
|
||||
maxCandidates: 10,
|
||||
index: 0,
|
||||
output: DEFAULT_OUTPUT_PATH,
|
||||
write: false,
|
||||
replace: false,
|
||||
name: '',
|
||||
format: 'entry',
|
||||
};
|
||||
|
||||
for (const arg of argv) {
|
||||
if (arg.startsWith('--limit=')) args.limit = Number(arg.split('=')[1] || 60);
|
||||
else if (arg.startsWith('--max-candidates=')) args.maxCandidates = Number(arg.split('=')[1] || 10);
|
||||
else if (arg.startsWith('--index=')) args.index = Number(arg.split('=')[1] || 0);
|
||||
else if (arg.startsWith('--output=')) args.output = arg.split('=').slice(1).join('=');
|
||||
else if (arg.startsWith('--name=')) args.name = arg.split('=').slice(1).join('=');
|
||||
else if (arg.startsWith('--format=')) args.format = arg.split('=').slice(1).join('=') || 'entry';
|
||||
else if (arg === '--write') args.write = true;
|
||||
else if (arg === '--replace') args.replace = true;
|
||||
}
|
||||
|
||||
return args;
|
||||
}
|
||||
|
||||
function pickCandidate(candidates, options = {}) {
|
||||
if (options.name) {
|
||||
const named = candidates.find(candidate => candidate.name === options.name);
|
||||
if (!named) throw new Error(`No extracted candidate named "${options.name}" was found.`);
|
||||
return named;
|
||||
}
|
||||
|
||||
if (!Number.isInteger(options.index) || options.index < 0 || options.index >= candidates.length) {
|
||||
throw new Error(`Candidate index ${options.index} is out of range for ${candidates.length} candidate(s).`);
|
||||
}
|
||||
return candidates[options.index];
|
||||
}
|
||||
|
||||
function readBenchmarkFile(pathname) {
|
||||
return JSON.parse(readFileSync(pathname, 'utf8'));
|
||||
}
|
||||
|
||||
function buildPreviewPayload(args, candidate, nextEntry, currentEntries) {
|
||||
const merged = mergeHistoricalBenchmarks(currentEntries, nextEntry, { replace: args.replace });
|
||||
|
||||
if (args.format === 'json-patch') {
|
||||
return {
|
||||
mode: 'preview',
|
||||
format: 'json-patch',
|
||||
output: args.output,
|
||||
candidateCount: null,
|
||||
selected: candidate.name,
|
||||
patch: createJsonPatch(currentEntries, nextEntry, { replace: args.replace }),
|
||||
};
|
||||
}
|
||||
|
||||
if (args.format === 'diff') {
|
||||
return {
|
||||
mode: 'preview',
|
||||
format: 'diff',
|
||||
output: args.output,
|
||||
selected: candidate.name,
|
||||
diff: renderUnifiedDiff(currentEntries, merged, args.output),
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
mode: 'preview',
|
||||
format: 'entry',
|
||||
output: args.output,
|
||||
selected: candidate.name,
|
||||
entry: nextEntry,
|
||||
};
|
||||
}
|
||||
|
||||
if (_isDirectRun) {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const history = await readForecastHistory(HISTORY_KEY, args.limit);
|
||||
const candidates = selectBenchmarkCandidates(history, { maxCandidates: args.maxCandidates });
|
||||
|
||||
if (candidates.length === 0) {
|
||||
console.error('No promotable forecast benchmark candidates are available yet.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const candidate = pickCandidate(candidates, args);
|
||||
const nextEntry = toHistoricalBenchmarkEntry(candidate);
|
||||
const current = readBenchmarkFile(args.output);
|
||||
|
||||
if (!args.write) {
|
||||
const preview = buildPreviewPayload(args, candidate, nextEntry, current);
|
||||
preview.candidateCount = candidates.length;
|
||||
console.log(JSON.stringify(preview, null, 2));
|
||||
} else {
|
||||
const merged = mergeHistoricalBenchmarks(current, nextEntry, { replace: args.replace });
|
||||
writeFileSync(args.output, `${JSON.stringify(merged, null, 2)}\n`, 'utf8');
|
||||
console.log(JSON.stringify({
|
||||
mode: args.replace ? 'replaced' : 'appended',
|
||||
output: args.output,
|
||||
selected: candidate.name,
|
||||
totalEntries: merged.length,
|
||||
}, null, 2));
|
||||
}
|
||||
}
|
||||
|
||||
export {
|
||||
materializeForecast,
|
||||
buildSummaryExpectation,
|
||||
buildItemExpectations,
|
||||
deriveThresholds,
|
||||
toHistoricalBenchmarkEntry,
|
||||
mergeHistoricalBenchmarks,
|
||||
createJsonPatch,
|
||||
renderUnifiedDiff,
|
||||
buildPreviewPayload,
|
||||
pickCandidate,
|
||||
parseArgs,
|
||||
DEFAULT_OUTPUT_PATH,
|
||||
};
|
||||
File diff suppressed because it is too large
Load Diff
@@ -8,7 +8,6 @@ const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
const BATCH_SIZE = 500;
|
||||
const R2_BUCKET_URL = 'https://api.cloudflare.com/client/v4/accounts/{acct}/r2/buckets/worldmonitor-data/objects/seed-data/military-bases-final.json';
|
||||
const CF_ACCOUNT_ID = 'c1dd10ed1008132d1e8d479b79a98b32';
|
||||
const MAX_RETRIES = 3;
|
||||
const RETRY_BASE_MS = 1000;
|
||||
const PROGRESS_INTERVAL = 5000;
|
||||
@@ -247,10 +246,11 @@ async function main() {
|
||||
|
||||
if (!dataPath) {
|
||||
const cfToken = process.env.CLOUDFLARE_R2_TOKEN || process.env.CLOUDFLARE_API_TOKEN || '';
|
||||
if (cfToken) {
|
||||
const cfAccountId = process.env.CLOUDFLARE_R2_ACCOUNT_ID || '';
|
||||
if (cfToken && cfAccountId) {
|
||||
console.log(' Local file not found — downloading from R2...');
|
||||
try {
|
||||
const r2Url = R2_BUCKET_URL.replace('{acct}', CF_ACCOUNT_ID);
|
||||
const r2Url = R2_BUCKET_URL.replace('{acct}', cfAccountId);
|
||||
const resp = await fetch(r2Url, {
|
||||
headers: { Authorization: `Bearer ${cfToken}` },
|
||||
signal: AbortSignal.timeout(60_000),
|
||||
@@ -267,6 +267,8 @@ async function main() {
|
||||
} catch (err) {
|
||||
console.log(` R2 download failed: ${err.message}`);
|
||||
}
|
||||
} else if (cfToken) {
|
||||
console.log(' R2 download skipped: missing CLOUDFLARE_R2_ACCOUNT_ID');
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -45,6 +45,7 @@ export const BOOTSTRAP_CACHE_KEYS: Record<string, string> = {
|
||||
gdeltIntel: 'intelligence:gdelt-intel:v1',
|
||||
correlationCards: 'correlation:cards-bootstrap:v1',
|
||||
securityAdvisories: 'intelligence:advisories-bootstrap:v1',
|
||||
forecasts: 'forecast:predictions:v2',
|
||||
};
|
||||
|
||||
export const BOOTSTRAP_TIERS: Record<string, 'slow' | 'fast'> = {
|
||||
@@ -63,4 +64,5 @@ export const BOOTSTRAP_TIERS: Record<string, 'slow' | 'fast'> = {
|
||||
iranEvents: 'fast', temporalAnomalies: 'fast', weatherAlerts: 'fast',
|
||||
spending: 'fast', gdeltIntel: 'fast', correlationCards: 'fast',
|
||||
securityAdvisories: 'slow',
|
||||
forecasts: 'fast',
|
||||
};
|
||||
|
||||
@@ -7,7 +7,7 @@ import type {
|
||||
} from '../../../../src/generated/server/worldmonitor/forecast/v1/service_server';
|
||||
import { getCachedJson } from '../../../_shared/redis';
|
||||
|
||||
const REDIS_KEY = 'forecast:predictions:v1';
|
||||
const REDIS_KEY = 'forecast:predictions:v2';
|
||||
|
||||
export const getForecasts: ForecastServiceHandler['getForecasts'] = async (
|
||||
_ctx: ServerContext,
|
||||
|
||||
@@ -1365,6 +1365,11 @@ export class DataLoaderManager implements AppModule {
|
||||
|
||||
async loadForecasts(): Promise<void> {
|
||||
try {
|
||||
const hydrated = getHydratedData('forecasts') as { predictions?: import('@/generated/client/worldmonitor/forecast/v1/service_client').Forecast[] } | undefined;
|
||||
if (hydrated?.predictions?.length) {
|
||||
this.callPanel('forecast', 'updateForecasts', hydrated.predictions);
|
||||
return;
|
||||
}
|
||||
const { fetchForecasts } = await import('@/services/forecast');
|
||||
const forecasts = await fetchForecasts();
|
||||
this.callPanel('forecast', 'updateForecasts', forecasts);
|
||||
|
||||
@@ -41,8 +41,10 @@ function injectStyles(): void {
|
||||
.fc-signal { color: var(--text-secondary, #999); font-size: 11px; padding: 1px 0; }
|
||||
.fc-signal::before { content: ''; display: inline-block; width: 6px; height: 1px; background: var(--text-secondary, #666); margin-right: 6px; vertical-align: middle; }
|
||||
.fc-cascade { font-size: 11px; color: var(--accent-color, #3b82f6); margin-top: 3px; }
|
||||
.fc-summary { font-size: 11px; color: var(--text-primary, #d7d7d7); margin: 6px 0 4px; line-height: 1.45; }
|
||||
.fc-scenario { font-size: 11px; color: var(--text-primary, #ccc); margin: 4px 0; font-style: italic; }
|
||||
.fc-hidden { display: none; }
|
||||
.fc-toggle-row { display: flex; flex-wrap: wrap; gap: 8px; margin-top: 6px; }
|
||||
.fc-toggle { cursor: pointer; color: var(--text-secondary, #888); font-size: 11px; }
|
||||
.fc-toggle:hover { color: var(--text-primary, #eee); }
|
||||
.fc-calibration { font-size: 10px; color: var(--text-secondary, #777); margin-top: 2px; }
|
||||
@@ -50,7 +52,17 @@ function injectStyles(): void {
|
||||
.fc-bar-fill { height: 100%; border-radius: 1.5px; }
|
||||
.fc-empty { padding: 20px; text-align: center; color: var(--text-secondary, #888); }
|
||||
.fc-projections { font-size: 10px; color: var(--text-secondary, #777); margin-top: 3px; font-variant-numeric: tabular-nums; }
|
||||
.fc-perspectives { margin-top: 4px; }
|
||||
.fc-detail { margin-top: 8px; padding-top: 8px; border-top: 1px solid var(--border-color, #2a2a2a); }
|
||||
.fc-detail-grid { display: grid; gap: 8px; }
|
||||
.fc-section { display: grid; gap: 4px; }
|
||||
.fc-section-title { color: var(--text-secondary, #888); font-size: 10px; text-transform: uppercase; letter-spacing: 0.08em; }
|
||||
.fc-section-copy { font-size: 11px; color: var(--text-primary, #d3d3d3); line-height: 1.45; }
|
||||
.fc-list-block { display: grid; gap: 4px; }
|
||||
.fc-list-item { font-size: 11px; color: var(--text-secondary, #a0a0a0); line-height: 1.4; }
|
||||
.fc-list-item::before { content: ''; display: inline-block; width: 6px; height: 1px; background: var(--text-secondary, #666); margin-right: 6px; vertical-align: middle; }
|
||||
.fc-chip-row { display: flex; flex-wrap: wrap; gap: 6px; }
|
||||
.fc-chip { border: 1px solid var(--border-color, #363636); border-radius: 999px; padding: 2px 8px; font-size: 10px; color: var(--text-secondary, #9a9a9a); background: rgba(255,255,255,0.02); }
|
||||
.fc-perspectives { margin-top: 2px; }
|
||||
.fc-perspective { font-size: 11px; color: var(--text-secondary, #999); padding: 2px 0; line-height: 1.4; }
|
||||
.fc-perspective strong { color: var(--text-primary, #ccc); font-weight: 600; }
|
||||
`;
|
||||
@@ -76,7 +88,9 @@ export class ForecastPanel extends Panel {
|
||||
|
||||
const toggle = target.closest('[data-fc-toggle]') as HTMLElement;
|
||||
if (toggle) {
|
||||
const details = toggle.nextElementSibling as HTMLElement;
|
||||
const card = toggle.closest('.fc-card');
|
||||
const panelId = toggle.dataset.fcToggle;
|
||||
const details = panelId ? card?.querySelector(`[data-fc-panel="${panelId}"]`) as HTMLElement | null : null;
|
||||
if (details) details.classList.toggle('fc-hidden');
|
||||
return;
|
||||
}
|
||||
@@ -133,8 +147,8 @@ export class ForecastPanel extends Panel {
|
||||
? `<div class="fc-cascade">Cascades: ${f.cascades.map(c => escapeHtml(c.domain)).join(', ')}</div>`
|
||||
: '';
|
||||
|
||||
const scenarioHtml = f.scenario
|
||||
? `<div class="fc-scenario">${escapeHtml(f.scenario)}</div>`
|
||||
const summaryHtml = (f.feedSummary || f.scenario)
|
||||
? `<div class="fc-summary">${escapeHtml(f.feedSummary || f.scenario)}</div>`
|
||||
: '';
|
||||
|
||||
const calibrationHtml = f.calibration?.marketTitle
|
||||
@@ -146,15 +160,7 @@ export class ForecastPanel extends Panel {
|
||||
? `<div class="fc-projections">24h: ${Math.round(proj.h24 * 100)}% | 7d: ${Math.round(proj.d7 * 100)}% | 30d: ${Math.round(proj.d30 * 100)}%</div>`
|
||||
: '';
|
||||
|
||||
const persp = f.perspectives;
|
||||
const perspectivesHtml = persp?.strategic
|
||||
? `<span class="fc-toggle" data-fc-toggle>Perspectives</span>
|
||||
<div class="fc-perspectives fc-hidden">
|
||||
<div class="fc-perspective"><strong>Strategic:</strong> ${escapeHtml(persp.strategic)}</div>
|
||||
<div class="fc-perspective"><strong>Regional:</strong> ${escapeHtml(persp.regional || '')}</div>
|
||||
<div class="fc-perspective"><strong>Contrarian:</strong> ${escapeHtml(persp.contrarian || '')}</div>
|
||||
</div>`
|
||||
: '';
|
||||
const detailHtml = this.renderDetail(f);
|
||||
|
||||
return `
|
||||
<div class="fc-card">
|
||||
@@ -165,13 +171,230 @@ export class ForecastPanel extends Panel {
|
||||
<div class="fc-bar"><div class="fc-bar-fill" style="width:${pct}%;background:${probColor}"></div></div>
|
||||
${projectionsHtml}
|
||||
<div class="fc-meta">${escapeHtml(f.region)} | ${escapeHtml(f.timeHorizon || '7d')} | <span class="${trendClass}">${f.trend || 'stable'}</span></div>
|
||||
${scenarioHtml}
|
||||
${perspectivesHtml}
|
||||
<span class="fc-toggle" data-fc-toggle>Signals (${(f.signals || []).length})</span>
|
||||
<div class="fc-signals fc-hidden">${signalsHtml}</div>
|
||||
${summaryHtml}
|
||||
<div class="fc-toggle-row">
|
||||
<span class="fc-toggle" data-fc-toggle="detail">Analysis</span>
|
||||
<span class="fc-toggle" data-fc-toggle="signals">Signals (${(f.signals || []).length})</span>
|
||||
</div>
|
||||
${detailHtml}
|
||||
<div class="fc-signals fc-hidden" data-fc-panel="signals">${signalsHtml}</div>
|
||||
${cascadesHtml}
|
||||
${calibrationHtml}
|
||||
</div>
|
||||
`;
|
||||
}
|
||||
|
||||
private renderList(items: string[] | undefined): string {
|
||||
if (!items || items.length === 0) return '';
|
||||
return `<div class="fc-list-block">${items.map(item => `<div class="fc-list-item">${escapeHtml(item)}</div>`).join('')}</div>`;
|
||||
}
|
||||
|
||||
private renderEvidence(items: Array<{ summary?: string; weight?: number }> | undefined): string {
|
||||
if (!items || items.length === 0) return '';
|
||||
return `<div class="fc-list-block">${items.map(item => {
|
||||
const suffix = typeof item.weight === 'number' ? ` (${Math.round(item.weight * 100)}%)` : '';
|
||||
return `<div class="fc-list-item">${escapeHtml(`${item.summary || ''}${suffix}`.trim())}</div>`;
|
||||
}).join('')}</div>`;
|
||||
}
|
||||
|
||||
private renderActors(items: Array<{
|
||||
name?: string;
|
||||
category?: string;
|
||||
role?: string;
|
||||
objectives?: string[];
|
||||
constraints?: string[];
|
||||
likelyActions?: string[];
|
||||
influenceScore?: number;
|
||||
}> | undefined): string {
|
||||
if (!items || items.length === 0) return '';
|
||||
return `<div class="fc-list-block">${items.map(actor => {
|
||||
const chips = [
|
||||
actor.category ? actor.category : '',
|
||||
typeof actor.influenceScore === 'number' ? `Influence ${Math.round(actor.influenceScore * 100)}%` : '',
|
||||
].filter(Boolean).map(chip => `<span class="fc-chip">${escapeHtml(chip)}</span>`).join('');
|
||||
const objective = actor.objectives?.[0] ? `<div class="fc-list-item"><strong>Objective:</strong> ${escapeHtml(actor.objectives[0])}</div>` : '';
|
||||
const constraint = actor.constraints?.[0] ? `<div class="fc-list-item"><strong>Constraint:</strong> ${escapeHtml(actor.constraints[0])}</div>` : '';
|
||||
const action = actor.likelyActions?.[0] ? `<div class="fc-list-item"><strong>Likely action:</strong> ${escapeHtml(actor.likelyActions[0])}</div>` : '';
|
||||
return `
|
||||
<div class="fc-section-copy">
|
||||
<strong>${escapeHtml(actor.name || 'Actor')}</strong>
|
||||
${chips ? `<div class="fc-chip-row" style="margin-top:4px;">${chips}</div>` : ''}
|
||||
${actor.role ? `<div class="fc-list-item">${escapeHtml(actor.role)}</div>` : ''}
|
||||
${objective}
|
||||
${constraint}
|
||||
${action}
|
||||
</div>
|
||||
`;
|
||||
}).join('')}</div>`;
|
||||
}
|
||||
|
||||
private renderBranches(items: Array<{
|
||||
kind?: string;
|
||||
title?: string;
|
||||
summary?: string;
|
||||
outcome?: string;
|
||||
projectedProbability?: number;
|
||||
rounds?: Array<{ round?: number; focus?: string; developments?: string[]; actorMoves?: string[] }>;
|
||||
}> | undefined): string {
|
||||
if (!items || items.length === 0) return '';
|
||||
return `<div class="fc-list-block">${items.map(branch => {
|
||||
const projected = typeof branch.projectedProbability === 'number'
|
||||
? `<span class="fc-chip">Projected ${Math.round(branch.projectedProbability * 100)}%</span>`
|
||||
: '';
|
||||
const rounds = (branch.rounds || []).slice(0, 3).map(round => {
|
||||
const developments = (round.developments || []).slice(0, 2).join(' ');
|
||||
const actorMoves = (round.actorMoves || []).slice(0, 1).join(' ');
|
||||
const copy = [developments, actorMoves].filter(Boolean).join(' ');
|
||||
return `<div class="fc-list-item"><strong>R${round.round || 0}:</strong> ${escapeHtml(copy || round.focus || '')}</div>`;
|
||||
}).join('');
|
||||
return `
|
||||
<div class="fc-section-copy">
|
||||
<strong>${escapeHtml(branch.title || branch.kind || 'Branch')}</strong>
|
||||
<div class="fc-chip-row" style="margin-top:4px;">${projected}</div>
|
||||
${branch.summary ? `<div class="fc-list-item">${escapeHtml(branch.summary)}</div>` : ''}
|
||||
${branch.outcome ? `<div class="fc-list-item"><strong>Outcome:</strong> ${escapeHtml(branch.outcome)}</div>` : ''}
|
||||
${rounds}
|
||||
</div>
|
||||
`;
|
||||
}).join('')}</div>`;
|
||||
}
|
||||
|
||||
private renderDetail(f: Forecast): string {
|
||||
const caseFile = f.caseFile;
|
||||
const sections: string[] = [];
|
||||
|
||||
if (f.scenario) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">Executive View</div>
|
||||
<div class="fc-section-copy fc-scenario">${escapeHtml(f.scenario)}</div>
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
if (caseFile?.baseCase) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">Base Case</div>
|
||||
<div class="fc-section-copy">${escapeHtml(caseFile.baseCase)}</div>
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
if (caseFile?.changeSummary || caseFile?.changeItems?.length) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">What Changed</div>
|
||||
${caseFile?.changeSummary ? `<div class="fc-section-copy">${escapeHtml(caseFile.changeSummary)}</div>` : ''}
|
||||
${caseFile?.changeItems?.length ? this.renderList(caseFile.changeItems) : ''}
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
if (caseFile?.worldState?.summary || caseFile?.worldState?.activePressures?.length || caseFile?.worldState?.stabilizers?.length || caseFile?.worldState?.keyUnknowns?.length) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">World State</div>
|
||||
${caseFile?.worldState?.summary ? `<div class="fc-section-copy">${escapeHtml(caseFile.worldState.summary)}</div>` : ''}
|
||||
${caseFile?.worldState?.activePressures?.length ? `<div class="fc-section-copy"><strong>Pressures:</strong></div>${this.renderList(caseFile.worldState.activePressures)}` : ''}
|
||||
${caseFile?.worldState?.stabilizers?.length ? `<div class="fc-section-copy"><strong>Stabilizers:</strong></div>${this.renderList(caseFile.worldState.stabilizers)}` : ''}
|
||||
${caseFile?.worldState?.keyUnknowns?.length ? `<div class="fc-section-copy"><strong>Key unknowns:</strong></div>${this.renderList(caseFile.worldState.keyUnknowns)}` : ''}
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
if (caseFile?.escalatoryCase || caseFile?.contrarianCase) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">Alternative Paths</div>
|
||||
${caseFile?.escalatoryCase ? `<div class="fc-section-copy"><strong>Escalatory:</strong> ${escapeHtml(caseFile.escalatoryCase)}</div>` : ''}
|
||||
${caseFile?.contrarianCase ? `<div class="fc-section-copy"><strong>Contrarian:</strong> ${escapeHtml(caseFile.contrarianCase)}</div>` : ''}
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
if (caseFile?.branches?.length) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">Simulated Branches</div>
|
||||
${this.renderBranches(caseFile.branches)}
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
if (caseFile?.supportingEvidence?.length) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">Supporting Evidence</div>
|
||||
${this.renderEvidence(caseFile.supportingEvidence)}
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
if (caseFile?.counterEvidence?.length) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">Counter Evidence</div>
|
||||
${this.renderEvidence(caseFile.counterEvidence)}
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
if (caseFile?.triggers?.length) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">Signals To Watch</div>
|
||||
${this.renderList(caseFile.triggers)}
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
if (caseFile?.actors?.length) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">Actors</div>
|
||||
${this.renderActors(caseFile.actors)}
|
||||
</div>
|
||||
`);
|
||||
} else if (caseFile?.actorLenses?.length) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">Actor Lenses</div>
|
||||
${this.renderList(caseFile.actorLenses)}
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
if (f.perspectives?.strategic) {
|
||||
sections.push(`
|
||||
<div class="fc-section">
|
||||
<div class="fc-section-title">Perspectives</div>
|
||||
<div class="fc-perspectives">
|
||||
<div class="fc-perspective"><strong>Strategic:</strong> ${escapeHtml(f.perspectives.strategic)}</div>
|
||||
<div class="fc-perspective"><strong>Regional:</strong> ${escapeHtml(f.perspectives.regional || '')}</div>
|
||||
<div class="fc-perspective"><strong>Contrarian:</strong> ${escapeHtml(f.perspectives.contrarian || '')}</div>
|
||||
</div>
|
||||
</div>
|
||||
`);
|
||||
}
|
||||
|
||||
const chips = [
|
||||
f.calibration?.marketTitle ? `Market: ${f.calibration.marketTitle}` : '',
|
||||
typeof f.priorProbability === 'number' ? `Prior: ${Math.round(f.priorProbability * 100)}%` : '',
|
||||
f.cascades?.length ? `Cascades: ${f.cascades.length}` : '',
|
||||
].filter(Boolean);
|
||||
|
||||
const chipHtml = chips.length > 0
|
||||
? `<div class="fc-section"><div class="fc-section-title">Context</div><div class="fc-chip-row">${chips.map(chip => `<span class="fc-chip">${escapeHtml(chip)}</span>`).join('')}</div></div>`
|
||||
: '';
|
||||
|
||||
return `
|
||||
<div class="fc-detail fc-hidden" data-fc-panel="detail">
|
||||
<div class="fc-detail-grid">
|
||||
${sections.join('')}
|
||||
${chipHtml}
|
||||
</div>
|
||||
</div>
|
||||
`;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,6 +18,7 @@ export interface Forecast {
|
||||
region: string;
|
||||
title: string;
|
||||
scenario: string;
|
||||
feedSummary: string;
|
||||
probability: number;
|
||||
confidence: number;
|
||||
timeHorizon: string;
|
||||
@@ -30,6 +31,7 @@ export interface Forecast {
|
||||
updatedAt: number;
|
||||
perspectives?: Perspectives;
|
||||
projections?: Projections;
|
||||
caseFile?: ForecastCase;
|
||||
}
|
||||
|
||||
export interface ForecastSignal {
|
||||
@@ -63,6 +65,62 @@ export interface Projections {
|
||||
d30: number;
|
||||
}
|
||||
|
||||
export interface ForecastCase {
|
||||
supportingEvidence: ForecastCaseEvidence[];
|
||||
counterEvidence: ForecastCaseEvidence[];
|
||||
triggers: string[];
|
||||
actorLenses: string[];
|
||||
baseCase: string;
|
||||
escalatoryCase: string;
|
||||
contrarianCase: string;
|
||||
changeSummary: string;
|
||||
changeItems: string[];
|
||||
actors: ForecastActor[];
|
||||
worldState?: ForecastWorldState;
|
||||
branches: ForecastBranch[];
|
||||
}
|
||||
|
||||
export interface ForecastCaseEvidence {
|
||||
type: string;
|
||||
summary: string;
|
||||
weight: number;
|
||||
}
|
||||
|
||||
export interface ForecastActor {
|
||||
id: string;
|
||||
name: string;
|
||||
category: string;
|
||||
role: string;
|
||||
objectives: string[];
|
||||
constraints: string[];
|
||||
likelyActions: string[];
|
||||
influenceScore: number;
|
||||
}
|
||||
|
||||
export interface ForecastWorldState {
|
||||
summary: string;
|
||||
activePressures: string[];
|
||||
stabilizers: string[];
|
||||
keyUnknowns: string[];
|
||||
}
|
||||
|
||||
export interface ForecastBranch {
|
||||
kind: string;
|
||||
title: string;
|
||||
summary: string;
|
||||
outcome: string;
|
||||
projectedProbability: number;
|
||||
rounds: ForecastBranchRound[];
|
||||
}
|
||||
|
||||
export interface ForecastBranchRound {
|
||||
round: number;
|
||||
focus: string;
|
||||
developments: string[];
|
||||
actorMoves: string[];
|
||||
probabilityShift: number;
|
||||
}
|
||||
|
||||
export interface FieldViolation {
|
||||
field: string;
|
||||
description: string;
|
||||
|
||||
@@ -18,6 +18,7 @@ export interface Forecast {
|
||||
region: string;
|
||||
title: string;
|
||||
scenario: string;
|
||||
feedSummary: string;
|
||||
probability: number;
|
||||
confidence: number;
|
||||
timeHorizon: string;
|
||||
@@ -30,6 +31,7 @@ export interface Forecast {
|
||||
updatedAt: number;
|
||||
perspectives?: Perspectives;
|
||||
projections?: Projections;
|
||||
caseFile?: ForecastCase;
|
||||
}
|
||||
|
||||
export interface ForecastSignal {
|
||||
@@ -63,6 +65,62 @@ export interface Projections {
|
||||
d30: number;
|
||||
}
|
||||
|
||||
export interface ForecastCase {
|
||||
supportingEvidence: ForecastCaseEvidence[];
|
||||
counterEvidence: ForecastCaseEvidence[];
|
||||
triggers: string[];
|
||||
actorLenses: string[];
|
||||
baseCase: string;
|
||||
escalatoryCase: string;
|
||||
contrarianCase: string;
|
||||
changeSummary: string;
|
||||
changeItems: string[];
|
||||
actors: ForecastActor[];
|
||||
worldState?: ForecastWorldState;
|
||||
branches: ForecastBranch[];
|
||||
}
|
||||
|
||||
export interface ForecastCaseEvidence {
|
||||
type: string;
|
||||
summary: string;
|
||||
weight: number;
|
||||
}
|
||||
|
||||
export interface ForecastActor {
|
||||
id: string;
|
||||
name: string;
|
||||
category: string;
|
||||
role: string;
|
||||
objectives: string[];
|
||||
constraints: string[];
|
||||
likelyActions: string[];
|
||||
influenceScore: number;
|
||||
}
|
||||
|
||||
export interface ForecastWorldState {
|
||||
summary: string;
|
||||
activePressures: string[];
|
||||
stabilizers: string[];
|
||||
keyUnknowns: string[];
|
||||
}
|
||||
|
||||
export interface ForecastBranch {
|
||||
kind: string;
|
||||
title: string;
|
||||
summary: string;
|
||||
outcome: string;
|
||||
projectedProbability: number;
|
||||
rounds: ForecastBranchRound[];
|
||||
}
|
||||
|
||||
export interface ForecastBranchRound {
|
||||
round: number;
|
||||
focus: string;
|
||||
developments: string[];
|
||||
actorMoves: string[];
|
||||
probabilityShift: number;
|
||||
}
|
||||
|
||||
export interface FieldViolation {
|
||||
field: string;
|
||||
description: string;
|
||||
|
||||
@@ -24,11 +24,37 @@ import {
|
||||
discoverGraphCascades,
|
||||
attachNewsContext,
|
||||
computeConfidence,
|
||||
computeHeadlineRelevance,
|
||||
computeMarketMatchScore,
|
||||
sanitizeForPrompt,
|
||||
parseLLMScenarios,
|
||||
validateScenarios,
|
||||
validatePerspectives,
|
||||
validateCaseNarratives,
|
||||
computeProjections,
|
||||
buildUserPrompt,
|
||||
buildForecastCase,
|
||||
buildForecastCases,
|
||||
buildPriorForecastSnapshot,
|
||||
buildChangeItems,
|
||||
buildChangeSummary,
|
||||
annotateForecastChanges,
|
||||
buildCounterEvidence,
|
||||
buildCaseTriggers,
|
||||
buildForecastActors,
|
||||
buildForecastWorldState,
|
||||
buildForecastBranches,
|
||||
buildActorLenses,
|
||||
scoreForecastReadiness,
|
||||
computeAnalysisPriority,
|
||||
rankForecastsForAnalysis,
|
||||
buildFallbackScenario,
|
||||
buildFallbackBaseCase,
|
||||
buildFallbackEscalatoryCase,
|
||||
buildFallbackContrarianCase,
|
||||
buildFeedSummary,
|
||||
buildFallbackPerspectives,
|
||||
populateFallbackNarratives,
|
||||
loadCascadeRules,
|
||||
evaluateRuleConditions,
|
||||
SIGNAL_TO_SOURCE,
|
||||
@@ -217,6 +243,19 @@ describe('calibrateWithMarkets', () => {
|
||||
calibrateWithMarkets([pred], { crypto: [] });
|
||||
assert.equal(pred.calibration, null);
|
||||
});
|
||||
|
||||
it('does not calibrate from unrelated same-region macro market', () => {
|
||||
const pred = makePrediction(
|
||||
'conflict', 'Middle East', 'Escalation risk: Iran',
|
||||
0.7, 0.6, '7d', [],
|
||||
);
|
||||
const markets = {
|
||||
geopolitical: [{ title: 'Will Netanyahu remain prime minister through 2026?', yesPrice: 20, source: 'polymarket', volume: 100000 }],
|
||||
};
|
||||
calibrateWithMarkets([pred], markets);
|
||||
assert.equal(pred.calibration, null);
|
||||
assert.equal(pred.probability, 0.7);
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeTrends', () => {
|
||||
@@ -463,7 +502,7 @@ describe('attachNewsContext', () => {
|
||||
assert.equal(corr, undefined);
|
||||
});
|
||||
|
||||
it('falls back to generic headlines when no match', () => {
|
||||
it('does not attach unrelated generic headlines when no match', () => {
|
||||
const preds = [makePrediction('conflict', 'Iran', 'test', 0.5, 0.5, '7d', [])];
|
||||
const news = { topStories: [
|
||||
{ primaryTitle: 'Unrelated headline about sports' },
|
||||
@@ -472,7 +511,7 @@ describe('attachNewsContext', () => {
|
||||
{ primaryTitle: 'Fourth unrelated story' },
|
||||
]};
|
||||
attachNewsContext(preds, news);
|
||||
assert.equal(preds[0].newsContext.length, 3); // fallback top-3
|
||||
assert.deepEqual(preds[0].newsContext, []);
|
||||
});
|
||||
|
||||
it('excludes commodity node names from matching (no false positives)', () => {
|
||||
@@ -509,6 +548,309 @@ describe('attachNewsContext', () => {
|
||||
attachNewsContext(preds, { topStories: [] }, null);
|
||||
assert.equal(preds[0].newsContext, undefined);
|
||||
});
|
||||
|
||||
it('prefers region-relevant headlines over generic domain-only matches', () => {
|
||||
const preds = [makePrediction('supply_chain', 'Red Sea', 'Shipping disruption: Red Sea', 0.6, 0.4, '7d', [])];
|
||||
const news = { topStories: [
|
||||
{ primaryTitle: 'Global shipping stocks rise despite broader market weakness' },
|
||||
{ primaryTitle: 'Red Sea shipping disruption worsens after new attacks' },
|
||||
{ primaryTitle: 'Freight rates react to Red Sea rerouting' },
|
||||
]};
|
||||
attachNewsContext(preds, news);
|
||||
assert.ok(preds[0].newsContext[0].includes('Red Sea'));
|
||||
assert.ok(preds[0].newsContext.every(h => /Red Sea|rerouting/i.test(h)));
|
||||
});
|
||||
});
|
||||
|
||||
describe('headline and market relevance helpers', () => {
|
||||
it('scores region-specific headlines above generic domain headlines', () => {
|
||||
const terms = ['Red Sea', 'Yemen'];
|
||||
const specific = computeHeadlineRelevance('Red Sea shipping disruption worsens after new attacks', terms, 'supply_chain');
|
||||
const generic = computeHeadlineRelevance('Global shipping shares rise in New York trading', terms, 'supply_chain');
|
||||
assert.ok(specific > generic);
|
||||
});
|
||||
|
||||
it('scores semantically aligned markets above broad regional ones', () => {
|
||||
const pred = makePrediction('conflict', 'Middle East', 'Escalation risk: Iran', 0.7, 0.5, '7d', []);
|
||||
const targeted = computeMarketMatchScore(pred, 'Will Iran conflict escalate before July?', ['Iran', 'Middle East']);
|
||||
const broad = computeMarketMatchScore(pred, 'Will Netanyahu remain prime minister through 2026?', ['Iran', 'Middle East']);
|
||||
assert.ok(targeted > broad);
|
||||
});
|
||||
});
|
||||
|
||||
describe('forecast case assembly', () => {
|
||||
it('buildForecastCase assembles evidence, triggers, and actors from current forecast data', () => {
|
||||
const pred = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.7, 0.42, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87 (critical)', weight: 0.4 },
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
]);
|
||||
pred.newsContext = ['Iran military drills intensify after border incident'];
|
||||
pred.calibration = { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.58, drift: 0.12, source: 'polymarket' };
|
||||
pred.cascades = [{ domain: 'market', effect: 'commodity price shock', probability: 0.41 }];
|
||||
pred.trend = 'falling';
|
||||
pred.priorProbability = 0.78;
|
||||
|
||||
const caseFile = buildForecastCase(pred);
|
||||
assert.ok(caseFile.supportingEvidence.some(item => item.type === 'cii'));
|
||||
assert.ok(caseFile.supportingEvidence.some(item => item.type === 'headline'));
|
||||
assert.ok(caseFile.supportingEvidence.some(item => item.type === 'market_calibration'));
|
||||
assert.ok(caseFile.supportingEvidence.some(item => item.type === 'cascade'));
|
||||
assert.ok(caseFile.counterEvidence.length >= 1);
|
||||
assert.ok(caseFile.triggers.length >= 1);
|
||||
assert.ok(caseFile.actorLenses.length >= 1);
|
||||
assert.ok(caseFile.actors.length >= 1);
|
||||
assert.ok(caseFile.worldState.summary.includes('Iran'));
|
||||
assert.ok(caseFile.worldState.activePressures.length >= 1);
|
||||
assert.equal(caseFile.branches.length, 3);
|
||||
});
|
||||
|
||||
it('buildForecastCases populates the case file for every forecast', () => {
|
||||
const a = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.7, 0.6, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87', weight: 0.4 },
|
||||
]);
|
||||
const b = makePrediction('market', 'Red Sea', 'Shipping price shock', 0.55, 0.5, '30d', [
|
||||
{ type: 'chokepoint', value: 'Red Sea risk: high', weight: 0.5 },
|
||||
]);
|
||||
buildForecastCases([a, b]);
|
||||
assert.ok(a.caseFile);
|
||||
assert.ok(b.caseFile);
|
||||
});
|
||||
|
||||
it('helper functions return structured case ingredients', () => {
|
||||
const pred = makePrediction('supply_chain', 'Red Sea', 'Supply chain disruption: Red Sea', 0.64, 0.35, '7d', [
|
||||
{ type: 'chokepoint', value: 'Red Sea disruption detected', weight: 0.5 },
|
||||
{ type: 'gps_jamming', value: 'GPS interference near Red Sea', weight: 0.2 },
|
||||
]);
|
||||
pred.trend = 'rising';
|
||||
pred.cascades = [{ domain: 'market', effect: 'supply shortage pricing', probability: 0.38 }];
|
||||
|
||||
const counter = buildCounterEvidence(pred);
|
||||
const triggers = buildCaseTriggers(pred);
|
||||
const structuredActors = buildForecastActors(pred);
|
||||
const worldState = buildForecastWorldState(pred, structuredActors, triggers, counter);
|
||||
const branches = buildForecastBranches(pred, {
|
||||
actors: structuredActors,
|
||||
triggers,
|
||||
counterEvidence: counter,
|
||||
worldState,
|
||||
});
|
||||
const actorLenses = buildActorLenses(pred);
|
||||
assert.ok(Array.isArray(counter));
|
||||
assert.ok(triggers.length >= 1);
|
||||
assert.ok(structuredActors.length >= 1);
|
||||
assert.ok(worldState.summary.includes('Red Sea'));
|
||||
assert.ok(worldState.activePressures.length >= 1);
|
||||
assert.equal(branches.length, 3);
|
||||
assert.ok(branches[0].rounds.length >= 3);
|
||||
assert.ok(actorLenses.length >= 1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('forecast evaluation and ranking', () => {
|
||||
it('scores evidence-rich forecasts above thin forecasts', () => {
|
||||
const rich = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.7, 0.62, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87 (critical)', weight: 0.4 },
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
{ type: 'theater', value: 'Middle East theater posture elevated', weight: 0.2 },
|
||||
]);
|
||||
rich.newsContext = ['Iran military drills intensify after border incident'];
|
||||
rich.calibration = { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.58, drift: 0.04, source: 'polymarket' };
|
||||
rich.cascades = [{ domain: 'market', effect: 'commodity price shock', probability: 0.41 }];
|
||||
rich.trend = 'rising';
|
||||
buildForecastCase(rich);
|
||||
|
||||
const thin = makePrediction('market', 'Europe', 'Energy stress: Europe', 0.7, 0.62, '7d', [
|
||||
{ type: 'prediction_market', value: 'Broad market stress chatter', weight: 0.2 },
|
||||
]);
|
||||
thin.trend = 'stable';
|
||||
buildForecastCase(thin);
|
||||
|
||||
const richScore = scoreForecastReadiness(rich);
|
||||
const thinScore = scoreForecastReadiness(thin);
|
||||
assert.ok(richScore.overall > thinScore.overall);
|
||||
assert.ok(richScore.groundingScore > thinScore.groundingScore);
|
||||
});
|
||||
|
||||
it('uses readiness to rank better-grounded forecasts ahead of thinner peers', () => {
|
||||
const rich = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.66, 0.58, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87 (critical)', weight: 0.4 },
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
]);
|
||||
rich.newsContext = ['Iran military drills intensify after border incident'];
|
||||
rich.calibration = { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.57, drift: 0.03, source: 'polymarket' };
|
||||
rich.trend = 'rising';
|
||||
buildForecastCase(rich);
|
||||
|
||||
const thin = makePrediction('market', 'Europe', 'Energy stress: Europe', 0.69, 0.58, '7d', [
|
||||
{ type: 'prediction_market', value: 'Broad market stress chatter', weight: 0.2 },
|
||||
]);
|
||||
thin.trend = 'stable';
|
||||
buildForecastCase(thin);
|
||||
|
||||
assert.ok(computeAnalysisPriority(rich) > computeAnalysisPriority(thin));
|
||||
|
||||
const ranked = [thin, rich];
|
||||
rankForecastsForAnalysis(ranked);
|
||||
assert.equal(ranked[0].title, rich.title);
|
||||
});
|
||||
});
|
||||
|
||||
describe('forecast change tracking', () => {
|
||||
it('builds prior snapshots with enough context for evidence diffs', () => {
|
||||
const pred = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.7, 0.6, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87 (critical)', weight: 0.4 },
|
||||
]);
|
||||
pred.newsContext = ['Iran military drills intensify after border incident'];
|
||||
pred.calibration = { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.58, drift: 0.04, source: 'polymarket' };
|
||||
const snapshot = buildPriorForecastSnapshot(pred);
|
||||
assert.equal(snapshot.id, pred.id);
|
||||
assert.deepEqual(snapshot.signals, ['Iran CII 87 (critical)']);
|
||||
assert.deepEqual(snapshot.newsContext, ['Iran military drills intensify after border incident']);
|
||||
assert.equal(snapshot.calibration.marketTitle, 'Will Iran conflict escalate before July?');
|
||||
});
|
||||
|
||||
it('annotates what changed versus the prior run', () => {
|
||||
const pred = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.72, 0.6, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87 (critical)', weight: 0.4 },
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
]);
|
||||
pred.newsContext = [
|
||||
'Iran military drills intensify after border incident',
|
||||
'Regional officials warn of retaliation risk',
|
||||
];
|
||||
pred.calibration = { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.64, drift: 0.04, source: 'polymarket' };
|
||||
buildForecastCase(pred);
|
||||
|
||||
const prior = {
|
||||
predictions: [{
|
||||
id: pred.id,
|
||||
probability: 0.58,
|
||||
signals: ['Iran CII 87 (critical)'],
|
||||
newsContext: ['Iran military drills intensify after border incident'],
|
||||
calibration: { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.53 },
|
||||
}],
|
||||
};
|
||||
|
||||
annotateForecastChanges([pred], prior);
|
||||
assert.match(pred.caseFile.changeSummary, /Probability rose from 58% to 72%/);
|
||||
assert.ok(pred.caseFile.changeItems.some(item => item.includes('New signal: 3 UCDP conflict events')));
|
||||
assert.ok(pred.caseFile.changeItems.some(item => item.includes('New reporting: Regional officials warn of retaliation risk')));
|
||||
assert.ok(pred.caseFile.changeItems.some(item => item.includes('Market moved from 53% to 64%')));
|
||||
});
|
||||
|
||||
it('marks newly surfaced forecasts clearly', () => {
|
||||
const pred = makePrediction('market', 'Europe', 'Energy stress: Europe', 0.55, 0.5, '30d', [
|
||||
{ type: 'prediction_market', value: 'Broad market stress chatter', weight: 0.2 },
|
||||
]);
|
||||
buildForecastCase(pred);
|
||||
const items = buildChangeItems(pred, null);
|
||||
const summary = buildChangeSummary(pred, null, items);
|
||||
assert.match(summary, /new in the current run/i);
|
||||
assert.ok(items[0].includes('New forecast surfaced'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('forecast narrative fallbacks', () => {
|
||||
it('buildUserPrompt keeps headlines scoped to each prediction', () => {
|
||||
const a = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.7, 0.6, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87', weight: 0.4 },
|
||||
]);
|
||||
a.newsContext = ['Iran military drills intensify'];
|
||||
a.projections = { h24: 0.6, d7: 0.7, d30: 0.5 };
|
||||
buildForecastCase(a);
|
||||
|
||||
const b = makePrediction('market', 'Europe', 'Gas price shock in Europe', 0.55, 0.5, '30d', [
|
||||
{ type: 'market', value: 'EU gas futures spike', weight: 0.3 },
|
||||
]);
|
||||
b.newsContext = ['European gas storage draw accelerates'];
|
||||
b.projections = { h24: 0.5, d7: 0.55, d30: 0.6 };
|
||||
buildForecastCase(b);
|
||||
|
||||
const prompt = buildUserPrompt([a, b]);
|
||||
assert.match(prompt, /\[0\][\s\S]*Iran military drills intensify/);
|
||||
assert.match(prompt, /\[1\][\s\S]*European gas storage draw accelerates/);
|
||||
assert.ok(!prompt.includes('Current top headlines:'));
|
||||
assert.match(prompt, /\[SUPPORTING_EVIDENCE\]/);
|
||||
assert.match(prompt, /\[ACTORS\]/);
|
||||
assert.match(prompt, /\[WORLD_STATE\]/);
|
||||
assert.match(prompt, /\[SIMULATED_BRANCHES\]/);
|
||||
});
|
||||
|
||||
it('populateFallbackNarratives fills missing scenario, perspectives, and case narratives', () => {
|
||||
const pred = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.7, 0.6, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87 (critical)', weight: 0.4 },
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
]);
|
||||
pred.trend = 'rising';
|
||||
populateFallbackNarratives([pred]);
|
||||
assert.match(pred.scenario, /Iran CII 87|central path/i);
|
||||
assert.ok(pred.perspectives?.strategic);
|
||||
assert.ok(pred.perspectives?.regional);
|
||||
assert.ok(pred.perspectives?.contrarian);
|
||||
assert.ok(pred.caseFile?.baseCase);
|
||||
assert.ok(pred.caseFile?.escalatoryCase);
|
||||
assert.ok(pred.caseFile?.contrarianCase);
|
||||
assert.equal(pred.caseFile?.branches?.length, 3);
|
||||
assert.ok(pred.feedSummary);
|
||||
});
|
||||
|
||||
it('fallback perspective references calibration when present', () => {
|
||||
const pred = makePrediction('market', 'Middle East', 'Oil price impact', 0.65, 0.5, '30d', [
|
||||
{ type: 'chokepoint', value: 'Hormuz disruption detected', weight: 0.5 },
|
||||
]);
|
||||
pred.calibration = { marketTitle: 'Will oil close above $90?', marketPrice: 0.62, drift: 0.03, source: 'polymarket' };
|
||||
const perspectives = buildFallbackPerspectives(pred);
|
||||
assert.match(perspectives.contrarian, /Will oil close above \$90/);
|
||||
});
|
||||
|
||||
it('fallback scenario stays concise and evidence-led', () => {
|
||||
const pred = makePrediction('infrastructure', 'France', 'Infrastructure cascade risk: France', 0.48, 0.4, '24h', [
|
||||
{ type: 'outage', value: 'France major outage', weight: 0.4 },
|
||||
]);
|
||||
const scenario = buildFallbackScenario(pred);
|
||||
assert.match(scenario, /France major outage/);
|
||||
assert.ok(scenario.length <= 500);
|
||||
});
|
||||
|
||||
it('fallback case narratives stay evidence-led and concise', () => {
|
||||
const pred = makePrediction('infrastructure', 'France', 'Infrastructure cascade risk: France', 0.48, 0.4, '24h', [
|
||||
{ type: 'outage', value: 'France major outage', weight: 0.4 },
|
||||
]);
|
||||
buildForecastCase(pred);
|
||||
const baseCase = buildFallbackBaseCase(pred);
|
||||
const escalatoryCase = buildFallbackEscalatoryCase(pred);
|
||||
const contrarianCase = buildFallbackContrarianCase(pred);
|
||||
assert.match(baseCase, /France major outage/);
|
||||
assert.ok(escalatoryCase.length <= 500);
|
||||
assert.ok(contrarianCase.length <= 500);
|
||||
});
|
||||
|
||||
it('buildFeedSummary stays compact and distinct from the deeper case output', () => {
|
||||
const pred = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.7, 0.6, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87 (critical)', weight: 0.4 },
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
]);
|
||||
buildForecastCase(pred);
|
||||
pred.caseFile.baseCase = 'Iran CII 87 (critical) and 3 UCDP conflict events keep the base path elevated over the next 7d with persistent force pressure.';
|
||||
const summary = buildFeedSummary(pred);
|
||||
assert.ok(summary.length <= 180);
|
||||
assert.match(summary, /Iran CII 87/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateCaseNarratives', () => {
|
||||
it('accepts valid case narratives', () => {
|
||||
const pred = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.7, 0.6, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87', weight: 0.4 },
|
||||
]);
|
||||
const valid = validateCaseNarratives([{
|
||||
index: 0,
|
||||
baseCase: 'Iran CII 87 remains the main anchor for the base path in the next 7d.',
|
||||
escalatoryCase: 'A further rise in Iran CII 87 and added conflict-event reporting would move risk materially higher.',
|
||||
contrarianCase: 'If no new corroborating headlines appear, the current path would lose support and flatten out.',
|
||||
}], [pred]);
|
||||
assert.equal(valid.length, 1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeConfidence', () => {
|
||||
@@ -628,7 +970,33 @@ describe('validateScenarios', () => {
|
||||
assert.equal(valid.length, 1);
|
||||
});
|
||||
|
||||
it('rejects scenario without signal reference', () => {
|
||||
it('accepts scenario with headline reference', () => {
|
||||
preds[0].newsContext = ['Iran military drills intensify after border incident'];
|
||||
const scenarios = [{ index: 0, scenario: 'Iran military drills intensify after border incident, keeping escalation pressure elevated over the next 7d.' }];
|
||||
const valid = validateScenarios(scenarios, preds);
|
||||
assert.equal(valid.length, 1);
|
||||
delete preds[0].newsContext;
|
||||
});
|
||||
|
||||
it('accepts scenario with market cue and trigger reference', () => {
|
||||
preds[0].calibration = { marketTitle: 'Will oil close above $90?', marketPrice: 0.62, drift: 0.03, source: 'polymarket' };
|
||||
preds[0].caseFile = {
|
||||
supportingEvidence: [],
|
||||
counterEvidence: [],
|
||||
triggers: ['A market repricing of 8-10 points would be a meaningful confirmation or rejection signal.'],
|
||||
actorLenses: [],
|
||||
baseCase: '',
|
||||
escalatoryCase: '',
|
||||
contrarianCase: '',
|
||||
};
|
||||
const scenarios = [{ index: 0, scenario: 'Will oil close above $90? remains a live market cue, and a market repricing of 8-10 points would confirm the current path.' }];
|
||||
const valid = validateScenarios(scenarios, preds);
|
||||
assert.equal(valid.length, 1);
|
||||
delete preds[0].calibration;
|
||||
delete preds[0].caseFile;
|
||||
});
|
||||
|
||||
it('rejects scenario without any evidence reference', () => {
|
||||
const scenarios = [{ index: 0, scenario: 'Tensions continue to rise in the region due to various geopolitical factors and ongoing disputes.' }];
|
||||
const valid = validateScenarios(scenarios, preds);
|
||||
assert.equal(valid.length, 0);
|
||||
|
||||
322
tests/forecast-history.test.mjs
Normal file
322
tests/forecast-history.test.mjs
Normal file
@@ -0,0 +1,322 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { describe, it } from 'node:test';
|
||||
|
||||
import {
|
||||
makePrediction,
|
||||
buildHistorySnapshot,
|
||||
buildForecastCase,
|
||||
} from '../scripts/seed-forecasts.mjs';
|
||||
|
||||
import {
|
||||
selectBenchmarkCandidates,
|
||||
summarizeObservedChange,
|
||||
} from '../scripts/extract-forecast-benchmark-candidates.mjs';
|
||||
|
||||
import {
|
||||
toHistoricalBenchmarkEntry,
|
||||
mergeHistoricalBenchmarks,
|
||||
createJsonPatch,
|
||||
buildPreviewPayload,
|
||||
} from '../scripts/promote-forecast-benchmark-candidate.mjs';
|
||||
|
||||
describe('forecast history snapshot', () => {
|
||||
it('buildHistorySnapshot stores a compact rolling snapshot', () => {
|
||||
const rich = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.7, 0.6, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 87 (critical)', weight: 0.4 },
|
||||
]);
|
||||
rich.newsContext = ['Iran military drills intensify after border incident'];
|
||||
buildForecastCase(rich);
|
||||
|
||||
const thin = makePrediction('market', 'Europe', 'Energy stress: Europe', 0.5, 0.4, '30d', [
|
||||
{ type: 'prediction_market', value: 'Broad market stress chatter', weight: 0.2 },
|
||||
]);
|
||||
buildForecastCase(thin);
|
||||
|
||||
const snapshot = buildHistorySnapshot({ generatedAt: 1234, predictions: [rich, thin] }, { maxForecasts: 1 });
|
||||
assert.equal(snapshot.generatedAt, 1234);
|
||||
assert.equal(snapshot.predictions.length, 1);
|
||||
assert.equal(snapshot.predictions[0].title, rich.title);
|
||||
assert.deepEqual(snapshot.predictions[0].signals[0], { type: 'cii', value: 'Iran CII 87 (critical)', weight: 0.4 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('forecast history candidate extraction', () => {
|
||||
it('summarizes observed change across consecutive snapshots', () => {
|
||||
const prior = {
|
||||
id: 'fc-conflict-1',
|
||||
domain: 'conflict',
|
||||
region: 'Iran',
|
||||
title: 'Escalation risk: Iran',
|
||||
probability: 0.5,
|
||||
confidence: 0.55,
|
||||
timeHorizon: '7d',
|
||||
trend: 'stable',
|
||||
signals: [{ type: 'cii', value: 'Iran CII 79 (high)', weight: 0.4 }],
|
||||
newsContext: ['Iran military drills intensify after border incident'],
|
||||
calibration: { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.45 },
|
||||
cascades: [],
|
||||
};
|
||||
const current = {
|
||||
...prior,
|
||||
probability: 0.68,
|
||||
trend: 'rising',
|
||||
signals: [
|
||||
...prior.signals,
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
],
|
||||
newsContext: [...prior.newsContext, 'Regional officials warn of retaliation risk'],
|
||||
calibration: { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.66 },
|
||||
};
|
||||
|
||||
const observed = summarizeObservedChange(current, prior);
|
||||
assert.equal(observed.deltaProbability, 0.18);
|
||||
assert.deepEqual(observed.newSignals, ['3 UCDP conflict events']);
|
||||
assert.deepEqual(observed.newHeadlines, ['Regional officials warn of retaliation risk']);
|
||||
assert.equal(observed.marketMove, 0.21);
|
||||
});
|
||||
|
||||
it('selects benchmark candidates from rolling history', () => {
|
||||
const newest = {
|
||||
generatedAt: Date.parse('2024-04-14T12:00:00Z'),
|
||||
predictions: [{
|
||||
id: 'fc-conflict-1',
|
||||
domain: 'conflict',
|
||||
region: 'Iran',
|
||||
title: 'Escalation risk: Iran',
|
||||
probability: 0.74,
|
||||
confidence: 0.64,
|
||||
timeHorizon: '7d',
|
||||
trend: 'rising',
|
||||
signals: [
|
||||
{ type: 'cii', value: 'Iran CII 79 (high)', weight: 0.4 },
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
],
|
||||
newsContext: [
|
||||
'Iran military drills intensify after border incident',
|
||||
'Regional officials warn of retaliation risk',
|
||||
],
|
||||
calibration: { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.71 },
|
||||
cascades: [],
|
||||
}],
|
||||
};
|
||||
const prior = {
|
||||
generatedAt: Date.parse('2024-04-13T12:00:00Z'),
|
||||
predictions: [{
|
||||
id: 'fc-conflict-1',
|
||||
domain: 'conflict',
|
||||
region: 'Iran',
|
||||
title: 'Escalation risk: Iran',
|
||||
probability: 0.46,
|
||||
confidence: 0.55,
|
||||
timeHorizon: '7d',
|
||||
trend: 'stable',
|
||||
signals: [
|
||||
{ type: 'cii', value: 'Iran CII 79 (high)', weight: 0.4 },
|
||||
],
|
||||
newsContext: ['Iran military drills intensify after border incident'],
|
||||
calibration: { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.45 },
|
||||
cascades: [],
|
||||
}],
|
||||
};
|
||||
|
||||
const candidates = selectBenchmarkCandidates([newest, prior], { maxCandidates: 5 });
|
||||
assert.equal(candidates.length, 1);
|
||||
assert.match(candidates[0].name, /escalation_risk_iran_2024_04_14/);
|
||||
assert.equal(candidates[0].observedChange.deltaProbability, 0.28);
|
||||
assert.ok(candidates[0].interestingness > 0.2);
|
||||
});
|
||||
|
||||
it('ignores headline churn when there is no meaningful state change', () => {
|
||||
const newest = {
|
||||
generatedAt: Date.parse('2024-04-14T12:00:00Z'),
|
||||
predictions: [{
|
||||
id: 'fc-conflict-1',
|
||||
domain: 'conflict',
|
||||
region: 'Iran',
|
||||
title: 'Escalation risk: Iran',
|
||||
probability: 0.46,
|
||||
confidence: 0.55,
|
||||
timeHorizon: '7d',
|
||||
trend: 'stable',
|
||||
signals: [
|
||||
{ type: 'cii', value: 'Iran CII 79 (high)', weight: 0.4 },
|
||||
{ type: 'news_corroboration', value: '6 headline(s) mention Iran or linked entities', weight: 0.15 },
|
||||
],
|
||||
newsContext: [
|
||||
'Regional officials warn of retaliation risk',
|
||||
'Fresh commentary on Iranian posture appears',
|
||||
],
|
||||
calibration: { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.46 },
|
||||
cascades: [],
|
||||
}],
|
||||
};
|
||||
const prior = {
|
||||
generatedAt: Date.parse('2024-04-13T12:00:00Z'),
|
||||
predictions: [{
|
||||
id: 'fc-conflict-1',
|
||||
domain: 'conflict',
|
||||
region: 'Iran',
|
||||
title: 'Escalation risk: Iran',
|
||||
probability: 0.455,
|
||||
confidence: 0.55,
|
||||
timeHorizon: '7d',
|
||||
trend: 'stable',
|
||||
signals: [
|
||||
{ type: 'cii', value: 'Iran CII 79 (high)', weight: 0.4 },
|
||||
{ type: 'news_corroboration', value: '60 headline(s) mention Iran or linked entities', weight: 0.15 },
|
||||
],
|
||||
newsContext: [
|
||||
'Earlier commentary on Iranian posture appears',
|
||||
],
|
||||
calibration: { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.455 },
|
||||
cascades: [],
|
||||
}],
|
||||
};
|
||||
|
||||
const candidates = selectBenchmarkCandidates([newest, prior], { maxCandidates: 5 });
|
||||
assert.equal(candidates.length, 0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('forecast benchmark promotion', () => {
|
||||
it('builds a historical benchmark entry with derived thresholds', () => {
|
||||
const newest = {
|
||||
generatedAt: Date.parse('2024-04-14T12:00:00Z'),
|
||||
predictions: [{
|
||||
id: 'fc-conflict-1',
|
||||
domain: 'conflict',
|
||||
region: 'Iran',
|
||||
title: 'Escalation risk: Iran',
|
||||
probability: 0.74,
|
||||
confidence: 0.64,
|
||||
timeHorizon: '7d',
|
||||
trend: 'rising',
|
||||
signals: [
|
||||
{ type: 'cii', value: 'Iran CII 79 (high)', weight: 0.4 },
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
],
|
||||
newsContext: [
|
||||
'Iran military drills intensify after border incident',
|
||||
'Regional officials warn of retaliation risk',
|
||||
],
|
||||
calibration: { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.71 },
|
||||
cascades: [],
|
||||
}],
|
||||
};
|
||||
const prior = {
|
||||
generatedAt: Date.parse('2024-04-13T12:00:00Z'),
|
||||
predictions: [{
|
||||
id: 'fc-conflict-1',
|
||||
domain: 'conflict',
|
||||
region: 'Iran',
|
||||
title: 'Escalation risk: Iran',
|
||||
probability: 0.46,
|
||||
confidence: 0.55,
|
||||
timeHorizon: '7d',
|
||||
trend: 'stable',
|
||||
signals: [
|
||||
{ type: 'cii', value: 'Iran CII 79 (high)', weight: 0.4 },
|
||||
],
|
||||
newsContext: ['Iran military drills intensify after border incident'],
|
||||
calibration: { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.45 },
|
||||
cascades: [],
|
||||
}],
|
||||
};
|
||||
|
||||
const [candidate] = selectBenchmarkCandidates([newest, prior], { maxCandidates: 5 });
|
||||
const entry = toHistoricalBenchmarkEntry(candidate);
|
||||
|
||||
assert.equal(entry.name, candidate.name);
|
||||
assert.equal(entry.thresholds.trend, 'rising');
|
||||
assert.match(entry.thresholds.changeSummaryIncludes[0], /rose from 46% to 74%/);
|
||||
assert.ok(entry.thresholds.overallMin <= entry.thresholds.overallMax);
|
||||
assert.ok(entry.thresholds.priorityMin <= entry.thresholds.priorityMax);
|
||||
assert.ok(entry.thresholds.changeItemsInclude.some(item => item.includes('New signal: 3 UCDP conflict events')));
|
||||
});
|
||||
|
||||
it('merges a promoted historical entry by append or replace', () => {
|
||||
const existing = [
|
||||
{ name: 'red_sea_shipping_disruption_2024_01_15', eventDate: '2024-01-15' },
|
||||
];
|
||||
const nextEntry = {
|
||||
name: 'iran_exchange_2024_04_14',
|
||||
eventDate: '2024-04-14',
|
||||
description: 'desc',
|
||||
forecast: {},
|
||||
thresholds: {},
|
||||
};
|
||||
|
||||
const appended = mergeHistoricalBenchmarks(existing, nextEntry);
|
||||
assert.equal(appended.length, 2);
|
||||
assert.equal(appended[1].name, 'iran_exchange_2024_04_14');
|
||||
|
||||
assert.throws(() => mergeHistoricalBenchmarks(appended, nextEntry), /already exists/);
|
||||
|
||||
const replaced = mergeHistoricalBenchmarks(appended, { ...nextEntry, description: 'updated' }, { replace: true });
|
||||
assert.equal(replaced.length, 2);
|
||||
assert.equal(replaced[1].description, 'updated');
|
||||
});
|
||||
|
||||
it('emits JSON patch previews and unified diffs without writing files', () => {
|
||||
const existing = [
|
||||
{
|
||||
name: 'red_sea_shipping_disruption_2024_01_15',
|
||||
eventDate: '2024-01-15',
|
||||
description: 'old',
|
||||
},
|
||||
];
|
||||
const candidate = {
|
||||
name: 'iran_exchange_2024_04_14',
|
||||
eventDate: '2024-04-14',
|
||||
description: 'Iran escalation risk jumps',
|
||||
priorForecast: {
|
||||
domain: 'conflict',
|
||||
region: 'Iran',
|
||||
title: 'Escalation risk: Iran',
|
||||
probability: 0.46,
|
||||
confidence: 0.55,
|
||||
timeHorizon: '7d',
|
||||
signals: [{ type: 'cii', value: 'Iran CII 79 (high)', weight: 0.4 }],
|
||||
},
|
||||
forecast: {
|
||||
domain: 'conflict',
|
||||
region: 'Iran',
|
||||
title: 'Escalation risk: Iran',
|
||||
probability: 0.74,
|
||||
confidence: 0.64,
|
||||
timeHorizon: '7d',
|
||||
trend: 'rising',
|
||||
signals: [
|
||||
{ type: 'cii', value: 'Iran CII 79 (high)', weight: 0.4 },
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
],
|
||||
newsContext: ['Regional officials warn of retaliation risk'],
|
||||
calibration: { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.71 },
|
||||
},
|
||||
};
|
||||
|
||||
const nextEntry = toHistoricalBenchmarkEntry(candidate);
|
||||
const patch = createJsonPatch(existing, nextEntry);
|
||||
assert.deepEqual(patch[0].op, 'add');
|
||||
assert.deepEqual(patch[0].path, '/1');
|
||||
|
||||
const jsonPreview = buildPreviewPayload(
|
||||
{ format: 'json-patch', output: '/tmp/forecast-historical-benchmark.json', replace: false },
|
||||
candidate,
|
||||
nextEntry,
|
||||
existing,
|
||||
);
|
||||
assert.equal(jsonPreview.format, 'json-patch');
|
||||
assert.equal(jsonPreview.patch[0].op, 'add');
|
||||
|
||||
const diffPreview = buildPreviewPayload(
|
||||
{ format: 'diff', output: '/tmp/forecast-historical-benchmark.json', replace: false },
|
||||
candidate,
|
||||
nextEntry,
|
||||
existing,
|
||||
);
|
||||
assert.equal(diffPreview.format, 'diff');
|
||||
assert.match(diffPreview.diff, /Escalation risk: Iran/);
|
||||
assert.match(diffPreview.diff, /Iran escalation risk jumps/);
|
||||
});
|
||||
});
|
||||
83
tests/forecast-trace-export.test.mjs
Normal file
83
tests/forecast-trace-export.test.mjs
Normal file
@@ -0,0 +1,83 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { describe, it } from 'node:test';
|
||||
|
||||
import {
|
||||
makePrediction,
|
||||
buildForecastCase,
|
||||
populateFallbackNarratives,
|
||||
buildForecastTraceArtifacts,
|
||||
} from '../scripts/seed-forecasts.mjs';
|
||||
|
||||
import {
|
||||
resolveR2StorageConfig,
|
||||
} from '../scripts/_r2-storage.mjs';
|
||||
|
||||
describe('forecast trace storage config', () => {
|
||||
it('resolves Cloudflare R2 trace env vars and derives the endpoint from account id', () => {
|
||||
const config = resolveR2StorageConfig({
|
||||
CLOUDFLARE_R2_ACCOUNT_ID: 'acct123',
|
||||
CLOUDFLARE_R2_TRACE_BUCKET: 'trace-bucket',
|
||||
CLOUDFLARE_R2_ACCESS_KEY_ID: 'abc',
|
||||
CLOUDFLARE_R2_SECRET_ACCESS_KEY: 'def',
|
||||
CLOUDFLARE_R2_REGION: 'auto',
|
||||
CLOUDFLARE_R2_TRACE_PREFIX: 'custom-prefix',
|
||||
CLOUDFLARE_R2_FORCE_PATH_STYLE: 'true',
|
||||
});
|
||||
assert.equal(config.bucket, 'trace-bucket');
|
||||
assert.equal(config.endpoint, 'https://acct123.r2.cloudflarestorage.com');
|
||||
assert.equal(config.region, 'auto');
|
||||
assert.equal(config.basePrefix, 'custom-prefix');
|
||||
assert.equal(config.forcePathStyle, true);
|
||||
});
|
||||
|
||||
it('falls back to a shared Cloudflare R2 bucket env var', () => {
|
||||
const config = resolveR2StorageConfig({
|
||||
CLOUDFLARE_R2_ACCOUNT_ID: 'acct123',
|
||||
CLOUDFLARE_R2_BUCKET: 'shared-bucket',
|
||||
CLOUDFLARE_R2_ACCESS_KEY_ID: 'abc',
|
||||
CLOUDFLARE_R2_SECRET_ACCESS_KEY: 'def',
|
||||
});
|
||||
assert.equal(config.bucket, 'shared-bucket');
|
||||
assert.equal(config.endpoint, 'https://acct123.r2.cloudflarestorage.com');
|
||||
});
|
||||
});
|
||||
|
||||
describe('forecast trace artifact builder', () => {
|
||||
it('builds manifest, summary, and per-forecast trace artifacts', () => {
|
||||
const a = makePrediction('conflict', 'Iran', 'Escalation risk: Iran', 0.74, 0.64, '7d', [
|
||||
{ type: 'cii', value: 'Iran CII 79 (high)', weight: 0.4 },
|
||||
{ type: 'ucdp', value: '3 UCDP conflict events', weight: 0.3 },
|
||||
]);
|
||||
a.newsContext = ['Regional officials warn of retaliation risk'];
|
||||
a.calibration = { marketTitle: 'Will Iran conflict escalate before July?', marketPrice: 0.71, drift: 0.03, source: 'polymarket' };
|
||||
a.trend = 'rising';
|
||||
buildForecastCase(a);
|
||||
|
||||
const b = makePrediction('supply_chain', 'Red Sea', 'Shipping disruption: Red Sea', 0.68, 0.59, '7d', [
|
||||
{ type: 'chokepoint', value: 'Red Sea disruption detected', weight: 0.5 },
|
||||
{ type: 'gps_jamming', value: 'GPS interference near Red Sea', weight: 0.2 },
|
||||
]);
|
||||
b.newsContext = ['Freight rates react to Red Sea rerouting'];
|
||||
b.trend = 'rising';
|
||||
buildForecastCase(b);
|
||||
|
||||
populateFallbackNarratives([a, b]);
|
||||
|
||||
const artifacts = buildForecastTraceArtifacts(
|
||||
{ generatedAt: Date.parse('2026-03-15T08:00:00Z'), predictions: [a, b] },
|
||||
{ runId: 'run-123' },
|
||||
{ basePrefix: 'forecast-runs', maxForecasts: 1 },
|
||||
);
|
||||
|
||||
assert.equal(artifacts.manifest.runId, 'run-123');
|
||||
assert.equal(artifacts.manifest.forecastCount, 2);
|
||||
assert.equal(artifacts.manifest.tracedForecastCount, 1);
|
||||
assert.match(artifacts.manifestKey, /forecast-runs\/2026\/03\/15\/run-123\/manifest\.json/);
|
||||
assert.match(artifacts.summaryKey, /forecast-runs\/2026\/03\/15\/run-123\/summary\.json/);
|
||||
assert.equal(artifacts.forecasts.length, 1);
|
||||
assert.equal(artifacts.summary.topForecasts[0].id, a.id);
|
||||
assert.ok(artifacts.forecasts[0].payload.caseFile.worldState.summary.includes('Iran'));
|
||||
assert.equal(artifacts.forecasts[0].payload.caseFile.branches.length, 3);
|
||||
assert.equal(artifacts.forecasts[0].payload.traceMeta.narrativeSource, 'fallback');
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user