mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
main
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Elie Habib
|
dec7b64b17 |
fix(unrest): proxy-only fetch + 3-attempt retry for GDELT (#3395)
* fix(unrest): proxy-only fetch + 3-attempt retry for GDELT Production logs showed PR #3362's 45s proxy timeout solved one failure mode (CONNECT-tunnel timeouts) but ~80% of ticks now fail in 3-14 seconds with either "Proxy CONNECT: HTTP/1.1 522 Server Error" (Cloudflare can't reach GDELT origin) or "Client network socket disconnected before secure TLS connection" (Decodo RSTs the handshake). These are fast-fails, not timeouts — no amount of timeout bumping helps. Two changes: 1. Drop the direct fetch entirely. Every direct attempt in 14h of logs errored with UND_ERR_CONNECT_TIMEOUT or ECONNRESET — 0% success since PR #3256 added the proxy fallback. The direct call costs ~8-30s per tick for nothing. 2. Wrap the proxy call in a 3-attempt retry with 1.5-3s jitter. Single-attempt per-tick success rate measured at ~18%; with 3 attempts that lifts to ~75%+ under the same Decodo↔Cloudflare flake rate, comfortably keeping seedAge under the 120m STALE_SEED threshold. Deeper structural fix (out of scope here): wire ACLED credentials on the Railway unrest service so GDELT isn't the single upstream. * test(unrest): cover GDELT proxy retry path + no-proxy hard-fail Address PR #3395 reviewer concerns: (1) "no automated coverage for the new retry path or the no-proxy path" Add scripts/seed-unrest-events.mjs DI seams (_proxyFetcher, _sleep, _jitter, _maxAttempts, _resolveProxyForConnect) and a 6-test suite at tests/seed-unrest-gdelt-fetch.test.mjs covering: 1. Single-attempt success — no retries fire. 2. 2 transient failures + 3rd-attempt success — recovers, returns JSON. 3. All attempts fail — throws LAST error, exact attempt count. 4. Malformed proxy body — SyntaxError short-circuits retry (deterministic parse failures shouldn't burn attempts). 5. Missing CONNECT proxy creds — fetchGdeltEvents throws clear "PROXY_URL env var is not set" pointer for ops, asserts NO proxy fetcher invocation (no wasted network). 6. End-to-end with retry — fetchGdeltEvents with one transient 522 recovers and aggregates events normally. Gate runSeed() entry-point with `import.meta.url === file://argv[1]` so tests can `import` the module without triggering a real seed run. (2) "review assumes Railway has Decodo creds; without them, fails immediately" Yes — that's intentional. Direct fetch had 0% success in production for weeks (every Railway tick errored UND_ERR_CONNECT_TIMEOUT or ECONNRESET) since PR #3256 added the proxy fallback. Reintroducing it as "soft" fallback would just add ~30s of latency + log noise per tick. What's improved here: the no-proxy error message now names the missing env var (PROXY_URL) so an operator who hits this in Railway logs has a direct pointer instead of a generic "GDELT requires proxy" string. |
||
|
Elie Habib
|
b68d98972a |
fix(unrest): bump GDELT proxy timeout 20s → 45s (#3362)
GDELT's v1 gkg_geojson endpoint is currently responding in ~19s (direct curl test: HTTP 200 at t=19.4s). With the old 20s proxy timeout the Decodo leg hits Cloudflare origin timeout and returns HTTP 522 on nearly every tick, so fetchGdeltEvents throws "both paths failed — proxy: HTTP/1.1 522 Server Error" and runSeed freezes seed-meta fetchedAt. Result: the unrest:events seed-meta stops advancing while Redis still holds the last-good payload — health.js reports STALE_SEED even though the seeder is running on schedule every 45 min. 4.5+ hours of consecutive failures observed in production logs overnight. Direct path has been chronically broken (UND_ERR_CONNECT_TIMEOUT in every tick since PR #3256 added the proxy fallback), so the proxy is the real fetch path. Giving it 45s absorbs GDELT's current degraded response time with headroom, without changing any other behavior. ACLED credentials remain unconfigured in this environment, so GDELT is effectively the single upstream — separate ops task to wire ACLED as a real second source. |
||
|
Elie Habib
|
65a1210531 |
fix(unrest): Decodo proxy fallback for GDELT + surface err.cause (#3256)
* fix(unrest): Decodo proxy fallback for GDELT + surface err.cause Background: unrestEvents went STALE_SEED when every tick logged "GDELT failed: fetch failed" (Railway log 2026-04-21). The bare "fetch failed" string hid the actual cause (DNS/TCP/TLS), so the outage was opaque. ACLED is disabled (no credentials) so GDELT is the sole live source — when it fails, the seed freezes. Changes: - fetchGdeltEvents: direct-first, Decodo proxy fallback via httpsProxyFetchRaw when PROXY_URL is configured. Mirrors imfFetchJson / _yahoo-fetch.mjs direct→proxy pattern. - Error messages now include err.cause.code (UND_ERR_CONNECT_TIMEOUT, ENOTFOUND, ECONNRESET, etc.) so the next outage surfaces the underlying transport error instead of "fetch failed". - Both-paths-failed error carries direct + proxy message so either can be diagnosed from a single log line. No behavior change on the happy path — direct fetch still runs first with the existing 30s AbortSignal timeout. * fix(unrest): address PR #3256 P2 review - describeErr: handle plain-string .cause (e.g. `{ cause: 'ENOTFOUND' }`) that would otherwise be silently dropped since a string has no .code/.errno/.message accessors. - fetchGdeltDirect: tag HTTP-status errors (!resp.ok) with httpStatus. fetchGdeltEvents skips the proxy hop for upstream HTTP errors since the proxy routes to the same GDELT endpoint — saves the 20s proxy timeout and avoids a pointless retry. Transport failures (DNS/TCP/TLS timeouts against Railway IPs) still trigger the proxy fallback, which is the motivating case. |
||
|
Elie Habib
|
044598346e |
feat(seed-contract): PR 2a — runSeed envelope dual-write + 91 seeders migrated (#3097)
* feat(seed-contract): PR 2a — runSeed envelope dual-write + 91 seeders migrated
Opt-in contract path in runSeed: when opts.declareRecords is provided, write
{_seed, data} envelope to the canonical key alongside legacy seed-meta:*
(dual-write). State machine: OK / OK_ZERO / RETRY with zeroIsValid opt.
declareRecords throws or returns non-integer → hard fail (contract violation).
extraKeys[*] support per-key declareRecords; each extra key writes its own
envelope. Legacy seeders (no declareRecords) entirely unchanged.
Migrated all 91 scripts/seed-*.mjs to contract mode. Each exports
declareRecords returning the canonical record count, and passes
schemaVersion: 1 + maxStaleMin (matched to api/health.js SEED_META, or 2.5x
interval where no registry entry exists). Contract conformance reports 84/86
seeders with full descriptor (2 pre-existing warnings).
Legacy seed-meta keys still written so unmigrated readers keep working;
follow-up slices flip health.js + readers to envelope-first.
Tests: 61/61 PR 1 tests still pass.
Next slices for PR 2:
- api/health.js registry collapse + 15 seed-bundle-*.mjs canonicalKey wiring
- reader migration (mcp, resilience, aviation, displacement, regional-snapshot)
- direct writers — ais-relay.cjs, consumer-prices-core publish.ts
- public-boundary stripSeedEnvelope + test migration
Plan: docs/plans/2026-04-14-002-fix-runseed-zero-record-lockout-plan.md
* fix(seed-contract): unwrap envelopes in internal cross-seed readers
After PR 2a enveloped 91 canonical keys as {_seed, data}, every script-side
reader that returned the raw parsed JSON started silently handing callers the
envelope instead of the bare payload. WoW baselines (bigmac, grocery-basket,
fear-greed) saw undefined .countries / .composite; seed-climate-anomalies saw
undefined .normals from climate:zone-normals:v1; seed-thermal-escalation saw
undefined .fireDetections from wildfire:fires:v1; seed-forecasts' ~40-key
pipeline batch returned envelopes for every input.
Fix: route every script-side reader through unwrapEnvelope(...).data. Legacy
bare-shape values pass through unchanged (unwrapEnvelope returns
{_seed: null, data: raw} for any non-envelope shape).
Changed:
- scripts/_seed-utils.mjs: import unwrapEnvelope; redisGet, readSeedSnapshot,
verifySeedKey all unwrap. Exported new readCanonicalValue() helper for
cross-seed consumers.
- 18 seed-*.mjs scripts with local redisGet-style helpers or inline fetch
patched to unwrap via the envelope source module (subagent sweep).
- scripts/seed-forecasts.mjs pipeline batch: parse() unwraps each result.
- scripts/seed-energy-spine.mjs redisMget: unwraps each result.
Tests:
- tests/seed-utils-envelope-reads.test.mjs: 7 new cases covering envelope
+ legacy + null paths for readSeedSnapshot and verifySeedKey.
- Full seed suite: 67/67 pass (was 61, +6 new).
Addresses both of user's P1 findings on PR #3097.
* feat(seed-contract): envelope-aware reads in server + api helpers
Every RPC and public-boundary reader now automatically strips _seed from
contract-mode canonical keys. Legacy bare-shape values pass through unchanged
(unwrapEnvelope no-ops on non-envelope shapes).
Changed helpers (one-place fix — unblocks ~60 call sites):
- server/_shared/redis.ts: getRawJson, getCachedJson, getCachedJsonBatch
unwrap by default. cachedFetchJson inherits via getCachedJson.
- api/_upstash-json.js: readJsonFromUpstash unwraps (covers api/mcp.ts
tool responses + all its canonical-key reads).
- api/bootstrap.js: getCachedJsonBatch unwraps (public-boundary —
clients never see envelope metadata).
Left intentionally unchanged:
- api/health.js / api/seed-health.js: read only seed-meta:* keys which
remain bare-shape during dual-write. unwrapEnvelope already imported at
the meta-read boundary (PR 1) as a defensive no-op.
Tests: 67/67 seed tests pass. typecheck + typecheck:api clean.
This is the blast-radius fix the PR #3097 review called out — external
readers that would otherwise see {_seed, data} after the writer side
migrated.
* fix(test): strip export keyword in vm.runInContext'd seed source
cross-source-signals-regulatory.test.mjs loads scripts/seed-cross-source-signals.mjs
via vm.runInContext, which cannot parse ESM `export` syntax. PR 2a added
`export function declareRecords` to every seeder, which broke this test's
static-analysis approach.
Fix: strip the `export` keyword from the declareRecords line in the
preprocessed source string so the function body still evaluates as a plain
declaration.
Full test:data suite: 5307/5307 pass. typecheck + typecheck:api clean.
* feat(seed-contract): consumer-prices publish.ts writes envelopes
Wrap the 5 canonical keys written by consumer-prices-core/src/jobs/publish.ts
(overview, movers:7d/30d, freshness, categories:7d/30d/90d, retailer-spread,
basket-series) in {_seed, data} envelopes. Legacy seed-meta:<key> writes
preserved for dual-write.
Inlined a buildEnvelope helper (10 lines) rather than taking a cross-package
dependency — consumer-prices-core is a standalone npm package. Documented the
four-file parity contract (mjs source, ts mirror, js edge mirror, this copy).
Contract fields: sourceVersion='consumer-prices-core-publish-v1', schemaVersion=1,
state='OK' (recordCount>0) or 'OK_ZERO' (legitimate zero).
Typecheck: no new errors in publish.ts.
* fix(seed-contract): 3 more server-side readers unwrap envelopes
Found during final audit:
- server/worldmonitor/resilience/v1/_shared.ts: resilience score reader
parsed cached GetResilienceScoreResponse raw. Contract-mode seed-resilience-scores
now envelopes those keys.
- server/worldmonitor/resilience/v1/get-resilience-ranking.ts: p05/p95
interval lookup parsed raw from seed-resilience-scores' extra-key path.
- server/worldmonitor/infrastructure/v1/_shared.ts: mgetJson() used for
count-source keys (wildfire:fires:v1, news:insights:v1) which are both
contract-mode now.
All three now unwrap via server/_shared/seed-envelope. Legacy shapes pass
through unchanged.
Typecheck clean.
* feat(seed-contract): ais-relay.cjs direct writes produce envelopes
32 canonical-key write sites in scripts/ais-relay.cjs now produce {_seed, data}
envelopes. Inlined buildEnvelope() (CJS module can't require ESM source) +
envelopeWrite(key, data, ttlSeconds, meta) wrapper. Enveloped keys span market
bootstrap, aviation, cyber-threats, theater-posture, weather-alerts, economic
spending/fred/worldbank, tech-events, corridor-risk, usni-fleet, shipping-stress,
social:reddit, wsb-tickers, pizzint, product-catalog, chokepoint transits,
ucdp-events, satellites, oref.
Left bare (not seeded data keys): seed-meta:* (dual-write legacy),
classifyCacheKey LLM cache, notam:prev-closed-state internal state,
wm:notif:scan-dedup flags.
Updated tests/ucdp-seed-resilience.test.mjs regex to accept both upstashSet
(pre-contract) and envelopeWrite (post-contract) call patterns.
* feat(seed-contract): 15 bundle files add canonicalKey for envelope gate
54 bundle sections across 12 files now declare canonicalKey alongside the
existing seedMetaKey. _bundle-runner.mjs (from PR 1) prefers canonicalKey
when both are present — gates section runs on envelope._seed.fetchedAt
read directly from the data key, eliminating the meta-outlives-data class
of bugs.
Files touched:
- climate (5), derived-signals (2), ecb-eu (3), energy-sources (6),
health (2), imf-extended (4), macro (10), market-backup (9),
portwatch (4), relay-backup (2), resilience-recovery (5), static-ref (2)
Skipped (14 sections, 3 whole bundles): multi-key writers, dynamic
templated keys (displacement year-scoped), or non-runSeed orchestrators
(regional brief cron, resilience-scores' 222-country publish, validation/
benchmark scripts). These continue to use seedMetaKey or their own gate.
seedMetaKey preserved everywhere — dual-write. _bundle-runner.mjs falls
back to legacy when canonicalKey is absent.
All 15 bundles pass node --check. test:data: 5307/5307. typecheck:all: clean.
* fix(seed-contract): 4 PR #3097 review P1s — transform/declareRecords mismatches + envelope leaks
Addresses both P1 findings and the extra-key seed-meta leak surfaced in review:
1. runSeed helper-level invariant: seed-meta:* keys NEVER envelope.
scripts/_seed-utils.mjs exports shouldEnvelopeKey(key) — returns false for
any key starting with 'seed-meta:'. Both atomicPublish (canonical) and
writeExtraKey (extras) gate the envelope wrap through this helper. Fixes
seed-iea-oil-stocks' ANALYSIS_META_EXTRA_KEY silently getting enveloped,
which broke health.js parsing the value as bare {fetchedAt, recordCount}.
Also defends against any future manual writeExtraKey(..., envelopeMeta)
call that happens to target a seed-meta:* key.
2. seed-token-panels canonical + extras fixed.
publishTransform returns data.defi (the defi panel itself, shape {tokens}).
Old declareRecords counted data.defi.tokens + data.ai.tokens + data.other.tokens
on the transformed payload → 0 → RETRY path → canonical market:defi-tokens:v1
never wrote, and because runSeed returned before the extraKeys loop,
market:ai-tokens:v1 + market:other-tokens:v1 stayed stale too.
New: declareRecords counts data.tokens on the transformed shape. AI_KEY +
OTHER_KEY extras reuse the same function (transforms return structurally
identical panels). Added isMain guard so test imports don't fire runSeed.
3. api/product-catalog.js cached reader unwraps envelope.
ais-relay.cjs now envelopes product-catalog:v2 via envelopeWrite(). The
edge reader did raw JSON.parse(result) and returned {_seed, data} to
clients, breaking the cached path. Fix: import unwrapEnvelope from
./_seed-envelope.js, apply after JSON.parse. One site — :238-241 is
downstream of getFromCache(), so the single reader fix covers both.
4. Regression lock tests/seed-contract-transform-regressions.test.mjs (11 cases):
- shouldEnvelopeKey invariant: seed-meta:* false, canonical true
- Token-panels declareRecords works on transformed shape (canonical + both extras)
- Explicit repro of pre-fix buggy signature returning 0 — guards against revert
- resolveRecordCount accepts 0, rejects non-integer
- Product-catalog envelope unwrap returns bare shape; legacy passes through
Verification:
- npm run test:data → 5318/5318 pass (was 5307 — 11 new regressions)
- npm run typecheck:all → clean
- node --check on every modified script
iea-oil-stocks canonical declareRecords was NOT broken (user confirmed during
review — buildIndex preserves .members); only its ANALYSIS_META_EXTRA_KEY
was affected, now covered generically by commit 1's helper invariant.
* fix(seed-contract): seed-token-panels validateFn also runs on post-transform shape
Review finding: fixing declareRecords wasn't sufficient — atomicPublish() runs
validateFn(publishData) on the transformed payload too. seed-token-panels'
validate() checked data.defi/.ai/.other on the transformed {tokens} shape,
returned false, and runSeed took the early skipped-write branch (before even
reaching the declareRecords RETRY logic). Net effect: same as before the
declareRecords fix — canonical + both extras stayed stale.
Fix: validate() now checks the canonical defi panel directly (Array.isArray
(data?.tokens) && has at least one t.price > 0). AI/OTHER panels are validated
implicitly by their own extraKey declareRecords on write.
Audited the other 9 seeders with publishTransform (bls-series, bis-extended,
bis-data, gdelt-intel, trade-flows, iea-oil-stocks, jodi-gas, sanctions-pressure,
forecasts): all validateFn's correctly target the post-transform shape. Only
token-panels regressed.
Added 4 regression tests (tests/seed-contract-transform-regressions.test.mjs):
- validate accepts transformed panel with priced tokens
- validate rejects all-zero-price tokens
- validate rejects empty/missing tokens
- Explicit pre-fix repro (buggy old signature fails on transformed shape)
Verification:
- npm run test:data → 5322/5322 pass (was 5318; +4 new)
- npm run typecheck:all → clean
- node --check clean
* feat(seed-contract): add /api/seed-contract-probe validation endpoint
Single machine-readable gate for 'is PR #3097 working in production'.
Replaces the curl/jq ritual with one authenticated edge call that returns
HTTP 200 ok:true or 503 + failing check list.
What it validates:
- 8 canonical keys have {_seed, data} envelopes with required data fields
and minRecords floors (fsi-eu, zone-normals, 3 token panels + minRecords
guard against token-panels RETRY regression, product-catalog, wildfire,
earthquakes).
- 2 seed-meta:* keys remain BARE (shouldEnvelopeKey invariant; guards
against iea-oil-stocks ANALYSIS_META_EXTRA_KEY-class regressions).
- /api/product-catalog + /api/bootstrap responses contain no '_seed' leak.
Auth: x-probe-secret header must match RELAY_SHARED_SECRET (reuses existing
Vercel↔Railway internal trust boundary).
Probe logic is exported (checkProbe, checkPublicBoundary, DEFAULT_PROBES) for
hermetic testing. tests/seed-contract-probe.test.mjs covers every branch:
envelope pass/fail on field/records/shape, bare pass/fail on shape/field,
missing/malformed JSON, Redis non-2xx, boundary seed-leak detection,
DEFAULT_PROBES sanity (seed-meta invariant present, token-panels minRecords
guard present).
Usage:
curl -H "x-probe-secret: $RELAY_SHARED_SECRET" \
https://api.worldmonitor.app/api/seed-contract-probe
PR 3 will extend the probe with a stricter mode that asserts seed-meta:*
keys are GONE (not just bare) once legacy dual-write is removed.
Verification:
- tests/seed-contract-probe.test.mjs → 15/15 pass
- npm run test:data → 5338/5338 (was 5322; +16 new incl. conformance)
- npm run typecheck:all → clean
* fix(seed-contract): tighten probe — minRecords on AI/OTHER + cache-path source header
Review P2 findings: the probe's stated guards were weaker than advertised.
1. market:ai-tokens:v1 + market:other-tokens:v1 probes claimed to guard the
token-panels extra-key RETRY regression but only checked shape='envelope'
+ dataHas:['tokens']. If an extra-key declareRecords regressed to 0, both
probes would still pass because checkProbe() only inspects _seed.recordCount
when minRecords is set. Now both enforce minRecords: 1.
2. /api/product-catalog boundary check only asserted no '_seed' leak — which
is also true for the static fallback path. A broken cached reader
(getFromCache returning null or throwing) could serve fallback silently
and still pass this probe. Now:
- api/product-catalog.js emits X-Product-Catalog-Source: cache|dodo|fallback
on the response (the json() helper gained an optional source param wired
to each of the three branches).
- checkPublicBoundary declaratively requires that header's value match
'cache' for /api/product-catalog, so a fallback-serve fails the probe
with reason 'source:fallback!=cache' or 'source:missing!=cache'.
Test updates (tests/seed-contract-probe.test.mjs):
- Boundary check reworked to use a BOUNDARY_CHECKS config with optional
requireSourceHeader per endpoint.
- New cases: served-from-cache passes, served-from-fallback fails with source
mismatch, missing header fails, seed-leak still takes precedence, bad
status fails.
- Token-panels sanity test now asserts minRecords≥1 on all 3 panels.
Verification:
- tests/seed-contract-probe.test.mjs → 17/17 pass (was 15, +2 net)
- npm run test:data → 5340/5340
- npm run typecheck:all → clean
|
||
|
Elie Habib
|
e825f3450d |
fix(unrest): bump GDELT fetch timeout from 15s to 30s (#1920)
GDELT API is responding in 18-21s from local; Railway routing likely slower. The 15s timeout causes consecutive fetch failures despite GDELT being up. |
||
|
Elie Habib
|
608eb42e4b |
fix(seeds): extend Redis TTLs to 6x cron interval across 6 standalone seeds (#1912)
All had TTL ≤ 2x cron interval — any missed Railway run or API timeout causes the key to expire and the panel to show unavailable. Worst case: natural-events had TTL=1h < maxStaleMin=2h — panel dark 60 min before health even alarmed. Changes: seed-natural-events: 3600 → 43200 (12h, 6x 2h cron) + health maxStaleMin 120 → 360 seed-insights: 1800 → 10800 (3h, 6x 30min cron) seed-internet-outages: 1800 → 10800 (3h, 6x 30min cron) seed-earthquakes: 3600 → 21600 (6h, 6x 1h cron) seed-unrest-events: 3600 → 16200 (4.5h, 6x 45min cron) seed-forecasts: 6300 → 21600 (6h, 6x 1h cron) |
||
|
Elie Habib
|
4008f56254 |
fix: log fetch error cause in seed retry/FATAL handlers (#1638)
* test: rewrite transit chart test as structural contract verification Replace fragile source-string extraction + new Function() compilation with structural pattern checks on the source code. Tests verify: - render() clears chart before content change - clearTransitChart() cancels timer, disconnects observer, destroys chart - MutationObserver setup for DOM readiness detection - Fallback timer for no-op renders (100-500ms range) - Both callbacks (observer + timer) clean up each other - Tab switch and collapse clear chart state - Mount function guards against missing element/data Replaces PR #1634's approach which was brittle (method body extraction, TypeScript cast stripping, sandboxed execution). * fix: log fetch error cause in seed retry and FATAL handlers Node 20 fetch() throws TypeError('fetch failed') with the real error hidden in err.cause (DNS, TLS, timeout). The current logging only shows 'fetch failed' which is useless for diagnosis. Now logs err.cause.message in both withRetry() retries and FATAL catch blocks. |
||
|
RepairYourTech
|
0420a54866 |
fix(acled): add OAuth token manager with automatic refresh (#1437)
* fix(acled): add OAuth token manager with automatic refresh ACLED access tokens expire every 24 hours, but WorldMonitor stores a static ACLED_ACCESS_TOKEN with no refresh logic — causing all ACLED API calls to fail after the first day. This commit adds `acled-auth.ts`, an OAuth token manager that: - Exchanges ACLED_EMAIL + ACLED_PASSWORD for an access token (24h) and refresh token (14d) via the official ACLED OAuth endpoint - Caches tokens in memory and auto-refreshes before expiry - Falls back to static ACLED_ACCESS_TOKEN for backward compatibility - Deduplicates concurrent refresh attempts - Degrades gracefully when no credentials are configured The only change to the existing `acled.ts` is replacing the synchronous `process.env.ACLED_ACCESS_TOKEN` read with an async call to the new `getAcledAccessToken()` helper. Fixes #1283 Relates to #290 * fix: address review feedback on ACLED OAuth PR - Use Redis (Upstash) as L2 token cache to survive Vercel Edge cold starts (in-memory cache retained as fast-path L1) - Add CHROME_UA User-Agent header on OAuth token exchange and refresh - Update seed script to use OAuth flow via getAcledToken() helper instead of raw process.env.ACLED_ACCESS_TOKEN - Add security comment to .env.example about plaintext password trade-offs - Sidecar ACLED_ACCESS_TOKEN case is a validation probe (tests user-provided value, not process.env) — data fetching delegates to handler modules * feat(sidecar): add ACLED_EMAIL/ACLED_PASSWORD to env allowlist and validation - Add ACLED_EMAIL and ACLED_PASSWORD to ALLOWED_ENV_KEYS set - Add ACLED_EMAIL validation case (store-only, verified with password) - Add ACLED_PASSWORD validation case with OAuth token exchange via acleddata.com/api/acled/user/login - On successful login, store obtained OAuth token in ACLED_ACCESS_TOKEN - Follows existing validation patterns (Cloudflare challenge handling, auth failure detection, User-Agent header) * fix: address remaining review feedback (duplicate OAuth, em dashes, emoji) - Extract shared ACLED OAuth helper into scripts/shared/acled-oauth.mjs - Remove ~55 lines of duplicate OAuth logic from seed-unrest-events.mjs, now imports getAcledToken from the shared helper - Replace em dashes with ASCII dashes in acled-auth.ts section comments - Replace em dash with parentheses in sidecar validation message - Remove emoji from .env.example security note Addresses koala73's second review: MEDIUM (duplicate OAuth), LOW (em dashes), LOW (emoji). * fix: align sidecar OAuth endpoint, fix L1/L2 cache, cleanup artifacts - Sidecar: switch from /api/acled/user/login (JSON) to /oauth/token (URL-encoded) to match server/_shared/acled-auth.ts exactly - acled-auth.ts: check L2 Redis when L1 is expired, not only when L1 is null (fixes stale L1 skipping fresher L2 from another isolate) - acled-oauth.mjs: remove stray backslash on line 9 - seed-unrest-events.mjs: remove extra blank line at line 13 --------- Co-authored-by: Elie Habib <elie.habib@gmail.com> Co-authored-by: RepairYourTech <30200484+RepairYourTech@users.noreply.github.com> |
||
|
Elie Habib
|
5e25bb1386 |
fix(health): resolve all critical health check failures (#1111)
## Summary - Reclassify 10 on-demand keys (BIS, supply chain, theater posture, etc.) from BOOTSTRAP → STANDALONE + ON_DEMAND to stop false CRITs - Fix seed-insights Railway OOM by correcting service-level settings - Unify LLM fallback chain (Groq → OpenRouter → Ollama) in seed-insights - Switch OpenRouter model to `openai/gpt-oss-safeguard-20b:nitro` - Fix GDELT v2/geo → v1/gkg_geojson for unrestEvents and positiveGeoEvents (v2 endpoint is dead) - Add seed-meta writes for marketQuotes/commodityQuotes in AIS relay (zero extra Yahoo calls) - Remove aggressive coord filter in cyber threats that dropped all threats when GeoIP rate-limited ## Health impact - 6 false CRITs → eliminated (reclassified as on-demand) - marketQuotes/commodityQuotes STALE_SEED → OK (seed-meta tracking) - unrestEvents EMPTY_DATA → OK (GDELT v1 fix) - positiveGeoEvents EMPTY_DATA → OK (GDELT v1 fix in relay) - cyberThreats resilience improved (coord filter removal) |
||
|
Elie Habib
|
f06db59720 |
fix: graceful degradation for seed scripts with missing keys or downed sources (#1045)
- seed-unrest-events: relax validation (ACLED missing + GDELT 404 = crash), console.warn → console.log for non-fatal failures - seed-natural-events: relax validation, console.error → console.log - seed-climate-anomalies: relax validation, console.error → console.log - seed-internet-outages: console.error → console.log for missing key Railway tags console.warn/error as severity:error, making healthy runs look like crashes in the dashboard. |
||
|
Elie Habib
|
78a14306d9 |
feat: add seed-first pattern to 15 RPC handlers with Railway seed scripts (#989)
Migrate handlers from direct external API calls to seed-first pattern: Railway cron seeds Redis → handlers read from Redis → fallback to live fetch if seed stale and SEED_FALLBACK_* env enabled. Handlers updated: earthquakes, fire-detections, internet-outages, climate-anomalies, unrest-events, cyber-threats, market-quotes, commodity-quotes, crypto-quotes, etf-flows, gulf-quotes, stablecoin-markets, natural-events, displacement-summary, risk-scores. Also adds: - scripts/_seed-utils.mjs (shared seed framework with atomic publish, distributed locks, retry, freshness metadata) - 13 seed scripts for Railway cron - api/seed-health.js monitoring endpoint - scripts/validate-seed-migration.mjs post-deploy validation - Restored multi-source CII in get-risk-scores (8 sources: ACLED, UCDP, outages, climate, cyber, fires, GPS, Iran) |