mirror of
https://github.com/koala73/worldmonitor.git
synced 2026-04-25 17:14:57 +02:00
58e42aadf951d8ea35403cbb60a6cb1792713d61
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sebastien Melki
|
58e42aadf9 |
chore(api): enforce sebuf contract + migrate drifting endpoints (#3207) (#3242)
* chore(api): enforce sebuf contract via exceptions manifest (#3207) Adds api/api-route-exceptions.json as the single source of truth for non-proto /api/ endpoints, with scripts/enforce-sebuf-api-contract.mjs gating every PR via npm run lint:api-contract. Fixes the root-only blind spot in the prior allowlist (tests/edge-functions.test.mjs), which only scanned top-level *.js files and missed nested paths and .ts endpoints — the gap that let api/supply-chain/v1/country-products.ts and friends drift under proto domain URL prefixes unchallenged. Checks both directions: every api/<domain>/v<N>/[rpc].ts must pair with a generated service_server.ts (so a deleted proto fails CI), and every generated service must have an HTTP gateway (no orphaned generated code). Manifest entries require category + reason + owner, with removal_issue mandatory for temporary categories (deferred, migration-pending) and forbidden for permanent ones. .github/CODEOWNERS pins the manifest to @SebastienMelki so new exceptions don't slip through review. The manifest only shrinks: migration-pending entries (19 today) will be removed as subsequent commits in this PR land each migration. * refactor(maritime): migrate /api/ais-snapshot → maritime/v1.GetVesselSnapshot (#3207) The proto VesselSnapshot was carrying density + disruptions but the frontend also needed sequence, relay status, and candidate_reports to drive the position-callback system. Those only lived on the raw relay passthrough, so the client had to keep hitting /api/ais-snapshot whenever callbacks were registered and fall back to the proto RPC only when the relay URL was gone. This commit pushes all three missing fields through the proto contract and collapses the dual-fetch-path into one proto client call. Proto changes (proto/worldmonitor/maritime/v1/): - VesselSnapshot gains sequence, status, candidate_reports. - GetVesselSnapshotRequest gains include_candidates (query: include_candidates). Handler (server/worldmonitor/maritime/v1/get-vessel-snapshot.ts): - Forwards include_candidates to ?candidates=... on the relay. - Separate 5-min in-memory caches for the candidates=on and candidates=off variants; they have very different payload sizes and should not share a slot. - Per-request in-flight dedup preserved per-variant. Frontend (src/services/maritime/index.ts): - fetchSnapshotPayload now calls MaritimeServiceClient.getVesselSnapshot directly with includeCandidates threaded through. The raw-relay path, SNAPSHOT_PROXY_URL, DIRECT_RAILWAY_SNAPSHOT_URL and LOCAL_SNAPSHOT_FALLBACK are gone — production already routed via Vercel, the "direct" branch only ever fired on localhost, and the proto gateway covers both. - New toLegacyCandidateReport helper mirrors toDensityZone/toDisruptionEvent. api/ais-snapshot.js deleted; manifest entry removed. Only reduced the codegen scope to worldmonitor.maritime.v1 (buf generate --path) — regenerating the full tree drops // @ts-nocheck from every client/server file and surfaces pre-existing type errors across 30+ unrelated services, which is not in scope for this PR. Shape-diff vs legacy payload: - disruptions / density: proto carries the same fields, just with the GeoCoordinates wrapper and enum strings (remapped client-side via existing toDisruptionEvent / toDensityZone helpers). - sequence, status.{connected,vessels,messages}: now populated from the proto response — was hardcoded to 0/false in the prior proto fallback. - candidateReports: same shape; optional numeric fields come through as 0 instead of undefined, which the legacy consumer already handled. * refactor(sanctions): migrate /api/sanctions-entity-search → LookupSanctionEntity (#3207) The proto docstring already claimed "OFAC + OpenSanctions" coverage but the handler only fuzzy-matched a local OFAC Redis index — narrower than the legacy /api/sanctions-entity-search, which proxied OpenSanctions live (the source advertised in docs/api-proxies.mdx). Deleting the legacy without expanding the handler would have been a silent coverage regression for external consumers. Handler changes (server/worldmonitor/sanctions/v1/lookup-entity.ts): - Primary path: live search against api.opensanctions.org/search/default with an 8s timeout and the same User-Agent the legacy edge fn used. - Fallback path: the existing OFAC local fuzzy match, kept intact for when OpenSanctions is unreachable / rate-limiting. - Response source field flips between 'opensanctions' (happy path) and 'ofac' (fallback) so clients can tell which index answered. - Query validation tightened: rejects q > 200 chars (matches legacy cap). Rate limiting: - Added /api/sanctions/v1/lookup-entity to ENDPOINT_RATE_POLICIES at 30/min per IP — matches the legacy createIpRateLimiter budget. The gateway already enforces per-endpoint policies via checkEndpointRateLimit. Docs: - docs/api-proxies.mdx — dropped the /api/sanctions-entity-search row (plus the orphaned /api/ais-snapshot row left over from the previous commit in this PR). - docs/panels/sanctions-pressure.mdx — points at the new RPC URL and describes the OpenSanctions-primary / OFAC-fallback semantics. api/sanctions-entity-search.js deleted; manifest entry removed. * refactor(military): migrate /api/military-flights → ListMilitaryFlights (#3207) Legacy /api/military-flights read a pre-baked Redis blob written by the seed-military-flights cron and returned flights in a flat app-friendly shape (lat/lon, lowercase enums, lastSeenMs). The proto RPC takes a bbox, fetches OpenSky live, classifies server-side, and returns nested GeoCoordinates + MILITARY_*_TYPE_* enum strings + lastSeenAt — same data, different contract. fetchFromRedis in src/services/military-flights.ts was doing nothing sebuf-aware. Renamed it to fetchViaProto and rewrote to: - Instantiate MilitaryServiceClient against getRpcBaseUrl(). - Iterate MILITARY_QUERY_REGIONS (PACIFIC + WESTERN) in parallel — same regions the desktop OpenSky path and the seed cron already use, so dashboard coverage tracks the analytic pipeline. - Dedup by hexCode across regions. - Map proto → app shape via new mapProtoFlight helper plus three reverse enum maps (AIRCRAFT_TYPE_REVERSE, OPERATOR_REVERSE, CONFIDENCE_REVERSE). The seed cron (scripts/seed-military-flights.mjs) stays put: it feeds regional-snapshot mobility, cross-source signals, correlation, and the health freshness check (api/health.js: 'military:flights:v1'). None of those read the legacy HTTP endpoint; they read the Redis key directly. The proto handler uses its own per-bbox cache keys under the same prefix, so dashboard traffic no longer races the seed cron's blob — the two paths diverge by a small refresh lag, which is acceptable. Docs: dropped the /api/military-flights row from docs/api-proxies.mdx. api/military-flights.js deleted; manifest entry removed. Shape-diff vs legacy: - f.location.{latitude,longitude} → f.lat, f.lon - f.aircraftType: MILITARY_AIRCRAFT_TYPE_TANKER → 'tanker' via reverse map - f.operator: MILITARY_OPERATOR_USAF → 'usaf' via reverse map - f.confidence: MILITARY_CONFIDENCE_LOW → 'low' via reverse map - f.lastSeenAt (number) → f.lastSeen (Date) - f.enrichment → f.enriched (with field renames) - Extra fields registration / aircraftModel / origin / destination / firstSeenAt now flow through where proto populates them. * fix(supply-chain): thread includeCandidates through chokepoint status (#3207) Caught by tsconfig.api.json typecheck in the pre-push hook (not covered by the plain tsc --noEmit run that ran before I pushed the ais-snapshot commit). The chokepoint status handler calls getVesselSnapshot internally with a static no-auth request — now required to include the new includeCandidates bool from the proto extension. Passing false: server-internal callers don't need per-vessel reports. * test(maritime): update getVesselSnapshot cache assertions (#3207) The ais-snapshot migration replaced the single cachedSnapshot/cacheTimestamp pair with a per-variant cache so candidates-on and candidates-off payloads don't evict each other. Pre-push hook surfaced that tests/server-handlers still asserted the old variable names. Rewriting the assertions to match the new shape while preserving the invariants they actually guard: - Freshness check against slot TTL. - Cache read before relay call. - Per-slot in-flight dedup. - Stale-serve on relay failure (result ?? slot.snapshot). * chore(proto): restore // @ts-nocheck on regenerated maritime files (#3207) I ran 'buf generate --path worldmonitor/maritime/v1' to scope the proto regen to the one service I was changing (to avoid the toolchain drift that drops @ts-nocheck from 60+ unrelated files — separate issue). But the repo convention is the 'make generate' target, which runs buf and then sed-prepends '// @ts-nocheck' to every generated .ts file. My scoped command skipped the sed step. The proto-check CI enforces the sed output, so the two maritime files need the directive restored. * refactor(enrichment): decomm /api/enrichment/{company,signals} legacy edge fns (#3207) Both endpoints were already ported to IntelligenceService: - getCompanyEnrichment (/api/intelligence/v1/get-company-enrichment) - listCompanySignals (/api/intelligence/v1/list-company-signals) No frontend callers of the legacy /api/enrichment/* paths exist. Removes: - api/enrichment/company.js, signals.js, _domain.js - api-route-exceptions.json migration-pending entries (58 remain) - docs/api-proxies.mdx rows for /api/enrichment/{company,signals} - docs/architecture.mdx reference updated to the IntelligenceService RPCs Verified: typecheck, typecheck:api, lint:api-contract (89 files / 58 entries), lint:boundaries, tests/edge-functions.test.mjs (136 pass), tests/enrichment-caching.test.mjs (14 pass — still guards the intelligence/v1 handlers), make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(leads): migrate /api/{contact,register-interest} → LeadsService (#3207) New leads/v1 sebuf service with two POST RPCs: - SubmitContact → /api/leads/v1/submit-contact - RegisterInterest → /api/leads/v1/register-interest Handler logic ported 1:1 from api/contact.js + api/register-interest.js: - Turnstile verification (desktop sources bypass, preserved) - Honeypot (website field) silently accepts without upstream calls - Free-email-domain gate on SubmitContact (422 ApiError) - validateEmail (disposable/offensive/typo-TLD/MX) on RegisterInterest - Convex writes via ConvexHttpClient (contactMessages:submit, registerInterest:register) - Resend notification + confirmation emails (HTML templates unchanged) Shared helpers moved to server/_shared/: - turnstile.ts (getClientIp + verifyTurnstile) - email-validation.ts (disposable/offensive/MX checks) Rate limits preserved via ENDPOINT_RATE_POLICIES: - submit-contact: 3/hour per IP (was in-memory 3/hr) - register-interest: 5/hour per IP (was in-memory 5/hr; desktop sources previously capped at 2/hr via shared in-memory map — now 5/hr like everyone else, accepting the small regression in exchange for Upstash-backed global limiting) Callers updated: - pro-test/src/App.tsx contact form → new submit-contact path - src-tauri/sidecar/local-api-server.mjs cloud-fallback rewrites /api/register-interest → /api/leads/v1/register-interest when proxying; keeps local path for older desktop builds - src/services/runtime.ts isKeyFreeApiTarget allows both old and new paths through the WORLDMONITOR_API_KEY-optional gate Tests: - tests/contact-handler.test.mjs rewritten to call submitContact handler directly; asserts on ValidationError / ApiError - tests/email-validation.test.mjs + tests/turnstile.test.mjs point at the new server/_shared/ modules Deleted: api/contact.js, api/register-interest.js, api/_ip-rate-limit.js, api/_turnstile.js, api/_email-validation.js, api/_turnstile.test.mjs. Manifest entries removed (58 → 56). Docs updated (api-platform, api-commerce, usage-rate-limits). Verified: npm run typecheck + typecheck:api + lint:api-contract (88 files / 56 entries) + lint:boundaries pass; full test:data (5852 tests) passes; make generate is zero-diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(pro-test): rebuild bundle for leads/v1 contact form (#3207) Updates the enterprise contact form to POST to /api/leads/v1/submit-contact (old path /api/contact removed in the previous commit). Bundle is rebuilt from pro-test/src/App.tsx source change in |
||
|
Elie Habib
|
56a792bbc4 |
docs(marketing): bump source-count claims from 435+ to 500+ (#3241)
Feeds.ts is at 523 entries after PR #3236 landed. The "435+" figure has been baked into marketing copy, docs, press kit, and localized strings for a long time and is now noticeably understated. Bump to 500+ as the new canonical figure. Also aligned three stale claims in less-visited docs: docs/getting-started.mdx 70+ RSS feeds => 500+ RSS feeds docs/ai-intelligence.mdx 344 sources => 500+ sources docs/COMMUNITY-PROMOTION-GUIDE 170+ news feeds => 500+ news feeds 170+ news sources => 500+ news sources And bumped the digest-dedup copy 400+ to 500+ (English + French locales + pro-test/index.html prerendered body) for consistency with the pricing and GDELT panels. Left alone on purpose (different metric): 22 services / 22 service domains 24 feeds (security-advisory seeder specifically) 31 sources (freshness tracker) 45 map layers Rebuilt /pro bundle so the per-locale chunks + prerendered index.html under public/pro/assets ship the new copy. 20 locales updated. |
||
|
Elie Habib
|
ced3e7058b |
perf(aviation): halve seed cadence to 30min + extend Redis TTLs (#3073)
* perf(aviation): halve seed cadence to 30min + extend Redis TTLs AviationStack upstream call volume has been climbing over the past few days. Halving the seed-aviation Railway cron from every 15 min to every 30 min cuts this seeder's contribution from ~576/day to ~288/day. With the longer interval, the Redis TTLs must span the new cron gap with buffer — otherwise keys expire between runs and the panel goes empty. Bumping OPS_TTL and NEWS_TTL from 300/900s to 2100s (35 min) gives a 5-minute buffer over the 30-min interval. The Railway cron schedule itself is changed on the dashboard (service a8e49386-64c1-4e1e-9f82-4eb69a55fce3), out of band from this PR. * perf(aviation): widen TTL buffer to 10min per review Greptile review flagged the 5-min buffer on 2100s TTL as tight. Worst case: serial AviationStack calls for all default airports can take ~63s, and the Railway cron can fire slightly late — a run that starts 29 min after the previous could write a key only seconds before the old one expires, risking a brief empty-panel window. Bump OPS_TTL and NEWS_TTL from 2100s to 2400s (40 min) to give a 10-minute buffer over the new 30-min cron interval. |
||
|
Fayez Bast
|
bb4f8dcb12 |
feat(climate): add WMO normals seeding and CO2 monitoring (#2531)
* feat(climate): add WMO normals seeding and CO2 monitoring * fix(climate): skip missing normals per-zone and align anomaly tooltip copy * fix(climate): remove normals from bootstrap and harden health/cache key wiring * feat(climate): version anomaly cache to v2, harden seed freshness, and align CO2/normal baselines |
||
|
Elie Habib
|
b7363b73c2 |
fix(health+economic-panel): unrestEvents threshold + FRED state render bug (#2074)
* fix(health): raise unrestEvents maxStaleMin 75→120, fix cron doc Actual Railway cron for seed-unrest-events is 45min (per seed script comment) not 15min (per stale architecture.mdx). With a 45min cron, the 75min threshold only gives 30min grace — any brief Railway hiccup triggered a false STALE_SEED WARN. 120min = 2h grace (2.67x cron). * fix(economic-panel): prevent render() from overwriting FRED error state When updateSpending() or updateBis() fires after loadFredData() shows an error, render() was calling setContent() which replaced the error UI with the noIndicatorData empty message. Fix: track FRED load state (loading/ok/error/retrying) inside EconomicPanel. renderIndicators() now uses this state to show the correct message. Replace showError()/showRetrying() calls in loadFredData() with setFredError()/setFredRetrying() which store state before re-rendering. Also fix circuit breaker name mismatch: loadFredData() was checking 'FRED Economic' but the breaker is named 'FRED Batch'. |
||
|
Elie Habib
|
e5db9a0700 |
docs: update Mintlify docs for recent features (#1693)
- finance-data.mdx: Add US Treasury customs revenue to Trade Policy section (Revenue tab, free API, FYTD comparison, desktop support) - features.mdx: Expand CMD+K section from 5 categories to full 55 panel coverage with organized subcategories - data-sources.mdx: Update advisory section for gold standard migration (Railway seed, 265-entry country map, relay proxy) - architecture.mdx: Add 12 missing seed scripts to Railway pipeline table (forecasts, conflict-intel, economy, supply-chain-trade, security-advisories, usni-fleet, gdelt-intel, research, etc.) - changelog.mdx + CHANGELOG.md: Add unreleased entries for customs revenue, advisory migration, CMD+K coverage, R2 traces, and 6 fixes |
||
|
Elie Habib
|
fe67111dc9 |
feat: harness engineering P0 - linting, testing, architecture docs (#1587)
* feat: harness engineering P0 - linting, testing, architecture docs
Add foundational infrastructure for agent-first development:
- AGENTS.md: agent entry point with progressive disclosure to deeper docs
- ARCHITECTURE.md: 12-section system reference with source-file refs and ownership rule
- Biome 2.4.7 linter with project-tuned rules, CI workflow (lint-code.yml)
- Architectural boundary lint enforcing forward-only dependency direction (lint-boundaries.mjs)
- Unit test CI workflow (test.yml), all 1083 tests passing
- Fixed 9 pre-existing test failures (bootstrap sync, deploy-config headers, globe parity, redis mocks, geometry URL, import.meta.env null safety)
- Fixed 12 architectural boundary violations (types moved to proper layers)
- Added 3 missing cache tier entries in gateway.ts
- Synced cache-keys.ts with bootstrap.js
- Renamed docs/architecture.mdx to "Design Philosophy" with cross-references
- Deprecated legacy docs/Docs_To_Review/ARCHITECTURE.md
- Harness engineering roadmap tracking doc
* fix: address PR review feedback on harness-engineering-p0
- countries-geojson.test.mjs: skip gracefully when CDN unreachable
instead of failing CI on network issues
- country-geometry-overrides.test.mts: relax timing assertion
(250ms -> 2000ms) for constrained CI environments
- lint-boundaries.mjs: implement the documented api/ boundary check
(was documented but missing, causing false green)
* fix(lint): scan api/ .ts files in boundary check
The api/ boundary check only scanned .js/.mjs files, missing the 25
sebuf RPC .ts edge functions. Now scans .ts files with correct rules:
- Legacy .js: fully self-contained (no server/ or src/ imports)
- RPC .ts: may import server/ and src/generated/ (bundled at deploy),
but blocks imports from src/ application code
* fix(lint): detect import() type expressions in boundary lint
- Move AppContext back to app/app-context.ts (aggregate type that
references components/services/utils belongs at the top, not types/)
- Move HappyContentCategory and TechHQ to types/ (simple enums/interfaces)
- Boundary lint now catches import('@/layer') expressions, not just
from '@/layer' imports
- correlation-engine imports of AppContext marked boundary-ignore
(type-only imports of top-level aggregate)
|
||
|
Elie Habib
|
09a582e089 |
fix(docs): correct numerical inaccuracies, add missing features and variants (#1472)
* fix(docs): correct variant count from 4 to 5 Updated overview.mdx and architecture.mdx to reflect all 5 platform variants (World Monitor, Tech Monitor, Happy Monitor, Finance Monitor, Commodity Monitor). Previously only 2 were listed in the table and the text said "four". * fix(docs): correct all numerical inaccuracies across documentation Updated counts verified against actual codebase: - AI datacenters: 111 → 313 - Undersea cables: 55 → 86 - Map layers: 45 → 49 - News sources: 80+ → 344 - Service domains: 22 → 24 - CII countries: 20 → 24 - Military bases: 210/220+ → 226 - Strategic ports: 61/83 → 62 - Airports: 30/107 → 111 - Chokepoints: 6/8 → 9 - Signal entities: 100+/600+ → 66 - Variant count: four → five (added Commodity Monitor) * docs(overview): add Happy, Finance, and Commodity Monitor sections Added detailed documentation sections for the three previously undocumented platform variants, including layers, panels, and news categories for each. * docs(features): document 13 previously undocumented features Added: trade routes, FIRMS fire detection, webcam surveillance, country brief, aviation intelligence panel, climate anomalies, displacement tracking, Gulf economies, WTO trade policy, central banks & BIS, market watchlist, NOTAM closure detection, and offline ML capabilities. * docs: standardize terminology, add cross-references, fix stale refs Phase 4: Renamed "News Importance Scoring" to "Headline Scoring", "Signal Correlation" to "Cross-Stream Correlation". Added cross-refs between CII/convergence, CORS/API-key, maritime/finance chokepoints. Deduplicated Population Exposure content. Clarified hotspot vs focal point distinction. Phase 5: Rewrote daily_stock_analysis as historical context. Added legacy note for api/*.js files. Added OREF 24h rolling history boost and GPS jamming classification thresholds to algorithms.mdx. Fixed orbital-surveillance tier table contradictions. Phase 6: Fixed orphaned markdown separator in orbital-surveillance. * fix(docs): catch remaining stale numbers in secondary doc files Fixed stale counts in data-sources.mdx, strategic-risk.mdx, documentation.mdx, ai-intelligence.mdx, PRESS_KIT.md, and roadmap-pro.md that were missed in the initial pass. * fix(docs): address review findings from fresh-eyes pass - desktop-app.mdx: "four variants" → "five", "22 services" → "24" - infrastructure-cascade.mdx: chokepoints 8 → 9, node total 279 → 350 - data-sources.mdx: chokepoints 8 → 9 - overview.mdx: remove duplicated intro sentence - getting-started.mdx: fix stale file tree comments (AI clusters, airports, entities) |
||
|
Elie Habib
|
1f38dea225 |
docs: restructure documentation into focused, navigable pages (#1465)
* docs: restructure documentation into focused, navigable pages (#docs-reorg) Break the 4096-line documentation.mdx monolith into 13 focused pages organized by feature area. Reorganize Mintlify navigation from 5 generic groups into 7 feature-based groups. Move Orbital Surveillance from Infrastructure to Map Layers where it belongs. - Extract: signal-intelligence, features, overview, hotspots, CII, geographic-convergence, strategic-risk, infrastructure-cascade, military-tracking, maritime-intelligence, natural-disasters, contributing, getting-started - Append to: architecture.mdx (9 sections), ai-intelligence.mdx (3 sections) - Fix legacy .md links in map-engine.mdx, maps-and-geocoding.mdx - Slim documentation.mdx to an 80-line index/hub page - Eliminate duplicate content that caused repeated headings * fix(docs): remove duplicate H1 headings from all Mintlify pages Mintlify auto-renders the frontmatter `title` as an H1, so having `# Title` in the body creates a doubled heading on every page. Remove the redundant H1 (and repeated description lines) from all 31 .mdx files. |
||
|
Elie Habib
|
60fa0c5907 |
fix(docs): escape angle brackets for MDX compatibility (#1452)
MDX interprets bare <digit patterns as JSX tags. Escape with < in architecture.mdx, data-sources.mdx, and documentation.mdx. |
||
|
Elie Habib
|
caa3e9f82f |
fix(docs): rename doc files to lowercase for Mintlify (#1451)
* fix(docs): rename doc files to lowercase kebab-case for Mintlify Mintlify serves pages at lowercase URLs. Uppercase filenames caused 404s on the Documentation tab. Renames all 18 doc files, updates docs.json references, and fixes internal cross-links. * fix(docs): rename .md to .mdx for Mintlify compatibility Mintlify expects .mdx files. Plain .md files were not being found, causing 404s on all documentation pages. * feat(docs): add navbar links and footer with site variants - Navbar: Live App, Tech, Finance, Blog links + GitHub CTA - Footer: World Monitor variants + Resources columns - Logo links back to worldmonitor.app |