Files
worldmonitor/scripts/seed-aviation.mjs
Elie Habib 34dfc9a451 fix(news): ground LLM surfaces on real RSS description end-to-end (#3370)
* feat(news/parser): extract RSS/Atom description for LLM grounding (U1)

Add description field to ParsedItem, extract from the first non-empty of
description/content:encoded (RSS) or summary/content (Atom), picking the
longest after HTML-strip + entity-decode + whitespace-normalize. Clip to
400 chars. Reject empty, <40 chars after strip, or normalize-equal to the
headline — downstream consumers fall back to the cleaned headline on '',
preserving current behavior for feeds without a description.

CDATA end is anchored to the closing tag so internal ]]> sequences do not
truncate the match. Preserves cached rss:feed:v1 row compatibility during
the 1h TTL bleed since the field is additive.

Part of fix: pipe RSS description end-to-end so LLM surfaces stop
hallucinating named actors (docs/plans/2026-04-24-001-...).

Covers R1, R7.

* feat(news/story-track): persist description on story:track:v1 HSET (U2)

Append description to the story:track:v1 HSET only when non-empty. Additive
— no key version bump. Old rows and rows from feeds without a description
return undefined on HGETALL, letting downstream readers fall back to the
cleaned headline (R6).

Extract buildStoryTrackHsetFields as a pure helper so the inclusion gate is
unit-testable without Redis.

Update the contract comment in cache-keys.ts so the next reader of the
schema sees description as an optional field.

Covers R2, R6.

* feat(proto): NewsItem.snippet + SummarizeArticleRequest.bodies (U3)

Add two additive proto fields so the article description can ride to every
LLM-adjacent consumer without a breaking change:

- NewsItem.snippet (field 12): RSS/Atom description, HTML-stripped,
  ≤400 chars, empty when unavailable. Wired on toProtoItem.
- SummarizeArticleRequest.bodies (field 8): optional article bodies
  paired 1:1 with headlines for prompt grounding. Empty array is today's
  headline-only behavior.

Regenerated TS client/server stubs and OpenAPI YAML/JSON via sebuf v0.11.1
(PATH=~/go/bin required — Homebrew's protoc-gen-openapiv3 is an older
pre-bundle-mode build that collides on duplicate emission).

Pre-emptive bodies:[] placeholders at the two existing SummarizeArticle
call sites in src/services/summarization.ts; U6 replaces them with real
article bodies once SummarizeArticle handler reads the field.

Covers R3, R5.

* feat(brief/digest): forward RSS description end-to-end through brief envelope (U4)

Digest accumulator reader (seed-digest-notifications.mjs::buildDigest) now
plumbs the optional `description` field off each story:track:v1 HGETALL into
the digest story object. The brief adapter (brief-compose.mjs::
digestStoryToUpstreamTopStory) prefers the real RSS description over the
cleaned headline; when the upstream row has no description (old rows in the
48h bleed, feeds that don't carry one), we fall back to the cleaned headline
so today behavior is preserved (R6).

This is the upstream half of the description cache path. U5 lands the LLM-
side grounding + cache-prefix bump so Gemini actually sees the article body
instead of hallucinating a named actor from the headline.

Covers R4 (upstream half), R6.

* feat(brief/llm): RSS grounding + sanitisation + 4 cache prefix bumps (U5)

The actual fix for the headline-only named-actor hallucination class:
Gemini 2.5 Flash now receives the real article body as grounding context,
so it paraphrases what the article says instead of filling role-label
headlines from parametric priors ("Iran's new supreme leader" → "Ali
Khamenei" was the 2026-04-24 reproduction; with grounding, it becomes
the actual article-named actor).

Changes:

- buildStoryDescriptionPrompt interpolates a `Context: <body>` line
  between the metadata block and the "One editorial sentence" instruction
  when description is non-empty AND not normalise-equal to the headline.
  Clips to 400 chars as a second belt-and-braces after the U1 parser cap.
  No Context line → identical prompt to pre-fix (R6 preserved).

- sanitizeStoryForPrompt extended to cover `description`. Closes the
  asymmetry where whyMatters was sanitised and description wasn't —
  untrusted RSS bodies now flow through the same injection-marker
  neutraliser before prompt interpolation. generateStoryDescription wraps
  the story in sanitizeStoryForPrompt before calling the builder,
  matching generateWhyMatters.

- Four cache prefixes bumped atomically to evict pre-grounding rows:
    scripts/lib/brief-llm.mjs:
      brief:llm:description:v1 → v2  (Railway, description path)
      brief:llm:whymatters:v2 → v3   (Railway, whyMatters fallback)
    api/internal/brief-why-matters.ts:
      brief:llm:whymatters:v6 → v7                (edge, primary)
      brief:llm:whymatters:shadow:v4 → shadow:v5  (edge, shadow)
  hashBriefStory already includes description in the 6-field material
  (v5 contract) so identity naturally drifts; the prefix bump is the
  belt-and-braces that guarantees a clean cold-start on first tick.

- Tests: 8 new + 2 prefix-match updates on tests/brief-llm.test.mjs.
  Covers Context-line injection, empty/dup-of-headline rejection,
  400-char clip, sanitisation of adversarial descriptions, v2 write,
  and legacy-v1 row dark (forced cold-start).

Covers R4 + new sanitisation requirement.

* feat(news/summarize): accept bodies + bump summary cache v5→v6 (U6)

SummarizeArticle now grounds on per-headline article bodies when callers
supply them, so the dashboard "News summary" path stops hallucinating
across unrelated headlines when the upstream RSS carried context.

Three coordinated changes:

1. SummarizeArticleRequest handler reads req.bodies, sanitises each entry
   through sanitizeForPrompt (same trust treatment as geoContext — bodies
   are untrusted RSS text), clips to 400 chars, and pads to the headlines
   length so pair-wise identity is stable.

2. buildArticlePrompts accepts optional bodies and interleaves a
   `    Context: <body>` line under each numbered headline that has a
   non-empty body. Skipped in translate mode (headline[0]-only) and when
   all bodies are empty — yielding a byte-identical prompt to pre-U6
   for every current caller (R6 preserved).

3. summary-cache-key bumps CACHE_VERSION v5→v6 so the pre-grounding rows
   (produced from headline-only prompts) cold-start cleanly. Extends
   canonicalizeSummaryInputs + buildSummaryCacheKey with a pair-wise
   bodies segment `:bd<hash>`; the prefix is `:bd` rather than `:b` to
   avoid colliding with `:brief:` when pattern-matching keys. Translate
   mode is headline[0]-only and intentionally does not shift on bodies.

Dedup reorder preserved: the handler re-pairs bodies to the deduplicated
top-5 via findIndex, so layout matches without breaking cache identity.

New tests: 7 on buildArticlePrompts (bodies interleave, partial fill,
translate-mode skip, clip, short-array tolerance), 8 on
buildSummaryCacheKey (pair-wise sort, cache-bust on body drift, translate
skip). Existing summary-cache-key assertions updated v5→v6.

Covers R3, R4.

* feat(consumers): surface RSS snippet across dashboard, email, relay, MCP + audit (U7)

Thread the RSS description from the ingestion path (U1-U5) into every
user-facing LLM-adjacent surface. Audit the notification producers so
RSS-origin and domain-origin events stay on distinct contracts.

Dashboard (proto snippet → client → panel):
- src/types/index.ts NewsItem.snippet?:string (client-side field).
- src/app/data-loader.ts proto→client mapper propagates p.snippet.
- src/components/NewsPanel.ts renders snippet as a truncated (~200 chars,
  word-boundary ellipsis) `.item-snippet` line under each headline.
- NewsPanel.currentBodies tracks per-headline bodies paired 1:1 with
  currentHeadlines; passed as options.bodies to generateSummary so the
  server-side SummarizeArticle LLM grounds on the article body.

Summary plumbing:
- src/services/summarization.ts threads bodies through SummarizeOptions
  → generateSummary → runApiChain → tryApiProvider; cache key now includes
  bodies (via U6's buildSummaryCacheKey signature).

MCP world-brief:
- api/mcp.ts pairs headlines with their RSS snippets and POSTs `bodies`
  to /api/news/v1/summarize-article so the MCP tool surface is no longer
  starved.

Email digest:
- scripts/seed-digest-notifications.mjs plain-text formatDigest appends
  a ~200-char truncated snippet line under each story; HTML formatDigestHtml
  renders a dim-grey description div between title and meta. Both gated
  on non-empty description (R6 — empty → today's behavior).

Real-time alerts:
- src/services/breaking-news-alerts.ts BreakingAlert gains optional
  description; checkBatchForBreakingAlerts reads item.snippet; dispatchAlert
  includes `description` in the /api/notify payload when present.

Notification relay:
- scripts/notification-relay.cjs formatMessage gated on
  NOTIFY_RELAY_INCLUDE_SNIPPET=1 (default off). When on, RSS-origin
  payloads render a `> <snippet>` context line under the title. When off
  or payload.description absent, output is byte-identical to pre-U7.

Audit (RSS vs domain):
- tests/notification-relay-payload-audit.test.mjs enforces file-level
  @notification-source tags on every producer, rejects `description:` in
  domain-origin payload blocks, and verifies the relay codepath gates
  snippet rendering under the flag.
- Tag added to ais-relay.cjs (domain), seed-aviation.mjs (domain),
  alert-emitter.mjs (domain), breaking-news-alerts.ts (rss).

Deferred (plan explicitly flags): InsightsPanel + cluster-producer
plumbing (bodies default to [] — will unlock gradually once news:insights:v1
producer also carries primarySnippet).

Covers R5, R6.

* docs+test: grounding-path note + bump pinned CACHE_VERSION v5→v6 (U8)

Final verification for the RSS-description-end-to-end fix:

- docs/architecture.mdx — one-paragraph "News Grounding Pipeline"
  subsection tracing parser → story:track:v1.description → NewsItem.snippet
  → brief / SummarizeArticle / dashboard / email / relay / MCP, with the
  empty-description R6 fallback rule called out explicitly.
- tests/summarize-reasoning.test.mjs — Fix-4 static-analysis pin updated
  to match the v6 bump from U6. Without this the summary cache bump silently
  regressed CI's pinned-version assertion.

Final sweep (2026-04-24):
- grep -rn 'brief:llm:description:v1' → only in the U5 legacy-row test
  simulation (by design: proves the v2 bump forces cold-start).
- grep -rn 'brief:llm:whymatters:v2/v6/shadow:v4' → no live references.
- grep -rn 'summary:v5' → no references.
- CACHE_VERSION = 'v6' in src/utils/summary-cache-key.ts.
- Full tsx --test sweep across all tests/*.test.{mjs,mts}: 6747/6747 pass.
- npm run typecheck + typecheck:api: both clean.

Covers R4, R6, R7.

* fix(rss-description): address /ce:review findings before merge

14 fixes from structured code review across 13 reviewer personas.

Correctness-critical (P1 — fixes that prevent R6/U7 contract violations):
- NewsPanel signature covers currentBodies so view-mode toggles that leave
  headlines identical but bodies different now invalidate in-flight summaries.
  Without this, switching renderItems → renderClusters mid-summary let a
  grounded response arrive under a stale (now-orphaned) cache key.
- summarize-article.ts re-pairs bodies with headlines BEFORE dedup via a
  single zip-sanitize-filter-dedup pass. Previously bodies[] was indexed by
  position in light-sanitized headlines while findIndex looked up the
  full-sanitized array — any headline that sanitizeHeadlines emptied
  mispaired every subsequent body, grounding the LLM on the wrong story.
- Client skips the pre-chain cache lookup when bodies are present, since
  client builds keys from RAW bodies while server sanitizes first. The
  keys diverge on injection content, which would silently miss the
  server's authoritative cache every call.

Test + audit hardening:
- Legacy v1 eviction test now uses the real hashBriefStory(story()) suffix
  instead of a literal "somehash", so a bug where the reader still queried
  the v1 prefix at the real key would actually be caught.
- tests/summary-cache-key.test.mts adds 400-char clip identity coverage so
  the canonicalizer's clip and any downstream clip can't silently drift.
- tests/news-rss-description-extract.test.mts renames the well-formed
  CDATA test and adds a new test documenting the malformed-]]> fallback
  behavior (plain regex captures, article content survives).

Safe_auto cleanups:
- Deleted dead SNIPPET_PUSH_MAX constant in notification-relay.cjs.
- BETA-mode groq warm call now passes bodies, warming the right cache slot.
- seed-digest shares a local normalize-equality helper for description !=
  headline comparison, matching the parser's contract.
- Pair-wise sort in summary-cache-key tie-breaks on body so duplicate
  headlines produce stable order across runs.
- buildSummaryCacheKey gained JSDoc documenting the client/server contract
  and the bodies parameter semantics.
- MCP get_world_brief tool description now mentions RSS article-body
  grounding so calling agents see the current contract.
- _shared.ts `opts.bodies![i]!` double-bang replaced with `?? ''`.
- extractRawTagBody regexes cached in module-level Map, mirroring the
  existing TAG_REGEX_CACHE pattern.

Deferred to follow-up (tracked for PR description / separate issue):
- Promote shared MAX_BODY constant across the 5 clip sites
- Promote shared truncateForDisplay helper across 4 render sites
- Collapse NewsPanel.{currentHeadlines, currentBodies} → Array<{title, snippet}>
- Promote sanitizeStoryForPrompt to shared/brief-llm-core.js
- Split list-feed-digest.ts parser helpers into sibling -utils.ts
- Strengthen audit test: forward-sweep + behavioral gate test

Tests: 6749/6749 pass. Typecheck clean on both configs.

* fix(summarization): thread bodies through browser T5 path (Codex #2)

Addresses the second of two Codex-raised findings on PR #3370:

The PR threaded bodies through the server-side API provider chain
(Ollama → Groq → OpenRouter → /api/news/v1/summarize-article) but the
local browser T5 path at tryBrowserT5 was still summarising from
headlines alone. In BETA_MODE that ungrounded path runs BEFORE the
grounded server providers; in normal mode it remains the last
fallback. Whenever T5-small won, the dashboard summary surface
regressed to the headline-only path — the exact hallucination class
this PR exists to eliminate.

Fix: tryBrowserT5 accepts an optional `bodies` parameter and
interleaves each body with its paired headline via a `headline —
body` separator in the combined text (clipped to 200 chars per body
to stay within T5-small's ~512-token context window). All three call
sites (BETA warm, BETA cold, normal-mode fallback) now pass the
bodies threaded down from generateSummary options.bodies.

When bodies is empty/omitted, the combined text is byte-identical to
pre-fix (R6 preserved).

On Codex finding #1 (story:track:v1 additive-only HSET keeps a body
from an earlier mention of the same normalized title), declining to
change. The current rule — "if this mention has a body, overwrite;
otherwise leave the prior body alone" — is defensible: a body from
mention A is not falsified by mention B being body-less (a wire
reprint doesn't invalidate the original source's body). A feed that
publishes a corrected headline creates a new normalized-title hash,
so no stale body carries forward. The failure window is narrow (live
story evolving while keeping the same title through hours of
body-less wire reprints) and the 7-day STORY_TTL is the backstop.
Opening a follow-up issue to revisit semantics if real-world evidence
surfaces a stale-grounding case.

* fix(story-track): description always-written to overwrite stale bodies (Codex #1)

Revisiting Codex finding #1 on PR #3370 after re-review. The previous
response declined the fix with reasoning; on reflection the argument
was over-defending the current behavior.

Problem: buildStoryTrackHsetFields previously wrote `description` only
when non-empty. Because story:track:v1 rows are collapsed by
normalized-title hash, an earlier mention's body would persist for up
to STORY_TTL (7 days) on subsequent body-less mentions of the same
story. Consumers reading `track.description` via HGETALL could not
distinguish "this mention's body" from "some mention's body from the
last week," silently grounding brief / whyMatters / SummarizeArticle
LLMs on text the current mention never supplied. That violates the
grounding contract advertised to every downstream surface in this PR.

Fix: HSET `description` unconditionally on every mention — empty
string when the current item has no body, real body when it does. An
empty value overwrites any prior mention's body so the row is always
authoritative for the current cycle. Consumers continue to treat
empty description as "fall back to cleaned headline" (R6 preserved).
The 7-day STORY_TTL and normalized-title hash semantics are unchanged.

Trade-off accepted: a valid body from Feed A (NYT) is wiped when Feed
B (AP body-less wire reprint) arrives for the same normalized title,
even though Feed A's body is factually correct. Rationale: the
alternative — keeping Feed A's body indefinitely — means the user
sees Feed A's body attributed (by proximity) to an AP mention at a
later timestamp, which is at minimum misleading and at worst carries
retracted/corrected details. Honest absence beats unlabeled presence.

Tests: new stale-body overwrite sequence test (T0 body → T1 empty →
T2 new body), existing "writes description when non-empty" preserved,
existing "omits when empty" inverted to "writes empty, overwriting."
cache-keys.ts contract comment updated to mark description as
always-written rather than optional.
2026-04-24 16:25:14 +04:00

1186 lines
64 KiB
JavaScript
Executable File
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env node
/**
* Consolidated aviation seeder. Writes four Redis keys from one cron tick:
*
* aviation:delays:intl:v3 — AviationStack per-airport delay aggregates (51 intl)
* aviation:delays:faa:v1 — FAA ASWS XML delays (30 US)
* aviation:notam:closures:v2 — ICAO NOTAM closures (60 global)
* aviation:news::24:v1 — RSS news prewarmer (list-aviation-news.ts cache)
*
* Also publishes notifications for new severe/major airport disruptions and new
* NOTAM closures via the standard wm:events:queue LPUSH + wm:notif:scan-dedup SETNX.
* Prev-alerted state is persisted to Redis so short-lived cron invocations don't
* re-notify on every tick.
*
* @notification-source: domain (aviation)
* publishNotificationEvent() calls in this file build payload.title from
* structured airport/ICAO/delay/NOTAM fields. Events are NOT RSS-origin
* and MUST NOT set payload.description. Enforced by
* tests/notification-relay-payload-audit.test.mjs.
*
* Supersedes: scripts/seed-airport-delays.mjs (deleted) + the in-process seed
* loops that used to live inside scripts/ais-relay.cjs (stripped). ais-relay still
* hosts the /aviationstack live proxy for user-triggered flight lookups.
*/
import { readFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
import {
loadEnvFile,
CHROME_UA,
runSeed,
writeExtraKeyWithMeta,
extendExistingTtl,
acquireLockSafely,
releaseLock,
getRedisCredentials,
} from './_seed-utils.mjs';
import { unwrapEnvelope } from './_seed-envelope-source.mjs';
loadEnvFile(import.meta.url);
// ─── Redis keys / TTLs ───────────────────────────────────────────────────────
const INTL_KEY = 'aviation:delays:intl:v3';
const FAA_KEY = 'aviation:delays:faa:v1';
const NOTAM_KEY = 'aviation:notam:closures:v2';
const NEWS_KEY = 'aviation:news::24:v1';
// Page-load hydration aggregate. Health (api/health.js BOOTSTRAP_KEYS.flightDelays)
// reads STRLEN here. Historically only written as a 1800s RPC side-effect inside
// list-airport-delays.ts — quiet user windows >30min would let it expire, tripping
// EMPTY (CRIT) even with healthy upstream feeds. Now produced canonically by this
// seeder; RPC keeps its write at the same TTL as a courtesy mid-tick refresh.
const BOOTSTRAP_KEY = 'aviation:delays-bootstrap:v1';
const INTL_TTL = 10_800; // 3h — survives ~5 consecutive missed 30min cron ticks
const FAA_TTL = 7_200; // 2h
const NOTAM_TTL = 7_200; // 2h
const NEWS_TTL = 2_400; // 40min
const BOOTSTRAP_TTL = 7_200; // 2h — matches FAA/NOTAM; survives ~4 missed cron ticks
// health.js expects these exact meta keys (api/health.js:222,223,269)
const INTL_META_KEY = 'seed-meta:aviation:intl';
const FAA_META_KEY = 'seed-meta:aviation:faa';
const NOTAM_META_KEY = 'seed-meta:aviation:notam';
// Notification dedup state (persisted so cron runs don't spam on every tick)
const AVIATION_PREV_ALERTED_KEY = 'notifications:dedup:aviation:prev-alerted:v1';
const NOTAM_PREV_CLOSED_KEY = 'notam:prev-closed-state:v1';
const PREV_STATE_TTL = 86_400; // 24h — longer than any realistic cron cadence
// ─── Unified airport registry ────────────────────────────────────────────────
// Each row declares: iata, icao, name, city, country, region, lat, lon (where
// known), and which data sources cover it:
// 'aviationstack' — AviationStack /v1/flights?dep_iata={iata}
// 'faa' — FAA ASWS XML filter matches this IATA
// 'notam' — ICAO NOTAM list includes this ICAO
// lat/lon/city are only required for rows with 'aviationstack' (feed the
// AirportDelayAlert envelope).
const AIRPORTS = [
// ── Americas — AviationStack + NOTAM ──
{ iata: 'YYZ', icao: 'CYYZ', name: 'Toronto Pearson', city: 'Toronto', country: 'Canada', lat: 43.6777, lon: -79.6248, region: 'americas', sources: ['aviationstack', 'notam'] },
{ iata: 'YVR', icao: 'CYVR', name: 'Vancouver International', city: 'Vancouver', country: 'Canada', lat: 49.1947, lon: -123.1792, region: 'americas', sources: ['aviationstack'] },
{ iata: 'MEX', icao: 'MMMX', name: 'Mexico City International', city: 'Mexico City', country: 'Mexico', lat: 19.4363, lon: -99.0721, region: 'americas', sources: ['aviationstack', 'notam'] },
{ iata: 'GRU', icao: 'SBGR', name: 'São PauloGuarulhos', city: 'São Paulo', country: 'Brazil', lat: -23.4356, lon: -46.4731, region: 'americas', sources: ['aviationstack', 'notam'] },
{ iata: 'EZE', icao: 'SAEZ', name: 'Ministro Pistarini', city: 'Buenos Aires', country: 'Argentina', lat: -34.8222, lon: -58.5358, region: 'americas', sources: ['aviationstack'] },
{ iata: 'BOG', icao: 'SKBO', name: 'El Dorado International', city: 'Bogotá', country: 'Colombia', lat: 4.7016, lon: -74.1469, region: 'americas', sources: ['aviationstack', 'notam'] },
{ iata: 'SCL', icao: 'SCEL', name: 'Arturo Merino Benítez', city: 'Santiago', country: 'Chile', lat: -33.3930, lon: -70.7858, region: 'americas', sources: ['aviationstack', 'notam'] },
// ── Americas — FAA + NOTAM (US only; many dual-covered with AviationStack too for intl flights) ──
{ iata: 'ATL', icao: 'KATL', name: 'HartsfieldJackson Atlanta', city: 'Atlanta', country: 'USA', region: 'americas', sources: ['faa', 'notam'] },
{ iata: 'ORD', icao: 'KORD', name: "Chicago O'Hare", city: 'Chicago', country: 'USA', region: 'americas', sources: ['faa', 'notam'] },
{ iata: 'DFW', icao: 'KDFW', name: 'Dallas/Fort Worth', city: 'Dallas', country: 'USA', region: 'americas', sources: ['faa', 'notam'] },
{ iata: 'DEN', icao: 'KDEN', name: 'Denver International', city: 'Denver', country: 'USA', region: 'americas', sources: ['faa', 'notam'] },
{ iata: 'LAX', icao: 'KLAX', name: 'Los Angeles International', city: 'Los Angeles', country: 'USA', region: 'americas', sources: ['faa', 'notam'] },
{ iata: 'JFK', icao: 'KJFK', name: 'John F. Kennedy International', city: 'New York', country: 'USA', region: 'americas', sources: ['faa', 'notam'] },
{ iata: 'SFO', icao: 'KSFO', name: 'San Francisco International', city: 'San Francisco', country: 'USA', region: 'americas', sources: ['faa', 'notam'] },
{ iata: 'SEA', icao: 'KSEA', name: 'SeattleTacoma International', city: 'Seattle', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'LAS', icao: 'KLAS', name: 'Harry Reid International', city: 'Las Vegas', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'MCO', icao: 'KMCO', name: 'Orlando International', city: 'Orlando', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'EWR', icao: 'KEWR', name: 'Newark Liberty International', city: 'Newark', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'CLT', icao: 'KCLT', name: 'Charlotte Douglas International', city: 'Charlotte', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'PHX', icao: 'KPHX', name: 'Phoenix Sky Harbor International', city: 'Phoenix', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'IAH', icao: 'KIAH', name: 'George Bush Intercontinental', city: 'Houston', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'MIA', icao: 'KMIA', name: 'Miami International', city: 'Miami', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'BOS', icao: 'KBOS', name: 'Logan International', city: 'Boston', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'MSP', icao: 'KMSP', name: 'MinneapolisSaint Paul International', city: 'Minneapolis', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'DTW', icao: 'KDTW', name: 'Detroit Metropolitan', city: 'Detroit', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'FLL', icao: 'KFLL', name: 'Fort LauderdaleHollywood', city: 'Fort Lauderdale', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'PHL', icao: 'KPHL', name: 'Philadelphia International', city: 'Philadelphia', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'LGA', icao: 'KLGA', name: 'LaGuardia', city: 'New York', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'BWI', icao: 'KBWI', name: 'Baltimore/Washington International', city: 'Baltimore', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'SLC', icao: 'KSLC', name: 'Salt Lake City International', city: 'Salt Lake City', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'SAN', icao: 'KSAN', name: 'San Diego International', city: 'San Diego', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'IAD', icao: 'KIAD', name: 'Washington Dulles International', city: 'Washington', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'DCA', icao: 'KDCA', name: 'Ronald Reagan Washington National', city: 'Washington', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'MDW', icao: 'KMDW', name: 'Chicago Midway International', city: 'Chicago', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'TPA', icao: 'KTPA', name: 'Tampa International', city: 'Tampa', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'HNL', icao: 'PHNL', name: 'Daniel K. Inouye International', city: 'Honolulu', country: 'USA', region: 'americas', sources: ['faa'] },
{ iata: 'PDX', icao: 'KPDX', name: 'Portland International', city: 'Portland', country: 'USA', region: 'americas', sources: ['faa'] },
// ── Europe — AviationStack + NOTAM ──
{ iata: 'LHR', icao: 'EGLL', name: 'London Heathrow', city: 'London', country: 'UK', lat: 51.4700, lon: -0.4543, region: 'europe', sources: ['aviationstack', 'notam'] },
{ iata: 'CDG', icao: 'LFPG', name: 'Paris Charles de Gaulle', city: 'Paris', country: 'France', lat: 49.0097, lon: 2.5479, region: 'europe', sources: ['aviationstack', 'notam'] },
{ iata: 'FRA', icao: 'EDDF', name: 'Frankfurt Airport', city: 'Frankfurt', country: 'Germany', lat: 50.0379, lon: 8.5622, region: 'europe', sources: ['aviationstack', 'notam'] },
{ iata: 'AMS', icao: 'EHAM', name: 'Amsterdam Schiphol', city: 'Amsterdam', country: 'Netherlands', lat: 52.3105, lon: 4.7683, region: 'europe', sources: ['aviationstack', 'notam'] },
{ iata: 'MAD', icao: 'LEMD', name: 'Adolfo Suárez MadridBarajas', city: 'Madrid', country: 'Spain', lat: 40.4983, lon: -3.5676, region: 'europe', sources: ['aviationstack', 'notam'] },
{ iata: 'FCO', icao: 'LIRF', name: 'Leonardo da VinciFiumicino', city: 'Rome', country: 'Italy', lat: 41.8003, lon: 12.2389, region: 'europe', sources: ['aviationstack', 'notam'] },
{ iata: 'MUC', icao: 'EDDM', name: 'Munich Airport', city: 'Munich', country: 'Germany', lat: 48.3537, lon: 11.7750, region: 'europe', sources: ['aviationstack'] },
{ iata: 'BCN', icao: 'LEBL', name: 'BarcelonaEl Prat', city: 'Barcelona', country: 'Spain', lat: 41.2974, lon: 2.0833, region: 'europe', sources: ['aviationstack'] },
{ iata: 'ZRH', icao: 'LSZH', name: 'Zurich Airport', city: 'Zurich', country: 'Switzerland', lat: 47.4647, lon: 8.5492, region: 'europe', sources: ['aviationstack', 'notam'] },
{ iata: 'IST', icao: 'LTFM', name: 'Istanbul Airport', city: 'Istanbul', country: 'Turkey', lat: 41.2753, lon: 28.7519, region: 'europe', sources: ['aviationstack', 'notam'] },
{ iata: 'VIE', icao: 'LOWW', name: 'Vienna International', city: 'Vienna', country: 'Austria', lat: 48.1103, lon: 16.5697, region: 'europe', sources: ['aviationstack', 'notam'] },
{ iata: 'CPH', icao: 'EKCH', name: 'Copenhagen Airport', city: 'Copenhagen', country: 'Denmark', lat: 55.6180, lon: 12.6508, region: 'europe', sources: ['aviationstack', 'notam'] },
{ iata: 'DUB', icao: 'EIDW', name: 'Dublin Airport', city: 'Dublin', country: 'Ireland', lat: 53.4264, lon: -6.2499, region: 'europe', sources: ['aviationstack'] },
{ iata: 'LIS', icao: 'LPPT', name: 'Humberto Delgado Airport', city: 'Lisbon', country: 'Portugal', lat: 38.7756, lon: -9.1354, region: 'europe', sources: ['aviationstack'] },
{ iata: 'ATH', icao: 'LGAV', name: 'Athens International', city: 'Athens', country: 'Greece', lat: 37.9364, lon: 23.9445, region: 'europe', sources: ['aviationstack'] },
{ iata: 'WAW', icao: 'EPWA', name: 'Warsaw Chopin Airport', city: 'Warsaw', country: 'Poland', lat: 52.1657, lon: 20.9671, region: 'europe', sources: ['aviationstack', 'notam'] },
// Europe NOTAM-only (no AviationStack coverage today)
{ iata: 'OSL', icao: 'ENGM', name: 'Oslo Gardermoen', city: 'Oslo', country: 'Norway', region: 'europe', sources: ['notam'] },
{ iata: 'ARN', icao: 'ESSA', name: 'Stockholm Arlanda', city: 'Stockholm', country: 'Sweden', region: 'europe', sources: ['notam'] },
{ iata: 'HEL', icao: 'EFHK', name: 'Helsinki-Vantaa', city: 'Helsinki', country: 'Finland', region: 'europe', sources: ['notam'] },
// ── APAC — AviationStack + NOTAM ──
{ iata: 'HND', icao: 'RJTT', name: 'Tokyo Haneda', city: 'Tokyo', country: 'Japan', lat: 35.5494, lon: 139.7798, region: 'apac', sources: ['aviationstack', 'notam'] },
{ iata: 'NRT', icao: 'RJAA', name: 'Narita International', city: 'Tokyo', country: 'Japan', lat: 35.7720, lon: 140.3929, region: 'apac', sources: ['aviationstack'] },
{ iata: 'PEK', icao: 'ZBAA', name: 'Beijing Capital', city: 'Beijing', country: 'China', lat: 40.0799, lon: 116.6031, region: 'apac', sources: ['aviationstack', 'notam'] },
{ iata: 'PVG', icao: 'ZSPD', name: 'Shanghai Pudong', city: 'Shanghai', country: 'China', lat: 31.1443, lon: 121.8083, region: 'apac', sources: ['aviationstack'] },
{ iata: 'HKG', icao: 'VHHH', name: 'Hong Kong International', city: 'Hong Kong', country: 'China', lat: 22.3080, lon: 113.9185, region: 'apac', sources: ['aviationstack', 'notam'] },
{ iata: 'SIN', icao: 'WSSS', name: 'Singapore Changi', city: 'Singapore', country: 'Singapore', lat: 1.3644, lon: 103.9915, region: 'apac', sources: ['aviationstack', 'notam'] },
{ iata: 'ICN', icao: 'RKSI', name: 'Incheon International', city: 'Seoul', country: 'South Korea', lat: 37.4602, lon: 126.4407, region: 'apac', sources: ['aviationstack', 'notam'] },
{ iata: 'BKK', icao: 'VTBS', name: 'Suvarnabhumi Airport', city: 'Bangkok', country: 'Thailand', lat: 13.6900, lon: 100.7501, region: 'apac', sources: ['aviationstack', 'notam'] },
{ iata: 'SYD', icao: 'YSSY', name: 'Sydney Kingsford Smith', city: 'Sydney', country: 'Australia', lat: -33.9461, lon: 151.1772, region: 'apac', sources: ['aviationstack', 'notam'] },
{ iata: 'DEL', icao: 'VIDP', name: 'Indira Gandhi International', city: 'Delhi', country: 'India', lat: 28.5562, lon: 77.1000, region: 'apac', sources: ['aviationstack', 'notam'] },
{ iata: 'BOM', icao: 'VABB', name: 'Chhatrapati Shivaji Maharaj', city: 'Mumbai', country: 'India', lat: 19.0896, lon: 72.8656, region: 'apac', sources: ['aviationstack'] },
{ iata: 'KUL', icao: 'WMKK', name: 'Kuala Lumpur International', city: 'Kuala Lumpur', country: 'Malaysia', lat: 2.7456, lon: 101.7099, region: 'apac', sources: ['aviationstack', 'notam'] },
{ iata: 'CAN', icao: 'ZGGG', name: 'Guangzhou Baiyun International', city: 'Guangzhou', country: 'China', lat: 23.3924, lon: 113.2988, region: 'apac', sources: ['aviationstack'] },
{ iata: 'TPE', icao: 'RCTP', name: 'Taiwan Taoyuan International', city: 'Taipei', country: 'Taiwan', lat: 25.0797, lon: 121.2342, region: 'apac', sources: ['aviationstack'] },
{ iata: 'MNL', icao: 'RPLL', name: 'Ninoy Aquino International', city: 'Manila', country: 'Philippines', lat: 14.5086, lon: 121.0197, region: 'apac', sources: ['aviationstack'] },
// APAC NOTAM-only
{ iata: 'KMG', icao: 'ZPPP', name: 'Kunming Changshui', city: 'Kunming', country: 'China', region: 'apac', sources: ['notam'] },
// ── MENA — AviationStack + NOTAM ──
{ iata: 'DXB', icao: 'OMDB', name: 'Dubai International', city: 'Dubai', country: 'UAE', lat: 25.2532, lon: 55.3657, region: 'mena', sources: ['aviationstack', 'notam'] },
{ iata: 'DOH', icao: 'OTHH', name: 'Hamad International', city: 'Doha', country: 'Qatar', lat: 25.2731, lon: 51.6081, region: 'mena', sources: ['aviationstack', 'notam'] },
{ iata: 'AUH', icao: 'OMAA', name: 'Abu Dhabi International', city: 'Abu Dhabi', country: 'UAE', lat: 24.4330, lon: 54.6511, region: 'mena', sources: ['aviationstack', 'notam'] },
{ iata: 'RUH', icao: 'OERK', name: 'King Khalid International', city: 'Riyadh', country: 'Saudi Arabia', lat: 24.9576, lon: 46.6988, region: 'mena', sources: ['aviationstack', 'notam'] },
{ iata: 'CAI', icao: 'HECA', name: 'Cairo International', city: 'Cairo', country: 'Egypt', lat: 30.1219, lon: 31.4056, region: 'mena', sources: ['aviationstack', 'notam'] },
{ iata: 'TLV', icao: 'LLBG', name: 'Ben Gurion Airport', city: 'Tel Aviv', country: 'Israel', lat: 32.0055, lon: 34.8854, region: 'mena', sources: ['aviationstack'] },
{ iata: 'AMM', icao: 'OJAI', name: 'Queen Alia International', city: 'Amman', country: 'Jordan', lat: 31.7226, lon: 35.9932, region: 'mena', sources: ['aviationstack', 'notam'] },
{ iata: 'KWI', icao: 'OKBK', name: 'Kuwait International', city: 'Kuwait City', country: 'Kuwait', lat: 29.2266, lon: 47.9689, region: 'mena', sources: ['aviationstack', 'notam'] },
{ iata: 'CMN', icao: 'GMMN', name: 'Mohammed V International', city: 'Casablanca', country: 'Morocco', lat: 33.3675, lon: -7.5898, region: 'mena', sources: ['aviationstack', 'notam'] },
// MENA NOTAM-only
{ iata: 'JED', icao: 'OEJN', name: 'King Abdulaziz', city: 'Jeddah', country: 'Saudi Arabia', region: 'mena', sources: ['notam'] },
{ iata: 'MED', icao: 'OEMA', name: 'Prince Mohammad bin Abdulaziz', city: 'Medina', country: 'Saudi Arabia', region: 'mena', sources: ['notam'] },
{ iata: 'DMM', icao: 'OEDF', name: 'King Fahd International', city: 'Dammam', country: 'Saudi Arabia', region: 'mena', sources: ['notam'] },
{ iata: 'SHJ', icao: 'OMSJ', name: 'Sharjah International', city: 'Sharjah', country: 'UAE', region: 'mena', sources: ['notam'] },
{ iata: 'BAH', icao: 'OBBI', name: 'Bahrain International', city: 'Manama', country: 'Bahrain', region: 'mena', sources: ['notam'] },
{ iata: 'MCT', icao: 'OOMS', name: 'Muscat International', city: 'Muscat', country: 'Oman', region: 'mena', sources: ['notam'] },
{ iata: 'BEY', icao: 'OLBA', name: 'BeirutRafic Hariri', city: 'Beirut', country: 'Lebanon', region: 'mena', sources: ['notam'] },
{ iata: 'DAM', icao: 'OSDI', name: 'Damascus International', city: 'Damascus', country: 'Syria', region: 'mena', sources: ['notam'] },
{ iata: 'BGW', icao: 'ORBI', name: 'Baghdad International', city: 'Baghdad', country: 'Iraq', region: 'mena', sources: ['notam'] },
{ iata: 'IKA', icao: 'OIIE', name: 'Imam Khomeini International', city: 'Tehran', country: 'Iran', region: 'mena', sources: ['notam'] },
{ iata: 'SYZ', icao: 'OISS', name: 'Shiraz International', city: 'Shiraz', country: 'Iran', region: 'mena', sources: ['notam'] },
{ iata: 'MHD', icao: 'OIMM', name: 'Mashhad International', city: 'Mashhad', country: 'Iran', region: 'mena', sources: ['notam'] },
{ iata: 'BND', icao: 'OIKB', name: 'Bandar Abbas International', city: 'Bandar Abbas', country: 'Iran', region: 'mena', sources: ['notam'] },
{ iata: 'TUN', icao: 'DTTA', name: 'TunisCarthage', city: 'Tunis', country: 'Tunisia', region: 'mena', sources: ['notam'] },
{ iata: 'ALG', icao: 'DAAG', name: 'Houari Boumediene', city: 'Algiers', country: 'Algeria', region: 'mena', sources: ['notam'] },
{ iata: 'TIP', icao: 'HLLT', name: 'Tripoli International', city: 'Tripoli', country: 'Libya', region: 'mena', sources: ['notam'] },
// ── Africa — AviationStack + NOTAM ──
{ iata: 'JNB', icao: 'FAOR', name: "O.R. Tambo International", city: 'Johannesburg', country: 'South Africa', lat: -26.1392, lon: 28.2460, region: 'africa', sources: ['aviationstack', 'notam'] },
{ iata: 'NBO', icao: 'HKJK', name: 'Jomo Kenyatta International', city: 'Nairobi', country: 'Kenya', lat: -1.3192, lon: 36.9278, region: 'africa', sources: ['aviationstack', 'notam'] },
{ iata: 'LOS', icao: 'DNMM', name: 'Murtala Muhammed International', city: 'Lagos', country: 'Nigeria', lat: 6.5774, lon: 3.3212, region: 'africa', sources: ['aviationstack', 'notam'] },
{ iata: 'ADD', icao: 'HAAB', name: 'Bole International', city: 'Addis Ababa', country: 'Ethiopia', lat: 8.9779, lon: 38.7993, region: 'africa', sources: ['aviationstack'] },
{ iata: 'CPT', icao: 'FACT', name: 'Cape Town International', city: 'Cape Town', country: 'South Africa', lat: -33.9715, lon: 18.6021, region: 'africa', sources: ['aviationstack'] },
// Africa NOTAM-only
{ iata: 'GBE', icao: 'FBSK', name: 'Sir Seretse Khama International', city: 'Gaborone', country: 'Botswana', region: 'africa', sources: ['notam'] },
];
// Derived per-source views (built once at module load)
const AVIATIONSTACK_LIST = AIRPORTS.filter(a => a.sources.includes('aviationstack'));
const FAA_LIST = AIRPORTS.filter(a => a.sources.includes('faa')).map(a => a.iata);
const NOTAM_LIST = AIRPORTS.filter(a => a.sources.includes('notam')).map(a => a.icao);
// iata → aviationstack-enriched meta (for building AirportDelayAlert envelopes
// with coordinates — aviationstack rows are the only ones with lat/lon).
const AIRPORT_META = Object.fromEntries(AVIATIONSTACK_LIST.map(a => [a.iata, a]));
// iata → FAA-row meta (icao/name/city/country for alert envelopes; no lat/lon
// by design — FAA rows are US-regional airports we don't render on the globe).
const FAA_META = Object.fromEntries(
AIRPORTS.filter(a => a.sources.includes('faa')).map(a => [a.iata, a]),
);
// Protobuf enum mappers (mirror ais-relay.cjs mappings; consumers parse strings)
const REGION_MAP = {
americas: 'AIRPORT_REGION_AMERICAS',
europe: 'AIRPORT_REGION_EUROPE',
apac: 'AIRPORT_REGION_APAC',
mena: 'AIRPORT_REGION_MENA',
africa: 'AIRPORT_REGION_AFRICA',
};
const DELAY_TYPE_MAP = {
ground_stop: 'FLIGHT_DELAY_TYPE_GROUND_STOP',
ground_delay: 'FLIGHT_DELAY_TYPE_GROUND_DELAY',
departure_delay: 'FLIGHT_DELAY_TYPE_DEPARTURE_DELAY',
arrival_delay: 'FLIGHT_DELAY_TYPE_ARRIVAL_DELAY',
general: 'FLIGHT_DELAY_TYPE_GENERAL',
closure: 'FLIGHT_DELAY_TYPE_CLOSURE',
};
const SEVERITY_MAP = {
normal: 'FLIGHT_DELAY_SEVERITY_NORMAL',
minor: 'FLIGHT_DELAY_SEVERITY_MINOR',
moderate: 'FLIGHT_DELAY_SEVERITY_MODERATE',
major: 'FLIGHT_DELAY_SEVERITY_MAJOR',
severe: 'FLIGHT_DELAY_SEVERITY_SEVERE',
};
const AVIATION_BATCH_CONCURRENCY = 10;
const AVIATION_MIN_FLIGHTS_FOR_CLOSURE = 10;
const RESOLVED_STATUSES = new Set(['cancelled', 'landed', 'active', 'arrived', 'diverted']);
// ─── Inline Upstash helpers (LPUSH + SETNX + GET/SET) ────────────────────────
// These aren't in _seed-utils.mjs (which focuses on SET/GET/EXPIRE). Pattern
// mirrors ais-relay.cjs upstashLpush/upstashSetNx/upstashSet/upstashGet so the
// notification queue + prev-state reads speak the same wire protocol.
async function upstashCommand(cmd) {
const { url, token } = getRedisCredentials();
const resp = await fetch(url, {
method: 'POST',
headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
body: JSON.stringify(cmd),
signal: AbortSignal.timeout(10_000),
});
if (!resp.ok) throw new Error(`Upstash ${cmd[0]} failed: HTTP ${resp.status}`);
return resp.json();
}
async function upstashGet(key) {
try {
const result = await upstashCommand(['GET', key]);
if (!result?.result) return null;
try { return JSON.parse(result.result); } catch { return null; }
} catch { return null; }
}
// Envelope-aware GET. runSeed wraps canonical keys in `{_seed, data}` when the
// seeder opts into the seed contract (declareRecords + envelopeMeta) — INTL_KEY
// is one such key (see runSeed call w/ declareRecords below). Bare values
// (FAA_KEY, NOTAM_KEY via writeExtraKey w/o envelopeMeta) pass through.
async function upstashGetUnwrapped(key) {
const raw = await upstashGet(key);
return unwrapEnvelope(raw).data;
}
async function upstashSet(key, value, ttlSeconds) {
try {
const serialized = typeof value === 'string' ? value : JSON.stringify(value);
const result = await upstashCommand(['SET', key, serialized, 'EX', String(ttlSeconds)]);
return result?.result === 'OK';
} catch { return false; }
}
async function upstashSetNx(key, value, ttlSeconds) {
try {
const serialized = typeof value === 'string' ? value : JSON.stringify(value);
const result = await upstashCommand(['SET', key, serialized, 'NX', 'EX', String(ttlSeconds)]);
return result?.result === 'OK' ? 'OK' : null;
} catch { return null; }
}
async function upstashLpush(key, value) {
try {
const serialized = typeof value === 'string' ? value : JSON.stringify(value);
const result = await upstashCommand(['LPUSH', key, serialized]);
return typeof result?.result === 'number' && result.result > 0;
} catch { return false; }
}
async function upstashDel(key) {
try {
const result = await upstashCommand(['DEL', key]);
return result?.result === 1;
} catch { return false; }
}
// ─── Notification publishing ─────────────────────────────────────────────────
// Mirrors ais-relay.cjs::publishNotificationEvent: LPUSH the event onto
// wm:events:queue, guarded by a SETNX dedup key (TTL = dedupTtl). On LPUSH
// failure, rollback the dedup key so the next run can retry.
function notifyHash(str) {
let h = 0;
for (let i = 0; i < str.length; i++) h = (Math.imul(31, h) + str.charCodeAt(i)) | 0;
return Math.abs(h).toString(36);
}
async function publishNotificationEvent({ eventType, payload, severity, variant, dedupTtl = 1800 }) {
try {
const variantSuffix = variant ? `:${variant}` : '';
const dedupKey = `wm:notif:scan-dedup:${eventType}${variantSuffix}:${notifyHash(`${eventType}:${payload.title ?? ''}`)}`;
const isNew = await upstashSetNx(dedupKey, '1', dedupTtl);
if (!isNew) {
console.log(`[Notify] Dedup hit — ${eventType}: ${String(payload.title ?? '').slice(0, 60)}`);
return;
}
const msg = JSON.stringify({ eventType, payload, severity, ...(variant ? { variant } : {}), publishedAt: Date.now() });
const ok = await upstashLpush('wm:events:queue', msg);
if (ok) {
console.log(`[Notify] Queued ${severity} event: ${eventType}${String(payload.title ?? '').slice(0, 60)}`);
} else {
console.warn(`[Notify] LPUSH failed for ${eventType} — rolling back dedup key`);
await upstashDel(dedupKey);
}
} catch (e) {
console.warn(`[Notify] publishNotificationEvent error (${eventType}):`, e?.message || e);
}
}
// ─── Section 1: AviationStack intl delays ────────────────────────────────────
const AVIATIONSTACK_URL = 'https://api.aviationstack.com/v1/flights';
function aviationDetermineSeverity(avgDelay, delayedPct) {
if (avgDelay >= 60 || (delayedPct && delayedPct >= 60)) return 'severe';
if (avgDelay >= 45 || (delayedPct && delayedPct >= 45)) return 'major';
if (avgDelay >= 30 || (delayedPct && delayedPct >= 30)) return 'moderate';
if (avgDelay >= 15 || (delayedPct && delayedPct >= 15)) return 'minor';
return 'normal';
}
async function fetchAviationStackSingle(apiKey, iata) {
const today = new Date().toISOString().slice(0, 10);
const url = `${AVIATIONSTACK_URL}?access_key=${apiKey}&dep_iata=${iata}&flight_date=${today}&limit=100`;
try {
const resp = await fetch(url, {
headers: { 'User-Agent': CHROME_UA },
signal: AbortSignal.timeout(10_000),
});
if (!resp.ok) {
console.warn(`[Aviation] ${iata}: HTTP ${resp.status}`);
return { ok: false, alert: null };
}
const json = await resp.json();
if (json.error) {
console.warn(`[Aviation] ${iata}: ${json.error.message}`);
return { ok: false, alert: null };
}
const flights = json?.data ?? [];
const alert = aviationAggregateFlights(iata, flights);
return { ok: true, alert };
} catch (err) {
console.warn(`[Aviation] ${iata}: fetch error: ${err?.message || err}`);
return { ok: false, alert: null };
}
}
function aviationAggregateFlights(iata, flights) {
if (flights.length === 0) return null;
const meta = AIRPORT_META[iata];
if (!meta) return null;
let delayed = 0, cancelled = 0, totalDelay = 0, resolved = 0;
for (const f of flights) {
if (RESOLVED_STATUSES.has(f.flight_status || '')) resolved++;
if (f.flight_status === 'cancelled') cancelled++;
if (f.departure?.delay && f.departure.delay > 0) {
delayed++;
totalDelay += f.departure.delay;
}
}
const total = resolved >= AVIATION_MIN_FLIGHTS_FOR_CLOSURE ? resolved : flights.length;
const cancelledPct = (cancelled / total) * 100;
const delayedPct = (delayed / total) * 100;
const avgDelay = delayed > 0 ? Math.round(totalDelay / delayed) : 0;
let severity, delayType, reason;
if (cancelledPct >= 80 && total >= AVIATION_MIN_FLIGHTS_FOR_CLOSURE) {
severity = 'severe'; delayType = 'closure';
reason = 'Airport closure / airspace restrictions';
} else if (cancelledPct >= 50 && total >= AVIATION_MIN_FLIGHTS_FOR_CLOSURE) {
severity = 'major'; delayType = 'ground_stop';
reason = `${Math.round(cancelledPct)}% flights cancelled`;
} else if (cancelledPct >= 20 && total >= AVIATION_MIN_FLIGHTS_FOR_CLOSURE) {
severity = 'moderate'; delayType = 'ground_delay';
reason = `${Math.round(cancelledPct)}% flights cancelled`;
} else if (cancelledPct >= 10 && total >= AVIATION_MIN_FLIGHTS_FOR_CLOSURE) {
severity = 'minor'; delayType = 'general';
reason = `${Math.round(cancelledPct)}% flights cancelled`;
} else if (avgDelay > 0) {
severity = aviationDetermineSeverity(avgDelay, delayedPct);
delayType = avgDelay >= 60 ? 'ground_delay' : 'general';
reason = `Avg ${avgDelay}min delay, ${Math.round(delayedPct)}% delayed`;
} else {
return null;
}
if (severity === 'normal') return null;
return {
id: `avstack-${iata}`,
iata,
icao: meta.icao,
name: meta.name,
city: meta.city,
country: meta.country,
location: { latitude: meta.lat, longitude: meta.lon },
region: REGION_MAP[meta.region] || 'AIRPORT_REGION_UNSPECIFIED',
delayType: DELAY_TYPE_MAP[delayType] || 'FLIGHT_DELAY_TYPE_GENERAL',
severity: SEVERITY_MAP[severity] || 'FLIGHT_DELAY_SEVERITY_NORMAL',
avgDelayMinutes: avgDelay,
delayedFlightsPct: Math.round(delayedPct),
cancelledFlights: cancelled,
totalFlights: total,
reason,
source: 'FLIGHT_DELAY_SOURCE_AVIATIONSTACK',
updatedAt: Date.now(),
};
}
async function seedIntlDelays() {
const apiKey = process.env.AVIATIONSTACK_API;
if (!apiKey) {
console.log('[Intl] No AVIATIONSTACK_API key — skipping');
return { alerts: [], healthy: false, skipped: true };
}
const t0 = Date.now();
const alerts = [];
let succeeded = 0, failed = 0;
for (let i = 0; i < AVIATIONSTACK_LIST.length; i += AVIATION_BATCH_CONCURRENCY) {
const chunk = AVIATIONSTACK_LIST.slice(i, i + AVIATION_BATCH_CONCURRENCY);
const results = await Promise.allSettled(
chunk.map(a => fetchAviationStackSingle(apiKey, a.iata)),
);
for (const r of results) {
if (r.status === 'fulfilled') {
if (r.value.ok) { succeeded++; if (r.value.alert) alerts.push(r.value.alert); }
else failed++;
} else {
failed++;
}
}
}
const healthy = AVIATIONSTACK_LIST.length < 5 || failed <= succeeded;
console.log(`[Intl] ${alerts.length} alerts (${succeeded} ok, ${failed} failed, healthy: ${healthy}) in ${((Date.now() - t0) / 1000).toFixed(1)}s`);
return { alerts, healthy, skipped: false };
}
// ─── Section 2: FAA delays (XML) ─────────────────────────────────────────────
const FAA_URL = 'https://nasstatus.faa.gov/api/airport-status-information';
function parseDelayTypeFromReason(reason) {
const r = reason.toLowerCase();
if (r.includes('ground stop')) return 'ground_stop';
if (r.includes('ground delay') || r.includes('gdp')) return 'ground_delay';
if (r.includes('departure')) return 'departure_delay';
if (r.includes('arrival')) return 'arrival_delay';
if (r.includes('clos')) return 'ground_stop';
return 'general';
}
function faaSeverityFromAvg(avgDelay) {
if (avgDelay >= 90) return 'severe';
if (avgDelay >= 60) return 'major';
if (avgDelay >= 30) return 'moderate';
if (avgDelay >= 15) return 'minor';
return 'normal';
}
function parseFaaXml(text) {
const delays = new Map();
const parseTag = (xml, tag) => {
const re = new RegExp(`<${tag}>(.*?)</${tag}>`, 'gs');
const out = [];
let m;
while ((m = re.exec(xml))) out.push(m[1]);
return out;
};
const getVal = (block, tag) => {
const m = block.match(new RegExp(`<${tag}>(.*?)</${tag}>`));
return m ? m[1].trim() : '';
};
for (const gd of parseTag(text, 'Ground_Delay')) {
const arpt = getVal(gd, 'ARPT');
if (arpt) {
delays.set(arpt, { airport: arpt, reason: getVal(gd, 'Reason') || 'Ground delay', avgDelay: parseInt(getVal(gd, 'Avg') || '30', 10), type: 'ground_delay' });
}
}
for (const gs of parseTag(text, 'Ground_Stop')) {
const arpt = getVal(gs, 'ARPT');
if (arpt) {
delays.set(arpt, { airport: arpt, reason: getVal(gs, 'Reason') || 'Ground stop', avgDelay: 60, type: 'ground_stop' });
}
}
for (const d of parseTag(text, 'Delay')) {
const arpt = getVal(d, 'ARPT');
if (arpt) {
const existing = delays.get(arpt);
if (!existing || existing.type !== 'ground_stop') {
const min = parseInt(getVal(d, 'Min') || '15', 10);
const max = parseInt(getVal(d, 'Max') || '30', 10);
delays.set(arpt, { airport: arpt, reason: getVal(d, 'Reason') || 'Delays', avgDelay: Math.round((min + max) / 2), type: parseDelayTypeFromReason(getVal(d, 'Reason') || '') });
}
}
}
for (const ac of parseTag(text, 'Airport')) {
const arpt = getVal(ac, 'ARPT');
if (arpt && FAA_LIST.includes(arpt)) {
delays.set(arpt, { airport: arpt, reason: 'Airport closure', avgDelay: 120, type: 'ground_stop' });
}
}
return delays;
}
async function seedFaaDelays() {
const t0 = Date.now();
const resp = await fetch(FAA_URL, {
headers: { Accept: 'application/xml', 'User-Agent': CHROME_UA },
signal: AbortSignal.timeout(15_000),
});
if (!resp.ok) throw new Error(`FAA HTTP ${resp.status}`);
const xml = await resp.text();
const faaDelays = parseFaaXml(xml);
const alerts = [];
for (const iata of FAA_LIST) {
const d = faaDelays.get(iata);
if (!d) continue;
const meta = FAA_META[iata];
alerts.push({
id: `faa-${iata}`,
iata,
icao: meta?.icao ?? '',
name: meta?.name ?? iata,
city: meta?.city ?? '',
country: meta?.country ?? 'USA',
location: { latitude: 0, longitude: 0 }, // FAA rows have no lat/lon in the registry
region: 'AIRPORT_REGION_AMERICAS',
delayType: `FLIGHT_DELAY_TYPE_${d.type.toUpperCase()}`,
severity: `FLIGHT_DELAY_SEVERITY_${faaSeverityFromAvg(d.avgDelay).toUpperCase()}`,
avgDelayMinutes: d.avgDelay,
delayedFlightsPct: 0,
cancelledFlights: 0,
totalFlights: 0,
reason: d.reason,
source: 'FLIGHT_DELAY_SOURCE_FAA',
updatedAt: Date.now(),
});
}
console.log(`[FAA] ${alerts.length} alerts in ${((Date.now() - t0) / 1000).toFixed(1)}s`);
return { alerts };
}
// ─── Section 3: NOTAM closures (ICAO) ────────────────────────────────────────
const ICAO_NOTAM_URL = 'https://dataservices.icao.int/api/notams-realtime-list';
const NOTAM_CLOSURE_QCODES = new Set(['FA', 'AH', 'AL', 'AW', 'AC', 'AM']);
// Restrictions: NOTAM Q-codes RA (restricted area) and RO (overfly prohibited)
// + restricted code45s and text patterns. Mirrors NOTAM_RESTRICTION_QCODES +
// the restriction-text regex in server/worldmonitor/aviation/v1/_shared.ts:29,
// :440-444 — keep in lockstep so seeded NOTAM data matches the live RPC's
// classifier.
const NOTAM_RESTRICTION_QCODES = new Set(['RA', 'RO']);
// Returns: Array of NOTAMs on success, null on quota exhaustion, [] on other errors.
async function fetchIcaoNotams() {
const apiKey = process.env.ICAO_API_KEY;
if (!apiKey) return [];
const locations = NOTAM_LIST.join(',');
const url = `${ICAO_NOTAM_URL}?api_key=${apiKey}&format=json&locations=${locations}`;
try {
const resp = await fetch(url, {
headers: { 'User-Agent': CHROME_UA },
signal: AbortSignal.timeout(30_000),
});
const body = await resp.text();
if (/reach call limit/i.test(body) || /quota.?exceed/i.test(body)) {
console.warn('[NOTAM] ICAO quota exhausted ("Reach call limit")');
return null;
}
if (!resp.ok) {
console.warn(`[NOTAM] ICAO HTTP ${resp.status}`);
return [];
}
const ct = resp.headers.get('content-type') || '';
if (ct.includes('text/html')) {
console.warn('[NOTAM] ICAO returned HTML (challenge page)');
return [];
}
try {
const data = JSON.parse(body);
return Array.isArray(data) ? data : [];
} catch {
console.warn('[NOTAM] Invalid JSON from ICAO');
return [];
}
} catch (err) {
console.warn(`[NOTAM] Fetch error: ${err?.message || err}`);
return [];
}
}
async function seedNotamClosures() {
if (!process.env.ICAO_API_KEY) {
console.log('[NOTAM] No ICAO_API_KEY — skipping');
return { closedIcaos: [], restrictedIcaos: [], reasons: {}, quotaExhausted: false, skipped: true };
}
const t0 = Date.now();
const notams = await fetchIcaoNotams();
if (notams === null) {
// Quota exhausted — don't blank the key; signal upstream to touch TTL.
return { closedIcaos: [], restrictedIcaos: [], reasons: {}, quotaExhausted: true, skipped: false };
}
const now = Math.floor(Date.now() / 1000);
const closedSet = new Set();
const restrictedSet = new Set();
const reasons = {};
for (const n of notams) {
const icao = n.itema || n.location || '';
if (!icao || !NOTAM_LIST.includes(icao)) continue;
if (n.endvalidity && n.endvalidity < now) continue;
const code23 = (n.code23 || '').toUpperCase();
const code45 = (n.code45 || '').toUpperCase();
const text = (n.iteme || '').toUpperCase();
const closureCode45 = code45 === 'LC' || code45 === 'AS' || code45 === 'AU' || code45 === 'XX' || code45 === 'AW';
const restrictionCode45 = code45 === 'RE' || code45 === 'RT';
const isClosureCode = NOTAM_CLOSURE_QCODES.has(code23) && closureCode45;
const isRestrictionCode = (NOTAM_RESTRICTION_QCODES.has(code23) || NOTAM_CLOSURE_QCODES.has(code23)) && restrictionCode45;
const isClosureText = /\b(AD CLSD|AIRPORT CLOSED|AIRSPACE CLOSED|AD NOT AVBL|CLSD TO ALL)\b/.test(text);
const isRestrictionText = /\b(RESTRICTED AREA|PROHIBITED AREA|DANGER AREA|TFR|TEMPORARY FLIGHT RESTRICTION)\b/.test(text);
// Closure wins over restriction for the same NOTAM (mirrors _shared.ts
// if/else chain at line 446-452).
if (isClosureCode || isClosureText) {
closedSet.add(icao);
reasons[icao] = n.iteme || 'Airport closure (NOTAM)';
} else if (isRestrictionCode || isRestrictionText) {
restrictedSet.add(icao);
reasons[icao] = n.iteme || 'Airspace restriction (NOTAM)';
}
}
const closedIcaos = [...closedSet];
const restrictedIcaos = [...restrictedSet];
console.log(`[NOTAM] ${notams.length} raw NOTAMs, ${closedIcaos.length} closures, ${restrictedIcaos.length} restrictions in ${((Date.now() - t0) / 1000).toFixed(1)}s`);
return { closedIcaos, restrictedIcaos, reasons, quotaExhausted: false, skipped: false };
}
// ─── Section 4: Aviation RSS news prewarmer ──────────────────────────────────
const AVIATION_RSS_FEEDS = [
{ url: 'https://www.flightglobal.com/rss', name: 'FlightGlobal' },
{ url: 'https://simpleflying.com/feed/', name: 'Simple Flying' },
{ url: 'https://aerotime.aero/feed', name: 'AeroTime' },
{ url: 'https://thepointsguy.com/feed/', name: 'The Points Guy' },
{ url: 'https://airlinegeeks.com/feed/', name: 'Airline Geeks' },
{ url: 'https://onemileatatime.com/feed/', name: 'One Mile at a Time' },
{ url: 'https://viewfromthewing.com/feed/', name: 'View from the Wing' },
{ url: 'https://www.aviationpros.com/rss', name: 'Aviation Pros' },
{ url: 'https://www.aviationweek.com/rss', name: 'Aviation Week' },
];
function parseRssItems(xml, sourceName) {
try {
const items = [];
const itemRegex = /<item[\s>]([\s\S]*?)<\/item>/gi;
let match;
while ((match = itemRegex.exec(xml)) !== null) {
const block = match[1];
const title = block.match(/<title[^>]*>([\s\S]*?)<\/title>/i)?.[1]?.replace(/<!\[CDATA\[([\s\S]*?)\]\]>/g, '$1').trim() || '';
const link = block.match(/<link[^>]*>([\s\S]*?)<\/link>/i)?.[1]?.replace(/<!\[CDATA\[([\s\S]*?)\]\]>/g, '$1').trim() || '';
const pubDate = block.match(/<pubDate[^>]*>([\s\S]*?)<\/pubDate>/i)?.[1]?.trim() || '';
const desc = block.match(/<description[^>]*>([\s\S]*?)<\/description>/i)?.[1]?.replace(/<!\[CDATA\[([\s\S]*?)\]\]>/g, '$1').trim() || '';
if (title && link) items.push({ title, link, pubDate, description: desc, _source: sourceName });
}
return items.slice(0, 30);
} catch {
return [];
}
}
async function seedAviationNews() {
const t0 = Date.now();
const now = Date.now();
const cutoff = now - 24 * 60 * 60 * 1000;
const allItems = [];
await Promise.allSettled(
AVIATION_RSS_FEEDS.map(async (feed) => {
try {
const resp = await fetch(feed.url, {
headers: { 'User-Agent': CHROME_UA, Accept: 'application/rss+xml, application/xml, text/xml, */*' },
signal: AbortSignal.timeout(8_000),
});
if (!resp.ok) return;
const xml = await resp.text();
allItems.push(...parseRssItems(xml, feed.name));
} catch { /* skip */ }
}),
);
const items = allItems.map((item) => {
let publishedAt = 0;
if (item.pubDate) try { publishedAt = new Date(item.pubDate).getTime(); } catch { /* skip */ }
if (publishedAt && publishedAt < cutoff) return null;
const snippet = (item.description || '').replace(/<[^>]+>/g, '').slice(0, 200);
return {
id: Buffer.from(item.link).toString('base64').slice(0, 32),
title: item.title, url: item.link, sourceName: item._source,
publishedAt: publishedAt || now, snippet, matchedEntities: [], imageUrl: '',
};
}).filter(Boolean).sort((a, b) => b.publishedAt - a.publishedAt);
console.log(`[News] ${items.length} articles from ${AVIATION_RSS_FEEDS.length} feeds in ${((Date.now() - t0) / 1000).toFixed(1)}s`);
return { items };
}
// ─── Section 5: Notification dispatch ────────────────────────────────────────
// Aviation: new entries into severe/major state trigger an aviation_closure notification.
// NOTAM: new ICAOs in the closed-set trigger a notam_closure notification.
// Both sources persist their prev-state to Redis so short-lived cron runs don't
// spam on every tick.
async function dispatchAviationNotifications(alerts) {
const severeAlerts = alerts.filter(a =>
a.severity === 'FLIGHT_DELAY_SEVERITY_SEVERE' || a.severity === 'FLIGHT_DELAY_SEVERITY_MAJOR',
);
const currentIatas = new Set(severeAlerts.map(a => a.iata).filter(Boolean));
const prev = await upstashGet(AVIATION_PREV_ALERTED_KEY);
const prevSet = new Set(Array.isArray(prev) ? prev : []);
const newAlerts = severeAlerts.filter(a => a.iata && !prevSet.has(a.iata));
// Persist current set for next tick's diff (24h TTL guards restarts).
await upstashSet(AVIATION_PREV_ALERTED_KEY, [...currentIatas], PREV_STATE_TTL);
for (const a of newAlerts.slice(0, 3)) {
await publishNotificationEvent({
eventType: 'aviation_closure',
payload: { title: `${a.iata}${a.city ? ` (${a.city})` : ''}: ${a.reason || 'Airport disruption'}`, source: 'AviationStack' },
severity: a.severity === 'FLIGHT_DELAY_SEVERITY_SEVERE' ? 'critical' : 'high',
variant: undefined,
dedupTtl: 14_400, // 4h
});
}
}
async function dispatchNotamNotifications(closedIcaos, reasons) {
const prev = await upstashGet(NOTAM_PREV_CLOSED_KEY);
const prevSet = new Set(Array.isArray(prev) ? prev : []);
const newClosures = closedIcaos.filter(icao => !prevSet.has(icao));
await upstashSet(NOTAM_PREV_CLOSED_KEY, closedIcaos, PREV_STATE_TTL);
for (const icao of newClosures.slice(0, 3)) {
await publishNotificationEvent({
eventType: 'notam_closure',
payload: { title: `NOTAM: ${icao}${reasons[icao] || 'Airport closure'}`, source: 'ICAO NOTAM' },
severity: 'high',
variant: undefined,
dedupTtl: 21_600, // 6h
});
}
}
// ─── Page-load bootstrap aggregate ───────────────────────────────────────────
// Mirror of the alerts-array assembly in
// server/worldmonitor/aviation/v1/list-airport-delays.ts (FAA + intl + NOTAM
// merge + Normal-operations filler from AIRPORTS). Keep the two builders in
// lockstep — when the RPC's NOTAM merge / filler shape / enum mapping changes,
// update both. Enum-string forms here match SEVERITY_MAP/DELAY_TYPE_MAP/REGION_MAP
// at the top of this file so consumers parse identically to the RPC's output.
const SEV_ORDER = ['normal', 'minor', 'moderate', 'major', 'severe'];
function buildNormalOpsAlert(airport) {
return {
id: `status-${airport.iata}`,
iata: airport.iata,
icao: airport.icao,
name: airport.name,
city: airport.city ?? '',
country: airport.country,
location: { latitude: airport.lat ?? 0, longitude: airport.lon ?? 0 },
region: REGION_MAP[airport.region] ?? 'AIRPORT_REGION_AMERICAS',
delayType: 'FLIGHT_DELAY_TYPE_GENERAL',
severity: 'FLIGHT_DELAY_SEVERITY_NORMAL',
avgDelayMinutes: 0,
delayedFlightsPct: 0,
cancelledFlights: 0,
totalFlights: 0,
reason: 'Normal operations',
source: 'FLIGHT_DELAY_SOURCE_COMPUTED',
updatedAt: Date.now(),
};
}
function buildNotamAlert(airport, reason, severity = 'severe', delayType = 'closure') {
const trimmed = reason.length > 200 ? reason.slice(0, 200) + '…' : reason;
return {
id: `notam-${airport.iata}`,
iata: airport.iata,
icao: airport.icao,
name: airport.name,
city: airport.city ?? '',
country: airport.country,
location: { latitude: airport.lat ?? 0, longitude: airport.lon ?? 0 },
region: REGION_MAP[airport.region] ?? 'AIRPORT_REGION_AMERICAS',
delayType: DELAY_TYPE_MAP[delayType] ?? 'FLIGHT_DELAY_TYPE_CLOSURE',
severity: SEVERITY_MAP[severity] ?? 'FLIGHT_DELAY_SEVERITY_SEVERE',
avgDelayMinutes: 0,
delayedFlightsPct: 0,
cancelledFlights: 0,
totalFlights: 0,
reason: trimmed,
source: 'FLIGHT_DELAY_SOURCE_NOTAM',
updatedAt: Date.now(),
};
}
function mergeNotamWithExistingAlert(airport, notamReason, existing, severity = 'severe', delayType = 'closure') {
if (!existing || existing.totalFlights === 0) {
return buildNotamAlert(airport, notamReason, severity, delayType);
}
const cancelRate = (existing.cancelledFlights / existing.totalFlights) * 100;
const notamCancelSev = cancelRate >= 50 ? 'severe' : cancelRate >= 25 ? 'major' : cancelRate >= 10 ? 'moderate' : 'minor';
const existingSevName = (existing.severity ?? '')
.replace('FLIGHT_DELAY_SEVERITY_', '').toLowerCase() || 'normal';
const effectiveSev = SEV_ORDER[Math.max(
SEV_ORDER.indexOf(existingSevName),
SEV_ORDER.indexOf(notamCancelSev),
SEV_ORDER.indexOf('moderate'), // notamFloor
)] ?? 'moderate';
const cancelText = `${Math.round(cancelRate)}% cxl`;
const reason = `NOTAM: ${notamReason.slice(0, 120)}${cancelText}`;
const trimmed = reason.length > 200 ? reason.slice(0, 200) + '…' : reason;
return {
...existing,
id: `notam-${airport.iata}`,
severity: SEVERITY_MAP[effectiveSev] ?? 'FLIGHT_DELAY_SEVERITY_MODERATE',
delayType: DELAY_TYPE_MAP[delayType] ?? 'FLIGHT_DELAY_TYPE_CLOSURE',
reason: trimmed,
source: 'FLIGHT_DELAY_SOURCE_NOTAM',
updatedAt: Date.now(),
};
}
// Parse src/config/airports.ts as text to recover the live RPC's MONITORED_AIRPORTS
// registry without a TS build step. The RPC iterates this for "Normal operations"
// filler; the seeder's local AIRPORTS list is a related-but-different set
// (carries `sources` for which feed covers each airport, omits some RPC-only
// entries, has some seeder-only entries). Today the two diverge by ~45 iata codes.
// We read both at runtime and union them by iata so the bootstrap covers every
// airport either registry knows about — match RPC output exactly + future-proof
// against drift in either direction.
//
// Memoised: read-once at module load. If parse fails (unexpected file shape, file
// missing in some packaging), we degrade to seeder's AIRPORTS only and warn —
// bootstrap is still produced, just without the RPC-only iata coverage.
let _monitoredAirportsCache = null;
function loadMonitoredAirportsFromConfigFile() {
if (_monitoredAirportsCache !== null) return _monitoredAirportsCache;
try {
const here = dirname(fileURLToPath(import.meta.url));
const path = join(here, '..', 'src', 'config', 'airports.ts');
const src = readFileSync(path, 'utf8');
const rows = [];
// Match rows of shape: { iata: 'XXX', icao: 'YYYY', name: '...', city: '...',
// country: '...', lat: N, lon: N, region: '...' }. Allows both single + double
// quotes for `name` (some rows use double-quote to embed apostrophes).
const rowRe = /\{\s*iata:\s*'([A-Z]{3})'\s*,\s*icao:\s*'([A-Z0-9]{3,4})'\s*,\s*name:\s*(?:'([^']*)'|"([^"]*)")\s*,\s*city:\s*(?:'([^']*)'|"([^"]*)")\s*,\s*country:\s*(?:'([^']*)'|"([^"]*)")\s*,\s*lat:\s*(-?\d+(?:\.\d+)?)\s*,\s*lon:\s*(-?\d+(?:\.\d+)?)\s*,\s*region:\s*'([a-z]+)'\s*\}/g;
let m;
while ((m = rowRe.exec(src)) !== null) {
rows.push({
iata: m[1],
icao: m[2],
name: m[3] ?? m[4],
city: m[5] ?? m[6],
country: m[7] ?? m[8],
lat: parseFloat(m[9]),
lon: parseFloat(m[10]),
region: m[11],
});
}
if (rows.length === 0) {
console.warn(`[Bootstrap] parsed 0 rows from ${path} — falling back to seeder AIRPORTS only`);
_monitoredAirportsCache = [];
return _monitoredAirportsCache;
}
_monitoredAirportsCache = rows;
return rows;
} catch (err) {
console.warn(`[Bootstrap] failed to parse src/config/airports.ts: ${err?.message || err} — falling back to seeder AIRPORTS only`);
_monitoredAirportsCache = [];
return _monitoredAirportsCache;
}
}
// Union the seeder's AIRPORTS with RPC's MONITORED_AIRPORTS by iata. Seeder rows
// win on conflict (they have the more recent canonical NOTAM/AviationStack meta).
// Logs a warning summary on first divergence so registry drift surfaces in cron
// logs without blocking writes.
let _filterRegistryWarnLogged = false;
function buildFillerRegistry() {
const monitored = loadMonitoredAirportsFromConfigFile();
const byIata = new Map();
for (const a of monitored) byIata.set(a.iata, a);
for (const a of AIRPORTS) byIata.set(a.iata, a); // seeder wins on conflict
if (!_filterRegistryWarnLogged) {
const seederIatas = new Set(AIRPORTS.map(a => a.iata));
const monitoredIatas = new Set(monitored.map(a => a.iata));
const monitoredOnly = [...monitoredIatas].filter(i => !seederIatas.has(i));
const seederOnly = [...seederIatas].filter(i => !monitoredIatas.has(i));
if (monitoredOnly.length > 0 || seederOnly.length > 0) {
console.warn(`[Bootstrap] registry drift: ${monitoredOnly.length} RPC-only iatas (${monitoredOnly.slice(0, 10).join(',')}${monitoredOnly.length > 10 ? '…' : ''}), ${seederOnly.length} seeder-only iatas (${seederOnly.slice(0, 10).join(',')}${seederOnly.length > 10 ? '…' : ''}). Bootstrap covers union of both (${byIata.size} airports).`);
}
_filterRegistryWarnLogged = true;
}
return [...byIata.values()];
}
// Build + write the page-load bootstrap aggregate. Pass `intlAlertsOverride` to
// use this-tick's intl from afterPublish (skips the Redis round-trip and avoids
// a one-tick lag); omit to fall back to the last-good intl in Redis (used by
// the pre-runSeed call so a current-tick intl failure still refreshes bootstrap).
async function writeDelaysBootstrap(intlAlertsOverride) {
try {
const [faaPayload, intlPayload, notamPayload] = await Promise.all([
upstashGetUnwrapped(FAA_KEY),
intlAlertsOverride ? Promise.resolve({ alerts: intlAlertsOverride }) : upstashGetUnwrapped(INTL_KEY),
upstashGetUnwrapped(NOTAM_KEY),
]);
const faaAlerts = Array.isArray(faaPayload?.alerts) ? faaPayload.alerts : [];
const intlAlerts = Array.isArray(intlPayload?.alerts) ? intlPayload.alerts : [];
const closedIcaos = Array.isArray(notamPayload?.closedIcaos) ? notamPayload.closedIcaos : [];
const restrictedIcaos = Array.isArray(notamPayload?.restrictedIcaos) ? notamPayload.restrictedIcaos : [];
const reasons = (notamPayload?.reasons && typeof notamPayload.reasons === 'object') ? notamPayload.reasons : {};
const allAlerts = [...faaAlerts, ...intlAlerts];
// Union of seeder AIRPORTS + RPC MONITORED_AIRPORTS so the bootstrap matches
// what the live RPC produces even when registries drift.
const fillerRegistry = buildFillerRegistry();
const existingIatas = new Set(allAlerts.map(a => a.iata));
const applyNotam = (icao, severity, delayType, fallback) => {
const airport = fillerRegistry.find(a => a.icao === icao);
if (!airport) return;
const reason = reasons[icao] || fallback;
if (existingIatas.has(airport.iata)) {
const idx = allAlerts.findIndex(a => a.iata === airport.iata);
if (idx >= 0) allAlerts[idx] = mergeNotamWithExistingAlert(airport, reason, allAlerts[idx], severity, delayType);
} else {
allAlerts.push(buildNotamAlert(airport, reason, severity, delayType));
existingIatas.add(airport.iata);
}
};
for (const icao of closedIcaos) applyNotam(icao, 'severe', 'closure', 'Airport closure (NOTAM)');
for (const icao of restrictedIcaos) applyNotam(icao, 'major', 'general', 'Airspace restriction (NOTAM)');
const alertedIatas = new Set(allAlerts.map(a => a.iata));
for (const airport of fillerRegistry) {
if (!alertedIatas.has(airport.iata)) allAlerts.push(buildNormalOpsAlert(airport));
}
const ok = await upstashSet(BOOTSTRAP_KEY, { alerts: allAlerts }, BOOTSTRAP_TTL);
if (ok) {
console.log(`[Bootstrap] wrote ${allAlerts.length} alerts to ${BOOTSTRAP_KEY} (faa=${faaAlerts.length}, intl=${intlAlerts.length}, notam-closed=${closedIcaos.length}, notam-restricted=${restrictedIcaos.length})`);
} else {
console.warn(`[Bootstrap] SET ${BOOTSTRAP_KEY} returned false`);
}
} catch (err) {
console.warn(`[Bootstrap] build/write error: ${err?.message || err}`);
}
}
// ─── Orchestration ───────────────────────────────────────────────────────────
// runSeed's primary key = INTL (largest spend, most-consumed). FAA + NOTAM +
// News are written as "extra keys" after the primary publish. Each has its own
// seed-meta override that matches api/health.js expectations.
// ─── Side-car seed runners ───────────────────────────────────────────────────
// Each secondary data source (FAA, NOTAM, news) seeds INDEPENDENTLY of the
// AviationStack intl path. A transient intl outage or missing AVIATIONSTACK_API
// MUST NOT freeze FAA/NOTAM/news writes — they have their own upstream sources
// (FAA ASWS, ICAO API, RSS) and their own consumers (list-airport-delays,
// loadNotamClosures, list-aviation-news).
//
// Each side-car: acquires its own Redis lock (distinct from intl's lock),
// fetches, writes data-key + seed-meta on success, extends TTL on failure,
// releases the lock in finally. Sequential so concurrent Railway cron fires
// don't stomp; each source's cost is independent so total wall time ≈ sum.
async function withLock(lockDomain, body) {
const runId = `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
const lockResult = await acquireLockSafely(lockDomain, runId, 120_000, { label: lockDomain });
if (lockResult.skipped) {
console.log(` ${lockDomain}: SKIPPED (Redis unavailable)`);
return;
}
if (!lockResult.locked) {
console.log(` ${lockDomain}: SKIPPED (lock held by another run)`);
return;
}
try {
await body();
} finally {
await releaseLock(lockDomain, runId);
}
}
async function runFaaSideCar() {
await withLock('aviation:faa', async () => {
try {
const faa = await seedFaaDelays();
if (faa?.alerts) {
await writeExtraKeyWithMeta(FAA_KEY, faa, FAA_TTL, faa.alerts.length, FAA_META_KEY);
console.log(`[FAA] wrote ${faa.alerts.length} alerts to ${FAA_KEY}`);
}
} catch (err) {
console.warn(`[FAA] fetch/write error: ${err?.message || err} — extending TTL`);
try { await extendExistingTtl([FAA_KEY, FAA_META_KEY], FAA_TTL); } catch {}
}
});
}
async function runNotamSideCar() {
await withLock('aviation:notam', async () => {
try {
const notam = await seedNotamClosures();
if (notam.skipped) return; // no ICAO_API_KEY
if (notam.quotaExhausted) {
// ICAO quota exhausted ("Reach call limit") — preserve the last known
// closure list by refreshing the data-key TTL + writing fresh meta
// with quotaExhausted=true. Keeps api/health.js (maxStaleMin: 240)
// green through the 24h backoff window. Matches pre-strip
// ais-relay.cjs:2805-2808 byte-for-byte.
try { await extendExistingTtl([NOTAM_KEY], NOTAM_TTL); } catch {}
try {
await upstashSet(NOTAM_META_KEY, { fetchedAt: Date.now(), recordCount: 0, quotaExhausted: true }, 604_800);
} catch (e) { console.warn(`[NOTAM] meta write error: ${e?.message || e}`); }
console.log(`[NOTAM] ICAO quota exhausted — extended data TTL + wrote fresh meta (quotaExhausted=true)`);
return;
}
await writeExtraKeyWithMeta(
NOTAM_KEY,
{ closedIcaos: notam.closedIcaos, restrictedIcaos: notam.restrictedIcaos, reasons: notam.reasons },
NOTAM_TTL,
notam.closedIcaos.length + notam.restrictedIcaos.length,
NOTAM_META_KEY,
);
console.log(`[NOTAM] wrote ${notam.closedIcaos.length} closures + ${notam.restrictedIcaos.length} restrictions to ${NOTAM_KEY}`);
try { await dispatchNotamNotifications(notam.closedIcaos, notam.reasons); }
catch (e) { console.warn(`[NOTAM] notify error: ${e?.message || e}`); }
} catch (err) {
console.warn(`[NOTAM] fetch/write error: ${err?.message || err} — extending TTL`);
try { await extendExistingTtl([NOTAM_KEY, NOTAM_META_KEY], NOTAM_TTL); } catch {}
}
});
}
async function runNewsSideCar() {
await withLock('aviation:news', async () => {
try {
const news = await seedAviationNews();
if (news?.items?.length > 0) {
await writeExtraKeyWithMeta(NEWS_KEY, news, NEWS_TTL, news.items.length);
console.log(`[News] wrote ${news.items.length} articles to ${NEWS_KEY}`);
}
} catch (err) {
console.warn(`[News] fetch/write error: ${err?.message || err}`);
}
});
}
// ─── Intl via runSeed ────────────────────────────────────────────────────────
// Intl (the paid-API, high-cost canonical) uses runSeed's full machinery:
// contract-mode envelope, retry-on-throw, graceful TTL-extend on failure,
// seed-meta freshness. When intl is unhealthy we throw to force runSeed into
// its catch path — which extends the INTL_KEY + seed-meta:aviation:intl TTLs
// and exits 0 without touching afterPublish. Consumers keep serving the
// last-good snapshot. FAA/NOTAM/news already ran via their side-cars and are
// independent — an intl outage does NOT freeze their freshness.
async function fetchIntl() {
const result = await seedIntlDelays();
if (!result.healthy || result.skipped) {
const why = result.skipped
? 'no AVIATIONSTACK_API key'
: 'systemic fetch failure (failures > successes)';
throw new Error(`intl unpublishable: ${why}`);
}
return result;
}
export function declareRecords(data) {
return data?.alerts?.length ?? 0;
}
// publishTransform reshapes seedIntlDelays' output into the canonical envelope
// shape consumers read ({ alerts: AirportDelayAlert[] }). declareRecords sees
// this transformed shape; afterPublish still receives the raw fetchIntl result.
function publishTransform(data) {
return { alerts: data?.alerts ?? [] };
}
async function afterPublishIntl(data) {
// CONTRACT: runSeed forwards the RAW fetchIntl() result here, NOT the
// publishTransform()'d shape. fetchIntl returns seedIntlDelays' output
// ({ alerts, healthy, skipped, ... }), so data.alerts is the same array
// publishTransform wraps into INTL_KEY. If publishTransform ever filters
// or mutates alerts (today it's a pass-through wrapper), this bootstrap
// write would silently diverge from INTL_KEY — keep them in lockstep.
try { await dispatchAviationNotifications(data.alerts); }
catch (e) { console.warn(`[Intl] notify error: ${e?.message || e}`); }
// Refresh the page-load bootstrap with this-tick intl. The pre-runSeed call
// in main() already wrote a bootstrap using last-good intl; this overwrite
// upgrades it to current.
await writeDelaysBootstrap(data?.alerts);
}
function validate(publishData) {
// Zero alerts is a valid steady state (no current airport disruptions) —
// but shape must be { alerts: [] } regardless.
return !!(publishData && Array.isArray(publishData.alerts));
}
// Entry point: run the three independent side-cars sequentially, then hand off
// to runSeed for intl. runSeed calls process.exit() on every terminal path, so
// it MUST be the last thing invoked in this file.
async function main() {
console.log('=== Aviation Seeder (side-cars + intl) ===');
await runFaaSideCar();
await runNotamSideCar();
await runNewsSideCar();
// Pre-runSeed bootstrap write: ensures the page-load aggregate refreshes even
// if intl fetch fails this tick (runSeed's catch-path skips afterPublish).
// Uses last-good intl from Redis; afterPublishIntl will overwrite with fresh
// intl on success.
await writeDelaysBootstrap();
return runSeed('aviation', 'intl', INTL_KEY, fetchIntl, {
validateFn: validate,
ttlSeconds: INTL_TTL,
sourceVersion: 'aviationstack',
schemaVersion: 3,
declareRecords,
publishTransform,
afterPublish: afterPublishIntl,
maxStaleMin: 90,
zeroIsValid: true,
});
}
main().catch((err) => {
const cause = err.cause ? ` (cause: ${err.cause.message || err.cause.code || err.cause})` : '';
console.error('FATAL:', (err.message || err) + cause);
process.exit(1);
});