Skip to content

fix: bundle a pinned, filtered airports snapshot and scale airport and port points with zoom (#9) - #23

Merged
GeoBradDev merged 19 commits into
mainfrom
issue-9-airports-preprocessed-data
Nov 8, 2026
Merged

GeoBradDev merged 19 commits into
mainfrom
issue-9-airports-preprocessed-data

Conversation

@GeoBradDev

Copy link
Copy Markdown
Owner

Airports no longer come from a 12.7 MB unpinned CSV on raw.githubusercontent.com parsed on the main thread. A local build script (npm run data:airports, scripts/build-airports.mjs) fetches OurAirports airports.csv pinned to commit 64af12951161ed52bc733eb4101224643c48850a, drops closed airports, heliports, seaplane bases, balloonports, unknown types, and rows without valid coordinates, and writes a compact columnar public/data/airports.json (1,967,855 bytes, 623,559 bytes gzip -9) plus public/data/airports.meta.json (source URL, commit, fetch time in UTC, per-type counts, license: public domain). Both files are committed; the app fetches /data/airports.json same-origin, like ports.geojson. CI does not run the script.

  • 86,347 source rows, 48,072 kept (large 1,175, medium 4,107, small 42,790); 38,275 dropped by type, 0 for coordinates.
  • Points layer draws only large and medium airports (5,282), sized in meters with a 1.5 to 8 px clamp. The AIRPORTS count is the number of points drawn.
  • The heatmap bins all 48,072 included airports (large, medium, and small).
  • Ports layer gets the same zoom scaling (6000 m, 2 to 8 px) so country labels stay legible at zoom 2 with all layers on.
  • The script's CSV parser is a strict RFC 4180 parser (quoted commas, doubled quotes, quoted newlines, CRLF, BOM; throws with a line number on malformed quoting) and refuses to write when nothing is included.
  • No Web Worker: parsing measured at about 3 to 4 ms for JSON.parse plus 2 to 5 ms for decode in the browser, so it never forms a long task (see evidence).

Plan

docs/plans/2026-11-07-issue-9-airports-preprocessed-data.md, first committed at 0e57bfe, last revised at d592ad9. Revisions during the work, each in the commit that departed: App test mocks moved into the hook step so every commit stays green; counts also record each upstream input type; the long task analysis and the decision not to add the Worker; review fixes (ports radius, strict parser and empty-output guard, realpath main guard, full UTC fetchedAt).

Test evidence

Gate (worktree, final tree before push), all exit 0:

  • npm ci: found 0 vulnerabilities
  • npm run lint: clean
  • npm test: Test Files 49 passed (49), Tests 437 passed (437)
  • npm run build: built (the existing chunk size warning over 500 kB predates this change)
  • npm audit --omit=dev --audit-level=high: found 0 vulnerabilities

Live drive (production preview, headless Chrome 157, --enable-unsafe-webgpu --use-webgpu-adapter=swiftshader, 1440 x 900), by the implementer and again by an independent verifier:

  • Network: no request to raw.githubusercontent.com (or any GitHub host); /data/airports.json same-origin, 200, 626,850 bytes transferred (gzip).
  • Panel: Airports: 5,282 FEATURES, Sea Ports: 1,081 FEATURES, Heatmap: READY, header SYS NOMINAL.
  • Main thread: JSON.parse of the real file 3.1 to 4.1 ms, decode 1.6 to 4.6 ms. Long tasks over 50 ms do occur during startup (longest 122 to 181 ms across runs), but none is parsing: with the airports response held 5 s they start only after res.json() resolves; a CPU profile puts 1.7 ms of them in decode and the rest in native WebGL work in deck.gl's first draw; and with the Airports and Heatmap layers unticked (data still fetched, parsed, decoded, heatmap computed) no long task follows the airports response. With airports blocked entirely, MapLibre and deck.gl init still produce 65 to 182 ms tasks.
  • Zoom 2, all layers on: country labels legible (UK, Denmark, Sweden, Norway, Germany, France, Austria, Italy, Greece checked; before the ports fix they were covered).
  • Zoom 6 and 8: clicking opened Frankfurt Main (large), Frankfurt-Egelsbach (medium), and Rotterdam and Oostende (ports) popups. Escape closes the popup and returns focus.
  • Retry (No loading, error, or degraded states: failed data loads leave an empty map and SYS NOMINAL #10): aborting the first /data/airports.json showed LOAD FAILED, RETRY, heatmap NO AIRPORT DATA, SYS DEGRADED: AIRPORTS FAILED; RETRY recovered to 5,282 features, heatmap READY, SYS NOMINAL. Same for ports. A malformed body shows LOAD FAILED with "Airport data is malformed" and recovers on retry.

Review findings

First round: three Claude finders (bugs and compliance; security and data safety; outside the diff) and two Fable finders (bugs and compliance; edge cases), then one confirmation finder. Fixed, each with a test committed red first:

  • Ports layer still 5 px pixel radius covered European country labels at zoom 2 with all layers on (coordinator, from the screenshot). Fixed: meters with 2 to 8 px clamp.
  • Lenient CSV parser: a stray quote in an unquoted field or an unterminated quote swallowed the rest of the file silently, and the build script would write the result, even zero rows (Claude security finder; Fable edge-case finder). Fixed: strict parser with line numbers, checkCounts guard before any write, warning on unknown types.
  • inputByType in a plain object lost a __proto__ type (Claude finder note). Fixed with a Map.
  • README claimed large airports draw at twice the radius of medium ones, false outside a narrow zoom band because of the pixel clamps (Claude outside-the-diff finder; Fable edge-case finder; Fable bugs finder note). Fixed wording.
  • Build script main guard silently did nothing when invoked through a symlinked path (Fable bugs finder note). Fixed: compares real paths.
  • fetchedAt was a UTC date without a zone marker, one day ahead of the local run (Fable edge-case finder). Fixed: full ISO timestamp, README says UTC.

Delta finder on the fixes: no findings. Verifier: no findings.

Dismissed or left as notes:

  • Historical design docs under docs/superpowers/ and older plans still describe the runtime CSV: historical records, not current docs.
  • decodeAirports does not check version; the plan did not ask for it.
  • Root-relative /data/airports.json breaks under a non-root Vite base: same as the existing ports URL, and no base is set.
  • The South Pole medium airport (lat -90) never renders in Web Mercator: pre-existing.
  • The empty-output guard does not catch a truncated download that still parses cleanly: the committed-data test (more than 40,000 rows) catches it in CI.
  • Parser error line numbers are off for bare-CR files inside quoted fields: the upstream file uses LF.

Docs reconciled

  • README.md: airports, ports, and heatmap bullets; network access; scripts table (npm run data:airports); CI note; data sources row (pinned commit, fetch time, filters, counts, sizes, public domain); project layout; known limitations.
  • src/config/sources.js: AIRPORTS_DATA_URL replaces OURAIRPORTS_CSV_URL, comment names the generated file, script, pinned commit, and license.
  • CLAUDE.md: no change needed because it only says data URLs live in src/config/sources.js, which is still true.
  • ADR: none.

Closes #9

🤖 Generated with Claude Code

https://claude.ai/code/session_01AELE7jB1i3uqqrHu3SjJjf

GeoBradDev and others added 19 commits November 7, 2026 22:37
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AELE7jB1i3uqqrHu3SjJjf
…#9)

Fetches airports.csv at ourairports-data commit 64af129, keeps large,
medium, and small airports with valid coordinates rounded to 3 decimals,
and writes public/data/airports.json (1,967,855 bytes, 623,559 gzip,
48,072 rows) plus airports.meta.json. CI does not run the script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AELE7jB1i3uqqrHu3SjJjf
Adds decodeAirports and selectMajorAirports, switches useAirports to the
bundled columnar JSON, and returns majorAirports computed once per load.
App test fetch mocks move to AIRPORTS_DATA_URL with JSON bodies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AELE7jB1i3uqqrHu3SjJjf
…ay legible (#9)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AELE7jB1i3uqqrHu3SjJjf
Regenerated with npm run data:airports; public/data/airports.json is
byte-identical to the committed file, only airports.meta.json changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AELE7jB1i3uqqrHu3SjJjf
@GeoBradDev
GeoBradDev merged commit 0377769 into main Nov 8, 2026
1 check passed
@GeoBradDev
GeoBradDev deleted the issue-9-airports-preprocessed-data branch November 8, 2026 05:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Airports layer downloads a 12.7 MB unpinned third-party CSV and parses it on the main thread

1 participant