Skip to content

fix: harden HTTP limits, redirects, ArcGIS paging, and parsers - #47

Merged
Reddimus merged 9 commits into
mainfrom
fix/redirect-size-limit
Nov 20, 2026
Merged

Reddimus merged 9 commits into
mainfrom
fix/redirect-size-limit

Conversation

@Reddimus

Copy link
Copy Markdown
Owner

Summary

Fixes from a full audit of the repo, for v0.4.8. Fixes #45.

  • HTTP:
  • ArcGIS:
    • One Web Mercator expansion budget and one byte cap cover each call, across pages and both fire-weather layers. Before this, a server that kept paging could make one call hold gigabytes.
    • ArcGIS 429/5xx errors in HTTP 200 bodies are retried per page.
    • A std::exception such as bad_alloc in any client call is ParseError instead of escaping.
  • Parsers:
    • Features whose fields are not an object are skipped.
    • "nan"/"inf" strings read as 0.
    • A null latestWkid falls back to wkid.
  • Errors and retry:
    • A Retry-After past int64 is capped.
    • Only 500-599 are ServerError.
    • The daily quota resets when the clock steps back.
  • Build:
    • Glaze is linked as a build-local target, so a packaged Glaze configures and installs.
    • GoogleTest and Benchmark are pinned by commit.
    • Linux CI installs libssl-dev so the new TLS redirect tests run.
  • Tools and docs:
    • The auto audit checks every auto.
    • The fixture checker recurses and rejects duplicates.
    • Release notes match the tag literally.
    • The security docs point at private vulnerability reporting, now enabled.

Public API changes:

  • The error codes above change.
  • The libcurl floor is now 7.86.
  • Exceptions not derived from std::exception, which only a custom transport can throw, still propagate.

Testing

  • make test, make lint, make lint-md, make fixtures-check, and make test-consumers pass
  • CHANGELOG.md has an [Unreleased] entry for user-visible changes
  • Public API changes are called out above

Where it ran:

  • macOS (arm64, Apple Clang 21, libcurl 8.7.1): all five gates, ASan/UBSan, TSan, and the universal arm64+x86_64 build at 13.4.
  • Linux, natively: GCC 15 with libcurl 8.18.
  • Linux, in ubuntu:24.04 (libcurl 8.5.0): GCC 13, Clang 18 with libc++, and clang-tidy-18.
  • On Linux, tests, sanitizers and consumers passed with the TLS tests enabled.

#45: reproduced on libcurl 8.5.0, 8.7.1, 8.18 and 8.23 before the fix, with new loopback tests.

Live: tools/verify_arcgis_metadata.py (39 layers) and every example pass on both machines.

Review: an independent review found three issues, fixed in the last commit, and the re-review approved.

…anent failures

Fixes #45. CURLOPT_MAXFILESIZE_LARGE compared every Content-Length in the
redirect chain, and encoded lengths, against the decoded-body limit. The
write callback only sees the final body, so the early refusal moves there.

Malformed URLs, disallowed schemes, and redirect loops were NetworkError,
which with_retry repeats. Retry-After values past int64 were ignored, and
statuses above 599 were ServerError.
…ferences

Found in the audit: conditional-intensity and day 1-2 fire-weather layers
kept features with null or missing fields, numeric strings "nan" and "inf"
reached public doubles, and a null latestWkid or geometry spatialReference
hid a valid reference. The locale test now drops dn so it can fail.
Glaze came in through glaze_SOURCE_DIR, which is empty when CMake finds an
installed Glaze, so that configuration failed. Linking the target behind
BUILD_LOCAL_INTERFACE keeps it out of the export set and brings Glaze's
MSVC options; its headers stay system headers.

Also test the MapServer and static-feed URL of each typed query, skip the
benchmark self-check's library probe on MSVC, and fix stale docs.
Private vulnerability reporting was off while SECURITY.md and the issue
templates sent reporters to it; it is now enabled and the docs say how it
works. GoogleTest and Benchmark are pinned by commit, the Makefile's
markdownlint-cli2 matches the CI action, and .gitignore covers more secret
files.
…gs literally

The explicit-type audit only saw auto at line start and treated any
iterator-looking substring as an exemption; it now checks each auto token
and exempts only an initializer that is one iterator call. The fixture
checker recurses and rejects duplicate entries, release notes match the tag
literally, and the tools run on Python 3.9.
An https request followed a redirect to plain http and returned that body
as a normal response. Redirects from https now allow only https, and the
refusal is InvalidRequest. A TLS loopback server (optional OpenSSL 3, test
only) covers it; Linux CI installs libssl-dev so those tests run.

libcurl 7.85's curl/header.h lacks extern "C", so curl_easy_nextheader
never linked there; 7.86 is the real floor.
…0 bodies

Each page got a fresh Web Mercator expansion budget and its own response
cap, so a server that kept paging could make one call hold gigabytes. One
budget now covers the call, both fire-weather layers included, and the
pages together may not exceed max_response_bytes.

ArcGIS 429 and 5xx errors sent with HTTP 200 are retried like the same
status, one page at a time; a missing code is no longer reported as 400;
and a std::exception such as bad_alloc in any client call is a ParseError
instead of escaping.
From independent review. libcurl fetches https:/host as absolute HTTPS, but
is_absolute_url wanted two slashes, so with a base_url the request went to
the plaintext base. Error bodies over 64 KiB are no longer parsed into a
JSON tree, and golden coverage now walks fixture subdirectories like the
checksum tool does.
clang-tidy's bugprone-implicit-widening-of-multiplication-result flagged
64 * 1024 being computed in int.
@Reddimus
Reddimus merged commit bfe099d into main Nov 20, 2026
14 checks passed
@Reddimus
Reddimus deleted the fix/redirect-size-limit branch November 20, 2026 21:50
@Reddimus Reddimus mentioned this pull request Nov 20, 2026
3 tasks done
Reddimus added a commit that referenced this pull request Nov 21, 2026
## Summary

Sets the version to 0.4.8, dates the changelog section, and moves the
README `GIT_TAG` to `v0.4.8`. The changes are in #47.

## Testing

- [x] `make test-consumers` passes (checks both README versions)
- [x] `make lint-md` passes
- [x] Public API changes: none beyond #47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HttpClient rejects a request when a redirect hop's Content-Length exceeds max_response_bytes, even though the final response body is tiny

1 participant