diff --git a/README.md b/README.md index 7f06c51..88f2633 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,8 @@ Methods use fixed API routes; constructing a client opens no network connection. The caller supplies a Foldr HTTPS site and an access token. Each client has its own HTTP session and identity. Foldr decides which shares and files that identity may access, including shares backed by another storage account. +Access tokens must be nonempty visible ASCII without whitespace; invalid values +are rejected before any request. ## Install @@ -131,8 +133,15 @@ For direct downloads, the client follows at most five redirects, sending **no Foldr Authorization or cookies** on any redirect hop, including the same host. Only HTTPS destinations without URL credentials are accepted. Omitted `direct` honors the share's setting; `direct=False` requests bytes through Foldr. -Signed storage URLs and Location headers -are credentials: keep them out of logs. Stream chunks use bounded buffering; +Choose `direct=False` when downloads need Foldr to check access on each new +request. It does not revoke storage URLs issued earlier; those can outlive an app +disconnect or permission change. An optional `destination_policy` callback on +`FoldrClient` checks every download redirect, and on `DirectUpload` checks each +storage send. See [client safety configuration](docs/client-safety.md). +Signed storage URLs and Location headers are credentials: keep them out of logs. +HTTPX/HTTPCore logs can expose these URLs even when exception messages are safe; +the opt-in `HTTPLogFilter` suppresses their records on configured output handlers. +Stream chunks use bounded buffering; collecting them into a list or joining them makes memory use the caller's concern. Content requests require identity encoding. A server that sends compressed content despite that request is rejected before decompression, preserving bounded memory @@ -251,6 +260,13 @@ bytes rather than being decompressed. Exception messages omit provider text and URLs; error attributes and headers may contain sensitive data, so redact before logging. Failures during an error body's transfer may surface as transport errors. +Successful JSON responses are limited to 8 MiB for `FoldrClient`/`FoldrSite` and +64 KiB for `FoldrOAuth`. Set the positive integer `max_json_bytes` constructor +option to adjust a limit. Unexpected compression is rejected before decoding; +oversized responses raise `ProtocolError` and close without retry. Failed +mutation/token responses retain `outcome_uncertain=True`. File streams are not +subject to the JSON limit. See [configuration examples](docs/client-safety.md). + There are no implicit retries, sleeps, snapshot restarts or permission checks. Foldr's 401/403/404 and storage connection errors reach the caller unchanged. Use structured type/code/status rather than matching human-readable detail. diff --git a/docs/api-reference.md b/docs/api-reference.md index 8ef3407..6b459ef 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -71,7 +71,8 @@ ETags remain opaque, including weak markers and quotes. ## Site bootstrap and OAuth -`FoldrSite(site, *, timeout=60.0, transport=None)` owns a credential-free session: +`FoldrSite(site, *, timeout=60.0, transport=None, max_json_bytes=8388608)` owns a +credential-free session: | Method | Endpoint | Result | | --- | --- | --- | @@ -82,7 +83,7 @@ ETags remain opaque, including weak markers and quotes. | `register(metadata)` | POST `/oauth/register`, JSON | ApiResponse 201 | `FoldrOAuth(site, *, client_id, client_secret=None, auth_method="none", timeout=60.0, -transport=None)` provides these explicit operations: +max_json_bytes=65536, transport=None)` provides these explicit operations: | Method | Protocol | | --- | --- | @@ -91,7 +92,8 @@ transport=None)` provides these explicit operations: | `refresh(refresh_token, *, scopes=None)` | One POST `/oauth/token`; returns rotated tokens for caller to save | | `revoke(token, *, token_type_hint=None)` | POST `/oauth/revoke`; accepts empty 200, returns None | -`DirectUpload(session, *, timeout=60.0, transport=None)` sends bytes to the exact +`DirectUpload(session, *, timeout=60.0, transport=None, destination_policy=None)` +sends bytes to the exact storage method/URL/headers from createUploadSession. `send(content, *, content_length, content_range=None)` returns status/headers with `data=None` and does not interpret storage response bodies. Use a separate context from Foldr; @@ -100,6 +102,25 @@ transport and support context management and `close()`. ## Request behavior and limits +`FoldrClient(site, *, access_token, timeout=60.0, transport=None, +max_json_bytes=8388608, destination_policy=None)` owns one authenticated session. +`max_json_bytes` is a positive integer limit on each successful JSON response; +it is enforced while reading actual bytes, regardless of Content-Length. +Compressed JSON is refused before decompression. OAuth defaults to 64 KiB; +API and site/discovery default to 8 MiB. These limits do not cap file streams, +change the 64 KiB error cap, or read bodies on HEAD/204/304 responses. + +`destination_policy` is an optional synchronous `Callable[[httpx.URL], bool]`. +It must return `True` to approve each download redirect or each direct-upload +send. Rejection or a callback exception raises a sanitized `ProtocolError` +before contacting that destination. It does not validate the initial Foldr site +or resolve DNS. With no callback, existing HTTPS/private storage support remains. + +`HTTPLogFilter()` is a standard logging filter, exported by `foldr_client`, that +suppresses HTTPX/HTTPCore records at all levels. It must be attached explicitly to +every applicable output handler; importing the client changes no logging setup. +See [client safety configuration](client-safety.md) for examples and limits. + Download `direct=None` omits the parameter and honors the share's setting; `direct=False` requests bytes through Foldr. Upload-session `content_type=None` omits that optional field. Optional query/body values set to `None` are generally diff --git a/docs/client-safety.md b/docs/client-safety.md new file mode 100644 index 0000000..959b252 --- /dev/null +++ b/docs/client-safety.md @@ -0,0 +1,127 @@ +# Client safety configuration + +The defaults require HTTPS, verify TLS, isolate credentials, and avoid automatic +retries. The settings below control JSON memory use, HTTP dependency logs, and +which storage destinations an application will contact. + +## JSON response limits + +| Client | Default maximum successful JSON body | +| --- | --- | +| `FoldrClient` | 8 MiB (8,388,608 bytes) | +| `FoldrSite` (including OpenAPI and registration) | 8 MiB | +| `FoldrOAuth` | 64 KiB (65,536 bytes) | + +Set `max_json_bytes` to a positive integer on the relevant constructor when an +expected response needs a different bound: + +```python +from foldr_client import FoldrClient, FoldrOAuth, FoldrSite + +with FoldrClient(site, access_token=token, max_json_bytes=4 * 1024 * 1024) as client: + account = client.me() + +with FoldrSite(site, max_json_bytes=16 * 1024 * 1024) as public: + contract = public.openapi().data + +with FoldrOAuth(site, client_id=client_id, max_json_bytes=128 * 1024) as oauth: + authorization = oauth.authorize(callback_uri, scopes=["foldr.read"]) +``` + +The limit counts actual serialized JSON bytes, not Content-Length. It bounds the +body retained for parsing; the parsed Python objects consume additional memory. +Unexpected compression is rejected before reading/decompression, even if the +compressed body is small. Oversize or malformed JSON raises `ProtocolError`, +closes the response and does not retry. On a mutation or token exchange, +`outcome_uncertain=True` means the request may already have taken effect: reconcile +the result rather than blindly replaying it. + +File downloads continue to stream without this JSON cap. Error bodies keep their +separate 64 KiB wire-byte cap. HEAD, 204 and 304 do not consume response bodies. +Custom transports are responsible for any buffering they perform before returning +a response; the client cannot prevent a transport from preloading a whole body. + +## Keep signed URLs out of dependency logs + +HTTPX INFO records contain complete request URLs. HTTPCore DEBUG records can +include headers and other sensitive details. Signed storage credentials can live +in URL paths as well as query strings, so removing a query string alone is not +sufficient. Safe exception messages do not sanitize these dependency logs. + +`HTTPLogFilter` suppresses all records from `httpx`, `httpcore` and their child +loggers. Unrelated application logs remain visible. For a new application's +startup logging configuration: + +```python +import logging +from foldr_client import HTTPLogFilter + +output = logging.StreamHandler() +output.addFilter(HTTPLogFilter()) +logging.basicConfig(level=logging.INFO, handlers=[output]) +``` + +If logging is already configured, `basicConfig` may do nothing: add the filter to +each existing output handler instead. This includes handlers attached directly to +dependency loggers, file handlers, telemetry handlers and queue handlers. Apply +the filter before records leave the process. A filter on the root logger alone +does not filter records propagated from descendant loggers. + +Importing or constructing the filter makes no global logging changes. Suppression +also removes HTTP dependency error diagnostics; SDK exceptions still reach the +caller. The filter does not redact application logs, custom transport logs, +external tracing, or printed response data. Never log tokens, authorization +contexts, upload sessions, signed URLs or raw sensitive error attributes yourself. + +## Restrict storage destinations when needed + +Both `FoldrClient` and `DirectUpload` accept `destination_policy`. It receives an +`httpx.URL` and must return the literal boolean `True` to approve a destination. +It runs for every download redirect, including subsequent and same-origin hops, +and before every direct-upload send, including each chunk. The existing HTTPS and +no-URL-credentials checks still apply. + +For an application with a known storage host: + +```python +import httpx +from foldr_client import DirectUpload, FoldrClient + + +def approved_storage(url: httpx.URL) -> bool: + return url.host == "storage.example.com" and url.port in (None, 443) + + +with FoldrClient(site, access_token=token, destination_policy=approved_storage) as client: + with client.stream_content(share_id, path) as stream: + for chunk in stream.iter_bytes(): + consume(chunk) + +with DirectUpload(upload_session, destination_policy=approved_storage) as storage: + storage.send(body, content_length=upload_session["size"]) +``` + +Use the hosts appropriate to your storage, including any legitimate CDN hops. +The callback sees the full sensitive URL: do not log it. Rejection, a non-boolean +approval, or a callback exception raises a generic `ProtocolError` before a +request to that destination, without consuming upload bytes. Earlier upload +chunks or requests may already have succeeded. Callback exceptions are suppressed +from normal formatted tracebacks so they cannot echo the signed URL. + +The policy is opt-in. Without it, HTTPS destinations including private storage +continue to work. It does not control the initial configured Foldr site, perform +DNS resolution, or prevent DNS rebinding. An application accepting untrusted +sites must validate them separately. Deployments requiring network isolation +also need appropriate egress controls; URL hostname checks are not a firewall. + +## Revocation and version guarantees + +`stream_content(..., direct=False)` requests Foldr-proxied bytes, allowing Foldr +to check access on each new request. It does not revoke previously issued storage +URLs, which can remain valid after an app disconnect or permission change. Direct +URL validity is backend-dependent; the API's `expires_at` is not a guaranteed +maximum lifetime. Neither mode can retract bytes already downloaded. + +The SDK preserves server ETags unchanged. Their concurrency guarantees depend on +the storage backend; weak SMB size/mtime tags are not strong content-version +identifiers. See [request keys and ETags](writes.md#request-keys-and-etags). diff --git a/docs/writes.md b/docs/writes.md index e095d44..1180530 100644 --- a/docs/writes.md +++ b/docs/writes.md @@ -35,6 +35,14 @@ locally because these methods require an explicit version. Other validation and conflict handling belong to the server. A failed request is raised to the caller; the client does not force the change or retry it. +Concurrency protection depends on the storage backend. Foldr can return weak +size/mtime ETags for SMB, where different contents of the same size within the +timestamp resolution may share a tag. The client preserves the tag, including +`W/`, exactly as the API requires; it does not turn it into a strong content +version or guarantee atomic server-side checks. Do not remove `W/` or substitute +a locally invented tag. Workflows needing stronger assurance must use server +and storage controls that provide it. + ## Upload and overwrite ```python @@ -204,3 +212,10 @@ retry or resume attempt. The session's explicit storage authorization headers are retained. Keep the session URL and headers out of logs and treat them as credentials. `cancel_upload_session(id)` provides explicit session cancellation; the server determines what cancellation can remove. + +Storage conditions such as a session-supplied `If-None-Match: *` are preserved +unchanged. A storage 412 is raised as `FoldrError`; the helper never removes a +condition or retries to force an upload. Create-only enforcement belongs to the +server/storage signature, not to a caller-controlled SDK. Configure +`destination_policy` and HTTP log filtering as described in +[client safety configuration](client-safety.md). diff --git a/src/foldr_client/__init__.py b/src/foldr_client/__init__.py index 54c2994..2e4fa92 100644 --- a/src/foldr_client/__init__.py +++ b/src/foldr_client/__init__.py @@ -9,6 +9,7 @@ StreamError, TransportError, ) +from .logging import HTTPLogFilter from .models import ApiResponse, ContentStream, Page from .oauth import Authorization, FoldrOAuth from .site import FoldrSite @@ -24,6 +25,7 @@ "FoldrError", "FoldrOAuth", "FoldrSite", + "HTTPLogFilter", "OAuthError", "Page", "ProtocolError", diff --git a/src/foldr_client/_destinations.py b/src/foldr_client/_destinations.py new file mode 100644 index 0000000..7c4561e --- /dev/null +++ b/src/foldr_client/_destinations.py @@ -0,0 +1,19 @@ +"""Opt-in application policy for storage requests; not a DNS or network firewall.""" + +from collections.abc import Callable + +import httpx + +from .errors import ProtocolError + + +def check_destination(url: str, policy: Callable[[httpx.URL], bool] | None) -> None: + """Require an explicit approval without exposing signed URLs in errors.""" + if policy is None: + return + try: + approved = policy(httpx.URL(url)) + except Exception: + raise ProtocolError("Storage destination policy failed") from None + if approved is not True: + raise ProtocolError("Storage destination rejected by policy") diff --git a/src/foldr_client/_http.py b/src/foldr_client/_http.py index a53762f..a40da89 100644 --- a/src/foldr_client/_http.py +++ b/src/foldr_client/_http.py @@ -69,16 +69,6 @@ def site_url(site: str) -> str: return site.rstrip("/") -def require_object(response: httpx.Response) -> dict[str, Any]: - try: - value = response.json() - except (ValueError, UnicodeError): - raise ProtocolError("Expected a JSON object") from None - if not isinstance(value, dict): - raise ProtocolError("Expected a JSON object") - return value - - def raise_response(response: httpx.Response, *, oauth: bool = False) -> None: body = bytearray() truncated = False @@ -120,7 +110,11 @@ def __init__( *, timeout: float | httpx.Timeout = 60.0, transport: httpx.BaseTransport | None = None, + max_json_bytes: int = 8 * 1024 * 1024, ): + if type(max_json_bytes) is not int or max_json_bytes <= 0: + raise ValueError("max_json_bytes must be a positive integer") + self.max_json_bytes = max_json_bytes # A private session per instance. Explicit Request objects never inherit # cookies, auth, URL credentials or headers from earlier responses. self.http = httpx.Client(timeout=timeout, transport=transport, trust_env=False) @@ -151,6 +145,33 @@ def send( except OSError: raise TransportError(operation, outcome_uncertain=uncertain) from None + def _json_value(self, response: httpx.Response, *, uncertain: bool) -> Any: + # Reject compression before reading: iter_bytes/read would decompress a + # small wire response into potentially unbounded memory. Content-Length + # is not trusted; enforce the limit against the bytes actually received. + if response.headers.get("content-encoding", "identity").strip().lower() != "identity": + raise ProtocolError( + "JSON response must honor Accept-Encoding: identity", outcome_uncertain=uncertain + ) + body = bytearray() + # Pre-buffered responses can be supplied by custom/test transports. The + # transport owns any buffering before this method receives the response. + chunks = ( + (response.content,) + if response.is_stream_consumed + else response.iter_raw(chunk_size=min(8192, self.max_json_bytes + 1)) + ) + for chunk in chunks: + if len(chunk) > self.max_json_bytes - len(body): + raise ProtocolError( + "JSON response exceeds max_json_bytes", outcome_uncertain=uncertain + ) + body.extend(chunk) + try: + return json.loads(body) + except (ValueError, UnicodeError, RecursionError): + raise ProtocolError("Expected a JSON response", outcome_uncertain=uncertain) from None + def read_response( self, response: httpx.Response, @@ -167,13 +188,7 @@ def read_response( if head or response.status_code in (204, 304): data = None else: - response.read() - try: - data = response.json() - except (ValueError, UnicodeError): - raise ProtocolError( - "Expected a JSON response", outcome_uncertain=uncertain - ) from None + data = self._json_value(response, uncertain=uncertain) if not isinstance(data, (dict, list)): raise ProtocolError( "Expected a JSON object or array", outcome_uncertain=uncertain @@ -197,8 +212,10 @@ def read_json( try: if response.status_code != 200: raise_response(response, oauth=oauth) - response.read() - return require_object(response) + data = self._json_value(response, uncertain=uncertain) + if not isinstance(data, dict): + raise ProtocolError("Expected a JSON object", outcome_uncertain=uncertain) + return data except httpx.HTTPError: raise TransportError(operation, outcome_uncertain=uncertain) from None finally: diff --git a/src/foldr_client/client.py b/src/foldr_client/client.py index a1d8452..5e36a83 100644 --- a/src/foldr_client/client.py +++ b/src/foldr_client/client.py @@ -1,13 +1,14 @@ """Synchronous operations specific to Foldr API v1.""" import json -from collections.abc import Iterator +from collections.abc import Callable, Iterator from contextlib import contextmanager from typing import Any from urllib.parse import quote, urljoin, urlsplit import httpx +from ._destinations import check_destination from ._http import HTTP, UploadContent, iter_content, raise_response, site_url from .errors import ProtocolError, TransportError from .models import ApiResponse, ContentStream, Page @@ -27,13 +28,20 @@ def __init__( access_token: str, timeout: float | httpx.Timeout = 60.0, transport: httpx.BaseTransport | None = None, + max_json_bytes: int = 8 * 1024 * 1024, + destination_policy: Callable[[httpx.URL], bool] | None = None, ): self.site = site_url(site) self.api_url = self.site + "/api/v1" - if not access_token or any(c in access_token for c in "\r\n"): - raise ValueError("access_token must be nonempty and contain no newlines") + if ( + not isinstance(access_token, str) + or not access_token + or any(not 33 <= ord(c) <= 126 for c in access_token) + ): + raise ValueError("access_token must be nonempty visible ASCII without whitespace") self._access_token = access_token - self._connection = HTTP(timeout=timeout, transport=transport) + self._destination_policy = destination_policy + self._connection = HTTP(timeout=timeout, transport=transport, max_json_bytes=max_json_bytes) def __enter__(self): return self @@ -649,6 +657,7 @@ def stream_content( raise ProtocolError( "Content redirect must use HTTPS without URL credentials" ) + check_destination(target, self._destination_policy) response.close() # Never restore credentials, even after a same-origin hop or a # redirect back to Foldr. No session cookies are inherited. diff --git a/src/foldr_client/logging.py b/src/foldr_client/logging.py new file mode 100644 index 0000000..5feffe3 --- /dev/null +++ b/src/foldr_client/logging.py @@ -0,0 +1,23 @@ +"""Opt-in protection against credentials in HTTP dependency logs.""" + +import logging + + +class HTTPLogFilter(logging.Filter): + """Suppress HTTPX and HTTPCore records, which can contain signed URLs. + + Attach an instance to every output handler that could receive these records, + including handlers attached directly to dependency loggers. Filtering a + parent logger does not filter records propagated from its descendants. + + Suppression covers all levels and the complete record: storage credentials + may occur in URL paths, query strings, headers, or exception details. Records + from unrelated loggers are unchanged. Importing or constructing this filter + never installs it or changes the application's logging configuration. + """ + + def filter(self, record: logging.LogRecord) -> bool: + return not any( + record.name == name or record.name.startswith(name + ".") + for name in ("httpx", "httpcore") + ) diff --git a/src/foldr_client/oauth.py b/src/foldr_client/oauth.py index 70bf2ac..87d7610 100644 --- a/src/foldr_client/oauth.py +++ b/src/foldr_client/oauth.py @@ -4,7 +4,7 @@ from collections.abc import Sequence from dataclasses import dataclass from typing import Any -from urllib.parse import parse_qs, quote_plus, urlencode, urlsplit +from urllib.parse import SplitResult, parse_qs, quote_plus, urlencode, urlsplit import httpx from oauthlib.oauth2 import WebApplicationClient @@ -13,6 +13,26 @@ from .errors import OAuthError, ProtocolError, TransportError +def _callback_parts(url: str) -> SplitResult: + """Parse without silently accepting userinfo or stripped URL controls.""" + if any(ord(char) <= 32 or ord(char) == 127 for char in url): + raise ValueError("Invalid callback URL") + parsed = urlsplit(url) + if not parsed.hostname or parsed.username is not None or parsed.password is not None: + raise ValueError("Invalid callback URL") + # Accessing port validates its spelling and range; preserve its original text. + _ = parsed.port + return parsed + + +def _callback_target(parsed: SplitResult) -> tuple[str, str, str]: + authority = parsed.netloc + if not authority.startswith("["): + host, separator, port = authority.partition(":") + authority = host.lower() + separator + port + return parsed.scheme, authority, parsed.path + + @dataclass(frozen=True, repr=False) class Authorization: """Caller-held, short-lived authorization context. Never log casually. @@ -43,6 +63,7 @@ def __init__( client_secret: str | None = None, auth_method: str = "none", timeout: float | httpx.Timeout = 60.0, + max_json_bytes: int = 64 * 1024, transport: httpx.BaseTransport | None = None, ): self.site = site_url(site) @@ -57,7 +78,7 @@ def __init__( raise ValueError("client_secret must match the authentication method") self._client_secret = client_secret self.auth_method = auth_method - self._connection = HTTP(timeout=timeout, transport=transport) + self._connection = HTTP(timeout=timeout, transport=transport, max_json_bytes=max_json_bytes) def __enter__(self): return self @@ -72,7 +93,10 @@ def authorize(self, redirect_uri: str, *, scopes: Sequence[str]) -> Authorizatio """Generate an S256 PKCE URL and state; caller opens URL and retains context.""" if isinstance(scopes, str): raise ValueError("scopes must be a sequence of scope strings") - parsed = urlsplit(redirect_uri) + try: + parsed = _callback_parts(redirect_uri) + except ValueError: + raise ValueError("Invalid redirect_uri") from None if ( not parsed.hostname or parsed.username is not None @@ -113,12 +137,12 @@ def exchange_callback( """ if authorization.site != self.site or authorization.client_id != self.client_id: raise ProtocolError("Authorization context belongs to another site or client") - actual, expected = urlsplit(callback_url), urlsplit(authorization.redirect_uri) - if actual.fragment or (actual.scheme, actual.netloc, actual.path) != ( - expected.scheme, - expected.netloc, - expected.path, - ): + try: + actual = _callback_parts(callback_url) + expected = _callback_parts(authorization.redirect_uri) + except ValueError: + raise ProtocolError("Malformed OAuth callback target") from None + if actual.fragment or _callback_target(actual) != _callback_target(expected): raise ProtocolError("OAuth callback target does not match redirect_uri") query = parse_qs(actual.query, keep_blank_values=True) for key, values in parse_qs(expected.query, keep_blank_values=True).items(): diff --git a/src/foldr_client/site.py b/src/foldr_client/site.py index 98cb784..84c7fe8 100644 --- a/src/foldr_client/site.py +++ b/src/foldr_client/site.py @@ -25,9 +25,10 @@ def __init__( *, timeout: float | httpx.Timeout = 60.0, transport: httpx.BaseTransport | None = None, + max_json_bytes: int = 8 * 1024 * 1024, ): self.site = site_url(site) - self._connection = HTTP(timeout=timeout, transport=transport) + self._connection = HTTP(timeout=timeout, transport=transport, max_json_bytes=max_json_bytes) def __enter__(self): return self diff --git a/src/foldr_client/uploads.py b/src/foldr_client/uploads.py index 8a0f83d..59813e7 100644 --- a/src/foldr_client/uploads.py +++ b/src/foldr_client/uploads.py @@ -1,12 +1,13 @@ """Send direct-upload bytes using only the storage credentials in a Foldr session.""" import re -from collections.abc import Iterable +from collections.abc import Callable, Iterable from typing import Any, BinaryIO from urllib.parse import urlsplit import httpx +from ._destinations import check_destination from ._http import HTTP, iter_content, raise_response from .errors import ProtocolError, TransportError from .models import ApiResponse @@ -19,6 +20,9 @@ class DirectUpload: and headers are credentials: retain them privately. This helper never takes a Foldr token, follows redirects, retries, or completes the Foldr session. Close with a context manager or ``close()``. Caller-owned streams stay open. + Optional ``destination_policy`` must return True to approve each send. It + receives a credential-bearing URL; keep it out of logs. Private storage is + permitted by default. A URL policy does not constrain DNS resolution. """ def __init__( @@ -27,6 +31,7 @@ def __init__( *, timeout: float | httpx.Timeout = 60.0, transport: httpx.BaseTransport | None = None, + destination_policy: Callable[[httpx.URL], bool] | None = None, ): url = session.get("url") parsed = urlsplit(url) if isinstance(url, str) else None @@ -60,6 +65,7 @@ def __init__( self._method = method self._size = size self._chunk_size = chunk_size + self._destination_policy = destination_policy self._connection = HTTP(timeout=timeout, transport=transport) def __enter__(self): @@ -121,6 +127,7 @@ def send( headers["Content-Length"] = str(content_length) if content_range is not None: headers["Content-Range"] = content_range + check_destination(self._url, self._destination_policy) response = self._connection.send( self._method, self._url, diff --git a/tests/test_destination_policy.py b/tests/test_destination_policy.py new file mode 100644 index 0000000..6a8d5f2 --- /dev/null +++ b/tests/test_destination_policy.py @@ -0,0 +1,162 @@ +"""Storage policies approve every request before consuming or sending bytes.""" + +import io +import traceback + +import httpx +import pytest + +from foldr_client import DirectUpload, FoldrClient, ProtocolError + +SITE = "https://foldr.example" +STORAGE = "https://storage.example/private-path?signature=secret" + + +def session(**overrides): + value = {"url": STORAGE, "method": "PUT", "headers": {}, "size": 5, "chunk_size": None} + value.update(overrides) + return value + + +def fail_policy(url): + raise RuntimeError(f"Cannot approve {url}") + + +@pytest.mark.parametrize("policy", [lambda url: False, lambda url: None, fail_policy]) +def test_upload_denial_neither_reads_content_nor_sends_request(policy): + class UnreadFile(io.BytesIO): + def read(self, size=-1): + pytest.fail("Denied upload must not read caller content") + + def handler(request): + pytest.fail("Denied upload must not contact storage") + + source = UnreadFile(b"hello") + with DirectUpload( + session(), destination_policy=policy, transport=httpx.MockTransport(handler) + ) as upload: + with pytest.raises(ProtocolError) as error: + upload.send(source, content_length=5) + assert not source.closed + assert not error.value.outcome_uncertain + formatted = "".join(traceback.format_exception(error.value)) + assert STORAGE not in formatted + assert "private-path" not in formatted + assert "signature=secret" not in formatted + + +def test_upload_policy_rechecks_each_piece_before_reading_it(): + destinations = [] + requests = [] + + def policy(url): + assert isinstance(url, httpx.URL) + destinations.append(url) + return len(destinations) == 1 + + def handler(request): + requests.append(request) + assert request.read() == b"hel" + return httpx.Response(202) + + source = io.BytesIO(b"lo") + with DirectUpload( + session(chunk_size=3), destination_policy=policy, transport=httpx.MockTransport(handler) + ) as upload: + upload.send(b"hel", content_length=3, content_range="bytes 0-2/5") + with pytest.raises(ProtocolError): + upload.send(source, content_length=2, content_range="bytes 3-4/5") + assert destinations == [httpx.URL(STORAGE), httpx.URL(STORAGE)] + assert len(requests) == 1 + assert source.tell() == 0 + + +@pytest.mark.parametrize("denied_hop", [1, 2]) +def test_download_policy_checks_each_hop_and_closes_denied_redirect(denied_hop): + requests = [] + checked = [] + bodies = [] + targets = [STORAGE, "https://169.254.169.254/private-path?signature=secret"] + + class UnreadBody(httpx.SyncByteStream): + closed = False + + def __iter__(self): + pytest.fail("Redirect body must not be consumed") + yield b"" + + def close(self): + self.closed = True + + def policy(url): + checked.append(url) + return len(checked) < denied_hop + + def handler(request): + requests.append(request) + assert len(requests) <= denied_hop + body = UnreadBody() + bodies.append(body) + return httpx.Response(302, headers={"Location": targets[len(requests) - 1]}, stream=body) + + with FoldrClient( + SITE, + access_token="private", + destination_policy=policy, + transport=httpx.MockTransport(handler), + ) as client: + with pytest.raises(ProtocolError) as error: + with client.stream_content("s", "/f"): + pytest.fail("Denied destination must not produce content") + assert len(requests) == denied_hop + assert checked == [httpx.URL(url) for url in targets[:denied_hop]] + assert all(body.closed for body in bodies) + assert "signature" not in str(error.value) + assert "private-path" not in str(error.value) + assert requests[0].headers["authorization"] == "Bearer private" + for request in requests[1:]: + assert "authorization" not in request.headers + + +def test_download_policy_failure_is_sanitized_before_storage_request(): + requests = [] + + def handler(request): + requests.append(request) + return httpx.Response(302, headers={"Location": STORAGE}) + + with FoldrClient( + SITE, + access_token="private", + destination_policy=fail_policy, + transport=httpx.MockTransport(handler), + ) as client: + with pytest.raises(ProtocolError) as error: + with client.stream_content("s", "/f"): + pytest.fail("Policy failure must deny the request") + assert len(requests) == 1 + formatted = "".join(traceback.format_exception(error.value)) + assert STORAGE not in formatted + assert "signature=secret" not in formatted + + +@pytest.mark.parametrize("policy", [None, lambda url: url.host == "10.0.0.1"]) +def test_private_storage_works_by_default_or_explicit_approval(policy): + seen = [] + + def handler(request): + seen.append(request) + if len(seen) == 1: + return httpx.Response(302, headers={"Location": "https://10.0.0.1/file"}) + assert "authorization" not in request.headers + return httpx.Response(200, content=b"hello") + + with FoldrClient( + SITE, + access_token="private", + destination_policy=policy, + transport=httpx.MockTransport(handler), + ) as client: + with client.stream_content("s", "/f") as content: + assert b"".join(content.iter_bytes()) == b"hello" + assert len(seen) == 2 diff --git a/tests/test_direct_upload.py b/tests/test_direct_upload.py index 04052b3..d6e9a70 100644 --- a/tests/test_direct_upload.py +++ b/tests/test_direct_upload.py @@ -46,6 +46,26 @@ def handler(request): assert response.data is None +def test_create_only_header_is_preserved_and_repeat_rejection_is_not_retried(): + seen = [] + + def handler(request): + seen.append(request) + assert request.headers["if-none-match"] == "*" + assert request.read() == b"hello" + return httpx.Response(201 if len(seen) == 1 else 412) + + with DirectUpload( + session(headers={"If-None-Match": "*"}), + transport=httpx.MockTransport(handler), + ) as upload: + assert upload.send(b"hello", content_length=5).status_code == 201 + with pytest.raises(FoldrError) as error: + upload.send(b"hello", content_length=5) + assert error.value.status_code == 412 + assert len(seen) == 2 + + def test_supplied_storage_auth_is_preserved_but_cookies_are_not_retained(): seen = [] diff --git a/tests/test_json_limits.py b/tests/test_json_limits.py new file mode 100644 index 0000000..18944ae --- /dev/null +++ b/tests/test_json_limits.py @@ -0,0 +1,171 @@ +"""Successful JSON is bounded before parsing, including token/mutation outcomes.""" + +import gzip + +import httpx +import pytest + +from foldr_client import FoldrClient, FoldrOAuth, FoldrSite, ProtocolError + +SITE = "https://files.example.com" +CASES = ("read", "envelope", "write", "oauth", "discovery", "registration") + + +class TrackedBody(httpx.SyncByteStream): + def __init__(self, content): + self.content = content + self.delivered = 0 + self.closed = False + + def __iter__(self): + for offset in range(0, len(self.content), 4096): + chunk = self.content[offset : offset + 4096] + self.delivered += len(chunk) + yield chunk + + def close(self): + self.closed = True + + +def invoke(case, handler, **options): + options["transport"] = httpx.MockTransport(handler) + if case in ("read", "envelope", "write"): + with FoldrClient(SITE, access_token="synthetic", **options) as client: + if case == "read": + return client.me() + if case == "envelope": + return client.list_fields().data + return client.create_folder("share", "/new", idempotency_key="key").data + if case == "oauth": + with FoldrOAuth(SITE, client_id="synthetic", **options) as oauth: + return oauth.refresh("synthetic-refresh") + with FoldrSite(SITE, **options) as site: + if case == "registration": + return site.register({"client_name": "Synthetic"}).data + return site.openapi().data + + +def response_status(case): + return 201 if case in ("write", "registration") else 200 + + +@pytest.mark.parametrize("case", CASES) +@pytest.mark.parametrize("declared_length", [None, "2", "99999999"]) +def test_success_limit_uses_actual_bytes_stops_early_and_closes(case, declared_length): + body = TrackedBody(b'{"data":"' + b"x" * 1024 * 1024 + b'"}') + calls = [] + + def handle(request): + calls.append(request) + headers = {} if declared_length is None else {"Content-Length": declared_length} + return httpx.Response(response_status(case), headers=headers, stream=body) + + with pytest.raises(ProtocolError) as caught: + invoke(case, handle, max_json_bytes=32768) + assert caught.value.outcome_uncertain is (case in ("write", "oauth", "registration")) + assert 32768 < body.delivered <= 32768 + 8192 + assert body.closed + assert len(calls) == 1 + + +@pytest.mark.parametrize("case", CASES) +@pytest.mark.parametrize("encoding", ["gzip", "deflate", "br", "gzip, identity"]) +def test_success_compression_is_rejected_without_consuming_body(case, encoding): + body = TrackedBody(gzip.compress(b'{"data":"' + b"x" * 1024 * 1024 + b'"}')) + + def handle(request): + assert request.headers["accept-encoding"] == "identity" + return httpx.Response( + response_status(case), headers={"Content-Encoding": encoding}, stream=body + ) + + with pytest.raises(ProtocolError) as caught: + invoke(case, handle) + assert caught.value.outcome_uncertain is (case in ("write", "oauth", "registration")) + assert body.delivered == 0 + assert body.closed + + +@pytest.mark.parametrize("case", CASES) +def test_exact_limit_success_is_accepted(case): + content = b'{"access_token":"a","token_type":"Bearer","refresh_token":"r"}' + body = TrackedBody(content) + + def handle(request): + return httpx.Response(response_status(case), stream=body) + + assert invoke(case, handle, max_json_bytes=len(content))["access_token"] == "a" + assert body.closed + + +@pytest.mark.parametrize("case", ["read", "discovery", "oauth"]) +def test_oauth_default_is_smaller_than_api_and_discovery(case): + content = ( + b'{"access_token":"a","token_type":"Bearer","refresh_token":"r","extension":"' + + b"x" * 70000 + + b'"}' + ) + + def handle(request): + return httpx.Response(200, stream=TrackedBody(content)) + + if case == "oauth": + with pytest.raises(ProtocolError) as caught: + invoke(case, handle) + assert caught.value.outcome_uncertain + assert invoke(case, handle, max_json_bytes=len(content))["access_token"] == "a" + else: + assert invoke(case, handle)["access_token"] == "a" + + +@pytest.mark.parametrize("case", CASES) +def test_prebuffered_custom_transport_success_still_obeys_limit(case): + def handle(request): + return httpx.Response(response_status(case), json={"secret": "x" * 100}) + + with pytest.raises(ProtocolError): + invoke(case, handle, max_json_bytes=32) + + +@pytest.mark.parametrize("case", ["read", "oauth", "discovery"]) +@pytest.mark.parametrize("limit", [0, -1, True, 1.5, None]) +def test_invalid_limit_is_rejected_before_network(case, limit): + def handle(request): + pytest.fail("Invalid configuration must not send a request") + + with pytest.raises(ValueError, match="max_json_bytes"): + invoke(case, handle, max_json_bytes=limit) + + +@pytest.mark.parametrize("case", ["write", "registration"]) +def test_malformed_mutation_success_remains_uncertain(case): + body = TrackedBody(b"not JSON") + with pytest.raises(ProtocolError) as caught: + invoke(case, lambda _: httpx.Response(response_status(case), stream=body)) + assert caught.value.outcome_uncertain + assert body.closed + + +def test_limit_does_not_cap_file_streams(): + with FoldrClient( + SITE, + access_token="synthetic", + max_json_bytes=2, + transport=httpx.MockTransport(lambda _: httpx.Response(200, content=b"file bytes")), + ) as client: + with client.stream_content("share", "/file") as stream: + assert b"".join(stream.iter_bytes()) == b"file bytes" + + +@pytest.mark.parametrize("character", [*(chr(code) for code in range(33)), "\x7f", "é"]) +def test_access_token_rejects_controls_whitespace_and_non_ascii_without_echoing(character): + token = "private" + character + "token" + with pytest.raises(ValueError) as caught: + FoldrClient(SITE, access_token=token) + assert "private" not in str(caught.value) + + +@pytest.mark.parametrize("token", ["", None, b"secret", 123]) +def test_access_token_requires_a_nonempty_string(token): + with pytest.raises(ValueError): + FoldrClient(SITE, access_token=token) diff --git a/tests/test_logging.py b/tests/test_logging.py new file mode 100644 index 0000000..09eeb3d --- /dev/null +++ b/tests/test_logging.py @@ -0,0 +1,126 @@ +"""HTTP logging protection is explicit and covers signed paths as well as queries.""" + +import io +import logging +import subprocess +import sys + +import httpx +import pytest + +from foldr_client import DirectUpload, FoldrClient +from foldr_client.logging import HTTPLogFilter + + +@pytest.mark.parametrize( + ("name", "allowed"), + [ + ("httpx", False), + ("httpx.client", False), + ("httpcore", False), + ("httpcore.http11", False), + ("httpcore.connection.deep", False), + ("httpx_app", True), + ("httpcore_metrics", True), + ("app.httpx", True), + ("foldr_client", True), + ("root", True), + ], +) +def test_filter_matches_only_http_dependency_logger_hierarchies(name, allowed): + record = logging.LogRecord(name, logging.CRITICAL, __file__, 1, "message", (), None) + assert HTTPLogFilter().filter(record) is allowed + + +@pytest.mark.parametrize("level", [logging.INFO, logging.DEBUG]) +def test_signed_download_and_upload_urls_are_suppressed_on_output_handlers( + monkeypatch, + caplog, + level, +): + filtered_output, raw_output = io.StringIO(), io.StringIO() + filtered_handler, raw_handler = ( + logging.StreamHandler(filtered_output), + logging.StreamHandler(raw_output), + ) + filtered_handler.addFilter(HTTPLogFilter()) + # Install directly on dependency loggers to exercise the common handler + # configuration that a root-logger filter alone would fail to protect. + for name in ("httpx", "httpcore", "foldr_test_app"): + logger = logging.getLogger(name) + monkeypatch.setattr(logger, "handlers", [filtered_handler, raw_handler]) + caplog.set_level(level, logger=name) + monkeypatch.setattr(logger, "disabled", False) + monkeypatch.setattr(logger, "propagate", False) + core_logger = logging.getLogger("httpcore.http11") + monkeypatch.setattr(core_logger, "handlers", []) + caplog.set_level(logging.NOTSET, logger="httpcore.http11") + monkeypatch.setattr(core_logger, "disabled", False) + monkeypatch.setattr(core_logger, "propagate", True) + download_url = "https://storage.example/download-secret?signature=download-query-secret" + upload_url = "https://storage.example/upload-secret?signature=upload-query-secret" + requests = [] + + def transport(request): + requests.append(request) + # MockTransport bypasses HTTPCore. Emit a representative dependency + # debug record here to cover headers and exception details too. + core_logger.debug("headers=%s exception=%s", "storage-header-secret", "detail-secret") + if request.url.host == "foldr.example": + return httpx.Response(302, headers={"Location": download_url}) + if request.method == "GET": + assert "authorization" not in request.headers + return httpx.Response(200, content=b"hello") + assert request.headers["x-storage-proof"] == "storage-header-secret" + assert request.read() == b"hello" + return httpx.Response(201) + + with FoldrClient( + "https://foldr.example", + access_token="foldr-token-secret", + transport=httpx.MockTransport(transport), + ) as client: + with client.stream_content("share", "/file") as content: + assert b"".join(content.iter_bytes()) == b"hello" + with DirectUpload( + { + "url": upload_url, + "method": "PUT", + "headers": {"X-Storage-Proof": "storage-header-secret"}, + "size": 5, + }, + transport=httpx.MockTransport(transport), + ) as upload: + upload.send(b"hello", content_length=5) + logging.getLogger("foldr_test_app").info("application event remains visible") + + assert len(requests) == 3 + assert download_url in raw_output.getvalue() + assert upload_url in raw_output.getvalue() + if level == logging.DEBUG: + assert "storage-header-secret" in raw_output.getvalue() + assert "detail-secret" in raw_output.getvalue() + assert filtered_output.getvalue() == "application event remains visible\n" + + +def test_import_and_construction_do_not_change_logging_configuration(): + # A fresh interpreter verifies the first import, not only a cached module. + script = """ +import logging + +loggers = [logging.getLogger(name) for name in ('', 'httpx', 'httpcore')] +for logger in loggers: + logger.addHandler(logging.NullHandler()) + logger.addFilter(logging.Filter()) + logger.setLevel(logging.WARNING) + +def snapshot(): + return [(tuple(item.handlers), tuple(item.filters), item.level, + item.propagate, item.disabled) for item in loggers] + +before = snapshot() +from foldr_client.logging import HTTPLogFilter +HTTPLogFilter() +assert snapshot() == before +""" + subprocess.run([sys.executable, "-c", script], check=True, capture_output=True, text=True) diff --git a/tests/test_oauth.py b/tests/test_oauth.py index 31ca010..7b6d479 100644 --- a/tests/test_oauth.py +++ b/tests/test_oauth.py @@ -107,6 +107,24 @@ def handle(request): assert len(requests) == 1 +@pytest.mark.parametrize("port", ["", ":443", ":0443"]) +def test_callback_hostname_case_preserves_original_registered_uri_in_exchange(port): + registered = "https://App.Example.COM" + port + "/Callback?session=caller-owned" + requests = [] + + def handle(request): + requests.append(request) + assert form(request)["redirect_uri"] == [registered] + return httpx.Response(200, json=TOKENS) + + with FoldrOAuth(SITE, client_id="id", transport=httpx.MockTransport(handle)) as oauth: + authorization = oauth.authorize(registered, scopes=["foldr.read"]) + query = urlsplit(callback(authorization)).query + actual = "https://APP.example.com" + port + "/Callback?" + query + assert oauth.exchange_callback(actual, authorization) == TOKENS + assert len(requests) == 1 + + @pytest.mark.parametrize( "updates", [ @@ -114,11 +132,13 @@ def handle(request): {"state": ""}, {"state": ["fixed-state", "fixed-state"]}, {"iss": "https://other.example.com"}, + {"iss": "https://FILES.example.com"}, {"iss": ""}, {"iss": [SITE, SITE]}, {"code": ""}, {"code": ["one", "two"]}, {"session": "wrong"}, + {"session": ["caller-owned", "caller-owned"]}, {"error": ["access_denied", "access_denied"]}, {"error": "access_denied"}, ], @@ -130,7 +150,10 @@ def handle(request): with FoldrOAuth(SITE, client_id="id", transport=httpx.MockTransport(handle)) as oauth: authorization = oauth.authorize(CALLBACK, scopes=["foldr.read"]) with pytest.raises(ProtocolError): - oauth.exchange_callback(callback(authorization, **updates), authorization) + oauth.exchange_callback( + callback(authorization, **updates).replace("app.example.com", "APP.EXAMPLE.COM"), + authorization, + ) @pytest.mark.parametrize("missing", ["state", "iss", "code"]) @@ -155,6 +178,18 @@ def handle(request): "https://evil.example.com/callback", "https://app.example.com/other", "https://app.example.com:444/callback", + "https://APP.EXAMPLE.COM:443/callback", + "https://APP.EXAMPLE.COM:/callback", + "https://APP.EXAMPLE.COM/Callback", + "https://APP.EXAMPLE.COM/call%62ack", + "https://APP.EXAMPLE.COM/callback/", + "https://secret@APP.EXAMPLE.COM/callback", + "https://user:secret@APP.EXAMPLE.COM/callback", + "https://APP.EXAMPLE.COM:secret/callback", + "https://APP.EXAMPLE.COM:65536/callback", + "https://[secret/callback", + "https://APP.EXAMPLE.COM/call\nback", + " https://APP.EXAMPLE.COM/callback", ], ) def test_callback_target_checked_before_http(target): @@ -164,8 +199,22 @@ def handle(request): with FoldrOAuth(SITE, client_id="id", transport=httpx.MockTransport(handle)) as oauth: authorization = oauth.authorize(CALLBACK, scopes=["foldr.read"]) query = urlsplit(callback(authorization)).query - with pytest.raises(ProtocolError): + with pytest.raises(ProtocolError) as caught: oauth.exchange_callback(target + "?" + query, authorization) + assert "secret" not in str(caught.value) + + +def test_callback_port_spelling_remains_exact(): + def handle(request): + pytest.fail("Changed port spelling must not exchange a code") + + with FoldrOAuth(SITE, client_id="id", transport=httpx.MockTransport(handle)) as oauth: + authorization = oauth.authorize( + "https://app.example.com:0443/callback?session=caller-owned", scopes=["foldr.read"] + ) + query = urlsplit(callback(authorization)).query + with pytest.raises(ProtocolError): + oauth.exchange_callback("https://APP.EXAMPLE.COM:443/callback?" + query, authorization) def test_callback_fragment_and_foreign_authorization_context_are_rejected(): @@ -273,6 +322,38 @@ def close(self): self.closed = True +@pytest.mark.parametrize("max_json_bytes", [None, 128]) +def test_token_response_limit_closes_body_and_preserves_uncertain_outcome(max_json_bytes): + calls = [] + limit = 64 * 1024 if max_json_bytes is None else max_json_bytes + + class OversizedBody(httpx.SyncByteStream): + closed = False + + def __iter__(self): + yield b'{"access_token":"' + b"x" * limit + b'","token_type":"Bearer"}' + + def close(self): + self.closed = True + + body = OversizedBody() + + def handle(request): + calls.append(request) + return httpx.Response(200, stream=body) + + options = {} if max_json_bytes is None else {"max_json_bytes": max_json_bytes} + with FoldrOAuth( + SITE, client_id="id", transport=httpx.MockTransport(handle), **options + ) as oauth: + with pytest.raises(ProtocolError) as caught: + oauth.refresh("refresh-secret") + assert caught.value.outcome_uncertain is True + assert "refresh-secret" not in str(caught.value) + assert len(calls) == 1 + assert body.closed + + @pytest.mark.parametrize("failure", ["timeout", "connect", "read"]) def test_refresh_transport_failures_are_uncertain_and_never_retried(failure): calls = [] @@ -409,6 +490,10 @@ def handle(request): "https://user:secret@app.example.com/callback", "https://app.example.com/callback#fragment", "javascript:alert(1)", + "https://app.example.com:secret/callback", + "https://app.example.com:65536/callback", + "https://[secret/callback", + "https://app.example.com/call\nback", ], ) def test_authorization_rejects_unsafe_redirect_uri(redirect):