Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ Methods use fixed API routes; constructing a client opens no network connection.
The caller supplies a Foldr HTTPS site and an access token. Each client has its
own HTTP session and identity. Foldr decides which shares and files that identity
may access, including shares backed by another storage account.
Access tokens must be nonempty visible ASCII without whitespace; invalid values
are rejected before any request.

## Install

Expand Down Expand Up @@ -131,8 +133,15 @@ For direct downloads, the client follows at most five redirects, sending **no
Foldr Authorization or cookies** on any redirect hop, including the same host.
Only HTTPS destinations without URL credentials are accepted. Omitted `direct`
honors the share's setting; `direct=False` requests bytes through Foldr.
Signed storage URLs and Location headers
are credentials: keep them out of logs. Stream chunks use bounded buffering;
Choose `direct=False` when downloads need Foldr to check access on each new
request. It does not revoke storage URLs issued earlier; those can outlive an app
disconnect or permission change. An optional `destination_policy` callback on
`FoldrClient` checks every download redirect, and on `DirectUpload` checks each
storage send. See [client safety configuration](docs/client-safety.md).
Signed storage URLs and Location headers are credentials: keep them out of logs.
HTTPX/HTTPCore logs can expose these URLs even when exception messages are safe;
the opt-in `HTTPLogFilter` suppresses their records on configured output handlers.
Stream chunks use bounded buffering;
collecting them into a list or joining them makes memory use the caller's concern.
Content requests require identity encoding. A server that sends compressed content
despite that request is rejected before decompression, preserving bounded memory
Expand Down Expand Up @@ -251,6 +260,13 @@ bytes rather than being decompressed. Exception messages omit provider text and
URLs; error attributes and headers may contain sensitive data, so redact before logging.
Failures during an error body's transfer may surface as transport errors.

Successful JSON responses are limited to 8 MiB for `FoldrClient`/`FoldrSite` and
64 KiB for `FoldrOAuth`. Set the positive integer `max_json_bytes` constructor
option to adjust a limit. Unexpected compression is rejected before decoding;
oversized responses raise `ProtocolError` and close without retry. Failed
mutation/token responses retain `outcome_uncertain=True`. File streams are not
subject to the JSON limit. See [configuration examples](docs/client-safety.md).

There are no implicit retries, sleeps, snapshot restarts or permission checks.
Foldr's 401/403/404 and storage connection errors reach the caller unchanged.
Use structured type/code/status rather than matching human-readable detail.
Expand Down
27 changes: 24 additions & 3 deletions docs/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,8 @@ ETags remain opaque, including weak markers and quotes.

## Site bootstrap and OAuth

`FoldrSite(site, *, timeout=60.0, transport=None)` owns a credential-free session:
`FoldrSite(site, *, timeout=60.0, transport=None, max_json_bytes=8388608)` owns a
credential-free session:

| Method | Endpoint | Result |
| --- | --- | --- |
Expand All @@ -82,7 +83,7 @@ ETags remain opaque, including weak markers and quotes.
| `register(metadata)` | POST `/oauth/register`, JSON | ApiResponse 201 |

`FoldrOAuth(site, *, client_id, client_secret=None, auth_method="none", timeout=60.0,
transport=None)` provides these explicit operations:
max_json_bytes=65536, transport=None)` provides these explicit operations:

| Method | Protocol |
| --- | --- |
Expand All @@ -91,7 +92,8 @@ transport=None)` provides these explicit operations:
| `refresh(refresh_token, *, scopes=None)` | One POST `/oauth/token`; returns rotated tokens for caller to save |
| `revoke(token, *, token_type_hint=None)` | POST `/oauth/revoke`; accepts empty 200, returns None |

`DirectUpload(session, *, timeout=60.0, transport=None)` sends bytes to the exact
`DirectUpload(session, *, timeout=60.0, transport=None, destination_policy=None)`
sends bytes to the exact
storage method/URL/headers from createUploadSession. `send(content, *,
content_length, content_range=None)` returns status/headers with `data=None` and
does not interpret storage response bodies. Use a separate context from Foldr;
Expand All @@ -100,6 +102,25 @@ transport and support context management and `close()`.

## Request behavior and limits

`FoldrClient(site, *, access_token, timeout=60.0, transport=None,
max_json_bytes=8388608, destination_policy=None)` owns one authenticated session.
`max_json_bytes` is a positive integer limit on each successful JSON response;
it is enforced while reading actual bytes, regardless of Content-Length.
Compressed JSON is refused before decompression. OAuth defaults to 64 KiB;
API and site/discovery default to 8 MiB. These limits do not cap file streams,
change the 64 KiB error cap, or read bodies on HEAD/204/304 responses.

`destination_policy` is an optional synchronous `Callable[[httpx.URL], bool]`.
It must return `True` to approve each download redirect or each direct-upload
send. Rejection or a callback exception raises a sanitized `ProtocolError`
before contacting that destination. It does not validate the initial Foldr site
or resolve DNS. With no callback, existing HTTPS/private storage support remains.

`HTTPLogFilter()` is a standard logging filter, exported by `foldr_client`, that
suppresses HTTPX/HTTPCore records at all levels. It must be attached explicitly to
every applicable output handler; importing the client changes no logging setup.
See [client safety configuration](client-safety.md) for examples and limits.

Download `direct=None` omits the parameter and honors the share's setting;
`direct=False` requests bytes through Foldr. Upload-session `content_type=None`
omits that optional field. Optional query/body values set to `None` are generally
Expand Down
127 changes: 127 additions & 0 deletions docs/client-safety.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Client safety configuration

The defaults require HTTPS, verify TLS, isolate credentials, and avoid automatic
retries. The settings below control JSON memory use, HTTP dependency logs, and
which storage destinations an application will contact.

## JSON response limits

| Client | Default maximum successful JSON body |
| --- | --- |
| `FoldrClient` | 8 MiB (8,388,608 bytes) |
| `FoldrSite` (including OpenAPI and registration) | 8 MiB |
| `FoldrOAuth` | 64 KiB (65,536 bytes) |

Set `max_json_bytes` to a positive integer on the relevant constructor when an
expected response needs a different bound:

```python
from foldr_client import FoldrClient, FoldrOAuth, FoldrSite

with FoldrClient(site, access_token=token, max_json_bytes=4 * 1024 * 1024) as client:
account = client.me()

with FoldrSite(site, max_json_bytes=16 * 1024 * 1024) as public:
contract = public.openapi().data

with FoldrOAuth(site, client_id=client_id, max_json_bytes=128 * 1024) as oauth:
authorization = oauth.authorize(callback_uri, scopes=["foldr.read"])
```

The limit counts actual serialized JSON bytes, not Content-Length. It bounds the
body retained for parsing; the parsed Python objects consume additional memory.
Unexpected compression is rejected before reading/decompression, even if the
compressed body is small. Oversize or malformed JSON raises `ProtocolError`,
closes the response and does not retry. On a mutation or token exchange,
`outcome_uncertain=True` means the request may already have taken effect: reconcile
the result rather than blindly replaying it.

File downloads continue to stream without this JSON cap. Error bodies keep their
separate 64 KiB wire-byte cap. HEAD, 204 and 304 do not consume response bodies.
Custom transports are responsible for any buffering they perform before returning
a response; the client cannot prevent a transport from preloading a whole body.

## Keep signed URLs out of dependency logs

HTTPX INFO records contain complete request URLs. HTTPCore DEBUG records can
include headers and other sensitive details. Signed storage credentials can live
in URL paths as well as query strings, so removing a query string alone is not
sufficient. Safe exception messages do not sanitize these dependency logs.

`HTTPLogFilter` suppresses all records from `httpx`, `httpcore` and their child
loggers. Unrelated application logs remain visible. For a new application's
startup logging configuration:

```python
import logging
from foldr_client import HTTPLogFilter

output = logging.StreamHandler()
output.addFilter(HTTPLogFilter())
logging.basicConfig(level=logging.INFO, handlers=[output])
```

If logging is already configured, `basicConfig` may do nothing: add the filter to
each existing output handler instead. This includes handlers attached directly to
dependency loggers, file handlers, telemetry handlers and queue handlers. Apply
the filter before records leave the process. A filter on the root logger alone
does not filter records propagated from descendant loggers.

Importing or constructing the filter makes no global logging changes. Suppression
also removes HTTP dependency error diagnostics; SDK exceptions still reach the
caller. The filter does not redact application logs, custom transport logs,
external tracing, or printed response data. Never log tokens, authorization
contexts, upload sessions, signed URLs or raw sensitive error attributes yourself.

## Restrict storage destinations when needed

Both `FoldrClient` and `DirectUpload` accept `destination_policy`. It receives an
`httpx.URL` and must return the literal boolean `True` to approve a destination.
It runs for every download redirect, including subsequent and same-origin hops,
and before every direct-upload send, including each chunk. The existing HTTPS and
no-URL-credentials checks still apply.

For an application with a known storage host:

```python
import httpx
from foldr_client import DirectUpload, FoldrClient


def approved_storage(url: httpx.URL) -> bool:
return url.host == "storage.example.com" and url.port in (None, 443)


with FoldrClient(site, access_token=token, destination_policy=approved_storage) as client:
with client.stream_content(share_id, path) as stream:
for chunk in stream.iter_bytes():
consume(chunk)

with DirectUpload(upload_session, destination_policy=approved_storage) as storage:
storage.send(body, content_length=upload_session["size"])
```

Use the hosts appropriate to your storage, including any legitimate CDN hops.
The callback sees the full sensitive URL: do not log it. Rejection, a non-boolean
approval, or a callback exception raises a generic `ProtocolError` before a
request to that destination, without consuming upload bytes. Earlier upload
chunks or requests may already have succeeded. Callback exceptions are suppressed
from normal formatted tracebacks so they cannot echo the signed URL.

The policy is opt-in. Without it, HTTPS destinations including private storage
continue to work. It does not control the initial configured Foldr site, perform
DNS resolution, or prevent DNS rebinding. An application accepting untrusted
sites must validate them separately. Deployments requiring network isolation
also need appropriate egress controls; URL hostname checks are not a firewall.

## Revocation and version guarantees

`stream_content(..., direct=False)` requests Foldr-proxied bytes, allowing Foldr
to check access on each new request. It does not revoke previously issued storage
URLs, which can remain valid after an app disconnect or permission change. Direct
URL validity is backend-dependent; the API's `expires_at` is not a guaranteed
maximum lifetime. Neither mode can retract bytes already downloaded.

The SDK preserves server ETags unchanged. Their concurrency guarantees depend on
the storage backend; weak SMB size/mtime tags are not strong content-version
identifiers. See [request keys and ETags](writes.md#request-keys-and-etags).
15 changes: 15 additions & 0 deletions docs/writes.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,14 @@ locally because these methods require an explicit version. Other validation and
conflict handling belong to the server. A failed request is raised to the caller;
the client does not force the change or retry it.

Concurrency protection depends on the storage backend. Foldr can return weak
size/mtime ETags for SMB, where different contents of the same size within the
timestamp resolution may share a tag. The client preserves the tag, including
`W/`, exactly as the API requires; it does not turn it into a strong content
version or guarantee atomic server-side checks. Do not remove `W/` or substitute
a locally invented tag. Workflows needing stronger assurance must use server
and storage controls that provide it.

## Upload and overwrite

```python
Expand Down Expand Up @@ -204,3 +212,10 @@ retry or resume attempt. The session's explicit storage authorization headers
are retained. Keep the session URL and headers out of logs and treat them as
credentials. `cancel_upload_session(id)` provides explicit session cancellation;
the server determines what cancellation can remove.

Storage conditions such as a session-supplied `If-None-Match: *` are preserved
unchanged. A storage 412 is raised as `FoldrError`; the helper never removes a
condition or retries to force an upload. Create-only enforcement belongs to the
server/storage signature, not to a caller-controlled SDK. Configure
`destination_policy` and HTTP log filtering as described in
[client safety configuration](client-safety.md).
2 changes: 2 additions & 0 deletions src/foldr_client/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
StreamError,
TransportError,
)
from .logging import HTTPLogFilter
from .models import ApiResponse, ContentStream, Page
from .oauth import Authorization, FoldrOAuth
from .site import FoldrSite
Expand All @@ -24,6 +25,7 @@
"FoldrError",
"FoldrOAuth",
"FoldrSite",
"HTTPLogFilter",
"OAuthError",
"Page",
"ProtocolError",
Expand Down
19 changes: 19 additions & 0 deletions src/foldr_client/_destinations.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
"""Opt-in application policy for storage requests; not a DNS or network firewall."""

from collections.abc import Callable

import httpx

from .errors import ProtocolError


def check_destination(url: str, policy: Callable[[httpx.URL], bool] | None) -> None:
"""Require an explicit approval without exposing signed URLs in errors."""
if policy is None:
return
try:
approved = policy(httpx.URL(url))
except Exception:
raise ProtocolError("Storage destination policy failed") from None
if approved is not True:
raise ProtocolError("Storage destination rejected by policy")
55 changes: 36 additions & 19 deletions src/foldr_client/_http.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,16 +69,6 @@ def site_url(site: str) -> str:
return site.rstrip("/")


def require_object(response: httpx.Response) -> dict[str, Any]:
try:
value = response.json()
except (ValueError, UnicodeError):
raise ProtocolError("Expected a JSON object") from None
if not isinstance(value, dict):
raise ProtocolError("Expected a JSON object")
return value


def raise_response(response: httpx.Response, *, oauth: bool = False) -> None:
body = bytearray()
truncated = False
Expand Down Expand Up @@ -120,7 +110,11 @@ def __init__(
*,
timeout: float | httpx.Timeout = 60.0,
transport: httpx.BaseTransport | None = None,
max_json_bytes: int = 8 * 1024 * 1024,
):
if type(max_json_bytes) is not int or max_json_bytes <= 0:
raise ValueError("max_json_bytes must be a positive integer")
self.max_json_bytes = max_json_bytes
# A private session per instance. Explicit Request objects never inherit
# cookies, auth, URL credentials or headers from earlier responses.
self.http = httpx.Client(timeout=timeout, transport=transport, trust_env=False)
Expand Down Expand Up @@ -151,6 +145,33 @@ def send(
except OSError:
raise TransportError(operation, outcome_uncertain=uncertain) from None

def _json_value(self, response: httpx.Response, *, uncertain: bool) -> Any:
# Reject compression before reading: iter_bytes/read would decompress a
# small wire response into potentially unbounded memory. Content-Length
# is not trusted; enforce the limit against the bytes actually received.
if response.headers.get("content-encoding", "identity").strip().lower() != "identity":
raise ProtocolError(
"JSON response must honor Accept-Encoding: identity", outcome_uncertain=uncertain
)
body = bytearray()
# Pre-buffered responses can be supplied by custom/test transports. The
# transport owns any buffering before this method receives the response.
chunks = (
(response.content,)
if response.is_stream_consumed
else response.iter_raw(chunk_size=min(8192, self.max_json_bytes + 1))
)
for chunk in chunks:
if len(chunk) > self.max_json_bytes - len(body):
raise ProtocolError(
"JSON response exceeds max_json_bytes", outcome_uncertain=uncertain
)
body.extend(chunk)
try:
return json.loads(body)
except (ValueError, UnicodeError, RecursionError):
raise ProtocolError("Expected a JSON response", outcome_uncertain=uncertain) from None

def read_response(
self,
response: httpx.Response,
Expand All @@ -167,13 +188,7 @@ def read_response(
if head or response.status_code in (204, 304):
data = None
else:
response.read()
try:
data = response.json()
except (ValueError, UnicodeError):
raise ProtocolError(
"Expected a JSON response", outcome_uncertain=uncertain
) from None
data = self._json_value(response, uncertain=uncertain)
if not isinstance(data, (dict, list)):
raise ProtocolError(
"Expected a JSON object or array", outcome_uncertain=uncertain
Expand All @@ -197,8 +212,10 @@ def read_json(
try:
if response.status_code != 200:
raise_response(response, oauth=oauth)
response.read()
return require_object(response)
data = self._json_value(response, uncertain=uncertain)
if not isinstance(data, dict):
raise ProtocolError("Expected a JSON object", outcome_uncertain=uncertain)
return data
except httpx.HTTPError:
raise TransportError(operation, outcome_uncertain=uncertain) from None
finally:
Expand Down
Loading
Loading