Repository navigation
fix(control): read data exports from D1 in pages - #372
Merged
Merged
Conversation
…lt limit From 4 November 2026, D1 fails a query whose result is over 50 MB. GET /me/export read all of a person's sessions (prompts of up to 100 000 characters each, twice) and their year of audit log in single unbounded queries; it now reads them 100 sessions or 2000 entries at a time. Every other D1 read is bounded by a key, a LIMIT or a small table.
kalaspuff
approved these changes
Nov 4, 2026
kalaspuff
deleted the
fctl/631-can-you-look-into-https-developers-cloud
branch
November 4, 2026 11:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cloudflare's D1 query result size limit took effect on 4 November 2026: D1 now fails any query whose result is over 50 MB (Durable Object SQLite isn't affected, and wrangler's imports and exports are exempt). I went through the control plane's D1 reads (about 280
.all()/.first()calls, all throughdb::q). D1 stores no BLOBs, and every read is bounded by a key, aLIMIT, or a small per-person or per-project table. The exception was the data export (GET /me/export,fctl account export). It read all of a person's sessions twice in one query each. Prompts can be up to 100,000 characters (up to ~400 KB), so a heavy user could pass 50 MB in a few hundred sessions. It also read a year of their audit log in one query. Those four reads now go through a newdb::all_paged, which reads 100 sessions or 2,000 audit entries at a time, withLIMIT/OFFSETover a total order.I checked it against
cargo xtask dev. I copied a session with a 5,000-character prompt 250 times and added 2,500 audit entries to the dev account, then ranfctl account export. The export had all 251 sessions and 2,512 audit entries, each once and in order.cargo xtask check lintpasses.Worth confirming
GET /sessions, at most 201 rows with prompts) stays well under 50 MB with real prompts, so I left it as it is. It could only pass 50 MB if nearly every session in a page had a prompt of 100,000 four-byte characters.fctl session
Note
Low risk. It changes only how the account data export reads from D1, in pages instead of single queries; nothing is written, and other endpoints are untouched.
What it does. D1 now fails any query whose result is over 50 MB, and the account export (
GET /me/export,fctl account export) read all of a person's sessions, with prompts, and a year of audit log in single queries. This addsdb::all_paged, which reads rows withLIMIT/OFFSETover a total order, and uses it for the four unbounded export reads, 100 sessions or 2,000 audit entries per page.Pages are not a snapshot. Each page is a separate query, so a session created while an export runs may be missed or read twice.
fctl review's first look at
166bace· 2 files, +70 −24 · Review agent settings