Skip to content

fix(control): read data exports from D1 in pages - #372

Merged
kalaspuff merged 1 commit into
mainfrom
fctl/631-can-you-look-into-https-developers-cloud
Nov 4, 2026
Merged

kalaspuff merged 1 commit into
mainfrom
fctl/631-can-you-look-into-https-developers-cloud

Conversation

@fctl-sh

@fctl-sh fctl-sh Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Cloudflare's D1 query result size limit took effect on 4 November 2026: D1 now fails any query whose result is over 50 MB (Durable Object SQLite isn't affected, and wrangler's imports and exports are exempt). I went through the control plane's D1 reads (about 280 .all()/.first() calls, all through db::q). D1 stores no BLOBs, and every read is bounded by a key, a LIMIT, or a small per-person or per-project table. The exception was the data export (GET /me/export, fctl account export). It read all of a person's sessions twice in one query each. Prompts can be up to 100,000 characters (up to ~400 KB), so a heavy user could pass 50 MB in a few hundred sessions. It also read a year of their audit log in one query. Those four reads now go through a new db::all_paged, which reads 100 sessions or 2,000 audit entries at a time, with LIMIT/OFFSET over a total order.

I checked it against cargo xtask dev. I copied a session with a 5,000-character prompt 250 times and added 2,500 audit entries to the dev account, then ran fctl account export. The export had all 251 sessions and 2,512 audit entries, each once and in order. cargo xtask check lint passes.

Worth confirming

  • Assumption: The session list (GET /sessions, at most 201 rows with prompts) stays well under 50 MB with real prompts, so I left it as it is. It could only pass 50 MB if nearly every session in a page had a prompt of 100,000 four-byte characters.
  • Decision: Pages of an export are separate queries, not one snapshot. A session started while an export runs may be left out or, rarely, read twice. Before this change, the export's different queries were already separate.

fctl session


Note

Low risk. It changes only how the account data export reads from D1, in pages instead of single queries; nothing is written, and other endpoints are untouched.

What it does. D1 now fails any query whose result is over 50 MB, and the account export (GET /me/export, fctl account export) read all of a person's sessions, with prompts, and a year of audit log in single queries. This adds db::all_paged, which reads rows with LIMIT/OFFSET over a total order, and uses it for the four unbounded export reads, 100 sessions or 2,000 audit entries per page.

Pages are not a snapshot. Each page is a separate query, so a session created while an export runs may be missed or read twice.

fctl review's first look at 166bace · 2 files, +70 −24 · Review agent settings

…lt limit

From 4 November 2026, D1 fails a query whose result is over 50 MB. GET /me/export read all of a person's sessions (prompts of up to 100 000 characters each, twice) and their year of audit log in single unbounded queries; it now reads them 100 sessions or 2000 entries at a time. Every other D1 read is bounded by a key, a LIMIT or a small table.
@fctl-sh
fctl-sh Bot requested a review from kalaspuff as a code owner November 4, 2026 08:58
@kalaspuff
kalaspuff merged commit 03ddefc into main Nov 4, 2026
11 checks passed
@kalaspuff
kalaspuff deleted the fctl/631-can-you-look-into-https-developers-cloud branch November 4, 2026 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant