Skip to content

fix(onlion): the manifest read is bounded too, it was the last unbounded step #147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wefio merged 2 commits into from
Nov 1, 2026
Merged

fix(onlion): the manifest read is bounded too, it was the last unbounded step #147

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7c3fc23
fix(onlion): the manifest read is bounded too, it was the last unboun…
wefio Nov 1, 2026
f4c6d21
fix(onlion): the early partial page uses its own counter object
wefio Nov 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 39 additions & 3 deletions onlion/backend/worker.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1205,6 +1205,8 @@ const SEARCH_DEADLINE_MS = 25_000;
const PENDING_READ_MS = 8_000;
/** How long /health waits for its storage probe before answering anyway. */
const HEALTH_PROBE_MS = 8_000;
/** How long a search waits for the manifest before answering a partial page. */
const MANIFEST_READ_MS = 8_000;

/** Features to embed and search per clause, bounded by `MAX_QUERY_FEATURES`.
*
Expand Down Expand Up @@ -1602,9 +1604,43 @@ async function search(env, input) {
// was absorbed by the CAS retry loop; as a pure read the path retries itself with a fresh
// manifest - and a 2026-10-30 15:26 /search 503 of 4.2 s is exactly the race this covered.
let manifestReads = 0;
let manifest = (await measure("manifest_read", () =>
loadManifest(env.MEMORY, keyFor(userId), userId),
)).manifest;
// Bounded, because it is the first object read of every search and it was the last unbounded one.
// Measured on the live endpoint 2026-11-01: one MEMORY.head stalled for over 45 s on /health, and
// a search issued right after an Add stalled past 90 s with the manifest read as the only step
// left before the bounded log tail. A slow bucket degrades the page; it does not hold the request
// until the platform kills it.
let manifest;
try {
manifest = (
await measure("manifest_read", () =>
hardDeadline(
() => loadManifest(env.MEMORY, keyFor(userId), userId),
Number(env.NMG_MANIFEST_READ_MS) > 0
? Number(env.NMG_MANIFEST_READ_MS)
: MANIFEST_READ_MS,
"manifest_read",
),
)
).manifest;
} catch (error) {
if (error?.code !== "embedding_deadline") throw error;
return searchAnswer({
candidates: [],
modes: new Set(),
partial: true,
degraded: "manifest_timeout",
started,
manifest: { parts: [], format: manifestFormat({ parts: [], format: "nmg-shards-v3" }) },
clauses,
contributions,
hasAppendOnly,
appendedById,
topK,
// The per-attempt counter is declared later, inside the attempt itself: nothing has been
// saved or changed when the manifest read is what failed.
persistence: { inline_delta_saves: 0, full_snapshot_saves: 0, changed_pages: 0 },
});
}
// Loaded on the first partition, not up front: a user whose writes are still in the log has no
// partitions at all (the harness's first questions are exactly that), and the manifest read is
// the only thing a search needs for them. Measured before this: /health at 25 ms with
Expand Down
Loading