Skip to content

perf(lexer): lowercase string prefixes only, so a cold first prompt fits - #231

Merged
jonruttan merged 1 commit into
mainfrom
perf/lexer-prefixes
Nov 6, 2026
Merged

jonruttan merged 1 commit into
mainfrom
perf/lexer-prefixes

Conversation

@jonruttan

Copy link
Copy Markdown
Owner

The painter's Lexer is made at a terminal session's first prompt. With an empty code cache, which is the state x-os's warm step runs in, that make compiles every state. It then records the plan for its cache group (%plan in x-lang's x/reader/lexer.x). The plan's lookups are linear walks done once per state, so its cost grows with the square of the number of states.

Measured peak footprint for x -l python at a terminal, from start to its first prompt, with an empty X_ASM_CACHE_DIR (macOS, guard.sh):

states %plan allocates peak
main (25 prefixes, both cases: 100 string rules) 686 43.0M objects 2,986 MB
this branch (9 lowercase prefixes: 36 string rules) 254 6.5M objects 745 MB

These are with x-lang v0.47.0, the pair x-os pins. The peak was the same on v0.44.0, v0.46.0, before #227 and after it, so it has not grown. On Linux it was the 2.68 GB x-bin killed in x-os's 3.6 GB podman VM.

Cost: an uppercase prefix (F"z", BR"w") now paints as a name, then its string. 09-line's expectation says so. Lexing for the interpreter is unaffected.

Verified on v0.46.0 with specs.sh: 09-line 7/0, 91-repl-editor 14/0, C0-syntax 5/0. On v0.47.0, a cold terminal session typed six lines (brackets, prefixed strings, a comprehension, number forms), checked every answer, and peaked at 745 MB.

The general fix is a linear %plan in x-lang. A prototype brought this lexer's peak to 1,697 MB and could go lower. It would need a release.

🤖 Generated with Claude Code

The painter's Lexer is made at a terminal session's first prompt.  With an
empty code cache that make compiles every state and then records the plan
for its cache group, and the plan's lookups are linear walks done once a
state, so its cost grows with the square of the states.  The 25 prefixes of
both cases gave 100 string rules and 686 states: the plan alone allocated
43M objects and the first prompt peaked at 2,986 MB (macOS footprint,
x-lang v0.47.0), which is what x-os's warm step was killed for in its
3.6 GB podman VM.  The 9 lowercase prefixes give 254 states and a cold
first prompt of 745 MB.  An uppercase prefix now paints as a name, then
its string.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jonruttan
jonruttan merged commit 47b3aba into main Nov 6, 2026
12 checks passed
@jonruttan
jonruttan deleted the perf/lexer-prefixes branch November 6, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant