Repository navigation
perf(lexer): lowercase string prefixes only, so a cold first prompt fits - #231
Merged
Merged
Conversation
The painter's Lexer is made at a terminal session's first prompt. With an empty code cache that make compiles every state and then records the plan for its cache group, and the plan's lookups are linear walks done once a state, so its cost grows with the square of the states. The 25 prefixes of both cases gave 100 string rules and 686 states: the plan alone allocated 43M objects and the first prompt peaked at 2,986 MB (macOS footprint, x-lang v0.47.0), which is what x-os's warm step was killed for in its 3.6 GB podman VM. The 9 lowercase prefixes give 254 states and a cold first prompt of 745 MB. An uppercase prefix now paints as a name, then its string. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The painter's Lexer is made at a terminal session's first prompt. With an empty code cache, which is the state x-os's warm step runs in, that make compiles every state. It then records the plan for its cache group (
%planin x-lang'sx/reader/lexer.x). The plan's lookups are linear walks done once per state, so its cost grows with the square of the number of states.Measured peak footprint for
x -l pythonat a terminal, from start to its first prompt, with an emptyX_ASM_CACHE_DIR(macOS, guard.sh):%planallocatesThese are with x-lang v0.47.0, the pair x-os pins. The peak was the same on v0.44.0, v0.46.0, before #227 and after it, so it has not grown. On Linux it was the 2.68 GB
x-binkilled in x-os's 3.6 GB podman VM.Cost: an uppercase prefix (
F"z",BR"w") now paints as a name, then its string. 09-line's expectation says so. Lexing for the interpreter is unaffected.Verified on v0.46.0 with
specs.sh: 09-line 7/0, 91-repl-editor 14/0, C0-syntax 5/0. On v0.47.0, a cold terminal session typed six lines (brackets, prefixed strings, a comprehension, number forms), checked every answer, and peaked at 745 MB.The general fix is a linear
%planin x-lang. A prototype brought this lexer's peak to 1,697 MB and could go lower. It would need a release.🤖 Generated with Claude Code