Skip to content

ADFA-6854: Lazy Kotlin VFS over JarListing - #2231

Draft
itsaky-adfa wants to merge 11 commits into
perf/ADFA-6854-jar-listingfrom
perf/ADFA-6854-kotlin-vfs
Draft

itsaky-adfa wants to merge 11 commits into
perf/ADFA-6854-jar-listingfrom
perf/ADFA-6854-kotlin-vfs

Conversation

@itsaky-adfa

@itsaky-adfa itsaky-adfa commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Draft, part 4 of 4 of the ADFA-6854 stack.

Fixes ADFA-6854. Do not merge. CI fails until the kt-android release, and the open items below block the stack.

Stack (bottom to top):

  1. Kotlin library scopes check membership by root
  2. javac package map shared per JAR
  3. JarListing: one packed index per JAR for zipfs2 and javac
  4. This PR: Lazy Kotlin VFS over JarListing

What

  • ListingJarFileSystem is the Kotlin LSP's jar:// VirtualFileSystem. It is backed by JarListingCache.SHARED and replaces IntelliJ's CoreJarFileSystem, which built a CoreJarVirtualFile for every entry of every JAR.
    • A VirtualFile is created only for entries something asks for. It stays the same object while referenced (weak-value cache per JAR).
    • Content is read through one file handle per JAR, at the listing's offsets.
    • A JAR replaced in place is reread through the cache's stat check. Old files then report isValid() == false, and handler generations keep a slow lookup from installing a stale handler.
  • LibraryRootsScope falls back to matching a file's JAR root by URL, so a replaced JAR's entries stay in their module.
  • KotlinApplicationEnvironmentPin installs the file system through a new kt-android hook, KotlinCoreApplicationEnvironment.jarFileSystemFactory (patch 0011). It does this before any application environment exists, and checks the result after acquiring it.

Measured, whole stack

OnePlus Pad 2, cold open of CoGo at stage ffaa75be84, heap dump after open. "stack" is this branch (3254dcc1db) with the locally built kt-android jar.

Copies of JAR entries stage stack
javac package structures (PR 2) 39.2 MB 26.6 MB
zipfs2 index, incl. JarListing (PR 3) 36.7 MB 13.7 MB
Kotlin JAR VFS (CoreJarHandler -> lazy VFS) 18.3 MB 0.1 MB
Kotlin library scopes (PR 1) 41.4 MB 0.3 MB
Total 135.6 MB 40.7 MB
Open timings and heap stage stack
Strongly reachable heap after open 495.7 MB 378.3 MB
Library index 131.3 s 109.2 s
Kotlin source index 919.9 s 527.8 s

The timings are from the a71e6da99c stack run, on battery (52%), while the baseline ran on the charger, so treat them as indicative. Heap figures are not affected.

A moto g34 milestone run (the device the earlier memory milestones were measured on) is in progress.

Open items (block merging)

  1. Scope completion regressed. In JavaModuleAccessibilityChecker.kt (warm open, scope completion with prefix jav) it takes 12.5 / 14.2 / 13.6 s, against 2.3 / 2.1 / 2.2 s on stage. Member completion there is unchanged (0.49 / 0.08 / 0.07 s against 0.58 / 0.04 / 0.05 s). Measured so far:

    • simpleperf during the slow completion: 83% of samples are in GC (HeapTaskDaemon);
    • the heap sits at 440-453 MB of 512 MB during completion on the stack, against 285-383 MB on stage;
    • heap dump before completion: soft-only reachable 114.8 MB on the stack against 42.4 MB on stage. Most of the growth is Kotlin known-class-name sets (18,974 LinkedHashSets held in ConcurrentHashMap caches that are only softly reachable).

    Inferred, not yet confirmed: with the JAR copies gone, softly cached LL FIR library sessions survive longer, refill the freed heap up to the limit, and completion then spends its time in GC. Next: find the holder of those soft references and cap or release it.

  2. javac residual, 26.6 MB. The shared package map now works (PR 2). What remains is javac's per-manager pathsAndContainersByLocationAndRelativeDirectory in each of 98 file managers. Reducing it needs a change inside javac or fewer file managers; 98 managers for one project may also mean managers pile up across syncs, still to be checked.

  3. kt-android release. subprojects/kotlin-analysis-api/build.gradle.kts still pins v2.3.255-dc58d97, which lacks the hook. One kt-android release with patch 0011 follows once the stack is verified, then the tag and SHA-256 bump lands here.

  4. Kotlin source indexing reported indexed=2025 of 2,027 scanned files on both stack runs (stage: 2,027). No index errors were logged; still to be explained.

Tests

  • ListingJarFileSystemTest (13): shape against CoreJarFileSystem, identity, laziness, STORED/DEFLATED/stub-prefix contents, rename and same-size replacement, handler races.
  • LibraryRootsScopeTest, including a test that the environment's JAR file system is ListingJarFileSystem.
  • JarListingCacheTest (11).
  • :lsp:kotlin:testV7DebugUnitTest 649/649 on the new file system (with the local kt-android jar).

@atlassian

atlassian Bot commented Nov 6, 2026

Copy link
Copy Markdown

To enable Code Reviewer, link your GitHub account to your Atlassian account.

This is a one-time task that takes less than a minute. Once your account is linked, resubmit the pull request to trigger a code review.

@itsaky-adfa
itsaky-adfa added this pull request to stack #2232 November 6, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant