Skip to content

Avoid a full workspace search per Python file event - #1999

Open
om singhal (Om-singhaI) wants to merge 1 commit into
microsoft:mainfrom
Om-singhaI:fix/1989-coalesce-workspace-python-scans
Open

om singhal (Om-singhaI) wants to merge 1 commit into
microsoft:mainfrom
Om-singhaI:fix/1989-coalesce-workspace-python-scans

Conversation

@Om-singhaI

Copy link
Copy Markdown

Summary

Since 1.42.0 the workspaceHasPython watcher calls findFiles for every created or deleted marker file. Each call starts its own rg --files over the workspace, so a branch switch that touches a thousand Python files starts a thousand searches at once, which is the freeze reported in #1989.

The change stays in workspacePythonContext.ts and follows the approach suggested on the issue:

  • a created marker file or an opened Python document sets the context to true without a search
  • deletes and workspace folder changes share one rescan through createSimpleDebounce (500 ms); deletes are skipped while the context is already false
  • only one search runs at a time, and a rescan requested meanwhile runs once after it
  • findFiles gets a cancellation token, cancelled when the result is stale or on dispose
  • setContext is only called when the value changes

The activation scan and the excludes are unchanged.

Fixes #1989

Testing

A small harness fired watcher events at registerWorkspacePythonContext and counted findFiles calls (total / peak concurrent):

Burst Before After
1000 creates 1000 / 1000 0 / 0
1000 deletes 1000 / 1000 1 / 1
1000 creates then 1000 deletes, in the order VS Code delivers a branch switch 2000 / 2000 1 / 1
50 folder changes 50 / 50 1 / 1

The final context value was the same before and after in every case.

  • npm run compile-tests, npm run lint, and a prettier check on the changed files
  • npm run unittest: 3521 passing, 12 pending
  • new fake timer tests cover the bursts, deletes during a running scan, a create racing a scan, and disposal; all of them fail without the fix

The workspaceHasPython watcher called findFiles for every created or
deleted marker file, and each call starts its own ripgrep process, so a
branch switch that touches a thousand Python files started a thousand
workspace searches at once.

A created marker file or an opened Python document now sets the context
to true without searching. Deletes and workspace folder changes share
one rescan, debounced by 500 ms. Only one search runs at a time, a
rescan requested meanwhile runs once after it, and a search is
cancelled when its result is stale or the extension is disposed. The
context is only set when its value changes.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked this branch on Windows 11 against the case in my #1989 comment: the home folder open as the workspace. Your test file passes (35 of 35) and fails on main (18 failures, for example 1,001 scans for 1,000 creates). I also ran a harness with your stubs, a 40 s fake scan and 5 minutes of temp-file events. A delete every 250 ms went from 1,201 findFiles calls, 161 at once, to 1 call. Create then delete every second went from 601 calls, 81 at once, to 301 calls, one at a time. That removes the cause of our commit-limit crash.

One gap: every delete while the key is true still schedules a rescan, so pytest-style churn starts a new rg each cycle. Keeping the Uri that findFiles returned (maxResults 1 gives exactly one) and rescanning only when a delete hits that Uri would make those deletes free.

@nikicat

Copy link
Copy Markdown

One case this doesn't cover yet: a single scan that never finishes.

Linux, VS Code 1.145.2, Python Environments 1.42.0. The workspace has no marker files at all (a Rust/shell repo), but it does hold a Wine prefix, and Wine's dosdevices/z: is a symlink to /. The scan the extension starts is:

rg --files --hidden --case-sensitive --no-require-git \
   -g '**/{*.py,*.pyi,*.ipynb,pyproject.toml,requirements.txt,Pipfile,pyvenv.cfg,mspythonconfig.json}' \
   -g '!**/{node_modules,.git,site-packages}/**' ... --no-ignore --follow --no-config --no-ignore-global
  • --follow comes from search.followSymlinks (default true), because legacy findFiles passes no followSymlinks option. The scan goes through z: and walks the whole filesystem.
  • --no-ignore means .gitignore doesn't stop it either.
  • maxResults: 1 only stops rg once it finds a match. A workspace with no Python never produces one, so rg runs until it has seen every file reachable from /. I saw each one take 60–96% of a core for minutes, with more piling up behind it.

With this PR there would be one such rg instead of a pile. But a rescan queued by a delete or a folder change waits for the running search, so those updates would also wait for as long as that scan lasts. Two suggestions:

  1. Bound the search: cancel the token after a timeout (a few seconds), and treat a timeout as "unknown, keep the current value" rather than false.
  2. Use findFiles2 with followSymlinks: false (and useIgnoreFiles on), so the marker check stays inside the workspace whatever the user's search settings are.

Workaround for anyone affected: "search.followSymlinks": false in user settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants