Skip to content

Make the pre-commit hooks about twice as fast and fix the sign-off line - #282

Open
joshwong-cs wants to merge 9 commits into
mainfrom
perf/faster-hooks
Open

joshwong-cs wants to merge 9 commits into
mainfrom
perf/faster-hooks

Conversation

@joshwong-cs

@joshwong-cs joshwong-cs commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Note

AI-Assisted (Opus 5.5) with profiling, tests. Human proposed solutions and AI implementation (Sonnet 5.5).

  • Make the security hooks twice as fast
  • Make this repo run its own hooks the same way every other repo does (no reason to hold itself to a different security model)
  • Fix how the commit-message hook adds the sign-off line
Before After
Commit-message hook 0.95s 0.43s
Security scan, code files only 3.1s 1.6s
Security scan with a text file containing personal data 3.1s 1.7s
Personal data scan of a 2000-line CSV 3.15s 1.74s

Why it was slow

  • Every scan waited about a second for proxy.py to shut down because it only checks for shutdown once a second. We now wake it up straight away.
  • The commit-message hook loaded the whole scanner just to add one line.
  • The hook checks GitHub for a newer release on every commit and it ran before the scans. Now it runs alongside them.
  • The personal data scan loaded spaCy's English model on every commit even when no files needed checking. When it did run, it held up trufflehog.
  • The model also ran steps whose output nothing used.
  • Everyone in the department runs them for every commit! It adds up!

Other changes

This repo's own hooks now run in the published image. Commits here are scanned exactly like commits anywhere else, with the same trufflehog version and English model. Contributors no longer need trufflehog installed to commit and make run-hook-python still works for testing locally. The catch is that Python packages come from the image, so a PR that adds one fails these hooks. CONTRIBUTING explains how to build the image from your branch so you can still commit.

The sign-off line is now added properly. The old code:

  • deleted every line starting with "Signed-off-by", including people's own sign-offs and ordinary sentences
  • separated the sign-off from lines like "Co-authored-by", so git stopped reading those as sign-off-style lines
  • added another blank line every time it ran
  • put the sign-off where our PR check can't see it when the message had a "---" line, and lost it entirely with git commit -v and a custom comment character

Now git adds the line, and only if it isn't already there.

aiohttp is gone. The release check was the only thing using it so it now uses Python's built-in urllib. (something something supply chain risks)

Commits

  1. Run this repo's own hooks in the published Docker image (chore: Run this repo's own hooks in the published Docker image #284)
  2. Load hook modules only when their subcommand runs
  3. Shut down proxy.py without waiting for its poll timeout
  4. Skip loading spaCy when no files need a personal data scan
  5. Run Presidio in a worker thread
  6. Disable unused spaCy components in Presidio
  7. Check for a newer release alongside the scans
  8. Replace aiohttp with urllib for the release check
  9. Import GitPython only for --github-action scans
  10. Add the Signed-off-by trailer with git interpret-trailers (fix: Add the Signed-off-by trailer with git interpret-trailers #283)

Noteworthy

  • The faster proxy.py shutdown relies on how proxy.py works internally. If that changes, scans just take an extra second again. It can't block a commit, and blocked sites are still blocked.
  • The model steps we skip are switched back on automatically if anyone re-enables the part of Presidio that needs them. Results on our test data didn't change.
  • Removed dependencies: aiohttp, the multidict pin, pytest-aiohttp and pytest-asyncio.
  • Known gap: if someone sets a custom comment character only in their global git config and amends without -v, each amend adds another sign-off line. The PR check still passes.
  • CONTRIBUTING also loses some trailing spaces on unrelated lines.

Type of change

  • Refactoring (made code better without changing its behaviour)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

How this has been tested

  • All 174 tests pass. Each commit passes on its own.
  • Ran both hooks in the published image with this code. Findings and exit codes match the current release.
  • Made real commits through the image's git and ran our PR check on each one. Sign-offs, co-authors, "---" lines, git commit -v and amends all pass.
  • Checked that building the image from a branch with make build-docker works for the local hooks.

Checklist

  • I have performed a self-review of my code
  • I have commented my code in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings

Reviewer Checklist

  • I have reviewed the PR and ensured no secret or sensitive values are present

@joshwong-cs
joshwong-cs requested a review from a team as a code owner November 17, 2026 10:34
@github-actions

Copy link
Copy Markdown

Your PR has commits that are missing the Signed-off-by trailer, which is added by the pre-commit hooks. To install them, run pre-commit install --install-hooks --overwrite -t commit-msg -t pre-commit, or follow the instructions at https://github.com/uktrade/github-standards/blob/main/README.md#usage. If the commit was made in the GitHub web UI, installing the hooks will not help. Only applying suggested changes and updating the branch are allowed there, so make the change locally instead

1 similar comment
@github-actions

Copy link
Copy Markdown

Your PR has commits that are missing the Signed-off-by trailer, which is added by the pre-commit hooks. To install them, run pre-commit install --install-hooks --overwrite -t commit-msg -t pre-commit, or follow the instructions at https://github.com/uktrade/github-standards/blob/main/README.md#usage. If the commit was made in the GitHub web UI, installing the hooks will not help. Only applying suggested changes and updating the branch are allowed there, so make the change locally instead

validate_scan only adds a commit trailer but cli.py imported the full
scanning stack (Presidio, spaCy, aiohttp, proxy.py) at startup.
Deferring each hook import to its subcommand cuts validate_scan from
0.95s to 0.43s.

Signed-off-by: BIST pre-commit check
proxy.py acceptors only check for shutdown after their 1-second
select() call returns, adding ~0.8s to every run_scan. Run a single
acceptor and wake it with a dummy connection once trufflehog exits to
avoid that wait.

Signed-off-by: BIST pre-commit check
The Presidio analyzer loaded on every run, even when no staged file had
an extension Presidio scans. Check paths first, and import and load
Presidio only when files need scanning, saving roughly 0.4s. Files are
scanned in a plain loop, as the per-file tasks never ran concurrently.

Signed-off-by: BIST pre-commit check
Loading and running the Presidio analyzer blocked the event loop. That
didn't slow the scan because trufflehog already starts first but it
held up every other task including the release check once it runs
alongside the scans. Each CSV is also read in one go rather than line
by line which is slightly faster on large files.

Signed-off-by: BIST pre-commit check
Presidio never uses spaCy's dependency parser and with only regex-based
recognizers enabled the named entity recognizer did no useful work
either. Disabling both roughly halves scan time with unchanged findings
on test data. NER stays enabled if an NLP-based recognizer is
re-enabled.

Signed-off-by: BIST pre-commit check
The GitHub release check ran before the scans, delaying them by up to
its 1-second timeout. Run it concurrently with the scans in run(),
scheduled after them so proxy.py forks first.

Signed-off-by: BIST pre-commit check
The release check was the only thing using aiohttp. Use urllib instead
and remove aiohttp, its multidict pin, pytest-aiohttp and
pytest-asyncio. This also saves about 0.06s per run_scan.

Signed-off-by: BIST pre-commit check
Importing GitPython spawns a `git version` subprocess, although it is
only used with --github-action. Defer the import so pre-commit scans
skip it.

Signed-off-by: BIST pre-commit check

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant