Best practices for scanning large numbers of small binary+text files — Nginx cache use case

[hsntgm]

Background / Use Case

I'm the author of a free WordPress plugin Nginx Cache Purge Preload that uses ripgrep as a RG Purge feature — as of v2.1.6 — to locate and purge specific Nginx cache files. I'd love input from maintainers or the community on whether my current ripgrep usage is well-optimized for this workload.

Workload characteristics:

• Nginx cache directories can contain 50k–500k+ files spread across a commonly 2-level hash subdirectory tree (e.g. /var/cache/nginx/ab/12/abc123)
• Each cache file is a small binary file, typically 2 KB – 64 KB
• The match target is always a KEY: line near the very top of each file (within the first ~4 KB), formatted like:

KEY: $scheme$request_method$host$request_uri

KEY: httpGETexample.com/my-post/

• Files are not text files — they have binary Nginx headers
• The target scan filesystem is sometimes a FUSE mount (bindfs), sometimes a RAM-backed tmpfs (/dev/shm), sometimes a persistent disk according to user env.

What I'm actually scanning inside each cache file ?

Nginx cache files have this structure:

[binary metadata header — ~300 bytes, nginx version-dependent]
KEY: httpGETexample.com/my-post/
[null bytes / padding]
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
...
[full HTTP response body]

So every cache file is a mixed binary+text file. I only ever care about one line: KEY:
That single line contains the cache key — typically $scheme$request_method$host$request_uri — which is what I match against to identify which URL this cache file belongs to.

Current core ripgrep commands used in plugin

1. Single-page purge — find cache file(s) for one or more specific URLs in one pass:
purge.php#L531

rg -m 1 --text -E none --no-unicode --no-messages --no-ignore --no-config --no-mmap --no-heading \
  '^KEY: .*(example\.com/post-a/$|example\.com/post-b/$)' \
  /var/cache/nginx/

This uses a combined alternation pattern so the directory is walked exactly once regardless of how many URLs need purging.

---

2. Full cache inventory (two-pass approach):
advanced.php#L1030 - advanced.php#L1217

Pass 1 — Collect redirect files to exclude them:

rg -F --text -l -m 1 -E none --no-unicode --no-heading --no-ignore --no-config --no-messages --no-mmap \
  -e 'Status: 301' -e 'Status: 302' \
  /var/cache/nginx/

Pass 2 — extract KEY: line from every non-redirect cache file:

rg --text -m 1 -E none --no-unicode --no-heading --no-ignore --no-config --no-messages --no-mmap \
  '^KEY: [^\r\n]+' \
  /var/cache/nginx/

---

3. Post-preload cache confirmation — verify a single URL was cached:
advanced.php#L829

rg -l -m 1 --text -E none --no-unicode --no-messages --no-ignore --no-config --no-mmap \
  '^KEY: .*example\.com/my-post/$' \
  /var/cache/nginx/

Questions

Q1: --no-mmap vs --mmap for small files
I'm using --no-mmap based on general advice that mmap has overhead for small files. Is this correct for files mostly under 32 KB? Or is there a size threshold where mmap becomes beneficial?

Q2: Is there a way to limit how many bytes ripgrep reads per file?
Since KEY: is always within the first ~4 KB of every cache file, I currently rely on -m 1 to stop reading after the first match within a file, but ripgrep still has to scan until it finds KEY: (or reaches EOF if absent). Any better approach here?

Q3: Thread count tuning for I/O-bound small-file workloads
Is there a recommended -j / --threads value for workloads that are heavily I/O-bound (many small files on tmpfs or FUSE) rather than CPU-bound? Does ripgrep auto-tune this, or should I expose a user-configurable -j option in my plugin?

Q4: Two-pass vs single-pass for the full cache inventory scan
Currently I do two passes (redirect exclusion + KEY extraction). A single pass with something like:

rg --text -m 1 --no-heading ... -e '^KEY: [^\r\n]+' -e 'Status: 30[12]' /var/cache/nginx/

...would return both match types per file in one walk, but I'd need to post-process output to separate them. Is a single-pass approach actually faster at this scale, or does the Linux dentry-cache warm-up from pass 1 make the two-pass approach net-faster?

Q5: Any other flags I'm missing or redundant for this workload?

---

The performance difference between ripgrep and the PHP iterator (read head 4KB) is much smaller than expected. This is precisely why I'm looking for further optimization opportunities.

Some test results here

Best~Hasan

[apoorva-01]

Your flags are already in good shape, so a few of these are "you're fine, stop worrying." Going in order:

Q1 (mmap): --no-mmap is a no-op in your commands. ripgrep only memory-maps when it's searching a single file you name directly. The moment you point it at a directory it switches to parallel traversal and doesn't mmap at all, so there's nothing to turn off. You can drop --no-mmap from every one of these. mmap only enters the picture for single large files, which isn't your workload.

Q2 (cap bytes per file): there's no "read at most N bytes then give up" flag. -m 1 is the right tool and it's already doing the work: it stops the file as soon as it hits the first match. Since KEY: sits ~300 bytes in, the scan to reach it is a few hundred bytes of a file you've already paid to open and read. That cost is noise. Don't chase it.

Q3 (threads): ripgrep auto-sets threads to your logical CPU count and parallelizes the tree walk, not the inside of one file. For many-small-files on tmpfs/FUSE your bottleneck is open()/stat()/read() latency, not CPU, so piling on threads past the default mostly buys you contention, not speed. Leave it on default. Only expose -j if a FUSE-heavy user actually measures a win, and let them opt in.

Q4 (one pass vs two): single pass is almost certainly faster at 50k-500k files. The warm dentry cache on pass 2 saves you some open/stat latency, but you still re-read every file's bytes a second time, and at this file count the per-file syscalls dominate. One walk with -e '^KEY: [^\r\n]+' -e 'Status: 30[12]' and splitting the output in PHP halves the file I/O. Worth measuring on your own test set, but I'd bet on the single pass.

Q5: nothing redundant except the --no-mmap from Q1. --no-config, --no-ignore, --no-messages, -E none, --no-unicode, -F for the fixed-string pass are all the right calls here.

On the "smaller-than-expected gap vs the PHP head-4KB reader": that's the tell that you're I/O-bound, not scan-bound. When the whole job is open a file, read a few KB, close, times 200k, ripgrep's fast matching engine barely gets to flex because it's waiting on the filesystem the same as your PHP loop is. The realistic win left is fewer syscalls (the single pass), not regex tuning. If the storage is the persistent-disk case, that's where a faster mount or more read parallelism would actually move the needle.

[hsntgm]

@apoorva-01

This is golden. I will drop all --no-mmap

Q1 (mmap): --no-mmap is a no-op in your commands. ripgrep only memory-maps when it's searching a single file you name directly. The moment you point it at a directory it switches to parallel traversal and doesn't mmap at all, so there's nothing to turn off. You can drop --no-mmap from every one of these. mmap only enters the picture for single large files, which isn't your workload.

Merged. psaux-it/nginx-fastcgi-cache-purge-and-preload@8347934

Q4 (one pass vs two): single pass is almost certainly faster at 50k-500k files. The warm dentry cache on pass 2 saves you some open/stat latency, but you still re-read every file's bytes a second time, and at this file count the per-file syscalls dominate. One walk with -e '^KEY: [^\r\n]+' -e 'Status: 30[12]' and splitting the output in PHP halves the file I/O. Worth measuring on your own test set, but I'd bet on the single pass.

Overall, I see the picture more clearly now. I previously thought that we were paying a cost with -m 1 for unmatched cases because ripgrep lacks a --max-bytes option to cap reads and would always scan to EOF—implying that a PHP recursive scan with a 4 KB cap would be equally performant. But it turns out that for files up to 64 KB, the entire file is already loaded into memory with a single read() syscall, so in my case there is no extra I/O penalty and no benefit from such read capping via ripgrep, since cache files are generally smaller than 64 KB.

Architecturally, I also use a single-pass rg walk to build an authoritative Transposition Table (with object caching). Every subsequent single-page cache purges hits the TT first as a fast path— a hit means an instant cache purge with no directory walk. On a TT miss I delegate to ripgrep for a full purge walk and update/rebuild TT.

Helped me squeeze every last drop of juice out of ripgrep by optimizing its flags.
Thank you.