Repository navigation
[diskann-inmem] Add support for filtered searches - #1601
Open
Mark Hildebrand (hildebrandmw) wants to merge 4 commits into
Open
Mark Hildebrand (hildebrandmw) wants to merge 4 commits into
Mark Hildebrand (hildebrandmw) wants to merge 4 commits into
Conversation
Mark Hildebrand (hildebrandmw)
requested review from
a team
and
a balanced review from Copilot
November 13, 2026 22:23
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Filtered search can deadlock when a user-supplied predicate modifies the same DashMap shard whose read lock is still held by the iterator.
1 open issue
What changed in this PR
Adds inline and multihop filtered searches to diskann-inmem, moving it closer to feature parity with the older in-memory implementation. Filtering uses epoch-protected ID mappings, while static-build integration tests remain separate.
Changes:
- Adds filtering support infrastructure, reusable read guards, and synchronous search execution.
- Routes ground-truth loading through shared binary readers.
- Refactors integration tests and updates configurations and baselines for filtered L2 searches.
| File | Description |
|---|---|
diskann/src/graph/search/multihop_filter_search.rs |
Makes multihop search parameters cloneable. |
diskann/src/graph/search/inline_filter_search.rs |
Adds adaptive-L accessors and cloneable search parameters. |
diskann/src/graph/glue.rs |
Adds a boolean-based filter decision constructor. |
diskann/src/error/ann_error.rs |
Converts staggered-reader errors into algorithm errors. |
diskann-tools/src/utils/search_index_utils.rs |
Uses the shared range ground-truth reader. |
diskann-inmem/src/repr/quantized/mod.rs |
Supports supplied read guards in quantized accessors. |
diskann-inmem/src/repr/mod.rs |
Defines read guards and updates accessor contracts. |
diskann-inmem/src/repr/internal/macros.rs |
Implements representation read-guard acquisition. |
diskann-inmem/src/repr/full.rs |
Threads read guards through full-precision accessors. |
diskann-inmem/src/provider/buildonly.rs |
Adapts static-build calls to updated accessor signatures. |
diskann-inmem/src/poll.rs |
Adds a tested single-poll synchronous executor. |
diskann-inmem/src/lib.rs |
Registers bitvector and polling modules. |
diskann-inmem/src/ids.rs |
Exposes ID-map iteration for predicate evaluation. |
diskann-inmem/src/epoch.rs |
Makes shared epoch guards available without quantization. |
diskann-inmem/src/bitvector.rs |
Adds a tested compact predicate bitvector. |
diskann-inmem/integration/support/mod.rs |
Registers filters and adds collection comparison support. |
diskann-inmem/integration/support/io.rs |
Adds buffered file opening with error context. |
diskann-inmem/integration/main.rs |
Separates static baselines and simplifies regression execution. |
diskann-inmem/integration/jsons/graph/static/f32.json |
Moves static builds to the partial integration job. |
diskann-inmem/integration/jsons/graph/static/f32-baseline.json |
Adds a dedicated static-build baseline. |
diskann-inmem/integration/jsons/graph/static/checks.json |
Adds static-build regression tolerances. |
diskann-inmem/integration/jsons/graph/spherical/two-bit-l2.json |
Adds two-bit spherical filtered L2 searches. |
diskann-inmem/integration/jsons/graph/spherical/two-bit-l2-baseline.json |
Records filtered-search results. |
diskann-inmem/integration/jsons/graph/spherical/two-bit-ip.json |
Migrates configuration; filtered runs remain empty. |
diskann-inmem/integration/jsons/graph/spherical/two-bit-ip-baseline.json |
Updates the baseline schema. |
diskann-inmem/integration/jsons/graph/spherical/two-bit-cosine.json |
Migrates configuration; filtered runs remain empty. |
diskann-inmem/integration/jsons/graph/spherical/two-bit-cosine-baseline.json |
Updates the baseline schema. |
diskann-inmem/integration/jsons/graph/spherical/one-bit-l2.json |
Adds one-bit spherical filtered L2 searches. |
diskann-inmem/integration/jsons/graph/spherical/one-bit-ip.json |
Migrates configuration; filtered runs remain empty. |
diskann-inmem/integration/jsons/graph/spherical/one-bit-ip-baseline.json |
Updates the baseline schema. |
diskann-inmem/integration/jsons/graph/spherical/one-bit-cosine.json |
Migrates configuration; filtered runs remain empty. |
diskann-inmem/integration/jsons/graph/spherical/one-bit-cosine-baseline.json |
Updates the baseline schema. |
diskann-inmem/integration/jsons/graph/spherical/four-bit-l2.json |
Adds four-bit spherical filtered L2 searches. |
diskann-inmem/integration/jsons/graph/spherical/four-bit-l2-baseline.json |
Records filtered-search results. |
diskann-inmem/integration/jsons/graph/spherical/four-bit-ip.json |
Migrates configuration; filtered runs remain empty. |
diskann-inmem/integration/jsons/graph/spherical/four-bit-ip-baseline.json |
Updates the baseline schema. |
diskann-inmem/integration/jsons/graph/spherical/four-bit-cosine.json |
Migrates configuration; filtered runs remain empty. |
diskann-inmem/integration/jsons/graph/spherical/four-bit-cosine-baseline.json |
Updates the baseline schema. |
diskann-inmem/integration/jsons/graph/scalar/two-bit-cosine.json |
Adds filtered searches for the configured L2 dataset. |
diskann-inmem/integration/jsons/graph/scalar/two-bit-cosine-baseline.json |
Records filtered-search results. |
diskann-inmem/integration/jsons/graph/scalar/one-bit-l2.json |
Adds one-bit scalar filtered L2 searches. |
diskann-inmem/integration/jsons/graph/scalar/four-bit-ip.json |
Migrates configuration; filtered runs remain empty. |
diskann-inmem/integration/jsons/graph/scalar/four-bit-ip-baseline.json |
Updates the baseline schema. |
diskann-inmem/integration/jsons/graph/scalar/eight-bit-l2.json |
Adds eight-bit scalar filtered L2 searches. |
diskann-inmem/integration/jsons/graph/scalar/eight-bit-l2-multi-insert.json |
Adds filtered searches after scalar multi-insert builds. |
diskann-inmem/integration/jsons/graph/scalar/eight-bit-l2-multi-insert-baseline.json |
Records multi-insert filtered-search results. |
diskann-inmem/integration/jsons/graph/scalar/eight-bit-l2-baseline.json |
Records filtered-search results. |
diskann-inmem/integration/jsons/graph/product/l2.json |
Adds product-quantized filtered L2 searches. |
diskann-inmem/integration/jsons/graph/product/ip.json |
Migrates configuration; filtered runs remain empty. |
diskann-inmem/integration/jsons/graph/product/ip-baseline.json |
Updates the baseline schema. |
diskann-inmem/integration/jsons/graph/product/cosine.json |
Migrates configuration; filtered runs remain empty. |
diskann-inmem/integration/jsons/graph/product/cosine-baseline.json |
Updates the baseline schema. |
diskann-inmem/integration/jsons/graph/full/integration-u8.json |
Adds filtered searches for unsigned-byte vectors. |
diskann-inmem/integration/jsons/graph/full/integration-u8-baseline.json |
Records unsigned-byte filtered-search results. |
diskann-inmem/integration/jsons/graph/full/integration-i8.json |
Adds filtered searches for signed-byte vectors. |
diskann-inmem/integration/jsons/graph/full/integration-i8-baseline.json |
Records signed-byte filtered-search results. |
diskann-inmem/integration/jsons/graph/full/integration-f32.json |
Adds filtered searches for full-precision vectors. |
diskann-inmem/integration/jsons/graph/full/integration-f32-multi-insert.json |
Adds filtered searches after full-precision multi-insert builds. |
diskann-inmem/integration/jsons/graph/full/integration-f32-multi-insert-baseline.json |
Records multi-insert filtered-search results. |
diskann-inmem/integration/jsons/graph/full/integration-f32-baseline.json |
Records full-precision filtered-search results. |
diskann-inmem/integration/jsons/graph/full/integration-f16.json |
Adds filtered searches for half-precision vectors. |
diskann-inmem/integration/jsons/graph/full/integration-f16-baseline.json |
Records half-precision filtered-search results. |
diskann-inmem/integration/index/stats.rs |
Consolidates statistics and regression checks. |
diskann-inmem/integration/index/ops.rs |
Loads labels and runs filtered-search recall checks. |
diskann-inmem/integration/index/mod.rs |
Registers refactored integration modules. |
diskann-inmem/integration/index/factory.rs |
Separates dynamic and static index construction. |
diskann-benchmark/src/utils/datafiles.rs |
Reuses shared fixed and staggered binary readers. |
Review effort: Balanced
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1601 +/- ##
==========================================
- Coverage 92.14% 91.22% -0.92%
==========================================
Files 621 628 +7
Lines 130552 131862 +1310
==========================================
+ Hits 120293 120297 +4
- Misses 10259 11565 +1306
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Mark Hildebrand (hildebrandmw)
force-pushed
the
mhildebr/filtered
branch
2 times, most recently
from
November 14, 2026 01:15
fbfd1d5 to
071a09b
Compare
Mark Hildebrand (hildebrandmw)
force-pushed
the
mhildebr/filtered
branch
from
November 14, 2026 01:36
071a09b to
b6e9e5a
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add support for
InlineFilterandMultihopFiltersearches todiskann-inmem.This PR is huge for a number of reasons:
diskann-inmem's integration test framework was structured to split the static build into its own benchmark. This is because I have no interest in supporting filtered search yet for the static build path and instead of special casing the un-supported methods and somehow trying to keep everything in-sync.diskann-label-filterordiskann-tools.It also adds much more robust integration test coverage for inline-filtered and multihop-filtered searches, which we are currently lacking.
In addition, I had to kind of fight infrastructure at every step and take on some technical debt. The goal here is to be able to deprecate inmem1 through near-feature-parity with inmem2, then fix-up the abstractions in
diskannto undo the workarounds needed in this PR.To help pretend it's a little more reviewable, I've broken it into 4 commits.
Commit 1: Support Loading Range Groundtruth
Filtered and range searches use the range-search file format since the number of items satisfying the predicate are not necessarily uniform.
We currently do not have a central shared utility for loading this format. This PR adds
read_staggered_bintodiskann_utils::iofor reading these types of files and reroutesdiskann-benchmark/src/utils/datafiles.rsanddiskann-tools/src/utils/search_index_utils.rsto use this shared helper.In addition, this PR adds a few helpers to
diskann:boolbased constructor forDecision.AdaptiveLand aClonebounds toInlineFilterSearchandMultihopFilterSearch.Commit 2: Add filtered search to
diskann-inmemUp to now, we've largely been relying on some form of
QueryLabelProviderfor predicate evaluation. As noted in the module level documentation, there is some magic needed for aQueryLabelProviderto work, especially in the presence of a dynamic index: it needs to somehow operate on the internal ID space and needs to handle the race window between the predicate evaluation and the distance evaluation.This commit addresses those issues in the following way:
The public filtering interface takes a function applied to external IDs and uses that to build a bit-vector over internal IDs.
Internal bitvector construction and subsequent use are all done under an epoch guard, which prevents ID recycling and thus a situation where an internal ID is marked as accepted, gets deleted, and then rebound to a new external ID (by preventing the rebinding).
This also naturally handles the evaluation time mismatch between the bit-vector and the distance. If an item is deleted after the evaluation of the bit-vector but before the distance calculation, the distance evaluation returns
Noneand the item is not returned.If the item is deleted after the distance evaluation, then final post-processing will not find an valid ID translation and it still will not be returned.
To help amortize the set-up cost while benchmarking algorithms, a scoped
Filteredis introduced that has theFilteredAccessorready-to-go. This then uses synchronous methods to run multiple inline-filtered and multihop-filtered searches.Synchronous methods can be used because we know the internally created futures will always run to completion on their first poll.
This last bit is direction I think
diskannanddiskann-inmemneed to move towards. By not usingDiskANNIndexas the public API and instead simply havingdiskannexport algorithms, we increase the flexibility of user code to do the most efficient and ergonomic thing.The section marked with the
HERE BE DRAGONSbanner shows how hard I had to fight against the currentdiskannAPI to make this work like I wanted. My longterm vision after we deprecate inmem1 is that we can reworkdiskannso these kinds of crazy hacks are no longer needed.Commit 3: Reworking Integration Tests
This consists of two parts: the first is adding predicate evaluation and the second is refactoring the integration tests.
For predicate evaluation, I opted to add a small inverted-index based evaluator for base labels and the simple query predicates in the test YFCC dataset. This is done to largely avoid taking dependencies on the bitmap computation in
diskann-toolsand transitivelydiskann-label-filter.For the second part, I kind of shot myself in the foot with #1589 with the static index shadowing the
Indextrait object. The refactor is generally pretty boring: theIndextrait gets split intoPartialIndex(with support for just insert, search, and counter retrieval), andIndex, which adds multi-insert and filtered search. The static index then gets a separate benchmark for justPartialIndex. All of the changes are basically shuffling things around so this works and then adding inline-filtered and multi-hop filtered searches to the fullIndexruns.Note that only the integration tests that use L2 are extended to have filtered search runs since that's all we have groundtruth for. All other runs just have empty arrays for their filtered searches.
Commit 4: Regenerating Baselines
This is just rerunning the integration tests with
DISKANN_TEST=overwriteto regenerate the expected results.