add bf_tree benchmark infrastructure

[JordanMaples]

Summary

Adds benchmark support for the bf_tree storage backend — both full-precision and spherical-quantized, in static (build-once) and streaming (insert/delete/search) modes.

What's included

Core crate fix (diskann/src/graph/index.rs):

• VisitedAndTopK ordering fix: inplace_delete_inner now pre-fetches the delete element before calling delete(), so that VisitedAndTopK can use the deleted vector as a search query. This is critical for hard-delete providers (bf-tree) where the data is erased on delete. Soft-delete providers (in-mem) are unaffected since data remains accessible either way. All three delete methods (OneHop, TwoHopAndOneHop, VisitedAndTopK) continue to work for both provider types.

Benchmark variants (4 tags):

• graph-index-bftree-full-precision-f32 — static build + search
• graph-index-bftree-stream-full-precision-f32 — streaming with runbook
• graph-index-build-bftree-spherical-quantization — static spherical (1/2/4-bit)
• graph-index-stream-bftree-spherical-quantization — streaming spherical (1/2/4-bit)

Bug fixes in diskann-bftree:

1. Hard delete with VisitedAndTopK support: bf-tree now performs true hard deletes via Delete::delete. The ordering fix in the core crate ensures the delete element is captured before erasure, enabling all three inplace_delete methods to work correctly.
2. Max-degree mismatch in append_vector: The old append_vector would resize the neighbor list to self.dim and write the count at dim - 1, but set_neighbors wrote a variable-length buffer. This inconsistency caused bf-tree page fragmentation and potential read errors when the two code paths produced different value sizes for the same key. Fix: both paths now write a fixed-size dim-element buffer using the same format (|neighbors|padding|count|).
3. Reusable buffer in hot path: set_neighbors and append_vector previously allocated on every call. Replaced with a caller-provided &mut [u8] scratch buffer, eliminating per-call heap allocations.

Benchmark infrastructure:

• Expose bf_tree internal config (buffer sizes, fanout, promotion rates) as JSON input fields with validation
• Shared helpers: bftree_parameters_from(), run_streaming() in dedicated util module
• GenericStats::empty replaced with Option<GenericStats> for maintain stats (bf_tree has no separate drop_deleted/release phases)
• Minimum circular buffer size validated early with a friendly error message (must be ≥ 2× leaf_page_size)

Example configs

See diskann-benchmark/example/graph-index-bftree*.json for ready-to-run configurations targeting test_data.

Tests on CosmosDBDevBox VM (16 vCPU 64GB RAM 2,048GB SSD)

Results from MSTuring-1M

Full Precision streaming:

 max_degree=64, L_build=128, alpha=1.2, 4GB bf_tree buffers, 4 build threads, 2 search threads

┌───────┬────────┬─────────┬───────────────────┬──────────────────────────┬────────────────────┐
│ Stage │ Op     │ Vectors │ Avg / p99 Latency │ Recall@10 (L=32 / L=200) │ QPS (L=32 / L=200) │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 1     │ Insert │ 1M      │ 11.9ms / 17.9ms   │ —                        │ —                  │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 2     │ Search │ —       │ —                 │ 82.8% / 97.0%            │ 460 / 99           │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 3     │ Delete │ 500K    │ 16.3ms / 20.9ms   │ —                        │ —                  │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 4     │ Search │ —       │ —                 │ 84.2% / 97.9%            │ 474 / 97           │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 5     │ Insert │ 500K    │ 15.2ms / 19.6ms   │ —                        │ —                  │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 6     │ Search │ —       │ —                 │ 83.9% / 97.4%            │ 430 / 92           │
└───────┴────────┴─────────┴───────────────────┴──────────────────────────┴────────────────────┘

Spherical 2-bit streaming:

 max_degree=64, L_build=128, alpha=1.2, 4GB bf_tree buffers, 4 build threads, 2 search threads

┌───────┬────────┬─────────┬───────────────────┬──────────────────────────┬────────────────────┐
│ Stage │ Op     │ Vectors │ Avg / p99 Latency │ Recall@10 (L=32 / L=200) │ QPS (L=32 / L=200) │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 1     │ Insert │ 1M      │ 4.8ms / 7.6ms     │ —                        │ —                  │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 2     │ Search │ —       │ —                 │ 74.7% / 94.0%            │ 1167 / 255         │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 3     │ Delete │ 500K    │ 6.9ms / 9.3ms     │ —                        │ —                  │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 4     │ Search │ —       │ —                 │ 73.4% / 94.1%            │ 1103 / 233         │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 5     │ Insert │ 500K    │ 6.0ms / 8.6ms     │ —                        │ —                  │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 6     │ Search │ —       │ —                 │ 75.1% / 94.3%            │ 1053 / 229         │
└───────┴────────┴─────────┴───────────────────┴──────────────────────────┴────────────────────┘

Spherical 4-bit streaming:

 max_degree=64, L_build=128, alpha=1.2, 4GB bf_tree buffers, 4 build threads, 2 search threads

┌───────┬────────┬─────────┬───────────────────┬──────────────────────────┬────────────────────┐
│ Stage │ Op     │ Vectors │ Avg / p99 Latency │ Recall@10 (L=32 / L=200) │ QPS (L=32 / L=200) │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 1     │ Insert │ 1M      │ 4.8ms / 7.7ms     │ —                        │ —                  │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 2     │ Search │ —       │ —                 │ 82.3% / 96.7%            │ 1127 / 253         │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 3     │ Delete │ 500K    │ 6.9ms / 9.5ms     │ —                        │ —                  │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 4     │ Search │ —       │ —                 │ 82.4% / 97.3%            │ 1095 / 231         │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 5     │ Insert │ 500K    │ 6.3ms / 9.1ms     │ —                        │ —                  │
├───────┼────────┼─────────┼───────────────────┼──────────────────────────┼────────────────────┤
│ 6     │ Search │ —       │ —                 │ 83.1% / 97.1%            │ 1019 / 217         │
└───────┴────────┴─────────┴───────────────────┴──────────────────────────┴────────────────────┘

Comparison across bit widths at L=200:

┌─────────────────┬────────┬─────┬────────────────┐
│ Variant         │ Recall │ QPS │ Insert Latency │
├─────────────────┼────────┼─────┼────────────────┤
│ Full Precision  │ 97.0%  │ 99  │ 11.9ms         │
├─────────────────┼────────┼─────┼────────────────┤
│ Spherical 4-bit │ 96.7%  │ 253 │ 4.8ms          │
├─────────────────┼────────┼─────┼────────────────┤
│ Spherical 2-bit │ 94.0%  │ 255 │ 4.8ms          │
└─────────────────┴────────┴─────┴────────────────┘

The configuration used in the above tests was enough to keep all of the data in memory and didn't overflow to disk. I also tested full-precision using 32mb of cb size and experienced a 2x performance degradation due to the disk lookups.

CI Notes

CI reports an increase in IR size of 11%, which is over the 5% allowed in the CI file. This is largely unavoidable given the amount of code added here, so that CI gate should be ignored for this PR.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 92.66667% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.38%. Comparing base (bc106df) to head (bdb1014).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
diskann-bftree/src/provider.rs	87.71%	7 Missing ⚠️
diskann-benchmark/src/inputs/graph_index.rs	82.35%	6 Missing ⚠️
diskann/src/graph/internal/prune.rs	0.00%	4 Missing ⚠️
diskann/src/graph/index.rs	91.89%	3 Missing ⚠️
diskann-benchmark/src/backend/index/benchmarks.rs	94.11%	1 Missing ⚠️
...ann-benchmark/src/backend/index/streaming/stats.rs	96.55%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1106      +/-   ##
==========================================
- Coverage   89.40%   89.38%   -0.03%     
==========================================
  Files         485      485              
  Lines       92079    92199     +120     
==========================================
+ Hits        82323    82408      +85     
- Misses       9756     9791      +35     

Flag	Coverage Δ
miri	89.38% <92.66%> (-0.03%)	⬇️
unittests	89.03% <92.66%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark/src/backend/index/build.rs	85.92% <100.00%> (ø)
diskann-benchmark/src/backend/index/mod.rs	100.00% <100.00%> (ø)
diskann-benchmark/src/backend/index/product.rs	100.00% <ø> (ø)
diskann-benchmark/src/backend/index/scalar.rs	100.00% <ø> (ø)
diskann-benchmark/src/backend/index/spherical.rs	100.00% <ø> (ø)
...mark/src/backend/index/streaming/full_precision.rs	93.54% <100.00%> (+0.07%)	⬆️
...n-benchmark/src/backend/index/streaming/managed.rs	97.56% <100.00%> (+0.09%)	⬆️
diskann-benchmark/src/inputs/mod.rs	81.25% <ø> (ø)
diskann-benchmark/src/main.rs	91.17% <ø> (ø)
diskann-benchmark/src/utils/streaming.rs	91.38% <ø> (ø)
... and 8 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds bf_tree-backed benchmark support to diskann-benchmark (full-precision + spherical quantization; static + streaming variants) and updates diskann-bftree to address inplace-delete behavior and reduce per-call allocations in neighbor serialization.

Changes:

• Add a new bftree input schema + backend benchmarks (static build/search and streaming runbook modes, including spherical quantization bit-width dispatch).
• Add delete-time vector caching in diskann-bftree to support inplace-delete pruning after the underlying storage entries are removed.
• Rework neighbor list serialization to avoid per-call heap allocations (stack buffer fast path).

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
diskann-bftree/src/quant.rs	Makes get_vector_sync available outside tests for delete-time caching.
diskann-bftree/src/provider.rs	Adds delete caches + accessor fallbacks for inplace-delete pruning; exposes clear_delete_caches().
diskann-bftree/src/neighbors.rs	Switches neighbor serialization to a stack buffer fast path.
diskann-benchmark/src/inputs/mod.rs	Centralizes PRINT_WIDTH + write_field! for consistent summaries.
diskann-benchmark/src/inputs/graph_index.rs	Exposes IndexBuild helpers for reuse by bf_tree inputs.
diskann-benchmark/src/inputs/bftree.rs	New bf_tree input schemas (FP + spherical; static + streaming) and display/validation.
diskann-benchmark/src/backend/index/streaming/stats.rs	Adds a bf_tree-only GenericStats::noop() helper for skipped maintenance steps.
diskann-benchmark/src/backend/index/mod.rs	Registers bf_tree benchmarks behind the bftree feature.
diskann-benchmark/src/backend/index/bftree/mod.rs	New bf_tree benchmark module + shared streaming runner.
diskann-benchmark/src/backend/index/bftree/full_precision.rs	Implements bf_tree full-precision static + streaming benchmarks.
diskann-benchmark/src/backend/index/bftree/spherical.rs	Implements bf_tree spherical quantization static + streaming benchmarks.
diskann-benchmark/example/graph-index-bftree.json	Example config for bf_tree full-precision static build/search.
diskann-benchmark/example/graph-index-bftree-stream.json	Example config for bf_tree full-precision streaming runbook.
diskann-benchmark/example/graph-index-bftree-spherical.json	Example config for bf_tree spherical quantization static build/search.
diskann-benchmark/example/graph-index-bftree-spherical-stream.json	Example config for bf_tree spherical quantization streaming runbook.
diskann-benchmark/Cargo.toml	Adds bf_tree dependency + feature flag.
Cargo.toml	Adds diskann-bftree to workspace dependencies.
Cargo.lock	Locks in the new dependency edge.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[JordanMaples]

@copilot re-evaluate please

[Copilot]

@copilot re-evaluate please

Re-evaluated: you’re right, those match result arms compile as written. I verified locally with cargo check -p diskann-bftree and did not find a moved-value compile error there, so no code change is needed for that feedback.

[hildebrandmw]

Thanks Jordan, this is looking good!

[magdalendobson]

I'm not sure I fully understand the bug in deletion. Why do we need to keep the deleted vector around for pruning its neighbors? If it's in someone's neighbor list after deletion, it should be ignored during pruning. Is there some difference in how bf-tree handles this versus how in-mem providers handle this that meant deletes needed to be handled differently?

[magdalendobson]

I think posting some experiments with a more complicated runbook would be a good idea, it looks like all the experimental results here are from simple_runbook.yaml which has just a few steps and might not be sufficient to catch bugs or perf issues. If you like I am happy to provide some suggestions for runbooks.

[magdalendobson]

I left some cosmetic comments but overall the structure and code look good to me. For the streaming I have a couple of concerns:

1. I would like to see experiments with a more complicated runbook to make sure we have caught any perf or correctness issues. I would especially want to make sure we've tested replace adequately.
2. I am not sure I understand why encountering deleted nodes during pruning and throwing a Transient error is a bug. This seems like intended behavior and the transient error can be appropriately ignored. If we instead leave the deleted vectors in the graph until the end of deletion and take them into account during pruning of other nodes, we're just wasting work by computing distances to them anyway. Can you explain the reasoning here if I am wrong?

Thanks!

[JordanMaples]

I'm not sure I fully understand the bug in deletion. Why do we need to keep the deleted vector around for pruning its neighbors? If it's in someone's neighbor list after deletion, it should be ignored during pruning. Is there some difference in how bf-tree handles this versus how in-mem providers handle this that meant deletes needed to be handled differently?

In-mem marks the vectors as deleted but keeps them around to be used during the inplace delete as the query point for finding replacement neighbors. bf_tree's design decision to hard delete would break this flow, which is why storing them temporarily is necessary. The bug I'm addressing was introduced during the bftree migration out of providers and into its own crate and we opted to go for the hard deletes.

[magdalendobson]

Could you add your new json examples as integration tests to https://github.com/microsoft/DiskANN/blob/main/diskann-benchmark/src/main.rs?

Now we have a runbook in our test data directory, we should have a norm that all example jsons should be wired up as integration tests.

[magdalendobson]

Could you add your new json examples as integration tests to https://github.com/microsoft/DiskANN/blob/main/diskann-benchmark/src/main.rs?

Now we have a runbook in our test data directory, we should have a norm that all example jsons should be wired up as integration tests.

And to add on to this, perhaps you could modify the README to mention adding example jsons as integration tests when you mention creating them?

[magdalendobson]

Thank you for bearing with the weirdness of soft deletes and slot recycling, and for dealing with the learning curve of big-ann-benchmarks and runbooks!

Now that the integration tests are present and you validated with a longer runbook, I am happy with this PR.