feat(eml-hnsw): v2 integrated pipeline — retention selector + SIMD rerank + PQ + progressive cascade (supersedes #353)#356
Open
feat(eml-hnsw): v2 integrated pipeline — retention selector + SIMD rerank + PQ + progressive cascade (supersedes #353)#356
Conversation
The execute_match() function previously collapsed all match results into a single ExecutionContext via context.bind(), which overwrote previous bindings. MATCH (n:Person) on 3 Person nodes returned only 1 row. This commit refactors the executor to use a ResultSet pipeline: - type ResultSet = Vec<ExecutionContext> - Each clause transforms ResultSet → ResultSet - execute_match() expands the set (one context per match) - execute_return() projects one row per context - execute_set/delete() apply to all contexts - Cross-product semantics for multiple patterns in one MATCH Also adds comprehensive tests: - test_match_returns_multiple_rows (the Issue #269 regression) - test_match_return_properties (verify correct values per row) - test_match_where_filter (WHERE correctly filters multi-row) - test_match_single_result (1 match → 1 row, no regression) - test_match_no_results (0 matches → 0 rows) - test_match_many_nodes (100 nodes → 100 rows, stress test) Co-Authored-By: claude-flow <ruv@ruv.net>
RETURN n.name now produces column "n.name" instead of "?column?". Property expressions (Expression::Property) are formatted as "object.property" for column naming, matching standard Cypher behavior. Co-Authored-By: claude-flow <ruv@ruv.net>
Built from commit b2347ce Platforms updated: - linux-x64-gnu - linux-arm64-gnu - darwin-x64 - darwin-arm64 - win32-x64-msvc 🤖 Generated by GitHub Actions
Built from commit 2adb949 Platforms updated: - linux-x64-gnu - linux-arm64-gnu - darwin-x64 - darwin-arm64 - win32-x64-msvc 🤖 Generated by GitHub Actions
Phase 2 of the ruvector remediation plan. Replaces simulated benchmarks with real measurements: - Python harness: hnswlib (C++) and numpy brute-force on same datasets - Rust test: ruvector-core HNSW with ground-truth recall measurement - Datasets: random-10K and random-100K, 128 dimensions - Metrics: QPS (p50/p95), recall@10 vs ground truth, memory, build time Key findings: - ruvector recall@10 is good: 98.3% (10K), 86.75% (100K) - ruvector QPS is 2.6-2.9x slower than hnswlib - ruvector build time is 2.2-5.9x slower than hnswlib - ruvector uses ~523MB for 100K vectors (10x raw data size) - All numbers are REAL — no hardcoded values, no simulation Co-Authored-By: claude-flow <ruv@ruv.net>
Built from commit 3b173a9 Platforms updated: - linux-x64-gnu - linux-arm64-gnu - darwin-x64 - darwin-arm64 - win32-x64-msvc 🤖 Generated by GitHub Actions
New crate: ruvector-eml-hnsw (6 modules, 93 tests) Patch: hnsw_rs/src/eml_distance.rs (integrated implementations) 1. Cosine Decomposition (EmlDistanceModel) — 10-30x distance speed Learns which dimensions discriminate, reduces O(384) to O(k) 2. Progressive Dimensionality (ProgressiveDistance) — 5-20x search Layer 2: 8-dim, Layer 1: 32-dim, Layer 0: full-dim 3. Adaptive ef (AdaptiveEfModel) — 1.5-3x search speed Per-query beam width from (norm, variance, graph_size, max_component) 4. Search Path Prediction (SearchPathPredictor) — 2-5x search K-means query regions → cached entry points, skip top-layer traversal 5. Rebuild Cost Prediction (RebuildPredictor) — operational efficiency Predicts recall degradation, triggers rebuild only when needed 6. PQ Distance Correction (PqDistanceCorrector) — DiskANN recall Learns PQ approximation error correction from exact/PQ pairs All backward compatible — untrained models fall back to standard behavior. Based on: Odrzywolel 2026, arXiv:2603.21852v2 Co-Authored-By: claude-flow <ruv@ruv.net>
Stage 1: micro-benchmarks (cosine decomp, adaptive ef, path prediction, rebuild prediction) — raw 16d L2 proxy is 9.3x faster than full 128d cosine, but EML model overhead makes fast_distance 2.1x slower. Stage 2: synthetic e2e (10K x 128d) — recall@10 drops to 0.1% on uniform random data because all dimensions are equally important. EML decomposition needs structured embeddings to work. Stage 3: real dataset — deferred, SIFT1M not available. Infrastructure in place to auto-run when dataset is downloaded. Stage 4: hypothesis test — DISPROVEN on random data (Spearman rho=0.013 vs required 0.95). Expected: uniform random has no discriminative dimensions. Real embeddings with PCA structure should score higher. Honest results: dimension reduction mechanism works, but EML model inference overhead and random-data limitations are documented clearly. Following shaal's methodology from PR #352. Co-Authored-By: claude-flow <ruv@ruv.net>
PR #353 added 6 standalone learned models but no consumer, so the selected-dims approach never reached any index. This commit closes that gap: - selected_distance.rs: plain cosine over learned dim subset (the corrected runtime path; the original fast_distance evaluated the EML tree per call and was 2.1x SLOWER than baseline, confirmed on ruvultra AMD 9950X). - hnsw_integration.rs: EmlHnsw wraps hnsw_rs::Hnsw, projects vectors to the learned subspace on add/search, keeps full-dim store for optional rerank. - tests/recall_integration.rs: end-to-end synthetic validation (rerank recall@10 >= 0.83 on structured data). - tests/sift1m_real.rs: Stage-3 gated real-data harness. Test counts: 70 unit + 3 recall_integration + 1 SIFT1M gated + 3 doctests (vs PR #353 body claim of 93 unit tests; actual on pr-353 pre-fix was 60). Stage-3 SIFT1M measured (50k base x 200 queries x 128d, selected_k=32, AMD 9950X): recall@10 reduced = 0.194 (PR #353 author expected ~0.85-0.95) recall@10 +rerank = 0.438 (fetch_k=50 too tight on real data) reduced HNSW p50 = 268.9 us reduced HNSW p95 = 361.8 us Finding: the mechanism is viable as a candidate pre-filter but requires (a) larger fetch_k (200-500), (b) SIMD-accelerated rerank (per PR #352), and (c) training on many more than 500-1000 samples for real embeddings. The synthetic ρ=0.958 claim does NOT reproduce on SIFT1M.
…rank + PQ + progressive cascade Supersedes the original PR #353 contribution with the combined result of six targeted experiments run on ruvultra (AMD Ryzen 9 9950X / 32T / 123 GB) against real SIFT1M (50k base × 200 queries). Integration gap is closed — this crate now has actual consumers (EmlHnsw, ProgressiveEmlHnsw, PqEmlHnsw), each with a real hnsw_rs-backed search path + rerank. ## Landing 1. EmlHnsw wrapper (base, from fix/eml-hnsw-integration) - Projects vectors to the learned subspace on insert/search, keeps full-dim store for rerank, exposes search_with_rerank(query, k, fetch_k, ef). - Fixes the fundamental "no consumer" problem in PR #353's original crate. 2. Tier 1B — SimSIMD rerank kernel - cosine_distance_simd backed by simsimd::SpatialSimilarity - 5.65× speedup at d=128 (59.1 ns → 10.5 ns), 6.22× at d=384 - Recall unchanged (Δ = 0.002, f32-vs-f64 accumulation noise) - Benchmark: benches/rerank_kernel.rs 3. Tier 1C — retention-objective selector - EmlDistanceModel::train_for_retention: greedy forward selection that maximizes recall@target_k on held-out queries - SIFT1M result at selected_k=32, fetch_k=200: pearson selector: recall@10 = 0.712 retention selector: recall@10 = 0.817 (+0.105, >3σ at n=200) - Training 37× slower but offline/one-shot 4. Tier 3A — ProgressiveEmlHnsw [8, 32, 128] cascade - Multi-index coarsest→finest, union + exact cosine rerank - SIFT1M: recall@10 = 0.984 at 961 µs p50 vs single-index 0.974 at ~1950 µs (2.0× latency improvement at matched recall) - Build cost 5.9× baseline — read-heavy workloads only 5. Tier 3B — PqEmlHnsw (8 subspaces × 256 centroids) + corrector - 64× memory reduction (512 B → 8 B per vector) - SIFT1M: rerank@10 = 0.9515, clears the ≥0.80 tier target - k-means converged cleanly (10-19 iterations per subspace, 25-iter cap never bound) - PqDistanceCorrector kept advisory-only: normalization against global max_pq_dist saturates on SIFT's O(10⁵) distance scale (MSE 1.4e9 → 6.4e10). Does not hurt recall because final rank is exact cosine. ## Measured evidence (all on ruvultra) See docs/adr/ADR-151-eml-hnsw-selected-dims.md for full context, acceptance criteria, and per-tier commit SHAs. Per-PR measured numbers are in GitHub issue #351 and PR #353 discussion. ## NOT included from PR #353 - EmlDistanceModel::fast_distance (EML tree per call): 2.35× SLOWER than scalar baseline on ruvultra. Kept as reference impl; not on any search path. See ADR-151 §Rejected Surface. - AdaptiveEfModel: 290 ns/query actual vs 3 ns claimed. Rejected until a <20 ns predictor is demonstrated. - Sliced Wasserstein rerank (Tier 2 experiment): 50.9× slower AND 38.1 pp worse than cosine rerank on SIFT. Cleanly falsified for gradient- histogram datasets. Documented in ADR-151 closed open-questions. ## Surface area - Default RuVector retrieval paths unchanged. - HnswIndex::new() and DbOptions::default() untouched. - EmlHnsw / ProgressiveEmlHnsw / PqEmlHnsw are explicitly constructed by callers opting into the approximate-then-exact pipeline. Co-Authored-By: swarm-coder <swarm@ruv.net> Co-Authored-By: Mathew Beane (aepod) <124563+aepod@users.noreply.github.com> Co-Authored-By: Ofer Shaal (shaal) <22901+shaal@users.noreply.github.com>
ruvnet
force-pushed
the
feat/eml-hnsw-optimizations-v2
branch
from
0ade479 to
db1c58b
Compare
This was referenced Apr 16, 2026
…ence Primary artifact for PR #356. Documents: - PR #353 claims vs measured reality on ruvultra (AMD 9950X) - v2 accepted surface (EmlHnsw, ProgressiveEmlHnsw, PqEmlHnsw, retention selector, SimSIMD rerank) - Rejected surface (fast_distance, AdaptiveEfModel, Sliced Wasserstein) - 6-tier swarm results: 4 passes, 1 clean falsification - SOTA v3 scope: 4-agent swarm in progress - Open questions with current status Co-Authored-By: Mathew Beane (aepod) <124563+aepod@users.noreply.github.com> Co-Authored-By: Ofer Shaal (shaal) <22901+shaal@users.noreply.github.com>
Owner
Author
v3 update (branch feat/eml-hnsw-optimizations-v3)Merge of four SOTA tiers on top of v2 ( Tier landing
Retention selector A/B (SIFT1M, selected_k=32)
Greedy wins: +10.4 pp over pearson. Beam gain is inside noise. Honest reframePlain What v3 does deliver:
Clean falsifications (kept in repo)
Files
Readinessv3 is ready for review. All 93 lib + 4 new core integration tests green on merged branch. Recommend reading ADR-151 §v3 SOTA Evidence first — it carries the honest-reframe framing this comment summarizes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Credit
This work builds directly on two outstanding upstream contributions:
EmlDistanceModel,ProgressiveDistance,AdaptiveEfModel,SearchPathPredictor,RebuildPredictor,PqDistanceCorrector), the gradient-freeeml-coretraining library, and the 4-stage proof chain methodology. Without @aepod's Stage 4 hypothesis ("EML is the teacher, not the runtime — use plain cosine on selected dims") this v2 would not exist. The architectural pivot described in his own PR #353 comment thread is exactly what this branch ships as callable code.UnifiedDistanceParamskernel, the four-stage proof methodology (adopted verbatim here), and the honest SIFT1M+GloVe measurement discipline all originated in his work. Tier 1B of this branch is a direct port of his SIMD cosine approach into the reduced-dim rerank stage.Both authors are credited as
Co-Authored-By:on the merged commit, and every piece of measured evidence below is traceable to one or both of their PRs.Supersedes #353
Rewrites the EML-HNSW contribution into a working integrated pipeline with measured SIFT1M numbers. The original PR shipped six standalone learned models but had no downstream consumer — the
ruvector-eml-hnswcrate compiled but its code never reached any RuVector HNSW path. This branch closes that gap and folds in the winning results from a six-experiment swarm run on ruvultra (AMD Ryzen 9 9950X / 32T / 123 GB) against real SIFT1M.What's in v2
EmlHnswwrapper aroundhnsw_rs::Hnsw+search_with_rerankcosine_distance_simd), after @shaal's PR #352 kernelEmlDistanceModel::train_for_retention— greedy forward selectionProgressiveEmlHnsw[8, 32, 128]multi-level cascade, using @aepod'sProgressiveDistancePqEmlHnsw8×256 Product Quantizer paired with @aepod'sPqDistanceCorrectorWhat's NOT in v2 (and why)
EmlDistanceModel::fast_distance(EML tree per call): measured 2.35× slower than scalar baseline. Kept as reference impl; not on any query-time path. This matches @aepod's own Stage-1 finding on his test hardware.AdaptiveEfModel: 290 ns/query actual overhead vs 3 ns claimed — too expensive to amortize against the ef-search work it would save.PqDistanceCorrectoris kept but held advisory-only: under training on SIFT1M it increased MSE (1.4e9 → 6.4e10) because feature normalization against a globalmax_pq_distsaturates on SIFT's O(10⁵) distance scale. Final rank is exact cosine so this does not hurt recall. Noted in ADR-151 as a design flaw with a proposed fix direction (per-vector exact normalization).Test surface
92 tests pass on the merged branch:
selected_distance,pq,pq_hnsw,progressive_hnsw,hnsw_integration; retained: all originalruvector-eml-hnswtests from @aepod's PR feat: EML-enhanced HNSW — 6 learned optimizations (10-30x distance, 2-5x search) #353)recall_integration)sift1m_real,retention_vs_pearson,progressive_sift1m,sift1m_pqbenches/rerank_kernel.rs)Reproducibility recipe (on any Linux box with rustc ≥ 1.80):
Coupling with #352
@shaal's PR #352 (unified SIMD kernel +
QuantizationConfig::Log) is strictly additive over this branch. Landing both captures the full effect: #352 accelerates the inner distance kernel, this branch adds the pre-filter stage that makes widefetch_kviable. See issue #351 for the cross-PR measurements.Surface area and compatibility
DbOptions::default()behavior unchanged.HnswIndex::new(...)and all existing RuVector retrieval paths unchanged.EmlHnsw/ProgressiveEmlHnsw/PqEmlHnsware explicitly constructed by callers opting into the approximate-then-exact pipeline.References
docs/adr/ADR-151-eml-hnsw-selected-dims.md) — acceptance matrix, per-tier measured numbers, closed/open questions.Closes #353 on merge. Cc @aepod @shaal for review — your work drove every measured result in this PR.