[REVIEW] Add OpenSearch backend to ``cuvs-bench`` by jrbourbeau · Pull Request #2012 · rapidsai/cuvs

jrbourbeau · 2026-04-10T18:30:27Z

Would still like to refine a bit but pushing up for early feedback

cc @singhmanas1 @janakivamaraju @afourniernv

xref #1907 which does something similar but for Elastic

EDIT: Okay, this is now ready for review (cc @jnke2016)

Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

jrbourbeau · 2026-04-10T18:59:30Z

-                    print(
-                        f"Build failed for {config.index_name}: {build_result.error_message}"
+            try:
+                backend.initialize()


Note to reviewers: This is a separate, but related change to the orchestrator. I noticed the initialize and cleanup methods weren't actually being called like their docstrings said they were suppose to. This is a big diff but most of it is just intending over the existing code into a try/except block. The functional change is calling initialize + cleanup

EDIT: FWIW supporting a context manager for backend setup / teardown seems reasonable. But that's also not something I wanted to handle in this PR.

Good catch. The initialize() and cleanup() hooks in BenchmarkBackend are defined in the abstract base class with docstrings describing their intended purpose, but neither method is actually called by the BenchmarkOrchestrator in _run_sweep() or _run_trial(). This was an oversight because the only backend implemented at the time was CppGoogleBenchmarkBackend, whose lifecycle is entirely self-contained within each subprocess call to build()/search(), making initialize()/cleanup() unnecessary for it.

EDIT: FWIW supporting a context manager for backend setup / teardown seems reasonable. But that's also not something I wanted to handle in this PR.

I Agree that context manager support is reasonable. The purpose of the pluggable backend API is precisely to enable these kinds of extensions without modifying the core orchestrator. Since initialize() and cleanup() are already part of the BenchmarkBackend interface, wiring them up in the orchestrator (whether as explicit calls or a context manager) would ensure the plugin system works end-to-end for future backends without requiring further modifications to the core

Overall, I think we're leaning a little too strongly on the local (individual backend) vs global (benchmark orchestrator) control and we should probably prioritize global control for all boilerplate things so that we're not having to reimplement them in each backend. I'm okay doing this in follow-ups, but the way we typically handle these types of things is to create a github issue that clearly explains the intended changes for the follow-up (explain it in a way that assumes someone else might have to pick it up in the future, just in case) and then reference a link to the issue in a todo comment in the code where the change is intended to be made. This makes sure 1) people reading the code are aware of the intentions, and 2) the issue doesn't get orphaned and forgotten.

jrbourbeau · 2026-04-10T18:59:59Z


+[tool.pytest.ini_options]
+markers = [
+    "integration: tests that require a live OpenSearch node (run with '-m integration')",


TODO: Make this more generic language (not just opensearch)

Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

…ch-opensearch Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

…ch-opensearch Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

jnke2016

The PR looks good me.

cjnolet · 2026-04-20T18:52:56Z

+
+    ext = os.path.splitext(path)[1].lower()
+    dtype = _DTYPE_FOR_EXT.get(ext, np.float32)
+    with open(path, "rb") as f:


I mentioned this in the elasticsearch PR as well- we should standardize/centralize these calls so that we're not having to manually implement them for every backend. The benchmark should be opening the files and passing the vectors or the file handle) down to the actual backends.

@jnke2016

Don't disagree with this comment (or the other ones about consolidating various utilities). IIUC it sounds like these will need deeper changes to the config loader / orchestrator specs. Is that something that's needed in this PR, or is a follow up to refactor fine?

Since both backends(elastic search and OpenSearch) are separately implementing this, a follow up PR consolidating both might be easier especially since both PRs are still open. Otherwise I can push a PR consolidating those various utilities (general purpose) based on both PRs which they can pull after it is merged

cjnolet · 2026-04-20T18:53:23Z

+    return data.reshape(n_rows, n_cols)
+
+
+def _load_yaml(path: str) -> Any:


This and the dataset config should also be getting loaded by the benchmark itself and not the backend. @jnke2016

cjnolet · 2026-04-20T18:53:40Z

+
+
+def _gather_algo_configs(
+    config_path: str, algorithm_configuration: Optional[str]


We need to centralize this @jnke2016

cjnolet · 2026-04-20T18:55:50Z

+
+        benchmark_configs: List[BenchmarkConfig] = []
+
+        for algo_file in algo_files:


Yeah, this is all boilerplate- exactly why this should be centralized. The resulting config dicts should be loaded from the files by the benchmark and passed down to the individual backends. It's important that this is standardized across benchmarks so that we don't have specific backends doing things drastically different than others (that just makes it super complicated for everyone. But also, if we find a bug in a spcific loader or decide to change how we do things generally, every single backend then needs to be updated individually instead of just changing it one place ) @jnke2016

So the loading pipeline (loading dataset YAML, gathering algorithm files, filtering by algorithm/group names etc ..) is a shared logic that should be centralized in the base ConfigLoader instead of having each backend implementing this.

cjnolet · 2026-04-20T18:59:24Z

+                method_params = {"nlist": build_param.get("nlist", 4)}
+            else:
+                raise ValueError(
+                    f"Unsupported faiss_method {faiss_method!r}. "


Yes! This is important. I believe i suggested that this be done in the eladticserch PR as well. Need to fail gracefully when the user configures somethign that's not supported (or loads the wrong file).

cjnolet · 2026-04-20T19:03:38Z

cc @navneet1v for thoughts / feedback on the opensearch backend for cuvs-bench.

Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

…ch-opensearch

Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

Add OpenSearch backend to cuvs-bench

b30b9e1

Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

jrbourbeau requested review from a team as code owners April 10, 2026 18:30

jrbourbeau requested a review from bdice April 10, 2026 18:30

github-project-automation Bot added this to Unstructured Data Processing Apr 10, 2026

jrbourbeau commented Apr 10, 2026

View reviewed changes

jrbourbeau added 2 commits April 10, 2026 14:13

Lint

a58363f

Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

Merge branch 'main' of https://github.com/rapidsai/cuvs into cuvs-ben…

734a83c

…ch-opensearch Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

jrbourbeau mentioned this pull request Apr 10, 2026

[DNM] Add OpenSearch deployment #2013

Draft

jrbourbeau added 2 commits April 13, 2026 11:43

Updates

ed7c1d0

Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

Merge branch 'main' of https://github.com/rapidsai/cuvs into cuvs-ben…

7131d39

…ch-opensearch Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

jrbourbeau changed the title ~~[WIP] Add OpenSearch backend to cuvs-bench~~ [REVIEW] Add OpenSearch backend to cuvs-bench Apr 13, 2026

jnke2016 self-requested a review April 20, 2026 18:43

jnke2016 approved these changes Apr 20, 2026

View reviewed changes

cjnolet reviewed Apr 20, 2026

View reviewed changes

cjnolet assigned jrbourbeau Apr 20, 2026

cjnolet added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Apr 20, 2026

cjnolet moved this to In Progress in Unstructured Data Processing Apr 20, 2026

jrbourbeau added 3 commits April 20, 2026 14:40

Updates

964a0df

Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

Merge branch 'main' of https://github.com/rapidsai/cuvs into cuvs-ben…

288d308

…ch-opensearch

Lint

887b2a3

Signed-off-by: James Bourbeau <jbourbeau@nvidia.com>

		return data.reshape(n_rows, n_cols)


		def _load_yaml(path: str) -> Any:



		def _gather_algo_configs(
		config_path: str, algorithm_configuration: Optional[str]


		benchmark_configs: List[BenchmarkConfig] = []

		for algo_file in algo_files:

Conversation

jrbourbeau commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jrbourbeau Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnke2016 Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

jnke2016 Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjnolet Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

jrbourbeau Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

jnke2016 left a comment

Choose a reason for hiding this comment

Uh oh!

cjnolet Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrbourbeau Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

jnke2016 Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjnolet Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjnolet Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

jnke2016 Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

cjnolet Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjnolet commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jrbourbeau commented Apr 10, 2026 •

edited

Loading

jrbourbeau Apr 10, 2026 •

edited

Loading

jnke2016 Apr 13, 2026 •

edited

Loading

cjnolet Apr 20, 2026 •

edited

Loading

jnke2016 Apr 20, 2026 •

edited

Loading

cjnolet Apr 20, 2026 •

edited

Loading

cjnolet Apr 20, 2026 •

edited

Loading