perf: cache per-cctx raw ocamldep references by (obj_name, kind)#14520
Merged
robinbb merged 1 commit intoJun 16, 2026
Merged
Conversation
This was referenced May 13, 2026
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
83a8648 to
d0903f5
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
ee1f8ce to
58ed81a
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
d0903f5 to
7205127
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
58ed81a to
ec1ccdb
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
7205127 to
e28b521
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
ec1ccdb to
8054767
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
e28b521 to
9b63c9b
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
8054767 to
a4b31bc
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
9b63c9b to
ec28e9e
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
2 times, most recently
from
bccbb29 to
2095cf4
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
ec28e9e to
7d86a46
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
2095cf4 to
5d6842c
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
7d86a46 to
aa40e40
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
5d6842c to
8a4a3d3
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
aa40e40 to
7fb8178
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
8a4a3d3 to
2d1b2ec
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
7fb8178 to
334e2a0
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
2d1b2ec to
b04e30d
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
334e2a0 to
b606c5f
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
b04e30d to
0849010
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
b606c5f to
90239f6
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
0849010 to
45fa463
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
90239f6 to
750995d
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
45fa463 to
419d452
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
750995d to
25b83ba
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
419d452 to
c9c3957
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
25b83ba to
fcf767c
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
c9c3957 to
b076dc4
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
from
fcf767c to
c93dbda
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
b076dc4 to
a525d86
Compare
robinbb
force-pushed
the
robinbb-14492-l7-filtered-includes-cache
branch
3 times, most recently
from
b2db8e6 to
0d280ae
Compare
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
a525d86 to
5f53a4d
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces a per-Compilation_context cache for “raw” ocamldep module-reference reads (as Action_builder.ts), keyed by (obj_name, kind), so multiple consumers within the same cctx can share the same builders when they traverse overlapping transitive dependencies. This is a performance-only layer in the per-module inter-library dependency filtering stack (#14492 / #4572).
Changes:
- Add
Compilation_context.Raw_refs.Keyand acached_raw_refsAPI to memoize raw-refAction_builder.ts within a cctx. - Store a
raw_refstable insideCompilation_context.tand implement the cache lookup/insert logic. - Update
Module_compilation.lib_deps_for_moduleto wrap rawocamldepreads withCompilation_context.cached_raw_refsusingConsumervsTransitivekeying.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
src/dune_rules/module_compilation.ml |
Reuses cached raw-ref builders when reading per-module/transitive raw ocamldep deps. |
src/dune_rules/compilation_context.mli |
Exposes Raw_refs.Key and the cached_raw_refs cache API. |
src/dune_rules/compilation_context.ml |
Implements the Raw_refs key/table and adds the per-cctx cache storage + accessor. |
robinbb
added a commit
that referenced
this pull request
Jun 13, 2026
Layer 7 of 9 of #14492. Pure performance. `Compilation_context.Filtered_includes` caches the `Action_builder.t` returned by `filtered_include_flags` keyed on `(lib_mode, kept_libs)`. Two modules in the same cctx that reach the same set of kept libs share one builder; `Action_builder.memoize` dedupes its evaluation. `Lib_mode.hash`: new — used by the cache key. Stack: rebases on #14518. Next: #14520. Part of #14492. Related to #4572. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Base automatically changed from
robinbb-14492-l7-filtered-includes-cache
to
robinbb-14492-test-stack-integration
June 13, 2026 21:21
[Compilation_context.Raw_refs] caches the [Action_builder.t] computed for each ocamldep raw-deps read inside a cctx. Two consumer modules that share trans_deps (or a consumer and one of its trans deps that share an [obj_name + ml_kind]) get the same builder. The cache short-circuits before constructing the builder; on hit, no allocation. [Raw_refs.Key] distinguishes the two read patterns the per-module filter uses: [Consumer] (the cctx-driving module's own deps, keyed by [ml_kind]) and [Transitive] (a dep module's deps, keyed by [cm_kind] because the impl/intf gating in [need_impl_deps_of] varies by cm_kind on the [Cmx]/opaque path). Conservatively-distinct keying — never collapse two semantically-different reads under one cache cell. [Compilation_context.cached_raw_refs t ~key ~compute] is the thin public surface: lookup, compute on miss, store, return the builder. [Module_compilation.lib_deps_for_module]: wraps the inline [read_dep_m_raw] body that the BFS uses for both the consumer's own and each trans dep's raw refs. No semantic change — the cache only deduplicates builder construction across calls within the same cctx. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
force-pushed
the
robinbb-14492-l8-raw-refs-cache
branch
from
5f53a4d to
ba1e3b1
Compare
Alizter
approved these changes
Jun 15, 2026
robinbb
merged commit 275751b
into
robinbb-14492-test-stack-integration
30 checks passed
robinbb
added a commit
that referenced
this pull request
Jun 16, 2026
Layer 9 of 9 of #14492. Performance + algorithm reference doc. `Lib.closure` is now defined as `Memo.exec` over a `Memo.create` keyed on `(linking, for_, libs)`. The per-module filter calls `Lib.closure` twice per consumer module (once for `direct_libs`, once for `must_glob_libs`); without memoisation every call re-traverses the dependency graph. The list-of-libs key is order- and multiplicity-sensitive — callers that share inputs (e.g. `lib_deps_for_module` at both call sites) need to canonicalise via `List.sort_uniq ~compare:Lib.compare`; the existing call sites already do. Adds `doc/dev/per-module-narrowing.md` (626 lines): a reference for the narrowing algorithm — frontier construction, BFS expansion through dep-lib references, classification into tight vs glob, soundness recovery (wrapped-closure, ppx-runtime, virtual-impl, stanza-`(flags -open)`), and the filtered-include-flag emit path. Intended as the implementer's first stop when revisiting `lib_deps_for_module` or `filtered_include_flags`. Stack: rebases on #14520. Final layer of the stack. Part of #14492. Related to #4572. --------- Signed-off-by: Robin Bate Boerop <me@robinbb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer 8 of 9 of #14492. Pure performance.
Compilation_context.Raw_refscaches theAction_builder.tcomputed for each ocamldep raw-deps read inside a cctx. Two consumer modules that share trans deps get the same builder.Raw_refs.Keydistinguishes the two read patterns the per-module filter uses:Consumer(the cctx-driving module's own deps, keyed byml_kind) andTransitive(a dep module's deps, keyed bycm_kindbecause the impl/intf gating inneed_impl_deps_ofvaries by cm_kind on theCmx/ opaque path).Module_compilation.lib_deps_for_module: wraps the inlineread_dep_m_rawbody withcached_raw_refs.Stack: rebases on #14519. Next: #14521.
Part of #14492. Related to #4572.