perf: memoise [Lib.closure] keyed on (linking, for_, libs)#14521
Merged
robinbb merged 4 commits intoJun 16, 2026
Merged
Conversation
This was referenced May 13, 2026
ee1f8ce to
58ed81a
Compare
215bfef to
bf10fd7
Compare
58ed81a to
ec1ccdb
Compare
bf10fd7 to
6470974
Compare
ec1ccdb to
8054767
Compare
4f3d261 to
8ed3374
Compare
robinbb
added a commit
that referenced
this pull request
May 16, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
added a commit
that referenced
this pull request
May 16, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in this PR (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
added a commit
that referenced
this pull request
May 18, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
8ed3374 to
356728c
Compare
robinbb
added a commit
that referenced
this pull request
May 20, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
added a commit
that referenced
this pull request
May 20, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
added a commit
that referenced
this pull request
May 21, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
8054767 to
a4b31bc
Compare
robinbb
added a commit
that referenced
this pull request
May 21, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
356728c to
e7aa8f5
Compare
robinbb
added a commit
that referenced
this pull request
May 22, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
a4b31bc to
bccbb29
Compare
e7aa8f5 to
a38b747
Compare
robinbb
added a commit
that referenced
this pull request
May 22, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
added a commit
that referenced
this pull request
May 22, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
bccbb29 to
2095cf4
Compare
a38b747 to
1ca1078
Compare
robinbb
added a commit
that referenced
this pull request
May 22, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
2095cf4 to
5d6842c
Compare
0849010 to
45fa463
Compare
0e29736 to
405579f
Compare
robinbb
added a commit
that referenced
this pull request
Jun 8, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
45fa463 to
419d452
Compare
robinbb
added a commit
that referenced
this pull request
Jun 8, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
405579f to
75bb1e9
Compare
robinbb
added a commit
that referenced
this pull request
Jun 8, 2026
Lifts the [can_filter] Melange opt-out installed by #14516 (L4). The BFS narrowing pipeline now activates for Melange consumer compiles on the same terms as OCaml — same [Lib_index], same wrapped-lib soundness recovery, same [must_glob_set] / [tight_set] split. L4 routed Melange through [deps_of_entries] (broad) on the rationale that the narrowing had been developed and tested with OCaml-mode in mind. The defensive gate has no remaining technical justification: the cross-library walk reads [ocamldep] output (mode-agnostic), the classification pipeline is keyed on [Lib_index] entries (already populated for libs compiled in either mode), and the only adjustment needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to round out per-module deps for [Melange Cmj] consumers. [Module_compilation.lib_deps_for_module]: drop the [match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard collapses to the four mode-independent predicates ([dep_graph] is real, the consumer module is in it, the module kind is filterable, the cctx is neither virtual nor a parameter). [Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange] disjunct from the singleton-stanza short-circuit. Single-module Melange stanzas with library deps now produce a real dep graph (the per-module filter consumes it). Comment reworded to drop the stale "Melange [can_filter]" forward-reference. [Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors [groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by the broad-dep path. The mli prose is rewritten — the "future Melange caller" caveat is gone, replaced with a positive description of all four [cm_kind] cases. [doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass narrowing" statements; the [can_filter] predicate list no longer includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to describe Melange's [want_cmj] participation. Soundness: the only edge surfaced by this layer is the alias-form issue already documented by [wrapped-internal-leak.t] on L4 — a consumer referencing [Foo__Bar] directly via the wrapped-lib mangled-form alias. GitHub code search on 2026-05-25 found zero matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__] outside Melange's own test suite; the pattern is essentially absent in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml] is the one known caller and is fixed in melange-re/melange to route through the supported [Melange_ppx.String_interp] wrapper. Tests: six existing Melange cram tests are promoted for output drift under the new path: - [depend-on-installed.t], [emit-installed.t], [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep (internal)] line in [--display=short] trace. - [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder of [ocamlc helper] vs [melc foo] in the build trace. No artefact change. - [missing-melc.t]: duplicated [melc not found] error block — the narrowing now triggers the consumer's per-module compile, which fails with the same melc-missing error. Exit code unchanged. Stack: rebases on #14521 (L9). Part of #14492. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
added a commit
that referenced
this pull request
Jun 8, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
419d452 to
c9c3957
Compare
robinbb
added a commit
that referenced
this pull request
Jun 8, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
75bb1e9 to
15a0200
Compare
robinbb
added a commit
that referenced
this pull request
Jun 8, 2026
Lifts the [can_filter] Melange opt-out installed by #14516 (L4). The BFS narrowing pipeline now activates for Melange consumer compiles on the same terms as OCaml — same [Lib_index], same wrapped-lib soundness recovery, same [must_glob_set] / [tight_set] split. L4 routed Melange through [deps_of_entries] (broad) on the rationale that the narrowing had been developed and tested with OCaml-mode in mind. The defensive gate has no remaining technical justification: the cross-library walk reads [ocamldep] output (mode-agnostic), the classification pipeline is keyed on [Lib_index] entries (already populated for libs compiled in either mode), and the only adjustment needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to round out per-module deps for [Melange Cmj] consumers. [Module_compilation.lib_deps_for_module]: drop the [match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard collapses to the four mode-independent predicates ([dep_graph] is real, the consumer module is in it, the module kind is filterable, the cctx is neither virtual nor a parameter). [Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange] disjunct from the singleton-stanza short-circuit. Single-module Melange stanzas with library deps now produce a real dep graph (the per-module filter consumes it). Comment reworded to drop the stale "Melange [can_filter]" forward-reference. [Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors [groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by the broad-dep path. The mli prose is rewritten — the "future Melange caller" caveat is gone, replaced with a positive description of all four [cm_kind] cases. [doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass narrowing" statements; the [can_filter] predicate list no longer includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to describe Melange's [want_cmj] participation. Soundness: the only edge surfaced by this layer is the alias-form issue already documented by [wrapped-internal-leak.t] on L4 — a consumer referencing [Foo__Bar] directly via the wrapped-lib mangled-form alias. GitHub code search on 2026-05-25 found zero matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__] outside Melange's own test suite; the pattern is essentially absent in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml] is the one known caller and is fixed in melange-re/melange to route through the supported [Melange_ppx.String_interp] wrapper. Tests: six existing Melange cram tests are promoted for output drift under the new path: - [depend-on-installed.t], [emit-installed.t], [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep (internal)] line in [--display=short] trace. - [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder of [ocamlc helper] vs [melc foo] in the build trace. No artefact change. - [missing-melc.t]: duplicated [melc not found] error block — the narrowing now triggers the consumer's per-module compile, which fails with the same melc-missing error. Exit code unchanged. Stack: rebases on #14521 (L9). Part of #14492. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
c9c3957 to
b076dc4
Compare
robinbb
added a commit
that referenced
this pull request
Jun 8, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
15a0200 to
1e32a97
Compare
robinbb
added a commit
that referenced
this pull request
Jun 8, 2026
Lifts the [can_filter] Melange opt-out installed by #14516 (L4). The BFS narrowing pipeline now activates for Melange consumer compiles on the same terms as OCaml — same [Lib_index], same wrapped-lib soundness recovery, same [must_glob_set] / [tight_set] split. L4 routed Melange through [deps_of_entries] (broad) on the rationale that the narrowing had been developed and tested with OCaml-mode in mind. The defensive gate has no remaining technical justification: the cross-library walk reads [ocamldep] output (mode-agnostic), the classification pipeline is keyed on [Lib_index] entries (already populated for libs compiled in either mode), and the only adjustment needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to round out per-module deps for [Melange Cmj] consumers. [Module_compilation.lib_deps_for_module]: drop the [match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard collapses to the four mode-independent predicates ([dep_graph] is real, the consumer module is in it, the module kind is filterable, the cctx is neither virtual nor a parameter). [Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange] disjunct from the singleton-stanza short-circuit. Single-module Melange stanzas with library deps now produce a real dep graph (the per-module filter consumes it). Comment reworded to drop the stale "Melange [can_filter]" forward-reference. [Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors [groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by the broad-dep path. The mli prose is rewritten — the "future Melange caller" caveat is gone, replaced with a positive description of all four [cm_kind] cases. [doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass narrowing" statements; the [can_filter] predicate list no longer includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to describe Melange's [want_cmj] participation. Soundness: the only edge surfaced by this layer is the alias-form issue already documented by [wrapped-internal-leak.t] on L4 — a consumer referencing [Foo__Bar] directly via the wrapped-lib mangled-form alias. GitHub code search on 2026-05-25 found zero matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__] outside Melange's own test suite; the pattern is essentially absent in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml] is the one known caller and is fixed in melange-re/melange to route through the supported [Melange_ppx.String_interp] wrapper. Tests: six existing Melange cram tests are promoted for output drift under the new path: - [depend-on-installed.t], [emit-installed.t], [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep (internal)] line in [--display=short] trace. - [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder of [ocamlc helper] vs [melc foo] in the build trace. No artefact change. - [missing-melc.t]: duplicated [melc not found] error block — the narrowing now triggers the consumer's per-module compile, which fails with the same melc-missing error. Exit code unchanged. Stack: rebases on #14521 (L9). Part of #14492. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
b076dc4 to
a525d86
Compare
robinbb
added a commit
that referenced
this pull request
Jun 9, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
1e32a97 to
6ecdd02
Compare
robinbb
added a commit
that referenced
this pull request
Jun 9, 2026
Lifts the [can_filter] Melange opt-out installed by #14516 (L4). The BFS narrowing pipeline now activates for Melange consumer compiles on the same terms as OCaml — same [Lib_index], same wrapped-lib soundness recovery, same [must_glob_set] / [tight_set] split. L4 routed Melange through [deps_of_entries] (broad) on the rationale that the narrowing had been developed and tested with OCaml-mode in mind. The defensive gate has no remaining technical justification: the cross-library walk reads [ocamldep] output (mode-agnostic), the classification pipeline is keyed on [Lib_index] entries (already populated for libs compiled in either mode), and the only adjustment needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to round out per-module deps for [Melange Cmj] consumers. [Module_compilation.lib_deps_for_module]: drop the [match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard collapses to the four mode-independent predicates ([dep_graph] is real, the consumer module is in it, the module kind is filterable, the cctx is neither virtual nor a parameter). [Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange] disjunct from the singleton-stanza short-circuit. Single-module Melange stanzas with library deps now produce a real dep graph (the per-module filter consumes it). Comment reworded to drop the stale "Melange [can_filter]" forward-reference. [Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors [groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by the broad-dep path. The mli prose is rewritten — the "future Melange caller" caveat is gone, replaced with a positive description of all four [cm_kind] cases. [doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass narrowing" statements; the [can_filter] predicate list no longer includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to describe Melange's [want_cmj] participation. Soundness: the only edge surfaced by this layer is the alias-form issue already documented by [wrapped-internal-leak.t] on L4 — a consumer referencing [Foo__Bar] directly via the wrapped-lib mangled-form alias. GitHub code search on 2026-05-25 found zero matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__] outside Melange's own test suite; the pattern is essentially absent in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml] is the one known caller and is fixed in melange-re/melange to route through the supported [Melange_ppx.String_interp] wrapper. Tests: six existing Melange cram tests are promoted for output drift under the new path: - [depend-on-installed.t], [emit-installed.t], [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep (internal)] line in [--display=short] trace. - [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder of [ocamlc helper] vs [melc foo] in the build trace. No artefact change. - [missing-melc.t]: duplicated [melc not found] error block — the narrowing now triggers the consumer's per-module compile, which fails with the same melc-missing error. Exit code unchanged. Stack: rebases on #14521 (L9). Part of #14492. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
added a commit
that referenced
this pull request
Jun 10, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
added a commit
that referenced
this pull request
Jun 10, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
There was a problem hiding this comment.
Pull request overview
This PR improves the performance of per-module library dependency narrowing by memoising Lib.closure on (linking, for_, libs), and adds a detailed developer reference documenting the narrowing algorithm introduced in the #14492 stack.
Changes:
- Memoise
Lib.closureviaMemo.createkeyed on(linking, for_, libs)to avoid repeated traversal of the library dependency graph. - Update
Lib.closureAPI docs to describe memoisation and how callers should canonicalise inputs for cache reuse. - Add
doc/dev/per-module-narrowing.md, a reference document describing the per-module narrowing algorithm and its soundness recoveries.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| src/dune_rules/lib.ml | Wraps Lib.closure in a Memo keyed by (linking, for_, libs) to cache closure computations. |
| src/dune_rules/lib.mli | Documents the new memoisation behaviour and caller expectations for canonicalising the input list. |
| doc/dev/per-module-narrowing.md | Adds an in-depth implementer-facing description of the narrowing pipeline and invariants. |
[Compilation_context.Raw_refs] caches the [Action_builder.t] computed for each ocamldep raw-deps read inside a cctx. Two consumer modules that share trans_deps (or a consumer and one of its trans deps that share an [obj_name + ml_kind]) get the same builder. The cache short-circuits before constructing the builder; on hit, no allocation. [Raw_refs.Key] distinguishes the two read patterns the per-module filter uses: [Consumer] (the cctx-driving module's own deps, keyed by [ml_kind]) and [Transitive] (a dep module's deps, keyed by [cm_kind] because the impl/intf gating in [need_impl_deps_of] varies by cm_kind on the [Cmx]/opaque path). Conservatively-distinct keying — never collapse two semantically-different reads under one cache cell. [Compilation_context.cached_raw_refs t ~key ~compute] is the thin public surface: lookup, compute on miss, store, return the builder. [Module_compilation.lib_deps_for_module]: wraps the inline [read_dep_m_raw] body that the BFS uses for both the consumer's own and each trans dep's raw refs. No semantic change — the cache only deduplicates builder construction across calls within the same cctx. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
The per-module filter calls [Lib.closure] twice per consumer module (once for [direct_libs], once for [must_glob_libs]) on each compile rule. Across a cctx, many modules pass overlapping inputs to these closures; without memoisation every call re-traverses the dependency graph. [Lib.closure] is now defined as [Memo.exec] over a [Memo.create] keyed on [(bool * Compilation_mode.t * t list)]. The list-of-libs key is order- and multiplicity-sensitive, so callers that share inputs need to canonicalise (sort by [Lib.compare]) for maximum cache reuse — [lib_deps_for_module] already does this at both call sites. A docstring on [val closure] notes the requirement. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Add [doc/dev/per-module-narrowing.md] describing the per-module library file dependency narrowing introduced in #14492 (split into PRs #14513..#14521 as layers L1..L9): - The motivation and soundness model. - The [can_filter] precondition and [has_virtual_impl] early-out. - The narrowing pipeline: read ocamldep raw refs → [referenced] → [Lib.closure] → cross-library BFS → classification → emit per-lib deps and filtered include flags. - The data structures used ([Lib_index], the per-cctx [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos). - Soundness fallbacks (wrapped libs, virtual impls, ppx runtime). - A source map locating each concern in [src/dune_rules/]. - A layer-by-layer summary of #14513..#14521. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Documents the fourth class of soundness recovery added in L5: the BFS's per-iteration step now extends the frontier with the modules named by each visited library's effective [-open] flags, in addition to the entry's impl + intf ocamldep raw refs. "Effective" covers both the library stanza's own [(flags ...)] and any [(env ...)] stanza that contributes default flags to the dep lib's directory; both injection paths are captured even when the library stanza is [:standard]. The reachability rule the BFS computes becomes a three-way disjunction (consumer references it; reached module's ocamldep names it; reached module's owning lib's effective flags open it). Updates the [cross_lib_tight_set] code snippet's signature (now threads [~mode]), the per-iteration description, the "Soundness recovery and known edge cases" list (adds a fourth class), the L5 layer-summary line, and the cost-characteristics list (adds the per-visited-lib effective-flags expansion). Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Alizter
approved these changes
Jun 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer 9 of 9 of #14492. Performance + algorithm reference doc.
Lib.closureis now defined asMemo.execover aMemo.createkeyed on(linking, for_, libs). The per-module filter callsLib.closuretwice per consumer module (once fordirect_libs, once formust_glob_libs); without memoisation every call re-traverses the dependency graph.The list-of-libs key is order- and multiplicity-sensitive — callers that share inputs (e.g.
lib_deps_for_moduleat both call sites) need to canonicalise viaList.sort_uniq ~compare:Lib.compare; the existing call sites already do.Adds
doc/dev/per-module-narrowing.md(626 lines): a reference for the narrowing algorithm — frontier construction, BFS expansion through dep-lib references, classification into tight vs glob, soundness recovery (wrapped-closure, ppx-runtime, virtual-impl, stanza-(flags -open)), and the filtered-include-flag emit path. Intended as the implementer's first stop when revisitinglib_deps_for_moduleorfiltered_include_flags.Stack: rebases on #14520. Final layer of the stack.
Part of #14492. Related to #4572.