feat: per-module library dependency filter via ocamldep BFS by robinbb · Pull Request #14516 · ocaml/dune

robinbb · 2026-05-13T18:22:59Z

Layer 4 of 9 of #14492. The core BFS. NOTE: this is the only layer that is expected to fail tests. This cannot be merged into 'main' without layer 5.

Module_compilation.lib_deps_for_module activates the tight branch: a per-module BFS (breadth-first search) over the cross-library dependency graph (cross_lib_tight_set) computes the dep-lib modules the consumer actually references. The compile rule sees only those cmi/cmx files. can_filter falls back to glob for Melange, dummy dep graphs, synthesised modules, non-filterable kinds, and consumer-side virtual / parameter cctxs.

Includes are still the cctx-wide -I / -H; filtered include flags ship in layer 6. Soundness recovery for wrapped libs / ppx-runtime / virtual-impl deps ships in layer 5.

Part of #14492. Related to #4572.

Layer 3 of 9 of #14492. Behaviour-equivalent refactor. `Compilation_context.Includes.make` drops `~opaque` and stops emitting `Hidden_deps`; `Includes.t` carries only `-I`/`-H` flags. `Module_compilation.lib_deps_for_module` (scaffold form: always-glob) and `lib_cm_deps` wire the dep set back in via `deps_of_entries`. `build_cm` and `ocamlc_i` are switched to consume `lib_cm_deps`; `Alias` / `Wrapped_compat` short-circuit to the cctx's now-flag-only `Includes`. `Lib_file_deps.deps_with_exts` removed — only used by the old `Cmx` arm of `Includes.make`. Combined effect: same `-I` / `-H`, same lib file deps. Stack: rebases on #14514. Next: #14516. Part of #14492. Related to #4572. Signed-off-by: Robin Bate Boerop <me@robinbb.com>

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Layer 3 of 9 of #14492. Behaviour-equivalent refactor. `Compilation_context.Includes.make` drops `~opaque` and stops emitting `Hidden_deps`; `Includes.t` carries only `-I`/`-H` flags. `Module_compilation.lib_deps_for_module` (scaffold form: always-glob) and `lib_cm_deps` wire the dep set back in via `deps_of_entries`. `build_cm` and `ocamlc_i` are switched to consume `lib_cm_deps`; `Alias` / `Wrapped_compat` short-circuit to the cctx's now-flag-only `Includes`. `Lib_file_deps.deps_with_exts` removed — only used by the old `Cmx` arm of `Includes.make`. Combined effect: same `-I` / `-H`, same lib file deps. Stack: rebases on #14514. Next: #14516. Part of #14492. Related to #4572. Signed-off-by: Robin Bate Boerop <me@robinbb.com>

Lifts the [can_filter] Melange opt-out installed by ocaml#14516 (L4). The BFS narrowing pipeline now activates for Melange consumer compiles on the same terms as OCaml — same [Lib_index], same wrapped-lib soundness recovery, same [must_glob_set] / [tight_set] split. L4 routed Melange through [deps_of_entries] (broad) on the rationale that the narrowing had been developed and tested with OCaml-mode in mind. The defensive gate has no remaining technical justification: the cross-library walk reads [ocamldep] output (mode-agnostic), the classification pipeline is keyed on [Lib_index] entries (already populated for libs compiled in either mode), and the only adjustment needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to round out per-module deps for [Melange Cmj] consumers. [Module_compilation.lib_deps_for_module]: drop the [match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard collapses to the four mode-independent predicates ([dep_graph] is real, the consumer module is in it, the module kind is filterable, the cctx is neither virtual nor a parameter). [Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange] disjunct from the singleton-stanza short-circuit. Single-module Melange stanzas with library deps now produce a real dep graph (the per-module filter consumes it). Comment reworded to drop the stale "Melange [can_filter]" forward-reference. [Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors [groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by the broad-dep path. The mli prose is rewritten — the "future Melange caller" caveat is gone, replaced with a positive description of all four [cm_kind] cases. [doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass narrowing" statements; the [can_filter] predicate list no longer includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to describe Melange's [want_cmj] participation. Soundness: the only edge surfaced by this layer is the alias-form issue already documented by [wrapped-internal-leak.t] on L4 — a consumer referencing [Foo__Bar] directly via the wrapped-lib mangled-form alias. GitHub code search on 2026-05-25 found zero matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__] outside Melange's own test suite; the pattern is essentially absent in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml] is the one known caller and is fixed in melange-re/melange to route through the supported [Melange_ppx.String_interp] wrapper. Tests: six existing Melange cram tests are promoted for output drift under the new path: - [depend-on-installed.t], [emit-installed.t], [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep (internal)] line in [--display=short] trace. - [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder of [ocamlc helper] vs [melc foo] in the build trace. No artefact change. - [missing-melc.t]: duplicated [melc not found] error block — the narrowing now triggers the consumer's per-module compile, which fails with the same melc-missing error. Exit code unchanged. Stack: rebases on ocaml#14521 (L9). Part of ocaml#14492. Signed-off-by: Robin Bate Boerop <me@robinbb.com>

Activates the tight branch in [lib_deps_for_module]: a per-module BFS over the cross-library dependency graph (built from each lib's [ocamldep -modules] output, normalised through [build_lib_index]'s post-pp module map) produces the set of dep-lib modules actually referenced by the consumer module. The compile rule sees only those [.cmi]/[.cmx] files; sibling-module recompilations on unreferenced dep-lib cmi changes drop out. Include flags are still the cctx-wide [-I]/[-H] in this layer; the filtered include flags ship separately. Wrapped-lib soundness recovery, virtual-impl gating on the deps side, ppx-runtime force-glob, and stanza/env [-open] BFS expansion ship in the next layer — this commit leaves 18 cram tests broken, all restored at L5: - 5 canonical symptom fixtures under per-module-lib-deps/: [auto-wrapped-child-reexport.t], [wrapped-closure-precision.t], [wrapped-reexport-via-open-flag.t] (wrapped recovery), [ppx-runtime-libraries.t] (ppx-runtime recovery), [virtual-library.t] (virtual-impl recovery). - 4 more per-module-lib-deps/ casualties pinned by baselines on main: [cinaps-pps-runtime-libs.t], [toplevel-pps-runtime-libs.t], [cross-lib-instrumentation-barrier.t/run.t], [wrapped-from-vlib-soundness.t]. - 7 pre-existing [virtual-libraries/] tests: [implements-external.t/run.t], [impl-private-modules-incremental.t], [impl-private-modules.t/run.t], [impl-using-vlib-modules.t/run.t], [preprocess.t/run.t], [unwrapped.t/run.t], [variants-sub-module.t/run.t]. - [melange/virtual_lib_compilation.t/run.t], [wrapped-transition-incremental.t]. [Module_compilation]: - [union_module_name_sets_mapped]: parallel fold over a list of [Module_name.Set.t] producers. - [module_kind_is_filterable]: predicate excluding kinds whose dep story is handled outside the BFS ([Root], [Wrapped_compat], [Impl_vmodule], [Virtual], [Parameter]). - [cross_lib_tight_set]: BFS expanding through the lib_index's [(lib, entry)] pairs, reading each entry's impl + intf [ocamldep] output. Non-tight-eligible libs terminate chains. - [lib_deps_for_module]: replaces the scaffold body. A [can_filter] guard (consumer-side virtual / parameter, dummy dep graph, module kind, [Module.has m ~ml_kind]) falls back to glob; otherwise runs the BFS, classifies libs via [Lib_file_deps.Lib_index.filter_libs_with_modules], and emits specific-file deps for tight libs + glob deps for non-tight / unreached-non-eligible libs. Returns the cctx-wide [Includes]; filtered include flags follow in a later layer. [Compilation_context.create]: peek [direct_requires] / [hidden_requires] and pass [has_library_deps] to [Dep_rules.rules]. Single-module stanzas with library deps now produce real dep graphs (the filter needs them). [Dep_rules.rules]: gate the singleton short-circuit on [(not has_library_deps) || for_ = Melange]. Other singletons fall through to the full dep-graph build. [Ocaml_flags]: [extract_open_module_names] surfaces [-open Foo] references (and the head module of dotted paths such as [-open Foo.Bar]) that ocamldep doesn't see; they join [BFS]'s initial frontier. [Virtual_rules]: [is_virtual_or_parameter] — true for virtual impls and parameter cctxs; used by [can_filter] to suppress per-module filtering on consumer cctxs whose dep story [Dep_rules] handles specially. [Parameterised_rules]: pass [~has_library_deps:true] to the [Dep_rules.rules] call; conservative — the dep-rules path here serves external parameterised libs whose dep set is built from generated sources. Tests: rebuild-precision promotions for the existing modified-test set in #14492 — cram outputs reflect the tighter dep / rebuild behavior that L4 already produces. New soundness test fixtures (and the two tests gated on filtered include flags, [per-module-include-flags.t] / [add-unreferenced-sibling-lib.t]) are deferred to their respective follow-ups. Signed-off-by: Robin Bate Boerop <me@robinbb.com>

Lifts the [can_filter] Melange opt-out installed by #14516 (L4). The BFS narrowing pipeline now activates for Melange consumer compiles on the same terms as OCaml — same [Lib_index], same wrapped-lib soundness recovery, same [must_glob_set] / [tight_set] split. L4 routed Melange through [deps_of_entries] (broad) on the rationale that the narrowing had been developed and tested with OCaml-mode in mind. The defensive gate has no remaining technical justification: the cross-library walk reads [ocamldep] output (mode-agnostic), the classification pipeline is keyed on [Lib_index] entries (already populated for libs compiled in either mode), and the only adjustment needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to round out per-module deps for [Melange Cmj] consumers. [Module_compilation.lib_deps_for_module]: drop the [match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard collapses to the four mode-independent predicates ([dep_graph] is real, the consumer module is in it, the module kind is filterable, the cctx is neither virtual nor a parameter). [Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange] disjunct from the singleton-stanza short-circuit. Single-module Melange stanzas with library deps now produce a real dep graph (the per-module filter consumes it). Comment reworded to drop the stale "Melange [can_filter]" forward-reference. [Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors [groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by the broad-dep path. The mli prose is rewritten — the "future Melange caller" caveat is gone, replaced with a positive description of all four [cm_kind] cases. [doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass narrowing" statements; the [can_filter] predicate list no longer includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to describe Melange's [want_cmj] participation. Soundness: the only edge surfaced by this layer is the alias-form issue already documented by [wrapped-internal-leak.t] on L4 — a consumer referencing [Foo__Bar] directly via the wrapped-lib mangled-form alias. GitHub code search on 2026-05-25 found zero matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__] outside Melange's own test suite; the pattern is essentially absent in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml] is the one known caller and is fixed in melange-re/melange to route through the supported [Melange_ppx.String_interp] wrapper. Tests: six existing Melange cram tests are promoted for output drift under the new path: - [depend-on-installed.t], [emit-installed.t], [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep (internal)] line in [--display=short] trace. - [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder of [ocamlc helper] vs [melc foo] in the build trace. No artefact change. - [missing-melc.t]: duplicated [melc not found] error block — the narrowing now triggers the consumer's per-module compile, which fails with the same melc-missing error. Exit code unchanged. Stack: rebases on #14521 (L9). Part of #14492. Signed-off-by: Robin Bate Boerop <me@robinbb.com>

Alizter · 2026-06-08T18:13:25Z

FTR for those reading, this PR is targeted at a staging branch and not main.

Alizter · 2026-06-08T18:13:54Z

There is some clean up we can do, but I think it will make sense to do later after L5-9 when considering to merge into main.

Lifts the [can_filter] Melange opt-out installed by #14516 (L4). The BFS narrowing pipeline now activates for Melange consumer compiles on the same terms as OCaml — same [Lib_index], same wrapped-lib soundness recovery, same [must_glob_set] / [tight_set] split. L4 routed Melange through [deps_of_entries] (broad) on the rationale that the narrowing had been developed and tested with OCaml-mode in mind. The defensive gate has no remaining technical justification: the cross-library walk reads [ocamldep] output (mode-agnostic), the classification pipeline is keyed on [Lib_index] entries (already populated for libs compiled in either mode), and the only adjustment needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to round out per-module deps for [Melange Cmj] consumers. [Module_compilation.lib_deps_for_module]: drop the [match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard collapses to the four mode-independent predicates ([dep_graph] is real, the consumer module is in it, the module kind is filterable, the cctx is neither virtual nor a parameter). [Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange] disjunct from the singleton-stanza short-circuit. Single-module Melange stanzas with library deps now produce a real dep graph (the per-module filter consumes it). Comment reworded to drop the stale "Melange [can_filter]" forward-reference. [Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors [groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by the broad-dep path. The mli prose is rewritten — the "future Melange caller" caveat is gone, replaced with a positive description of all four [cm_kind] cases. [doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass narrowing" statements; the [can_filter] predicate list no longer includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to describe Melange's [want_cmj] participation. Soundness: the only edge surfaced by this layer is the alias-form issue already documented by [wrapped-internal-leak.t] on L4 — a consumer referencing [Foo__Bar] directly via the wrapped-lib mangled-form alias. GitHub code search on 2026-05-25 found zero matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__] outside Melange's own test suite; the pattern is essentially absent in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml] is the one known caller and is fixed in melange-re/melange to route through the supported [Melange_ppx.String_interp] wrapper. Tests: six existing Melange cram tests are promoted for output drift under the new path: - [depend-on-installed.t], [emit-installed.t], [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep (internal)] line in [--display=short] trace. - [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder of [ocamlc helper] vs [melc foo] in the build trace. No artefact change. - [missing-melc.t]: duplicated [melc not found] error block — the narrowing now triggers the consumer's per-module compile, which fails with the same melc-missing error. Exit code unchanged. Stack: rebases on #14521 (L9). Part of #14492. Signed-off-by: Robin Bate Boerop <me@robinbb.com>

## Summary Lifts the `can_filter` Melange opt-out installed by #14516 (L4). The BFS per-module narrowing pipeline now activates for Melange consumer compiles on the same terms as OCaml — same `Lib_index`, same wrapped-lib soundness recovery, same `must_glob_set` / `tight_set` split. `Module_compilation.lib_deps_for_module` drops the `match Lib_mode.of_cm_kind cm_kind` arm from `can_filter`. `Dep_rules.rules` drops the `|| Compilation_mode.equal for_ Melange` disjunct from the singleton-stanza short-circuit. `Lib_file_deps.deps_of_entry_modules` gains a `want_cmj` arm symmetric with `want_cmx`, so `Melange Cmj` consumers see per-module `.cmj` deps in addition to `.cmi`. Six Melange cram tests are promoted for output drift (additional `ocamldep (internal)` trace lines + one duplicated `melc not found` error in `missing-melc.t`). No artefact changes; same exit codes. ## Stack Rebases on #14521 (L9). Part of #14492. ## Validation - All 112 `test/blackbox-tests/test-cases/melange/*.t` pass after promotion. - melange-re/melange 6.0.1-54 unit-test suite: 84/84 OUnit tests pass against this branch with melange-re/melange's `test/unit-tests/ounit_unicode_tests.ml` re-routed through `Melange_ppx.String_interp` (a 3-line patch that's a no-op for upstream Melange and that the project is welcome to absorb). - Pristine 6.0.1-54 fails with `module String_interp is an alias for module Melange_ppx__String_interp, which is missing` — the same alias-form issue already documented by `wrapped-internal-leak.t` on L4. The pattern is essentially absent outside Melange's own ppx (zero GitHub code-search hits for `= Core__`, `= Async__`, `= Ppxlib__`, `= Bonsai__`). ## Fixes Part of #14492. --------- Signed-off-by: Robin Bate Boerop <me@robinbb.com>

robinbb self-assigned this May 13, 2026

This was referenced May 13, 2026

refactor: route lib file deps through [lib_deps_for_module] #14515

Merged

fix: soundness recovery for wrapped libs, ppx-runtime, virtual-impl deps #14517

Merged

Filter inter-library dependencies per module #14492

Open

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch 3 times, most recently from d3418da to 246a914 Compare May 14, 2026 00:35

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from da056f0 to a0893d7 Compare May 14, 2026 00:35

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch from 246a914 to d88da28 Compare May 14, 2026 19:07

robinbb force-pushed the robinbb-14492-l4-tight-branch branch 3 times, most recently from 4e63363 to 4c95b95 Compare May 15, 2026 01:31

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch 2 times, most recently from 8dcb05b to 60b92c8 Compare May 20, 2026 01:01

robinbb force-pushed the robinbb-14492-l4-tight-branch branch 2 times, most recently from 48451fc to e65efd4 Compare May 20, 2026 01:16

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch from 60b92c8 to dde100d Compare May 20, 2026 20:13

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from e65efd4 to 6fc605c Compare May 20, 2026 20:16

robinbb requested a review from Alizter May 20, 2026 20:18

robinbb assigned Alizter May 20, 2026

robinbb marked this pull request as ready for review May 20, 2026 20:41

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch from dde100d to 63c3651 Compare May 21, 2026 01:34

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from 6fc605c to b95dd0f Compare May 21, 2026 01:34

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch from 63c3651 to 416e701 Compare May 21, 2026 20:46

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from b95dd0f to d5a41a6 Compare May 21, 2026 20:47

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch from 416e701 to 1c1953d Compare May 22, 2026 21:16

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from d5a41a6 to 73246cc Compare May 22, 2026 21:44

Base automatically changed from robinbb-14492-l3-includes-refactor to robinbb-14492-test-stack-integration May 23, 2026 04:51

robinbb force-pushed the robinbb-14492-test-stack-integration branch from c8ab405 to 57f1c08 Compare May 23, 2026 05:17

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from ec8180d to 662e008 Compare May 23, 2026 21:24

robinbb requested a review from Copilot May 23, 2026 21:24

Copilot started reviewing on behalf of robinbb May 23, 2026 21:24 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

Comment thread src/dune_rules/module_compilation.ml

Comment thread src/dune_rules/module_compilation.ml

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from 662e008 to 9126ed9 Compare May 24, 2026 18:57

robinbb mentioned this pull request May 25, 2026

feat: per-module narrowing for Melange consumers #14732

Merged

robinbb force-pushed the robinbb-14492-test-stack-integration branch from 57f1c08 to b1dc605 Compare May 29, 2026 00:11

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from 9126ed9 to 070f9a3 Compare May 29, 2026 00:11

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from 070f9a3 to 11fc94d Compare June 8, 2026 01:05

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from 11fc94d to 4b05435 Compare June 8, 2026 01:39

Alizter approved these changes Jun 8, 2026

View reviewed changes

robinbb merged commit 4637e98 into robinbb-14492-test-stack-integration Jun 8, 2026
21 of 22 checks passed

robinbb deleted the robinbb-14492-l4-tight-branch branch June 8, 2026 18:15

robinbb mentioned this pull request Jun 8, 2026

test: cross-library .cmi baseline for dotted -open path #15056

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: per-module library dependency filter via ocamldep BFS#14516

feat: per-module library dependency filter via ocamldep BFS#14516
robinbb merged 1 commit into
robinbb-14492-test-stack-integrationfrom
robinbb-14492-l4-tight-branch

robinbb commented May 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Alizter commented Jun 8, 2026

Uh oh!

Alizter commented Jun 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

robinbb commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Alizter commented Jun 8, 2026

Uh oh!

Alizter commented Jun 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

robinbb commented May 13, 2026 •

edited

Loading