Skip to content

perf: memoise [Lib.closure] keyed on (linking, for_, libs)#14521

Merged
robinbb merged 4 commits into
robinbb-14492-test-stack-integrationfrom
robinbb-14492-l9-lib-closure-memo
Jun 16, 2026
Merged

perf: memoise [Lib.closure] keyed on (linking, for_, libs)#14521
robinbb merged 4 commits into
robinbb-14492-test-stack-integrationfrom
robinbb-14492-l9-lib-closure-memo

Conversation

@robinbb

@robinbb robinbb commented May 13, 2026

Copy link
Copy Markdown
Collaborator

Layer 9 of 9 of #14492. Performance + algorithm reference doc.

Lib.closure is now defined as Memo.exec over a Memo.create keyed on (linking, for_, libs). The per-module filter calls Lib.closure twice per consumer module (once for direct_libs, once for must_glob_libs); without memoisation every call re-traverses the dependency graph.

The list-of-libs key is order- and multiplicity-sensitive — callers that share inputs (e.g. lib_deps_for_module at both call sites) need to canonicalise via List.sort_uniq ~compare:Lib.compare; the existing call sites already do.

Adds doc/dev/per-module-narrowing.md (626 lines): a reference for the narrowing algorithm — frontier construction, BFS expansion through dep-lib references, classification into tight vs glob, soundness recovery (wrapped-closure, ppx-runtime, virtual-impl, stanza-(flags -open)), and the filtered-include-flag emit path. Intended as the implementer's first stop when revisiting lib_deps_for_module or filtered_include_flags.

Stack: rebases on #14520. Final layer of the stack.

Part of #14492. Related to #4572.

@robinbb robinbb self-assigned this May 13, 2026
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from ee1f8ce to 58ed81a Compare May 14, 2026 00:36
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch 2 times, most recently from 215bfef to bf10fd7 Compare May 14, 2026 18:38
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from 58ed81a to ec1ccdb Compare May 14, 2026 23:58
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from bf10fd7 to 6470974 Compare May 14, 2026 23:58
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from ec1ccdb to 8054767 Compare May 15, 2026 02:57
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch 2 times, most recently from 4f3d261 to 8ed3374 Compare May 16, 2026 03:59
robinbb added a commit that referenced this pull request May 16, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request May 16, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in this PR (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request May 18, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 8ed3374 to 356728c Compare May 18, 2026 00:07
robinbb added a commit that referenced this pull request May 20, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request May 20, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request May 21, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from 8054767 to a4b31bc Compare May 21, 2026 01:34
robinbb added a commit that referenced this pull request May 21, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 356728c to e7aa8f5 Compare May 21, 2026 01:34
robinbb added a commit that referenced this pull request May 22, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from a4b31bc to bccbb29 Compare May 22, 2026 03:29
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from e7aa8f5 to a38b747 Compare May 22, 2026 03:29
robinbb added a commit that referenced this pull request May 22, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request May 22, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from bccbb29 to 2095cf4 Compare May 22, 2026 23:57
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from a38b747 to 1ca1078 Compare May 22, 2026 23:57
robinbb added a commit that referenced this pull request May 22, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from 2095cf4 to 5d6842c Compare May 23, 2026 05:18
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from 0849010 to 45fa463 Compare June 8, 2026 01:05
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 0e29736 to 405579f Compare June 8, 2026 01:05
robinbb added a commit that referenced this pull request Jun 8, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from 45fa463 to 419d452 Compare June 8, 2026 01:39
robinbb added a commit that referenced this pull request Jun 8, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 405579f to 75bb1e9 Compare June 8, 2026 01:39
robinbb added a commit that referenced this pull request Jun 8, 2026
Lifts the [can_filter] Melange opt-out installed by #14516 (L4). The
BFS narrowing pipeline now activates for Melange consumer compiles on
the same terms as OCaml — same [Lib_index], same wrapped-lib soundness
recovery, same [must_glob_set] / [tight_set] split.

L4 routed Melange through [deps_of_entries] (broad) on the rationale
that the narrowing had been developed and tested with OCaml-mode in
mind. The defensive gate has no remaining technical justification: the
cross-library walk reads [ocamldep] output (mode-agnostic), the
classification pipeline is keyed on [Lib_index] entries (already
populated for libs compiled in either mode), and the only adjustment
needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to
round out per-module deps for [Melange Cmj] consumers.

[Module_compilation.lib_deps_for_module]: drop the
[match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard
collapses to the four mode-independent predicates ([dep_graph] is real,
the consumer module is in it, the module kind is filterable, the cctx
is neither virtual nor a parameter).

[Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange]
disjunct from the singleton-stanza short-circuit. Single-module Melange
stanzas with library deps now produce a real dep graph (the per-module
filter consumes it). Comment reworded to drop the stale "Melange
[can_filter]" forward-reference.

[Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric
with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps
now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors
[groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by
the broad-dep path. The mli prose is rewritten — the "future Melange
caller" caveat is gone, replaced with a positive description of all
four [cm_kind] cases.

[doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass
narrowing" statements; the [can_filter] predicate list no longer
includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to
describe Melange's [want_cmj] participation.

Soundness: the only edge surfaced by this layer is the alias-form
issue already documented by [wrapped-internal-leak.t] on L4 — a
consumer referencing [Foo__Bar] directly via the wrapped-lib
mangled-form alias. GitHub code search on 2026-05-25 found zero
matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__]
outside Melange's own test suite; the pattern is essentially absent
in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml]
is the one known caller and is fixed in melange-re/melange to route
through the supported [Melange_ppx.String_interp] wrapper.

Tests: six existing Melange cram tests are promoted for output drift
under the new path:

- [depend-on-installed.t], [emit-installed.t],
  [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep
  (internal)] line in [--display=short] trace.
- [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder
  of [ocamlc helper] vs [melc foo] in the build trace. No artefact
  change.
- [missing-melc.t]: duplicated [melc not found] error block — the
  narrowing now triggers the consumer's per-module compile, which
  fails with the same melc-missing error. Exit code unchanged.

Stack: rebases on #14521 (L9). Part of #14492.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request Jun 8, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from 419d452 to c9c3957 Compare June 8, 2026 18:40
robinbb added a commit that referenced this pull request Jun 8, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 75bb1e9 to 15a0200 Compare June 8, 2026 18:40
robinbb added a commit that referenced this pull request Jun 8, 2026
Lifts the [can_filter] Melange opt-out installed by #14516 (L4). The
BFS narrowing pipeline now activates for Melange consumer compiles on
the same terms as OCaml — same [Lib_index], same wrapped-lib soundness
recovery, same [must_glob_set] / [tight_set] split.

L4 routed Melange through [deps_of_entries] (broad) on the rationale
that the narrowing had been developed and tested with OCaml-mode in
mind. The defensive gate has no remaining technical justification: the
cross-library walk reads [ocamldep] output (mode-agnostic), the
classification pipeline is keyed on [Lib_index] entries (already
populated for libs compiled in either mode), and the only adjustment
needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to
round out per-module deps for [Melange Cmj] consumers.

[Module_compilation.lib_deps_for_module]: drop the
[match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard
collapses to the four mode-independent predicates ([dep_graph] is real,
the consumer module is in it, the module kind is filterable, the cctx
is neither virtual nor a parameter).

[Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange]
disjunct from the singleton-stanza short-circuit. Single-module Melange
stanzas with library deps now produce a real dep graph (the per-module
filter consumes it). Comment reworded to drop the stale "Melange
[can_filter]" forward-reference.

[Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric
with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps
now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors
[groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by
the broad-dep path. The mli prose is rewritten — the "future Melange
caller" caveat is gone, replaced with a positive description of all
four [cm_kind] cases.

[doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass
narrowing" statements; the [can_filter] predicate list no longer
includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to
describe Melange's [want_cmj] participation.

Soundness: the only edge surfaced by this layer is the alias-form
issue already documented by [wrapped-internal-leak.t] on L4 — a
consumer referencing [Foo__Bar] directly via the wrapped-lib
mangled-form alias. GitHub code search on 2026-05-25 found zero
matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__]
outside Melange's own test suite; the pattern is essentially absent
in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml]
is the one known caller and is fixed in melange-re/melange to route
through the supported [Melange_ppx.String_interp] wrapper.

Tests: six existing Melange cram tests are promoted for output drift
under the new path:

- [depend-on-installed.t], [emit-installed.t],
  [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep
  (internal)] line in [--display=short] trace.
- [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder
  of [ocamlc helper] vs [melc foo] in the build trace. No artefact
  change.
- [missing-melc.t]: duplicated [melc not found] error block — the
  narrowing now triggers the consumer's per-module compile, which
  fails with the same melc-missing error. Exit code unchanged.

Stack: rebases on #14521 (L9). Part of #14492.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from c9c3957 to b076dc4 Compare June 8, 2026 23:27
robinbb added a commit that referenced this pull request Jun 8, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 15a0200 to 1e32a97 Compare June 8, 2026 23:27
robinbb added a commit that referenced this pull request Jun 8, 2026
Lifts the [can_filter] Melange opt-out installed by #14516 (L4). The
BFS narrowing pipeline now activates for Melange consumer compiles on
the same terms as OCaml — same [Lib_index], same wrapped-lib soundness
recovery, same [must_glob_set] / [tight_set] split.

L4 routed Melange through [deps_of_entries] (broad) on the rationale
that the narrowing had been developed and tested with OCaml-mode in
mind. The defensive gate has no remaining technical justification: the
cross-library walk reads [ocamldep] output (mode-agnostic), the
classification pipeline is keyed on [Lib_index] entries (already
populated for libs compiled in either mode), and the only adjustment
needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to
round out per-module deps for [Melange Cmj] consumers.

[Module_compilation.lib_deps_for_module]: drop the
[match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard
collapses to the four mode-independent predicates ([dep_graph] is real,
the consumer module is in it, the module kind is filterable, the cctx
is neither virtual nor a parameter).

[Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange]
disjunct from the singleton-stanza short-circuit. Single-module Melange
stanzas with library deps now produce a real dep graph (the per-module
filter consumes it). Comment reworded to drop the stale "Melange
[can_filter]" forward-reference.

[Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric
with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps
now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors
[groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by
the broad-dep path. The mli prose is rewritten — the "future Melange
caller" caveat is gone, replaced with a positive description of all
four [cm_kind] cases.

[doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass
narrowing" statements; the [can_filter] predicate list no longer
includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to
describe Melange's [want_cmj] participation.

Soundness: the only edge surfaced by this layer is the alias-form
issue already documented by [wrapped-internal-leak.t] on L4 — a
consumer referencing [Foo__Bar] directly via the wrapped-lib
mangled-form alias. GitHub code search on 2026-05-25 found zero
matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__]
outside Melange's own test suite; the pattern is essentially absent
in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml]
is the one known caller and is fixed in melange-re/melange to route
through the supported [Melange_ppx.String_interp] wrapper.

Tests: six existing Melange cram tests are promoted for output drift
under the new path:

- [depend-on-installed.t], [emit-installed.t],
  [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep
  (internal)] line in [--display=short] trace.
- [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder
  of [ocamlc helper] vs [melc foo] in the build trace. No artefact
  change.
- [missing-melc.t]: duplicated [melc not found] error block — the
  narrowing now triggers the consumer's per-module compile, which
  fails with the same melc-missing error. Exit code unchanged.

Stack: rebases on #14521 (L9). Part of #14492.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from b076dc4 to a525d86 Compare June 9, 2026 14:50
robinbb added a commit that referenced this pull request Jun 9, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 1e32a97 to 6ecdd02 Compare June 9, 2026 14:50
robinbb added a commit that referenced this pull request Jun 9, 2026
Lifts the [can_filter] Melange opt-out installed by #14516 (L4). The
BFS narrowing pipeline now activates for Melange consumer compiles on
the same terms as OCaml — same [Lib_index], same wrapped-lib soundness
recovery, same [must_glob_set] / [tight_set] split.

L4 routed Melange through [deps_of_entries] (broad) on the rationale
that the narrowing had been developed and tested with OCaml-mode in
mind. The defensive gate has no remaining technical justification: the
cross-library walk reads [ocamldep] output (mode-agnostic), the
classification pipeline is keyed on [Lib_index] entries (already
populated for libs compiled in either mode), and the only adjustment
needed in [deps_of_entry_modules] is a symmetric [want_cmj] arm to
round out per-module deps for [Melange Cmj] consumers.

[Module_compilation.lib_deps_for_module]: drop the
[match Lib_mode.of_cm_kind cm_kind] arm from [can_filter]. The guard
collapses to the four mode-independent predicates ([dep_graph] is real,
the consumer module is in it, the module kind is filterable, the cctx
is neither virtual nor a parameter).

[Dep_rules.rules]: drop the [|| Compilation_mode.equal for_ Melange]
disjunct from the singleton-stanza short-circuit. Single-module Melange
stanzas with library deps now produce a real dep graph (the per-module
filter consumes it). Comment reworded to drop the stale "Melange
[can_filter]" forward-reference.

[Lib_file_deps.deps_of_entry_modules]: add a [want_cmj] arm symmetric
with [want_cmx]. For [cm_kind = Melange Cmj] consumers, per-module deps
now include the dep lib's [.cmj] in addition to its [.cmi]; mirrors
[groups_for_cm_kind]'s [Melange Cmj -> [Cmi; Cmj]] expansion used by
the broad-dep path. The mli prose is rewritten — the "future Melange
caller" caveat is gone, replaced with a positive description of all
four [cm_kind] cases.

[doc/dev/per-module-narrowing.md]: drop the two "Melange paths bypass
narrowing" statements; the [can_filter] predicate list no longer
includes an [Ocaml _] arm. Rewrite the trailing-section paragraph to
describe Melange's [want_cmj] participation.

Soundness: the only edge surfaced by this layer is the alias-form
issue already documented by [wrapped-internal-leak.t] on L4 — a
consumer referencing [Foo__Bar] directly via the wrapped-lib
mangled-form alias. GitHub code search on 2026-05-25 found zero
matches for [= Core__], [= Async__], [= Ppxlib__], [= Bonsai__]
outside Melange's own test suite; the pattern is essentially absent
in the OCaml ecosystem. Melange's [test/unit-tests/ounit_unicode_tests.ml]
is the one known caller and is fixed in melange-re/melange to route
through the supported [Melange_ppx.String_interp] wrapper.

Tests: six existing Melange cram tests are promoted for output drift
under the new path:

- [depend-on-installed.t], [emit-installed.t],
  [emit-installed-two-modes.t], [emit-private.t]: +1 [ocamldep
  (internal)] line in [--display=short] trace.
- [melange-conditional-modules.t]: +1 [ocamldep (internal)] + reorder
  of [ocamlc helper] vs [melc foo] in the build trace. No artefact
  change.
- [missing-melc.t]: duplicated [melc not found] error block — the
  narrowing now triggers the consumer's per-module compile, which
  fails with the same melc-missing error. Exit code unchanged.

Stack: rebases on #14521 (L9). Part of #14492.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request Jun 10, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request Jun 10, 2026
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the performance of per-module library dependency narrowing by memoising Lib.closure on (linking, for_, libs), and adds a detailed developer reference documenting the narrowing algorithm introduced in the #14492 stack.

Changes:

  • Memoise Lib.closure via Memo.create keyed on (linking, for_, libs) to avoid repeated traversal of the library dependency graph.
  • Update Lib.closure API docs to describe memoisation and how callers should canonicalise inputs for cache reuse.
  • Add doc/dev/per-module-narrowing.md, a reference document describing the per-module narrowing algorithm and its soundness recoveries.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
src/dune_rules/lib.ml Wraps Lib.closure in a Memo keyed by (linking, for_, libs) to cache closure computations.
src/dune_rules/lib.mli Documents the new memoisation behaviour and caller expectations for canonicalising the input list.
doc/dev/per-module-narrowing.md Adds an in-depth implementer-facing description of the narrowing pipeline and invariants.

Comment thread src/dune_rules/lib.ml
Comment thread src/dune_rules/lib.mli Outdated
robinbb added 3 commits June 13, 2026 14:24
[Compilation_context.Raw_refs] caches the [Action_builder.t] computed
for each ocamldep raw-deps read inside a cctx. Two consumer modules
that share trans_deps (or a consumer and one of its trans deps that
share an [obj_name + ml_kind]) get the same builder. The cache
short-circuits before constructing the builder; on hit, no allocation.

[Raw_refs.Key] distinguishes the two read patterns the per-module
filter uses: [Consumer] (the cctx-driving module's own deps, keyed by
[ml_kind]) and [Transitive] (a dep module's deps, keyed by [cm_kind]
because the impl/intf gating in [need_impl_deps_of] varies by cm_kind
on the [Cmx]/opaque path). Conservatively-distinct keying — never
collapse two semantically-different reads under one cache cell.

[Compilation_context.cached_raw_refs t ~key ~compute] is the thin
public surface: lookup, compute on miss, store, return the builder.

[Module_compilation.lib_deps_for_module]: wraps the inline
[read_dep_m_raw] body that the BFS uses for both the consumer's own
and each trans dep's raw refs. No semantic change — the cache only
deduplicates builder construction across calls within the same cctx.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
The per-module filter calls [Lib.closure] twice per consumer module
(once for [direct_libs], once for [must_glob_libs]) on each compile
rule. Across a cctx, many modules pass overlapping inputs to these
closures; without memoisation every call re-traverses the dependency
graph.

[Lib.closure] is now defined as [Memo.exec] over a [Memo.create]
keyed on [(bool * Compilation_mode.t * t list)]. The list-of-libs key
is order- and multiplicity-sensitive, so callers that share inputs
need to canonicalise (sort by [Lib.compare]) for maximum cache reuse —
[lib_deps_for_module] already does this at both call sites. A
docstring on [val closure] notes the requirement.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread doc/dev/per-module-narrowing.md
Documents the fourth class of soundness recovery added in L5: the
BFS's per-iteration step now extends the frontier with the modules
named by each visited library's effective [-open] flags, in
addition to the entry's impl + intf ocamldep raw refs. "Effective"
covers both the library stanza's own [(flags ...)] and any
[(env ...)] stanza that contributes default flags to the dep lib's
directory; both injection paths are captured even when the library
stanza is [:standard]. The reachability rule the BFS computes
becomes a three-way disjunction (consumer references it; reached
module's ocamldep names it; reached module's owning lib's effective
flags open it).

Updates the [cross_lib_tight_set] code snippet's signature (now
threads [~mode]), the per-iteration description, the "Soundness
recovery and known edge cases" list (adds a fourth class), the L5
layer-summary line, and the cost-characteristics list (adds the
per-visited-lib effective-flags expansion).

Signed-off-by: Robin Bate Boerop <me@robinbb.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread src/dune_rules/lib.ml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants