Skip to content

Add a new link_resource_set attribute#687

Open
JSGette wants to merge 2 commits intobazelbuild:mainfrom
JSGette:jsgette/configurable_ram_for_linking
Open

Add a new link_resource_set attribute#687
JSGette wants to merge 2 commits intobazelbuild:mainfrom
JSGette:jsgette/configurable_ram_for_linking

Conversation

@JSGette
Copy link
Copy Markdown

@JSGette JSGette commented Apr 13, 2026

Closes #628. The former attempt (#636) was aiming to adjust the heuristic, this approach is more flexible as there is no one size fits all solution.

Problem

The hardcoded _resource_set heuristic in finalize_link_action.bzl underestimates memory for large link actions, causing Bazel to over-schedule parallel links and trigger OOM kills in constrained environments. There is no way for users to override this except for setting --jobs to extremely low numbers that doesn't always make sense.

Solution

Add a link_resource_set attr.string_dict() to all rules that produce linking actions (cc_binary, cc_test, cc_library, cc_shared_library). It accepts "memory" (MB) and "cpu" keys, both optional. An empty dict (the default) preserves the existing heuristic — zero behavior change for anyone who doesn't set the attr.

cc_binary(
    name = "my_binary",
    srcs = [...],
    link_resource_set = {"memory": "8192", "cpu": "2"},
)

Platform-specific values work via select():

    link_resource_set = select({
        "@platforms//os:linux": {"memory": "8192", "cpu": "4"},
        "//conditions:default": {},
    }),

Modern linkers (LLD, mold, gold) are multi-threaded. Telling Bazel about CPU usage prevents over-scheduling concurrent links that each internally use multiple cores.

Implementation note

Bazel's actions.run(resource_set=...) rejects closures — only top-level def statements are accepted (to prevent retention of analysis-phase data). This means user-provided values cannot be turned into a callback dynamically. The current implementation uses a pre-built lookup table of top-level functions (resource_sets.bzl), similar to bazel-lib's resource_set_for. User values are rounded up to the nearest supported level (memory: powers of 2 from 256–65536 MB; cpu: 1, 2, 4, 8, 16).

Open question for maintainers: bazel-lib already provides a well-tested resource_set_for(memory, cpu) helper with broader coverage. Would you prefer adding aspect_build/bazel-lib as a dependency to reuse it, or keeping the self-contained lookup table? Given the ongoing effort to reduce dependencies in rules_cc (#492, #584), this PR avoids adding one — but happy to switch if you'd rather not maintain a separate table.

@JSGette
Copy link
Copy Markdown
Author

JSGette commented Apr 13, 2026

@meisterT what about this approach?

@fmeum
Copy link
Copy Markdown
Collaborator

fmeum commented Apr 13, 2026

What do you think of making the resource usage (or the constants for a formula computing it) part of a cc_toolchain? Do you really have individual targets with very specific resource utilization?

@armandomontanez
Copy link
Copy Markdown
Collaborator

This should probably be tackled more generally:
bazelbuild/bazel#25792

What do you think of making the resource usage (or the constants for a formula computing it) part of a cc_toolchain? Do you really have individual targets with very specific resource utilization?

Yeah, I think it'd make more sense to make this heuristic configurable per-toolchain with a callback if we put something in rules_cc specifically.

@keith
Copy link
Copy Markdown
Member

keith commented Apr 13, 2026

I've submitted bazelbuild/bazel#29284 which would go towards the more generic version of this. the one thing that wouldn't be solved by that is the resources scaling with the input set. which seems like a pretty important piece

@JSGette
Copy link
Copy Markdown
Author

JSGette commented Apr 14, 2026

What do you think of making the resource usage (or the constants for a formula computing it) part of a cc_toolchain? Do you really have individual targets with very specific resource utilization?

Yes, that's even better. Though, sometimes there are some very heavy sources (e.g. AUTOSAR generated cpp files) that would benefit from this, but it's an edge case in general. Making it configurable on the toolchain level is more versatile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

resource_set for linking actions is too conservative

4 participants