Add a new link_resource_set attribute#687
Conversation
|
@meisterT what about this approach? |
|
What do you think of making the resource usage (or the constants for a formula computing it) part of a |
|
This should probably be tackled more generally:
Yeah, I think it'd make more sense to make this heuristic configurable per-toolchain with a callback if we put something in rules_cc specifically. |
|
I've submitted bazelbuild/bazel#29284 which would go towards the more generic version of this. the one thing that wouldn't be solved by that is the resources scaling with the input set. which seems like a pretty important piece |
Yes, that's even better. Though, sometimes there are some very heavy sources (e.g. AUTOSAR generated cpp files) that would benefit from this, but it's an edge case in general. Making it configurable on the toolchain level is more versatile |
Closes #628. The former attempt (#636) was aiming to adjust the heuristic, this approach is more flexible as there is no one size fits all solution.
Problem
The hardcoded
_resource_setheuristic infinalize_link_action.bzlunderestimates memory for large link actions, causing Bazel to over-schedule parallel links and trigger OOM kills in constrained environments. There is no way for users to override this except for setting--jobsto extremely low numbers that doesn't always make sense.Solution
Add a
link_resource_setattr.string_dict()to all rules that produce linking actions (cc_binary,cc_test,cc_library,cc_shared_library). It accepts"memory"(MB) and"cpu"keys, both optional. An empty dict (the default) preserves the existing heuristic — zero behavior change for anyone who doesn't set the attr.Platform-specific values work via select():
Modern linkers (LLD, mold, gold) are multi-threaded. Telling Bazel about CPU usage prevents over-scheduling concurrent links that each internally use multiple cores.
Implementation note
Bazel's
actions.run(resource_set=...)rejects closures — only top-leveldefstatements are accepted (to prevent retention of analysis-phase data). This means user-provided values cannot be turned into a callback dynamically. The current implementation uses a pre-built lookup table of top-level functions (resource_sets.bzl), similar to bazel-lib'sresource_set_for. User values are rounded up to the nearest supported level (memory: powers of 2 from 256–65536 MB; cpu: 1, 2, 4, 8, 16).Open question for maintainers:
bazel-libalready provides a well-testedresource_set_for(memory, cpu)helper with broader coverage. Would you prefer addingaspect_build/bazel-libas a dependency to reuse it, or keeping the self-contained lookup table? Given the ongoing effort to reduce dependencies inrules_cc(#492, #584), this PR avoids adding one — but happy to switch if you'd rather not maintain a separate table.