Skip to content

history: finalize export records on their owning node#3716

Open
crazy-max wants to merge 4 commits intodocker:masterfrom
crazy-max:history-fix-finalize
Open

history: finalize export records on their owning node#3716
crazy-max wants to merge 4 commits intodocker:masterfrom
crazy-max:history-fix-finalize

Conversation

@crazy-max
Copy link
Member

@crazy-max crazy-max commented Mar 11, 2026

needs #3724
fixes docker/build-push-action#1388
closes #3344
closes docker/build-push-action#1392
closes docker/bake-action#336

This fixes history export --finalize for multi-node builders such as the Kubernetes driver.

The failure happens when the same logical build ref appears more than once in history, once per node, and export finalization is not applied to the node that owns the selected record. In that case Buildx can later try to export a record whose trace descriptor is still not readable and fail with NotFound while loading the trace descriptor:

+ docker buildx --builder=buildx-test-c882461ff920a3349084436f290222c8 history ls
}BUILD ID                    NAME                       STATUS      CREATED AT               DURATION   
kh4h2q8qavh3iqe7bvpz869gy   buildx-output.PdsmBG11XK   Completed   Less than a second ago   0.2s       
z8xx18nyoptmh8l0ylyo5bqat   buildx-output.PdsmBG11XK   Completed   1 second ago             1.1s       
z8xx18nyoptmh8l0ylyo5bqat   buildx-output.PdsmBG11XK   Completed   1 second ago             0.9s       
+ docker buildx --builder=buildx-test-c882461ff920a3349084436f290222c8 history export --finalize --all --output /tmp/buildx-output.PdsmBG11XK/records.dockerbuild
ERROR: failed to export: failed to export record: failed to load trace descriptor: failed to get info: NotFound: rpc error: code = NotFound desc = content digest sha256:c257101ad2a226de341bdab5149018946bbea30e8290135b15c9ccf5755f5f60: not found

This keeps the existing finalize behavior for records that do not yet expose a trace descriptor, but changes finalization to follow record ownership. Buildx now deduplicates selected records by (ref, node), sends finalize to the owning node for each selected record, re-queries records before export, and collapses duplicate per-node entries down to one logical record per ref.

It also fixes bundle export to use all available node content stores instead of stopping at the first one. That allows the exported records to resolve their referenced content correctly across multi-node builders during archive creation.

@crazy-max crazy-max changed the title History fix finalize history: always finalize selected records before export Mar 11, 2026
@crazy-max
Copy link
Member Author

Hum still fails, looking at the logs: https://github.com/docker/buildx/actions/runs/22947651040/job/66604243143?pr=3716#step:9:321

+ docker buildx --builder=buildx-test-30a7ab764361dd1f4482285765e20405 history ls
}BUILD ID                    NAME                       STATUS      CREATED AT               DURATION   
0v82njsviycrtwp3hicwi7ma1   buildx-output.9R23X274Na   Completed   Less than a second ago   0.1s       
i2hwfladqm5qikdpkv1tqrd7x   buildx-output.9R23X274Na   Completed   1 second ago             1.2s       
i2hwfladqm5qikdpkv1tqrd7x   buildx-output.9R23X274Na   Completed   1 second ago             1.0s       
+ docker buildx --builder=buildx-test-30a7ab764361dd1f4482285765e20405 history export --finalize --all --output /tmp/buildx-output.9R23X274Na/records.dockerbuild
ERROR: failed to export: failed to export record: failed to load trace descriptor: failed to get info: NotFound: rpc error: code = NotFound desc = content digest sha256:da1a7755c4ee696f92ea39b1f92465e758d344f0ac6262061c21dea33432d81e: not found

history ls shows the same build ref twice, once per node. That means export is trying to bundle per-node records for a multi-node build, and one node is advertising a trace digest that its content store still cannot serve or smth around that?

@crazy-max crazy-max force-pushed the history-fix-finalize branch from bfa30ad to 7b2fc84 Compare March 11, 2026 10:28
@crazy-max crazy-max changed the title history: always finalize selected records before export history: fix export fallback for finalized multi-node kubernetes builds Mar 11, 2026
@crazy-max crazy-max force-pushed the history-fix-finalize branch from 7b2fc84 to c08d761 Compare March 11, 2026 10:42
@crazy-max crazy-max changed the title history: fix export fallback for finalized multi-node kubernetes builds history: fix export fallback for finalized multi-node builds Mar 11, 2026
@crazy-max crazy-max force-pushed the history-fix-finalize branch from c08d761 to 546ef34 Compare March 11, 2026 10:53
@crazy-max crazy-max requested a review from tonistiigi March 11, 2026 10:58
@crazy-max crazy-max force-pushed the history-fix-finalize branch 2 times, most recently from b8b84bf to a3f74ef Compare March 11, 2026 11:39
@crazy-max crazy-max marked this pull request as ready for review March 11, 2026 11:43
@crazy-max crazy-max added this to the v0.33.0 milestone Mar 11, 2026
@crazy-max crazy-max force-pushed the history-fix-finalize branch 2 times, most recently from 9bf67d2 to ba2f865 Compare March 11, 2026 17:30
@crazy-max crazy-max changed the title history: fix export fallback for finalized multi-node builds history: finalize export records on their owning node Mar 11, 2026
@crazy-max crazy-max force-pushed the history-fix-finalize branch 2 times, most recently from 51f5b32 to 3a4686a Compare March 11, 2026 17:54
@crazy-max crazy-max marked this pull request as draft March 11, 2026 17:58
@crazy-max crazy-max force-pushed the history-fix-finalize branch 2 times, most recently from 31ed9f1 to be4a2c2 Compare March 12, 2026 08:30
@crazy-max
Copy link
Member Author

crazy-max commented Mar 12, 2026

Tried to add integration tests but can't make it works with docker runtime atm: https://github.com/docker/buildx/actions/runs/22994422408/job/66762935425#step:7:2418

Will work on multi-nodes support in our integration tests as follow-up: crazy-max@35166b9

Edit: opened #3724 and rebased on top of it

@crazy-max crazy-max marked this pull request as ready for review March 12, 2026 09:25
@crazy-max crazy-max force-pushed the history-fix-finalize branch from 0c6e103 to 6d100df Compare March 12, 2026 10:49
@crazy-max crazy-max marked this pull request as draft March 12, 2026 10:49
@crazy-max crazy-max force-pushed the history-fix-finalize branch 9 times, most recently from 33926f5 to 3675c74 Compare March 12, 2026 12:52
@crazy-max crazy-max marked this pull request as ready for review March 12, 2026 13:29
@crazy-max crazy-max force-pushed the history-fix-finalize branch 3 times, most recently from 5095d06 to e0b8929 Compare March 12, 2026 18:38
@crazy-max crazy-max force-pushed the history-fix-finalize branch from e0b8929 to 8b1bba8 Compare March 13, 2026 08:45
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
@crazy-max crazy-max force-pushed the history-fix-finalize branch from 8b1bba8 to 3fc8c06 Compare March 13, 2026 08:46
@github-actions github-actions bot removed the area/ci label Mar 13, 2026
crazy-max and others added 3 commits March 13, 2026 09:51
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Post Build Step hanging since v6.17.0 with kubernetes driver

2 participants