Skip to content

File race in download_url() when parallel threads download the same monorepo sdist #1142

@smoparth

Description

@smoparth

In build-parallel, when multiple sub-packages share the same monorepo sdist (e.g., MLServer runtimes like mlserver-sklearn, mlserver-lightgbm), they can land in the same topological level and all try to download the same tarball simultaneously. The first thread to finish renames the .tmp file, and the others fail with:

[Errno 2] No such file or directory: '.../downloads/mlserver-v1.7.1.tar.gz.tmp' -> '.../downloads/mlserver-v1.7.1.tar.gz'

Root cause
sources.py download_url() uses a shared temp filename (outfile + ".tmp") for all threads downloading the same file. Thread A downloads and renames it; Thread B's rename fails because the .tmp is already gone.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions