Verify indirect deps reachable on incremental run#20735
Open
ilevkivskyi wants to merge 2 commits intopython:masterfrom
Open
Verify indirect deps reachable on incremental run#20735ilevkivskyi wants to merge 2 commits intopython:masterfrom
ilevkivskyi wants to merge 2 commits intopython:masterfrom
Conversation
This comment has been minimized.
This comment has been minimized.
Contributor
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #19477
This is another surprising correctness hole in the indirect dependencies logic. We implicitly assume everywhere that indirect dependencies are subset of transitive ones. Although it is true when the indirect dependencies are initially calculated, there is no guarantee this will stay true after an incremental update where some import structure has changed.
Currently, we only handle the situations where an indirect dependency is removed from the graph completely, but if it is still in the graph, but now brought in by a different dependent, this may cause out-of-order cache loading, and thus a bug or a crash. It is interesting that although in theory it is kind of a big deal, in practice it is very hard to create such scenario. So far I found only two complicated scenarios (see tests).
Implementation note: although this additional check is needed for rare edge cases, it is quite computationally expensive. I tried few options (including only computing import deltas), but the only viable/robust option seems to be an honest recursive check with some caching. I tested this on
torch(it has ~1000 small SCCs plus single SCC with around 1000 modules), and the performance penalty is negligible there. So hopefully it should be fine even for large code bases.