partr: fix lost tasks when threads race to grow a pool's multiqueue - #62372
Merged
Conversation
`multiq_size` reads `tpheaps = heaps[tp]` before taking `heaps_lock[tp]`, and then re-checks `length(tpheaps)` on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning `:interactive` tasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it with `heaps[tp] = newheaps`, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable from `heaps[tp]`, so no thread can ever pop it, and anything waiting on it hangs. With `-t 8,4` and a burst of cross-pool spawn/fetch pairs this reproduces within seconds. Re-read `heaps[tp]` under the lock so the re-check and the `copyto!` operate on the current vector, making the grow-if-needed check correct and the publication monotonic. Fixes JuliaLang#62144 Co-Authored-By: Claude <claude@users.noreply.github.com>
IanButterworth
added
multithreading
KristofferC
pushed a commit
that referenced
this pull request
Jul 18, 2026
…62372) `multiq_size` reads `tpheaps = heaps[tp]` before taking `heaps_lock[tp]`, and then re-checks `length(tpheaps)` on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning `:interactive` tasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it with `heaps[tp] = newheaps`, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable from `heaps[tp]`, so no thread can ever pop it, and anything waiting on it hangs. With `-t 8,4` and a burst of cross-pool spawn/fetch pairs this reproduces within seconds. Re-read `heaps[tp]` under the lock so the re-check and the `copyto!` operate on the current vector, making the grow-if-needed check correct and the publication monotonic. Fixes #62144 Co-authored-by: Claude <claude@users.noreply.github.com> (cherry picked from commit 733410f)
KristofferC
pushed a commit
that referenced
this pull request
Jul 18, 2026
…62372) `multiq_size` reads `tpheaps = heaps[tp]` before taking `heaps_lock[tp]`, and then re-checks `length(tpheaps)` on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning `:interactive` tasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it with `heaps[tp] = newheaps`, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable from `heaps[tp]`, so no thread can ever pop it, and anything waiting on it hangs. With `-t 8,4` and a burst of cross-pool spawn/fetch pairs this reproduces within seconds. Re-read `heaps[tp]` under the lock so the re-check and the `copyto!` operate on the current vector, making the grow-if-needed check correct and the publication monotonic. Fixes #62144 Co-authored-by: Claude <claude@users.noreply.github.com> (cherry picked from commit 733410f)
KristofferC
removed
the
backport 1.13
KristofferC
pushed a commit
that referenced
this pull request
Jul 21, 2026
…62372) `multiq_size` reads `tpheaps = heaps[tp]` before taking `heaps_lock[tp]`, and then re-checks `length(tpheaps)` on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning `:interactive` tasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it with `heaps[tp] = newheaps`, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable from `heaps[tp]`, so no thread can ever pop it, and anything waiting on it hangs. With `-t 8,4` and a burst of cross-pool spawn/fetch pairs this reproduces within seconds. Re-read `heaps[tp]` under the lock so the re-check and the `copyto!` operate on the current vector, making the grow-if-needed check correct and the publication monotonic. Fixes #62144 Co-authored-by: Claude <claude@users.noreply.github.com> (cherry picked from commit 733410f)
KristofferC
pushed a commit
that referenced
this pull request
Jul 21, 2026
…62372) `multiq_size` reads `tpheaps = heaps[tp]` before taking `heaps_lock[tp]`, and then re-checks `length(tpheaps)` on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning `:interactive` tasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it with `heaps[tp] = newheaps`, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable from `heaps[tp]`, so no thread can ever pop it, and anything waiting on it hangs. With `-t 8,4` and a burst of cross-pool spawn/fetch pairs this reproduces within seconds. Re-read `heaps[tp]` under the lock so the re-check and the `copyto!` operate on the current vector, making the grow-if-needed check correct and the publication monotonic. Fixes #62144 Co-authored-by: Claude <claude@users.noreply.github.com> (cherry picked from commit 733410f)
KristofferC
pushed a commit
that referenced
this pull request
Jul 24, 2026
…62372) `multiq_size` reads `tpheaps = heaps[tp]` before taking `heaps_lock[tp]`, and then re-checks `length(tpheaps)` on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning `:interactive` tasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it with `heaps[tp] = newheaps`, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable from `heaps[tp]`, so no thread can ever pop it, and anything waiting on it hangs. With `-t 8,4` and a burst of cross-pool spawn/fetch pairs this reproduces within seconds. Re-read `heaps[tp]` under the lock so the re-check and the `copyto!` operate on the current vector, making the grow-if-needed check correct and the publication monotonic. Fixes #62144 Co-authored-by: Claude <claude@users.noreply.github.com> (cherry picked from commit 733410f)
DilumAluthge
pushed a commit
that referenced
this pull request
Jul 25, 2026
…62372) `multiq_size` reads `tpheaps = heaps[tp]` before taking `heaps_lock[tp]`, and then re-checks `length(tpheaps)` on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning `:interactive` tasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it with `heaps[tp] = newheaps`, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable from `heaps[tp]`, so no thread can ever pop it, and anything waiting on it hangs. With `-t 8,4` and a burst of cross-pool spawn/fetch pairs this reproduces within seconds. Re-read `heaps[tp]` under the lock so the re-check and the `copyto!` operate on the current vector, making the grow-if-needed check correct and the publication monotonic. Fixes #62144 Co-authored-by: Claude <claude@users.noreply.github.com> (cherry picked from commit 733410f)
KristofferC
pushed a commit
that referenced
this pull request
Aug 7, 2026
…62372) `multiq_size` reads `tpheaps = heaps[tp]` before taking `heaps_lock[tp]`, and then re-checks `length(tpheaps)` on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning `:interactive` tasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it with `heaps[tp] = newheaps`, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable from `heaps[tp]`, so no thread can ever pop it, and anything waiting on it hangs. With `-t 8,4` and a burst of cross-pool spawn/fetch pairs this reproduces within seconds. Re-read `heaps[tp]` under the lock so the re-check and the `copyto!` operate on the current vector, making the grow-if-needed check correct and the publication monotonic. Fixes #62144 Co-authored-by: Claude <claude@users.noreply.github.com> (cherry picked from commit 733410f)
KristofferC
pushed a commit
that referenced
this pull request
Aug 8, 2026
…62372) `multiq_size` reads `tpheaps = heaps[tp]` before taking `heaps_lock[tp]`, and then re-checks `length(tpheaps)` on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning `:interactive` tasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it with `heaps[tp] = newheaps`, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable from `heaps[tp]`, so no thread can ever pop it, and anything waiting on it hangs. With `-t 8,4` and a burst of cross-pool spawn/fetch pairs this reproduces within seconds. Re-read `heaps[tp]` under the lock so the re-check and the `copyto!` operate on the current vector, making the grow-if-needed check correct and the publication monotonic. Fixes #62144 Co-authored-by: Claude <claude@users.noreply.github.com> (cherry picked from commit 733410f)
KristofferC
pushed a commit
that referenced
this pull request
Aug 8, 2026
…62372) `multiq_size` reads `tpheaps = heaps[tp]` before taking `heaps_lock[tp]`, and then re-checks `length(tpheaps)` on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning `:interactive` tasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it with `heaps[tp] = newheaps`, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable from `heaps[tp]`, so no thread can ever pop it, and anything waiting on it hangs. With `-t 8,4` and a burst of cross-pool spawn/fetch pairs this reproduces within seconds. Re-read `heaps[tp]` under the lock so the re-check and the `copyto!` operate on the current vector, making the grow-if-needed check correct and the publication monotonic. Fixes #62144 Co-authored-by: Claude <claude@users.noreply.github.com> (cherry picked from commit 733410f)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #62144
Claude:
multiq_sizereadstpheaps = heaps[tp]before takingheaps_lock[tp], and then re-checkslength(tpheaps)on that stale local after acquiring the lock. When several threads concurrently perform the first inserts into a pool (e.g. many default-pool tasks spawning:interactivetasks at once), each of them sees the pre-growth length, so each one in turn allocates a fresh heaps vector and publishes it withheaps[tp] = newheaps, replacing the vector the previous thread just published. Any task inserted into one of the replaced vectors in that window is orphaned: it is unreachable fromheaps[tp], so no thread can ever pop it, and anything waiting on it hangs. With-t 8,4and a burst of cross-pool spawn/fetch pairs this reproduces within seconds.Re-read
heaps[tp]under the lock so the re-check and thecopyto!operate on the current vector, making the grow-if-needed check correct and the publication monotonic.