[release-1.13] reland the scheduler task and backport the interrupt fixes and scheduler optimization - #62665
Merged
KristofferC merged 16 commits intoAug 12, 2026
Conversation
… with a OncePerThread" This reverts commit aaeedb1.
This reverts commit cd8c9ef.
This reverts commit aabfe94.
An attempt at fixing interrupt handling, also some specific interrupt hardening from manual tests, and adds interrupt tests. Fixes JuliaLang#58689 Closes JuliaLang#58849 Developed with Claude Fable 5: ---- Since JuliaLang#57544, an idle thread parks in a per-thread internal scheduler task. A SIGINT is always delivered to thread 1, which is almost always parked when the signal arrives, so the resulting `InterruptException` landed in the scheduler task's `wait_forever`, where it was reported as a confusing `Internal Task ERROR: InterruptException` and then dropped. As a result Ctrl-C no longer reached user code: - scripts blocked in `sleep`/IO could not be interrupted at all, - the REPL printed internal task errors on every Ctrl-C, - `Distributed.interrupt` (which just sends SIGINT to the worker process) became a silent no-op, breaking remote interrupts in Distributed, Malt.jl/Pluto, and IJulia. 1.12 and 1.13 worked around this by reverting JuliaLang#57544 and its follow-ups; master still had the scheduler task and the broken behavior. **`base/task.jl`** — Each thread now remembers the last user task that yielded into the scheduler while going idle (never recording a completed task, so this does not delay collection of done tasks — the problem scheduler task, it is re-thrown into a task that can meaningfully observe it instead of being dropped: the REPL backend if it is evaluating user code; silently dropped at an idle REPL prompt; otherwise the last idle task, falling back to the root task. Expected failures of the redirect (the victim raced to be rescheduled, or a second interrupt arrived mid-switch) drop the interrupt; anything unexpected is still reported. Delivery remains best-effort, as it always has been; robust cancellation is left to JuliaLang#60281. **`src/gf.c`** — Interrupting a process that is compiling (e.g. Ctrl-C during `Pkg.test`) frequently threw the `InterruptException` into type inference via the safepoint, unwinding the compiler mid-flight ("`Internal error: during type inference of ...`", an abort in assertion builds, and a lost interrupt). The inference entry point is now signal-atomic, so the interrupt is deferred and rethrown once compiler state is consistent. Forced interrupts (repeated Ctrl-C) bypass the deferral as before. **`stdlib/REPL/src/REPL.jl`** — An interrupt forwarded to the REPL backend just as user code finished evaluating (the forwarder checks `in_eval`, but eval can complete before the throw lands) was raised at `take!(backend.repl_channel)` and tore down the whole REPL session. The backend loop now ignores a stray `InterruptException` there and keeps serving. New regression tests in `test/misc.jl` (pty-driven REPL + subprocess scenarios, Unix-only) and `stdlib/REPL/test/repl.jl`, all derived from the issue reports: | Scenario | 1.11 | master before | master after | |---|---|---|---| | SIGINT at idle REPL prompt | ok | internal-error noise | ok | | SIGINT during REPL `sleep` loop | ok | noise + interrupt works | ok | | SIGINT to `julia -e 'sleep(600)'` | exits after 2nd SIGINT | never exits | exits cleanly on 1st | | `Distributed.interrupt` of a busy worker | `RemoteException` | silent no-op | `RemoteException` | | SIGINT during `Pkg.test`-style run (compiling) | — | inference internal error / abort | clean `InterruptException` | The fix is platform-independent (the Windows delivery path in `signals-win.c` lands in the same scheduler task), but the tests are Unix-only since sending a console Ctrl-C from the test harness on Windows requires `GenerateConsoleCtrlEvent`/`CREATE_NEW_PROCESS_GROUP`, which the spawn API doesn't expose. This also adds the Unix portion of the CI coverage requested in JuliaLang#58849, and likely fixes the crash class in JuliaLang#50045 (unverified, needs network). Fixes JuliaLang#58689 Fixes JuliaLang#29369 Fixes JuliaLang#43451 Closes JuliaLang#58849 Fixes JuliaLang#50045 --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit fc7ca1b)
…61826) Fixes JuliaLang#61820 Fixes JuliaLang#50425 Linux - Ryzen 9 5950X <img width="1560" height="720" alt="image" src="https://github.com/user-attachments/assets/e3f667fb-6fed-46ed-837f-dfd1b8dd925a" /> Linux - Ryzen Threadripper PRO 7995WX 96-Cores <img width="1560" height="720" alt="image" src="https://github.com/user-attachments/assets/359e75b8-1a7e-4596-99a7-e7e5878f1b4d" /> Windows - i7-8700 <img width="1560" height="720" alt="image" src="https://github.com/user-attachments/assets/1f9f4671-3e9a-45a7-b196-04dd779ed2f3" /> macOS - M2 Pro 6 p cores <img width="1560" height="720" alt="image" src="https://github.com/user-attachments/assets/0a5f7ea3-d443-4b23-a42b-b4ab3ee9940b" /> Developed with Claude: --- `schedule` for a non-sticky task previously broadcast a wake to every thread via `jl_wakeup_thread(-1)`, performing a per-thread lock/signal/unlock under `wakeup_thread`'s loop. Per-insert cost was linear in `jl_n_threads`, and on systems where the producer can be preempted (e.g. SMT + oversubscribed thread count on Windows/Linux) every iteration hit the kernel park/unpark path, producing the >100x slowdown reported in JuliaLang#61820. Add `jl_wakeup_threadpool(tpid)`, which wakes at most one sleeping thread in the target pool, with a round-robin start hint to spread wake load. Workers re-check the queue before sleeping (the existing store-buffering dance), so bursty inserts naturally wake additional consumers across the per-insert calls without a broadcast. Restricting wakes to the task's own threadpool is also a correctness improvement, since `Partr.multiq_deletemin` only ever returns tasks from the caller's pool -- waking out-of-pool threads was pure overhead. The round-robin start hint is sharded across 64 cache-padded stripes indexed by the producing thread's tid. A single global atomic counter became the dominant cost of `@spawn` at high producer counts on multi-die parts (e.g. Ryzen 5950X dual-CCD); striping eliminates the cross-CCD ping-pong without changing the round-robin semantics. `enq_work` already routes the three "wake one specific thread" cases to a single-target `jl_wakeup_thread(tid)` rather than a broadcast; this PR only changes the remaining multiqueue-insert path: | Case | Wake call | | --- | --- | | Sticky task | `jl_wakeup_thread(tid)` — single specific thread (unchanged) | | `:foreign` threadpool | `jl_wakeup_thread(tid)` — single specific thread (unchanged) | | Single-thread pool (e.g. `-t N,1` interactive) | `jl_wakeup_thread(tid)` — single specific thread (unchanged) | | Multiqueue insert | `jl_wakeup_threadpool(tpid)` — one thread in pool (**new**) | The `tid == -1` broadcast branch inside `wakeup_thread` is no longer reached from `enq_work` and has no in-tree callers, but is retained for external `JL_DLLEXPORT`'d `jl_wakeup_thread(-1)` users. Co-authored-by: GitHub Copilot with Claude --------- Co-authored-by: Claude <claude@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> (cherry picked from commit a8f97b1)
…ang#62318) When multiq_deletemin finds that the highest-priority task in the chosen heap is sticky to a different thread, it cannot claim the task and moves on to look for other work. If the thread that task is pinned to is asleep, nothing wakes it, so the task can stall until some other event happens to rouse that thread. Before continuing the scan, read the task's owning thread id and wake it so it can come pick up its own work. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit a6d8ef9)
…e, so interrupts reach user code (JuliaLang#62298) (cherry picked from commit c9566b8)
…a no-op (JuliaLang#62371) (cherry picked from commit cd463b8)
…ps (JuliaLang#62385) Followup to fix flaky tests from JuliaLang#62069 Re. JuliaLang#62069 (comment) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit d4f184f)
See JuliaLang#62069 (comment) (cherry picked from commit 2c7d2b1)
…liaLang#62471) (cherry picked from commit d4ccf40)
…kport On master, JuliaLang#62069 made `jl_type_infer` signal-atomic because JuliaLang#61255 had removed `jl_typeinf_lock`. release-1.13 still holds `jl_typeinf_lock` across inference (the JuliaLang#60689 backport), and taking that lock already defers asynchronous signals for its duration, so the extra guard is redundant here. Keeping 1.13's existing lock-based deferral avoids diverging from the branch's behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uliaLang#62389) A non-sticky task whose threadpool has no threads falls through enq_work's single-thread check into Partr.multiq_insert. There multiq_size finds length(heaps) == 0 and, computing heap_c * nt <= heap_p with nt == 0, returns 0 without ever sizing the heaps. cong(0) then yields index 1 (formerly `seed % 0`, which is undefined behavior: SIGFPE on x86, converted into a spurious DivideError; a silent garbage index on architectures like PowerPC where integer division by zero does not trap), and indexing the empty heap vector throws: fatal: error thrown and no exception handler available. BoundsError(a=Array{Base.Partr.taskheap, (0,)}[], i=(357873774,)) This is reachable during every sysimage build: _finish_julia_init moves the bootstrap thread into the interactive pool, leaving the default pool with zero threads, and the Downloads stdlib's precompile workload (download("file://" * @__FILE__)) spawns an errormonitor task into the default pool from Curl's socket_callback. On x86 the resulting DivideError happens to be swallowed, so the task is silently lost and the build proceeds; on FreeBSD/powerpc64le the garbage index aborts the sysimage build. Found while porting Julia to FreeBSD/powerpc64le. Fix it in two places: - enq_work: if the task's pool has no threads, leave the task unqueued. This makes the previously accidental behavior explicit and deterministic, and keeps unrunnable tasks from being reachable from Partr.heaps during sysimage serialization (a task queued there while bootstrapping ends up serialized into the image and aborts the subsequent load). - multiq_size: treat nt == 0 as 1 at both reads so an insert into an empty pool can never index unsized heaps, should it be reached by another path (e.g. jl_set_task_threadpoolid users). Co-authored-by: Jameson Nash <vtjnash@gmail.com> (cherry picked from commit e4a1b6e)
…g#60463) On x86-64 Windows, we're unfortunately pretty reliant on this timer to avoid (otherwise unavoidable) deadlocks between `RtlLookupFunctionEntry` and many other internal functions in the Windows runtime (incl. `RtlAllocateHeap` and `LdrLoadDll`). For unknown reasons, `RegisterWaitForSingleObject` seems to be quite bad about dropping the callback on the floor in contended situations (we deadlock but the timer callback is simply never called). This alternative API appears to be better-behaved. This does not solve the deadlock identified by @xal-0 in JuliaLang#60454 (comment) (or several others) (cherry picked from commit 8a7223e)
This adds a queue so that the DLL processing can be deferred to a profiling thread if the `jl_in_stackwalk` lock is contended. The problem was: It is possible for a thread to be suspended for profiling in the middle of a DLL load / unload event, in which case our watchdog may wake the thread up and it will finish loading the DLL. However, the load will immediately try to load symbols for the DLL and hit a deadlock trying to obtain the `jl_in_stackwalk` lock still held by the profiling thread. See JuliaLang#60463 (comment) for more information. Locally this resolves the rest of JuliaLang#60306 for me, at least for the MWE I have. (cherry picked from commit 7eb1edb)
IanButterworth
marked this pull request as ready for review
August 9, 2026 14:03
44 tasks
IanButterworth
changed the base branch from
release-1.13
to
backports-release-1.13
August 11, 2026 16:23
KristofferC
merged commit Aug 12, 2026
9064067
into
JuliaLang:backports-release-1.13
10 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since Keno's cancellation overhaul (#60281 and follow-ups) is not going to be backported to 1.13, this PR instead brings 1.13's interrupt handling to the quite healthy state (with new interrupt tests passing robustly) that master reached just before that overhaul landed.
Claude:
Reland (revert of #61065)
Reverts the four commits of #61065, restoring the scheduler-task work that was reverted on this branch while Ctrl-C delivery was broken (the breakage is fixed by the backports below):
Scheduler: Use a "scheduler" task for thread sleep(Scheduler: Use a "scheduler" task for thread sleep #57544)simplify wait()(simplifywait()#58595)make sched_task more robust(make sched_task more robust #59669)Backports
In master order:
@spawn#61826 — scheduler: avoid O(nthreads) wake-storm on every@spawn(already labeledbackport 1.13; tasks: only switch to the scheduler task when the current task is done, so interrupts reach user code #62298/scheduler: fall back to a pool wake when the targeted thread wake is a no-op #62371 build on it)enq_workpath modified by tasks: only switch to the scheduler task when the current task is done, so interrupts reach user code #62298/scheduler: fall back to a pool wake when the targeted thread wake is a no-op #62371)TimerQueueAPI to register Profile watchdog #60463 — [Windows] UseTimerQueueAPI to register Profile watchdogThe last two complete the Windows profiler-deadlock fix chain (#60056 is already on this branch): with the scheduler task relanded, the Profile debuginfo-registration test deadlocks on Windows CI exactly as it did on master before these fixes (#60042, #60306).
#62372 was also considered but is already on this branch via an earlier backport.
Backport adaptations
jl_type_inferbecause Add function to emit multiple CodeInstances to the JIT atomically (remove jl_typeinf_lock) #61255 removedjl_typeinf_lock; 1.13 still takes that lock (the Temporarily reintroduce a global type inference lock #60689 backport), which already defers signals for its duration, so the guard is dropped in a separate commit on top of the faithful cherry-pick.multiq_deleteminwas restructured by the labeled-break syntax PR syntax: Add labeled block break #60481, which can't come to 1.13; the hunks are placed into 1.13's@goto retrystructure instead.mach_safepoint_trampoline(from [macOS] Handle GC safepoint on-thread #61341, not backported) doesn't exist on 1.13; theeh == NULLdeferral guard is applied to 1.13's equivalent decision point incatch_mach_exception_raise.src/llvm-julia-task-dispatcher.h(pre-Add function to emit multiple CodeInstances to the JIT atomically (remove jl_typeinf_lock) #61255 name/structure): samedispatcher_sigdefer_guard, placed around the inline-run/drain path andwork_until.parse_repl_input_linedoesn't exist on 1.13; the BasicREPL loop keepsBase.parse_input_line.jl_unw_initkeeps its(void)from_signal_handler;line (master had dropped the parameter); the Fix windows profiler deadlock #60056 test-enablement hunk reduces to nothing since master later re-skipped that stress test.