Skip to content

[release-1.13] reland the scheduler task and backport the interrupt fixes and scheduler optimization - #62665

Merged
KristofferC merged 16 commits into
JuliaLang:backports-release-1.13from
IanButterworth:ib/interrupt-backports-1.13
Aug 12, 2026
Merged

[release-1.13] reland the scheduler task and backport the interrupt fixes and scheduler optimization#62665
KristofferC merged 16 commits into
JuliaLang:backports-release-1.13from
IanButterworth:ib/interrupt-backports-1.13

Conversation

@IanButterworth

@IanButterworth IanButterworth commented Aug 8, 2026

Copy link
Copy Markdown
Member

Since Keno's cancellation overhaul (#60281 and follow-ups) is not going to be backported to 1.13, this PR instead brings 1.13's interrupt handling to the quite healthy state (with new interrupt tests passing robustly) that master reached just before that overhaul landed.

Claude:


Reland (revert of #61065)

Reverts the four commits of #61065, restoring the scheduler-task work that was reverted on this branch while Ctrl-C delivery was broken (the breakage is fixed by the backports below):

Backports

In master order:

The last two complete the Windows profiler-deadlock fix chain (#60056 is already on this branch): with the scheduler task relanded, the Profile debuginfo-registration test deadlocks on Windows CI exactly as it did on master before these fixes (#60042, #60306).

#62372 was also considered but is already on this branch via an earlier backport.

Backport adaptations

IanButterworth and others added 16 commits August 8, 2026 10:46
An attempt at fixing interrupt handling, also some specific interrupt
hardening from manual tests, and adds interrupt tests.

Fixes JuliaLang#58689
Closes JuliaLang#58849

Developed with Claude Fable 5:

----

Since JuliaLang#57544, an idle thread parks in a per-thread internal scheduler
task. A SIGINT is always delivered to thread 1, which is almost always
parked when the signal arrives, so the resulting `InterruptException`
landed in the scheduler task's `wait_forever`, where it was reported as
a confusing `Internal Task ERROR: InterruptException` and then dropped.
As a result Ctrl-C no longer reached user code:

- scripts blocked in `sleep`/IO could not be interrupted at all,
- the REPL printed internal task errors on every Ctrl-C,
- `Distributed.interrupt` (which just sends SIGINT to the worker
process) became a silent no-op, breaking remote interrupts in
Distributed, Malt.jl/Pluto, and IJulia.

1.12 and 1.13 worked around this by reverting JuliaLang#57544 and its follow-ups;
master still had the scheduler task and the broken behavior.

**`base/task.jl`** — Each thread now remembers the last user task that
yielded into the scheduler while going idle (never recording a completed
task, so this does not delay collection of done tasks — the problem
scheduler task, it is re-thrown into a task that can meaningfully
observe it instead of being dropped: the REPL backend if it is
evaluating user code; silently dropped at an idle REPL prompt; otherwise
the last idle task, falling back to the root task. Expected failures of
the redirect (the victim raced to be rescheduled, or a second interrupt
arrived mid-switch) drop the interrupt; anything unexpected is still
reported. Delivery remains best-effort, as it always has been; robust
cancellation is left to JuliaLang#60281.

**`src/gf.c`** — Interrupting a process that is compiling (e.g. Ctrl-C
during `Pkg.test`) frequently threw the `InterruptException` into type
inference via the safepoint, unwinding the compiler mid-flight
("`Internal error: during type inference of ...`", an abort in assertion
builds, and a lost interrupt). The inference entry point is now
signal-atomic, so the interrupt is deferred and rethrown once compiler
state is consistent. Forced interrupts (repeated Ctrl-C) bypass the
deferral as before.

**`stdlib/REPL/src/REPL.jl`** — An interrupt forwarded to the REPL
backend just as user code finished evaluating (the forwarder checks
`in_eval`, but eval can complete before the throw lands) was raised at
`take!(backend.repl_channel)` and tore down the whole REPL session. The
backend loop now ignores a stray `InterruptException` there and keeps
serving.

New regression tests in `test/misc.jl` (pty-driven REPL + subprocess
scenarios, Unix-only) and `stdlib/REPL/test/repl.jl`, all derived from
the issue reports:

| Scenario | 1.11 | master before | master after |
|---|---|---|---|
| SIGINT at idle REPL prompt | ok | internal-error noise | ok |
| SIGINT during REPL `sleep` loop | ok | noise + interrupt works | ok |
| SIGINT to `julia -e 'sleep(600)'` | exits after 2nd SIGINT | never
exits | exits cleanly on 1st |
| `Distributed.interrupt` of a busy worker | `RemoteException` | silent
no-op | `RemoteException` |
| SIGINT during `Pkg.test`-style run (compiling) | — | inference
internal error / abort | clean `InterruptException` |

The fix is platform-independent (the Windows delivery path in
`signals-win.c` lands in the same scheduler task), but the tests are
Unix-only since sending a console Ctrl-C from the test harness on
Windows requires `GenerateConsoleCtrlEvent`/`CREATE_NEW_PROCESS_GROUP`,
which the spawn API doesn't expose.

This also adds the Unix portion of the CI coverage requested in JuliaLang#58849,
and likely fixes the crash class in JuliaLang#50045 (unverified, needs network).

Fixes JuliaLang#58689
Fixes JuliaLang#29369
Fixes JuliaLang#43451
Closes JuliaLang#58849
Fixes JuliaLang#50045

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit fc7ca1b)
…61826)

Fixes JuliaLang#61820
Fixes JuliaLang#50425

Linux - Ryzen 9 5950X
<img width="1560" height="720" alt="image"
src="https://github.com/user-attachments/assets/e3f667fb-6fed-46ed-837f-dfd1b8dd925a"
/>

Linux - Ryzen Threadripper PRO 7995WX 96-Cores
<img width="1560" height="720" alt="image"
src="https://github.com/user-attachments/assets/359e75b8-1a7e-4596-99a7-e7e5878f1b4d"
/>

Windows - i7-8700
<img width="1560" height="720" alt="image"
src="https://github.com/user-attachments/assets/1f9f4671-3e9a-45a7-b196-04dd779ed2f3"
/>

macOS - M2 Pro 6 p cores
<img width="1560" height="720" alt="image"
src="https://github.com/user-attachments/assets/0a5f7ea3-d443-4b23-a42b-b4ab3ee9940b"
/>

Developed with Claude:

---

`schedule` for a non-sticky task previously broadcast a wake to every
thread via
`jl_wakeup_thread(-1)`, performing a per-thread lock/signal/unlock under
`wakeup_thread`'s loop. Per-insert cost was linear in `jl_n_threads`,
and on
systems where the producer can be preempted (e.g. SMT + oversubscribed
thread
count on Windows/Linux) every iteration hit the kernel park/unpark path,
producing the >100x slowdown reported in JuliaLang#61820.

Add `jl_wakeup_threadpool(tpid)`, which wakes at most one sleeping
thread in the
target pool, with a round-robin start hint to spread wake load. Workers
re-check
the queue before sleeping (the existing store-buffering dance), so
bursty inserts
naturally wake additional consumers across the per-insert calls without
a
broadcast.

Restricting wakes to the task's own threadpool is also a correctness
improvement,
since `Partr.multiq_deletemin` only ever returns tasks from the caller's
pool --
waking out-of-pool threads was pure overhead.

The round-robin start hint is sharded across 64 cache-padded stripes
indexed by
the producing thread's tid. A single global atomic counter became the
dominant
cost of `@spawn` at high producer counts on multi-die parts (e.g. Ryzen
5950X
dual-CCD); striping eliminates the cross-CCD ping-pong without changing
the
round-robin semantics.

`enq_work` already routes the three "wake one specific thread" cases to
a
single-target `jl_wakeup_thread(tid)` rather than a broadcast; this PR
only
changes the remaining multiqueue-insert path:

| Case | Wake call |
| --- | --- |
| Sticky task | `jl_wakeup_thread(tid)` — single specific thread
(unchanged) |
| `:foreign` threadpool | `jl_wakeup_thread(tid)` — single specific
thread (unchanged) |
| Single-thread pool (e.g. `-t N,1` interactive) |
`jl_wakeup_thread(tid)` — single specific thread (unchanged) |
| Multiqueue insert | `jl_wakeup_threadpool(tpid)` — one thread in pool
(**new**) |

The `tid == -1` broadcast branch inside `wakeup_thread` is no longer
reached
from `enq_work` and has no in-tree callers, but is retained for external
`JL_DLLEXPORT`'d `jl_wakeup_thread(-1)` users.

Co-authored-by: GitHub Copilot with Claude

---------

Co-authored-by: Claude <claude@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
(cherry picked from commit a8f97b1)
…ang#62318)

When multiq_deletemin finds that the highest-priority task in the chosen
heap is sticky to a different thread, it cannot claim the task and moves
on to look for other work. If the thread that task is pinned to is
asleep, nothing wakes it, so the task can stall until some other event
happens to rouse that thread.

Before continuing the scan, read the task's owning thread id and wake it
so it can come pick up its own work.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit a6d8ef9)
…e, so interrupts reach user code (JuliaLang#62298)

(cherry picked from commit c9566b8)
…ps (JuliaLang#62385)

Followup to fix flaky tests from JuliaLang#62069

Re. JuliaLang#62069 (comment)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit d4f184f)
…kport

On master, JuliaLang#62069 made `jl_type_infer` signal-atomic
because JuliaLang#61255 had removed `jl_typeinf_lock`. release-1.13
still holds `jl_typeinf_lock` across inference (the JuliaLang#60689
backport), and taking that lock already defers asynchronous signals for
its duration, so the extra guard is redundant here. Keeping 1.13's
existing lock-based deferral avoids diverging from the branch's behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uliaLang#62389)

A non-sticky task whose threadpool has no threads falls through
enq_work's single-thread check into Partr.multiq_insert. There
multiq_size finds length(heaps) == 0 and, computing
heap_c * nt <= heap_p with nt == 0, returns 0 without ever sizing the
heaps. cong(0) then yields index 1 (formerly `seed % 0`, which is
undefined behavior: SIGFPE on x86, converted into a spurious
DivideError; a silent garbage index on architectures like PowerPC
where integer division by zero does not trap), and indexing the empty
heap vector throws:

    fatal: error thrown and no exception handler available.
    BoundsError(a=Array{Base.Partr.taskheap, (0,)}[], i=(357873774,))

This is reachable during every sysimage build: _finish_julia_init
moves the bootstrap thread into the interactive pool, leaving the
default pool with zero threads, and the Downloads stdlib's precompile
workload (download("file://" * @__FILE__)) spawns an errormonitor
task into the default pool from Curl's socket_callback. On x86 the
resulting DivideError happens to be swallowed, so the task is silently
lost and the build proceeds; on FreeBSD/powerpc64le the garbage index
aborts the sysimage build. Found while porting Julia to
FreeBSD/powerpc64le.

Fix it in two places:

- enq_work: if the task's pool has no threads, leave the task
  unqueued. This makes the previously accidental behavior explicit
  and deterministic, and keeps unrunnable tasks from being reachable
  from Partr.heaps during sysimage serialization (a task queued there
  while bootstrapping ends up serialized into the image and aborts
  the subsequent load).

- multiq_size: treat nt == 0 as 1 at both reads so an insert into an
  empty pool can never index unsized heaps, should it be reached by
  another path (e.g. jl_set_task_threadpoolid users).

Co-authored-by: Jameson Nash <vtjnash@gmail.com>
(cherry picked from commit e4a1b6e)
…g#60463)

On x86-64 Windows, we're unfortunately pretty reliant on this timer to
avoid (otherwise unavoidable) deadlocks between `RtlLookupFunctionEntry`
and many other internal functions in the Windows runtime (incl.
`RtlAllocateHeap` and `LdrLoadDll`).

For unknown reasons, `RegisterWaitForSingleObject` seems to be quite bad
about dropping the callback on the floor in contended situations (we
deadlock but the timer callback is simply never called). This
alternative API appears to be better-behaved.

This does not solve the deadlock identified by @xal-0 in
JuliaLang#60454 (comment)
(or several others)

(cherry picked from commit 8a7223e)
This adds a queue so that the DLL processing can be deferred to a
profiling thread if the `jl_in_stackwalk` lock is contended.

The problem was: It is possible for a thread to be suspended for
profiling in the middle of a DLL load / unload event, in which case our
watchdog may wake the thread up and it will finish loading the DLL.
However, the load will immediately try to load symbols for the DLL and
hit a deadlock trying to obtain the `jl_in_stackwalk` lock still held by
the profiling thread.

See
JuliaLang#60463 (comment)
for more information.

Locally this resolves the rest of
JuliaLang#60306 for me, at least for the
MWE I have.

(cherry picked from commit 7eb1edb)
@IanButterworth
IanButterworth marked this pull request as ready for review August 9, 2026 14:03
@KristofferC KristofferC mentioned this pull request Aug 11, 2026
44 tasks
@IanButterworth
IanButterworth changed the base branch from release-1.13 to backports-release-1.13 August 11, 2026 16:23
@IanButterworth IanButterworth added the don't squash Don't squash merge label Aug 11, 2026
@KristofferC
KristofferC merged commit 9064067 into JuliaLang:backports-release-1.13 Aug 12, 2026
10 checks passed
@IanButterworth
IanButterworth deleted the ib/interrupt-backports-1.13 branch August 12, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

don't squash Don't squash merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants