objcache: Try to make it less likely to hang - #62363
Conversation
We're already trying to do this, so I'll have to see why it isn't working as intended. I would think it's the write transaction we have to open in order to create the databases if they don't exist (GPT's commit messages also suggest this but the changes seem way more complicated than I would think necessary). |
| State->CachePath = std::move(*CachePath); | ||
| State->Phase.store(ObjCachePhase::Initializing, memory_order_release); | ||
|
|
||
| #ifndef __clang_gcanalyzer__ |
There was a problem hiding this comment.
I'm going to start making a CI check which autofails every time GPT keeps inserting this hack instead of fixing its bugs
|
@xal-0 points out that LMDB 1.0 has some license headers that reference a dual use license that does not appear to be included. I have asked upstream to clarify in https://bugs.openldap.org/show_bug.cgi?id=10540, but we should hold this until we find out. |
|
Upstream confirms the license is vestigal. |
LMDB 1.0 adds robust-lock support for platforms without robust POSIX mutexes (SysV semaphores on Apple/BSD), so that locks held by a process that dies are recovered by the next process instead of wedging the cache forever. Select the robust lock backend explicitly with MDB_USE_ROBUST. LMDB 1.0's Windows remapping and incremental-copy code does not compile with MinGW (it treats LARGE_INTEGER as a scalar and passes pointer types that do not match the native APIs), so add a compatibility patch. Also make libjulia-codegen depend on the bundled archive so that it is relinked when the dependency is rebuilt. Upstream has confirmed that the vestigial dual-license header in the 1.0 sources does not apply: https://bugs.openldap.org/show_bug.cgi?id=10540 Co-authored-by: Codex <codex@openai.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The objcache's LMDB environment contains locks that are shared with other julia processes, so a misbehaving (or suspended, or dead) process could previously wedge an innocent one, in several ways: - Opening the environment and creating the databases takes the write lock and ran on the first thread to call ObjCache::get, blocking compilation behind whatever process held it. All initialization now happens on the writer thread, and the databases are opened read-only when they already exist, so a wedged writer elsewhere does not even prevent cache hits. - Shutdown joined the writer thread, so a writer stuck on the write lock prevented exit. The writer thread is now detached and shares ownership of the cache state; it exits on its own once it observes shutdown. The write queue is bounded, dropping writes when the writer cannot keep up. - Each lookup took LMDB's reader-table lock to reserve a reader slot. A single read transaction is now reserved up front (on the writer thread) and renewed/reset on the lookup path, which requires no locking. In addition, mdb_reader_check now runs during initialization to recover reader slots from dead processes; together with LMDB 1.0's robust locks this also recovers the write lock from processes that died holding it. Since initialization is now asynchronous, whether a module is compiled cache-portably is frozen per materialization (CompileFn receives a Cacheable flag), and the pipeline-level TLS lowering decision uses the process-wide isGloballyEnabled(), which unlike the previous OCache.isEnabled() is meaningful at JIT construction time. The cache directory moves to objcache-lmdb1 since LMDB 1.0 cannot open 0.9 data files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5238920 to
87401b5
Compare
|
1.0 bump is still needed for sysv semaphores, but the hangs I was observing themselves turn out to be a kernel bug. Closing this and will open a separate PR for the 1.0 bump and patches to be conservative when we know the objcache will hang. |
|
Upstream license cleanup in LMDB/lmdb@7834588 for reference. |
As mentioned in #62363 (comment), I had misidentified the reason for the hangs. Turns out there's a kernel limitation/bug that causes pthread_mutex to not provide mutual exclusion across pid namespaces. The LMDB 1.0 bump is still required to get robust cleanup on macos. To workaround the linux issue for now, patch LMDB to refuse opening if it's being used across pid namespaces or on a remote file system. Also provide a convenient notice in the REPL banner so that users that run into this (e.g. on HPC clusters) will know to move their cache somewhere else. --------- Co-authored-by: Keno Fischer <Keno@users.noreply.github.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
3 participants
I've been observing a fair number of julia processes get wedged because of a misbehaving earlier julia process. Now granted, I have a much higher fraction of misbheaving julia processes than the average user, but I still want to try to do better. This: