[Issue] Massive Memory Leak / OOM Crash on initial Trakt Import (~1100 items)

[drgoyo]

Describe the bug

Critical memory leak / OOM (Out of Memory) crash during the initial Trakt library synchronization. The container consumed all available host RAM (48 GB), leading to a kernel panic/OOM-kill of essential host processes (Unraid WebGUI/php-fpm). This occurred on the latest Unraid 7 release.

To Reproduce

1. Install Riven using the :dev tag.
2. Configure a Trakt account with a large watchlist/collection (approx. 1100 items).
3. Start the initial synchronization/scrape process.
4. Monitor RAM usage: It spikes uncontrollably until system saturation.

Expected behavior

Memory usage should remain stable during metadata processing. Large imports should be batched or memory should be cleared using aggressive garbage collection to stay within reasonable limits (e.g., 2-4 GB).

Environment

• OS: Unraid 7.2.3 (Kernel: 6.12.54-Unraid)
• CPU: AMD Ryzen 5 PRO 4650G
• RAM: 48 GB DDR4
• Installation Type: Docker Compose
• Image: spoked/riven:dev (Pulled Feb 15, 2026)

Riven Configuration (Environment Variables)

• RIVEN_FILESYSTEM_CACHE_MAX_SIZE_MB=10240
• RIVEN_FILESYSTEM_CHUNK_SIZE_MB=16
• RIVEN_FILESYSTEM_FETCH_AHEAD_CHUNKS=10

Logs (Host OOM-Killer evidence from Unraid 7.2.3)

Feb 15 12:12:43 Tower php-fpm[14355]: [WARNING] [pool www] child 221973 exited on signal 9 (SIGKILL) after 12.973920 seconds from start
Feb 15 12:12:56 Tower php-fpm[14355]: [WARNING] [pool www] child 222249 exited on signal 9 (SIGKILL) after 12.992889 seconds from start
Feb 15 12:13:09 Tower php-fpm[14355]: [WARNING] [pool www] child 222258 exited on signal 9 (SIGKILL) after 13.290010 seconds from start

[drgoyo]

Context

For context, my library has about 42,000 items, and at the time of the crash, roughly 16,661 items were sitting in the Indexed state.

The Symptoms

The OS OOM killer is terminating the riven-backend process because it hits roughly 20GB+ RES / 54GB VIRT memory (Memory cgroup out of memory: Killed process...). Right before this happens, the APScheduler completely freezes for over 40 minutes, flooding the logs with:
Run time of job "EventManager.submit_job..." was missed by 0:41:59

Ruling Out Other Causes

I ran some diagnostics inside the container during the freeze.

• It is not a network socket leak (netstat showed only 6 ESTABLISHED and 0 CLOSE_WAIT sockets).
• It is not a thread leak (stable at 9 active threads).
• There are no hanging Postgres queries or deadlocks.

Assumed Causes

1. SQLAlchemy Identity Map Explosion: When the scheduler or EventManager queries the database to process the backlog of pending items (those 16k+ Indexed episodes), it seems to load the entire result set into memory at once without chunking. Loading thousands of complex nested ORM objects (Season -> Episode -> MediaItem) explodes the SQLAlchemy Identity Map. This massive synchronous load blocks the Python GIL and the single-threaded Trio event loop for 40+ minutes. This starves the APScheduler and balloons the RAM until the OS kills the container.
2. Unmanaged File Cache: I also checked /riven/cache and found 12,484 files consuming 8.0GB. It looks like the cache mechanism is missing a TTL/LRU eviction policy, causing a secondary storage/inode leak that adds to the overall system pressure.

Proposal to Fix (Upcoming Release)

To fix this, the queries feeding the event loop need to be paginated:

• Use .yield_per(500) combined with .execution_options(stream_results=True) for these batch queries.
• Ensure session.expunge_all() is called after a batch is submitted to the worker pool to clear the Identity Map and allow the GC to free the memory.
• Add a hard limit or TTL cleanup for /riven/cache.

Workaround

I will try to run a workaround:
Use SQL to move all 16k Indexed items to Unreleased (to hide them from the active queue), clear the cache directory, and restart. Then, feed them back into the queue in manageable chunks:

docker exec -it riven-db psql -U postgres -d riven -c "UPDATE \"MediaItem\" SET last_state = 'Indexed' WHERE id IN (SELECT id FROM \"MediaItem\" WHERE last_state = 'Unreleased' LIMIT 500);"