Memory-in-the-Loop fork local run results for across sessions memory references - has anyone tried this direction?

[habanwer]

Hi @karpathy and autoresearch community,

A month ago I opened a PR #302 that was a bit noisy (sorry again @svlandeg — I took your feedback seriously and closed it).

Since then I’ve cleaned everything up into a dedicated fork that focuses on one core idea - how to enable an agent with a way (or more) to enjoy a Memory-in-the-Loop (MIL) or references states - and reflecting on @karpathy's tweet on English as programming language.

In the latest run, the agent writes a compact sessions/memory.md after every run (best val_bpb ever + top kept experiments + last-run context). It reads this file at the start of the next experiment, so it remembers its own past lessons instead of rediscovering them from scratch - On an RTX 5000 with the standard 5-minute budget the agent quickly converged on a strong direction: drastically shrinking depth (down to 1) and maximizing tokens processed beats larger models. It improved the baseline val_bpb from 3.13 all the way down to 1.62 while staying comfortably inside time and VRAM limits. You can see the full experiment trace in the attached progress.png (and the raw results.tsv in the repo: https://github.com/habanwer/autoresearch-MIL)

The fork also adds clean JSON configs (ground.json + model.json), explicit file-ownership governance (user-owned vs agent-owned files), multi GPU platform detection (Volta → Blackwell + Windows support), and follows the DEITY Principles framework for transparent human-AI collaboration. Repo: https://github.com/habanwer/autoresearch-MIL

Has anyone else experimented with giving the agent this kind of explicit long-term memory across runs? Curious if this direction has obvious failure modes I haven’t hit yet, or if others have tried similar patterns and seen different trade-offs on their hardware.

Thanks again for the inspirational project — watching an agent build on their own findings has been genuinely informative and inspirational.

Best,
Shehab