Skip to content

CI: bump Build devel timeout, it times out on the free runner tier#38286

Open
JPL11 wants to merge 2 commits into
commaai:masterfrom
JPL11:build-devel-timeout
Open

CI: bump Build devel timeout, it times out on the free runner tier#38286
JPL11 wants to merge 2 commits into
commaai:masterfrom
JPL11:build-devel-timeout

Conversation

@JPL11

@JPL11 JPL11 commented Jul 4, 2026

Copy link
Copy Markdown

build release currently fails on every fork PR: the Build devel step hits its 1-minute timeout on the ubuntu-24.04 runners that fork PRs are routed to (verified twice on #38285: killed at 72s both times, deterministic). It copies the full tree including LFS models, which fits in a minute on the namespace runners master uses but no longer on the free tier — fork PRs from late June (#38242, #38258) passed the same step at 55-59s, so recent tree growth tipped it over.

3 minutes keeps the step bounded while giving the free tier ~2.5x headroom.

Copilot AI review requested due to automatic review settings July 4, 2026 05:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts CI to be more reliable on GitHub-hosted ubuntu-24.04 runners used for fork PRs by increasing the allowed time for the Build devel step, which now deterministically exceeds the previous 1-minute limit.

Changes:

  • Increase the Build devel step timeout from 1 minute to 3 minutes in the main CI workflow.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Process replay diff report

Replays driving segments through this PR and compares the behavior to master.
Please review any changes carefully to ensure they are expected.

✅ 0 changed, 66 passed, 0 errors

@adeebshihadeh

Copy link
Copy Markdown
Contributor

Why does this need more than a minute? Anything we can optimize?

@JPL11

JPL11 commented Jul 4, 2026

Copy link
Copy Markdown
Author

Profiled it on the free runner (phase timestamps are already in the script):

phase time
git setup + checkout devel ~1s
copy files (~2 GB, mostly big_driving_supercombo.onnx) ~7s
git add + commit ~64s

The commit is zlib-compressing ~2 GB of model chunks into loose objects on one slow core. Since object IDs are content-addressed before compression and this repo is a throwaway build artifact, core.compression 0 is free: 74s → 29s measured on ubuntu-24.04 (bench run). Pushed that here.

I also tried hardlinking the file copy (another ~9s off) but it had a flaky interaction I didn't fully chase down, so I left it out.

Kept the timeout at 3 min for slack against free-runner variance (they swing ±40%) — happy to drop it back to 1 if you'd rather, 29s fits.

@therepanic

Copy link
Copy Markdown
Contributor

I suppose it's not just the Build devel in tests/build release that's failing now. Also tests / unit tests (push) and tests / build release (push)

See therepanic@ee11c24

@JPL11

JPL11 commented Jul 10, 2026

Copy link
Copy Markdown
Author

Those are two different failures, and only one of them is this PR's.

Your build release failure is exactly what this PR fixes: the log dies at [-] copying files T=1 right at the 60s mark, so the timeout-minutes: 1 on Build devel killed it mid copy (your runner was even slower than mine, which died later, during the commit phase).

Your unit tests failure is unrelated to the timeout. The root cause is in your log: ImportError: libGLESv2.so.2: cannot open shared object file when loading _raylib_cffi_headless. The comma-deps-raylib headless backend links against GLES, and tools/setup_dependencies.sh installs no GL libraries at all, so it only works where libGLESv2.so.2 happens to be preinstalled (comma's namespace runner image, typical dev machines). Vanilla ubuntu-24.04 GitHub runners do not have it, which is why master is green upstream while forks fail. Workaround: sudo apt-get install -y libgles2 before running tests. The proper fix is probably adding it to the apt list in setup_dependencies.sh. That deserves its own PR to keep this one minimal; feel free to take it since you hit it, otherwise I can file it.

@therepanic

Copy link
Copy Markdown
Contributor

are you an agent?

@JPL11

JPL11 commented Jul 10, 2026

Copy link
Copy Markdown
Author

Human, I do use Claude Code to pull CI logs and traced the ImportError chain though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants