Skip to content

perf(ssh): cut warm SSH connect from ~97s to ~19s on high-latency links#9015

Open
nasagong wants to merge 4 commits into
stablyai:mainfrom
nasagong:perf/ssh-connect-speed
Open

perf(ssh): cut warm SSH connect from ~97s to ~19s on high-latency links#9015
nasagong wants to merge 4 commits into
stablyai:mainfrom
nasagong:perf/ssh-connect-speed

Conversation

@nasagong

Copy link
Copy Markdown

Summary

Fixes #9014

SSH connect time on a ~150ms RTT link drops from ~97s to ~19s (measured, warm relay). Three focused commits:

  1. Per-step startup timing for relay establish — main-side timeStartupStep (mirrors the renderer helper's -done/-failed/durationMs conventions) wrapping the relay establish phases and hook-install probes. Gated behind ORCA_STARTUP_DIAGNOSTICS=1, zero effect when disabled. These timings located the bottleneck below.
  2. Concurrent remote agent-hook installs (the headline) — the 16 per-agent hook installers ran serially, each paying its own SFTP round-trip chain on every connect: 93.5s at 150ms RTT. Concurrency is safe because installers touch disjoint per-agent paths, shared script-dir writes are atomic (tmp+rename, content-gated), and mkdirpRemote already tolerates concurrent mkdir. Measured 93.5s → 11.0s.
  3. Eager reconnect at window load — the renderer only reaches ssh:connect after its full hydration chain, serializing the SSH handshake behind startup. Main now starts the same reconnects at did-finish-load; the renderer's own reconnect joins via the existing connectInFlight dedup, so state transitions are unchanged. The eager pass waits for the renderer's explicit credential-listener-ready signal and is skipped entirely without it (fallback = exactly the old behavior). Passphrase-flagged targets keep deferring to tab focus.
Step (per target, warm relay) Before After
Managed hook install (16 installers) 93.5s 11.0s
Reconnect start after renderer hydration window load + ~100ms
Total until connected ~97s ~19s

Screenshots

No visual change.

Testing

  • pnpm lint
  • pnpm typecheck
  • pnpm test
  • pnpm build
  • Added or updated high-quality tests that would catch regressions, or explained why tests were not needed

New tests: concurrent-installer result order and per-agent failure isolation; eager-reconnect gating (no ready signal → eager skipped entirely, late signal ignored), passphrase-target skip, and renderer join dedup on the in-flight connect. Also manually verified against a real Linux remote over ~150ms RTT across repeated cold starts, including two targets connecting concurrently and session restore with live terminals.

AI Review Report

Reviewed with Claude Code and a Codex adversarial review (final verdict: approve). Main risks checked:

  • Cross-platform (macOS/Linux/Windows): no new platform-dependent code, shortcuts, labels, or path handling; hook installers keep their existing Windows-remote skip; Electron usage limited to an existing did-finish-load hook in main.
  • SSH/remote/local: local flows untouched; remote changes are ordering/concurrency only, no protocol or wire changes.
  • Agents/integrations: per-agent installer failure isolation and result ordering preserved (regression-tested); no provider-specific logic touched.
  • Performance: concurrency rides the existing single SFTP channel; no new SSH channels, so MaxSessions-limited remotes are unaffected.

Codex initially flagged that an eager connect could race the renderer's credential-request listener registration and lose a prompt (invisible 120s wait). Resolved deterministically: eager waits for an explicit listener-ready IPC signal and is skipped without it; Codex re-review approved.

Security Audit

  • No new credential handling, secrets, or auth paths; the credential-listener-ready gate strictly narrows the window where a prompt could be lost.
  • New IPC surface is a single no-payload ssh:credentialListenerReady invoke (signal only, idempotent, no data in or out).
  • No command execution or path construction added; hook installs reuse the existing SFTP writers (atomic tmp+rename) unchanged.
  • Renderer call is wrapped so a stale preload (dev hot-reload skew) degrades to skipping eager reconnect instead of crashing.

Notes

nasagong added 3 commits July 16, 2026 18:28
Adds a main-side timeStartupStep helper mirroring the renderer's
-done/-failed and durationMs conventions, and wraps the relay establish
phases plus the managed-hook install probes. Gated behind
ORCA_STARTUP_DIAGNOSTICS=1; these timings located the SSH cold-connect
bottlenecks addressed in the follow-up commits.
The 16 per-agent hook installers ran serially, each paying its own SFTP
round trips: ~93.5s per connect at 150ms RTT, dominating SSH connect
time. Concurrent execution is safe because installers touch disjoint
per-agent paths and shared script-dir writes are atomic (tmp+rename).
Measured 93.5s -> 11.0s on the same host.

Adds a regression test for result order and per-agent failure isolation
under concurrency.
Renderer startup only reaches its ssh:connect step after the full
hydration chain, serializing SSH handshake and relay deploy behind it.
Main now starts the same reconnects at did-finish-load; the renderer's
own reconnect joins them via connectInFlight, so state transitions are
unchanged.

The eager pass waits for the renderer's explicit
credential-listener-ready signal and is skipped entirely without one,
so an unexpected credential prompt can never fire before its listener
exists. Targets that last required a prompt keep deferring to tab
focus.
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a5406489-0943-46b6-bfd5-336f48e1a2fd

📥 Commits

Reviewing files that changed from the base of the PR and between e1f010b and cba2dee.

📒 Files selected for processing (2)
  • src/main/ipc/ssh.ts
  • src/main/window/attach-main-window-services.test.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/main/window/attach-main-window-services.test.ts
  • src/main/ipc/ssh.ts

📝 Walkthrough

Walkthrough

Remote managed hook installers now run concurrently while returning ordered per-agent statuses. SSH startup reconnects are gated on renderer credential-listener readiness, exposed through preload and renderer IPC wiring and triggered after the main window loads. Relay establishment and remote hook installation now record startup timing and failure diagnostics. Tests cover installer ordering, reconnect control flow, startup wiring, and updated IPC mocks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main SSH performance change in the PR.
Description check ✅ Passed The description matches the template and includes summary, screenshots, testing, AI review, security audit, and notes.
Linked Issues check ✅ Passed The changes address #9014 by parallelizing hook installs and overlapping reconnect startup while preserving credential-prompt safeguards.
Out of Scope Changes check ✅ Passed The added diagnostics, tests, and preload API wiring are all directly supporting the SSH performance work.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/main/window/attach-main-window-services.test.ts (1)

234-245: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert that the load callback invokes eager reconnect.

This only proves a did-finish-load callback exists and does not throw; an empty callback would pass. Mock or spy on eagerReconnectSshTargetsFromShutdown, invoke loadHandler, and assert exactly one call.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: b3ceb040-30e8-4495-b4c0-947fe7396f72

📥 Commits

Reviewing files that changed from the base of the PR and between 377082e and e1f010b.

📒 Files selected for processing (13)
  • src/main/agent-hooks/remote-hook-service-installers.test.ts
  • src/main/agent-hooks/remote-managed-hook-installers.ts
  • src/main/ipc/ssh.test.ts
  • src/main/ipc/ssh.ts
  • src/main/ssh/ssh-relay-session.ts
  • src/main/startup/startup-diagnostics.ts
  • src/main/window/attach-main-window-services.test.ts
  • src/main/window/attach-main-window-services.ts
  • src/preload/api-types.ts
  • src/preload/index.ts
  • src/renderer/src/hooks/useIpcEvents.test.ts
  • src/renderer/src/hooks/useIpcEvents.ts
  • src/renderer/src/web/web-preload-api.ts

Comment thread src/main/ipc/ssh.ts
…mers

Applies CodeRabbit review on stablyai#9015:

- whenSshCredentialListenerReady left its waiter in the array after the
  timeout resolved, and resetSshHandlerStateForTests dropped waiters
  without clearing their timers, retaining dead callbacks and 10s timer
  handles past test cleanup. Waiters now carry their timer, remove
  themselves as the timeout fires, and are cancelled on reset.
- The did-finish-load wiring test only asserted the callback did not
  throw, so an empty callback would have passed. It now mocks the ssh
  module and asserts eagerReconnectSshTargetsFromShutdown is called
  exactly once, and not before the event.
@AmethystLiang

Copy link
Copy Markdown
Contributor

thank you for the PR! will get it in ASAP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: SSH connect takes ~95s on high-latency links

3 participants