[Intermediate] Retry transient object-store failures

[larsspinetta12]

Complexity

Intermediate — needs more product judgment or multi-file design. Still largely testable with fakes; live cloud optional unless noted.

Context

providers/aws.py has no retry — a dropped connection fails the whole run.

Done when

• Transient errors retry with backoff.
• Permanent ones (NoSuchBucket, AccessDenied) fail immediately.
• Covered by a test with a fake client (no cloud).
• Slash/gateway replies stay free of provider detail (CWE-209); failures log server-side.

---

Worth repeating in every issue

1. Read platform/filestorage/ports.py first — the four-method ObjectStore protocol is the entire contract; the engine depends on nothing else. Fake stores are enough for most work.
2. The credential deny-list in syncable.py is a security boundary. Any change touching it, or what push() walks, must keep the planted-secret tests green in tests/filestorage/test_remote_sync.py (LEAKED_SECRET / canary in integrations.json must appear in no stored object).
3. Surfaces share platform.filestorage.operations — put behavior there; keep CLI / /remote-sync / gateway thin. Slash and gateway replies are an external surface: never echo str(exc) / provider detail into chat (CWE-209). Local CLI may show detail.
4. Prefer StrEnum for new closed vocab (see platform/filestorage/enums.py). Env var names stay in config/constants/filestorage.py.
5. Org-scoped gateway turns currently refuse on purpose — do not re-enable org sync without the namespaced-keys issue (Advanced Feature/tools refactor #13).

Source: community contribution ideas after merge of feat/laptop-s3-sync. Engine contract: platform/filestorage/ports.py.

[dhakad22klx]

I would like to work on this. @larsspinetta12 can you assign this to me ?

[santhiprakash]

Progress update: implemented the AWS provider retry/backoff fix and opened PR #4584 from . Verification: ============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/ubuntu/Projects/open-source
collected 0 items

============================ no tests ran in 0.00s ============================= passed (3 passed, 43 deselected).

[santhiprakash]

Progress update: implemented the AWS provider retry/backoff fix and opened PR #4584 from issue/4555-retry-transient-object-store-failures. Verification: uv run pytest tests/filestorage/test_remote_sync.py -k 'aws' passed (3 passed, 43 deselected).

[santhiprakash]

Hi maintainers — following @muddlebee's guidance on #4584: please assign this issue to me (santhiprakash) if it's still available and you'd like me to continue the transient object-store retry work.

I already had a draft fix + tests (closed PR #4584) and will only reopen/send a PR after assignment, per CONTRIBUTING. Happy to step aside if another contributor is preferred.

[dhakad22klx]

Hi, @santhiprakash I already commented, but I am not sure if anyone else can comment/claim if a comment already exists. Hi @muddlebee @larsspinetta12 may i get this issue assigned to me ?. I've already started working on this.

[dhakad22klx]

Hi, Closed my PR because there was extra commit from previous PR (by mistake).Apologize for the mistake.

[dhakad22klx]

Hi, do we need an explicit retry mechanism, or configuring boto3's built-in one enough?. I had implemented explicit retry mechanism but greptile review (PR : #4606 ) shows overlapping with built-in layer.

config = Config(
   retries = {
      'total_max_attempts': 10,
      'mode': 'standard'
   }
)