Skip to content

Prevent deleted Lambda auto-annotation requests from overwriting a newer same-task request - #11048

Open
ksm463 wants to merge 3 commits into
cvat-ai:developfrom
ksm463:topic/lambda-aba-fix
Open

Prevent deleted Lambda auto-annotation requests from overwriting a newer same-task request#11048
ksm463 wants to merge 3 commits into
cvat-ai:developfrom
ksm463:topic/lambda-aba-fix

Conversation

@ksm463

@ksm463 ksm463 commented Aug 15, 2026

Copy link
Copy Markdown

Motivation and context

Fixes #11016.

DELETE /api/lambda/requests/{id} unconditionally deleted the RQ job hash, including
while the job was STARTED. Because Lambda auto-annotation derives a deterministic
RQ id from the task (create:task{id} style), deleting a running job immediately frees
that id for reuse while the old work-horse is still alive and still writing annotations.

That allows two failure modes for the same task:

  • the old execution finishes after a new request was created under the same id and
    overwrites the newer request's progress/result;
  • the old execution's annotation writes land on top of the new run's output, which
    surfaces to users as duplicated or mixed annotations.

This PR implements the minimal-conservative policy (option 1) proposed in #11016:
a running request is no longer cancelable, so its id can never be recycled underneath it.

  • LambdaQueue.delete_job() takes the same get_rq_lock_for_job(queue, pk) lock that
    enqueue() uses, so a DELETE can no longer race a concurrent POST for the same task.
  • STARTED409 Conflict, and the hash is left untouched.
  • QUEUED / DEFERRED / SCHEDULEDcancel() then delete(), so a queued request
    can still be dropped and immediately re-submitted for the same task.
  • terminal states (FINISHED / FAILED / CANCELED / STOPPED) → deleted as before.
  • LambdaJob.delete() is removed; RequestViewSet.destroy() was its only caller.

Permission checks are unchanged: check_object_permissions() still runs on the
fetch_job() result, and the status is then re-checked inside the lock.

Behavior change worth flagging. DELETE on a running request now returns 409
where it previously returned 204, so the UI's "Cancel" button on an in-progress
auto-annotation will surface an error instead of appearing to succeed. That appearance
was not accurate before this change — the DELETE removed the hash but never stopped the
work-horse, which is exactly the bug above. Genuinely interrupting a running job needs
the larger stop-and-ack design (option 2 in #11016: progress/annotation write guards,
a stop command, idempotent handling). I kept this PR to the minimal fix and am happy to
follow up with option 2, or to adjust the UI message here, if you prefer.

How has this been tested?

Server unit tests, run in the cvat_ci container against
cvat.settings.testing:

python manage.py test --settings cvat.settings.testing \
  cvat.apps.lambda_manager.tests.test_lambda

Two tests that were previously @skip("Fail: add mock") are now implemented and pass:

  • test_api_v2_lambda_requests_create_two_requests — characterizes the existing
    enqueue() lock: POSTing a second request for the same task while the first is
    STARTED returns 409.
  • test_api_v2_lambda_requests_delete_not_finished_request — the regression test for
    this fix. It asserts that DELETE on a STARTED request is rejected, that the job hash
    survives, that a second request for the same task stays blocked until the first reaches
    a terminal state, and that afterwards the second request runs exactly once and reuses
    the same deterministic id.

Because cvat.settings.testing forces every RQ queue to ASYNC=False, a normal request
completes inline and can never be observed as STARTED. The tests therefore monkeypatch
the queue's enqueue_job to persist the job in STARTED without running it, and flip
the status explicitly to reproduce the lifecycle without a live worker or a real Nuclio
function. No Nuclio deployment is required to run them.

Verified against the fix reverted: test_api_v2_lambda_requests_delete_not_finished_request
fails with AssertionError: 204 not found in (400, 409), i.e. it reproduces the reported
bug rather than merely passing.

Full cvat.apps.lambda_manager.tests.test_lambda run is green.

Checklist

  • I submit my changes into the develop branch
  • I have created a changelog fragment
  • I have updated the documentation accordingly
  • I have added tests to cover my changes
  • I have linked related issues

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Running Lambda auto-annotation can overwrite or replay a newer same-task request after deletion

1 participant