fix(runners,tasks): duplicate dispatch, terminal status overwrite, and HA finalization leaks

[cursor]

Summary

Fixes three critical runner/task correctness bugs still present on develop after PR #4023/#4024. These were previously identified by the bug-inspection automation but never merged.

Bugs and impact

1. Duplicate dispatch on ErrAllRunnersBusy

When every runner is at capacity, the task server requeued tasks by calling Enqueue while they still sat in the running set, then sent EventTypeRequeued from a defer. A concurrent queue tick could ClaimAndDequeue the task in that window and start a second dispatch on the same TaskRunner.

Trigger: Launch multiple remote tasks when all runners are busy; a periodic queue tick can double-dispatch the same task.

2. Stopped → failed after terminated_jobs emergency stop

On the runner client, when applyTerminatedJobs emergency-stops a job and Run() unwinds with an error, the error path overwrote an already-terminal stopped status with failed.

Trigger: Server tells runner to emergency-stop a reassigned/orphaned job; runner reports failed instead of stopped, breaking workflow edges and user-visible status.

3. HA pool state leak in failTaskRunnerLost

When the reconciler won the finalize lock but the DB already had a terminal status (e.g. runner report on another node), failTaskRunnerLost returned early without completing finalization. Pool/Redis state (running set, claims) and autorun children could leak.

Trigger: HA deployment; runner goes offline while another node finalizes the task; reconciler observes finished DB row but leaves stale pool state.

Root cause

• Requeue paths (ErrAllRunnersBusy, offline runner requeue, undispatched requeue) enqueued before releasing running/active bookkeeping.
• Runner job_pool error handler did not respect an already-finished status after emergency stop.
• failTaskRunnerLost treated an already-finished DB status as a no-op instead of completing finalizeRemoteTaskLocked.

Fix

• Release running/active bookkeeping (onTaskStop) before enqueueing on all requeue paths; notify pool synchronously with EventTypeRequeued.
• Add finalizeAfterRun on the runner client to preserve terminal status when Run() returns an error.
• When failTaskRunnerLost sees a finished status, call finalizeRemoteTaskLocked to release pool state.
• Harden offline requeue with pre-mutation DB re-check and rollback on persist failure.
• Move End != nil cleanup in finalizeRemoteTaskLocked outside HA-only guard.

Validation

• go test ./services/tasks/... ./services/runners/... ./api/runners/... — all pass
• New/updated tests: TestTaskRunner_ErrAllRunnersBusy_ReleasesRunningBeforeEnqueue, TestRunningJob_finalizeAfterRun_KeepsFinishedStatusOnRunError, reconciler HA finalization tests