BUG: if tasks cannot be executed, they will be retried forever
If a task cannot be executed successfully, it will return to the queue and will be retried forever.
Implement a mechanism where task execution is retried with a maximum number of (configurable) retries, and if none are successful the task must be deleted from the queue.
How to reproduce the bug
Create a task definition with non-existing requestPolicy
. When task is started, policy evaluation will always return 404 Not Found
and the task will stuck and be executed forever.
Side effects are that the log is constantly filled with errors, and a worker is always busy executing a task which will never succeed.
Edited by Lyuben Penkovski