BUG: if tasks cannot be executed, they will be retried forever

If a task cannot be executed successfully, it will return to the queue and will be retried forever.

Implement a mechanism where task execution is retried with a maximum number of (configurable) retries, and if none are successful the task must be deleted from the queue.

How to reproduce the bug

Create a task definition with non-existing requestPolicy. When task is started, policy evaluation will always return 404 Not Found and the task will stuck and be executed forever.

Side effects are that the log is constantly filled with errors, and a worker is always busy executing a task which will never succeed.

Edited by Lyuben Penkovski