[WIP] feat: add support for native retry policies

[dejanzele]

This work is a continuation of the work done by @Sovietaced and huge shoutout and big thanks for doing the initial work!

What type of PR is this?

Feature

What this PR does / why we need it

Adds native preemption retry support to Armada, allowing preempted jobs to be automatically requeued instead of failing permanently. This improves cluster utilization and user experience by giving preemptible workloads a chance to complete even when temporarily displaced by higher-priority jobs.

When a job is preempted by a higher-priority job, it currently fails permanently. Users must manually resubmit preempted jobs, which is:

• Tedious for batch workloads with many jobs
• Error-prone (users may not notice preemptions)
• Wasteful of already-completed work in long-running jobs

Jobs can now opt-in to automatic retry after preemption. When a preemptible job is preempted:

1. The run is marked as preempted (terminal state for the run)
2. If retries are enabled and retry count is not exhausted, the job is requeued
3. The job gets a new run when resources become available
4. After exhausting retries, the job fails permanently

Server configuration:

submission:
  preemptionRetry:
    # Enable/disable the feature globally. When false, jobs with
    # preemption retry annotations are rejected at submission.
    enabled: true
    # Maximum retry count users can request via annotations.
    # If omitted, no upper bound is enforced.
    maxRetryCount: 10

Scheduler configuration:

scheduling:
  preemptionRetry:
    # Enable/disable preemption retry processing.
    enabled: true
    # Default retry count for jobs without annotation.
    # If omitted, defaults to 0 (no retries unless job specifies).
    defaultRetryCount: 3

Example job which sets custom retry rules:

queue: default
jobSetId: my-batch-job
jobs:
  - priority: 0
    namespace: default
    annotations:
      armadaproject.io/preemptionRetryEnabled: "true"
      armadaproject.io/preemptionMaxRetryCount: "3"
    podSpec:
      priorityClassName: preemptible
      containers:
        - name: worker
          image: my-image:latest
          resources:
            requests:
              cpu: "4"
              memory: "8Gi"

All jobs in a gang must have consistent preemption retry configuration:

• Same preemptionRetryEnabled value
• Same preemptionMaxRetryCount value

Which issue(s) this PR fixes

Fixes #4340

Special notes for your reviewer

[Sovietaced]

One thing we ran into recently is that there can also be collision on service names and ingress names. This would only affect folks opting into those features so can probably be done in a follow up pull request.

[Sovietaced]

Another issue that we see daily that seems to completely disrupt scheduling is the fact that there is no concept of a gang generation. This specifically seems to happen when the scheduler has performed a preemption on a gang.

Later the scheduler will do some logical schedule and preempt logic in the scheduler but it will see some old pods from the original gang schedule that were preempted still on the nodes and then we get error messages like

scheduler.go:202 scheduling cycle failure error="gang runner-1328d9002fde4882bcb-n3-0-n3-0-dn0-0 was partially evicted: 2 out of 3 jobs evicted" cycleNumber=1030057

I believe that it misinterprets real pods on the nodes from the previous gang generation as related to the logical schedule/preempt it does as part of the regular scheduling algorithm and then blows up.