Skip to content

add default queue name to mpi jobs #1175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
iamvigneshwars wants to merge 5 commits into
base: main
Choose a base branch
from
Open

add default queue name to mpi jobs #1175

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
33cb705
fix(charts): update default queuename clusterpolicy to check if mpijo…
iamvigneshwars Feb 20, 2026
9f62e96
fix(charts): update chainsaw test for default-queuename policy
iamvigneshwars Feb 20, 2026
cba6e45
chore(charts): bump workflows version from 0.13.38 to 0.13.39
iamvigneshwars Feb 20, 2026
547aa3a
fix(charts): update sessionspaces chainsaw test with mpijob queue-name
iamvigneshwars Feb 20, 2026
4b9e093
chore(sessionspaces): bump sessionspaces version from 0.3.19 to 0.3.20
iamvigneshwars Feb 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/sessionspaces/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: sessionspaces
description: Namespace controller for creating session namespaces
type: application
version: 0.3.19
version: 0.3.20
appVersion: 0.1.3
dependencies:
- name: common
Expand Down
2 changes: 2 additions & 0 deletions charts/sessionspaces/test-policy/pod-securitycontext/chainsaw-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,8 @@ spec:
kind: MPIJob
metadata:
name: test-mpijob
labels:
kueue.x-k8s.io/queue-name: default-queue
ownerReferences:
- name: test-workflow
kind: Workflow
Expand Down
2 changes: 1 addition & 1 deletion charts/workflows/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: workflows
description: Data Analysis workflow orchestration
type: application
version: 0.13.38
version: 0.13.39
dependencies:
- name: argo-workflows
repository: https://argoproj.github.io/argo-helm
Expand Down
19 changes: 19 additions & 0 deletions charts/workflows/templates/default-queuename-clusterpolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,22 @@ spec:
operator: AnyNotIn
value:
- default-queue

Copy link
Collaborator

@davehadley davehadley Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just confirming that I understand correctly:

The problem is that it is possible for MPIJob to start pods without kyverno policies being applied.

This PR fixes the problem caused by the set-default-queuename rule not being applied. We force the label to be explicitly set in MPIJob and that gets propagated down to the pods that it creates.

That's OK since we expect MPIJob to only be used by "experts" so we don't mind if they need to add a little extra boiler plate.

What about other kyverno policies? Can they being skipped as well by MPIJob pods?

In particular, I'm concerned about any security related policies (eg pod-securitycontext, apply-posix-uid-label etc).

Copy link
Contributor Author

@iamvigneshwars iamvigneshwars Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other Kyverno pod policies still apply to MPIJob created pods (I've included mpijob pods in my previous PR's). The only issue was with the set-default-queuename rule is if the queue label was not applied, the pods could start but would bypass Kueue.
This fixes that by requiring the label to be explicitly set on the MPIJob so it propagates to the pods.
All other policies pod security context are still applied, and pods will not start if those policies fail.

- name: validate-mpijob-queuename
match:
any:
- resources:
kinds:
- kubeflow.org/*/MPIJob
operations:
- CREATE
- UPDATE
validate:
message: "The label kueue.x-k8s.io/queue-name must be default-queue"
deny:
conditions:
all:
- key: "{{ `{{ request.object.metadata.labels.\"kueue.x-k8s.io/queue-name\" || '' }}` }}"
operator: AnyNotIn
value:
- default-queue
93 changes: 93 additions & 0 deletions charts/workflows/test-policy/queue-name/chainsaw-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,3 +62,96 @@ spec:
podMetadata:
labels:
kueue.x-k8s.io/queue-name: default-queue

- name: mpijob-missing-queue-is-rejected
try:
- apply:
expect:
- check:
($error != null): true
resource:
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: mpijob-missing-queue
spec:
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
containers:
- name: launcher
image: docker.io/library/busybox:latest
Worker:
replicas: 1
template:
spec:
containers:
- name: worker
image: docker.io/library/busybox:latest

- name: mpijob-default-queue-is-allowed
try:
- apply:
resource:
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: mpijob-default-queue
labels:
kueue.x-k8s.io/queue-name: default-queue
spec:
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
containers:
- name: launcher
image: docker.io/library/busybox:latest
Worker:
replicas: 1
template:
spec:
containers:
- name: worker
image: docker.io/library/busybox:latest
- assert:
resource:
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: mpijob-default-queue
labels:
kueue.x-k8s.io/queue-name: default-queue

- name: mpijob-custom-queue-is-rejected
try:
- apply:
expect:
- check:
($error != null): true
resource:
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: mpijob-custom-queue
labels:
kueue.x-k8s.io/queue-name: custom-queue
spec:
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
containers:
- name: launcher
image: docker.io/library/busybox:latest
Worker:
replicas: 1
template:
spec:
containers:
- name: worker
image: docker.io/library/busybox:latest