Add support for preferredDuringSchedulingIgnoredDuringExecution

README.md

@@ -8,7 +8,7 @@ Namespace Node Affinity is a Kubernetes mutating webhook which provides the abil
 
 It is a replacement for the [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) admission controller and it is useful when using a managed k8s control plane such as [GKE](https://cloud.google.com/kubernetes-engine) or [EKS](https://aws.amazon.com/eks) where you do not have the ability to enable additional admission controller plugins and the [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) might not be available. The only admission controller plugin required to run the namespace-node-affinity mutating webhook is the `MutatingAdmissionWebhook` which is already enabled on most managed Kubernetes services such as [EKS](https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html).
 
-It might still be useful on [AKS](https://azure.microsoft.com/en-gb/services/kubernetes-service/) where the [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) admission controller is [readily available](https://docs.microsoft.com/en-us/azure/aks/faq#what-kubernetes-admission-controllers-does-aks-support-can-admission-controllers-be-added-or-removed) as using `namespace-node-affinity` allows a litte bit more flexibility than the node selector by allowing you to set node affinity (only `requiredDuringSchedulingIgnoredDuringExecution` is supported for now) for all pods in the namespace.
+It might still be useful on [AKS](https://azure.microsoft.com/en-gb/services/kubernetes-service/) where the [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) admission controller is [readily available](https://docs.microsoft.com/en-us/azure/aks/faq#what-kubernetes-admission-controllers-does-aks-support-can-admission-controllers-be-added-or-removed) as using `namespace-node-affinity` allows a litte bit more flexibility than the node selector by allowing you to set node affinity for all pods in the namespace.
 
 # Deployment
 
@@ -47,7 +47,13 @@ To enable the namespace-node-affinity mutating webhook on a namespace you simply
 kubectl label ns my-namespace namespace-node-affinity=enabled
 ```
 
-Each namespace with the `namespace-node-affinity=enabled` label will also need an entry in the `ConfigMap` where the configuration for the webhook is stored. The config for each namespace can be in either JSON or YAML format and must have at least one of `nodeSelectorTerms` or `tolerations`. The `nodeSelectorTerms` from the config will be added as `requiredDuringSchedulingIgnoredDuringExecution` node affinity type to each pod that is created in the labeled namespace. An example configuration can be found in [examples/sample_configmap.yaml](/examples/sample_configmap.yaml).
+Each namespace with the `namespace-node-affinity=enabled` label will also need an entry in the `ConfigMap` where the configuration for the webhook is stored. The config for each namespace can be in either JSON or YAML format and must have at least one of `nodeSelectorTerms`, `preferredNodeSelectorTerms`, or `tolerations`.
+
+The `nodeSelectorTerms` from the config will be added as `requiredDuringSchedulingIgnoredDuringExecution` node affinity type to each pod that is created in the labeled namespace. This is a hard requirement and pods will only be scheduled on nodes that satisfy all the specified terms.
+
+The `preferredNodeSelectorTerms` from the config will be added as soft/preferred node affinity rules to each pod. The scheduler will try to satisfy these preferences but will still schedule the pod even if no nodes match. Each preferred term has a weight (1-100) that influences scheduling decisions.
+
+An example configuration can be found in [examples/sample_configmap.yaml](/examples/sample_configmap.yaml).
 
 More information on how node affinity works can be found [here](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity).
 More information on how taints and tolerations work can be found [here](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/).
@@ -68,13 +74,13 @@ time="2021-09-03T17:32:16Z" level=info msg="Received AdmissionReview: {...}
 time="2021-09-03T17:32:16Z" level=error msg="missing configuration: for testing-ns-e"
 ```
 
- * Both `nodeSelectorTerms` and `tolerations` are missing from the entry for the namespace in the `ConfigMap`
+ * Both `nodeSelectorTerms`, `preferredNodeSelectorTerms` and `tolerations` are missing from the entry for the namespace in the `ConfigMap`
 ```
 time="2021-09-03T17:38:46Z" level=info msg="Received AdmissionReview: {...}
-time="2021-09-03T17:38:46Z" level=error msg="invalid configuration: at least one of nodeSelectorTerms or tolerations needs to be specified for testing-ns-d"
+time="2021-09-03T17:38:46Z" level=error msg="invalid configuration: at least one of nodeSelectorTerms, preferredNodeSelectorTerms or tolerations needs to be specified for testing-ns-d"
 ```
 
- * Invalid `nodeSelectorTerms` or `tolerations` in the `namespace-node-affinity` `ConfigMap`
+ * Invalid `nodeSelectorTerms`, `preferredNodeSelectorTerms` or `tolerations` in the `namespace-node-affinity` `ConfigMap`
 ```
 time="2021-04-10T09:40:59Z" level=info msg="Received AdmissionReview: {...}
 time="2021-04-10T09:40:59Z" level=error msg="invalid configuration: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field NamespaceConfig.nodeSelectorTerms of type []v1.NodeSelectorTerm"

examples/sample_configmap.yaml

@@ -35,3 +35,38 @@ data:
       - key: "example-key"
         operator: "Exists"
         effect: "NoSchedule"
+  testing-ns-preferred: |
+    preferredNodeSelectorTerms:
+      - weight: 100
+        preference:
+          matchExpressions:
+          - key: spot
+            operator: In
+            values:
+            - "true"
+      - weight: 50
+        preference:
+          matchExpressions:
+          - key: zone
+            operator: In
+            values:
+            - us-west-2a
+  testing-ns-combined: |
+    nodeSelectorTerms:
+      - matchExpressions:
+        - key: dedicated
+          operator: In
+          values:
+          - "true"
+    preferredNodeSelectorTerms:
+      - weight: 100
+        preference:
+          matchExpressions:
+          - key: spot
+            operator: In
+            values:
+            - "true"
+    tolerations:
+      - key: "spot"
+        operator: "Exists"
+        effect: "NoSchedule"

injector/injector.go

@@ -31,11 +31,13 @@ type PatchPath string
 // PatchPath values
 const (
 	// affinity
-	CreateAffinity              = "/spec/affinity"
-	CreateNodeAffinity          = "/spec/affinity/nodeAffinity"
-	AddRequiredDuringScheduling = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution"
-	AddNodeSelectorTerms        = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms"
-	AddToNodeSelectorTerms      = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/-"
+	CreateAffinity                  = "/spec/affinity"
+	CreateNodeAffinity              = "/spec/affinity/nodeAffinity"
+	AddRequiredDuringScheduling     = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution"
+	AddNodeSelectorTerms            = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms"
+	AddToNodeSelectorTerms          = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/-"
+	AddPreferredNodeSelectorTerms   = "/spec/affinity/nodeAffinity/preferredDuringSchedulingIgnoredDuringExecution"
+	AddToPreferredNodeSelectorTerms = "/spec/affinity/nodeAffinity/preferredDuringSchedulingIgnoredDuringExecution/-"
 	// tolerations
 	CreateTolerations = "/spec/tolerations"
 	AddTolerations    = "/spec/tolerations/-"
@@ -64,9 +66,10 @@ type JSONPatch struct {
 
 // NamespaceConfig is the per-namespace configuration
 type NamespaceConfig struct {
-	NodeSelectorTerms []corev1.NodeSelectorTerm `json:"nodeSelectorTerms"`
-	Tolerations       []corev1.Toleration       `json:"tolerations"`
-	ExcludedLabels    map[string]string         `json:"excludedLabels"`
+	NodeSelectorTerms          []corev1.NodeSelectorTerm        `json:"nodeSelectorTerms"`
+	PreferredNodeSelectorTerms []corev1.PreferredSchedulingTerm `json:"preferredNodeSelectorTerms"`
+	Tolerations                []corev1.Toleration              `json:"tolerations"`
+	ExcludedLabels             map[string]string                `json:"excludedLabels"`
 }
 
 // Injector handles AdmissionReview objects
@@ -175,8 +178,8 @@ func (m *Injector) configForNamespace(namespace string) (*NamespaceConfig, error
 	err = yamlUnmarshal([]byte(namespaceConfigString), config)
 	if err != nil {
 		return nil, fmt.Errorf("%w: %s", ErrInvalidConfiguration, err)
-	} else if config.NodeSelectorTerms == nil && config.Tolerations == nil {
-		return nil, fmt.Errorf("%w: at least one of nodeSelectorTerms or tolerations needs to be specified for %s", ErrInvalidConfiguration, namespace)
+	} else if config.NodeSelectorTerms == nil && config.PreferredNodeSelectorTerms == nil && config.Tolerations == nil {
+		return nil, fmt.Errorf("%w: at least one of nodeSelectorTerms, preferredNodeSelectorTerms or tolerations needs to be specified for %s", ErrInvalidConfiguration, namespace)
 	}
 
 	return config, nil
@@ -208,6 +211,62 @@ func buildTolerationsPath(podSpec corev1.PodSpec) PatchPath {
 	return AddTolerations
 }
 
+func buildPreferredAffinityPath(podSpec corev1.PodSpec) PatchPath {
+	if podSpec.Affinity == nil {
+		return CreateAffinity
+	} else if podSpec.Affinity.NodeAffinity == nil {
+		return CreateNodeAffinity
+	} else if podSpec.Affinity.NodeAffinity.PreferredDuringSchedulingIgnoredDuringExecution == nil {
+		return AddPreferredNodeSelectorTerms
+	}
+	return AddToPreferredNodeSelectorTerms
+}
+
+func buildPreferredAffinityPatch(path PatchPath, preferredTerm corev1.PreferredSchedulingTerm) JSONPatch {
+	patch := JSONPatch{
+		Op:    "add",
+		Path:  path,
+		Value: preferredTerm,
+	}
+
+	return patch
+}
+
+// Returns a patch that initialises the PodSpec's PreferredDuringSchedulingIgnoredDuringExecution array as an empty array, if it does not exist
+func buildPreferredAffinityInitPatch(podSpec corev1.PodSpec) (JSONPatch, error) {
+	path := buildPreferredAffinityPath(podSpec)
+
+	patch := JSONPatch{
+		Op:   "add",
+		Path: path,
+	}
+
+	patchAffinity := &corev1.Affinity{
+		NodeAffinity: &corev1.NodeAffinity{
+			PreferredDuringSchedulingIgnoredDuringExecution: []corev1.PreferredSchedulingTerm{},
+		},
+	}
+
+	switch path {
+	case AddToPreferredNodeSelectorTerms:
+		// Array for PreferredDuringSchedulingIgnoredDuringExecution already exists. Do nothing
+		return JSONPatch{}, nil
+	case AddPreferredNodeSelectorTerms:
+		// PreferredDuringSchedulingIgnoredDuringExecution array missing, add it
+		patch.Value = patchAffinity.NodeAffinity.PreferredDuringSchedulingIgnoredDuringExecution
+	case CreateNodeAffinity:
+		// Adds NodeAffinity with PreferredDuringSchedulingIgnoredDuringExecution
+		patch.Value = patchAffinity.NodeAffinity
+	case CreateAffinity:
+		// Adds Affinity with NodeAffinity and PreferredDuringSchedulingIgnoredDuringExecution
+		patch.Value = patchAffinity
+	default:
+		return JSONPatch{}, fmt.Errorf("%w: invalid patch path", ErrFailedToCreatePatch)
+	}
+
+	return patch, nil
+}
+
 func buildNodeSelectorTermPatch(path PatchPath, nodeSelectorTerm corev1.NodeSelectorTerm) JSONPatch {
 	patch := JSONPatch{
 		Op:    "add",
@@ -278,6 +337,22 @@ func buildPatch(config *NamespaceConfig, podSpec corev1.PodSpec) ([]byte, error)
 		}
 	}
 
+	if config.PreferredNodeSelectorTerms != nil {
+		initPatch, err := buildPreferredAffinityInitPatch(podSpec)
+		if err != nil {
+			return nil, err
+		}
+		if (initPatch != JSONPatch{}) {
+			patches = append(patches, initPatch)
+		}
+
+		for _, preferredTerm := range config.PreferredNodeSelectorTerms {
+			preferredAffinityPatch := buildPreferredAffinityPatch(AddToPreferredNodeSelectorTerms, preferredTerm)
+
+			patches = append(patches, preferredAffinityPatch)
+		}
+	}
+
 	if config.Tolerations != nil {
 		tolerationsPatchPath := buildTolerationsPath(podSpec)
 		for _, toleration := range config.Tolerations {