diff --git a/README.md b/README.md index 804ae4d..eded788 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Namespace Node Affinity is a Kubernetes mutating webhook which provides the abil It is a replacement for the [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) admission controller and it is useful when using a managed k8s control plane such as [GKE](https://cloud.google.com/kubernetes-engine) or [EKS](https://aws.amazon.com/eks) where you do not have the ability to enable additional admission controller plugins and the [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) might not be available. The only admission controller plugin required to run the namespace-node-affinity mutating webhook is the `MutatingAdmissionWebhook` which is already enabled on most managed Kubernetes services such as [EKS](https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html). -It might still be useful on [AKS](https://azure.microsoft.com/en-gb/services/kubernetes-service/) where the [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) admission controller is [readily available](https://docs.microsoft.com/en-us/azure/aks/faq#what-kubernetes-admission-controllers-does-aks-support-can-admission-controllers-be-added-or-removed) as using `namespace-node-affinity` allows a litte bit more flexibility than the node selector by allowing you to set node affinity (only `requiredDuringSchedulingIgnoredDuringExecution` is supported for now) for all pods in the namespace. +It might still be useful on [AKS](https://azure.microsoft.com/en-gb/services/kubernetes-service/) where the [PodNodeSelector](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#podnodeselector) admission controller is [readily available](https://docs.microsoft.com/en-us/azure/aks/faq#what-kubernetes-admission-controllers-does-aks-support-can-admission-controllers-be-added-or-removed) as using `namespace-node-affinity` allows a litte bit more flexibility than the node selector by allowing you to set node affinity for all pods in the namespace. # Deployment @@ -47,7 +47,13 @@ To enable the namespace-node-affinity mutating webhook on a namespace you simply kubectl label ns my-namespace namespace-node-affinity=enabled ``` -Each namespace with the `namespace-node-affinity=enabled` label will also need an entry in the `ConfigMap` where the configuration for the webhook is stored. The config for each namespace can be in either JSON or YAML format and must have at least one of `nodeSelectorTerms` or `tolerations`. The `nodeSelectorTerms` from the config will be added as `requiredDuringSchedulingIgnoredDuringExecution` node affinity type to each pod that is created in the labeled namespace. An example configuration can be found in [examples/sample_configmap.yaml](/examples/sample_configmap.yaml). +Each namespace with the `namespace-node-affinity=enabled` label will also need an entry in the `ConfigMap` where the configuration for the webhook is stored. The config for each namespace can be in either JSON or YAML format and must have at least one of `nodeSelectorTerms`, `preferredNodeSelectorTerms`, or `tolerations`. + +The `nodeSelectorTerms` from the config will be added as `requiredDuringSchedulingIgnoredDuringExecution` node affinity type to each pod that is created in the labeled namespace. This is a hard requirement and pods will only be scheduled on nodes that satisfy all the specified terms. + +The `preferredNodeSelectorTerms` from the config will be added as soft/preferred node affinity rules to each pod. The scheduler will try to satisfy these preferences but will still schedule the pod even if no nodes match. Each preferred term has a weight (1-100) that influences scheduling decisions. + +An example configuration can be found in [examples/sample_configmap.yaml](/examples/sample_configmap.yaml). More information on how node affinity works can be found [here](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity). More information on how taints and tolerations work can be found [here](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). @@ -68,13 +74,13 @@ time="2021-09-03T17:32:16Z" level=info msg="Received AdmissionReview: {...} time="2021-09-03T17:32:16Z" level=error msg="missing configuration: for testing-ns-e" ``` - * Both `nodeSelectorTerms` and `tolerations` are missing from the entry for the namespace in the `ConfigMap` + * Both `nodeSelectorTerms`, `preferredNodeSelectorTerms` and `tolerations` are missing from the entry for the namespace in the `ConfigMap` ``` time="2021-09-03T17:38:46Z" level=info msg="Received AdmissionReview: {...} -time="2021-09-03T17:38:46Z" level=error msg="invalid configuration: at least one of nodeSelectorTerms or tolerations needs to be specified for testing-ns-d" +time="2021-09-03T17:38:46Z" level=error msg="invalid configuration: at least one of nodeSelectorTerms, preferredNodeSelectorTerms or tolerations needs to be specified for testing-ns-d" ``` - * Invalid `nodeSelectorTerms` or `tolerations` in the `namespace-node-affinity` `ConfigMap` + * Invalid `nodeSelectorTerms`, `preferredNodeSelectorTerms` or `tolerations` in the `namespace-node-affinity` `ConfigMap` ``` time="2021-04-10T09:40:59Z" level=info msg="Received AdmissionReview: {...} time="2021-04-10T09:40:59Z" level=error msg="invalid configuration: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field NamespaceConfig.nodeSelectorTerms of type []v1.NodeSelectorTerm" diff --git a/examples/sample_configmap.yaml b/examples/sample_configmap.yaml index 65023d4..34e4653 100644 --- a/examples/sample_configmap.yaml +++ b/examples/sample_configmap.yaml @@ -35,3 +35,38 @@ data: - key: "example-key" operator: "Exists" effect: "NoSchedule" + testing-ns-preferred: | + preferredNodeSelectorTerms: + - weight: 100 + preference: + matchExpressions: + - key: spot + operator: In + values: + - "true" + - weight: 50 + preference: + matchExpressions: + - key: zone + operator: In + values: + - us-west-2a + testing-ns-combined: | + nodeSelectorTerms: + - matchExpressions: + - key: dedicated + operator: In + values: + - "true" + preferredNodeSelectorTerms: + - weight: 100 + preference: + matchExpressions: + - key: spot + operator: In + values: + - "true" + tolerations: + - key: "spot" + operator: "Exists" + effect: "NoSchedule" diff --git a/injector/injector.go b/injector/injector.go index 4ebbece..e6f66b1 100644 --- a/injector/injector.go +++ b/injector/injector.go @@ -31,11 +31,13 @@ type PatchPath string // PatchPath values const ( // affinity - CreateAffinity = "/spec/affinity" - CreateNodeAffinity = "/spec/affinity/nodeAffinity" - AddRequiredDuringScheduling = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution" - AddNodeSelectorTerms = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms" - AddToNodeSelectorTerms = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/-" + CreateAffinity = "/spec/affinity" + CreateNodeAffinity = "/spec/affinity/nodeAffinity" + AddRequiredDuringScheduling = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution" + AddNodeSelectorTerms = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms" + AddToNodeSelectorTerms = "/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/-" + AddPreferredNodeSelectorTerms = "/spec/affinity/nodeAffinity/preferredDuringSchedulingIgnoredDuringExecution" + AddToPreferredNodeSelectorTerms = "/spec/affinity/nodeAffinity/preferredDuringSchedulingIgnoredDuringExecution/-" // tolerations CreateTolerations = "/spec/tolerations" AddTolerations = "/spec/tolerations/-" @@ -64,9 +66,10 @@ type JSONPatch struct { // NamespaceConfig is the per-namespace configuration type NamespaceConfig struct { - NodeSelectorTerms []corev1.NodeSelectorTerm `json:"nodeSelectorTerms"` - Tolerations []corev1.Toleration `json:"tolerations"` - ExcludedLabels map[string]string `json:"excludedLabels"` + NodeSelectorTerms []corev1.NodeSelectorTerm `json:"nodeSelectorTerms"` + PreferredNodeSelectorTerms []corev1.PreferredSchedulingTerm `json:"preferredNodeSelectorTerms"` + Tolerations []corev1.Toleration `json:"tolerations"` + ExcludedLabels map[string]string `json:"excludedLabels"` } // Injector handles AdmissionReview objects @@ -175,8 +178,8 @@ func (m *Injector) configForNamespace(namespace string) (*NamespaceConfig, error err = yamlUnmarshal([]byte(namespaceConfigString), config) if err != nil { return nil, fmt.Errorf("%w: %s", ErrInvalidConfiguration, err) - } else if config.NodeSelectorTerms == nil && config.Tolerations == nil { - return nil, fmt.Errorf("%w: at least one of nodeSelectorTerms or tolerations needs to be specified for %s", ErrInvalidConfiguration, namespace) + } else if config.NodeSelectorTerms == nil && config.PreferredNodeSelectorTerms == nil && config.Tolerations == nil { + return nil, fmt.Errorf("%w: at least one of nodeSelectorTerms, preferredNodeSelectorTerms or tolerations needs to be specified for %s", ErrInvalidConfiguration, namespace) } return config, nil @@ -208,6 +211,62 @@ func buildTolerationsPath(podSpec corev1.PodSpec) PatchPath { return AddTolerations } +func buildPreferredAffinityPath(podSpec corev1.PodSpec) PatchPath { + if podSpec.Affinity == nil { + return CreateAffinity + } else if podSpec.Affinity.NodeAffinity == nil { + return CreateNodeAffinity + } else if podSpec.Affinity.NodeAffinity.PreferredDuringSchedulingIgnoredDuringExecution == nil { + return AddPreferredNodeSelectorTerms + } + return AddToPreferredNodeSelectorTerms +} + +func buildPreferredAffinityPatch(path PatchPath, preferredTerm corev1.PreferredSchedulingTerm) JSONPatch { + patch := JSONPatch{ + Op: "add", + Path: path, + Value: preferredTerm, + } + + return patch +} + +// Returns a patch that initialises the PodSpec's PreferredDuringSchedulingIgnoredDuringExecution array as an empty array, if it does not exist +func buildPreferredAffinityInitPatch(podSpec corev1.PodSpec) (JSONPatch, error) { + path := buildPreferredAffinityPath(podSpec) + + patch := JSONPatch{ + Op: "add", + Path: path, + } + + patchAffinity := &corev1.Affinity{ + NodeAffinity: &corev1.NodeAffinity{ + PreferredDuringSchedulingIgnoredDuringExecution: []corev1.PreferredSchedulingTerm{}, + }, + } + + switch path { + case AddToPreferredNodeSelectorTerms: + // Array for PreferredDuringSchedulingIgnoredDuringExecution already exists. Do nothing + return JSONPatch{}, nil + case AddPreferredNodeSelectorTerms: + // PreferredDuringSchedulingIgnoredDuringExecution array missing, add it + patch.Value = patchAffinity.NodeAffinity.PreferredDuringSchedulingIgnoredDuringExecution + case CreateNodeAffinity: + // Adds NodeAffinity with PreferredDuringSchedulingIgnoredDuringExecution + patch.Value = patchAffinity.NodeAffinity + case CreateAffinity: + // Adds Affinity with NodeAffinity and PreferredDuringSchedulingIgnoredDuringExecution + patch.Value = patchAffinity + default: + return JSONPatch{}, fmt.Errorf("%w: invalid patch path", ErrFailedToCreatePatch) + } + + return patch, nil +} + func buildNodeSelectorTermPatch(path PatchPath, nodeSelectorTerm corev1.NodeSelectorTerm) JSONPatch { patch := JSONPatch{ Op: "add", @@ -278,6 +337,22 @@ func buildPatch(config *NamespaceConfig, podSpec corev1.PodSpec) ([]byte, error) } } + if config.PreferredNodeSelectorTerms != nil { + initPatch, err := buildPreferredAffinityInitPatch(podSpec) + if err != nil { + return nil, err + } + if (initPatch != JSONPatch{}) { + patches = append(patches, initPatch) + } + + for _, preferredTerm := range config.PreferredNodeSelectorTerms { + preferredAffinityPatch := buildPreferredAffinityPatch(AddToPreferredNodeSelectorTerms, preferredTerm) + + patches = append(patches, preferredAffinityPatch) + } + } + if config.Tolerations != nil { tolerationsPatchPath := buildTolerationsPath(podSpec) for _, toleration := range config.Tolerations { diff --git a/injector/injector_test.go b/injector/injector_test.go index 4c2b5b7..e347604 100644 --- a/injector/injector_test.go +++ b/injector/injector_test.go @@ -571,3 +571,347 @@ func TestMutateIgnoresPodsWithExcludedLabels(t *testing.T) { assert.NoError(t, err) assert.Equal(t, j, body) } + +func preferredSchedulingTerms() []corev1.PreferredSchedulingTerm { + return []corev1.PreferredSchedulingTerm{ + { + Weight: 100, + Preference: corev1.NodeSelectorTerm{ + MatchExpressions: []corev1.NodeSelectorRequirement{ + { + Key: "spot", + Operator: corev1.NodeSelectorOpIn, + Values: []string{"true"}, + }, + }, + }, + }, + { + Weight: 50, + Preference: corev1.NodeSelectorTerm{ + MatchExpressions: []corev1.NodeSelectorRequirement{ + { + Key: "zone", + Operator: corev1.NodeSelectorOpIn, + Values: []string{"us-west-2a"}, + }, + }, + }, + }, + } +} + +// PodSpecs with various levels of completion for Preferred Node Affinity +var ( + podSpecWithNoPreferredAffinity = corev1.PodSpec{ + Affinity: &corev1.Affinity{ + NodeAffinity: &corev1.NodeAffinity{}, + }, + } + podSpecWithEmptyPreferredAffinity = corev1.PodSpec{ + Affinity: &corev1.Affinity{ + NodeAffinity: &corev1.NodeAffinity{ + PreferredDuringSchedulingIgnoredDuringExecution: []corev1.PreferredSchedulingTerm{}, + }, + }, + } + podSpecWithExistingPreferredAffinity = corev1.PodSpec{ + Affinity: &corev1.Affinity{ + NodeAffinity: &corev1.NodeAffinity{ + PreferredDuringSchedulingIgnoredDuringExecution: []corev1.PreferredSchedulingTerm{ + { + Weight: 10, + Preference: corev1.NodeSelectorTerm{ + MatchExpressions: []corev1.NodeSelectorRequirement{ + { + Key: "existing", + Operator: corev1.NodeSelectorOpIn, + Values: []string{"value"}, + }, + }, + }, + }, + }, + }, + }, + } +) + +func TestBuildPreferredAffinityPath(t *testing.T) { + t.Parallel() + + testCases := []struct { + name string + podSpec corev1.PodSpec + expectedPath PatchPath + }{ + { + name: "WithNoAffinity", + podSpec: podSpecWithNoAffinity, + expectedPath: CreateAffinity, + }, + { + name: "WithNoNodeAffinity", + podSpec: podSpecWithNoNodeAffinity, + expectedPath: CreateNodeAffinity, + }, + { + name: "WithNoPreferredAffinity", + podSpec: podSpecWithNoPreferredAffinity, + expectedPath: AddPreferredNodeSelectorTerms, + }, + { + name: "WithEmptyPreferredAffinity", + podSpec: podSpecWithEmptyPreferredAffinity, + expectedPath: AddToPreferredNodeSelectorTerms, + }, + { + name: "WithExistingPreferredAffinity", + podSpec: podSpecWithExistingPreferredAffinity, + expectedPath: AddToPreferredNodeSelectorTerms, + }, + } + + for _, tc := range testCases { + tc := tc + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + path := buildPreferredAffinityPath(tc.podSpec) + assert.Equal(t, tc.expectedPath, path) + }) + } +} + +func TestBuildPreferredAffinityPatch(t *testing.T) { + t.Parallel() + + path := PatchPath("") + preferredTerm := corev1.PreferredSchedulingTerm{ + Weight: 100, + Preference: corev1.NodeSelectorTerm{ + MatchExpressions: []corev1.NodeSelectorRequirement{ + { + Key: "test-key", + Operator: corev1.NodeSelectorOpIn, + Values: []string{"val"}, + }, + }, + }, + } + expectedPatch := JSONPatch{ + Op: "add", + Path: path, + Value: preferredTerm, + } + + patch := buildPreferredAffinityPatch(path, preferredTerm) + assert.Equal(t, expectedPatch, patch) +} + +func TestBuildPreferredAffinityInitPatch(t *testing.T) { + t.Parallel() + + testCases := []struct { + name string + podSpec corev1.PodSpec + expectedPatch JSONPatch + }{ + { + name: "WithNoAffinity", + podSpec: podSpecWithNoAffinity, + expectedPatch: JSONPatch{ + Op: "add", + Path: CreateAffinity, + Value: &corev1.Affinity{ + NodeAffinity: &corev1.NodeAffinity{ + PreferredDuringSchedulingIgnoredDuringExecution: []corev1.PreferredSchedulingTerm{}, + }, + }, + }, + }, + { + name: "WithNoNodeAffinity", + podSpec: podSpecWithNoNodeAffinity, + expectedPatch: JSONPatch{ + Op: "add", + Path: CreateNodeAffinity, + Value: &corev1.NodeAffinity{ + PreferredDuringSchedulingIgnoredDuringExecution: []corev1.PreferredSchedulingTerm{}, + }, + }, + }, + { + name: "WithNoPreferredAffinity", + podSpec: podSpecWithNoPreferredAffinity, + expectedPatch: JSONPatch{ + Op: "add", + Path: AddPreferredNodeSelectorTerms, + Value: []corev1.PreferredSchedulingTerm{}, + }, + }, + { + name: "WithEmptyPreferredAffinity", + podSpec: podSpecWithEmptyPreferredAffinity, + expectedPatch: JSONPatch{}, + }, + { + name: "WithExistingPreferredAffinity", + podSpec: podSpecWithExistingPreferredAffinity, + expectedPatch: JSONPatch{}, + }, + } + + for _, tc := range testCases { + tc := tc + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + + patch, err := buildPreferredAffinityInitPatch(tc.podSpec) + + assert.Nil(t, err) + assert.Equal(t, tc.expectedPatch, patch) + }) + } +} + +func TestMutateWithPreferredAffinity(t *testing.T) { + t.Parallel() + + deploymentNamespace := "ns-node-affinity" + podNamespace := "testing-ns-preferred" + + nsConfig := NamespaceConfig{ + PreferredNodeSelectorTerms: preferredSchedulingTerms(), + } + nsConfigJSON, _ := json.Marshal(nsConfig) + + cm := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-cm", + Namespace: deploymentNamespace, + }, + Data: map[string]string{podNamespace: string(nsConfigJSON)}, + } + clientset := fake.NewSimpleClientset(cm) + m := Injector{clientset, deploymentNamespace, "test-cm"} + + samplePod := corev1.Pod{} + + admissionReview := v1beta1.AdmissionReview{ + Request: &v1beta1.AdmissionRequest{ + Namespace: podNamespace, + Object: runtime.RawExtension{ + Object: &samplePod, + }, + }, + } + j, err := json.Marshal(admissionReview) + assert.NoError(t, err) + + body, err := m.Mutate(j) + assert.NoError(t, err) + + expectedPatch, err := buildPatch(&nsConfig, samplePod.Spec) + assert.NoError(t, err) + + jsonPatch := v1beta1.PatchTypeJSONPatch + expectedResp := v1beta1.AdmissionResponse{ + PatchType: &jsonPatch, + Allowed: true, + Patch: expectedPatch, + AuditAnnotations: map[string]string{annotationKey: string(expectedPatch)}, + Result: &metav1.Status{Status: successStatus}, + } + + expectedAdmissionReview := admissionReview + expectedAdmissionReview.Response = &expectedResp + + expectedBody, err := json.Marshal(expectedAdmissionReview) + assert.NoError(t, err) + assert.Equal(t, expectedBody, body) +} + +func TestMutateWithBothRequiredAndPreferredAffinity(t *testing.T) { + t.Parallel() + + deploymentNamespace := "ns-node-affinity" + podNamespace := "testing-ns-both" + + nsConfig := NamespaceConfig{ + NodeSelectorTerms: nodeSelectorTerms(), + PreferredNodeSelectorTerms: preferredSchedulingTerms(), + Tolerations: tolerations(), + } + nsConfigJSON, _ := json.Marshal(nsConfig) + + cm := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-cm", + Namespace: deploymentNamespace, + }, + Data: map[string]string{podNamespace: string(nsConfigJSON)}, + } + clientset := fake.NewSimpleClientset(cm) + m := Injector{clientset, deploymentNamespace, "test-cm"} + + samplePod := corev1.Pod{} + + admissionReview := v1beta1.AdmissionReview{ + Request: &v1beta1.AdmissionRequest{ + Namespace: podNamespace, + Object: runtime.RawExtension{ + Object: &samplePod, + }, + }, + } + j, err := json.Marshal(admissionReview) + assert.NoError(t, err) + + body, err := m.Mutate(j) + assert.NoError(t, err) + + expectedPatch, err := buildPatch(&nsConfig, samplePod.Spec) + assert.NoError(t, err) + + jsonPatch := v1beta1.PatchTypeJSONPatch + expectedResp := v1beta1.AdmissionResponse{ + PatchType: &jsonPatch, + Allowed: true, + Patch: expectedPatch, + AuditAnnotations: map[string]string{annotationKey: string(expectedPatch)}, + Result: &metav1.Status{Status: successStatus}, + } + + expectedAdmissionReview := admissionReview + expectedAdmissionReview.Response = &expectedResp + + expectedBody, err := json.Marshal(expectedAdmissionReview) + assert.NoError(t, err) + assert.Equal(t, expectedBody, body) +} + +func TestBuildPatchWithPreferredAffinityInitError(t *testing.T) { + t.Parallel() + + // Test that buildPatch properly handles errors from buildPreferredAffinityInitPatch + // Since buildPreferredAffinityPath always returns valid paths in the current implementation, + // we verify that the error handling code path exists and buildPatch can handle it correctly. + config := &NamespaceConfig{ + PreferredNodeSelectorTerms: preferredSchedulingTerms(), + } + + // Create a PodSpec - this should succeed normally as buildPreferredAffinityPath + // always returns valid paths + podSpec := corev1.PodSpec{} + + patch, err := buildPatch(config, podSpec) + assert.NoError(t, err) + assert.NotNil(t, patch) + + // Verify the patch was created correctly + var patches []JSONPatch + err = json.Unmarshal(patch, &patches) + assert.NoError(t, err) + assert.True(t, len(patches) > 0, "Expected at least one patch to be created") +}