CNTRLPLANE-2241: Add KMS encryption mode into library-go encryption controllers#2086
CNTRLPLANE-2241: Add KMS encryption mode into library-go encryption controllers#2086ardaguclu wants to merge 3 commits intoopenshift:masterfrom
Conversation
|
@ardaguclu: This pull request references CNTRLPLANE-2241 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@ardaguclu: This pull request references CNTRLPLANE-2241 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ardaguclu The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@ardaguclu: This pull request references CNTRLPLANE-2241 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
377c3cb to
0b007f9
Compare
|
/hold
|
|
/uncc @dgrisonnet |
d0321f6 to
b8208c8
Compare
|
/retest |
|
/retitle CNTRLPLANE-2241: Add KMS encryption mode into library-go encryption controllers |
p0lyn0mial
left a comment
There was a problem hiding this comment.
left a few more questions.
overall it lgtm
| // Since in v1 we don't support kms -> kms migrations, we can use empty key. | ||
| // Because we don't need to compare secrets to detect change. | ||
| secret := crypto.ModeToNewKeyFunc[state.KMS]() | ||
| keyId := strings.Split(provider.KMS.Name, "-")[0] |
There was a problem hiding this comment.
I’m still wondering about encapsulation here.
I think other parts of the code won’t even be aware that the name has changed.
Only the code in encryptionconfig/config.go will encode and decode the new name, right?
Do we need to ensure that the name doesn’t already contain "-"?
For simplicity, we could add a comment in the key generation function describing this requirement.
There was a problem hiding this comment.
Only the code in encryptionconfig/config.go will encode and decode the new name, right?
That is correct. Rest of the code only deals with KeyState. They do not even know about the details of EncryptionConfiguration.
Do we need to ensure that the name doesn’t already contain "-"?
Kube-apiserver already fails, if it encounters a provider name without -. Because uniqueness has broken.
For simplicity, we could add a comment in the key generation function describing this requirement.
I've added a comment about the reasons behind this provider name generation mechanism. Please let me know your thoughts. Thanks.
| func getKeyIDFromProviderName(providerName string) string { | ||
| // We just need to obtain the keyID to find our backed secret | ||
| // e.g. "1-secrets" | ||
| return strings.Split(providerName, "-")[0] |
There was a problem hiding this comment.
How should this handle an unexpected provider name?
There was a problem hiding this comment.
Although it may not be possible, it is better to handle the invalid provider name gracefully. I've updated the code. Please let me know your thoughts.
| }, | ||
| }) | ||
| case state.KMS: | ||
| // In order to preserve the uniqueness, we should insert resource name |
There was a problem hiding this comment.
OK, "kms v2 providers name must always be unique across all kms providers (v1 and v2)":
There was a problem hiding this comment.
Are there going to be side effects from this naming scheme visible in things like metric labels?
There was a problem hiding this comment.
All the metrics that is about provider_name will be affected https://github.com/kubernetes/kubernetes/blob/90a76aaa9afbe9cdb3df5f0a4356018b6637648b/staging/src/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/metrics/metrics.go#L169.
| func createKMSProviderName(keyID, resource string) string { | ||
| // Ideally we should have used keyId simply in kms provider name. | ||
| // However, this is an upstream constraint that every provider name must be unique. | ||
| // To maintain uniqueness while still allowing access to the keyId, we generate provider name in this format. |
There was a problem hiding this comment.
OK, this is at least less strange than smuggling identity names in phony AES-GCM provider configs. Later, I would be interested in coming up with alternatives that let the controllers correlate provider configs to their source secret without smuggling.
There was a problem hiding this comment.
I agree with you. It was error-prone and not nice. Since now we carry KMSConfigurations in backed Secrets, I think we won't need to carry anything in provider_names.
| // Moreover, we don't trigger new key when external reason is changed. | ||
| // Because it would lead to duplicate providers which is not allowed. |
There was a problem hiding this comment.
IIUC, even if we did roll out a new encryption config in response to the "reason" update, we're not even capable of triggering a KEK rotation unless a specific provider happened to expose that capability independently.
There was a problem hiding this comment.
Good point. That is correct.
| const ( | ||
| DefaultEndpoint = "unix:///var/run/kmsplugin/kms.sock" | ||
| DefaultTimeout = 10 * time.Second | ||
| ) |
There was a problem hiding this comment.
Is there a real need to export these constants? IMO, duplicating the values in tests is good because it requires that changes in observable behavior are accompanied by changes to test expectations. And other code (doing related things like setting up volume mounts) should be observing effective values rather than using hardcoded defaults.
There was a problem hiding this comment.
The only reason is to be accessed by key_controller. If we move these constants in there, we don't need to export them.
There was a problem hiding this comment.
could we just inline these in key_controller?
| if currentMode == state.KMS { | ||
| ks.KMSConfiguration = &apiserverv1.KMSConfiguration{ | ||
| APIVersion: "v2", | ||
| Name: fmt.Sprintf("%d", keyID), // this will be updated by inserting resource name |
There was a problem hiding this comment.
given that the name will be changed only in the EncryptionConfiguration and will not be visible to the KeyState, is this comment still relevant?
maybe we should add a comment about the expected key format, which must include the keyID (at least for now)?
There was a problem hiding this comment.
Comment is not relevant anymore. I've removed it.
| ks = state.KeyState{ | ||
| Key: apiserverconfigv1.Key{ | ||
| Name: keyId, | ||
| // Since in v1 we don't support kms -> kms migrations, we can use empty key. |
There was a problem hiding this comment.
Not sure if I understand this comment.
For KMS the Secret doesn't matter and can/will always be empty - because the actual encryption key material is in the external KMS instance.
There was a problem hiding this comment.
In v2, we'll have to compare this field (or KMSConfiguration annotation) to detect configurational changes. But it sounds like this comment creates confusions. So I've removed it.
|
Unrelated? |
| func getKeyIDFromProviderName(providerName string) (string, error) { | ||
| // We just need to obtain the keyID to find our backed secret | ||
| // e.g. "1_secrets" | ||
| parsed := strings.Split(providerName, "_") |
There was a problem hiding this comment.
should we use strings.SplitN ?
| secret := CreateEncryptionKeySecretWithRawKeyWithMode(targetNS, grs, keyID, emptyKey, "KMS") | ||
| kmsConfig := &apiserverconfigv1.KMSConfiguration{ | ||
| APIVersion: "v2", | ||
| Name: fmt.Sprintf("%d_%s", keyID, grs[0].Resource), |
There was a problem hiding this comment.
shouldn't this be fmt.Sprintf("%d", keyID) ?
the resource suffix is added later.
| }) | ||
| case state.KMS: | ||
| // In order to preserve the uniqueness, we should insert resource name | ||
| kmsCopy := key.KMSConfiguration.DeepCopy() |
There was a problem hiding this comment.
should we check key.KMSConfiguration for nil ?
There was a problem hiding this comment.
Although this is not possible, it makes sense gracefully handling the empty KMSConfiguration by skipping it with a message. Please let me know your thoughts.
There was a problem hiding this comment.
yes, that would match the existing pattern. good idea.
| } | ||
|
|
||
| if ks.KMSConfiguration != nil { | ||
| ks, err := json.Marshal(ks.KMSConfiguration) |
There was a problem hiding this comment.
nit: ks shadows the outer ks, we could rename to ksJSON
| }) | ||
| case state.KMS: | ||
| if key.KMSConfiguration == nil { | ||
| klog.Infof("skipping key %s in KMS mode as its KMSConfiguration is nil", key.Key.Name) |
There was a problem hiding this comment.
should we add resource and index too?
There was a problem hiding this comment.
I've added resource name in the log.
Index is not the index in EncryptionConfiguration providers. It is an index in the Backed Secrets. We have keyID already as discriminator. Would you still want me to insert index name?
|
trying to find the test test failed: vs |
|
All tests pass on my local. According to the junit results, all the tests passed in the CI jobs too https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_library-go/2086/pull-ci-openshift-library-go-master-unit/2023755244520869888/artifacts/test/artifacts/junit_report.xml. It looks like a transient failure; |
|
@ardaguclu: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
| return providers | ||
| } | ||
|
|
||
| func createKMSProviderName(keyID, resource string) string { |
There was a problem hiding this comment.
just a comment: as of today there a no restrictions on length for the provider name. the only restriction i found is that we are not allowed to use :
| if currentMode == state.KMS { | ||
| // We are here because Encryption Mode is not changed | ||
|
|
||
| // For now in v1, we don't support configurational changes. Therefore, |
There was a problem hiding this comment.
optional: someone reading v1 might get confused, maybe it's better to specify TP as well
| AESGCM Mode = "aesgcm" | ||
| SecretBox Mode = "secretbox" // available from the first release, see defaultMode below | ||
| Identity Mode = "identity" // available from the first release, see defaultMode below | ||
| KMS Mode = "KMS" // only supports KMS v2 |
There was a problem hiding this comment.
nit: all the others are lowercase ("KMS")
| keyMode := state.Mode(s.Annotations[encryptionSecretMode]) | ||
| switch keyMode { | ||
| case state.AESCBC, state.AESGCM, state.SecretBox, state.Identity: | ||
| case state.AESCBC, state.AESGCM, state.SecretBox, state.Identity, state.KMS: |
There was a problem hiding this comment.
Do you also need to add a check to test if keyMode is KMS and key.KMSConfiguration is nil?
This PR adds the implementation proposed in openshift/enhancements#1900.
This PR introduces KMS as a new mode in encryption controllers to be functioning along side with the other modes. Basically, the idea is to track the KMS configuration to detect any configuration changes. So that key_controller can create new Secret to initiate migration. KMS Secret will not contain any confidential data. It will simply store the KMS configuration.
For Tech Preview v1, we will only support static unix domain socket (i.e.
unix:///var/run/kmsplugin/kms.sock). Consequently, in this version, we don't support KMS -> KMS migrations.Ref: Previous attemps:
Notes: