Missing ownerReferences on resources created by operator

[Philbow]

Dear maintainers,

all operators in our customers clusters are installed via a gitops approach using OpenShift GitOps (ArgoCD) and the operators-installer.

After cert-manager operator installation we detected, that OpenShift GitOps does not correctly identify resources created by the cert-manager operator itself and tries to delete these, as automatic pruning was enabled. With disabled pruning our OpenShift GitOps instance indicates an out of sync state on the managed cert-manager instance (resources created, that are not defined via our gitops repository).

Reason for this seems to be a missing metadata.ownerReference on all resources created by the operator itself. Kubernetes uses this field to keep track of the managing component, which should be set according to the best practices.

All other operator installed via the OpenShift marketplace (OLM), reference the ClusterServiceVersion resource in the ownerReference, to keep track of automatically created resources. Here an example from the OpenShift Pipelines Operator (webhook deployment resource), that is correctly recognized as a dependency in OpenShift GitOps:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    olm.deployment-spec-hash: 4ZTxOziFrU9OpCxA21IKr6rMlYGisUgO0yXer3
    olm.managed: 'true'
    olm.owner: openshift-pipelines-operator-rh.v1.19.3
    olm.owner.kind: ClusterServiceVersion
    olm.owner.namespace: pipelines-operator
    operator.tekton.dev/release: v0.76.0
    operators.coreos.com/openshift-pipelines-operator-rh.pipelines-operator: ''
    version: v0.76.0
  name: tekton-operator-webhook
  namespace: pipelines-operator
  ownerReferences:
    - apiVersion: operators.coreos.com/v1alpha1
      blockOwnerDeletion: false
      controller: false
      kind: ClusterServiceVersion
      name: openshift-pipelines-operator-rh.v1.19.3
      uid: a17e8225-96bc-42c2-a4c4-c1a9f19e334a
spec:
  replicas: 1
  ...

As soon as an ownerReference is defined and the corresponding parent resource is managed by gitOps (e.g. ClusterServiceVersion), OpenShift GitOps automatically recognizes this resource as part of cert-managers application.

Here is the full list of missing ownerReferences, where a missing ownerReference was identified:

rbac.authorization.k8s.io/v1/ClusterRole - cert-manager-cluster-view
rbac.authorization.k8s.io/v1/ClusterRole - cert-manager-controller-approve:cert-manager-io
rbac.authorization.k8s.io/v1/ClusterRoleBinding - cert-manager-controller-orders
rbac.authorization.k8s.io/v1/RoleBinding - cert-manager-cainjector:leaderelection
rbac.authorization.k8s.io/v1/ClusterRole  - cert-managercontroller-challenges
v1/Service - cert-manager
v1/Service - cert-manager-cainjector
v1/Service - cert-manager-webhook
rbac.authorization.k8s.io/v1/ClusterRoleBinding - cert-manager-cainjector
admissionregistration.k8s.io/v1/MutatingWebhookConfiguration - cert-manager-webhook
admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration - cert-manager-webhook
apiextensions.k8s.io/v1/CustomResourceDefinition - issuers.cert-manager.io 
v1/ServiceAccount - cert-manager-cainjector
rbac.authorization.k8s.io/v1/ClusterRoleBinding - cert-manager-controller-certificatesigningrequests
rbac.authorization.k8s.io/v1/ClusterRole - cert-manager-edit
rbac.authorization.k8s.io/v1/ClusterRoleBinding - cert-manager-controller-approve:cert-manager-io
rbac.authorization.k8s.io/v1/ClusterRole - cert-manager-controller-clusterissuers
rbac.authorization.k8s.io/v1/ClusterRole - cert-manager-controller-certificates
apiextensions.k8s.io/v1/CustomResourceDefinition - clusterissuers.cert-manager.io 
rbac.authorization.k8s.io/v1/ClusterRole  - cert-manager-controller-ingress-shim
apps/v1/Deployment - cert-manager
rbac.authorization.k8s.io/v1/RoleBinding - cert-manager-cert-manager-tokenrequest
rbac.authorization.k8s.io/v1/ClusterRoleBinding - cert-manager-webhook:subjectaccessreviews
v1/ServiceAccount - cert-manager-webhook
rbac.authorization.k8s.io/v1/Role - cert-manager-cainjector:leaderelection
rbac.authorization.k8s.io/v1/Role - cert-manager-tokenrequest
rbac.authorization.k8s.io/v1/Role - cert-manager-webhook:dynamic-serving
rbac.authorization.k8s.io/v1/Role - cert-manager:leaderelection
rbac.authorization.k8s.io/v1/ClusterRole - cert-manager-controller-certificatesigningrequests
rbac.authorization.k8s.io/v1/RoleBinding - cert-manager-webhook:dynamic-serving
rbac.authorization.k8s.io/v1/RoleBinding - cert-manager:leaderelection
rbac.authorization.k8s.io/v1/ClusterRole - cert-manager-controller-issuers
rbac.authorization.k8s.io/v1/ClusterRole - cert-manager-controller-orders
rbac.authorization.k8s.io/v1/ClusterRole - cert-manager-view
rbac.authorization.k8s.io/v1/ClusterRole - cert-manager-webhook:subjectaccessreviews
apps/v1/Deployment - cert-manager-cainjector
apps/v1/Deployment - cert-manager-webhook
rbac.authorization.k8s.io/v1/ClusterRoleBinding - cert-manager-controller-certificates
rbac.authorization.k8s.io/v1/ClusterRoleBinding - cert-manager-controller-challenges
rbac.authorization.k8s.io/v1/ClusterRoleBinding - cert-manager-controller-clusterissuers
rbac.authorization.k8s.io/v1/ClusterRoleBinding - cert-manager-controller-ingress-shim
rbac.authorization.k8s.io/v1/ClusterRoleBinding - cert-manager-controller-issuers
v1/ServiceAccount - cert-manager

Please add an appropriate ownerReference to support Kubernetes best practices and OpenShift GitOps / ArgoCD deployments. This could be either the 'ClusterServiceVersion' or 'CertManager' CRD.

BR,

Philipp

[Philbow]

Hi,

I did a bit more analysis on the issue, as this argoCD behaviour was new to me.

By default, when resources are created in a namespace outside of gitOps, argoCD ignores these as orphaned resources.

However argoCD uses the label app.kubernetes.io/instance to track all resources, that belong to an application. As my argoCD application was called cert-manager, it started to track resources, that were outside of gitOps as they had the same label / value pair.

For now I have renamed the application, in order to get rid of this problem.

Still the ownerReference might be something, worth a looking into the future to match Kubernetes best practices. It would also have solved my problem, by referencing the parent owner in gitOps (instead of all subresources).

Feel free to close this issue at your own decision ;).

BR,

Philipp

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[ibotty]

Just stumbled on this issue.
This is intentional. See https://cert-manager.io/docs/usage/certificate/#cleaning-up-secrets-when-certificates-are-deleted

I am sure you can configure the cluster certmanager to pass --enable-certificate-owner-ref in .spec.controllerConfig (or as a last means in .spec.unsupportedConfigOverrides).

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[bharath-b-rh]

This behavior is by design. By not setting ownerReferences, we avoid automatic garbage collection that could lead to unintended downtime for applications relying on these certificates. For now, cleanup remains a manual process to ensure data persistence. We are tracking a feature request to make this behavior configurable/declarative in the future.