feat: add affinity, toleration, and nodeSelector for operator deployment

[a1994sc]

Describe what this PR does

Adds affinity, toleration, and nodeSelector values for operator deployment

Fixes: #issue_number

Future concerns

List items that are not part of the PR and do not impact it's
functionality, but are work items that can be taken up subsequently.

Checklist:

• Commit Message Formatting: Commit titles and messages follow
  guidelines in the developer
  guide.
• Reviewed the developer guide on Submitting a Pull
  Request
• Pending release
  notes
  updated with breaking and/or notable changes for the next major release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[Madhu-1]

@a1994sc Ci is failing due to checks, the helm chart is auto generated using helmify, please check if we have option in helmify to set these values or else we need to add these to the manager https://github.com/ceph/ceph-csi-operator/blob/main/config/manager/manager.yaml to get it auto generated as well

[a1994sc]

@Madhu-1 gotcha, I will check and report back

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

[a1994sc]

Sorry got a bit swamped with other matters, but still trying to look into this... I thought that upgrading to v0.4.19 of helmify might habd helped, but it looks like it does not respect having multiple service accounts, so may need to address that before upgrading... May take a bit longer to address this PR then I thought

[JPaja]

I had luck with nodeSelector field by adding empty nodeSelector to BUILD_INSTALLER_OVERLAY

    - op: add
      path: /spec/template/spec/nodeSelector
      value: {}

Sadly this doesn't work for toleration and affinity

Full BUILD_INSTALLER_OVERLAY

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: $(NAMESPACE)
namePrefix: $(NAME_PREFIX)
patches:
- patch: |-
    - op: add
      path: /spec/template/spec/containers/0/env/-
      value:
        name: CSI_SERVICE_ACCOUNT_PREFIX
        value: $(NAME_PREFIX)
    - op: add
      path: /spec/template/spec/containers/0/env/-
      value:
        name: WATCH_NAMESPACE
        value: $(WATCH_NAMESPACE)
    - op: add
      path: /spec/template/spec/nodeSelector
      value: {}
  target:
    kind: Deployment
    name: controller-manager
images:
- name: controller
  newName: ${IMG}

[JPaja]

Okay so toleration defaults was added 2 months ago
arttor/helmify#175

And now I created helmify pr to add support for affinity
arttor/helmify#189

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

[a1994sc]

@JPaja thank you so much for taking up the torch on this, please let me know if you need any support

[JPaja]

@JPaja thank you so much for taking up the torch on this, please let me know if you need any support

Sadly no one is reviewing pr that adds affinity. I guess this or should drop affinity for now