fix: Optionally set provisioning.cattle.io/externally-managed annotation

[yiannistri]

What this PR does / why we need it:

This PR adds a feature gate that controls whether the provisioning.cattle.io/externally-managed annotation is set in imported clusters. By default, the annotation is not set (disabled) and the user needs to opt-in (enable) in order for the annotation to be added.

To test this feature gate, a new test suite was introduced that installs the Providers chart on a new management cluster. The chart has been updated as part of this PR to no longer install CAAPF by default. The test sets the RKE2_CNI env var to "" so that RKE2 defaults to installing Canal as its CNI without relying on CAAPF (using none means no CNI is installed.

Which issue(s) this PR fixes:
Fixes #2156
Fixes #2179

Special notes for your reviewer:
Please also review related docs PR

Checklist:

• squashed commits into logical changes
• includes documentation
• adds unit tests
• adds or updates e2e tests

[anmazzotti]

I'm a bit worried about the changes here:

1. I think this is a regression of #622 probably leading to registering 2 fleet-agents
2. CAAPF is still enabled by default in the providers chart. Having the related Turtles flag off by default creates an inconsistency.
3. There is no documentation PR for this new feature.
4. There is no e2e test coverage for this new feature.
5. Documentation updates are required for this major change, like:

• https://turtles.docs.rancher.com/turtles/stable/en/user/caapf.html
• https://turtles.docs.rancher.com/turtles/stable/en/user/gitops.html
• https://turtles.docs.rancher.com/turtles/stable/en/user/clusterclass.html
• https://github.com/rancher/turtles-product-docs/blob/main/versions/v0.26/modules/en/pages/user/providers.adoc

[salasberryfin]

1. I think this is a regression of #622 probably leading to registering 2 fleet-agents

As stated in the original issue #2156, this feature and using CAAPF are mutually exclusive and this should be properly documented.

Documentation

There must be proper documentation that informs users of the mutually exclusive nature of these features:

• Default behavior does not support using CAAPF reliably.
• Opting to use CAAPF means that Fleet is no longer managed by Rancher but CAAPF.

One could argue that we had a bug (or a pending change) in Turtles after removing CAAPF from the Turtles chart, since users are no longer getting the Fleet agent installed on their workload clusters.

For any existing user of Turtles+CAAPF (if there are any) that wants to continue using the add-on provider, they should follow the documentation to set the feature gate accordingly.

2. CAAPF is still enabled by default in the providers chart. Having the related Turtles flag off by default creates an inconsistency.

Good catch. This should be disabled by default from now on: only users that explicitly want to use CAAPF should have it installed. #2179

3. There is no documentation PR for this new feature.

4. Documentation updates are required for this major change, like:

This is a valid concern. My suggested approach to documentation here is:

• Document this new feature and the default behavior (including what I mentioned in my response to question 1.).
• Add a note to any example in the documentation that requires CAAPF (technically all of them for now) that the previous point is a requirement to use the add-on provider.
• Progressively we will add new examples that don't rely on CAAPF (and this will probably be given more visibility in the docs) while maintaining the existing CAAPF-specific examples, until the time comes to fully remove them.

Only an idea. Feel free to suggest alternatives.

[anmazzotti]

@salasberryfin I believe that everything should be addressed in the same PR. Splitting #2179 is leading to an inconsistent environment.

All changes are dependent on each other.

Also I don't entirely understand right now the impact of this change on current users.
What does it mean to disable CAAPF for already provisioned Clusters? How are users going to manage the existing Fleet Clusters that have been created by CAAPF? How can they migrate away from it? And do we have any test coverage to validate this scenario?

[salasberryfin]

@salasberryfin I believe that everything should be addressed in the same PR. Splitting #2179 is leading to an inconsistent environment.

All changes are dependent on each other.

Makes sense to me. Can you add this to the current PR @yiannistri?

Also I don't entirely understand right now the impact of this change on current users. What does it mean to disable CAAPF for already provisioned Clusters? How are users going to manage the existing Fleet Clusters that have been created by CAAPF? How can they migrate away from it? And do we have any test coverage to validate this scenario?

It's important to note that CAAPF is not removed. Users are free to keep using it (and will be for months) but they will be required to explicitly configure the charts to use the add-on provider:

• rancher-turtles using this feature.
• rancher-turtles-providers enabling CAAPF.

To increase our confidence on this feature I'm thinking we can extend the existing Chart upgrade test and add a validation to confirm that there's no duplication of Fleet clusters after upgrading and explicitly setting the feature to let Fleet be externally managed. Would this help to address the impact of this change?

We have plenty of time until CAAPF is removed to carefully plan a migration strategy for users of CAAPF.

[anmazzotti]

Removing request for changes to let other people review.

Letting other people review.

[kkaempf]

#2156 was changed from enhancement to bug.
This PR's title should be adapted accordingly (remove feat: 😉 )