Use Konnectivity for pod network access from bootstrap KAS

[mdbooth]

Enables kube-apiserver running on the bootstrap node to access the pod network,
specifically to enable access to webhooks running in the cluster.

Changes:

• Adds a new static Konnectivity server pod running on the bootstrap node
• Configures the bootstrap KAS to use its local Konnectivity server for
  outbound cluster traffic
• Add a daemonset deployed into the cluster to run Konnectivity agent on every
  cluster node
• Removes daemonset automatically in bootstrap teardown

┌──────────────────────┐               ┌──────────────────────┐
│  KAS (bootstrap)     │               │  Konnectivity Agent  │
│         │            │               │         │            │
│         ▼            │               │         │            │
│  UDS Socket          │               │         │            │
│         │            │               │         │            │
│         ▼            │   TCP:8091    │         ▼            │
│  Konnectivity Server │◄──────────────│  Connect to server   │
│         │            │               │         │            │
│         ▼            │               │         │            │
│  Route to Pod IPs    │               │  Pod Network Access  │
└──────────────────────┘               └──────────────────────┘

Slack thread: https://redhat-internal.slack.com/archives/C68TNFWA2/p1769796074939399?thread_ts=1769601349.942229&cid=C68TNFWA2

tl;dr This is a PoC, but it works (on AWS) in my manual testing.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rwsu for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[patrickdillon]

Awesome!

[patrickdillon]

konnectivity cleanup/delete should be moved to the end of the bootkube script. The cleanup should be part of bootstrapping, so that bootstrapping is not done until konnectivity is cleaned up--rather than cleanup being part of bootstrap destroy.

At present, at least in most cases (I'm not aware of any exceptions), we can successfully provision clusters without the installer connecting to the API LB.

[mdbooth]

As long as we don't dismantle konnectivity until the after the bootstrap KAS is gone this is fine.

[mdbooth]

Incidentally, I also wondered if it might be cleaner to put all these resources in an openshift-bootstrap namespace. Then we can just delete the namespace on teardown instead of having to keep track of multiple objects.

Also, if you think there's any mileage in moving the agent definition to an asset, we could have bootstrap.sh just write its IP to a configmap instead of creating the daemonset.

[mdbooth]

/test ?

[mdbooth]

/test e2e-aws-default-config e2e-aws-ovn-techpreview

[openshift-ci]

@mdbooth: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.