Upgrade Karpenter

[seanmorton]

Context

• Karpenter is our node autoscaler, for VMs virt-handler is a daemon set starting after a node is provisioned, mutating nodes with labels and resource capacities that Karpenter can't know upfront making VMs unschedulable
• We maintained a patch for 2 years ignoring static resource/capacity requirements and regex-based node selectors, we don't want to keep maintaining
• Karpenter just released a NodeOverlay CRD allowing us to flag static custom resource/capacity requirements upfront for nodes we want eligible through well-known labels, that solves half of the problem

Goals

Upgrade Karpenter from our "old" patch v1.3.3x to v1.7.x (latest) replacing:

• IGNORED_RESOURCE_REQUESTS with NodeOverlay now it's built-in
• IGNORED_NODE_SELECTOR_REQUIREMENTS with an additive way of handling node selectors having dynamic label keys (e.g., scheduling.node.kubevirt.io/tsc-frequency-2999998000=true and initiate buy-in with Karpenter team

Patch we applied to Karpenter

Diff is available here

Forks are available on the following repos:

• https://github.com/pelotech/karpenter
• https://github.com/pelotech/karpenter-provider-aws

How we configured the custom patch

env:
   - name: IGNORED_RESOURCE_REQUESTS
      value: "devices.kubevirt.io/kvm,devices.kubevirt.io/tun,devices.kubevirt.io/vhost-net"
   - name: IGNORED_NODE_SELECTOR_REQUIREMENTS
      value: "scheduling.node.kubevirt.io/tsc-frequency-*"

(source)

We call then this configured Karpenter fork in clusters where we need it as on:

• https://github.com/pelotech/infrastructure

What's new with Karpenter 1.7.x

This would indicate to Karpenter these custom capacity requirements are available to any eligible node returned by the AWS API; in this case, we make eligible any metal instance size

apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
 name: kubevirt
spec:
  requirements:
  - key: karpenter.k8s.aws/instance-size
    operator: In
    values: ["metal"]
  capacity: 
    devices.kubevirt.io/kvm: 1k
    devices.kubevirt.io/tun: 1k
    devices.kubevirt.io/vhost-net: 1k

How to reproduce the scheduling.node.kubevirt.io/tsc-frequency-* issue

To interact with VMs, install virtctl

1. Start a VM (no node selector constraint because it's the first VM boot, libvirt XML will at that point initialize and hardcode the value read from the node)
2. Stop your VM: kubectl virt stop $vm_name
3. Resume your VM: kubectl virt start $vm_name

(I don't remember if this tsc-frequency node selector constraint appears only on Windows VMs)

[tallquist10]

@chomatdam and I did pretty extensive testing in the core-prod cluster to recreate the tsc-frequency issue. The annotation was only found on Windows images, out of the different ones we tested. We tested starting and stopping VMs, stopping nodes directly using EC2 console, and using kubectl to delete nodes, and all of the VMs correctly scheduled onto nodes. As a result, foundation v2.9.0 uses regular Karpenter 1.7.1 instead of a custom patch. Barring issues in client clusters similar to before, at which point we would pursue a patch to enable patterns for node selectors, this is considered complete.

[tallquist10]

Update as of 2025-10-28

In ot-prod-1, Karpenter 1.7.1 is quite reliable, enduring scenarios like shutdown of multiple nodes, as well as quick cycles of labs spinning up and down. In general, it's very good at spinning up new nodes. In 3 cases over the last ~2 weeks, Karpenter has not been able to gracefully spin up new nodes, and labs were stuck in the scheduling phase with seemingly no means of recovering. Spinning the lab down and redeploying it fixes the issue. One thing we tried was adding various tsc-frequency labels that we've previously seen so that they're applied to the nodes, and then we're more likely to have nodes available with the required tsc-frequency node selector. Unfortunately, the labels are removed from the node upon sync because the tsc-frequency labels are assigned to nodes as a result of their ACTUAL tsc-frequency. The nodes sync on a schedule, so those labels will only consist of tsc-frequency values that are available during the lifespan of that node.

Additionally, bugs in the kube-ovn-controller interfere with pods spinning up, typically in instances where AddressConflict errors arise. This has caused a few false positives with pod issues that could have seemingly been caused by Karpenter. Additionally, interactions between Kubevirt and Karpenter in how labels are added to nodes, as well as nodeSelectors for the pods and VMIs, add complexity that has hindered me fromn finding a clear culprit at this time.

The current theory is that if we remove the windows.10.virtio VirtualMachineClusterPreferences from the lab-netspec Helm chart, we can remove the tsc-frequency node selectors from the pods, but because metal nodes have the tsc-frequency on them inherently, there is a chance that this may not fix the issue either.