Neuvector, a Private CA, and Keycloak (Possible Fix)

[cookew]

Problem

Neuvector does not have a method to use a private CA certificate (PEM encoded) when using OIDC authentication. For example, having Neuvector use Keycloak that is enforcing TLS and is using a certificate issued from a private certificate authority.

Reference: When does Keycloak require SSL? section in a SUSE support document.

A few possible Fixes

To allow Neuvector to authenticate to Keycloak using a private CA, you have to somehow get your private CA certificate into the /etc/ssl/certs/ca-certificates.crt (a list of PEM encoded certificates) file in the neuvector-controller-pod deployment.

A few options are:

1. Mount your CA certificate as the /etc/ssl/certs/ca-certificates.crt file from a configmap, secret, or another volume.
2. Using a trust-manager bundle, mount the bundle as the /etc/ssl/certs/ca-certificates.crt file from a configmap, secret, or another volume.
3. Use an init container and a small emptyDir volume, add your certificate to the already existing /etc/ssl/certs/ca-certificates.crt file.

Possible Issue

I am unsure if using only one certificate as the /etc/ssl/certs/ca-certificates.crt file will break anything else. A lot of things in Linux fall back to using this file, which is probably why this works to fix OIDC. If there is anything else in the neuvector-controller-pod deployment that pulls anything from a public website, it could potential fail when using TLS because it would not have all the normal pulic certificates. The /etc/ssl/certs/ca-certificates.crt file is normally updated with some sort of update-ca-certificates script, but that cannot be used in a properly secured pod because the filesystem should normally be read-only and the pod should be running as a non-root user.

Ideas to make it easier for users who use private CAs with OIDC

1. Add a way to specify custom certificates to be used when using OIDC, like there is for SAML and LDAP.
2. Add a method into the Helm chart for adding certificates into the /etc/ssl/certs/ca-certificates.crt file.
3. Update the documentation for using a private CA certificate with OIDC.
4. Update the When does Keycloak require SSL? section in the SUSE support document to link to the updated documentation.

Other Keycloak Notes

The issuer Issuer URL in the oidcinitcfg.yaml file for Keycloak should be everything up to the .well-known part. I couldn't find anything about if the .well-known part of the OIDC issuer URL needed to be included in the online documentation for Neuvector.

This is correct, if your realm name is myrealm: https://keycloak.example.com/realms/myrealm

This is incorrect: https://keycloak.example.com/realms/myrealm/.well-known/openid-configuration

Example Manifests

This is mostly taken from my setup in a working environment.

Tested with:

• Kubernetes 1.35.0
• Keycloak 26.5.2
• Neuvector Helm Chart 2.8.10
• Neuvector 5.4.8

# kustomization.yaml
# Kustomize file to deploy everything
# kustomize build --enable-helm . | kubectl apply -f -
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: neuvector
resources:
# Neuvector configuration file
- neuvector-init.yaml
# cert-manager certificate requests
- neuvector-certs.yaml

# Use only ONE patch!!! Comment out the other!!!
# There are notes in the patches for how to use them.
patches:
# This patch is for using only your ca certificate (Fix method 1 and 2)
# - path: neuvector-controller-pod-ca-mount.yaml
#   target:
#     group: apps
#     kind: Deployment
#     name: neuvector-controller-pod
#     version: v1
# This patch is for using a sidecar container to combine the existing certs in /etc/ssl/certs with yours (Fix method 3)
 - path: neuvector-controller-pod-init-container.yaml
  target:
    group: apps
    kind: Deployment
    name: neuvector-controller-pod
    version: v1

# https://github.com/neuvector/neuvector-helm
helmCharts:
- name: core
  includeCRDs: true
  namespace: neuvector
  releaseName: neuvector
  repo: https://neuvector.github.io/neuvector-helm/
  valuesFile: values.yaml
  version: 2.8.10

# neuvector-controller-pod-ca-mount.yaml
# Using trust-manager, or any config map
# This adds a volumeMount to the container
- op: add
  path: /spec/template/spec/containers/0/volumeMounts/-
  value:
    mountPath: /etc/ssl/certs/ca-certificates.crt
    name: ca
    readOnly: true
    # Change this to the key name in the config map that holds the CA certificate
    subPath: ca-bundle.pem
# This adds the volume to the pod, which is used by the container
- op: add
  path: /spec/template/spec/volumes/-
  value:
    configMap:
      defaultMode: 420
      # Change this to your config map name that holds the private CA certificates
      name: tm-ca-bundle
    name: ca

# neuvector-controller-pod-init-container.yaml
# These four patches add private CA certificates to the neuvector-controller pod
# 1. Using init container to append private CA certificates
- op: add
  path: /spec/template/spec/initContainers/-
  value:
    args:
    - |
      echo Certificates in /private-ca-certificates:
      ls -al /private-ca-certificates
      echo Existing CA certificates in /etc/ssl/certs:
      ls -al /etc/ssl/certs/
      echo Combining CA certificates
      for cert in /etc/ssl/certs/*.{0,pem} /private-ca-certificates/*
      do
        echo "Adding cert: $cert"
        cat "${cert}" >> /new-ca-certificates/ca-certificates.crt
        echo "" >> /new-ca-certificates/ca-certificates.crt
      done
      echo New CA certificates file:
      ls -al /new-ca-certificates
    command: ["/usr/bin/bash", "-c"]
    image: docker.io/neuvector/controller:5.4.8
    name: ca-certificate-fixer
    volumeMounts:
    - mountPath: /new-ca-certificates
      name: new-ca-certificates
      readOnly: false
    - mountPath: /private-ca-certificates
      name: private-ca-certificates
      readOnly: true

# 2. The volume to hold the new CA certificates
- op: add
  path: /spec/template/spec/volumes/-
  value:
    emptyDir:
      medium: Memory
      sizeLimit: 5Mi
    name: new-ca-certificates

# 3. The volume for the private CA certificates
- op: add
  path: /spec/template/spec/volumes/-
  value:
    configMap:
      defaultMode: 420
      # Change this to your config map name that holds the private CA certificates
      name: tm-ca-bundle
    name: private-ca-certificates

# 4. Mount the volume to hold the combined CA certificates
- op: add
  path: /spec/template/spec/containers/0/volumeMounts/-
  value:
    mountPath: /etc/ssl/certs/ca-certificates.crt
    name: new-ca-certificates
    readOnly: true
    subPath: ca-certificates.crt

# values.yaml
# The values file for Helm
# https://github.com/neuvector/neuvector-helm/tree/master/charts/core

# I use my own cert-manager to generate the certs
autoGenerateCert: false
controller:
  certificate:
    pemFile: tls.crt
    secret: neuvector-controller-cert
  disruptionBudget: 0
  pvc:
    enabled: true
    storageClass: ceph-filesystem
    capacity: 1Gi
cve:
  adapter:
    enabled: true
defaultValidityPeriod: 40
manager:
  certificate:
    secret: neuvector-manager-cert
    pemFile: tls.crt
  ingress:
    annotations:
      cert-manager.io/cluster-issuer: my-cert-manager
      ingress.cilium.io/tls-passthrough: "enabled"
    enabled: true
    host: neuvector.examle.com
    ingressClassName: cilium
    secretName: neuvector-ingress-manager-cert
    tls: true
  probes:
    enabled: true
  route:
    enabled: false
  svc:
    type: ClusterIP

# neuvector-certs.yaml
# Generate the certificates
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: neuvector-controller-cert
spec:
  commonName: neuvector-controller-cert
  dnsNames:
  - "neuvector-controller.example.com"
  - "neuvector-svc-controller.neuvector.svc.cluster.local"
  - "neuvector-svc-controller.neuvector.svc"
  duration: 960h
  isCA: false
  issuerRef:
    kind: ClusterIssuer
    name: my-cert-manager
  secretName: neuvector-controller-cert
  usages:
  - client auth
  - server auth
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: neuvector-manager-cert
spec:
  commonName: neuvector-manager-cert
  dnsNames:
  - "neuvector.example.com"
  - "neuvector-service-webui.neuvector.svc.cluster.local"
  - "neuvector-service-webui.neuvector.svc"
  duration: 960h
  isCA: false
  issuerRef:
    kind: ClusterIssuer
    name: my-cert-manager
  secretName: neuvector-manager-cert
  usages:
  - client auth
  - server auth

# neuvector-init.yaml
# https://open-docs.neuvector.com/deploying/production/configmap/
# You should be using bitnami sealed-secrets or some other secure method of deploying this to your cluster due to a secret string being contained within the secret.
# Work will need to be done in Keycloak to map roles or groups, to the client_roles claim, or however you decide to use roles and groups.
# I have a neuvector-admin client role that I assign to users. This client role is mapped to the client_roles claim. I do not pass any groups through to Neuvector in any claim.
apiVersion: v1
kind: Secret
metadata:
  name: neuvector-init
stringData:
  oidcinitcfg.yaml: |
    always_reload: true
    Issuer: https://keycloak.example.com/realms/myrealm
    Client_ID: neuvector
    # The secret comes from Keycloak
    Client_Secret: 1234567890abcdef1234567890abcdef
    Group_Claim: client_roles
    Scopes:
      - openid
      - profile
      - email
    Enable: true
    Default_Role: admin
    group_mapped_roles:
      - group: neuvector-admin
        global_role: admin
	# I don't know why Neuvector has this twice (one up above and this one in all lowercase) on their web page. Seems weird.
    group_claim: client_roles
  passwordprofileinitcfg.yaml: |
    always_reload: true
    active_profile_name: default
    pwd_profiles:
    - name: default
      comment: default from configMap
      min_len: 8
      min_uppercase_count: 1
      min_lowercase_count: 1
      min_digit_count: 1
      min_special_count: 1
      enable_block_after_failed_login: false
      block_after_failed_login_count: 0
      block_minutes: 0
      enable_password_expiration: false
      password_expire_after_days: 0
      enable_password_history: false
      password_keep_history_count: 0
      session_timeout: 300
  sysinitcfg.yaml: |
    always_reload: true
    New_Service_Policy_Mode: Discover
    New_Service_Profile_Baseline: zero-drift
    Syslog_Level: Info
    Syslog_in_json: true
    Auth_By_Platform: true
    single_cve_per_syslog: true
    syslog_cve_in_layers: true
    Monitor_Service_Mesh: true
    Xff_Enabled: true
    Net_Service_Status: false
    Net_Service_Policy_Mode: Discover
    Disable_Net_Policy: true
    Scanner_Autoscale:
      Strategy: immediate
      Min_Pods: 1
      Max_Pods: 3
    No_Telemetry_Report: true
    Mode_Auto_D2M: true
    Mode_Auto_D2M_Duration: 3600
    Mode_Auto_M2P: false
    Mode_Auto_M2P_Duration: 3600
    Scan_Config:
      Auto_Scan: true
    Unused_Group_Aging: 24
  userinitcfg.yaml: |
    always_reload: true
    users:
    - Fullname: admin
      Password: neuvector
      Role: admin
      Timeout: 600
type: stringData