Feature/28 provision additional certificates

[DiCanio]

Adds a routine to provision custom certificates in PEM format. This is necessary when deploying the node to an environment where requests are being intercepted for inspection causing SSL termination by leveraging custom certificates (e.g. self-signed certificates). The routine bundles all provisioned certificates into a single .pem file and makes that available via a ConfigMap.

Forwards these custom certificates to the message broker component. Without this change the component is not able to connect to the hub when deployed in an environment resembling the one described above.

[DiCanio]

@brucetony , @mjugl I'll take a look at the result-service and the node-ui for further adjustments later on. Will ping you once this is done for a second check.

[mjugl]

@brucetony , @mjugl I'll take a look at the result-service and the node-ui for further adjustments later on. Will ping you once this is done for a second check.

Thanks for the effort!

The result service sends requests using the httpx library. It uses Mozilla's CA bundle by default. Unfortunately you can't just provide an environment variable which points to extra CA certificates like in Node.js. They have to be added to the container's CA bundle.

The way I handled this so far was to create a new Dockerfile which is based on the image of the application that I want to add the certificate to. For Alpine, I extended the Dockerfile like so:

RUN chmod 0644 /usr/local/share/ca-certificates/*.crt && \
    update-ca-certificates && \
    rm /etc/ssl/cert.pem && \
    ln -s /etc/ssl/certs/ca-certificates.crt /etc/ssl/cert.pem

If we can't build new images in the deployment process, then I'll have to think of a workaround (e.g. making the above command part of the container's entrypoint script).

[DiCanio]

@brucetony , @mjugl I'll take a look at the result-service and the node-ui for further adjustments later on. Will ping you once this is done for a second check.

Thanks for the effort!

The result service sends requests using the httpx library. It uses Mozilla's CA bundle by default. Unfortunately you can't just provide an environment variable which points to extra CA certificates like in Node.js. They have to be added to the container's CA bundle.

The way I handled this so far was to create a new Dockerfile which is based on the image of the application that I want to add the certificate to. For Alpine, I extended the Dockerfile like so:

RUN chmod 0644 /usr/local/share/ca-certificates/*.crt && \
    update-ca-certificates && \
    rm /etc/ssl/cert.pem && \
    ln -s /etc/ssl/certs/ca-certificates.crt /etc/ssl/cert.pem

If we can't build new images in the deployment process, then I'll have to think of a workaround (e.g. making the above command part of the container's entrypoint script).

If it's not a major task (haven't had a look at the project) we could also try using the truststore package as detailed on the official site. This way the native OS trust stores are used and we can simply inject additional certificates.
What remains is the question whether this package only makes use of the bundled stores which would require a call to update-ca-certificates.

[mjugl]

If it's not a major task (haven't had a look at the project) we could also try using the truststore package as detailed on the official site. This way the native OS trust stores are used and we can simply inject additional certificates. What remains is the question whether this package only makes use of the bundled stores which would require a call to update-ca-certificates.

Yes, that's entirely possible and amounts to just a couple new lines of code. The FLAME Hub Python Client library permits custom httpx.Client instances, so I would simply do that and add additional arguments (base URL for the API to be called and auth flow implementation). All other services using the FLAME Hub Python Client library could simply copy-and-paste this behavior.

I could also add this behavior on the library level but I would honestly leave it up to the application using the library.

[DiCanio]

If it's not a major task (haven't had a look at the project) we could also try using the truststore package as detailed on the official site. This way the native OS trust stores are used and we can simply inject additional certificates. What remains is the question whether this package only makes use of the bundled stores which would require a call to update-ca-certificates.

Yes, that's entirely possible and amounts to just a couple new lines of code. The FLAME Hub Python Client library permits custom httpx.Client instances, so I would simply do that and add additional arguments (base URL for the API to be called and auth flow implementation). All other services using the FLAME Hub Python Client library could simply copy-and-paste this behavior.

I could also add this behavior on the library level but I would honestly leave it up to the application using the library.

That sounds good 👌🏻
Feel free to add to this PR here as soon as the changes in the upstream project have been made. If time is scarce I can also help incorporating it here.

[DiCanio]

@brucetony Since the UI is using a Node environment as well and is also making outbound requests I simply mirrored the approach used for the message broker component.

[mjugl]

Just published ghcr.io/privateaim/node-result-service:dev-20250314T080932Z which uses the system CA bundle to verify outbound requests.

All that's left is to call update-ca-certificates. The only idea I have in mind is:

• add an entrypoint script to the Node Result Service Docker image
• add a call to update-ca-certificates when running the script on container start-up
• optionally hide this call behind an environment variable which is set when deploying via Helm (but this can be left out for simplicity sake)

Any better ideas?

Edit: Just wrote up a PoC for the suggested approach. It works, but it requires the container to be run as root. The entrypoint script would drop privileges for the server process if it is run as root, but that would still cause exec'ing into the container to happen as root. This is the entrypoint script:

#!/bin/sh

_print_info() {
    printf "\033[1m[$0] [-]\033[0m %s\n" "$1"
}

_print_error() {
    printf "\033[31;1m[$0] [!]\033[0m %s\n" "$1"
}

if [ -n "${FORCE_UPDATE_CA_CERTIFICATES}" ];
then
  if [ "$(id -u)" -ne "0" ];
  then
    _print_error "update-ca-certificates requires to be run as root"
    exit 1
  fi

  _print_info "Provisioning additional certificates"

  update-ca-certificates
  rm /etc/ssl/cert.pem
  ln -s /etc/ssl/certs/ca-certificates.crt /etc/ssl/cert.pem
fi

if [ "$(id -u)" -eq "0" ];
then
  _print_info "Dropping privileges"
  su - nonroot
fi

_print_info "$(printf "%s " "Running server with arguments:" "${@}")"
exec /usr/local/bin/python -m uvicorn project.main:app "${@}"

[DiCanio]

Just published ghcr.io/privateaim/node-result-service:dev-20250314T080932Z which uses the system CA bundle to verify outbound requests.

All that's left is to call update-ca-certificates. The only idea I have in mind is:

• add an entrypoint script to the Node Result Service Docker image
• add a call to update-ca-certificates when running the script on container start-up
• optionally hide this call behind an environment variable which is set when deploying via Helm (but this can be left out for simplicity sake)

Any better ideas?

Edit: Just wrote up a PoC for the suggested approach. It works, but it requires the container to be run as root. The entrypoint script would drop privileges for the server process if it is run as root, but that would still cause exec'ing into the container to happen as root. This is the entrypoint script:

#!/bin/sh

_print_info() {
    printf "\033[1m[$0] [-]\033[0m %s\n" "$1"
}

_print_error() {
    printf "\033[31;1m[$0] [!]\033[0m %s\n" "$1"
}

if [ -n "${FORCE_UPDATE_CA_CERTIFICATES}" ];
then
  if [ "$(id -u)" -ne "0" ];
  then
    _print_error "update-ca-certificates requires to be run as root"
    exit 1
  fi

  _print_info "Provisioning additional certificates"

  update-ca-certificates
  rm /etc/ssl/cert.pem
  ln -s /etc/ssl/certs/ca-certificates.crt /etc/ssl/cert.pem
fi

if [ "$(id -u)" -eq "0" ];
then
  _print_info "Dropping privileges"
  su - nonroot
fi

_print_info "$(printf "%s " "Running server with arguments:" "${@}")"
exec /usr/local/bin/python -m uvicorn project.main:app "${@}"

We need to think of a solution that doesn't require running any sort of update-ca-certificates command as it requires elevated rights (due to changing system related properties). If this is going to be deployed to any k8s instance within a DIC that has a Pod Security Admission Controller configured then non-root containers might be enforced rendering this solution non-functional.

Would offer to check if adding the certificate as is will be sufficient already. Could do that throughout the day tomorrow.