Disable OpenTelemetry SDK during installation

[ThorstenHans]

This PR disables OTEL SDK during installation (otherwise executing spin commands will fail on GH Actions)

The PR also adds a small GitHub Action to run the installer whenever it is changed as part of a PR targeting main

[itowlson]

This looks fine apart from one formatting nit, but I do wonder if this is patching the problem in the wrong place. If merely having the OTel SDK enabled causes Spin to have ructions, that seems like a bug in Spin, and should be fixed at source. But maybe I'm misunderstanding the issue.

[itowlson]

cc @calebschoepp

[calebschoepp]

@ThorstenHans can you elaborate on why Spin is having issues when the OTEL SDK is not disabled?

[ThorstenHans]

@itowlson @calebschoepp sure thing, let me share some more context.

GitHub Action Runners have OTEL environment variables set, during installation of spin, we also execute a bunch of commands such as:

spin-docs/downloads/install.sh

Line 153 in 2be2247

./spin --version

spin-docs/downloads/install.sh

Line 163 in 2be2247

./spin templates install --git "https://github.com/spinframework/spin" --upgrade

spin-docs/downloads/install.sh

Line 169 in 2be2247

./spin plugins update

etc.

The spin telemetry crate recognizes that those OTEL variables are defined and fails when it tries to initialize the telemetry machinery.

This is not unique to GitHub Actions, I recently stumbled upon a DevContainer base image that had those env vars set as well.

By setting OTEL_SDK_DISABLED to true, I could tell spin not to initialize telemetry because of this

https://github.com/spinframework/spin/blob/9ec77fd2429b06fdeaac1a1b99b63799bb2f8e05/crates/telemetry/src/env.rs#L22

This issue won't be hit when using the setup action created by @rajatjindal ! However, I build a spin-devcontainer and used GH Actions to do so... Obviously, I want more than just the spin binary in that DevContainer, I want the full-fledged installation with all the templates and all the plugins... that's why I decided to use the installer script.

For reference: Spin DevContainer -> https://github.com/ThorstenHans/spin-devcontainer

[calebschoepp]

That's helpful @ThorstenHans. Do you have an example of the errors? I would expect that if the dev container is setting valid OTEL env vars things should just work.

[itowlson]

@ThorstenHans are you saying that GH Actions and the devcontainer are both setting e.g. OTEL_EXPORTER_OTLP_ENDPOINT but the value is garbage and Spin is erroring trying to contact it? or something?

And... I see your devcontainer already has ENV OTEL_SDK_DISABLED=true. The check you link should be picking that up (via the && !otel_sdk_disabled() clause) - are you finding that it doesn't?

To reiterate what Caleb said, it would be great if you could paste the actual error you're seeing - thanks!

[ThorstenHans]

@calebschoepp @itowlson I looked into this again. It ONLY happens if one pre-builds a DevContainer (both on GH or locally).

Using the installer script straight from within a GitHub Action works as expected, but when adding devcontainer-cli to the mix it fails.

I've created a small repo that demonstrates this: https://github.com/ThorstenHans/tmp--spin-installer-script/

There are two actions in this repo:

1. A simple action, that just installs Spin on the action runner ... works as expected (https://github.com/ThorstenHans/tmp--spin-installer-script/actions/runs/22217340385)
2. Another action, which uses devcontainer.json and devcontainer-cli to prebuilt the devcontainer image on the GitHub Action runner. You can see this fail (https://github.com/ThorstenHans/tmp--spin-installer-script/actions/runs/22217340419/job/64264025785)

The devcontainer.json is using a pinned version of my Spin DevContainer Feature (https://github.com/ThorstenHans/devcontainer-features/blob/f6d32a02ae5acf8fdc4d57345b190605d2e69bcd/src/spin/install.sh) In more recent versions of my Spin DevContainer Feature, I set OTEL_SDK_DISABLED as a workaround.

However, users that pre-built a devcontainer without using my Spin DevContainer Feature will run into the same error.

I don't know if we consider this as an edge-case or not, but at least it verifies that spin picks up OTEL variables that are set by a 3rd party

[ThorstenHans]

And... I see your devcontainer already has ENV OTEL_SDK_DISABLED=true. The check you link should be picking that up (via the && !otel_sdk_disabled() clause) - are you finding that it doesn't?

@itowlson Yeah, my DevContainer is setting that, to prevent DevContainer pre-builds from failing. IMO, our installer script could set that so that other DevContainer authors doesn't have to

[itowlson]

Sorry I am being a dolt, @ThorstenHans, but I still don't understand why Spin would works in a devcontainer except during installation. What is special about installation time that we need to disable OTel then but not at other times?

I would strongly prefer that we fix this in Spin, because users should be able to install their own plugins and templates in their devcontainers: they shouldn't be reliant on our specific install script. And because I would like us to understand what is happening rather than kludging past it with a global disable-enable. (Of course if nothing else works, or it's going to take a long time to fix it at source, then we'll do the kludge.)

@calebschoepp The error from Thorsten's failing build link is:

3.693 Error: Failed to initialize telemetry
  3.693 
  3.693 Caused by:
  3.693    0: failed to initialize otel tracing
  3.693    1: Exporter invalid URI invalid format encountered the following error(s): otlp

So I guess the devcontainer is... setting the exporter URI to garbage during initialisation and fixing it up later? I am not sure, and I don't understand either OTel or the devcontainer lifecycle well enough to have a sense for where this is all happening...!

Should we make "bad exporter URI" less fatal? What do other applications do?

[calebschoepp]

I would strongly prefer that we fix this in Spin, because users should be able to install their own plugins and templates in their devcontainers: they shouldn't be reliant on our specific install script. And because I would like us to understand what is happening rather than kludging past it with a global disable-enable. (Of course if nothing else works, or it's going to take a long time to fix it at source, then we'll do the kludge.)

+1

Hmmm it's not immediately obvious to me what's going wrong. @ThorstenHans could you log what the value of the relevant OTEL environment variables so we can see what they are. Likely Ivan is right that we shouldn't fail so fatally in the case that we're handed garbage in these OTEL env vars — we should probably just log a tracing WARN or ERROR and move on. Still it would be good to understand what the value we're barfing on is.