Version 0.41.0 and above break Otel Tracing for Python Applications

[matanyall]

Bug Description

Traces stopped being received after upgrading from Hatchet SDK 0.40.0a9 to 0.41.0 (and later versions including 0.45.0b1). Traces appear to be getting "blackholed".

Reproduction

1. Set up custom OTEL trace provider (code below)
2. Upgrade Hatchet SDK from 0.40.0a9 to 0.41.0+
3. Traces stop appearing in collector

import base64
import os
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def init_otlp(app_name: str = "my-app"):
    resource = Resource.create(attributes={"service.name": app_name})
    tracer = TracerProvider(resource=resource)
    trace.set_tracer_provider(tracer)
    
    # Configure OTLP exporter
    exporter = OTLPSpanExporter(
        endpoint="https://my-collector/v1/traces",
        headers={"authorization": "Basic xxx"}
    )
    tracer.add_span_processor(BatchSpanProcessor(exporter))

def init_api_instrumentors(app: FastAPI):
    FastAPIInstrumentor().instrument_app(app, tracer_provider=trace.get_tracer_provider())

Hypothesis

The SDK might be setting its own global trace provider, preventing custom trace provider configuration.

Versions

• Working: 0.40.0a9
• Not working: 0.41.0+
• Last stable: 0.38.3

[matanyall]

Hacky workaround, I set up telemetry before I import the hatchet sdk. Not a huge fan, but seems like a weird python module situation, not sure if there is a fix.