Skip to content
This repository was archived by the owner on Mar 26, 2025. It is now read-only.

Feat: Hatchet OTel Instrumentor#313

Merged
mrkaye97 merged 33 commits intomainfrom
feat--otel-instrumetor
Feb 11, 2025
Merged

Feat: Hatchet OTel Instrumentor#313
mrkaye97 merged 33 commits intomainfrom
feat--otel-instrumetor

Conversation

@mrkaye97
Copy link
Contributor

@mrkaye97 mrkaye97 commented Feb 7, 2025

Easier to review without whitespace because of all the indentation from removing spans

Reworking our OTel setup to instead follow a bunch of other "instrumentor" patterns. See the example for usage - I'll add docs too (modify the existing ones). This will replace our current setup. Tested a bunch with Honeycomb and everything looked great to me 🥳 - this should also let us extend our instrumentation pretty easily without mucking around in the internals of the SDK too much.

I also changed the OTel setup to be an "extra", so you can install the deps with pip install hatchet-sdk[otel] if you want them, but otherwise don't need to.

TODO:

  • Propagate traceparent through on event push, bulk push, run workflow, etc.
  • Test if spans created inside of hatchet steps work correctly
  • Add comments about keeping type sigs in sync
  • Improve handling of args / kwargs in wrappers
  • Add run workflow, async run workflow, bulk push, etc. to examples
  • Add tests

@mrkaye97 mrkaye97 requested review from abelanger5 and grutt February 8, 2025 04:03
@mrkaye97 mrkaye97 marked this pull request as ready for review February 8, 2025 04:03
Comment on lines +30 to +42
def test_push_event(hatchet: Hatchet, worker: Worker) -> None:
key = "otel:event"
payload = {"test": "test"}

with tracer.start_as_current_span("push_event"):
event = hatchet.event.push(
event_key=key,
payload=payload,
options=create_push_options(),
)

assert event.key == key
assert event.payload == json.dumps(payload)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the point of these tests is to make sure the existing code in the SDK runs without any issues if we use the instrumentor (in this case with a no-op tracer). can extend these more later if they end up being valuable

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ran all these and checked out the traces in Honeycomb. E.g.:

Screenshot 2025-02-07 at 9 05 05 PM

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This basically follows how asyncio does its instrumentation - was super easy to implement, really similar to mocking in tests



class HatchetInstrumentor(BaseInstrumentor): # type: ignore[misc]
OTEL_TRACEPARENT_KEY = "traceparent"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want to namespace this i.e. hatchet__traceparent to prevent the very unlikely collision someone is using this key?

I think we use the hatchet__ prefix for things like event id already

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems reasonable - I'm actually not sure. My sense is leaving this as just traceparent is fine because we'll need to rename the key anyways, but do you think that it's likely people will ever pass this as a field in the metadata and not intend for it to be used this way?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it's highly unlikely... and only breaks things if they use this key AND feature

@mrkaye97 mrkaye97 requested a review from grutt February 11, 2025 02:50
@mrkaye97 mrkaye97 merged commit 9ad90a6 into main Feb 11, 2025
6 checks passed
@mrkaye97 mrkaye97 deleted the feat--otel-instrumetor branch February 11, 2025 13:48
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Version 0.41.0 and above break Otel Tracing for Python Applications

3 participants