add Custom Trace Attributes to extend observability #708

Spherrrical · 2026-01-27T21:46:47Z

Introduces Custom Trace Attributes to allow developers to extend business logic!

Today, traces include request path and system metadata, but not application-specific identifiers. This makes it hard to troubleshoot per-tenant, per-workspace, or per-user issues.

#707

salmanap · 2026-01-28T00:22:33Z

One quick question - why wouldn't these trace attributes live in the customers application code. Are these attributes part of the LLM call, if so why? Is the idea that these multi-tenant attributes must be sent up for tracking certain customers?

Wouldn't it make more sense to simply accept and x-plano-trace-customer-attribute- prefix and rotate through all those and write them to the trace. Why build a static config that needs to get update all the time?

Spherrrical · 2026-01-28T18:56:04Z

One quick question - why wouldn't these trace attributes live in the customers application code. Are these attributes part of the LLM call, if so why? Is the idea that these multi-tenant attributes must be sent up for tracking certain customers?

These attributes are intentionally configured at the gateway so we can consistently attach tenant/workspace/user metadata to every trace without requiring each customer app/SDK to know our trace schema or emit spans themselves. They’re not part of the LLM payload; we read them from the incoming request headers and add them to the trace for operational filtering across routing/LLM/agent spans. the goal is multi‑tenant attribution so we can slice traces by customer/tenant/user reliably across all requests per the customers choice.

Wouldn't it make more sense to simply accept and x-plano-trace-customer-attribute- prefix and rotate through all those and write them to the trace. Why build a static config that needs to get update all the time?

We use a gateway allowlist instead of a wildcard prefix so trace schemas stay stable and auditable. The config maps header → attribute key with type parsing (str, int, etc.), which prevents sensitive or arbitary headers from being logged and keeps queries consistent. A prefix would let any client inject new "keys" and leak data/over permissive in data.

…ests configs

Spherrrical · 2026-02-06T01:10:41Z

Implemented prefix-based trace attribute extraction and removed the old per-header schema. The gateway now reads tracing.custom_attribute_prefixes and captures any matching request headers as OTEL attributes, trimming the prefix so x-katanemo-workspace-id becomes workspace.id, with values recorded as strings. Updated config/schema and demo files to use the new prefix model, added a prefix-mismatch example in test.rest, and updated unit tests to match the new behavior.

To enable the custom prefix:

tracing:
  random_sampling: 100
  custom_attribute_prefixes:
    - x-katanemo-

Headers must contain x-[identifier] for attribute to pass through OTEL by configuration.

salmanap

Left some comments on documentation and the name of the config attribute. Please fix those and this is ready to ship

salmanap · 2026-02-06T15:44:54Z

config/arch_config_schema.yaml

      trace_arch_internal:
        type: boolean
-      additionalProperties: false
+      custom_attribute_prefixes:


we should probably call this span_attribute_header_perfixes

salmanap · 2026-02-06T16:45:58Z

crates/brightstaff/src/tracing/custom_attributes.rs

+            None => continue,
+        };
+
+        let suffix = header_name.strip_prefix(prefix).unwrap_or("");


There are some semantics here that I don't think we've made explicit to the developer that we will transform x-user-id to user.id in the traces. We should add a comment here and add a section in the docs to talk about this.

…refixes in configuration and related handlers

adilhafeez

Thanks for putting this togehter. I do have some suggestions,

Name can be a bit better while I don't have an answer here but it should be sth like. Notice the static block too,

tracing:
  ...
  span_attributes: # or trace_attributes or just attributes
    header_prefixes: # or from_header
      - x-katanemo-
    static:
      environment: production
      service.version: "1.0.0"
      deployment.region: us-west-2

Placement of attributes in span - I see that in the screenshot attached to plano(llm) span. For a single request will we see these attributes in all spans that are created in a trace? I think these attributes should be part of just the root trace.

adilhafeez · 2026-02-06T18:48:22Z

config/arch_config_schema.yaml

        type: integer
      trace_arch_internal:
        type: boolean
-      additionalProperties: false


we still need additionalProperties: false here

adilhafeez · 2026-02-06T19:00:05Z

crates/brightstaff/src/handlers/agent_chat_completions.rs

+    let custom_attrs = extract_custom_trace_attributes(
+        &request_headers,
+        tracing_config
+            .as_ref()
+            .as_ref()
+            .and_then(|tracing| tracing.span_attribute_header_prefixes.as_deref()),
+    );



should go other way where only look at request headers if custom tracing attributes are set - this will optimize the run time

adilhafeez · 2026-02-06T19:01:00Z

crates/brightstaff/src/handlers/agent_chat_completions.rs

+        for (key, value) in &custom_attrs {
+            span_builder = span_builder.with_attribute(key, value);
+        }


why is it repeating - I saw another instance of this code at line 285

adilhafeez · 2026-02-06T19:01:54Z

crates/brightstaff/src/handlers/llm.rs

+// ! we reached the limit of the number of arguments for a function
+#[allow(clippy::too_many_arguments)]
 pub async fn llm_chat(
    request: Request<hyper::body::Incoming>,
    router_service: Arc<RouterService>,
    full_qualified_llm_provider_url: String,
    model_aliases: Arc<Option<HashMap<String, ModelAlias>>>,
    llm_providers: Arc<RwLock<LlmProviders>>,
    trace_collector: Arc<TraceCollector>,
+    tracing_config: Arc<Option<Tracing>>, // ! right here
    state_storage: Option<Arc<dyn StateStorage>>,


these comments are of no value - remove them

adilhafeez · 2026-02-06T19:03:10Z

crates/brightstaff/src/handlers/llm.rs

+    let custom_attrs = extract_custom_trace_attributes(
+        &request_headers,
+        tracing_config
+            .as_ref()
+            .as_ref()
+            .and_then(|tracing| tracing.span_attribute_header_prefixes.as_deref()),
+    );
    let request_id: String = match request_headers


go from other way that is if span_attributes are defined only then extract the attributes from header

adilhafeez · 2026-02-06T19:05:01Z

crates/brightstaff/src/main.rs

    ));

    let model_aliases = Arc::new(arch_config.model_aliases.clone());
+    let tracing_config = Arc::new(arch_config.tracing.clone());


don't send entire object may just send custom tracing block

…ust related handlers

… clarify configuration

salmanap · 2026-02-06T21:59:43Z

crates/brightstaff/src/handlers/agent_chat_completions.rs

        (String::new(), None)
    };

+    let apply_custom_attrs = |mut builder: SpanBuilder| {


I don't think you want a lambda function in the file. Feels a bit misplaced. I think we should ask ourselves, whats the canonical way to append attributes after a span has been built. A simple for loop feels okay. Although I might move some of that functionality in commons

salmanap · 2026-02-06T22:01:12Z

crates/brightstaff/src/handlers/llm.rs

-            .as_ref()
-            .and_then(|tracing| tracing.span_attribute_header_prefixes.as_deref()),
-    );
+    let mut header_prefixes: Option<&[String]> = None;


I think we can merge this into a single utility that both llm.rs and agent_chat_completions.rs work off of. Else its duplicate code and will be maintained in two places.

Spherrrical self-assigned this Jan 27, 2026

Spherrrical changed the title ~~add Custom Trace Attributes (#707)~~ add Custom Trace Attributes to extend observability Jan 27, 2026

Spherrrical added 2 commits February 5, 2026 16:39

add custom trace attributes

f61d720

refactor: prefix custom trace attributes and update schema handlers t…

40c959e

…ests configs

Spherrrical force-pushed the musa/custom-trace-attributes branch from 03605d9 to 40c959e Compare February 6, 2026 01:07

salmanap mentioned this pull request Feb 6, 2026

feat(tracing): enrich OTEL spans with business identifiers from reque… #725

Closed

salmanap approved these changes Feb 6, 2026

View reviewed changes

Spherrrical added 2 commits February 6, 2026 09:33

refactor: rename custom_attribute_prefixes to span_attribute_header_p…

4b70658

…refixes in configuration and related handlers

docs: add section on custom span attributes

0e95212

adilhafeez requested changes Feb 6, 2026

View reviewed changes

Spherrrical added 2 commits February 6, 2026 11:53

refactor: update tracing configuration to use span attributes and adj…

d015b40

…ust related handlers

docs: custom span attributes section to include static attributes and…

21cf046

… clarify configuration

Spherrrical requested a review from adilhafeez February 6, 2026 21:37

salmanap reviewed Feb 6, 2026

View reviewed changes

add Custom Trace Attributes to extend observability #708

Are you sure you want to change the base?

add Custom Trace Attributes to extend observability #708

Uh oh!

Conversation

Spherrrical commented Jan 27, 2026

Uh oh!

salmanap commented Jan 28, 2026

Uh oh!

Spherrrical commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Spherrrical commented Feb 6, 2026

Uh oh!

salmanap left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adilhafeez left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Spherrrical commented Jan 28, 2026 •

edited

Loading

adilhafeez left a comment •

edited

Loading