Skip to content

Conversation

@carlos-marchal-ph
Copy link
Contributor

We were counting cache write tokens incorrectly when using Anthropic with Langchain.

@carlos-marchal-ph carlos-marchal-ph requested a review from a team January 2, 2026 17:16
@carlos-marchal-ph carlos-marchal-ph self-assigned this Jan 2, 2026
@carlos-marchal-ph carlos-marchal-ph added bug Something isn't working team/llm-analytics labels Jan 2, 2026
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Jan 2, 2026

What Changed

The core issue was in posthog/ai/langchain/callbacks.py where the Anthropic-specific token accounting logic was only subtracting cache read tokens from input tokens, but not cache write tokens. According to Anthropic's API semantics, their input_tokens field includes all tokens (including cache tokens), but PostHog's cost calculation expects them to be separate.

Key changes:

  1. Token calculation fix: Modified the Anthropic token normalization to subtract both cache_read_tokens AND cache_write_tokens from input_tokens
  2. Provider detection: Uses consistent logic matching the plugin-server - exact match on provider name OR substring match on model name
  3. Comprehensive tests: Added three new test functions to verify the cache token subtraction behavior works correctly for Anthropic while leaving other providers unchanged

The fix ensures that for Anthropic providers:

normalized_input_tokens = reported_input_tokens - cache_read_tokens - cache_write_tokens

While for other providers like OpenAI, input_tokens already excludes cache tokens as expected.

Testing Coverage

The PR includes thorough test coverage with scenarios testing:

  • Cache write token subtraction for Anthropic provider
  • Combined cache read + write token subtraction for Anthropic
  • Verification that non-Anthropic providers don't have cache tokens subtracted
  • Edge cases with missing or zero cache token values

Why This Matters

This fix ensures accurate billing calculations and usage analytics for teams using Anthropic models through LangChain. Previously, cache creation costs were being double-counted, leading to inflated metrics and incorrect cost attribution.

The change is backwards compatible and only affects the internal token accounting logic for Anthropic providers - the external API remains unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working team/llm-analytics

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants