Skip to content

perf: Implement HTTP client connection pooling in LLM clients #436

@AlexMikhalev

Description

@AlexMikhalev

Issue Description

New HTTP clients are created for each LLM request instead of reusing connections, causing significant overhead.

Location

crates/terraphim_service/src/llm.rs (lines 285, 540, 569)

Current Code

```rust
// Creating new client for each request
let client = reqwest::Client::new();
// ... use client once ...
```

Impact

  • CRITICAL priority - Performance
  • Creates new connection pool for every client initialization
  • 30-50% connection overhead for repeated requests
  • Increased memory usage

Recommended Fix

```rust
use once_cell::sync::Lazy;
use reqwest::Client;

static HTTP_CLIENT: Lazy = Lazy::new(|| {
Client::builder()
.pool_max_idle_per_host(10)
.pool_idle_timeout(Duration::from_secs(90))
.timeout(Duration::from_secs(120))
.build()
.unwrap()
});

// Use: &*HTTP_CLIENT instead of creating new clients
```

Expected Improvement

  • 30-50% reduction in connection overhead
  • Reduced memory allocations
  • Faster repeated requests

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions