feat: LLM smart router with embedding-based prompt classification#11
Merged
sblanchard merged 15 commits intomainfrom Feb 21, 2026
Merged
feat: LLM smart router with embedding-based prompt classification#11sblanchard merged 15 commits intomainfrom
sblanchard merged 15 commits intomainfrom
Conversation
Add smart router configuration types: RouterConfig (with Default), RoutingProfile enum (Auto/Eco/Premium/Free/Reasoning), ModelTier enum (Simple/Complex/Reasoning/Free), ClassifierConfig, TierConfig, and RouterThresholds. Add optional router field to LlmConfig.
Introduces EmbeddingClassifier that uses cosine similarity between prompt embeddings and pre-computed tier centroids (Simple, Complex, Reasoning) to route prompts to the appropriate model tier. Includes an in-memory LRU cache with TTL eviction, threshold-based escalation rules, and agentic prompt detection via length heuristics.
Pure synchronous routing functions that resolve model strings from routing profiles, classified tiers, and tier configuration. Supports explicit model bypass, profile-to-tier mapping, classifier-driven Auto mode, and cross-tier fallback chains.
Thread-safe DecisionLog backed by parking_lot::Mutex and VecDeque that evicts the oldest entry at capacity. Provides `record()` and `recent()` for observability of smart-router routing choices.
Add SmartRouterState to AppState and update resolve_provider() to check the smart router when no explicit model override is provided. The return type now includes an optional model name so the router can specify which model within a provider to use. The resolution order becomes: explicit override -> smart router -> agent models -> role defaults -> any provider.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire routing_profile through the Schedule model, create/update API, and schedule runner so scheduled runs can use a specific router profile instead of the global default. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add routing_profile field to Schedule, CreateScheduleRequest, and UpdateScheduleRequest TypeScript types. Wire a "Routing Profile" dropdown into both the create form (Schedules.vue) and edit form (ScheduleDetail.vue) with options: Default (inherit), Auto, Eco, Premium, Free, and Reasoning.
Full round-trip validation of the routing pipeline without requiring Ollama or any external services. Covers all five routing profiles, explicit model bypass, tier fallback chains, and decision log recording with capacity eviction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/v1/router/status,/v1/router/classify,/v1/router/decisions,/v1/router/config)[llm.router]is absentNew files
crates/providers/src/classifier.rs— embedding classifier with cache (792 lines)crates/providers/src/smart_router.rs— pure routing resolution logic (233 lines)crates/providers/src/decisions.rs— thread-safe decisions ring buffer (111 lines)crates/gateway/src/api/router.rs— router API endpoints (229 lines)crates/providers/tests/router_integration.rs— integration tests (252 lines)Modified files
crates/domain/src/config/llm.rs— RoutingProfile, ModelTier, RouterConfig typescrates/gateway/src/runtime/mod.rs— resolve_provider() with smart router stepcrates/gateway/src/runtime/turn.rs— routing_profile on TurnInputcrates/gateway/src/bootstrap.rs— router initialization at startupcrates/gateway/src/api/schedules.rs— routing_profile on schedule CRUD + validationclient.ts,Settings.vue,Schedules.vue,ScheduleDetail.vueTest plan
[llm.router]in config.toml, verify Settings page shows Router card