RouteLLM

RouteLLM is an open-source framework from LMSYS that dynamically routes queries between high-performance and low-cost models to optimize inference budgets.

Developed by the LMSYS Org team (the creators of Chatbot Arena), RouteLLM uses learned routers to decide if a query requires a powerhouse like GPT-4o or can be handled by a smaller model like Llama-3-8B. The system leverages preference data to train specialized routers (including matrix factorization and BERT-based classifiers) that help users maintain 95% of a top-tier model's quality while slashing costs by over 50%. It functions as a drop-in proxy for OpenAI-compatible APIs: developers simply swap their base URL to begin balancing performance against budget automatically across diverse production workloads.

https://github.com/lm-sys/RouteLLM