Candle-vLLM

A high-performance Rust implementation of PagedAttention and vLLM features built on the Candle ML framework.

Candle-vLLM brings memory-efficient inference to the Rust ecosystem by porting the PagedAttention algorithm from the original Python vLLM project. It leverages the Candle framework to bypass Python runtime overhead, offering a streamlined stack for deploying Large Language Models like Llama 3 and Mistral. The project focuses on high throughput via efficient KV cache management (reducing fragmentation) and provides a clean API for developers who require the safety and speed of Rust in production environments.

https://github.com/EricLBuehler/candle-vllm