xLLM-AI/xllm

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

About the project

xLLM is an inference engine, meaning software that runs trained AI models to produce outputs from inputs, for large language, vision-language, diffusion, and recommendation models on different AI accelerators. Organizations use it to deploy these models with high-throughput and low-latency inference. The catalogue entries provide skills and instructions for working with xLLM.

This repository also configures its own agents. See what xllm tells them →