inference

It abstracts the complexity of model loading, quantization, batching, and hardware allocation behind a unified REST API, so application code written against OpenAI's API can be redirected to self-hosted models without code changes.

The platform supports a wide range of model backends including llama.cpp for CPU and quantized GPU inference, vLLM for high-throughput GPU serving, transformers for research flexibility, and specialized backends for embedding models (sentence-transformers) and image generation (Stable Diffusion).

Hardware support spans NVIDIA CUDA, AMD ROCm, Apple Silicon Metal, Intel GPUs, and CPU inference, with automatic device selection based on available hardware. Model management is handled through a web UI and Python client that covers downloading from Hugging Face, model configuration, deployment, and monitoring.

Xinference is open-source under the Apache 2.0 license and is developed by Xorbits, a distributed computing company.

It is used by organizations that need full control over their AI inference stack for data privacy, cost optimization, compliance with data residency requirements, or customization of serving configurations beyond what managed API providers allow.

Its compatibility with the OpenAI API format means migration from cloud APIs to self-hosted inference can be achieved by changing a base URL and API key rather than rewriting application code.