kserve

Originally developed as KFServing within the Kubeflow project, KServe provides a unified Custom Resource Definition (CRD) for defining inference services that handles model loading, auto-scaling (including scale-to-zero), canary rollouts, and multi-model serving across frameworks including TensorFlow, PyTorch, scikit-learn, XGBoost, Hugging Face Transformers, and vLLM for LLM serving.

The platform implements the V2 Inference Protocol (Open Inference Protocol), providing a standardized REST and gRPC API for model inference that decouples application code from the specific serving backend a model served via TensorFlow Serving, Triton Inference Server, or a custom predictor all expose the same interface.

KServe's transformer and explainer components allow pre-processing, post-processing, and explainability logic to be deployed alongside the model as separate containers in a coordinated inference graph, keeping model servers focused on inference computation.

KServe is a CNCF incubating project and is used as the production model serving layer in enterprise MLOps platforms built on Kubernetes, including those at major technology companies and financial institutions.

It is particularly relevant for organizations that have standardized on Kubernetes for application infrastructure and want a serving solution that integrates with their existing observability stack (Prometheus, Grafana, Jaeger) and GitOps workflows.

The project is maintained by contributors from Google, Bloomberg, IBM, and the broader Kubeflow community.