kong

Originally designed as a reverse proxy and API management layer, Kong has evolved to support AI-specific routing patterns including load balancing across multiple LLM providers, semantic caching of AI responses, rate limiting per model, and prompt injection detection making it a natural infrastructure layer for teams deploying AI-powered applications at scale.

The gateway is built on top of OpenResty (Nginx + Lua) and handles high-throughput traffic with minimal latency overhead.

Kong's plugin architecture allows teams to add capabilities declaratively authentication via API keys, JWT, or OAuth; request transformation; logging to Datadog, Splunk, or custom sinks; circuit breaking; and A/B routing for canary deployments.

With Kong AI Gateway features, these same patterns apply to LLM traffic: routing between OpenAI, Anthropic, Azure OpenAI, and self-hosted models based on cost, latency, or availability.

Kong is available as an open-source project (Kong Gateway OSS), a self-managed enterprise product (Kong Gateway Enterprise), and a managed cloud service (Konnect). The open-source version covers the majority of API gateway use cases and is freely deployable on Kubernetes via Helm charts or as a Docker container.

Kong is used across financial services, healthcare, and technology companies that require a single control plane for API governance across hundreds of services and multiple cloud environments.