Is an LLM gateway the same as an AI gateway?

Mostly. AI gateway is the broader name and can also cover image, speech and embedding models. LLM gateway refers to language models. Many products use both names for the same service.

Do I need an LLM gateway?

A gateway helps once several applications, teams or models are involved, because it puts keys, limits and cost records in one place. A single application that calls one provider can often call it directly.

Does a gateway add latency?

It adds a network step between your software and the provider. How much time that adds depends on where the gateway runs and what it does with each request, so measure it with your own traffic.

What is the difference between an LLM gateway and an LLM router?

A gateway carries requests to providers and applies the controls around them. A router chooses which model should answer. A router can run inside a gateway, as Conifer’s does.

Learn

What is an LLM gateway?

An LLM gateway is a service that sits between your software and the companies that serve AI models. This page explains what a gateway does, how it differs from a proxy or a router, and what to check when you choose one.

See the Conifer gateway

What an LLM gateway does

Without a gateway, each application keeps its own key for each AI provider and calls each provider’s API directly. A gateway puts one address in front of all of them. Applications send every model request to the gateway with one key, and the gateway forwards each request to a provider that serves the model.

One address: Applications call one endpoint, usually on an interface they already support, such as the OpenAI or Anthropic API. Moving to another model means changing the model name, not the code.
Keys and access: The gateway holds the provider credentials. People and services get their own gateway keys, which can be limited to certain models and revoked one at a time.
Failover: When a provider is down, rate-limited or slow, the gateway can retry the request with another provider or move it to another model, depending on how it is set up.
Spending limits: Budgets and rate limits can be set for each key, team or request, and checked before a request reaches a provider.
Usage and cost: Because every request passes through it, the gateway can record which model served each request, how many tokens it used and what it cost.
Caching: Some gateways reuse earlier answers to repeated requests, or manage the providers’ prompt caches, to cut cost and waiting time.

Every model request passes through one place, so the same keys, limits and records apply to all of them.

Gateway, proxy and router

The three names overlap, and vendors use them loosely. This is how they are usually meant.

LLM proxy: Forwards a request to a provider and returns the answer, often translating between API formats. Many gateways started as proxies, and the two names are often used for the same product.
LLM gateway: A proxy with the controls around it: keys, limits, failover, logs and cost records, across many providers at once. AI gateway is the broader name and can also cover image, speech and embedding models.
LLM router: Decides which model should answer a request, so the caller does not have to name one. A router can run inside a gateway or on its own. What is an LLM router? explains how routers decide.

Hosted or self-hosted

Gateways are offered in two main ways, and some vendors offer both.

Hosted: The vendor runs the gateway. You change a base URL and a key, and the vendor handles uptime, scaling and, with some vendors, the provider accounts. Requests pass through the vendor’s servers, so its data handling terms apply.
Self-hosted: You run the gateway in your own cloud or data centre, often from open-source code. Requests, logs and provider keys stay in your infrastructure, and your team deploys, upgrades and scales it, usually with a database alongside.
Hybrid: The part that handles requests runs in your environment, and the vendor runs the management side.

What to check when you choose one

Interfaces: Which APIs it accepts. Coding tools and SDKs each expect a particular one, such as OpenAI Chat Completions, OpenAI Responses or Anthropic Messages.
Named models: Whether a request that names a model is always served by that model, and what happens when it cannot be: a retry with another provider, a switch to another model, or an error.
Who bills for models: Whether you pay each provider directly with your own keys, buy model usage through the gateway, or can do both, and what fee applies to each.
Cost figures: Whether the cost shown is the amount actually charged or an estimate from a price table, and whether it comes back on each response or only appears in a dashboard.
Data: What the gateway stores, such as prompts, responses or only request details, and for how long.
Running it: For a self-hosted gateway, the people and systems it needs. For a hosted one, the vendor’s uptime record, regions and contract terms.

How the Conifer gateway works

Conifer runs a hosted LLM gateway for more than 230 AI models. A request that names a model is served by that model or refused with a clear error, and every response states the model that answered and what the request cost. The gateway page has the details.

Connect: Change the base URL and the key in an existing OpenAI or Anthropic client.
OPENAI_BASE_URL=https://api.conifer.build/v1 OPENAI_API_KEY=$CONIFER_API_KEY
Failover: If a provider fails, the gateway serves the same model from another provider at the same price. Another model is tried only if you list it as a fallback yourself, up to three.
Limits: Set a ceiling on a single request, a spend cap on each key and a monthly limit on the account. A request over a ceiling is refused before any provider is called.
Billing: Usage is charged per token at the catalog price, from a prepaid balance. You can also put your own Azure OpenAI resource behind the same key, and Conifer adds no fee on that usage.

Routing explains named requests, failover and the cost receipt, Billing and caching covers metering, and Replace your provider key has the setup steps.

Conifer Gateway: One API key and endpoint for every model in the catalog, with the model and cost stated on every response.
Conifer vs OpenRouter: The two gateways side by side, from each company's own documentation.
Conifer vs Portkey: The two gateways side by side, from each company's own documentation.
Conifer vs Helicone: The two gateways side by side, from each company's own documentation.
Conifer vs LiteLLM: The two gateways side by side, from each company's own documentation.
All comparisons: Conifer compared with eleven other LLM gateways and routers, from their own documentation.

Questions

What is an LLM gateway?: An LLM gateway is a service between your software and the companies that serve AI models. Applications send every request to one address with one key. The gateway forwards each request to a provider that serves the model, applies limits, handles failover and records usage and cost.
Is an LLM gateway the same as an AI gateway?: Mostly. AI gateway is the broader name and can also cover image, speech and embedding models. LLM gateway refers to language models. Many products use both names for the same service.
Do I need an LLM gateway?: A gateway helps once several applications, teams or models are involved, because it puts keys, limits and cost records in one place. A single application that calls one provider can often call it directly.
Does a gateway add latency?: It adds a network step between your software and the provider. How much time that adds depends on where the gateway runs and what it does with each request, so measure it with your own traffic.
What is the difference between an LLM gateway and an LLM router?: A gateway carries requests to providers and applies the controls around them. A router chooses which model should answer. A router can run inside a gateway, as Conifer’s does.