As generative AI transitions from experimental prototypes to mission-critical enterprise systems, organizations face a rapidly expanding web of AI models, agent frameworks, and contextual backends. Developers rarely rely on a single model provider anymore; instead, they mix and match specialized models based on cost, latency, context window size, or regional compliance. At the same time, AI agents require access to real-world enterprise databases, internal APIs, and Model Context Protocol (MCP) servers to perform meaningful tasks.

This explosion of endpoints creates significant hurdles for platform and security teams. Managing API keys scattered across application repositories, enforcing rate limits, preventing prompt injection, tracking token consumption across departments, and securing agent tool calls can quickly spin out of control.

To solve these challenges, Microsoft introduced the public preview of the AI Gateway tier for Azure API Management. Built on the enterprise-grade foundation of Azure API Management, this dedicated tier delivers a purpose-built runtime and portal experience tailored specifically for AI workloads. In this article, we’ll explore the architecture, real-world use cases, and hands-on deployment of the AI Gateway tier.

The AI Gateway tier is a fully managed runtime boundary and governance plane positioned between your applications (or AI agents) and backend AI assets. Instead of application code directly targeting distinct vendor endpoints with hardcoded API keys, all calls route through a single, secure gateway endpoint using a gateway-scoped runtime access key (api-key).

+-----------------------------------------------------------------------------------+
| Application & Agent Workloads                                                     |
|                                                                                   |
|  +--------------------+        OpenAI / Anthropic Compatible Format               |
|  | AI Coding Agents   | ----------------------------+                             |
|  | Cloud Native Apps  |                             |                             |
|  +--------------------+                             v                             |
|                                         +-----------------------+                 |
|                                         | Azure API Management  |                 |
|                                         |     AI Gateway        |                 |
|                                         +-----------------------+                 |
|                                                     |                             |
|            +----------------------------------------+-------------------+         |
|            |                        |                                   |         |
|            v                        v                                   v         |
|   +------------------+    +-------------------+               +------------------+|
|   | Microsoft Foundry|    | Multi-Cloud LLMs  |               |  MCP Servers &   ||
|   | & Azure OpenAI   |    | Bedrock / Vertex  |               | Enterprise Tools ||
|   +------------------+    +-------------------+               +------------------+|
+-----------------------------------------------------------------------------------+

Imagine an enterprise platform engineering team serving five independent development squads building internal copilots.

In this practical section, we will deploy an AI Gateway resource, establish secure VNet private networking, run the setup wizard to import models and MCP tools, apply rate-limiting policy cards, and execute a request through the gateway.

During its public preview phase, the AI Gateway tier is available with the following baseline parameters:

You can provision an AI Gateway resource using the Azure CLI or Azure Developer CLI (azd). Execute the following command to deploy a new gateway resource in a supported region:

# Register required resource providers
az provider register --namespace Microsoft.ApiManagement

# Create a resource group in a preview region
az group create --name rg-aigateway-prod --location swedencentral

# Provision the AI Gateway tier instance
az resource create \
  --resource-group rg-aigateway-prod \
  --namespace Microsoft.ApiManagement \
  --resource-type service \
  --name apim-aigateway-sweden \
  --location swedencentral \
  --sku name=AiGateway \
  --properties '{}'

Production AI workloads demand strict network isolation. The recommended pattern involves joining the gateway to a virtual network with dedicated subnets for integration and private endpoints:

The portal includes an automated setup wizard that scans your Azure subscriptions to discover existing model deployments and MCP servers.

While policies can be toggled visually in the portal, platform engineering teams can define them declaratively using JSON configuration for Infrastructure as Code workflows.

Here is an example policy definition enforcing a Token Rate Limit of 100,000 tokens per minute and turning on Azure AI Content Safety for a published model asset:

{
  "name": "gpt-4o-governance-policy",
  "properties": {
    "targetAsset": "gpt-4o",
    "policies": {
      "tokenRateLimit": {
        "enabled": true,
        "tokensPerMinute": 100000,
        "counterKey": "context.Subscription.Id"
      },
      "contentSafety": {
        "enabled": true,
        "severityThreshold": "Medium",
        "blockCategories": ["Hate", "Sexual", "SelfHarm", "Violence"]
      },
      "fallback": {
        "enabled": true,
        "secondaryModel": "gpt-4o-mini"
      }
    }
  }
}

Once your models are published and a runtime key is generated from the portal’s Keys page, client applications can call the gateway using standard OpenAI or Anthropic SDKs simply by altering the base_url.

import os
from openai import OpenAI

# Initialize client pointing to the AI Gateway runtime endpoint
client = OpenAI(
    base_url="https://apim-aigateway-sweden.azure-api.net/models/openai/v1",
    api_key="aigw_live_8f92a1b3c4d5e6f7a8b9c0d1e2f3a4b5"  # AI Gateway Runtime Key
)

# Submit a chat completion request through the gateway
response = client.chat.completions.create(
    model="gpt-4o",  # The exact published model name configured on the gateway
    messages=[
        {"role": "system", "content": "You are a helpful platform engineering assistant."},
        {"role": "user", "content": "Explain the architectural benefits of an AI Gateway tier."}
    ],
    temperature=0.7
)

print(f"Response: {response.choices[0].message.content}")
print(f"Prompt Tokens: {response.usage.prompt_tokens}")
print(f"Completion Tokens: {response.usage.completion_tokens}")

Because the gateway emits OpenTelemetry data with standard GenAI semantic attributes over OTLP, platform teams gain immediate visibility in Application Insights or Grafana:

In this article, you explored the AI Gateway tier for Azure API Management. You learned how it bridges the gap between AI application teams and platform engineers by providing a unified gateway for multi-provider models and federated MCP servers. You also implemented a complete setup using the setup wizard, applied private networking patterns, configured OTLP observability, and applied governance policies via policy cards.

You now have a solid foundation for centralizing, securing, and observing AI workloads across your entire cloud estate. From here, you can start integrating custom MCP servers created from OpenAPI specifications, implement private networking with Azure Private Endpoints, or connect Microsoft Foundry-hosted agents directly to governed gateway tools.

Thank you for taking the time to go through this post and making it to the end. Stay tuned, because we’ll keep continuing providing more content on topics like this in the future.