Proxy Azure OpenAI Requests

Azure OpenAI Service is a fully managed service that provides unified REST API access to OpenAI's language models, such as GPT-4 and GPT-3.5-Turbo. They can be easily integrated into applications to add capabilities such as content generation, text completion, semantic search, and more.

This guide will walk you through the process of integrating APISIX with Azure OpenAI Service.

Prerequisite(s)

• Have an Azure account and log into the Azure portal.
• Install Docker.
• Install cURL to send requests to the services for validation.
• Follow the Getting Started Tutorial to start a new APISIX instance in Docker or on Kubernetes.

Request Access to Azure OpenAI Service

Before proceeding, you should first request access to Azure OpenAI Service as part of Microsoft's commitment to responsible AI by filling out a registration form.

Please request the access for GPT-3.5, GPT-3.5 Turbo, GPT-4, GPT-4 Turbo, and/or Embeddings Models to follow along.

Deploy an Azure OpenAI Service

Once the access is granted, search for Azure AI services, navigate to the Azure OpenAI in the left panel, and click Create Azure OpenAI:

Fill out the project and instance details:

In the Network tab, select the All networks option, or adjust accordingly per your infrastructure:

Continue with the setup until the deployment is complete:

Obtain API Information

Go to the Azure AI Foundry and select Chat. Click View Code:

Switch the command to curl and select the key authentication tab:

Copy the generated curl command, endpoint, and API key.

You can optionally save your API key to an environment variable:

export AZ_OPENAI_API_KEY=57cha9ee8e8a89a12c0aha174f180f4   # replace with your API key

Create a Route to Azure OpenAI

Create a route and configure the ai-proxy plugin as such:

Admin API

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "ai-proxy-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai-compatible",
        "auth": {
          "header": {
            "api-key": "'"$AZ_OPENAI_API_KEY"'"
          }
        },
        "options: {
          "model": "gpt-4"
        },
        "override": {
          "endpoint": "https://api7-auzre-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview"
        }
      }
    }
  }'

❶ Set the provider to openai-compatible when integrating with OpenAI-compatible API with a custom endpoint.

❷ Attach Azure OpenAI API key in the header.

❸ Override with the Azure OpenAI endpoint.

ADC

adc.yaml

services:
  - name: Azure OpenAI Service
    routes:
      - uris:
          - /anything
        name: azure-openai-route
        plugins:
          ai-proxy:
            provider: openai-compatible
            auth:
              header:
                api-key: 57cha9ee8e8a89a12c0aha174f180f4
            options:
              model: gpt-4
            override:
              endpoint: https://api7-auzre-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview

❶ Set the provider to openai-compatible when integrating with OpenAI-compatible API with a custom endpoint.

❷ Attach Azure OpenAI API key in the header.

❸ Override with the Azure OpenAI endpoint.

Synchronize the configuration to APISIX:

adc sync -f adc.yaml

Ingress Controller

Gateway API

azure-openai-route.yaml

apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: ingress-apisix
  name: ai-proxy-plugin-config
spec:
  plugins:
    - name: ai-proxy
      config:
        provider: openai-compatible
        auth:
          header:
            api-key: 57cha9ee8e8a89a12c0aha174f180f4
        options:
          model: gpt-4
        override:
          endpoint: https://api7-auzre-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: ingress-apisix
  name: azure-openai-route
spec:
  parentRefs:
  - name: apisix
  rules:
  - matches: 
    - path:
        type: Exact
        value: /anything
    filters:
    - type: ExtensionRef
      extensionRef:
        group: apisix.apache.org
        kind: PluginConfig
        name: ai-proxy-plugin-config

APISIX CRD

azure-openai-route.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: ingress-apisix
  name: azure-openai-route
spec:
  ingressClassName: apisix
  http:
    - name: azure-openai-route
      match:
        paths:
          - /anything
      plugins:
      - name: ai-proxy
        enable: true
        config:
          provider: openai-compatible
          auth:
            header:
              api-key: 57cha9ee8e8a89a12c0aha174f180f4
          options:
            model: gpt-4
          override:
            endpoint: https://api7-auzre-openai.openai.azure.com/openai/deployments/gpt-4/chat/completions?api-version=2024-02-15-preview

❶ Set the provider to openai-compatible when integrating with OpenAI-compatible API with a custom endpoint.

❷ Attach Azure OpenAI API key in the header.

❸ Override with the Azure OpenAI endpoint.

Apply the configuration to your cluster:

kubectl apply -f azure-openai-route.yaml

Verify

Send a request with the following prompts to the route:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "system",
        "content": "You are an AI assistant that helps people find information."
      },
      {
        "role": "user",
        "content": "Write me a 50-word introduction for Apache APISIX."
      }
    ]
  }'

You should receive a response similar to the following:

{
  "choices": [
    {
      ...,
      "message": {
        "content": "Apache APISIX is a modern, cloud-native API gateway built to handle high-performance and low-latency use cases. It offers a wide range of features, including load balancing, rate limiting, authentication, and dynamic routing, making it an ideal choice for microservices and cloud-native architectures.",
        "role": "assistant"
      }
    }
  ],
  ...
}

Next Steps

You have now learned how to integrate APISIX with Azure OpenAI Service. See Azure OpenAI Service REST API reference to learn more.

If you would like to stream the Azure API response, you can set the stream parameter to true for Azure OpenAI Service and use the proxy-buffering plugin in APISIX to disable NGINX's proxy_buffering directive, which avoids the server-sent events (SSE) being buffered.

In addition, you can integrate more capabilities that APISIX offers, such as rate limiting and caching, to improve system availability and user experience.