Proxy OpenAI Requests
OpenAI provides access to state-of-the-art AI models, such as GPT-3, for various applications including natural language processing, text generation, and more. Integrating OpenAI's APIs into your applications can unlock powerful capabilities for text analysis, content generation, and other AI-driven tasks.
APISIX provides capabilities for secret management, response streaming, rate limiting, and more, making it an excellent choice for proxying requests from OpenAI's API endpoints.
This guide will show you how to configure APISIX to integrate with OpenAI APIs to proxy user requests and model responses, using the ai-proxy
plugin.
Prerequisite(s)
- Install Docker.
- Install cURL to send requests to the services for validation.
- Follow the Getting Started Tutorial to start a new APISIX instance in Docker or on Kubernetes.
Obtain an OpenAI API Key
Create an OpenAI account and an API key before proceeding. You can optionally save the key to an environment variable as such:
export OPENAI_API_KEY=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 # replace with your API key
Create a Route to OpenAI API
Create a route with the ai-proxy
plugin as such:
- Admin API
- ADC
- Ingress Controller
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '{
"id": "openai-chat",
"uri": "/anything",
"plugins": {
"ai-proxy": {
"provider": "openai",
"auth": {
"header": {
"Authorization": "Bearer '"$OPENAI_API_KEY"'"
}
},
"options": {
"model": "gpt-3.5-turbo"
}
}
}
}'
❶ Set the provider to openai
, which will proxy requests to the OpenAI endpoint.
❷ Attach OpenAPI API key to Authorization
request header.
❸ Set the model to gpt-3.5-turbo
.
services:
- name: OpenAI Service
routes:
- uris:
- /anything
name: openai-chat
plugins:
ai-proxy:
provider: openai
auth:
header:
Authorization: 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
options:
model: "gpt-3.5-turbo"
❶ Set the provider to openai
, which will proxy requests to the OpenAI endpoint.
❷ Attach OpenAPI API key to Authorization
request header.
❸ Set the model to gpt-3.5-turbo
.
Synchronize the configuration to APISIX:
adc sync -f adc.yaml
Create a Kubernetes manifest file to configure a route:
- Gateway API
- APISIX CRD
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
name: ai-proxy-plugin-config
spec:
plugins:
- name: ai-proxy
config:
provider: openai
auth:
header:
Authorization: "Bearer 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"
options:
model: "gpt-3.5-turbo"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: openai-chat
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: ai-proxy-plugin-config
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: openai-route
spec:
ingressClassName: apisix
http:
- name: openai-route
match:
paths:
- /anything
plugins:
- name: ai-proxy
enable: true
config:
provider: openai
auth:
header:
Authorization: "Bearer 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"
options:
model: "gpt-3.5-turbo"
❶ Set the provider to openai
, which will proxy requests to the OpenAI endpoint.
❷ Attach OpenAPI API key to Authorization
request header.
❸ Set the model to gpt-3.5-turbo
.
Apply the configuration to your cluster:
kubectl apply -f openai-route.yaml
Verify
Send a request with the following prompts to the route:
curl "http://127.0.0.1:9080/anything" -X POST \
-H "Content-Type: application/json" \
-d '{
"messages": [
{
"role": "system",
"content": "You are a computer scientist."
},
{
"role": "user",
"content": "Explain in one sentence what a Turing machine is."
}
]
}'
You should receive a response similar to the following:
{
...,
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "A Turing machine is an abstract mathematical model that manipulates symbols on an infinite tape according to a set of rules, representing the concept of a general-purpose computer."
},
"logprobs": null,
"finish_reason": "stop"
}
],
...
}
See OpenAI's API specifications for more information about how to compose the request.
Next Steps
You have now learned how to integrate APISIX with OpenAI. See OpenAI's API reference to learn more about OpenAI's capabilities.
If you would like to integrate with OpenAI's streaming API, you can use the proxy-buffering
plugin to disable NGINX's proxy_buffering
directive to avoid server-sent events (SSE) being buffered.
In addition, you can integrate more capabilities that APISIX offers, such as rate limiting and caching, to improve system availability and user experience.