Skip to main content
API7.ai Logo

ai-rag

The ai-rag plugin implements the retrieval step of a Retrieval-Augmented Generation (RAG) request flow. It generates an embedding from the request and performs a vector search. It then adds the retrieved content to the protocol-specific LLM input and removes the ai_rag object before proxying the request.

The current implementation supports Azure OpenAI for embeddings and Azure AI Search for vector search. Use the ai-proxy plugin in the same request flow to proxy the augmented request to the LLM provider. The plugin does not create or populate a search index; prepare the index and its content before sending requests through APISIX.

Behavior by Request Format​

The plugin enriches Chat Completions, Responses API, Anthropic Messages, and Bedrock Converse requests using each protocol's native prompt structure.

The gateway identifies each request by checking URI-specific rules before body-only rules:

  • Bedrock Converse requires a URI ending in /converse and a messages array.
  • Anthropic Messages requires a URI ending in /v1/messages.
  • Responses API requires a URI ending in /v1/responses and an input field.
  • Chat Completions uses a messages array.
  • Embeddings uses input after the earlier rules do not match.
  • Other non-empty JSON objects use passthrough after none of the earlier rules match.
Request formatContext enrichment
Bedrock ConverseAppends the retrieved context as a user message in messages.
Anthropic MessagesAppends the retrieved context as a user message in messages.
Responses APIAppends the retrieved context to input.
Chat CompletionsAppends the retrieved context as a user message in messages.
EmbeddingsDoes not enrich the request. The nested ai_rag.embeddings object configures the embedding input used internally for retrieval.
Other JSON (passthrough)Does not enrich the request.

Examples​

To follow the example, create an Azure account and complete the following setup:

  • In Microsoft Foundry, create or open an Azure OpenAI resource and deploy a chat model and an embedding model. Record the Azure OpenAI resource name, API key, and both deployment names.
  • Complete Microsoft's basic vector workflow to create the vectest index and load the sample data. Its defaults create the contentVector field with 1,024 dimensions, matching this example. If you use another index, replace the index, field, and dimension values throughout the workflow.
  • Obtain the Azure AI Search API key and service endpoint.

Save the API keys and endpoints to environment variables:

# replace with your values

export AZ_OPENAI_DOMAIN=https://your-openai-resource.openai.azure.com
export AZ_OPENAI_API_KEY=replace-with-azure-openai-api-key
export AZ_CHAT_DEPLOYMENT=replace-with-chat-deployment-name
export AZ_EMBEDDING_DEPLOYMENT=replace-with-embedding-deployment-name
export AZ_CHAT_ENDPOINT=${AZ_OPENAI_DOMAIN}/openai/v1/chat/completions
export AZ_EMBEDDINGS_ENDPOINT=${AZ_OPENAI_DOMAIN}/openai/v1/embeddings

export AZ_AI_SEARCH_SVC_DOMAIN=https://your-search-service.search.windows.net
export AZ_AI_SEARCH_KEY=replace-with-azure-ai-search-api-key
export AZ_AI_SEARCH_INDEX=vectest
export AZ_AI_SEARCH_ENDPOINT=${AZ_AI_SEARCH_SVC_DOMAIN}/indexes/${AZ_AI_SEARCH_INDEX}/docs/search?api-version=2024-07-01

Enrich Azure OpenAI Responses​

This example uses ai-proxy to send augmented requests to Azure OpenAI. Before proxying, ai-rag generates embeddings and searches Azure AI Search for context.

Create a route with ai-rag and ai-proxy:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d @- <<EOF
  {
    "id": "ai-rag-route",
    "uri": "/rag",
    "plugins": {
      "ai-rag": {
        "embeddings_provider": {
          "azure_openai": {
            "endpoint": "$AZ_EMBEDDINGS_ENDPOINT",
            "api_key": "$AZ_OPENAI_API_KEY"
          }
        },
        "vector_search_provider": {
          "azure_ai_search": {
            "endpoint": "$AZ_AI_SEARCH_ENDPOINT",
            "api_key": "$AZ_AI_SEARCH_KEY"
          }
        }
      },
      "ai-proxy": {
        "provider": "openai-compatible",
        "auth": {
          "header": {
            "api-key": "$AZ_OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "$AZ_CHAT_DEPLOYMENT"
        },
        "override": {
          "endpoint": "$AZ_CHAT_ENDPOINT"
        }
      }
    }
  }
EOF

Send a chat request with the embedding deployment, vector field, and retrieval input in the ai_rag object:

curl "http://127.0.0.1:9080/rag" -X POST \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{
  "messages": [
    {
      "role": "user",
      "content": "Which Azure services are good for DevOps?"
    }
  ],
  "ai_rag": {
    "vector_search": {
      "fields": "contentVector"
    },
    "embeddings": {
      "model": "$AZ_EMBEDDING_DEPLOYMENT",
      "input": "Which Azure services are good for DevOps?",
      "dimensions": 1024
    }
  }
}
EOF

You should receive an HTTP/1.1 200 OK response whose answer uses content retrieved from the configured index. The response contains fields similar to these:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Azure DevOps and related services support source control, build pipelines, testing, and deployment workflows...",
        "role": "assistant"
      }
    }
  ],
  "object": "chat.completion"
}