Skip to main content
API7.ai Logo
Version: 3.9.x

Implement RAG at the Gateway Layer

Retrieval-Augmented Generation (RAG) grounds LLM responses in context retrieved from a knowledge base. API7 AI Gateway can perform this retrieval before proxying a request, which centralizes the enrichment flow instead of requiring each application to implement it. The current ai-rag plugin uses Azure OpenAI for embeddings and Azure AI Search for vector search.

Overview​

A RAG request follows this flow:

  1. Client sends a chat request to API7 AI Gateway.
  2. Gateway uses ai-rag to generate embeddings and run vector search against Azure AI Search.
  3. Gateway injects retrieved context into the prompt.
  4. Gateway forwards the enriched request to Azure OpenAI through ai-proxy.
  5. Client receives a grounded answer.

Prerequisites​

  • cURL is installed on the client used for validation.
  • An API7 Gateway instance is running with the ai-proxy and ai-rag plugins available.
  • An Azure OpenAI resource in Microsoft Foundry with chat and embedding deployments. Record the resource name, API key, and both deployment names.
  • An Azure AI Search service has a populated index with a vector field whose name and dimensions match the request. The examples use contentVector with 1,024 dimensions, matching Microsoft's basic vector workflow.
  • ADC is installed and connected to the target Gateway Group if you use the ADC configuration.

Configure the RAG Plugin​

Configure ai-rag and ai-proxy on the same route so retrieval and generation happen in one request path.

Obtain a Dashboard token, then export it with the target gateway group and an existing service ID. The quickstart Gateway Group uses default. If needed, first create or reuse a service.

export API_KEY=replace-with-dashboard-token
export GATEWAY_GROUP_ID=default
export SERVICE_ID=replace-with-service-id

Create the route through the Admin API:

curl -k "https://localhost:7443/apisix/admin/routes?gateway_group_id=${GATEWAY_GROUP_ID}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -d @- <<EOF
  {
    "id": "rag-azure",
    "service_id": "$SERVICE_ID",
    "paths": ["/v1/chat/completions"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai-compatible",
        "auth": {
          "header": {
            "api-key": "YOUR_AZURE_OPENAI_KEY"
          }
        },
        "options": {
          "model": "YOUR_CHAT_DEPLOYMENT"
        },
        "override": {
          "endpoint": "https://YOUR-RESOURCE.openai.azure.com/openai/v1/chat/completions"
        }
      },
      "ai-rag": {
        "embeddings_provider": {
          "azure_openai": {
            "endpoint": "https://YOUR-RESOURCE.openai.azure.com/openai/v1/embeddings",
            "api_key": "YOUR_AZURE_OPENAI_KEY"
          }
        },
        "vector_search_provider": {
          "azure_ai_search": {
            "endpoint": "https://YOUR-SEARCH.search.windows.net/indexes/YOUR-INDEX/docs/search?api-version=2024-07-01",
            "api_key": "YOUR_AZURE_SEARCH_KEY"
          }
        }
      }
    }
  }
EOF

❶ Set provider to openai-compatible so the chat deployment name remains in the Azure OpenAI v1 request body.

❷ Set the Azure OpenAI v1 Chat Completions endpoint.

❸ Configure ai-rag with Azure OpenAI for embeddings and Azure AI Search for vector retrieval. The embedding endpoint uses the same Azure OpenAI resource at /openai/v1/embeddings.

For the full configuration reference, see ai-rag and ai-proxy.

Validate the Configuration​

Send a request that includes the ai_rag field. This example requests a 1,024-dimension embedding; change it to match the Azure AI Search vector field, or omit dimensions when the field uses the embedding deployment's default dimensions.

curl "http://127.0.0.1:9080/v1/chat/completions" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": "YOUR_CHAT_DEPLOYMENT",
    "messages": [
      {
        "role": "user",
        "content": "Based on our internal docs, what are the main capabilities of API7 AI Gateway?"
      }
    ],
    "ai_rag": {
      "vector_search": {
        "fields": "contentVector"
      },
      "embeddings": {
        "model": "YOUR_EMBEDDING_DEPLOYMENT",
        "input": "API7 AI Gateway capabilities overview",
        "dimensions": 1024
      }
    }
  }'

❶ Use a question that depends on your indexed knowledge base so you can confirm grounding behavior.

❷ ai_rag.vector_search.fields identifies the vector field used for retrieval.

❸ ai_rag.embeddings identifies the embedding deployment and supplies the text and dimensions used for vector retrieval. The dimensions must match the vector field in the search index.

You should receive a standard chat completion response with an answer grounded in your indexed documents. Compared with a request without ai_rag, the answer should be more specific and aligned with your internal knowledge base.

Best Practices​

  • Keep your knowledge base and index up to date. Stale documents reduce answer quality.
  • Keep the embedding dimensions aligned with the Azure AI Search vector field.
  • Monitor token usage closely. Added context increases prompt tokens and cost.

Next Steps​