Skip to main content
API7.ai Logo

LlamaIndex

LlamaIndex is a framework for building applications that combine LLM calls with data connectors, indexes, retrieval, and query workflows. AISIX fits at the model request boundary, while LlamaIndex continues to own document loading, indexing, retrieval, and response assembly.

The LlamaIndex OpenAI integration provides OpenAIResponses, an LLM adapter for the OpenAI Responses API with a custom API base. Use that adapter when a LlamaIndex application should send Responses API requests through AISIX.

This guide uses LlamaIndex Python with llama-index-llms-openai. You will configure OpenAIResponses with an AISIX proxy URL, caller API key, and model alias.

Prerequisites​

Before starting, prepare the following:

  • A Python environment supported by LlamaIndex.
  • A running AISIX gateway with the proxy listener available.
  • An AISIX caller API key.
  • A model alias the caller API key can access through the Responses API.

If AISIX is already deployed for your organization, obtain the gateway URL, model alias, and caller API key from the team that manages it. Otherwise, follow the Open-Source AISIX Gateway Quickstart or AISIX Cloud Quickstart, or contact API7 for Hybrid Cloud access.

Configure LlamaIndex​

Install the OpenAI LLM integration if the application does not already include it:

pip install llama-index-llms-openai

Set the values that the LlamaIndex application will use:

# Replace with your values
export AISIX_BASE_URL="http://127.0.0.1:3000/v1"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export AISIX_MODEL="gpt-4o-prod"
export AISIX_CONTEXT_WINDOW="128000"

Create an OpenAIResponses client with the AISIX values:

import os

from llama_index.core.llms import ChatMessage
from llama_index.llms.openai import OpenAIResponses

llm = OpenAIResponses(
  model=os.environ["AISIX_MODEL"],
  api_base=os.environ["AISIX_BASE_URL"],
  api_key=os.environ["AISIX_API_KEY"],
  context_window=int(os.environ["AISIX_CONTEXT_WINDOW"]),
)

response = llm.chat([
  ChatMessage(role="user", content="Write one sentence about retrieval systems."),
])
print(response.message.content)

The model value is the AISIX model alias, not the upstream provider model ID. Set AISIX_CONTEXT_WINDOW to the context size of the model behind the alias so LlamaIndex does not need to infer it from the alias name. AISIX governs the model request path, while the LlamaIndex application continues to own data loading, indexing, retrieval, and response assembly.

Verify the Integration​

Run the script from the shell where the AISIX environment variables are set.

When the request succeeds, verify the following results:

  • The script prints a chat response.
  • AISIX records a successful POST /v1/responses request for the selected model alias.

If you use the AISIX Cloud control plane, verify the request in the AISIX gateway logs. For standalone gateways, use your configured logs, metrics, or upstream provider logs.

If the request fails, first confirm that the caller API key can access the selected model alias and that api_base points to the AISIX proxy API root with /v1.

LlamaIndex also provides OpenAI-compatible chat adapters, including OpenAILike, for applications that still require Chat Completions compatibility. Use those adapters only when an existing workflow or integration depends on the Chat Completions route; for new OpenAI-family LlamaIndex integrations, prefer OpenAIResponses.

LlamaIndex applications often combine retrieval, chat, tool calling, and structured response parsing. Validate the exact workflow before relying on gateway policy or telemetry for that path.

Next Steps​

  • Responses API: review gateway-facing request behavior.
  • OpenAI-Compatible API: use the Chat Completions route when an existing LlamaIndex workflow requires it.
  • Embeddings: configure embedding traffic if your LlamaIndex workflow uses AISIX for embeddings.
  • Response Caching: cache eligible non-streaming chat responses.