LlamaIndex

LlamaIndex is a framework for building applications that combine LLM calls with data connectors, indexes, retrieval, and query workflows. AISIX fits at the model request boundary, while LlamaIndex continues to own document loading, indexing, retrieval, and response assembly.

The LlamaIndex OpenAI-like integration provides OpenAILike, an LLM adapter for custom OpenAI-compatible endpoints. Use that adapter when a LlamaIndex application should send model requests through AISIX.

This guide uses LlamaIndex Python with llama-index-llms-openai-like. You will configure OpenAILike with an AISIX proxy URL, caller API key, and model alias.

Prerequisites

Before starting, prepare the following:

• A Python environment supported by LlamaIndex.
• A running AISIX gateway with the proxy listener available.
• An AISIX caller API key.
• A model alias the caller API key can access through the OpenAI-Compatible API.

Configure LlamaIndex

Install the OpenAI-like LLM integration if the application does not already include it:

pip install llama-index-llms-openai-like

Set the values that the LlamaIndex application will use:

# Replace with your values
export AISIX_BASE_URL="http://127.0.0.1:3000/v1"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export AISIX_MODEL="gpt-4o-prod"

Create an OpenAILike client with the AISIX values:

import os

from llama_index.core.llms import ChatMessage
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
  model=os.environ["AISIX_MODEL"],
  api_base=os.environ["AISIX_BASE_URL"],
  api_key=os.environ["AISIX_API_KEY"],
  is_chat_model=True,
  is_function_calling_model=False,
)

response = llm.chat([
  ChatMessage(role="user", content="Write one sentence about retrieval systems."),
])
print(response.message.content)

The model value is the AISIX model alias, not the upstream provider model ID. AISIX governs the model request path, while the LlamaIndex application continues to own data loading, indexing, retrieval, and response assembly. Set is_function_calling_model to True only when the selected alias supports OpenAI-compatible tool calling.

Verify the Route

Run the script from the shell where the AISIX environment variables are set.

When the request succeeds, verify the following results:

• The script prints a chat response.
• AISIX records a successful POST /v1/chat/completions request for the selected model alias.

If you use AISIX Cloud or an on-premises managed control plane, verify the request in the managed gateway logs. For self-hosted gateways, use your configured logs, metrics, or upstream provider logs.

Handle Compatibility Issues

If a request fails, first confirm that the caller API key can access the selected model alias and that api_base points to the AISIX proxy API root with /v1.

LlamaIndex applications often combine retrieval, chat, tool calling, and structured response parsing. Set is_function_calling_model to True only when the selected alias supports OpenAI-compatible tool calling, and validate the exact workflow before relying on gateway policy or telemetry for that path.

Next Steps

• OpenAI-Compatible API: review gateway-facing request behavior.
• Embeddings: configure embedding traffic if your LlamaIndex workflow uses AISIX for embeddings.
• Response Caching: cache eligible non-streaming chat responses.