Haystack

Haystack is a framework for building LLM applications with pipelines, retrieval, indexing, document processing, agents, and generators. AISIX fits at the model request boundary, while Haystack continues to own pipeline execution and retrieval.

The Haystack OpenAIResponsesChatGenerator component supports custom OpenAI-compatible deployments through api_base_url. Use that setting when a Haystack pipeline should send Responses API requests through AISIX.

This guide uses Haystack with OpenAIResponsesChatGenerator. You will configure the generator with an AISIX proxy URL, caller API key, and model alias.

Prerequisites

Before starting, prepare the following:

• A Python environment supported by Haystack.
• A running AISIX gateway with the proxy listener available.
• An AISIX caller API key.
• A model alias the caller API key can access through the Responses API.

Configure Haystack

Install Haystack if the application does not already include it:

pip install haystack-ai

Set the values that the Haystack application will use:

# Replace with your values
export AISIX_BASE_URL="http://127.0.0.1:3000/v1"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export AISIX_MODEL="gpt-4o-prod"

Create an OpenAIResponsesChatGenerator with the AISIX values:

import os

from haystack.components.generators.chat import OpenAIResponsesChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret

generator = OpenAIResponsesChatGenerator(
  model=os.environ["AISIX_MODEL"],
  api_base_url=os.environ["AISIX_BASE_URL"],
  api_key=Secret.from_token(os.environ["AISIX_API_KEY"]),
)

response = generator.run([
  ChatMessage.from_user("Write one sentence about retrieval pipelines."),
])

print(response["replies"][0].text)

The model value is the AISIX model alias, not the upstream provider model ID. AISIX governs the model request path, while Haystack still owns pipeline components, prompt builders, retrievers, ranking components, and response handling.

Haystack also provides OpenAIChatGenerator for Chat Completions. Use that component when a pipeline needs broad OpenAI-compatible chat support instead of Responses-specific features.

Verify the Integration

Run the script from the shell where the AISIX environment variables are set.

When the request succeeds, verify the following results:

• The script prints a chat response.
• AISIX records a successful POST /v1/responses request for the selected model alias.

If you use AISIX Cloud or an on-premises managed control plane, verify the request in the managed gateway logs. For self-hosted gateways, use your configured logs, metrics, or upstream provider logs.

If the request fails, first confirm that the caller API key can access the selected model alias and that api_base_url points to the AISIX proxy API root with /v1.

Haystack applications often combine generators with retrievers, ranking components, tools, and structured output. Validate the exact pipeline before relying on gateway policy or telemetry for streaming, tool calls, or structured output.

Next Steps

• Responses API: review gateway-facing Responses behavior.
• OpenAI-Compatible API: use Chat Completions when a workflow needs broad OpenAI-compatible support.
• Embeddings: configure embedding traffic if your Haystack workflow uses AISIX for embeddings.
• Rerank: configure rerank traffic if your Haystack pipeline uses rerank models through AISIX.