Audio Input and Output with Chat Completions

Audio-capable chat models can accept recorded audio inside a message and return generated audio in the same Chat Completions response. This fits turn-based applications that need a model to understand or answer with audio while keeping the OpenAI-compatible POST /v1/chat/completions request shape.

For standalone transcription, translation, or speech generation, use Speech and Audio. For interactive, bidirectional sessions, use the Realtime API.

AISIX preserves the OpenAI chat-audio request and response fields when the selected upstream uses a compatible OpenAI-shaped API. It does not translate these fields into another provider's native audio protocol.

Prerequisites

Before starting, prepare the following:

• A running AISIX gateway that can serve proxy requests.
• A caller API key that can access the model alias.
• A model alias backed by an audio-capable Chat Completions model. Its provider key must use the openai or azure-openai adapter, and the upstream must implement the OpenAI chat-audio request and response shape.
• A local WAV recording named question.wav for the audio-input example.
• curl, jq, Python 3, and the base64, file, and tr command-line utilities for the examples.

If you have not configured the upstream yet, follow OpenAI, Azure OpenAI, or Bring Your Own Endpoint. Choose a currently supported upstream audio model and create an AISIX alias for it.

For a routing alias, every eligible target must use one of these adapters and support the same chat-audio fields. A fallback to a text-only or differently shaped provider can lose the audio content or fail upstream.

Export the gateway connection and request values:

# AISIX_PROXY has no trailing slash or endpoint path such as /v1.
# The local quickstarts use http://127.0.0.1:3000.
export AISIX_PROXY="YOUR_AISIX_GATEWAY_ORIGIN"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export AISIX_MODEL="audio-chat-prod"

Generate an Audio Response

Request both text and audio output, then save the complete response:

curl -sS -X POST "${AISIX_PROXY}/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"${AISIX_MODEL}"'",
    "modalities": ["text", "audio"],
    "audio": {
      "voice": "alloy",
      "format": "wav"
    },
    "messages": [
      {
        "role": "user",
        "content": "Say: Your AISIX audio route is working."
      }
    ]
  }' > chat-audio-response.json

AISIX replaces the alias with the upstream model ID before dispatch. The upstream chooses which voices and output formats it accepts.

Inspect the returned audio metadata without printing the base64 payload:

jq '{
  model,
  audio: (.choices[0].message.audio | {
    id,
    transcript,
    expires_at,
    encoded_characters: (.data | length)
  })
}' chat-audio-response.json

The response keeps model set to the AISIX alias. The audio object contains the provider's audio identifier, base64 data, transcript, and expiration timestamp when the upstream returns those fields.

Decode the generated WAV file with the Python standard library:

python3 - <<'PY'
import base64
import json

with open("chat-audio-response.json", encoding="utf-8") as response_file:
  response = json.load(response_file)

audio = response["choices"][0]["message"]["audio"]
with open("aisix-chat-audio.wav", "wb") as audio_file:
  audio_file.write(base64.b64decode(audio["data"]))

print(audio.get("transcript", ""))
PY

file aisix-chat-audio.wav

The final command should identify a WAV file. If you request another format, use a matching filename and media player.

Send Audio in a Message

Encode a local recording and place it in an input_audio content block. This example asks the model to answer with audio so you can use the same response handling as the previous request:

base64 < question.wav | tr -d '\n' | \
  jq -Rs \
    --arg model "$AISIX_MODEL" \
    '{
    model: $model,
    modalities: ["text", "audio"],
    audio: {
      voice: "alloy",
      format: "wav"
    },
    messages: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Answer the question in this recording."
          },
          {
            type: "input_audio",
            input_audio: {
              data: .,
              format: "wav"
            }
          }
        ]
      }
    ]
  }' > chat-audio-input.json

curl -sS -X POST "${AISIX_PROXY}/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  --data @chat-audio-input.json \
  > chat-audio-response.json

AISIX forwards the typed content blocks unchanged through an OpenAI-compatible adapter. It does not decode the recording or convert it to another provider's audio-input shape.

Understand Current Behavior

Chat audio uses the ordinary Chat Completions authentication, model access, routing, retry, and request telemetry path. The audio-specific behavior has these boundaries:

• Use a non-streaming request. Omit stream or set it to false. AISIX preserves message.audio on a complete response, but it does not currently return streamed delta.audio chunks.
• Keep the provider protocol compatible. The openai and azure-openai adapters preserve the audio request fields and the non-streaming message.audio object. Other adapters can reduce typed message content to text and do not translate the audio fields into a provider-native voice API.
• Apply guardrails to text separately from audio. Input guardrails can inspect the text content blocks, and output guardrails can inspect ordinary returned message text. They do not inspect the bytes in input_audio, the generated audio data, or the transcript nested inside message.audio.
• Treat Cloud audio cost as an upstream detail. AISIX records the normalized prompt and completion totals the upstream reports, but it does not retain separate audio-token counts. AISIX Cloud pricing therefore cannot apply distinct text-token and audio-token rates to the same Chat Completions request.

Because audio is base64-encoded inside JSON, request and response bodies are larger than the underlying binary files. Account for that expansion when setting client, proxy, or load-balancer body limits.

Troubleshoot Chat Audio

If the response succeeds but has no message.audio, check that the request includes the audio modality and an audio object, and that the upstream model supports audio output through Chat Completions. A text-only model can accept the HTTP request but reject or ignore unsupported audio fields upstream.

If AISIX returns an upstream decode or provider error, call the same upstream model directly with the provider's documented request shape. Confirm the model ID, voice, format, and audio-input encoding before changing gateway policy.

If the request works without streaming but produces no audio when stream is enabled, keep the request non-streaming. Use the Realtime API when the application needs incremental, bidirectional audio.

Next Steps

You have now sent and received audio inside an OpenAI-compatible chat request. Continue with Speech and Audio for standalone transcription or speech synthesis, or Realtime API for interactive voice sessions.