Audio Input and Output with Chat Completions

Audio-capable chat models can accept recorded audio inside a message and return generated audio in the same Chat Completions response. This fits turn-based applications that need a model to understand or answer with audio while keeping the OpenAI-compatible POST /v1/chat/completions request shape.

For standalone transcription, translation, or speech generation, use Speech and Audio. For interactive, bidirectional sessions, use the Realtime API.

AISIX preserves the OpenAI chat-audio request and response fields when the selected upstream uses a compatible OpenAI-shaped API. It does not translate these fields into another provider's native audio protocol.

Prerequisites

Before starting, prepare the following:

• A running AISIX gateway that can serve proxy requests.
• A caller API key that can access the model alias.
• A model alias backed by an audio-capable Chat Completions model. Its provider key must use the openai or azure-openai adapter, and the upstream must implement the OpenAI chat-audio request and response shape.
• A local WAV recording named question.wav for the audio-input example.
• curl, jq, Python 3, and the base64, file, and tr command-line utilities for the examples.

If you have not configured the upstream yet, follow OpenAI, Azure OpenAI, or Bring Your Own Endpoint. Choose a currently supported upstream audio model and create an AISIX alias for it.

For a routing alias, every eligible target must use one of these adapters and support the same chat-audio fields. A fallback to a text-only or differently shaped provider can lose the audio content or fail upstream.

Export the gateway connection and request values:

# AISIX_PROXY has no trailing slash or endpoint path such as /v1.
# The local quickstarts use http://127.0.0.1:3000.
export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"
export AISIX_MODEL="audio-chat-prod"

Generate an Audio Response

Request both text and audio output, then save the complete response:

curl -sS -X POST "${AISIX_PROXY}/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"${AISIX_MODEL}"'",
    "modalities": ["text", "audio"],
    "audio": {
      "voice": "alloy",
      "format": "wav"
    },
    "messages": [
      {
        "role": "user",
        "content": "Say: Your AISIX audio route is working."
      }
    ]
  }' > chat-audio-response.json

AISIX replaces the alias with the upstream model ID before dispatch. The upstream chooses which voices and output formats it accepts.

Inspect the returned audio metadata without printing the base64 payload:

jq '{
  model,
  audio: (.choices[0].message.audio | {
    id,
    transcript,
    expires_at,
    encoded_characters: (.data | length)
  })
}' chat-audio-response.json

The response keeps model set to the AISIX alias. The audio object contains the provider's audio identifier, base64 data, transcript, and expiration timestamp when the upstream returns those fields.

Decode the generated WAV file with the Python standard library:

python3 - <<'PY'
import base64
import json

with open("chat-audio-response.json", encoding="utf-8") as response_file:
  response = json.load(response_file)

audio = response["choices"][0]["message"]["audio"]
with open("aisix-chat-audio.wav", "wb") as audio_file:
  audio_file.write(base64.b64decode(audio["data"]))

print(audio.get("transcript", ""))
PY

file aisix-chat-audio.wav

The final command should identify a WAV file. If you request another format, use a matching filename and media player.

Send Audio in a Message

Encode a local recording and place it in an input_audio content block. This example asks the model to answer with audio so you can use the same response handling as the previous request:

base64 < question.wav | tr -d '\n' | \
  jq -Rs \
    --arg model "$AISIX_MODEL" \
    '{
    model: $model,
    modalities: ["text", "audio"],
    audio: {
      voice: "alloy",
      format: "wav"
    },
    messages: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Answer the question in this recording."
          },
          {
            type: "input_audio",
            input_audio: {
              data: .,
              format: "wav"
            }
          }
        ]
      }
    ]
  }' > chat-audio-input.json

curl -sS -X POST "${AISIX_PROXY}/v1/chat/completions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  --data @chat-audio-input.json \
  > chat-audio-response.json

An OpenAI-compatible adapter forwards the typed content blocks unchanged. It does not decode the recording.

Understand Current Behavior

Chat audio uses the ordinary Chat Completions authentication, model access, routing, retry, and request telemetry path. The audio-specific behavior has these boundaries:

• Stream or not. A complete response preserves message.audio. A streamed response relays delta.audio in the upstream's own shape, including its id, transcript, data, and trailing expires_at chunks, and with stream_options.include_usage the final usage chunk carries the audio token counts. This applies to OpenAI and OpenAI-compatible upstreams, Azure OpenAI, and the OpenAI-compatible routes of Amazon Bedrock and Vertex AI.
• Keep the provider protocol compatible. OpenAI-compatible adapters preserve the audio request fields and the message.audio object. A native-protocol upstream, such as Anthropic, Claude on Vertex AI or Amazon Bedrock, Vertex AI Gemini, or Bedrock Converse, receives an input_audio part as its own media block when it accepts that kind of media. When it does not, as with audio toward Claude, AISIX answers 400 naming the part as messages[i].content[j] instead of dropping it. AISIX does not translate the audio-output fields into a provider-native voice API.
• Apply guardrails to text separately from audio. Input guardrails can inspect the text content blocks, and output guardrails can inspect ordinary returned message text. They do not inspect the bytes in input_audio, the generated audio data, or the transcript nested inside message.audio or a streamed delta.audio.
• Treat audio cost as an upstream detail. The response's usage carries the audio token counts the upstream reported, and the usage event records them as prompt_audio_tokens and completion_audio_tokens. Control-plane pricing has no separate audio-token rate, so it bills them at the request's prompt and completion rates.

Because audio is base64-encoded inside JSON, request and response bodies are larger than the underlying binary files. Account for that expansion when setting client, proxy, or load-balancer body limits.

Troubleshoot Chat Audio

If the response succeeds but has no message.audio, check that the request includes the audio modality and an audio object, and that the upstream model supports audio output through Chat Completions. A text-only model can accept the HTTP request but reject or ignore unsupported audio fields upstream.

If AISIX returns an upstream decode or provider error, call the same upstream model directly with the provider's documented request shape. Confirm the model ID, voice, format, and audio-input encoding before changing gateway policy.

Use the Realtime API when the application needs incremental, bidirectional audio.

Next Steps

You have now sent and received audio inside an OpenAI-compatible chat request. Continue with Speech and Audio for standalone transcription or speech synthesis, or Realtime API for interactive voice sessions.