# Audio Input and Output with Chat Completions

Audio-capable chat models can accept recorded audio inside a message and return generated audio in the same Chat Completions response. This fits turn-based applications that need a model to understand or answer with audio while keeping the OpenAI-compatible `POST /v1/chat/completions` request shape.

For standalone transcription, translation, or speech generation, use [Speech and Audio](https://docs.api7.ai/ai-gateway/endpoints/audio.md). For interactive, bidirectional sessions, use the [Realtime API](https://docs.api7.ai/ai-gateway/endpoints/realtime.md).

AISIX preserves the OpenAI chat-audio request and response fields when the selected upstream uses a compatible OpenAI-shaped API. It does not translate these fields into another provider's native audio protocol.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

Before starting, prepare the following:

* A running AISIX gateway that can serve proxy requests.
* A caller API key that can access the model alias.
* A model alias backed by an audio-capable Chat Completions model. Its provider key must use the `openai` or `azure-openai` adapter, and the upstream must implement the OpenAI chat-audio request and response shape.
* A local WAV recording named `question.wav` for the audio-input example.
* `curl`, `jq`, Python 3, and the `base64`, `file`, and `tr` command-line utilities for the examples.

If you have not configured the upstream yet, follow [OpenAI](https://docs.api7.ai/ai-gateway/providers/openai.md), [Azure OpenAI](https://docs.api7.ai/ai-gateway/providers/azure-openai.md), or [Bring Your Own Endpoint](https://docs.api7.ai/ai-gateway/providers/bring-your-own-endpoint.md). Choose a currently supported upstream audio model and create an AISIX alias for it.

For a routing alias, every eligible target must use one of these adapters and support the same chat-audio fields. A fallback to a text-only or differently shaped provider can lose the audio content or fail upstream.

Export the gateway connection and request values:

```
# AISIX_PROXY has no trailing slash or endpoint path such as /v1.

# The local quickstarts use http://127.0.0.1:3000.

export AISIX_PROXY="YOUR_AISIX_GATEWAY_URL"

export AISIX_API_KEY="YOUR_CALLER_API_KEY"

export AISIX_MODEL="audio-chat-prod"
```

## Generate an Audio Response[​](#generate-an-audio-response "Direct link to Generate an Audio Response")

Request both text and audio output, then save the complete response:

```
curl -sS -X POST "${AISIX_PROXY}/v1/chat/completions" \

  -H "Authorization: Bearer ${AISIX_API_KEY}" \

  -H "Content-Type: application/json" \

  -d '{

    "model": "'"${AISIX_MODEL}"'",

    "modalities": ["text", "audio"],

    "audio": {

      "voice": "alloy",

      "format": "wav"

    },

    "messages": [

      {

        "role": "user",

        "content": "Say: Your AISIX audio route is working."

      }

    ]

  }' > chat-audio-response.json
```

AISIX replaces the alias with the upstream model ID before dispatch. The upstream chooses which voices and output formats it accepts.

Inspect the returned audio metadata without printing the base64 payload:

```
jq '{

  model,

  audio: (.choices[0].message.audio | {

    id,

    transcript,

    expires_at,

    encoded_characters: (.data | length)

  })

}' chat-audio-response.json
```

The response keeps `model` set to the AISIX alias. The `audio` object contains the provider's audio identifier, base64 data, transcript, and expiration timestamp when the upstream returns those fields.

Decode the generated WAV file with the Python standard library:

```
python3 - <<'PY'

import base64

import json



with open("chat-audio-response.json", encoding="utf-8") as response_file:

  response = json.load(response_file)



audio = response["choices"][0]["message"]["audio"]

with open("aisix-chat-audio.wav", "wb") as audio_file:

  audio_file.write(base64.b64decode(audio["data"]))



print(audio.get("transcript", ""))

PY



file aisix-chat-audio.wav
```

The final command should identify a WAV file. If you request another format, use a matching filename and media player.

## Send Audio in a Message[​](#send-audio-in-a-message "Direct link to Send Audio in a Message")

Encode a local recording and place it in an `input_audio` content block. This example asks the model to answer with audio so you can use the same response handling as the previous request:

```
base64 < question.wav | tr -d '\n' | \

  jq -Rs \

    --arg model "$AISIX_MODEL" \

    '{

    model: $model,

    modalities: ["text", "audio"],

    audio: {

      voice: "alloy",

      format: "wav"

    },

    messages: [

      {

        role: "user",

        content: [

          {

            type: "text",

            text: "Answer the question in this recording."

          },

          {

            type: "input_audio",

            input_audio: {

              data: .,

              format: "wav"

            }

          }

        ]

      }

    ]

  }' > chat-audio-input.json



curl -sS -X POST "${AISIX_PROXY}/v1/chat/completions" \

  -H "Authorization: Bearer ${AISIX_API_KEY}" \

  -H "Content-Type: application/json" \

  --data @chat-audio-input.json \

  > chat-audio-response.json
```

An OpenAI-compatible adapter forwards the typed content blocks unchanged. It does not decode the recording.

## Understand Current Behavior[​](#understand-current-behavior "Direct link to Understand Current Behavior")

Chat audio uses the ordinary Chat Completions authentication, model access, routing, retry, and request telemetry path. The audio-specific behavior has these boundaries:

* **Stream or not.** A complete response preserves `message.audio`. A streamed response relays `delta.audio` in the upstream's own shape, including its `id`, `transcript`, `data`, and trailing `expires_at` chunks, and with `stream_options.include_usage` the final usage chunk carries the audio token counts. This applies to OpenAI and OpenAI-compatible upstreams, Azure OpenAI, and the OpenAI-compatible routes of Amazon Bedrock and Vertex AI.
* **Keep the provider protocol compatible.** OpenAI-compatible adapters preserve the audio request fields and the `message.audio` object. A native-protocol upstream, such as Anthropic, Claude on Vertex AI or Amazon Bedrock, Vertex AI Gemini, or Bedrock Converse, receives an `input_audio` part as its own media block when it accepts that kind of media. When it does not, as with audio toward Claude, AISIX answers `400` naming the part as `messages[i].content[j]` instead of dropping it. AISIX does not translate the audio-output fields into a provider-native voice API.
* **Apply guardrails to text separately from audio.** Input guardrails can inspect the text content blocks, and output guardrails can inspect ordinary returned message text. They do not inspect the bytes in `input_audio`, the generated audio data, or the transcript nested inside `message.audio` or a streamed `delta.audio`.
* **Treat audio cost as an upstream detail.** The response's `usage` carries the audio token counts the upstream reported, and the usage event records them as [`prompt_audio_tokens` and `completion_audio_tokens`](https://docs.api7.ai/ai-gateway/observability/metrics-and-usage-events.md#audio-and-image-tokens). Control-plane pricing has no separate audio-token rate, so it bills them at the request's prompt and completion rates.

Because audio is base64-encoded inside JSON, request and response bodies are larger than the underlying binary files. Account for that expansion when setting client, proxy, or load-balancer body limits.

## Troubleshoot Chat Audio[​](#troubleshoot-chat-audio "Direct link to Troubleshoot Chat Audio")

If the response succeeds but has no `message.audio`, check that the request includes the audio modality and an `audio` object, and that the upstream model supports audio output through Chat Completions. A text-only model can accept the HTTP request but reject or ignore unsupported audio fields upstream.

If AISIX returns an upstream decode or provider error, call the same upstream model directly with the provider's documented request shape. Confirm the model ID, voice, format, and audio-input encoding before changing gateway policy.

Use the [Realtime API](https://docs.api7.ai/ai-gateway/endpoints/realtime.md) when the application needs incremental, bidirectional audio.

## Next Steps[​](#next-steps "Direct link to Next Steps")

You have now sent and received audio inside an OpenAI-compatible chat request. Continue with [Speech and Audio](https://docs.api7.ai/ai-gateway/endpoints/audio.md) for standalone transcription or speech synthesis, or [Realtime API](https://docs.api7.ai/ai-gateway/endpoints/realtime.md) for interactive voice sessions.
