Speech and Audio

Applications use audio to transcribe or translate recorded speech, generate spoken output, add audio to a model turn, or sustain a live conversation. These workflows need different request and delivery models.

AISIX gives each workflow its own interface. Across them, the gateway authenticates callers, resolves model aliases, applies the access restrictions and rate limits supported by that interface, and records request telemetry.

Choose an Audio Interface

Choose the endpoint based on how audio participates in the application:

Goal	Endpoint
Transcribe or translate a recorded file	POST /v1/audio/transcriptions or POST /v1/audio/translations
Generate speech from text as a standalone task	POST /v1/audio/speech
Send audio in a chat message or receive audio in a complete chat response	POST /v1/chat/completions
Exchange audio over an interactive WebSocket session	GET /v1/realtime

The rest of this guide covers the standalone /v1/audio/* routes for transcription, translation, and speech generation.

AISIX resolves the caller-facing model alias, applies access checks and supported text guardrails, and forwards the request to an upstream that supports the same audio route. It preserves the endpoint-specific request and response shapes instead of converting them into a chat-style format.

Prerequisites

Before starting, prepare the following:

• A running AISIX gateway that can serve proxy requests.
• A caller API key that can access the model alias.
• A model alias backed by a provider and model that support the audio route you want to call.

Export the gateway connection and caller API key used by the examples:

# AISIX_PROXY has no trailing slash or endpoint path such as /v1.
# The local quickstarts use http://127.0.0.1:3000.
export AISIX_PROXY="YOUR_AISIX_GATEWAY_ORIGIN"
export AISIX_API_KEY="YOUR_CALLER_API_KEY"

Send a Transcription Request

Transcription is a multipart/form-data upload, not a JSON body. Send the audio file in the file field and the AISIX model alias in the model field:

curl -sS -X POST "${AISIX_PROXY}/v1/audio/transcriptions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -F "file=@meeting.wav" \
  -F "model=transcribe-prod"

For a successful request, the upstream returns the transcript:

{
  "text": "The quick brown fox jumps over the lazy dog."
}

AISIX rebuilds the multipart form with the upstream model ID and preserves the remaining fields. A configured input guardrail can block or mask a text-bearing prompt field before AISIX sends the form upstream.

To translate non-English speech into English text instead, send the same form to /v1/audio/translations.

Choose a Response Format

The optional response_format field selects the transcript representation. For a successful request, AISIX preserves the upstream response body and content type unless a configured output guardrail blocks or masks the transcript. The requested representation therefore remains intact when no guardrail changes it:

response_format	Response content type	Body
json (default)	application/json	{"text": "..."}
verbose_json	application/json	Transcript plus duration, language, and per-segment timings
text	text/plain	The transcript alone
srt	text/plain	SubRip subtitles with cue timings
vtt	text/plain	WebVTT subtitles with cue timings

Handle transcription responses by content type rather than assuming JSON. Only json and verbose_json produce a JSON body:

curl -sS -X POST "${AISIX_PROXY}/v1/audio/transcriptions" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -F "file=@meeting.wav" \
  -F "model=transcribe-prod" \
  -F "response_format=srt"

Format support is a property of the upstream model, not of AISIX. If a model rejects a format, inspect the error returned by AISIX and the provider's model documentation. Do not rely on the error body being a byte-for-byte copy of the upstream response.

Transcription Streaming

Some transcription models accept stream=true and return server-sent events. AISIX relays those events as the upstream produces them, so a client receives the transcript incrementally instead of waiting for the whole request to finish. The events pass through unchanged, and AISIX reads usage from the terminal event as they do.

An output guardrail that can block or mask a transcript is the exception. Such a guardrail has to inspect the whole transcript before any of it reaches the caller. AISIX therefore holds the response back, scans it, and then releases or blocks it — the same protection a non-streamed request gets. A guardrail in monitor mode never blocks, so it does not hold the response back; it observes the transcript once the stream ends.

Streaming support is model-specific. For example, OpenAI's file-transcription guide uses gpt-transcribe models for streaming, while the official SDK specification states that whisper-1 ignores stream. Check the upstream model documentation before relying on streamed transcription events.

Send a Speech Request

Send a speech-generation request through the gateway proxy with the AISIX model alias in the request body:

curl -sS -X POST "${AISIX_PROXY}/v1/audio/speech" \
  -H "Authorization: Bearer ${AISIX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-prod",
    "input": "Hello from AISIX.",
    "voice": "alloy"
  }' \
  --output aisix-speech.mp3

For a successful request, the output file should contain the audio bytes returned by the upstream provider. Handle the response as a binary file, not as a chat-style JSON response. AISIX forwards the audio as the provider produces it, so a client that plays the response can start on the first bytes instead of waiting for the whole file.

Check that the file was written as audio output:

file aisix-speech.mp3

You should see output that identifies the file as audio. The exact wording depends on the operating system and the upstream response format:

aisix-speech.mp3: MPEG ADTS, layer III, v2, 160 kbps, 24 kHz, Monaural

Audio Endpoint Behavior

Audio endpoints do not all use the same request or response shape:

Endpoint	Request body	Response body
Transcriptions	Multipart form with an audio file and model alias	Upstream transcription result, in the requested response_format
Translations	Multipart form with an audio file and model alias	Upstream translation result, in the requested response_format
Speech	JSON body with model alias, text input, and voice	Binary audio bytes

For transcription and translation requests, AISIX rebuilds the multipart form with the upstream model ID before forwarding it. It preserves the other form fields, including the uploaded file name and content type when they are present.

For speech requests, AISIX rewrites the model field in the JSON body and forwards the remaining request fields to the upstream provider.

For successful requests, the gateway preserves the upstream response body and content type unless a transcript guardrail changes or blocks the output. Clients should handle transcription and translation according to the requested response_format, and speech as binary audio output.

Provider Support

Audio support depends on the resolved provider and model. AISIX does not translate audio formats across provider families.

Use these routes with upstreams that expose matching OpenAI-style audio endpoints. If the upstream does not support the requested audio route, the failure is a provider capability or base-URL issue, not a caller-authentication issue.

Usage and Guardrail Behavior

Successful audio requests are attributed in gateway usage events. Token counts are populated only when the upstream response includes recognized token usage.

Transcription and translation requests can also report the audio length. AISIX reads the duration from a supported upstream response or, when needed, measures the uploaded file. This allows AISIX Cloud to price models billed by duration rather than by tokens, including requests that use a non-JSON response_format. Set the rate in Model Pricing. Speech requests report no audio duration, so AISIX duration pricing does not apply to them.

Input guardrails can inspect, block, or mask the input text on speech requests and the optional prompt field on transcription and translation requests before AISIX calls the provider. Output guardrails can inspect, block, or mask transcript text. If an output guardrail blocks a transcript, AISIX still records the billed usage because the upstream has already processed the audio.

Uploaded audio bytes and generated speech bytes are not scanned as text.

Troubleshoot Audio Requests

If a speech request succeeds but the client expects JSON, adjust the response handling. Speech returns audio bytes.

If a transcription or translation request returns 400 from AISIX or the upstream, check the multipart form construction. The request must include a model field and the expected audio file field.

If a speech guardrail does not block a request, check the request text. Speech guardrails inspect the input text, not the generated audio bytes.

If a requested transcription format fails, confirm that the resolved provider and model support it. If streamed transcription events arrive only after the request finishes, check whether the model is attached to an output guardrail that blocks or masks transcripts. Such a guardrail holds the response back by design. Also confirm that the upstream model supports stream=true.

Next Steps

You have now seen how AISIX forwards OpenAI-style audio requests and where audio response handling differs from JSON proxy routes.

Continue with Audio Input and Output with Chat Completions for audio inside a chat turn or Realtime API for an interactive session. Use Passthrough Routes for a provider-native route that AISIX does not model directly.