ax-audio

Audio I/O Codegen Rules (@ax-llm/ax)

Use this skill for audio in Ax. Pick the smallest audio surface that matches the job:

• Use ai.transcribe(...) for batch speech-to-text.
• Use ai.speak(...) for batch text-to-speech.
• Use speech:audio signature outputs for structured programs that should return synthesized audio artifacts.
• Use .chat() audio config for conversational or realtime audio turns.

Core Rules

• Input :audio is an audio input value: { data, format?, mimeType?, sampleRate?, channels? }.
• Output :audio is a scripted audio artifact. The model returns plain text for that field; Ax synthesizes it after structured output parsing.
• Output audio JSON schema is model-facing string, not a binary object.
• Agents transcribe input audio fields before planner/executor/responder stages by default, so agent stages see text instead of base64 audio.
• Realtime and conversational audio still use .chat() and modelConfig.audio.
• Batch signature audio artifacts use forward-time speech options, not modelConfig.audio.

Direct Batch APIs

import { ai } from '@ax-llm/ax';

const llm = ai({ name: 'openai', apiKey: process.env.OPENAI_APIKEY! });

const transcript = await llm.transcribe({
  audio: { data: base64Wav, format: 'wav' },
  model: 'gpt-4o-mini-transcribe',
  language: 'en',
  prompt: 'Product support call',
});

const speech = await llm.speak({
  text: transcript.text,
  model: 'gpt-4o-mini-tts',
  voice: 'alloy',
  format: 'mp3',
});

console.log(transcript.text);
console.log(speech.data);
console.log(speech.transcript);

Providers without the requested batch audio capability throw AxMediaNotSupportedError.

OpenAI's speak() defaults to gpt-4o-mini-tts with the alloy voice and mp3. format: 'pcm' (or 'pcm16') asks OpenAI for its pcm output, 16-bit PCM at 24 kHz. Mistral's speak() defaults to voxtral-mini-tts-2603 and sends the voice as voice_id.

OpenAI also offers gpt-transcribe (AxAIOpenAIModel.GPTTranscribe), billed per minute of audio; it accepts responseFormat: 'json' or 'text' only.