ax-audio

Audio I/O Codegen Rules (@ax-llm/ax)

Use this skill for audio in Ax. Pick the smallest audio surface that matches the job:

• Use ai.transcribe(...) for batch speech-to-text.
• Use ai.speak(...) for batch text-to-speech.
• Use speech:audio signature outputs for structured programs that should return synthesized audio artifacts.
• Use .chat() audio config for conversational or realtime audio turns.

Core Rules

• Input :audio is an audio input value: { data, format?, mimeType?, sampleRate?, channels? }.
• Output :audio is a scripted audio artifact. The model returns plain text for that field; Ax synthesizes it after structured output parsing.
• Output audio JSON schema is model-facing string, not a binary object.
• Agents transcribe input audio fields before planner/executor/responder stages by default, so agent stages see text instead of base64 audio.
• Realtime and conversational audio still use .chat() and modelConfig.audio.
• Batch signature audio artifacts use forward-time speech options, not modelConfig.audio.

Direct Batch APIs

import { ai } from '@ax-llm/ax';

const llm = ai({ name: 'openai', apiKey: process.env.OPENAI_APIKEY! });

const transcript = await llm.transcribe({
  audio: { data: base64Wav, format: 'wav' },
  model: 'gpt-4o-mini-transcribe',
  language: 'en',
  prompt: 'Product support call',
});

const speech = await llm.speak({
  text: transcript.text,
  model: 'gpt-4o-mini-tts',
  voice: 'alloy',
  format: 'mp3',
});

console.log(transcript.text);
console.log(speech.data);
console.log(speech.transcript);

Providers without the requested batch audio capability throw AxMediaNotSupportedError.

Signature Audio Artifacts

import { ai, ax } from '@ax-llm/ax';

const llm = ai({ name: 'openai', apiKey: process.env.OPENAI_APIKEY! });
const say = ax('question:string -> speech:audio, summary:string');

const result = await say.forward(
  llm,
  { question: 'Explain retries in one sentence.' },
  {
    speech: {
      speak: { voice: 'alloy', format: 'mp3' },
      fields: {
        speech: { voice: 'alloy' },
      },
    },
  }
);

console.log(result.summary);
console.log(result.speech.data);
console.log(result.speech.mimeType);
console.log(result.speech.transcript);