nvidia-speech-nim

NVIDIA Speech NIM

Use this skill when an agent must plan or implement a speech workflow with NVIDIA Speech NIM microservices rather than a generic speech API. Treat NVIDIA Speech NIM as a deployable containerized speech stack: the application talks to independent ASR, TTS, and NMT NIM containers over HTTP, gRPC, or WebSocket APIs, and the agent remains responsible for model/profile choice, orchestration, rights checks, audio custody, and QA.

Facts in this skill were verified from official NVIDIA sources on 2026-07-10 unless noted otherwise.

First classify the job

Separate the user's request into one or more speech capabilities:

• ASR/STT: audio to text for transcription, captions, subtitles, call analytics, meeting notes, voice-agent input, or speech-to-text translation where an ASR model supports translation.
• TTS: text to speech for narration, localized voice-over, conversational output, accessibility, kiosk/agent responses, or approved voice cloning.
• NMT: text translation between languages. NVIDIA's documented Speech NIM translation route is a separate NMT container, not hidden inside every ASR or TTS call.
• Speech-to-speech translation: a pipeline that chains ASR -> NMT -> TTS. Do not present it as a single monolithic Speech NIM unless the selected model/card explicitly exposes an end-to-end speech-to-speech endpoint.
• Full-duplex voice chat: a distinct Nemotron VoiceChat-style model if explicitly selected; do not assume ordinary ASR + TTS NIM provides duplex conversational behavior by itself.

If the user asks only for "speech NIM," keep ASR, TTS, and NMT together as a speech-and-voice provider decision. If the user asks for a narrow transcription or narration integration, scope to the needed container but keep the route boundaries visible.

Documented capability map

Use these as documented facts, not preferences.

• NVIDIA Speech NIM microservices deploy ASR, TTS, and NMT models as standalone containers. The application calls the NIM APIs; it does not call the raw model directly. Source: How NVIDIA Speech NIM Microservices Work.
• ASR NIM converts speech to text and supports streaming and offline modes. Streaming returns partial transcripts as audio arrives; offline processes a complete file and returns a complete transcript. Source: About NVIDIA ASR NIM.
• TTS NIM converts text to speech and supports offline synthesis and streaming synthesis. Offline returns a complete audio response and the gRPC path is subject to a 4 MB message-size limit; streaming returns audio chunks and is the route for lower time-to-first-audio or long text. Source: About NVIDIA TTS NIM.
• NMT NIM translates text between languages. NVIDIA documents real-time translation as a chain of ASR NIM, NMT NIM, and TTS NIM that the application orchestrates. Source: NVIDIA Speech NIM overview and About NVIDIA NMT NIM.
• ASR APIs: HTTP REST for simple offline transcription/translation, gRPC for batch and streaming recognition, and WebSocket for low-latency realtime sessions. Source: ASR API references.
• TTS APIs: HTTP REST for voice listing and offline/streaming synthesis, gRPC for batch and streaming synthesis, and WebSocket for realtime interactive synthesis. Source: TTS API references.
• ASR HTTP accepts WAV, OPUS, and FLAC for /v1/audio/transcriptions; either language or model is required. /v1/audio/translations is supported only by models with translation capability, such as Canary or Whisper in NVIDIA's REST documentation. Source: ASR HTTP REST API.
• TTS HTTP /v1/audio/synthesize returns a WAV file; /v1/audio/synthesize_online streams raw 16-bit signed LPCM chunks without a WAV header. Text is limited to 2,000 normalized characters per request on the documented HTTP endpoints. Source: TTS HTTP REST API.
• TTS voice names come from /v1/audio/list_voices and follow Model.LOCALE.Speaker or Model.LOCALE.Speaker.Emotion; emotion suffix availability varies by speaker. Source: TTS HTTP REST API.
• Zero-shot voice cloning is documented for Magpie TTS Zeroshot and Magpie TTS Flow. Both require access approval; the reference prompt should be a clear 16-bit mono WAV at 22.05 kHz or higher, about 3-10 seconds. Flow requires a transcript and is offline-only; Zeroshot supports streaming and offline. Source: Cloning a Voice with Zero-Shot TTS.
• NIM containers expose health, version, metadata, model, and Prometheus-style metrics endpoints. Use readiness checks before sending production audio. Sources: ASR HTTP REST API, TTS HTTP REST API, and Observability.