speech-studio AGENTS.md

Agent Instructions

This file is for any AI coding agent working in this repo (Claude, Codex, Cursor, Aider, etc.).

Project

speech-studio — Speech Studio, a Soniqo project. Open-source desktop app for content creators.

MVP scope. Voice cloning + adjusting a cloned voice over a video timeline + emotional markers (style / prosody tags on the synthesized speech). The first end-to-end story:

1. Drop a short reference clip → clone the speaker.
2. Drop a video → extract / line up the existing dialogue.
3. Rewrite or re-record lines in the cloned voice, with inline emotion markers (e.g. <whisper>, <excited>, <sad>).
4. Preview against the video; export muxed output.

Status: the clone → script → synthesize → export pipeline works end to end. macOS runs the MLX engines — CosyVoice3 by default, with VoxCPM2, Qwen3-TTS, Chatterbox, OmniVoice, Indic-Mio, and Fish Audio S2 Pro selectable from the toolbar; Windows/Linux run VoxCPM2 and Indic-Mio through speech-core's LiteRT backend.

Stack

Tauri (Rust shell + web frontend) wrapping the Soniqo speech engines.

• Rust process — Tauri app, owns the window, menu, file pickers, IPC, model lifecycle, file I/O. Talks to a voice-cloning TTS backend through a sidecar chosen at compile time per OS:
  • macOS (Apple Silicon) — speech-swift (Swift / MLX) via the swift-sidecar/ binary. CosyVoice3 is the default cloning engine; VoxCPM2, Qwen3-TTS, Chatterbox, OmniVoice, Indic-Mio, and Fish Audio S2 Pro are also selectable (all MLX).
  • Windows / Linux (x86_64) — speech-core (C++) via the core-sidecar/ binary, cloning + cloned-voice TTS with the VoxCPM2 and Indic-Mio LiteRT models through the C ABIs in include/speech_core/voxcpm2_c.h and include/speech_core/indic_mio_c.h.
  • v1+ — speech-core's broader C ABI (speech_core_c.h) for STT (Parakeet), VAD (Silero), noise suppression (DeepFilterNet3), audio utilities.
• Web frontend — React + Vite, rendered into the OS WebView (WKWebView on macOS, WebView2 on Windows, WebKitGTK on Linux). Owns the video timeline, voice-clone manager, script editor with emotion markers, and waveform views. Talks to Rust via Tauri invoke() commands and events.
• Bridge mechanism — a stateful sidecar binary bundled with the app. Tauri spawns it; Rust talks to it over stdin/stdout using an NDJSON protocol (one JSON object per line each way). The sidecar loads the model once and keeps it resident across calls, so per-line synthesis after warmup is fast. The protocol (ping / init_model / per-engine synthesize_*) is implemented by both sidecars — the Swift sidecar handles every macOS engine (synthesize_voxcpm2 / _cosyvoice / _chatterbox / _icl / _indic_mio), and the C++ sidecar handles synthesize_voxcpm2 and synthesize_indic_mio — so SidecarManager (src-tauri/src/lib.rs) only varies the binary path per OS. init_model carries the selected engine. Code: swift-sidecar/ (Swift package, macOS) and core-sidecar/ (CMake C++, Windows/Linux).