Multimodal-voice-assistant: Instructions file for Claude Code

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

A multi-modal AI voice assistant that supports DeepSeek (default), OpenAI, Anthropic Claude, and LM Studio (local). Uses Whisper for local speech-to-text and configurable TTS (OpenAI streaming or Kokoro). Listens for the wake word "nova", then processes voice commands with support for clipboard extraction, web search, and optional screenshot/webcam analysis via tool calling.

Configuration loads from a gitignored .env file via python-dotenv (see assistant/__init__.py — loaded before any submodule imports).

Running the Assistant

pip install -r requirements.txt
# or: pip install -e '.[dev]'

# Configure via .env (recommended)
cp .env.example .env  # then fill DEEPSEEK_API_KEY=...

# Run
python run.py            # entry-point script
python -m assistant      # equivalent

Override the provider via env vars or .env:

LLM_PROVIDER=openai       OPENAI_API_KEY=sk-...
LLM_PROVIDER=local        LOCAL_LLM_BASE_URL=http://localhost:1234/v1
LLM_PROVIDER=anthropic    ANTHROPIC_API_KEY=sk-ant-...
LLM_PROVIDER=deepseek     DEEPSEEK_API_KEY=sk-...   # default

Testing

pytest                                      # all suites
pytest tests/test_tool_loop.py -v           # tool-loop seam (fake provider)

Tests use a conftest.py that seeds safe env defaults so importing assistant does not require a real API key or LM Studio instance.

Architecture

Modular package architecture in the assistant/ directory.

Directory Structure

assistant/
├── __init__.py           # Package init, exports VoiceAssistant, main
├── __main__.py           # Entry point: python -m assistant
├── core.py               # Main VoiceAssistant class and orchestration
├── config/
│   └── settings.py       # All env vars, ModelCatalog, defaults
├── providers/
│   ├── llm/
│   │   ├── base.py           # Abstract LLMProvider class
│   │   ├── openai_provider.py
│   │   ├── local_provider.py
│   │   └── anthropic_provider.py
│   └── tts/
│       ├── base.py           # Abstract TTSProvider class
│       ├── openai_tts.py
│       └── kokoro_tts.py
├── tools/
│   ├── registry.py       # ToolRegistry class
│   ├── vision_tools.py   # Screenshot, webcam tools
│   ├── search_tools.py   # DuckDuckGo search
│   └── clipboard_tools.py
├── context/
│   ├── conversation.py   # EnhancedConversationContext
│   └── mcp.py            # ContextProvider, MCPContextProvider
├── speech/
│   ├── recognition.py    # Whisper transcription
│   └── audio.py          # Audio playback
├── media/
│   ├── screenshot.py     # PIL screenshot capture
│   └── webcam.py         # Pygame webcam capture
└── utils/
    ├── logging.py        # Rich logging
    └── messages.py       # Message normalization