jarvis CLAUDE.md

Project: jarvis

Agent-facing project instructions. Read this before touching the codebase. Full historical detail — every milestone, decision, and post-mortem — lives in docs/MILESTONES.md.

Goal

A Windows desktop voice assistant. An always-on microphone listens for the wake word "Jarvis" / "Hey Jarvis", transcribes the spoken question locally, sends the text to the Claude API with a Jarvis-personality system prompt, and reads the response back through the speakers via TTS. Inspired by Tony Stark's J.A.R.V.I.S.: courteous, dryly witty, concise.

The interesting part is not the voice loop — it is everything hung off it: an agentic tool layer (36 tools), proactive background monitors, a vision/security subsystem, a phone client, and a set of engineering conventions strict enough to keep an always-on process honest.

Stack / Tools

• Language: Python 3.12 on Windows.
  • Windows-native, not WSL: audio device access. WSL2 audio bridging is unreliable for always-on real-time listening.
• Wake word: openWakeWord
  • MIT, fully local, no API key or account. Ships a pre-trained hey_jarvis model. CPU-only, ~3-5% continuous utilization.
  • Slightly higher false-positive rate than commercial alternatives (Porcupine); tunable via a confidence threshold. Porcupine was the original pick but now requires a company email, which rules it out for an open personal project.
• Speech-to-text: faster-whisper
  • Local Whisper on CPU; the small/base model is plenty for short commands. No internet required — question audio never leaves the machine.
  • Optionally offloaded to a CUDA box on the LAN over a small HTTP server (cuts transcription 5-10 s → 1.5-2.5 s); falls back to local on any failure.
• LLM: Anthropic Python SDK (anthropic)
  • Default claude-sonnet-5 (CLAUDE_MODEL overrides). Haiku for the cheap background jobs (session summarizer, prediction miner). Opus only if a request genuinely needs more reasoning.
  • Thinking is explicit. Sonnet 5 runs adaptive thinking when the thinking param is omitted. Voice and background paths pass thinking={"type": "disabled"} (latency, plus small max_tokens budgets would be truncated by an unplanned thinking block); engineer mode passes {"type": "adaptive"}.
  • Prompt caching on the system prompt — it is reused every turn. Per-turn volatile context (clock, speaker identity) rides a second, uncached system block so it never invalidates the cache.
  • Streaming so TTS can start before the reply is complete.
• Text-to-speech: edge-tts primary, pyttsx3 fallback.
  • Edge is a free Microsoft online voice, surprisingly good. pyttsx3 is offline (Windows SAPI) — the graceful degradation path when Edge is unreachable.
• Audio I/O: sounddevice — cleaner than PyAudio, handles streaming well.
• UI: pystray tray icon (four states) + a customtkinter console window.
• Vision: opencv-python + ultralytics (YOLOv8n person detection) + face_recognition/dlib for the enrolled-face auth path.
• Acoustic classification: PANNs Cnn14 (panns_inference) — 527 AudioSet classes, ~0.2 s CPU inference.
• Speaker ID: Resemblyzer d-vectors (256-d), cosine similarity.
• Env: python-dotenv. Async: asyncio around the listen → process → respond cycle; background subsystems are daemon threads.