TalkWithMe: Instructions file for Codex

TalkWithMe — Agent Instructions

Run the app

source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
deactivate

Requires a locally running llama.cpp server with an OpenAI-compatible API (or any OpenAI-compatible LLM, local or remote). TTS and STT servers are optional. Open http://localhost:8000 in a browser.

A remote LLM that needs an API key is supported: set the TALKWITHME_LLM_API_KEY env var or create an llm_api_key file in the project root (copy llm_api_key.example) — see the README "LLM API key (remote LLMs)" section. The key is never in settings.yaml, never in the UI, and never logged.

Testing — pytest, fully offline

source .venv/bin/activate
pip install -r requirements-dev.txt
python -m pytest # config in pytest.ini
deactivate

• No servers needed. The suite is hermetic: every external HTTP endpoint (LLM, TTS, STT, MCP) is faked via tests/factories.py (fake httpx.AsyncClients, config factories, SSE helpers). It must run — and pass — with nothing but Python installed.
• Isolation: tests/conftest.py has an autouse fixture that points every module-level global at per-test tmp_path state: the config caches in app/config.py, _PERSISTENCE_ROOT (in both app/persistence.py and app/routers/persistence.py — the router imported it by value, so it needs its own patch), the session singleton, the MCP tool registry, the TTS capabilities cache in app/services/tts_client.py, the LLM API key cache in app/services/llm_auth.py (its _PROJECT_ROOT is re-pointed to tmp_path and TALKWITHME_LLM_API_KEY is deleted from the environment, so a test's key — or the developer's real one — never leaks into another test), and the once-per-URL cleartext-warning dedupe set in app/services/llm.py. Real settings.yaml / Personas/ / chatrooms.yaml / chatrooms/ / llm_api_key data is never read or written. If you add a new module-level global to the app, add it to that fixture.
• Lifespan: the client fixture uses TestClient without the startup lifespan, because the lifespan re-reads the real YAML files (clobbering test caches) and attempts MCP discovery. Tests that exercise the lifespan do so explicitly with a local TestClient in a with block and monkeypatched load_*/load_tools (see tests/test_main.py).
• Coverage map: test_config.py (config models + YAML load/save), test_persona_store.py (persona directory discovery: frontmatter, file ops, language/avatar helpers, memory-file append/purge/dedupe, YAML→dir migration), test_builtin.py (the built-in add_memory tool: registry, availability gating, save/error paths), test_models.py (API request/response models), test_persistence.py (disk persistence + audio staging), test_mime.py (the shared deterministic MIME→extension helper in app/services/mime.py: the rule table plus the never-consult-the-OS-mime-database invariant — the macOS .weba bug class), test_session_manager.py, test_llm.py (SSE parsing + agentic tool loop incl. the mid-stream abort retry/LLMStreamAborted + the LLM API key header + the http:// cleartext warning), test_llm_auth.py (LLM API key resolution: env-var/file priority, the llm_api_key = <value> file format, lazy load, the never-logged invariant), test_mcp_client.py, test_tool_registry.py, test_tts_stt_clients.py, test_tts_fixtures.py (the two copies of the four capabilities snapshots — pytest's tests/factories.py vs the plain-Node tests/fixtures/*.json — must not drift; re-copy both from tts-serve in the same commit), test_chat_sse.py (the /api/chat SSE endpoint — LLM stubbed, selection/persistence/echo/tool events for real), test_main.py, test_docs.py (the AGENTS.md API endpoints table must match the routes actually registered on the app — run it after adding/removing endpoints and update the table), one test_routers_*.py per API router, test_persona_form.js (plain Node, not part of pytest — see below: the persona editor form logic in static/persona.js), test_tts_settings.js (plain Node, not part of pytest: the dynamic TTS parameter section in static/tts-params.js + the TTS wiring in static/settings.js), test_speaking_highlight.js (plain Node, not part of pytest: the speaking-bubble highlight — the playback queues in static/tts.js + setSpeakingHighlight/playPersistedAudio in static/chat.js), and test_stop_button.js (plain Node, not part of pytest: the top-bar stop button — a temporary "mute" over the playback queues: the button tracks the activeAudioSources set and is enabled exactly while some source plays, stop halts every active source and disables the button immediately (before the async onended fires), muted items are still fetched and persisted with their replay buttons but never play (both the non-streaming queue and the streaming sentence pipeline; the TTS fetch pipeline itself is untouched), and the mute is cleared by the next user prompt (exercised through the real sendMessage() with a stubbed SSE stream) or a manual replay click — an item whose playback start falls after the lift plays normally).
• Frontend tests (Node): tests/test_persona_form.js runs with plain Node 20+ — no npm packages, no network: node tests/test_persona_form.js. It evaluates the real static/utils.js + state.js + persona.js in a fresh vm.Context per test against a minimal DOM stub, stubs fetch to capture the multipart FormData, and drives the real openPersonaForm(), Remove-button listeners, file-input change handlers, and submitPersonaForm(). It exists to lock in the invariants that remove_avatar_image / remove_reference_audio are sent only after an explicit "Remove" click — never derived from server-side file presence (that bug silently deleted a persona's avatar and reference audio on every plain text save) — and, by the same rule, that clear_memories is sent only after an explicit "Clear saved memories" click while memory_size is always sent (the update endpoint requires it; an omitted value must not silently reset the persona's memory budget). tests/test_tts_settings.js uses the same harness (fresh vm.Context per test, minimal DOM stub, stubbed fetch) but loads static/utils.js + state.js + tts-params.js + settings.js and drives the real openSettings(), the TTS enabled/Base-URL change listeners, and form submission against the real tts-serve capabilities snapshots in tests/fixtures/. It locks in the T9 widget table (incl. "null default ⇒ not a slider" and the integer-span rule), the app-managed-fields exclusion (text/audio_base64/reference_text/language are never rendered), the "blank = let the engine decide" collection semantics (blank optionals omitted; checkboxes/sliders/selects-with-default always sent), the array parameter widget (flat scalar item lists, tts-serve schema v2: a fixed item count renders one labeled row per item — the engine's item_labels[i] when the spec advertises a well-formed list (exactly one non-empty string per item), else the name[i] fallback, which mirrors the validators' item naming; malformed labels degrade to the fallback and never to the JSON hatch — all-blank omits the parameter, a partially filled one is a validation error and blank slots are never filled with invented values; variable-size or malformed array specs fall to the raw-JSON escape hatch — tested with synthetic docs since no shipped snapshot has an array param yet), the T10 version gate, and the pass-through rule that a save with no capabilities document sends the saved tts.parameters unchanged rather than {} (never silently wipe). It also locks in the reconnect feature (Refresh button + Base-URL change listener probe the field's url via ?base_url=; a scheme-less url gets the specific 422 line, not "not reachable"; a same-URL refetch keeps in-flight slider edits instead of re-filling from saved values, only a real engine switch shows the "edits discarded" note, and opening the dialog is a fresh connect — the in-memory doc is cleared on open so unsaved edits from a cancelled session cannot leak across opens), the "Reset to defaults" button above the parameter list (re-renders the section from an empty saved-values map — the first-connect state: no network traffic, a no-op when no document is renderable so the no-doc save pass-through still protects the saved parameters, local until Save — an unsaved reset is discarded on reopen while a same-URL refresh keeps the reset state on screen — and a save after a reset sends the always-sent widgets at the engine's declared defaults while every blankable field is omitted, dropping the saved customization), and — critically — the DOM stub's children are array-like non-Array lists (HTMLCollection mimic), so the suite catches the regression class where code gates on Array.isArray(element.children): in real browsers that silently collects zero parameter values on every save, wiping tts.parameters (the save→reopen round-trip test reproduces the user-visible symptom: values reverting to the engine's doc defaults). Both files are kept out of the pytest suite on purpose: pytest must stay runnable with nothing but Python installed.
• Two expected deprecation warnings: a clean run is "all tests pass", even when the warnings summary contains exactly two third-party deprecation warnings. Both come from Starlette 1.x's own testclient (pulled in via the fastapi.testclient import), not from app or test code: a StarletteDeprecationWarning that the httpx-based testclient will one day be replaced by a new httpx2 package, and an anyio.abc.BlockingPortal alias deprecation raised from inside starlette/testclient.py. Neither is harmful today — the testclient works fine on httpx as of Starlette 1.6 — so don't chase them and don't "fix" app/test code in response. The eventual resolution, if it ever comes, is a future Starlette/httpx2 bump, not a code change here.
• Rules:
  • Every code change must be followed by a clean run: .venv/bin/python -m pytest, all green. No exceptions, no skipped tests. Changes to static/persona.js (or anything else covered by the Node tests) additionally require node tests/test_persona_form.js all green; changes touching the dynamic TTS section (static/tts-params.js, the TTS wiring in static/settings.js, or the capabilities fixtures) additionally require node tests/test_tts_settings.js all green; changes to the speaking-highlight paths (the playback queues in static/tts.js, or setSpeakingHighlight/beginSpeaking/endSpeaking/playPersistedAudio in static/chat.js) additionally require node tests/test_speaking_highlight.js all green; changes to the stop-button paths (the stop button in static/tts.js — stopAudioPlayback/stopAllAudioSources/updateStopButtonUI/playAudioSource — or the mute clearing in sendMessage/playPersistedAudio in static/chat.js) additionally require node tests/test_stop_button.js all green.
  • New functionality or API endpoints require new tests in the matching test_*.py file before the change is complete.
  • API tests use the client fixture + config caches (re-point app.config._settings_cache / _personas_cache / _chatrooms_cache via monkeypatch); router-level stubs are applied at the router's import site (e.g. app.routers.chat.stream_chat), since routers import service functions by name.
  • All fake responses in tests/factories.py attach an httpx.Request before returning: the httpx versions this project supports (0.24–0.28) raise RuntimeError from response.url / raise_for_status() if no request is attached to a Response, and the public .request getter itself raises when unset (check the private response._request attribute instead).