lemonade AGENTS.md

AGENTS.md

This file provides guidance to agent driven code reviews when working with this repository.

Project Overview

Lemonade is a local LLM server providing GPU and NPU acceleration for running large language models on consumer hardware. It exposes OpenAI-compatible, Ollama-compatible, and Anthropic-compatible REST APIs, plus a WebSocket Realtime API. It supports multiple backends: llama.cpp, FastFlowLM, RyzenAI, whisper.cpp, stable-diffusion.cpp, Kokoro TTS, and Moonshine.

Architecture

Executables

• lemond — Pure HTTP server. Handles REST API, routes requests to backends, manages model loading/unloading. Configured via config.json in the lemonade cache directory. CLI args: [cache_dir] [--port PORT] [--host HOST].
• lemonade — CLI client (src/cpp/cli/). Commands: list, pull, delete, run, status, logs, launch, backends, scan, etc. Communicates with router via HTTP. Discovers running server via UDP beacon.
• LemonadeServer.exe (Windows) — SUBSYSTEM:WINDOWS GUI app that embeds lemond and shows a system tray icon. Auto-starts via Windows startup folder.
• lemonade-tray (macOS/Linux) — Lightweight tray client that connects to a running lemond. Platform code in src/cpp/tray/platform/.

Backend Abstraction

WrappedServer (src/cpp/include/lemon/wrapped_server.h) is the abstract base class. Each backend inherits it and implements load(), unload(), chat_completion(), completion(), responses(), and optionally install() / download_model(). Backends run as subprocesses — Lemonade forwards HTTP requests to them.

Backend	Class	Capabilities	Device	Purpose
llama.cpp	LlamaCppServer	Completion, Embeddings, Reranking	GPU	LLM inference — CPU/GPU (Vulkan, ROCm, Metal)
FastFlowLM	FastFlowLMServer	Completion, Embeddings, Audio	NPU	NPU inference (multi-modal: LLM, ASR, embeddings)
RyzenAI	RyzenAIServer	Completion	NPU	Hybrid NPU inference
vLLM	VLLMServer	Completion	GPU	LLM inference — ROCm on AMD iGPU/dGPU (Linux). Experimental, validated only on gfx1151 (Strix Halo).
whisper.cpp	WhisperServer	Audio	CPU	Audio transcription
stable-diffusion.cpp	SdServer	Image	CPU	Image generation, editing, variations
Kokoro	KokoroServer	TTS	CPU	Text-to-speech
Moonshine	MoonshineServer	Audio	CPU	Streaming speech-to-text (ONNX-based)