local-model-serving

Local Model Serving

Use this skill whenever a task needs local LLM inference on this machine. It is engine- and model-generic: the measured numbers come from a Qwen3.8-27B-class dense model (8-bit, ~28-30 GB resident) on a 128 GB Apple Silicon Mac, but the decision rules and settings transfer to any model served by the same two engine families.

The Two Engine Families (same model, two backends)

	MLX + speculative decoding (e.g. mlx-dspark)	llama.cpp (e.g. LM Studio)
Model file	MLX 8-bit (e.g. mlx-community/…-8bit) + drafter	GGUF Q8_0 (e.g. lmstudio-community/…-GGUF)
Port	e.g. http://127.0.0.1:8080/v1	e.g. http://127.0.0.1:1234/v1
Context	model native (e.g. 262,144)	model native
Thinking-off knob	"enable_thinking": false	"reasoning_effort": "none"
Short-context decode	~1.7-2.2× faster (measured)	baseline
Long-context (100k+)	slower (spec-decode verify rounds double the huge-KV cost)	~1.9× faster decode, ~linear prefill

Both are OpenAI-compatible (/v1/chat/completions), so the same client code works against either — only the base URL and the thinking-off parameter differ.

Drafters (DSpark, DFlash 2, and similar speculative-decoding heads) are modes of the MLX family, not a third engine. Do not install NVIDIA-only stacks (SGLang, vLLM) or a third MLX GUI to A/B a drafter. A better drafter can raise short-context decode; it does not remove the long-context crossover, because every verify round still walks the full KV. Use the drafter the local serving policy or helper names. llama.cpp may be LM Studio or llama-server; that is still the same family.

First: The Memory Rule (crash prevention)

Never run both engines with a model resident at the same time. On a 128 GB machine, two resident 28 GB models plus the OS drove the system into ~20-46 GB of swap and crushed both engines' throughput (a long run was aborted, another finished at half speed). The same exclusivity rule applies on a 64 GB machine that can fit only one ~28 GB model. Each engine fits alone; both do not.