local-inference-tuning

Local Inference Tuning

Use this skill to design a local inference setup that fits the actual machine, model format, storage policy, and target workflow.

This skill may inspect hardware and local model inventory. Do not remove models, download large weights, change global package manager config, expose network services beyond loopback, or write to system-wide config unless the user explicitly approves.

Hardware Audit

Collect the minimum useful facts:

• OS and architecture;
• CPU / SoC / GPU;
• RAM or unified memory;
• available accelerators: Metal, CUDA, ROCm, Vulkan, CPU-only;
• free disk on the approved model/cache volume;
• thermal class if visible: desktop, laptop, fanless laptop;
• existing engines: Ollama, llama.cpp, MLX, vLLM, LM Studio, Docker;
• existing local models and formats: GGUF, MLX, safetensors, Ollama manifests.

Use compact commands. On macOS, prefer:

sysctl -n machdep.cpu.brand_string
sysctl -n hw.memsize
system_profiler SPHardwareDataType
system_profiler SPDisplaysDataType
df -h
ollama list

Engine Selection

Choose the engine by hardware and model format:

• MLX / Rapid-MLX: best default for Apple Silicon and MLX-format models. Prefer when the user wants an OpenAI-compatible local endpoint, batching, prompt/prefix cache, and Apple GPU kernels.
• llama.cpp / llama-cpp-python: best fallback for GGUF models, especially existing Ollama blobs or fully local files. Use Metal on Apple Silicon, CUDA on NVIDIA, CPU only as fallback.
• Ollama: best simple model manager and casual local endpoint. Keep when ease of use matters more than fine-grained batching/KV tuning.
• vLLM: best for NVIDIA CUDA servers with enough VRAM and concurrent serving workloads. Do not choose for Apple Silicon laptops.
• TensorRT-LLM: specialized NVIDIA deployment path; only use when the user has server NVIDIA hardware and deployment maturity.

If a lower-overhead engine already meets the goal, do not introduce a heavier stack.

Model Fit Estimate