llama-cpp

Kheish Compatibility

This skill is repo-local and stays inactive until explicitly activated.

When the original instructions refer to legacy tool names, use these Kheish mappings:

• terminal => bash
• web_extract => web_fetch, plus web_search when discovery is needed
• search_files => grep_search and glob_search
• browser_* tools require a browser-capable surfaced tool or MCP; if none is available, use the closest available surface and say so explicitly

When the instructions mention local helper files, resolve them from ${KHEISH_SKILL_DIR}.

llama.cpp

Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware.

When to use llama.cpp

Use llama.cpp when:

• Running on CPU-only machines
• Deploying on Apple Silicon (M1/M2/M3/M4)
• Using AMD or Intel GPUs (no CUDA)
• Edge deployment (Raspberry Pi, embedded systems)
• Need simple deployment without Docker/Python

Use TensorRT-LLM instead when:

• Have NVIDIA GPUs (A100/H100)
• Need maximum throughput (100K+ tok/s)
• Running in datacenter with CUDA

Use vLLM instead when:

• Have NVIDIA GPUs
• Need Python-first API
• Want PagedAttention

Quick start

Installation

# macOS/Linux
brew install llama.cpp

# Or build from source
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# With Metal (Apple Silicon)
make LLAMA_METAL=1

# With CUDA (NVIDIA)
make LLAMA_CUDA=1

# With ROCm (AMD)
make LLAMA_HIP=1

Download model

# Download from HuggingFace (GGUF format)
huggingface-cli download \
    TheBloke/Llama-2-7B-Chat-GGUF \
    llama-2-7b-chat.Q4_K_M.gguf \
    --local-dir models/

# Or convert from HuggingFace
python convert_hf_to_gguf.py models/llama-2-7b-chat/

Run inference

# Simple chat
./llama-cli \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    -p "Explain quantum computing" \
    -n 256  # Max tokens

# Interactive chat
./llama-cli \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --interactive