minicpm5-deploy-llama-cpp

Deploy MiniCPM5-1B with llama.cpp

CPU / edge / consumer-GPU deployment via the released GGUF artifacts. The artifacts work directly with vanilla llama.cpp and every downstream runtime (Ollama / LM Studio / llama-cpp-python).

Required input

Var	Example	Default
GGUF_REPO	openbmb/MiniCPM5-1B-GGUF	required
QUANT	Q4_K_M (657 MB, recommended) / Q8_0 (1.1 GB) / F16 (2.1 GB)	Q4_K_M
NGL	99 (all layers on GPU) / 0 (CPU only)	99 if NVIDIA GPU, else 0
CTX	8192 (default) up to 131072 (128 K)	8192

Steps

1. Install llama.cpp

# macOS
brew install llama.cpp

# Linux / cross-platform: pre-built binary
curl -fsSL https://github.com/ggerganov/llama.cpp/releases/latest/download/llama-cli-linux.tar.gz | tar -xz
# OR build from source:
git clone --depth=1 https://github.com/ggerganov/llama.cpp.git && cd llama.cpp
mkdir build && cd build
cmake .. -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release    # CPU-only: omit GGML_CUDA=ON
cmake --build . --config Release -j $(nproc) --target llama-cli llama-server

2. Download the GGUF

mkdir -p ~/minicpm5 && cd ~/minicpm5
huggingface-cli download ${GGUF_REPO} MiniCPM5-1B-${QUANT}.gguf --local-dir .

3a. Interactive chat (CLI)

llama-cli -m MiniCPM5-1B-${QUANT}.gguf \
    -n 2048 --temp 0.7 --top-p 0.95 -ngl ${NGL} -c ${CTX}

3b. OpenAI-compatible HTTP server

llama-server -m MiniCPM5-1B-${QUANT}.gguf \
    --port 8080 -ngl ${NGL} -c ${CTX} --jinja

4. Validate

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniCPM5-1B",
        "messages": [{"role":"user","content":"1+1=?"}],
        "temperature": 0.7, "top_p": 0.95, "max_tokens": 64
    }'

Expected: "2" in the reply.

Sampling defaults

Mode	--temp	--top-p
Think	0.9	0.95
No-think	0.7	0.95

Choosing a quant