minicpm5-deploy-arclight

Deploy ArcLight from Source (CPU)

ArcLight is a lightweight C/C++ LLM inference framework for unified-memory systems. The recommended path is to build from source, then run a GGUF model with al-gen, al-chat, or al-ppl.

Required input

Var	Example	Default
MODEL	/path/to/MiniCPM5-2B-Q4_0.gguf	required
PROMPT	"Hello!"	"Hello!"
THREADS	4	choose for the target CPU
NUMA_MODE	none or tp	none for first run
NODES	1, 2, 4	1 with NUMA_MODE=none
MAX_GEN	256	256

Steps

1. Build from source

git clone https://github.com/OpenBMB/ArcLight.git
cd ArcLight

cmake -B build -DARCLIGHT_BACKEND=AUTO
cmake --build build --config Release -j 32

Use ARCLIGHT_BACKEND=AUTO by default. Set it explicitly only when needed:

• X86: force the x86 backend
• NEON: force the ARM NEON backend
• NONE: build without architecture-specific backend code

2. Prepare a GGUF model

ArcLight uses GGUF checkpoints from llama.cpp. The nnml backend only loads f32 / f16 / q4_0 / q8_0 / q6_K / q8_K tensor types — Q4_K_M is not supported.

Supported model families: MiniCPM5-2B, Qwen3, Llama2.

For first validation use the released Q8_0 (openbmb/MiniCPM5-2B-GGUF), or quantize an unreleased Q4_0 yourself from the F16:

huggingface-cli download openbmb/MiniCPM5-2B-GGUF MiniCPM5-2B-F16.gguf --local-dir .
llama-quantize ./MiniCPM5-2B-F16.gguf ./MiniCPM5-2B-Q4_0.gguf Q4_0

3A. One-shot generation

./build/al-gen \
    --model "${MODEL}" \
    --prompt "${PROMPT}" \
    --numa none --nodes 1 \
    --threads ${THREADS} \
    --max_length 4096 \
    --max_gen ${MAX_GEN}

3B. Interactive chat

./build/al-chat \
    --model "${MODEL}" \
    --numa none --nodes 1 \
    --threads ${THREADS} \
    --max_length 4096 \
    --max_gen ${MAX_GEN}

To seed the first turn:

./build/al-chat \
    --model "${MODEL}" \
    --prompt "${PROMPT}" \
    --numa none --nodes 1 \
    --threads ${THREADS}