ggml

ggml

Overview

ggml is a minimalistic C tensor computation library powering llama.cpp and many other ML inference engines. It provides:

• A define-and-run computation graph model (similar to TensorFlow 1.x)
• CPU, CUDA, Metal, Vulkan, WebGPU, and other hardware backends
• 35+ quantization formats (Q4_0, Q8_0, Q5_K, MXFP4, NVFP4, TQ1_0, Q1_0, etc.)
• GGUF binary file format for model weights and metadata
• Automatic differentiation and AdamW/SGD optimizers
• Zero runtime allocations — all memory is pre-reserved

Version: v0.15.3 Language: C (C++ optional) License: MIT Repo: https://github.com/ggml-org/ggml

Quick Start

#include "ggml.h"
#include "ggml-cpu.h"
#include "ggml-backend.h"

int main(void) {
    struct ggml_init_params params = {
        .mem_size   = 64 * 1024 * 1024,  // 64 MB scratch buffer
        .mem_buffer = NULL,
        .no_alloc   = false,
    };
    struct ggml_context * ctx = ggml_init(params);

    struct ggml_tensor * a = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4);
    struct ggml_tensor * b = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 4);
    struct ggml_tensor * c = ggml_add(ctx, a, b);

    struct ggml_cgraph * gf = ggml_new_graph(ctx);
    ggml_build_forward_expand(gf, c);

    ggml_backend_t backend = ggml_backend_cpu_init();
    ggml_backend_graph_compute(backend, gf);

    ggml_backend_free(backend);
    ggml_free(ctx);
    return 0;
}

Core Concepts

• ggml_context — memory pool that owns all tensors; freed all at once
• ggml_tensor — N-D array (max 4 dims); stores type, shape, strides, and a data pointer
• ggml_cgraph — lazy computation graph; ops are recorded then executed via a backend
• ggml_backend_t — execution engine (CPU, CUDA, Metal, …); use ggml_backend_load_all() to discover available hardware
• ggml_backend_sched_t — multi-device scheduler that splits a graph across backends automatically
• GGUF — binary model format: metadata key-value store + packed tensor data