ort

ONNX Runtime in Rust (ort 2.x)

Overview

ort is the Rust binding to ONNX Runtime. Use it when you need to run ONNX-format models locally — text embedders, speech models, classifiers — with hardware-accelerated execution providers (CoreML on Apple Silicon, CUDA on Linux/Windows). The ort 2.x API changed significantly from 1.x; all patterns here target ort = "=2.0.0-rc.12" (the version pinned by fastembed-rs).

Reference implementation: crates/trusty-common/src/embedder/fast_embedder.rs

Quick Start

1. Cargo.toml setup

Library crate (default-features = false avoids linking the bundled ORT runtime unnecessarily; consumers opt into ONNX via a feature):

# Why: pin to the exact rc version fastembed-rs requires to avoid ABI mismatch.
# What: base features give std + the 24-API-level surface; coreml is macOS-only.
[dependencies.ort]
version = "=2.0.0-rc.12"
default-features = false
features = ["std", "api-24"]
optional = true

# macOS: add CoreML EP (builds to .mlmodelc, runs on ANE/GPU).
[target.'cfg(target_os = "macos")'.dependencies.ort]
version = "=2.0.0-rc.12"
default-features = false
features = ["coreml", "std", "api-24"]
optional = true

# ndarray interop (zero-copy tensor views).
ndarray = { version = "0.16", optional = true }

Binary / daemon crate (load-dynamic avoids statically linking libonnxruntime into the binary, reducing binary size when the shared lib is already present):

[dependencies.ort]
version = "=2.0.0-rc.12"
default-features = false
features = ["load-dynamic", "std", "api-24"]

# For CUDA GPU inference on Linux:
# features = ["load-dynamic", "cuda", "std", "api-24"]

ndarray tensor interop (add alongside ort):

# Why: ort's TensorRef::from_array_view needs ndarray's ArrayView.
# What: zero-copy conversion between ndarray arrays and ort Tensors.
ndarray = { version = "0.16", features = ["std"] }
# or in Cargo.toml features = ["std", "ndarray"] on the ort dep directly

Core Concepts