on-device-verification

On-device verification

A device result is done when three things hold:

1. the model compiles and runs on the accelerator you claim it runs on,
2. the output matches the source model numerically and on a task-level gate,
3. the record names the device, the runtime version, and the residency line. A number without those is not reproducible and not a result.

Host-side checks (the CompiledModel checker used in gpu-clean-conversion) exercise the host GPU. The device has its own shader compiler, its own precision behavior, and its own memory ceiling — every failure mode in the table below was hit by a model that had already passed on the host. The Python API below runs on the host only (ai-edge-litert ships wheels for Linux, macOS and Windows); on a phone the same inputs go through the app's inference class or a push-run-pull binary (last bullet of "Watch for").

Loop

1. Dump references from the source model, once. Fixed inputs — one real sample plus fixed-seed random — saved as .npy next to the recipe (dump_*_ref.py). These are ground truth for every later step; regenerate them only when the source model changes.

2. Run the same inputs on the host runtime, CPU then GPU, then on the device. One argument switches the accelerator (ai-edge-litert 2.2.0, 2.3.0):

from ai_edge_litert.compiled_model import CompiledModel
from ai_edge_litert.hardware_accelerator import HardwareAccelerator

model = CompiledModel.from_file(
    "model.tflite", hardware_accel=HardwareAccelerator.GPU)  # or .CPU

The device-CPU run is the control. If it already diverges from the source dump, the problem is the conversion, not the GPU — go back to gpu-clean-conversion. A full worked example of the A/B lives in this repo at samples/litert/speech_recognition/convert/verify_tflite.py (its ai_edge_torch.compiled_model import no longer resolves with litert-torch 0.9.4; read it as ai_edge_litert.compiled_model).

Ask for the strict accelerator. Compiling with HardwareAccelerator.CPU | HardwareAccelerator.GPU permits partial delegation and hides fallback; use the combined mode only to discover which ops fell back after a strict GPU compile fails. Strict is per graph, not per op: when the GPU cannot compile the graph, from_file raises (Some ops are not accelerated. Add kLiteRtHwAcceleratorCpu … in 2.3.0); when the delegate declines one node while building (TopK requires src tensor C dimension to be divisible by 4), that node runs on the CPU with only the log line, the call returns, and is_fully_accelerated() still reads true (measured with the 2.2.0 and 2.3.0 wheels on a Mac). The log is the residency signal.