deploy-edge-ai-model

Deploy Edge AI Model

See Extended Examples for complete configuration files, quantization scripts, and benchmark templates.

Deploy ML models to edge devices with optimized inference, hardware acceleration, and on-device model management.

When to Use

• Deploying LLMs (Gemma 4, Phi, Llama) to mobile devices via Google AI Edge Gallery
• Converting models to TensorFlow Lite or ONNX for on-device inference
• Quantizing models to INT8/INT4 for reduced memory and faster inference
• Building Android/iOS apps with local AI capabilities
• Selecting hardware delegates (GPU, NPU, DSP, Hexagon, CoreML)
• Benchmarking inference latency and memory on target devices
• Deploying MediaPipe tasks (vision, text, audio) to mobile or embedded platforms

Inputs

• Required: Trained model (SavedModel, PyTorch, ONNX, or Hugging Face checkpoint)
• Required: Target platform (Android, iOS, Linux embedded, browser)
• Required: Target device constraints (RAM, storage, compute capability)
• Optional: Calibration dataset for post-training quantization
• Optional: Google AI Edge Gallery configuration for LLM deployment
• Optional: Hardware delegate preferences (GPU, NPU, CPU-only)

Procedure

Step 1: Evaluate Model for Edge Deployment

Assess model size, latency requirements, and target device capabilities.

# assess_model.py
import os
import tensorflow as tf

def assess_model_for_edge(saved_model_path, target_ram_mb=4096):
    """Evaluate whether a model is suitable for edge deployment."""
    model = tf.saved_model.load(saved_model_path)

    # Check model size on disk
    model_size_mb = sum(
        os.path.getsize(os.path.join(dp, f))
        for dp, _, filenames in os.walk(saved_model_path)
        for f in filenames
    ) / (1024 * 1024)

    print(f"Model size: {model_size_mb:.1f} MB")
    print(f"Target RAM: {target_ram_mb} MB")
    print(f"Size/RAM ratio: {model_size_mb / target_ram_mb:.2%}")

    if model_size_mb > target_ram_mb * 0.25:
        print("WARNING: Model exceeds 25% of device RAM - quantization recommended")
        return False
    return True