modal-serverless-gpu

Modal Serverless GPU

Comprehensive guide to running ML workloads on Modal's serverless GPU cloud platform.

When to use Modal

Use Modal when:

• Running GPU-intensive ML workloads without managing infrastructure
• Deploying ML models as auto-scaling APIs
• Running batch processing jobs (training, inference, data processing)
• Need pay-per-second GPU pricing without idle costs
• Prototyping ML applications quickly
• Running scheduled jobs (cron-like workloads)

Key features:

• Serverless GPUs: T4, L4, A10G, L40S, A100, H100, H200, B200 on-demand
• Python-native: Define infrastructure in Python code, no YAML
• Auto-scaling: Scale to zero, scale to 100+ GPUs instantly
• Sub-second cold starts: Rust-based infrastructure for fast container launches
• Container caching: Image layers cached for rapid iteration
• Web endpoints: Deploy functions as REST APIs with zero-downtime updates

Use alternatives instead:

• RunPod: For longer-running pods with persistent state
• Lambda Labs: For reserved GPU instances
• SkyPilot: For multi-cloud orchestration and cost optimization
• Kubernetes: For complex multi-service architectures

Quick start

Installation

pip install modal
modal setup  # Opens browser for authentication

Hello World with GPU

import modal

app = modal.App("hello-gpu")

@app.function(gpu="T4")
def gpu_info():
    import subprocess
    return subprocess.run(["nvidia-smi"], capture_output=True, text=True).stdout

@app.local_entrypoint()
def main():
    print(gpu_info.remote())

Run: modal run hello_gpu.py

Basic inference endpoint

import modal

app = modal.App("text-generation")
image = modal.Image.debian_slim().pip_install("transformers", "torch", "accelerate")

@app.cls(gpu="A10G", image=image)
class TextGenerator:
    @modal.enter()
    def load_model(self):
        from transformers import pipeline
        self.pipe = pipeline("text-generation", model="gpt2", device=0)

    @modal.method()
    def generate(self, prompt: str) -> str:
        return self.pipe(prompt, max_length=100)[0]["generated_text"]

@app.local_entrypoint()
def main():
    print(TextGenerator().generate.remote("Hello, world"))