modal

modal Best Practices

Modal is the definitive platform for deploying AI/ML workloads. To leverage its full potential – sub-second cold starts, instant autoscaling, and GPU acceleration – you must adhere to these best practices. This guide cuts through the noise, providing the exact patterns your team will use daily.

Code Organization and Structure

A well-structured Modal application is modular, explicit, and easy to debug.

1. Centralize Your modal.Stub

Always define a single, well-named modal.Stub at the top level of your main application file. This Stub is the entry point for all your Modal functions, images, and volumes.

❌ BAD: Multiple Stub definitions or generic names

# my_module_a.py
import modal
stub_a = modal.Stub("my-app-part-a") # Don't do this

# my_module_b.py
import modal
stub_b = modal.Stub("my-app-part-b") # Or this

✅ GOOD: Single, descriptive Stub

# src/my_ml_app/app.py
import modal

# Define the stub for your entire application
# Use a clear, unique name for your project/service
stub = modal.Stub("my-inference-service")

# All modal.Functions, Images, and Volumes will be attached to this stub

2. Modularize Your Application

For larger applications, separate your core logic (e.g., model loading, inference pipeline) into distinct Python modules. Import these modules into your main app.py where your modal.Functions are defined. This keeps your Modal definitions clean and your business logic testable.

# src/my_ml_app/model_loader.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

class ModelLoader:
    def __init__(self, model_id: str):
        self.model_id = model_id
        self.model = None
        self.tokenizer = None

    def load(self):
        if self.model is None:
            print(f"Loading model {self.model_id}...")
            self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)
            self.model = AutoModelForCausalLM.from_pretrained(self.model_id, torch_dtype=torch.bfloat16)
            print("Model loaded.")
        return self.model, self.tokenizer

# src/my_ml_app/app.py
import modal
from .model_loader import ModelLoader # Relative import for modularity

stub = modal.Stub("my-inference-service")

# Define your image and volumes here (see sections below)
inference_image = modal.Image.from_registry("nvcr.io/nvidia/pytorch:23.09-py3") \
    .pip_install("torch", "transformers")

model_volume = modal.Volume.from_name("my-llm-weights", create_if_missing=True)

@stub.function(image=inference_image, volumes={"/models": model_volume})
def generate_text(prompt: str):
    model_id = "mistralai/Mistral-7B-Instruct-v0.2"
    # Lazy load the model inside the function
    model_loader = ModelLoader(model_id)
    model, tokenizer = model_loader.load()
    
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=100)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

@stub.local_entrypoint()
def main():
    print(generate_text.remote("Hello, my name is"))