local-vram-orchestrator

local-vram-orchestrator

Operating manual for running heavy AI models on a single consumer GPU without hitting CUDA out of memory. Built for the constraint of ~6GB VRAM (RTX 4050 class) where two large models cannot coexist in memory.

Core principle: one heavy model on the GPU at a time

On 6GB you cannot hold an LLM + SD + Whisper resident simultaneously. The architecture must serialize GPU-intensive work: load → run → fully unload → load next. Concurrency is for I/O and CPU work, never for two resident models.

1. Aggressive VRAM release

torch.cuda.empty_cache() alone does not free memory still referenced by a live Python object. You must delete the references first, then collect, then empty the cache — in that order.

import gc
import torch

def release_model(model):
    """Fully evict a model from VRAM. Call before loading the next heavy model."""
    try:
        model.to("cpu")          # move weights off-GPU first (helps fragmentation)
    except Exception:
        pass
    del model                    # drop the Python reference
    gc.collect()                 # collect any cyclic refs holding tensors
    torch.cuda.empty_cache()     # return freed blocks to the driver
    torch.cuda.ipc_collect()     # release cross-process cached allocations

Common leak sources to delete explicitly: the model, the pipeline wrapper, optimizer/scheduler objects, any output tensors still in local scope, and **inputs dicts moved to CUDA. A single tensor retained in a closure or a logging list pins the whole allocation block.

# After inference, before unloading:
del outputs, inputs
gc.collect(); torch.cuda.empty_cache()

Verify, don't trust

Always confirm memory actually dropped — agent-reported "freed" is not evidence.

def vram_report(tag=""):
    a = torch.cuda.memory_allocated() / 1024**2
    r = torch.cuda.memory_reserved() / 1024**2
    print(f"[VRAM {tag}] allocated={a:.0f}MB reserved={r:.0f}MB")