caddis-plugin: Instructions file for GitHub Copilot

GPU Integration Guidelines

Overview

The project has on-premise GPU infrastructure for AI/ML workloads. This guide covers patterns for efficient GPU utilization.

Current GPU Workloads

Workload	Tool	GPU Usage
Speech-to-Text	Whisper	High
Text Embeddings	Sentence Transformers	Medium
LLM Inference	Ollama	High
Data Processing	RAPIDS (optional)	Medium

Environment Setup

Check GPU Availability

import torch

def check_gpu():
    """Check GPU availability and print info."""
    if torch.cuda.is_available():
        print(f"GPU: {torch.cuda.get_device_name(0)}")
        print(f"Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
        return True
    else:
        print("No GPU available, using CPU")
        return False

# Usage
HAS_GPU = check_gpu()
DEVICE = "cuda" if HAS_GPU else "cpu"

Environment Variables

# .env file
CUDA_VISIBLE_DEVICES=0          # Which GPU to use (0, 1, etc.)
PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512  # Memory optimization
OLLAMA_NUM_GPU=1                # For Ollama models

Whisper Transcription

Basic Usage

import whisper
from pathlib import Path
from loguru import logger

def transcribe_audio(audio_path: str, model_size: str = "base") -> dict:
    """
    Transcribe audio using Whisper.
    
    Args:
        audio_path: Path to audio file (mp3, wav, m4a)
        model_size: tiny, base, small, medium, large
    
    Returns:
        Dict with 'text' and 'segments'
    """
    logger.info(f"Loading Whisper model: {model_size}")
    model = whisper.load_model(model_size)
    
    logger.info(f"Transcribing: {audio_path}")
    result = model.transcribe(
        audio_path,
        language="en",  # Force English
        verbose=False
    )
    
    return {
        'text': result['text'],
        'segments': result['segments'],
        'language': result['language']
    }

# Model size guide:
# - tiny:   ~1GB VRAM, fastest, lowest accuracy
# - base:   ~1GB VRAM, good balance for short audio
# - small:  ~2GB VRAM, better accuracy
# - medium: ~5GB VRAM, good for complex audio
# - large:  ~10GB VRAM, best accuracy, slowest