ollama-stack

Ollama Stack

Deploy a local LLM stack for offline and privacy-first workflows.

When to Use This Skill

Use this skill when:

• Setting up private/local LLM inference for development
• Building air-gapped AI environments
• Running models on personal hardware (Mac, Linux, Windows with GPU)
• Creating team-shared inference endpoints
• Prototyping before committing to cloud LLM APIs

Prerequisites

• 8 GB+ RAM (16 GB+ recommended for 7B+ models)
• For GPU acceleration: NVIDIA GPU with 6 GB+ VRAM, or Apple Silicon Mac
• Docker (for containerized deployment)
• 20 GB+ disk for model storage

Quick Start

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Start the server
ollama serve

# Pull and run a model
ollama pull llama3.1:8b
ollama run llama3.1:8b "Explain Kubernetes pods in one paragraph"

# List available models
ollama list

# Pull specific quantization
ollama pull llama3.1:8b-instruct-q4_K_M

Model Selection Guide

Model	Size	VRAM	Best For
llama3.1:8b	4.7 GB	6 GB	General chat, coding
llama3.1:70b	40 GB	48 GB	Complex reasoning
codellama:13b	7.4 GB	10 GB	Code generation
mistral:7b	4.1 GB	6 GB	Fast general tasks
mixtral:8x7b	26 GB	32 GB	High-quality MoE
nomic-embed-text	274 MB	1 GB	Embeddings for RAG
llava:13b	8 GB	10 GB	Vision + text
deepseek-coder-v2:16b	9 GB	12 GB	Code generation
qwen2.5:14b	9 GB	12 GB	Multilingual, reasoning

Docker Compose — Full Stack

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_MAX_LOADED_MODELS=2
      - OLLAMA_FLASH_ATTENTION=1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    volumes:
      - webui_data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=true
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY:-change-me-in-production}
      - DEFAULT_MODELS=llama3.1:8b
    depends_on:
      ollama:
        condition: service_healthy

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    restart: unless-stopped
    ports:
      - "4000:4000"
    volumes:
      - ./litellm-config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml"]
    depends_on:
      ollama:
        condition: service_healthy

volumes:
  ollama_data:
  webui_data: