unsloth-mcp-server: Skill for Claude Code

Model Deployment

Complete guide for exporting, optimizing, and deploying fine-tuned LLMs to production environments.

Overview

After fine-tuning your model with Unsloth, deploy it efficiently:

• GGUF export - For llama.cpp, Ollama, local inference
• vLLM deployment - For high-throughput production serving
• HuggingFace Hub - For sharing and version control
• Quantization - Reduce size while maintaining quality
• Platform selection - Choose the right infrastructure
• Monitoring - Track performance and costs

Quick Start

Export to GGUF (Ollama/llama.cpp)

from unsloth import FastLanguageModel

# Load your fine-tuned model
model, tokenizer = FastLanguageModel.from_pretrained(
    "./fine_tuned_model",
    max_seq_length=2048
)

# Export to GGUF format
model.save_pretrained_gguf(
    "./gguf_output",
    tokenizer,
    quantization_method="q4_k_m"  # 4-bit quantization
)

# Use with Ollama
# ollama create my-model -f ./gguf_output/Modelfile
# ollama run my-model

Deploy with vLLM

from unsloth import FastLanguageModel

# Save for vLLM
model.save_pretrained("./vllm_model")
tokenizer.save_pretrained("./vllm_model")

# Start vLLM server
# python -m vllm.entrypoints.openai.api_server \
#   --model ./vllm_model \
#   --tensor-parallel-size 1 \
#   --dtype bfloat16

Push to HuggingFace Hub

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "./fine_tuned_model",
    max_seq_length=2048
)

# Push to Hub
model.push_to_hub(
    "your-username/model-name",
    token="hf_...",
    private=False
)
tokenizer.push_to_hub(
    "your-username/model-name",
    token="hf_..."
)

Export Formats

1. GGUF (llama.cpp / Ollama)

Best for: Local deployment, edge devices, CPU inference

# Export with different quantization levels
quantization_methods = {
    "q4_k_m": "4-bit, medium quality (recommended)",
    "q5_k_m": "5-bit, higher quality",
    "q8_0": "8-bit, near-original quality",
    "f16": "16-bit float, full quality",
    "f32": "32-bit float, highest quality"
}

# Export
model.save_pretrained_gguf(
    "./gguf_output",
    tokenizer,
    quantization_method="q4_k_m"
)

# Creates:
# - model-q4_k_m.gguf (quantized model)
# - Modelfile (for Ollama)