unsloth-mcp-server: Skill for Claude Code

Unsloth Tokenizer Tools

Expert guidance for tokenizer comparison, analysis, and integration using Unsloth's tokenizer utilities.

Overview

Unsloth provides powerful tools for:

• Comparing tokenizers - Measure token efficiency across different tokenizers
• Analyzing tokenization - Understand how text is tokenized
• Integration - Use tokenizers with Unsloth models
• Optimization - Find the most efficient tokenizer for your use case

Note: For SuperBPE tokenizer training, see the dedicated superbpe skill.

Quick Start

Compare Two Tokenizers

from unsloth.tokenizer import compare_tokenizers

results = compare_tokenizers(
    text="The quick brown fox jumps over the lazy dog",
    tokenizer1="meta-llama/Llama-3.2-1B",
    tokenizer2="gpt2"
)

print(f"Llama-3.2: {results['tokenizer1']['tokens']} tokens")
print(f"GPT-2: {results['tokenizer2']['tokens']} tokens")
print(f"Difference: {results['reduction']}")

Analyze Tokenization

from unsloth.tokenizer import analyze_tokenization

analysis = analyze_tokenization(
    text="Your text here...",
    tokenizer="meta-llama/Llama-3.2-1B"
)

print(f"Total tokens: {analysis['total_tokens']}")
print(f"Unique tokens: {analysis['unique_tokens']}")
print(f"Token breakdown: {analysis['tokens']}")

Batch Compare Multiple Tokenizers

from unsloth.tokenizer import compare_multiple_tokenizers

tokenizers = [
    "meta-llama/Llama-3.2-1B",
    "meta-llama/Llama-3.2-3B",
    "mistralai/Mistral-7B-v0.1",
    "gpt2",
    "./tokenizers/custom_superbpe.json"
]

results = compare_multiple_tokenizers(
    text="Your text...",
    tokenizers=tokenizers
)

# Results sorted by efficiency
for result in results:
    print(f"{result['name']}: {result['tokens']} tokens")

Use Cases

1. Find Most Efficient Tokenizer

Problem: Need to choose the best tokenizer for your use case

Solution:

# Test multiple candidates
candidates = [
    "meta-llama/Llama-3.2-1B",
    "mistralai/Mistral-7B-v0.1",
    "google/gemma-2b",
    "./tokenizers/custom_superbpe.json"
]

# Use representative sample
sample_text = get_production_sample(size_kb=100)

results = compare_multiple_tokenizers(
    text=sample_text,
    tokenizers=candidates
)

# Choose most efficient
best = min(results, key=lambda x: x['tokens'])
print(f"Best tokenizer: {best['name']} with {best['tokens']} tokens")