groq-inference

Groq — Ultra-Fast LLM Inference

Groq runs open-weight LLMs on custom LPU (Language Processing Unit) hardware, delivering the lowest inference latency in the industry — up to 1000+ tokens/sec. OpenAI-compatible API, free tier, pay-per-token pricing. Inference only, no training.

When to Use Groq

Use when:

• You need the fastest possible inference latency (real-time chat, agents, interactive apps)
• You want open-weight models (Llama, Qwen, DeepSeek) without managing GPUs
• You need an OpenAI-compatible drop-in replacement with open models
• You want free-tier access for prototyping
• You need fast Whisper transcription or vision with Llama 4

Don't use when:

• You need to fine-tune or train models (inference-only)
• You need proprietary models (GPT-4o, Claude, Gemini)
• You need embeddings or image generation APIs (not offered)

Feature	Groq	Together AI	Fireworks	Replicate
Hardware	Custom LPU	NVIDIA GPUs	NVIDIA GPUs	NVIDIA GPUs
Speed (Llama 70B)	~280 tok/s	~100 tok/s	~100 tok/s	~50 tok/s
Free tier	Yes	Yes	Yes	No
Training	No	Yes	Yes	Yes

Credential Setup

Credentials are auto-injected by openscience when connected via the dashboard.

# Verify credentials
[ -n "$GROQ_API_KEY" ] && echo "GROQ_API_KEY set" || echo "NOT SET"

If not set: add your Groq key in Customize → Models or export GROQ_API_KEY locally.

Quick Start

pip install groq
export GROQ_API_KEY="gsk_..."   # https://console.groq.com/keys

from groq import Groq

client = Groq()
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in one paragraph."}
    ],
    temperature=0.7,
    max_completion_tokens=512,
)
print(response.choices[0].message.content)

Chat Completions