cli-inference

CLI Inference

Use this workflow for uv run mlx_vlm.generate and related command-line inference tasks.

First Checks

1. Identify the model ID or local path, modality, prompt, media files, and expected output.
2. Prefer an existing local model path or cached Hugging Face model when reproducing. Do not download a large model unless the user asks.
3. Check model-specific docs first when the family has a README under mlx_vlm/models/<family>/README.md.
4. Use uv run mlx_vlm.generate --help to verify current flags before giving a final command.

Command Patterns

Text:

uv run mlx_vlm.generate \
  --model <model-or-path> \
  --prompt "Write a short answer." \
  --max-tokens 128

Image:

uv run mlx_vlm.generate \
  --model <model-or-path> \
  --image /path/to/image.jpg \
  --prompt "Describe this image." \
  --max-tokens 128

Audio or multimodal:

uv run mlx_vlm.generate \
  --model <model-or-path> \
  --image /path/to/image.jpg \
  --audio /path/to/audio.wav \
  --prompt "Describe what you see and hear." \
  --max-tokens 128

More Flags & Modalities

uv run mlx_vlm.generate --help is the source of truth (the CLI lives in the mlx_vlm/generate/ package). Beyond --model/--prompt/--image/--audio/--max-tokens, common flags:

• --chat — interactive multi-turn session.
• --adapter-path — apply a LoRA adapter.
• --eos-tokens and sampling flags — use greedy/low-temp when debugging quality.
• --trust-remote-code — needed for some custom processors.
• --kv-bits / --kv-quant-scheme / --max-kv-size — quantized/bounded KV cache.
• --draft-model / --draft-kind — speculative decoding.
• Image generation/editing: --output-modality image, --output <file>, and the --diffusion-* flags drive diffusion image models — not just text-out VLMs.

Reproducibility Rules

• Include the exact command, model ID/path, media file type and size, Python version, package version or git commit, and full error.
• Use low-temperature or greedy settings when debugging quality or regressions.
• Bound output with --max-tokens.
• Preserve shell quoting exactly, especially prompts with JSON, XML-like thinking tokens, or newlines.
• If the failure depends on an image/audio/video file, record dimensions, duration, codec, and whether a small synthetic input reproduces it.