MLX-VLM is a Python package for running inference and fine-tuning vision-language models, which process images and text together, on Mac computers using MLX. It is used by developers working with multimodal models and includes command-line, web, server, scripting, and training workflows.
Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add Blaizzy/mlx-vlm --skill add-new-modelgit clone --depth 1 https://github.com/Blaizzy/mlx-vlmWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/blaizzy/mlx-vlm/add-new-model)<a href="https://agentmods.dev/skills/blaizzy/mlx-vlm/add-new-model"><img src="https://agentmods.dev/badge/skills/blaizzy/mlx-vlm/add-new-model.svg" alt="Measured on agentmods" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00083 | $0.01087 |
| Opus 5 | $0.00042 | $0.00544 |
| Sonnet 5 | $0.00017 | $0.00217 |
| Haiku 4.5 | $0.00008 | $0.00109 |
Grade A, and why
add-new-model scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 64 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Add a New Model
Use this workflow to port a new model to MLX-VLM.
Layout Rules
- New model lives in
mlx_vlm/models/<model_type>/, and the main file is named after theconfig.jsonmodel_type(e.g.model_type: "llava"→mlx_vlm/models/llava/llava.py). The loader resolves the arch by importingmlx_vlm.models.<model_type>(seeMODEL_REMAPPINGinmlx_vlm/utils.pyfor aliases). - Split by concern like the existing families:
language.py,vision.py,config.py,processing_*.py. A new kernel/helper goes in its own file in the model dir, not insidelanguage.py. - Start from a similar existing model in
mlx_vlm/models/and adapt — don't write from scratch.
Steps
- Confirm weights are safetensors. If not, convert them first (HF safetensors converter), then proceed.
- Copy a close relative as scaffolding (same attention/vision style). Rename to the new
model_type. - Write
config.py. AModelConfigdataclass; give every new field a backward-compatible default (None/0/False) so existing configs still load unchanged. Add inline# comments. - Map layer/weight names. Determine them by one of:
- the Transformers implementation, if you know it;
- loading the weights and printing key names;
- reading
model.safetensors.index.jsonin the HF repo.
- Wire the forward pass (embeddings → vision/audio encoder → projector → language model), reusing shared helpers (
prompt_utils.py, processors) where possible. - Convert to MLX to get loadable weights —
Skill("mlx-vlm-skills:convert-quantize"). Upload them to tomlx-community(which is self-add) on HF if there is no usable official repo. If the model is already usable directly from HF, you can skip this step and just use the repo directly. - Add a test class in
mlx_vlm/tests/test_models.py(e.g.TestMyModel) — a tiny random-weight config, a shape/forward check, and (if applicable) an exactness check against a reference path in the degenerate limit. Do not create a standalone test file. - Add a README in the model dir with a short description of the model, supported HF repos, and example usage.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 64 lines · 83 tokens per session scan A a6f2376a7ddc
add-new-model is a skill published in the GitHub repository Blaizzy/mlx-vlm (5,477 stars, last pushed yesterday), licensed MIT. It adds 83 tokens to every session and 1,087 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
mlx-model-porting
Guides and validates architecture-aware ports of PyTorch/Hugging Face models to Apple MLX, inspects existing local MLX projects, and plans evidence-gated optimizations for Apple Silicon. Use when the user asks to run, port, convert, inspect, quantize, benchmark, or fix a model (LLM, VLM, audio/TTS/ASR, diffusion, SSM…
batch-processing-clinical-text
Run large-scale batch NER, PII extraction, or de-identification over many clinical notes on-device with OpenMed, with sharding, checkpointing, resumability, and append-only JSONL output. Use when the user needs to process a corpus or folder of notes, de-identify a dataset, run NER over thousands of documents, build a…
bridging-presidio-and-spacy
Combine OpenMed clinical NLP with Microsoft Presidio, spaCy, or LangChain through OpenMed's built-in interop adapter registry (openmed.interop). Covers the lazy adapter registry (availableadapters, getadapter, adapterspec), the presidio/spacy/langchain pip extras, and the verified callables — Presidio…
exporting-to-fhir
Convert OpenMed NER output (entities from openmed.analyzetext) into FHIR R4 resources — Condition, MedicationStatement, Observation — using OpenMed's built-in FHIR R4 export helpers in openmed.clinical.exporters. Covers the verified CodeableConcept builder (coding, codeableconcept, systemuri), deterministic fullUrl…
authoring-model-cards
Generate a model card for an OpenMed clinical NER or de-identification model documenting intended use, quantitative metrics, subgroup performance, limitations, and a medical-device disclaimer for clinical AI governance. Use when the user wants to write or update a model card, a README model section, or governance…
building-gold-corpus
Scaffold a synthetic gold-standard annotation project for evaluating OpenMed NER and de-identification models — label schema, annotation guidelines, BRAT or Label Studio config, and disjoint train/dev/test splits. Use when the user wants to create eval fixtures, set up annotation, define a label set, write guidelines…