model-integration

Goal

Integrate a new model into diffusers end-to-end, to full numerical parity with the reference implementation — one workflow at a time.

Setup — gather before starting

Before writing any code, gather info in this order:

1. Reference repo — ask for the github link. If they've already set it up locally, ask for the path. Otherwise, ask what setup steps are needed (install deps, download checkpoints, set env vars, etc.) and run through them before proceeding.
2. Inference script — ask for a runnable end-to-end script for a basic workflow first (e.g. T2V). Then ask what other workflows they want to support (I2V, V2V, etc.) and agree on the full implementation order together.
3. Standard vs modular — default to modular. Modular Diffusers is the preferred implementation for new pipelines; the standard DiffusionPipeline is still supported but no longer the default. We prefer modular especially for models that don't fit a fixed task-based structure (modality baked into the checkpoint) or that are actively evolving.

Ask step 3 as an AskUserQuestion, with modular marked as the recommended default.

Once you have everything, confirm the plan with the user before implementing — state exactly what you'll do, e.g. "I'll integrate model X with pipeline Y based on your script, and verify the model matches the reference before considering it done."

Then work through the Integration checklist below

Integration checklist

A pipeline in Diffusers (be it standard or modular) will have multiple components. These components can be models, schedulers, processors, etc.

• Transformer model
  • Implement the model with from_pretrained support (conventions: models.md)
  • Convert weights (see Weight / Checkpoint Conversion)
  • Parity test against the reference (internal, not shipped — see Model parity test)
  • Register in the relevant __init__.py files (lazy imports)
  • Model-level tests (see Testing)
• VAE (if applicable) — reuse an existing AutoencoderKL* if possible; if a new one is needed, follow the same sub-steps as the transformer
• Scheduler — reuse an existing scheduler, or add a custom one
• Pipeline
  • Implement the pipeline — see modular.md for modular pipeline, or pipelines.md for standard pipeline
  • Add a LoRA mixin if applicable
  • Register in the relevant __init__.py files (lazy imports)
  • Pipeline-level tests (see Testing)
• Docs — see File structure
• Style — make style and make quality