LLaVA-OneVision-2: Skill for OpenCode

Purpose / 用途

Use this skill when merging a standalone ViT encoder and LLM into a unified LlavaOnevision2 HuggingFace checkpoint, and when validating that the merged weights and inference outputs are consistent with the originals.

当需要将独立的 ViT encoder 和 LLM 合并成统一的 LlavaOnevision2 HuggingFace checkpoint，并验证合并后权重和推理输出与原始模型一致时，使用这个 skill。

Prerequisites / 前置条件

• Container llava_megatron_container_ax running with GPU access
• All paths below assume execution inside the container at /workspace/LLaVA-OneVision-2
• PYTHONPATH=transformers_impl:. must be set for all Python commands
• For large models, use tmpfs (/train_tmp) for I/O performance

容器 llava_megatron_container_ax 需启动并有 GPU 访问权限。以下所有路径假设在容器内 /workspace/LLaVA-OneVision-2 执行。所有 Python 命令需设置 PYTHONPATH=transformers_impl:.。大模型建议用内存盘 /train_tmp。

Architecture / 架构

What merge_ov2 does / merge_ov2 做了什么

ViT encoder (e.g. onevision_encoder_patch16_0424)
  + LLM (e.g. Qwen3-4B-Instruct-2507)
  + Processor (e.g. lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct)
  → Unified LlavaOnevision2ForConditionalGeneration checkpoint

Key transformations during merge:

合并时的关键转换：

1. ViT weights are prefixed with visual. (e.g. encoder.layers.0.self_attn.q_proj.weight → visual.encoder.layers.0.self_attn.qkv.weight)
2. QKV fusion: separate q_proj / k_proj / v_proj are concatenated into fused self_attn.qkv (introduces ~1e-7 bf16 divergence)
3. LLM weights are prefixed with language_model. (e.g. model.layers.0.self_attn.q_proj.weight → language_model.model.layers.0.self_attn.q_proj.weight)
4. Adapter (multi_modal_projector) is randomly initialized if no adapter checkpoint is provided
5. layernorm_post from ViT is dropped (not used in LlavaOnevision2)
6. class_embedding may not exist in some ViT encoders (e.g. patch16 variant)

Source code layout / 源码结构

transformers_impl/merge_ov2/
├── __main__.py          # CLI entry point
├── cli.py               # Argument parsing for merge / validate / dry-run
├── remap.py             # Weight key remapping logic
├── loader.py            # Weight loading from source checkpoints
├── save.py              # Save merged checkpoint
├── io.py                # I/O utilities
├── utils.py             # Shared utilities
├── variants/
│   ├── dense.py         # Dense model variant
│   └── moe.py           # MoE model variant
└── validators/
    ├── vit_layerwise.py   # ViT layer-wise weight validator
    ├── vit_blockorder.py  # ViT block-order validator (patch14+sms=2 only)
    ├── llm_parallel.py    # LLM parallel validator
    ├── llm_sequential.py  # LLM sequential validator
    └── e2e.py             # End-to-end validator