LLaVA-OneVision-2: Skill for OpenCode

Purpose / 用途

Use this skill when reasoning about attention boundaries in the LLaVA-OneVision2 forward pass — specifically how cu_lengths and patch_positions control attention at different stages of the model.

在分析 LLaVA-OneVision2 前向传播中的 attention 边界时使用这个 skill——具体来说，cu_lengths 和 patch_positions 如何在模型的不同阶段控制 attention。

This skill is specifically for:

• Understanding the difference between ViT-level and LLM-level attention control
• Debugging packed vs non-packed attention behavior
• Reasoning about cross-sample isolation in packed sequences
• Understanding why patch_positions grouping does NOT carry into the LLM

这个 skill 专门用于：

• 理解 ViT 层和 LLM 层 attention 控制的区别
• 调试 packed 和 non-packed 的 attention 行为
• 分析 packed 序列中跨样本隔离机制
• 理解为什么 patch_positions 的分组不会延续到 LLM 中

---

Key Files / 关键文件

File	Role
aiak_training_llm/train/pretrain/pretrain_llava_onevision2.py	Forward function — decides packed vs non-packed path based on cu_lengths shape
aiak_training_llm/train/sft/utils.py	_get_packed_sequence_params() — builds PackedSeqParams from attention_mask for SFT
aiak_training_llm/data/multimodal/task_encoder.py	batch() — sets cu_lengths to [[0]] (dummy) for non-packed, or stacks real cu_lengths for packed
aiak_training_llm/data/multimodal/task_encoder.py	pack_selected_samples() — constructs cu_lengths = [0, len_1, len_1+len_2, ...] for offline packed data
aiak_training_llm/models/llava_onevision2/onevision_encoder_model.py	ViT encoder — uses patch_positions for local/shared attention
aiak_training_llm/data/multimodal/qwen2vl_task_encoder.py	process_sft_qa() — generates patch_positions from image_grid_thw

---

Core Concept: Two Independent Attention Control Mechanisms / 核心概念：两套独立的 Attention 控制机制

Overview Diagram / 概览图

┌─────────────────────────────────────────────────┐
│  ViT Encoder                                    │
│                                                 │
│  Control: patch_positions (temporal dimension)  │
│  Effect:  Local/shared attention                │
│           e.g. 4 images share one attention     │
│           window via same temporal index        │
│                                                 │
│  Output:  visual embeddings                     │
└──────────────────┬──────────────────────────────┘
                   │  (embeddings replace image
                   │   placeholder tokens)
                   ▼
┌─────────────────────────────────────────────────┐
│  LLM Decoder                                    │
│                                                 │
│  Control: cu_lengths (cumulative sub-seq lens)  │
│  Effect:  Determines attention domain           │
│                                                 │
│  NON-PACKED: cu_lengths == [[0]]                │
│    → full causal attention                      │
│    → ALL tokens see ALL previous tokens         │
│    → patch_positions grouping is GONE           │
│                                                 │
│  PACKED: cu_lengths = [0, a, a+b, ...]          │
│    → block-diagonal causal attention            │
│    → sub-sequences isolated from each other     │
│    → within each sub-seq: full causal           │
└─────────────────────────────────────────────────┘