OpenCUA: Instructions file for Claude Code

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

OpenCUA is an open-source framework for scaling computer-use agent data and foundation models. It consists of:

• AgentNet: Large-scale dataset (22.6K+ tasks) across Windows, macOS, and Ubuntu
• AgentNetTool: Cross-platform GUI recording and annotation tool (Git submodule in tool/)
• OpenCUA Models: End-to-end foundation models (7B, 32B, 72B variants) based on Qwen2.5-VL

Commands

Environment Setup

conda create -n opencua python=3.10
conda activate opencua

# Model inference dependencies
pip install -r model/requirement.txt

# Data processing dependencies
pip install -r data/data-process/requirements.txt

# Headless server support (for video processing)
sudo apt-get install -y xvfb

Model Inference

HuggingFace:

cd model/inference
python huggingface_inference.py

vLLM Server (recommended for production):

CUDA_VISIBLE_DEVICES=1,2,3,4 vllm serve /path/to/OpenCUA-32B \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --served-model-name opencua-32b \
  --host 0.0.0.0 \
  --port 8000

Data Processing Pipeline

Stage 1: Extract raw trajectories from video/events:

cd data/data-process
xvfb-run -a ./scripts/extract_raw.sh -1    # all samples
xvfb-run -a ./scripts/extract_raw.sh 10    # first 10 samples

Stage 2: Standardize to unified schema:

xvfb-run -a ./scripts/raw_to_standardized.sh -1

Stage 3: Generate CoT (requires API_KEY env var):

cd data/cot-generate
export API_KEY="your-api-key"
python gen_cot.py \
  --traj_path ./gen_cot_example/raw_example.jsonl \
  --image_folder ./gen_cot_example/images \
  --output_dir ./gen_cot_example/output/tasks \
  --model claude-3-7-sonnet-20250219

Visualization

cd data/vis
streamlit run app.py -- --traj_file <TRAJ_JSONL> --meta_file <META_JSONL> --image_dir <IMAGES>