computer-use

Computer Use with the CUA SDK

• Prefer a dedicated connector or API. Use Computer Use only for UI state or interactions the dedicated interface does not expose.
• Perform Computer Use through node_repl and the typed ComputerUse API. Do not use generic callTool, direct driver imports, AppleScript, JXA, or synthesized-input utilities.
• Observe the exact current window before acting. Prefer current element tokens over screenshot coordinates; use coordinates only when accessibility is incomplete and the screenshot provides the target.
• Treat an action result as delivery evidence, not task completion. Decide from fresh state and require stable postcondition evidence.

Setup

If node_repl is unavailable, run these commands yourself:

qwen mcp add --scope user node-repl npx -y @qwen-code/[email protected]
npm install --no-save --package-lock=false @qwen-code/[email protected]

Tell the user to restart Qwen Code, then stop. If only the SDK import is missing, run the second command and retry.

Create one persistent client per REPL kernel:

globalThis.computer = await (
  await import('@qwen-code/cua-sdk/computer-use')
).ComputerUse.create();
globalThis.cuaRevisions ??= new Map();

Target and observe

Use listApps({signal:nodeRepl.signal}) and filter in JavaScript; print only likely matches. After selecting a real PID, call listWindows({pid,signal:nodeRepl.signal}) and choose from returned metadata. Never guess a PID, window ID, element token, or coordinate. If the app is not running, start it with ordinary Node.js process APIs and refresh the lists.

Maintain one revision cursor per window surface. The first observation has no base; later observations use only the last revision actually consumed for that same surface:

globalThis.observeCuaWindow = async (target, options = {}) => {
  const key = `${target.pid}:${target.windowId}`;
  const state = await computer.observeWindow({
    ...target,
    ...options,
    baseRevisionId: cuaRevisions.get(key),
    signal: nodeRepl.signal,
  });
  if (state.revisionId) cuaRevisions.set(key, state.revisionId);
  return state;
};