jev-desktop

jev-desktop

This skill requires a clone of the agent-desktop repository (the scripts live in scripts/jev/ and ship in no installed package) and a TYPESAFE_API_KEY.

Two entry points over the same screen reading. run.mjs takes a goal and drives until it is met. act.mjs takes one step and hands the plan back to you. Neither puts the tree in your context.

The loop

node scripts/jev/run.mjs --app Finder "open the Applications folder"
node scripts/jev/run.mjs --app TextEdit --cursor --text "A line to write." "write the supplied sentence into the document"

Each turn reads the screen, asks for one operation and a target for that operation in the same request, carries it out, and reads again. It prints one JSON line per turn and one when it stops (shown trimmed).

{"turn":{"step":1,"operation":"CLICK","target":"treeitem \"Applications\"",
         "confidence":1,"ok":true,"delivery":"delivered_verified","changed":true}}
{"stop":"done","confidence":0.96}

The operations are CLICK, TYPE_TEXT, CHECK, UNCHECK, EXPAND, COLLAPSE, SCROLL, DRILL, WIDEN, WAIT, DONE and BLOCKED. Only the ones something on screen can receive are offered: DRILL for a region that reports it holds more, WIDEN only after a DRILL. SCROLL always scrolls down. delivery is waited for WAIT and looked for DRILL and WIDEN.

The target is asked once per operation. click_target is chosen from the elements that advertise Click, type_text_target from those that advertise SetValue, and so on. Only the head matching the chosen operation is read, so a target can never be incompatible with the verb that acts on it.

Looking is an operation. A window is read with --skeleton first, so a dense document costs the same first look as a small panel. A region that was cut off reports how much it holds, and DRILL pins it as the root for later turns. WIDEN gives the whole window back.

CHECK and UNCHECK instead of a toggle. Both are idempotent: a box already in the wanted state stays there.