jev-desktop

jev-desktop

Two entry points over the same screen reading. run.mjs takes a goal and drives until it is met. act.mjs takes one step and hands the plan back to you. Neither puts the tree in your context.

The loop

node scripts/jev/run.mjs --app Finder "open the Applications folder"
node scripts/jev/run.mjs --app TextEdit --cursor --text "A line to write." "write the supplied sentence into the document"

Each turn reads the screen, asks for one operation and a target for that operation in the same request, carries it out, and reads again. It prints one JSON line per turn and one when it stops.

{"turn":{"step":1,"operation":"CLICK","target":"treeitem \"Applications\"",
         "confidence":1,"ok":true,"delivery":"delivered_verified","changed":true}}
{"stop":"done","confidence":0.96}

The operations are CLICK, TYPE_TEXT, CHECK, UNCHECK, EXPAND, COLLAPSE, SCROLL, DRILL, WIDEN, WAIT, DONE and BLOCKED. Only the ones something on screen can receive are offered.

The target is asked once per operation. click_target is chosen from the elements that advertise Click, type_text_target from those that advertise SetValue, and so on. Only the head matching the chosen operation is read, so a target can never be incompatible with the verb that acts on it. That is the whole reason the loop does not need to correct itself afterwards.

Looking is an operation. A window is read with --skeleton first, so a document holding four thousand elements costs the same first look as a panel holding thirty. A region that was cut off reports how much it holds, and DRILL pins it as the root for later turns. WIDEN gives the whole window back.

CHECK and UNCHECK instead of a toggle. Both are idempotent, so a box already in the wanted state stays there and the policy never reasons about the current one.

You supply the text. Nothing here writes a value, so a run never puts a string on screen that you did not choose. Pass --text once per value and they are consumed in order, or pass a function as the text option and it is asked for each field with that field's description. When there is no value left to give, TYPE_TEXT is not offered at all, so the run stops rather than inventing one.