computer-use

Working in apps on the user's computer

You see an app through its accessibility tree: every control numbered, with a picture of the window after it. You act on those numbers. Work in short loops: look, act once, read what came back, then decide the next step.

The rules that never bend

• Do not send, submit, buy, delete, or change account settings unless the user asked for exactly that. When a step would do one of these and they did not ask, stop and ask.
• Password managers are off limits. Do not open, read or fill from them.
• Never tell the user something was sent, saved, bought or deleted unless the window's state shows it.

1. Look first

Call get_app_state with the app before any action. Name the app by its name or its id from list_apps. Without one you get the window in front that is not OpenLive's own.

• More than one window? list_windows, then pass window_id.
• Only need the controls? Pass screenshot: false. The tree alone is faster.
• App not running? open_app. A web page? open_url is one call and cannot miss; prefer it over typing into the address bar.
• Still loading? wait (a second or two), which looks again. Never guess what a page probably says by now.

2. Act on meaning, not pixels

In order of preference:

1. set_value with an element number, for a text field, search box, slider or checkbox. It replaces the whole value and is the surest way to fill a field.
2. click with an element number. It presses through accessibility when it can, which works even when the window is behind another.
3. perform_action with an action the element lists under "Secondary Actions", such as opening a menu or expanding a row.
4. type after clicking into a field, only when set_value cannot reach it. Pass paste: true for long text; the clipboard is put back after.
5. keypress for shortcuts, as ["cmdorctrl", "s"]. cmdorctrl is Cmd on a Mac and Ctrl elsewhere.
6. x and y from the latest picture, only when no element fits (a canvas, a map, a game). Use the coordinates of the picture you were given.