screen-ocr-operator

Screen OCR Operator (Commander Mode)

Act as the commander: you issue ADB commands, the vision model looks at screenshots and returns a concrete operation plan. The goal is to finish screen tasks with the fewest possible screenshot/OCR round-trips.

Roles

• Commander (you): capture screenshots, call the vision model, execute ADB taps/input, verify.
• Eyes (vision model): OCR the screen, identify UI elements, return coordinates and steps.
• Hands (ADB): input tap, input text, input keyevent, am start, etc.

Key optimization rules

1. Ask the model for a complete plan, not a single action.
   • Give the model the task + current screenshot.
   • Ask it to return a JSON list of actions with coordinates:

     {
       "actions": [
         {"type": "tap", "x": 630, "y": 1090},
         {"type": "text", "value": "hello"},
         {"type": "key", "key": "ENTER"}
       ],
       "verify": "expected screen after actions"
     }
     

2. One model call per meaningful milestone.
   • Don't screenshot + OCR after every tap.
   • Only re-ask when the next step depends on a changed screen or after a risky action.
3. Use Android APIs for deterministic checks first.
   • dumpsys window | grep mCurrentFocus → which app is foreground.
   • dumpsys input_method | grep mInputShown → is keyboard open.
   • uiautomator dump → native UI nodes/bounds when available.
   • These are faster and more reliable than OCR for state checks.
4. Shrink screenshots before sending to the model.
   • Capture full screen, then downscale to max width ~720px and save as JPEG quality ~80.
   • Smaller payload = much faster API round-trip.
5. Batch ADB commands.
   • Combine independent shell commands into one adb shell invocation.
   • Keep sleeps short (sleep 0.5–1) and only wait when the UI actually needs time.
6. Verify at the end (or at major checkpoints).
   • Final screenshot + one model confirmation is usually enough.
   • For long tasks, verify after each phase, not after every tap.