phone-harness

phone-harness

The main helpers are listed in Helpers.

Direct control of a phone from Python scripts. The same helpers drive every kind of phone below; what differs is how each one sees and touches the screen, and that is what the per-phone sections below are for. Read Which phone, the shared Working method, and then only the section for the phone in front of you.

Setup

Check what this machine can do (OS, shell, install rights, network, attached phones; phone-harness doctor does the phone part), tell the user in a sentence or two which phone you can drive with what is here, and use the best one that actually runs. A phone on the desk wins; the cloud needs no hardware and is always the fallback.

• If something is missing, name it and the one step the user can take.
• Ask the user only to approve sign-in and to pick a phone.
• Detail: iPhone, Android, Cloud, API.

Helpers

• ensure_device() - Screen bounds once the device is usable, or raises with what the user has to do.
• screen(min_confidence=0.3) - Visible text as {text, confidence, source, x, y, w, h} boxes. On Android the boxes come from the accessibility tree; on iPhone Mirroring they come from recognizing text.
• screenshot(path=None) - Capture the screen to a PNG, print its path, and return it.
• find_text(query, exact=False) - Visible text boxes matching query.
• tap_text(query, index=0, exact=False) - Tap the center of visible text matching query.
• tap(x, y) - Tap one point.
• long_press(x, y, duration=0.8) - Press and hold a point.
• type_text(text, delay=0.03, keystrokes=False) - Type into the focused field.
• press(combo) - Press a key, such as press("enter").
• open_app(name) - Launch an app and return the app id that launched.
• open_url(url) - Open an https link in the phone's browser, whole. Cloud phones only for now.
• push_file(path) - Send a file to the phone; it opens in the phone's browser under its own name. Cloud phones only for now.
• swipe(direction, distance=0.4, at=None) - Flick. direction is finger motion: "up", "down", "left", or "right".
• scroll(direction="down", amount=0.3, at=None) - Scroll. direction is the content to reveal: "down", "up", "left", or "right".
• scroll_screen(direction="down", amount=0.6, settle=2.5, moved_thresh=None, at=None) - One scroll, then the text on screen before and after.
• scroll_until(done, direction="down", amount=0.6, max_scrolls=60, settle=2.5, at=None) - Scroll until done(boxes) is truthy or the screen stops changing.
• scroll_collect(extract=None, key=None, direction="down", amount=0.6, max_scrolls=400, end_after=3, settle=2.5, on_progress=None, at=None) - Scroll a list and return {items, stop, scrolls}.
• home() - Go to the Home Screen.
• back() - System Back, where the phone has one.
• app_switcher() - Open the app switcher.
• wait_stable(timeout=6.0, interval=0.5, settle=2) - Wait until the screen stops changing.
• wait_for_text(query, timeout=10.0, exact=False, interval=0.5) - Poll until query is visible; return its box, or None.
• wait_for_app(app_id, timeout=10.0, interval=0.3) - Poll until app_id is the foreground app.
• ui() - The accessibility tree, where the device has one.
• tap_ui(query, index=0, exact=False, clickable_only=False) - Tap an element in the accessibility tree.
• current_app() - Foreground app id, where the device exposes one.
• list_apps(include_system=False) - Installed app ids.
• screen_info() - {window, frontmost, img_px}: bounds, focus, and capture size.
• find_window() - Screen bounds {x, y, w, h, id}, or None if there is no device.
• image_point(x, y, image_size=None) - Convert a screenshot pixel to a screen point.
• tap_image_point(x, y, image_size=None) - Tap a point taken from a screenshot.
• connection_state() - Backend state string; "ready" means usable.
• shell(cmd, binary=False, timeout=60) - Run a shell command on the phone, where the device has a shell.