find-ui-element

AI Builder - Find UI Element

Locate any clickable or interactive UI element in a screenshot — buttons, links, dropdowns, checkboxes, radio buttons, text input areas, etc. Uses a vision-first approach:

1. find quickly from the full image,
2. verify with a small close-up crop,
3. if wrong, retry with fixed-grid crops in nearby regions.

When to Use This Skill

• Locating any clickable element (button, link, dropdown, checkbox, radio button) in a screenshot
• Finding text input areas (search boxes, form fields, text areas) for typing automation
• Getting pixel coordinates of a UI element for computer-use tasks
• Verifying element positions before automated interaction

Your Roles in This Skill

• QA Engineer: Visually inspect screenshots, locate target elements, and verify bounding box accuracy.

Role Communication

As an expert in your assigned roles, you must announce your actions before performing them using the following format:

As a {Role}, I will {action description}

This communication pattern ensures transparency and allows for human-in-the-loop oversight at key decision points.

Instructions

Follow these steps in order.

Retry policy:

• MAX_TRIES = 4 total attempts (1 initial + up to 3 retries)
• Stop early when the verification crop clearly confirms the element
• If all tries fail, report failure explicitly (element may be absent or not visually identifiable)

Step 1: Load and View the Screenshot

Load the screenshot and note its dimensions. Then visually inspect the full image to estimate where the target element is.

core/bin/python -c "from PIL import Image; img = Image.open('INPUT_PATH'); print(f'Dimensions: {img.size[0]}x{img.size[1]}')"

Step 2: Estimate Coordinates by Vision

Look at the full screenshot and estimate the bounding box (x1, y1, x2, y2) of the target element based on visual inspection. Use the image dimensions to guide your estimate.

Step 3: Draw and Verify (Small Area First)

Draw the bounding box, then crop around it to verify.