MCP-OsAutomation: Instructions file for Claude Code

MCP WebAutomation - Claude Code Instructions

Environment Setup

• System: Single monitor setup with terminal-based workflow
• Primary Interface: Terminal/CLI applications preferred
• Display: One screen with multiple applications via Alt+Tab switching

Tools Available

Tool	Purpose
analyze_screen()	DUAL MODE: Get coordinates OR extract text
click_element()	Click by coordinates or text
type_text()	Type text with options
press_key()	Keyboard shortcuts
emergency_stop()	Stop all automation

Workflow Protocol

1. Task Analysis & Program Discovery

Before performing any task:

# Step 1: Switch to see available programs
mcp.call_tool("press_key", {"key": "tab", "modifiers": ["alt"]})

# Step 2: Analyze current screen to understand what's available
screen = mcp.call_tool("analyze_screen", {"include_ocr": False})

# Step 3: Check if required programs are running via bash
result = mcp.call_tool("bash_command", {"command": "pgrep -l firefox|chrome|code|terminal"})

2. Program Navigation Strategy

# Navigate through applications using Alt+Tab until target found
def find_target_program(target_program):
    max_attempts = 10
    for attempt in range(max_attempts):
        screen = mcp.call_tool("analyze_screen", {"include_ocr": False})

        if target_program.lower() in screen["screen_description"].lower():
            return True

        # Switch to next program
        mcp.call_tool("press_key", {"key": "tab", "modifiers": ["alt"]})

    return False

3. Task Execution Priority

1. PREFER BASH: If task can be done via terminal commands
2. GUI FALLBACK: Use desktop automation only when necessary
3. HYBRID APPROACH: Combine bash + GUI for optimal efficiency

4. Complete Task Workflow

def execute_user_task(user_request):
    # Step 1: Analyze what user wants
    # Step 2: Check if bash can handle it
    if can_use_bash(user_request):
        return execute_bash_solution(user_request)

    # Step 3: Find required GUI program
    target_program = determine_required_program(user_request)

    # Step 4: Navigate to program
    if find_target_program(target_program):
        return execute_gui_task(user_request)
    else:
        # Step 5: Launch program if not running
        launch_program(target_program)
        return execute_gui_task(user_request)