marionette-flutter-drive-app

marionette_flutter: drive an app

Marionette lets an AI agent drive a running Flutter app (debug or profile — never release) the way a real user would: read the screen, tap, type, scroll, screenshot, read logs, hot reload. It has two halves:

• marionette_flutter — the package you add to the app. It registers a VM service extension per action (ext.flutter.marionette.*).
• The bridge — either marionette_mcp (MCP tools, e.g. connect, tap, get_interactive_elements) or marionette_cli (shell commands, e.g. marionette tap --key ...). Nearly the same capabilities, different transport — the CLI additionally offers record-video, with no MCP equivalent (see the capability table, below). Prefer the MCP tools when they're in your tool list; fall back to the CLI in restricted environments (enterprise policy, a shell-only agent) — run marionette help-ai once at the start of such a session and follow its reference for exact syntax.

Everything shared between the two applies to either transport — action names differ only in casing (get_interactive_elements vs get-interactive-elements).

Preparing the app

connect (or any CLI command) only works if the target app initialized MarionetteBinding. If you're not sure whether it has, check main.dart (or ask) — skip the rest of this section if it's already there and you just need to drive the app, and jump to When to use this.

flutter pub add marionette_flutter

void main() {
  if (kDebugMode) {
    MarionetteBinding.ensureInitialized();
  } else {
    WidgetsFlutterBinding.ensureInitialized();
  }
  runApp(const MyApp());
}

This is all a standard Material-widgets app needs. It runs inside main(), so any time you add or change it, hot restart — a hot reload doesn't re-execute main(), so the change will look like it did nothing. Gate every piece of Marionette wiring behind kDebugMode (or !kReleaseMode — see Device-config sweeps, below) so none of it ships in a release build: both are compile-time constants the release branch survives cleanly.