simulang-js: Instructions file for Claude Code

@simular-ai/simulang-js

Node.js bindings for the Rust simulang-rs crate (via napi-rs). Cross-platform automation of desktops and Android devices: apps, windows, accessibility trees, mouse/keyboard, screenshots, clipboard, audio, and VLM/LLM/STT model access.

This file ships in the npm tarball alongside index.d.ts and is versioned with it.

Where the API is documented

Read index.d.ts first — it is the source of truth. Every class, function, and enum is fully typed (~2100 lines) and carries JSDoc covering idioms, lifecycle rules, platform quirks, and inter-API trade-offs that types alone can't express. The JSDoc is generated from doc comments, so the per-symbol guidance there is authoritative — trust it over any restatement elsewhere.

Mental model

• Machine is the entry point. Machine.local() is the desktop this process runs on; Machine.android(endpoint) is an Android device driven over adb from the host. Apps, windows, and accessibility nodes are obtained from a machine and stay bound to it; mouse, keyboard, and clipboard access are flat methods on the machine itself (moveMouse, typeText, getClipboardString, …). The same code drives any backend — write against Machine and the handles it returns, and only reach for machine.asAndroid() when an operation has no desktop counterpart.
• Calls into the native module are synchronous. No Promises are returned; Result::Err on the Rust side is translated to a thrown JS exception by napi-rs, so JS callers try/catch as usual.
• Many objects are handles to platform resources (windows, audio devices, accessibility trees, file/directory handles). Their lifetime matters; dropping them can free the underlying resource.
• Check canSee(), canUseAccessibility(), and canControl() before a run. Each returns null when this process can see the screen, use accessibility trees, or control other apps, and otherwise a reason that names what stands in the way and the fix. None of them opens a permission prompt.
• AskModel.askJson returns one JSON value and checks the reply locally against the schema you pass.
• Decisions are one DecisionModel.decide(state, questions) call. Name each question with noul, choice, or score. The same name is a property of the result: answers.urgent.probability, answers.dept.choice, answers.severity.nearest().
• searchRelative(candidates, landmarks, relation) is a free function (same as Rust search_relative). Keeps candidates whose box sits in a given position relative to at least one landmark — the fixed boxes you measure against. Not a tree walk and not scoped to a window or node. Both arguments are AccessibilityNode arrays from any windows or applications — find them with scoredSearch, a VLM + nodeAtPoint, find, or any other means. Boxes are compared in global-desktop coordinates. relation is (candidate, landmark) => boolean — compose BoundingBox predicates however you want (isBelow + overlapsX = under it, sharing its horizontal span; sameRow / sameColumn take a tolerance). Candidates with no box are omitted from the result; order is preserved.
• Coordinates live on the global desktop: top-left origin at (0, 0) on the primary monitor (on Android, the device screen), in OS-native units — the unit is not the same on every platform:
  • Windows / Linux / Android: physical pixels (raw hardware pixels).
  • macOS: logical points — on a 2× Retina display one point spans two hardware pixels, so a 1920×1080-logical screen is 1920×1080 here, not 3840×2160.