AGENT_PATTERNS

OmniLink agent patterns

Cross-demo design patterns extracted from the OmniSim agent builds (husky_maze, mission_captain, warehouse_foreman, warehouse_patrol). These are the choices that paid off — each one earned its place by being the lesson learned from a specific failure mode documented in that demo's docs/RESULTS.md.

On the warehouse_* citations below. The Warehouse Foreman, Warehouse Picker and Patrol Squad demos were removed from the tree on 2026-07-19 (their results predated the b18bd7a3 teleport-recovery removal and were never re-measured). The citations are kept because they are the provenance of these patterns — each names the run that produced the lesson. They are historical references, not paths you can open; recover them from git history if you need the detail. The patterns themselves are live and apply to every new agent.

This is the doc to read before starting a new agent. The default for every new demo should be: do the thing this doc recommends; if you need to deviate, write down why in the demo's docs.

---

1. Perception-as-tool, not pixels-to-LLM

Pattern. Vision belongs in a sidecar process that runs deterministic image analysis and exposes a structured-tag API. The agent calls a tool, gets back JSON like {tag_color: "green", marker_fraction: 0.0038, marker_centroid: {x_norm: 0.49, y_norm: 0.62}}. The agent NEVER sees pixels in the steady-state path.

Why. A 320×240 PNG attached as image_url part costs ~12 k input tokens. A structured tag digest is ~80 tokens. 150× per-query difference. Plus: deterministic across runs (the LLM may misread on any given snap), preserves cacheability (the per-turn input doesn't drown out the cacheable system prefix), and faster (no LLM-pixel-decoding latency).

How.

• Eye sidecar process (e.g. husky_eye) owns the OmniSim Camera. Pure-Python BGRA classifier returns {tag_color, marker_pixels, marker_fraction, marker_centroid, color_fractions, ...} per camera view.
• Bridge proxies sidecar /scan so the agent only knows about one HTTP target.
• Agent gets a scan_for_tag (or scan_surroundings etc.) tool with description like "PREFERRED tag-identification tool — never sends pixels to you. ~150× cheaper than read_camera."
• Keep read_camera as an explicit FALLBACK tool for genuinely ambiguous frames. Discriminator argument still holds (the agent CAN read pixels when the structured digest is uncertain), the cost story still holds (agents prefer the cheap path).