vision

Vision Input Agents (Proposed — Not Yet Implemented)

This is a design proposal, not a shipped feature. Nothing described below exists in the codebase yet — this file exists to visualize the interface before writing any code. Do not follow these examples expecting them to work.

ActiveHarness could let agents accept images as input alongside a text prompt — "look at this and tell me X" — reusing the model chain, hooks, retry/fallback, and system_prompt machinery that already exists for text agents.

Proposed interface

class ReceiptAgent < ActiveHarness::Agent
  vision true

  system_prompt "You are an expenses auditor. Flag anything that looks like a duplicate or a personal (non-business) purchase."

  model do
    use      provider: :openai,    model: "gpt-4o-mini"
    fallback provider: :anthropic, model: "claude-haiku-4-5-20251001"
  end
end

result = ReceiptAgent.call(
  input: "Does this receipt look legitimate?",
  image: "/path/to/receipt.jpg"
)

result.output # => "This looks like a standard restaurant receipt. Nothing suspicious — itemized total matches the sum."

• vision true — class-level flag, same shape as image true/transcribe true. Validates that every model in the chain has "vision" in its Pricing categories (this category already exists in the registry today — it's derived from a model's input modalities, the mirror image of "imggen", which is derived from output modalities. No changes needed on the pricing side to support this validation).
• image: — a new keyword at the call site, alongside the existing input:, context:, params:, memory:, models:, token:, stream:. Accepts:
  • a single local file path ("/path/to/photo.jpg")
  • an array of paths for multiple images in one message: image: ["front.jpg", "back.jpg"]
  • a value that already looks like an http(s):// URL is passed straight through as a URL reference instead of being read and base64-encoded — saves bandwidth when the image is already hosted somewhere.
• @input stays exactly what it already is for a normal agent: the text part of the prompt. This is the main difference from image true/transcribe true, where @input had to be repurposed (image prompt text, or an audio file path) because those are single-purpose, non-chat endpoints. Vision is a chat call with a richer message body, so @input keeps its normal meaning.