dotPilot: Agent for Claude Code

Using images with an agent

This tutorial shows you how to use images with an agent, allowing the agent to analyze and respond to image content.

Prerequisites

For prerequisites and installing NuGet packages, see the Create and run a simple agent step in this tutorial.

::: zone pivot="programming-language-csharp"

Passing images to the agent

You can send images to an agent by creating a ChatMessage that includes both text and image content. The agent can then analyze the image and respond accordingly.

First, create an AIAgent that is able to analyze images.

AIAgent agent = new AzureOpenAIClient(
    new Uri("https://<myresource>.openai.azure.com"),
    new AzureCliCredential())
    .GetChatClient("gpt-4o")
    .AsAIAgent(
        name: "VisionAgent",
        instructions: "You are a helpful agent that can analyze images");

Next, create a ChatMessage that contains both a text prompt and an image URL. Use TextContent for the text and UriContent for the image.

ChatMessage message = new(ChatRole.User, [
    new TextContent("What do you see in this image?"),
    new UriContent("https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", "image/jpeg")
]);

Run the agent with the message. You can use streaming to receive the response as it is generated.

Console.WriteLine(await agent.RunAsync(message));

This will print the agent's analysis of the image to the console.

::: zone-end ::: zone pivot="programming-language-python"

Passing images to the agent

You can send images to an agent by creating a ChatMessage that includes both text and image content. The agent can then analyze the image and respond accordingly.

First, create an agent that is able to analyze images.

import asyncio
from agent_framework.azure import AzureOpenAIChatClient
from azure.identity import AzureCliCredential

agent = AzureOpenAIChatClient(credential=AzureCliCredential()).as_agent(
    name="VisionAgent",
    instructions="You are a helpful agent that can analyze images"
)