multimodal

Using images with an agent

This tutorial shows you how to use images with an agent, allowing the agent to analyze and respond to image content.

::: zone pivot="programming-language-csharp"

Passing images to the agent

You can send images to an agent by creating a ChatMessage that includes both text and image content. The agent can then analyze the image and respond accordingly.

First, create an AIAgent that is able to analyze images.

AIAgent agent = new AIProjectClient(
    new Uri("<your-foundry-project-endpoint>"),
    new DefaultAzureCredential())
    .AsAIAgent(
        model: "gpt-4o",
        name: "VisionAgent",
        instructions: "You are a helpful agent that can analyze images");

[!WARNING] DefaultAzureCredential is convenient for development but requires careful consideration in production. In production, consider using a specific credential (e.g., ManagedIdentityCredential) to avoid latency issues, unintended credential probing, and potential security risks from fallback mechanisms.

Next, create a ChatMessage that contains both a text prompt and an image URL. Use TextContent for the text and UriContent for the image.

ChatMessage message = new(ChatRole.User, [
    new TextContent("What do you see in this image?"),
    new UriContent("https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", "image/jpeg")
]);

Run the agent with the message. You can use streaming to receive the response as it is generated.