general-plot-digitizer

General Plot Digitizer

Goal

Extract calibrated numeric X-Y data from images of experimental spectra (Raman, XRD, UV-Vis, IR, NMR, etc.) using a deterministic "Agent-in-the-Loop" workflow.

The labor is divided between two models:

1. Vision-Language Model (Visual Sensor): Reads the image and returns a rich, unstructured narrative description of axes, colors, and visual obstacles. It does not produce JSON.
2. Coding Agent (Translator & Executor): Translates the VLM narrative into a precise metadata.json, runs the CV pipeline, inspects the overlay, and iterates until the curve is correctly isolated.

Instructions

Phase 1: Visual Inspection (VLM)

Do not attempt to generate JSON with the VLM. It acts only as a visual sensor.

1. Generate grid overlay:

# Env: base-agent
python .agents/skills/general-plot-digitizer/scripts/plot_utils.py plot.png --draw-grid

This produces plot_grid.png with a labeled pixel grid for precise coordinate reading.

1. Prompt the VLM to analyze plot_grid.png (not the raw image). Use the built-in vision capabilities or the notify_user VLM inspection tool. Provide the prompt guidelines from scripts/vlm_prompt_template.txt.

2. Expected VLM output — a natural-language report covering:

   • Axis labels, numeric ranges, and directions (is X reversed?).
   • Bounding box of the data region in pixels (read from the grid).
   • Color and style of each target curve (hex guess from the pixels on the line itself).
   • Pixel bounding boxes of all visual obstacles (legends, text annotations, gridlines, tick marks) that overlap the data curves.
   • Trace quality hints: thin/needle-like, thick/noisy, anti-aliased, JPEG artifacts.

Phase 2: Metadata Construction (Coding Agent)

Read the VLM narrative and construct metadata.json. Schema: resources/metadata_schema.json.

Required fields:

{
  "plot_title": "",
  "x_axis_label": "Wavelength (nm)",
  "y_axis_label": "Absorbance",
  "x_tick_min": 400, "x_tick_max": 800,
  "y_tick_min": 0, "y_tick_max": 1,
  "x_calibration_points": [
    { "pixel": 70, "value": 400 },
    { "pixel": 450, "value": 800 }
  ],
  "x_scale": "linear", "y_scale": "linear",
  "bounding_box": {"x_min": 72, "y_min": 28, "x_max": 452, "y_max": 318},
  "x_reversed": false, "y_reversed": false,
  "spectrum_type": "UV-Vis",
  "curves": [{"label": "sample", "color_hint": "#1f77b4"}],
  "text_regions": [{"x_min": 300, "y_min": 50, "x_max": 400, "y_max": 80, "label": "legend"}]
}