fiftyone-dataset-inference

Run Model Inference on FiftyOne Datasets

Key Directives

ALWAYS follow these rules:

1. Check if dataset exists first

list_datasets()

If the dataset doesn't exist, use the fiftyone-dataset-import skill to load it first.

2. Set context before operations

set_context(dataset_name="my-dataset")

3. Launch App for inference

The App must be running to execute inference operators:

launch_app(dataset_name="my-dataset")

4. Ask user for field names

Always confirm with the user:

• Which model to use
• Label field name for predictions (e.g., predictions, detections, embeddings)

5. Close app when done

close_app()

Workflow

Step 1: Verify Dataset Exists

list_datasets()

If the dataset is not in the list:

• Ask the user for the data location
• Use the fiftyone-dataset-import skill to import the data first
• Return to this workflow after import completes

Step 2: Load Dataset and Review

set_context(dataset_name="my-dataset")
dataset_summary(name="my-dataset")

Review:

• Sample count
• Media type
• Existing label fields

Step 3: Launch App

launch_app(dataset_name="my-dataset")

Step 4: Discover and Apply Model

Ask the user about the task, model, or type of data they're using (detection, classification, segmentation, embeddings, or a specific model name); note users may give a 'tool name' (see Path B). Then determine the path:

Path A — Zoo model (most common)

ALWAYS first fetch the live model list — never assume what's available:

get_operator_schema(operator_uri="@voxel51/zoo/apply_zoo_model")

Pick the right model from the schema's model enum, then apply:

execute_operator(
    operator_uri="@voxel51/zoo/apply_zoo_model",
    params={
        "tab": "BUILTIN",
        "model": "<model-name-from-schema>",
        "label_field": "predictions"
    }
)

Path B — Plugin operator

If the user mentions a specific tool (e.g. CLIP similarity, SAM, a third-party model), check installed operators first:

list_operators(builtin_only=False)

Find the matching operator, inspect its schema, then execute it:

get_operator_schema(operator_uri="@org/plugin/operator")
execute_operator(operator_uri="@org/plugin/operator", params={...})