4. Evaluate a Segmentation Model

1. Context

A. Pre-requisite

A Projecton Picsellia allowing you to host Experiment and DatasetVersion.
A DatasetVersionconfigured in SEGMENTATION and annotated.
Have an Experimentwith a DatasetVersion attached to it - you can add the test alias to this DatasetVerison.
A Segmentation local model or a SegmentationModelVersion

📘
If you want to integrate your local custom Segmentation model into Picsellia you can checkout this tutorial 👉 Migrate your Models to Picsellia

B. Variables

Let's say that:

The Project is called Documentation Project
The Experimentis called my_experiment
The DatasetVersionattached is called test

C. Setup

You need to have a post-processing function that will return the class_name predicted and its confidence_threshold and polygons formatted in list of (x,y).

def predict(input: Image, model: TF.Model/PT.Model):
  	
    preprocessed_input = pre_process(input)
    prediction = model(preprocessed_input)
    
    class_name, confidence_treshold, denormalized_polygons = post_process(prediction) 
    return class_name, confidence_treshold, denormalized_polygons

You should also create a script that will initialize PicselliaClientconnection and fetch your Project,Experiment, DatasetVersion.

from picsellia import Client 

client = Client(api_token, organization_name, host='https://app.picsellia.com')

project = client.get_project(name='Documentation Project')
experiment = project.get_experiment(name='my_experiment')

testing_dataset = experiment.get_dataset('test')

We also need to create a dictionary matching class_names and the Label objects from Picsellia in order to attach the good Label. Something like that:

{
  "cat": PicselliaLabel(Python Object),
  "dog": PicselliaLabel(Python Object)
}

picsellia_labels_name = testing_dataset.list_labels()

label_matching = {k.name: k for k in picsellia_labels_name}

2. Implementing the Model Testing

Let's take a look at the Experiment add_evaluation() method:

add_evaluation(
   asset: Asset, add_type: Union[str,
   AddEvaluationType] = AddEvaluationType.REPLACE,
   rectangles: Optional[List[Tuple[int, int, int, int, Label, float]]] = None,
   polygons: Optional[List[Tuple[List[List[int]], Label, float]]] = None,
   classifications: Optional[List[Tuple[Label, float]]] = None
)

Let's dive into 3 of the arguments:

asset: Asset (Meaning that you can only have one evaluation by Asset)
add_type: It's an enum with these possibilities : (KEEP/REPLACE) the default is REPLACE. KEEP will keep the existing Evaluation if it exists.
polygons: it's a list of Tuple, the Tuple being ([[x1,y1], ..., [xn, yn]], Label, confidence_score)

Let's wrap everything together with a YOLOv8 segmentor from Ultralytics, here is the snippet from HuggingFace:

from ultralytics import YOLO

# Load a pretrained YOLOv8n model
model = YOLO('yolov8n-seg.pt')

# Run inference on an image
results = model('bus.jpg')  # results list

# View results
for r in results:
    print(r.masks)  # print the Boxes object containing the detection bounding boxes

Let's format this in order to integrate Picsellia into this:

import numpy as np
from picsellia import Client
from picsellia.sdk.asset import Asset
from picsellia.types.enums import InferenceType
from ultralytics import YOLO

client = Client(api_token="", organization_name="")
project = client.get_project(name='Documentation Project')
experiment = project.get_experiment(name='my_experiment')
testing_dataset = experiment.get_dataset('test')
picsellia_labels_name = dataset.list_labels()
label_matching = {k.name: k for k in picsellia_labels_name}


model = model = YOLO('yolov8n-seg.pt')

def postprocess(results):
    r = results[0].cpu()
    confs = r.boxes.conf.numpy().astype(np.float)
    polygons = list(map(lambda polygon: polygon.astype(int), r.masks.xy))
    classes_index = r.boxes.cls.numpy().astype(int)
    return classes_index, polygons, confs
      
for asset in dataset.list_assets():
    results = model(
    classes_idx, polygons, confs = postprocess(results)
    evaluated_polygons = []
    for idx, polygon, conf in list(zip(classes_idx, polygons, confs)):
        cls_name = model.names[idx]
        evaluated_polygons.append((polygon, label_matching[cls_name], conf))
    experiment.add_evaluation(asset, polygons=evaluated_polygons)
    
experiment.compute_evaluations_metrics(inference_type=InferenceType.SEGMENTATION)