4. Evaluate a Segmentation Model

1. Context

A. Pre-requisite

  • A Projecton Picsellia allowing you to host Experiment and DatasetVersion.
  • A DatasetVersionconfigured in SEGMENTATION and annotated.
  • Have an Experimentwith a DatasetVersion attached to it - you can add the test alias to this DatasetVerison.
  • A Segmentation local model or a SegmentationModelVersion


If you want to integrate your local custom Segmentation model into Picsellia you can checkout this tutorial ๐Ÿ‘‰ Migrate your Models to Picsellia

B. Variables

Let's say that:

  • The Project is called Documentation Project
  • The Experimentis called my_experiment
  • The DatasetVersionattached is called test

C. Setup

You need to have a post-processing function that will return the class_name predicted and its confidence_treshold and boxes formatted in denormalized x, y, w, h.

def predict(input: Image, model: TF.Model/PT.Model):
    preprocessed_input = pre_process(input)
    prediction = model(preprocessed_input)
    class, conf, denormalized_polygons = post_process(prediction) 
    return class_name, confidence_treshold, denormalized_polygons

You should also create a script that will initialize PicselliaClientconnexion and fetch your Project,Experiment, DatasetVersion.

from picsellia import Client 

client = Client(api_token, organization_name, host='https://app.picsellia.com')

project = client.get_project(name='Documentation Project')
experiment = project.get_experiment(name='my_experiment')

testing_dataset = experiment.get_dataset('test')

We also need to create a dictionary matching class_names and the Label objects from Picsellia in order to attach the good Label. Something like that:

  "cat": PicselliaLabel(Python Object),
  "dog": PicselliaLabel(Python Object)
picsellia_labels_name = testing_dataset.list_labels()

label_matching = {k.name: k for k in picsellia_labels_name}

2. Implementing the Model Testing

Let's take a look at the Experiment add_evaluation() method:

   asset: Asset, add_type: Union[str,
   AddEvaluationType] = AddEvaluationType.REPLACE,
   rectangles: Optional[List[Tuple[int, int, int, int, Label, float]]] = None,
   polygons: Optional[List[Tuple[List[List[int]], Label, float]]] = None,
   classifications: Optional[List[Tuple[Label, float]]] = None

Let's dive into 3 of the arguments:

  • asset: Asset (Meaning that you can only have one evaluation by Asset)
  • add_type: It's an enum with these possibilities : (KEEP/REPLACE) the default is REPLACE. KEEP will keep the existing Evaluation if it exists.
  • polygons: it's a list of list of Tuple, the Tuple being ([[x, y],[x1,y1], ..., [xn, yn]]Label, confidence_score)

Let's wrap everything together with a YOLOv8 segmentor from Ultralytics, here is the snippet from HuggingFace:

from ultralytics import YOLO

# Load a pretrained YOLOv8n model
model = YOLO('yolov8n-seg.pt')

# Run inference on an image
results = model('bus.jpg')  # results list

# View results
for r in results:
    print(r.masks)  # print the Boxes object containing the detection bounding boxes

Let's format this in order to integrate Picsellia into this:

import numpy as np
from picsellia import Client
from picsellia.sdk.asset import Asset
from picsellia.types.enums import InferenceType
from ultralytics import YOLO

client = Client(api_token="", organization_name="")
project = client.get_project(name='Documentation Project')
experiment = project.get_experiment(name='my_experiment')
testing_dataset = experiment.get_dataset('test')
picsellia_labels_name = dataset.list_labels()
label_matching = {k.name: k for k in picsellia_labels_name}

model = model = YOLO('yolov8n-seg.pt')

def postprocess(results):
    r = results[0].cpu()
    confs = r.boxes.conf.numpy().astype(np.float)
    masks = list(map(lambda polygon: polygon.astype(int), r.masks.xy))
    classes_index = r.boxes.cls.numpy().astype(int)
    return classes_index, masks, confs
for asset in dataset.list_assets():
    results = model(
    classes_idx, boxes, confs = postprocess(results)
    evaluated_polygons = []
    for idx, mask, conf in list(zip(classes_idx, boxes, confs)):
        cls_name = model.names[idx]
        evaluated_polygons.append((label_matching[cls_name], box, conf))
    experiment.add_evaluation(asset, polygons=evaluated_polygons)