4. Evaluate a Segmentation Model
1. Context
A. Pre-requisite
- A
Project
on Picsellia allowing you to host Experiment and DatasetVersion. - A
DatasetVersion
configured in SEGMENTATION and annotated. - Have an
Experiment
with aDatasetVersion
attached to it - you can add the test alias to thisDatasetVerison
. - A Segmentation local model or a Segmentation
ModelVersion
If you want to integrate your local custom Segmentation model into Picsellia you can checkout this tutorial 👉 Migrate your Models to Picsellia
B. Variables
Let's say that:
- The
Project
is called Documentation Project - The
Experiment
is called my_experiment - The
DatasetVersion
attached is called test
C. Setup
You need to have a post-processing function that will return the class_name predicted and its confidence_threshold and polygons formatted in list of (x,y).
def predict(input: Image, model: TF.Model/PT.Model):
preprocessed_input = pre_process(input)
prediction = model(preprocessed_input)
class_name, confidence_treshold, denormalized_polygons = post_process(prediction)
return class_name, confidence_treshold, denormalized_polygons
You should also create a script that will initialize PicselliaClient
connection and fetch your Project
,Experiment
, DatasetVersion
.
from picsellia import Client
client = Client(api_token, organization_name, host='https://app.picsellia.com')
project = client.get_project(name='Documentation Project')
experiment = project.get_experiment(name='my_experiment')
testing_dataset = experiment.get_dataset('test')
We also need to create a dictionary matching class_names and the Label objects from Picsellia in order to attach the good Label
. Something like that:
{
"cat": PicselliaLabel(Python Object),
"dog": PicselliaLabel(Python Object)
}
picsellia_labels_name = testing_dataset.list_labels()
label_matching = {k.name: k for k in picsellia_labels_name}
2. Implementing the Model Testing
Let's take a look at the Experiment add_evaluation()
method:
add_evaluation(
asset: Asset, add_type: Union[str,
AddEvaluationType] = AddEvaluationType.REPLACE,
rectangles: Optional[List[Tuple[int, int, int, int, Label, float]]] = None,
polygons: Optional[List[Tuple[List[List[int]], Label, float]]] = None,
classifications: Optional[List[Tuple[Label, float]]] = None
)
Let's dive into 3 of the arguments:
- asset: Asset (Meaning that you can only have one evaluation by
Asset
) - add_type: It's an enum with these possibilities : (KEEP/REPLACE) the default is REPLACE. KEEP will keep the existing
Evaluation
if it exists. - polygons: it's a list of Tuple, the Tuple being ([[x1,y1], ..., [xn, yn]],
Label
, confidence_score)
Let's wrap everything together with a YOLOv8 segmentor from Ultralytics, here is the snippet from HuggingFace:
from ultralytics import YOLO
# Load a pretrained YOLOv8n model
model = YOLO('yolov8n-seg.pt')
# Run inference on an image
results = model('bus.jpg') # results list
# View results
for r in results:
print(r.masks) # print the Boxes object containing the detection bounding boxes
Let's format this in order to integrate Picsellia into this:
import numpy as np
from picsellia import Client
from picsellia.sdk.asset import Asset
from picsellia.types.enums import InferenceType
from ultralytics import YOLO
client = Client(api_token="", organization_name="")
project = client.get_project(name='Documentation Project')
experiment = project.get_experiment(name='my_experiment')
testing_dataset = experiment.get_dataset('test')
picsellia_labels_name = dataset.list_labels()
label_matching = {k.name: k for k in picsellia_labels_name}
model = model = YOLO('yolov8n-seg.pt')
def postprocess(results):
r = results[0].cpu()
confs = r.boxes.conf.numpy().astype(np.float)
polygons = list(map(lambda polygon: polygon.astype(int), r.masks.xy))
classes_index = r.boxes.cls.numpy().astype(int)
return classes_index, polygons, confs
for asset in dataset.list_assets():
results = model(
classes_idx, polygons, confs = postprocess(results)
evaluated_polygons = []
for idx, polygon, conf in list(zip(classes_idx, polygons, confs)):
cls_name = model.names[idx]
evaluated_polygons.append((polygon, label_matching[cls_name], conf))
experiment.add_evaluation(asset, polygons=evaluated_polygons)
experiment.compute_evaluations_metrics(inference_type=InferenceType.SEGMENTATION)
Updated 11 months ago