Evaluation

The purpose of creating a Project and performing several Experiment in its frame is basically to train the best-performing ModelVersion. To do so, we saw on the previous page how to log Metrics in the Experiment Tracking dashboard, giving you the insights needed to assess the quality of the training performed. In addition to the Metrics, Picsellia also lets you compare, on a dedicated bench of images, the predictions made by the freshly trained model with the ground truth — this is called an Evaluation.

By leveraging the Evaluation interface, you can get a precise overview of the model's behavior, get performance metrics on each evaluated image, and easily identify the contexts in which the model performs well or not.

With Metrics and Evaluation, you'll have all the insights needed to assess the quality of the trained ModelVersion, understand why it behaves the way it does, and identify ways to improve its performance in a further Experiment.

1. Create an Evaluation

The creation of Evaluation is done by your training script. It needs to get a bunch of images unused during the training phase and already annotated, in order to use them as ground truth.

This bunch of images can come from the training script splitting the DatasetVersion attached to the Experiment, or from a DatasetVersion dedicated to Evaluation and attached to the Experiment with an alias indicating its purpose.

The script then makes the just-trained ModelVersion infer on the evaluation images and sends the result to the Picsellia Experiment.

The outcome is that, for each evaluation image, you can compare the prediction made by the ModelVersion with the ground truth pulled from the original DatasetVersion. You'll also have access to evaluation metrics computed by Picsellia based on the comparison of predictions and ground truth.

The creation of Evaluation is already integrated into almost all the training scripts attached to ModelVersion from the Public Registry.

However, if you are using your own training script integrated with Picsellia, you can use this dedicated tutorial, detailing how to create and log Evaluation within your Picsellia Experiment.

2. Evaluation interface

From an Experiment, you can access the Evaluations tab, as shown below:

After Experiment creation, the Evaluation interface is empty. It is only once the training script has been executed successfully that Evaluation are created in the Evaluation interface.

The Evaluation interface offers a way to visualize and explore Evaluation with the same philosophy and features as the Datalake to browse among data, or a DatasetVersion to browse among Asset.

A. Evaluation visualization

a. Evaluation views

You'll find three different views:

• Grid view, letting you visualize many Evaluation at a time.
• Table view, letting you quickly visualize many Evaluation and their associated information (filename, width, height, detection performances).
• Details view, letting you visualize one Evaluation in detail (full-screen image display and associated information).

Depending on the view, you can still modify the aspect ratio (for Grid and Details views) and the size of the thumbnail displayed (for Grid view only).

b. Evaluation Shape

For each Evaluation, you can visualize the image, as well as the superposition of the prediction made by the trained ModelVersion and the ground truth inherited from the DatasetVersion.

The ground truth Shape are displayed in green, whereas the predicted Shape are red. For each Shape, the associated Label is displayed along with the confidence score in the case of predictions.

To ensure the smoothest visualization possible, you can select which elements to display on your image.

You can choose to display, on your Evaluation image, only the ground truth, only the predictions, or both. You can also filter the Shape to display by Label or by confidence score (in the case of predictions).

c. Evaluation metrics

As explained previously, for each Evaluation, Picsellia compares ground truth and prediction to compute different metrics, allowing you to assess the quality of the prediction made by the ModelVersion on that particular image.

Those metrics can be visualized in the Table and Details views, as shown below:

For each Evaluation, in addition to the filename, asset ID, and creation date, you can access:

• The number of ground truth Shape vs. the number of predicted Shape.
• Average precision score for different IoUs.
• Average recall score for different IoUs.

B. Evaluation ordering

To access ordering or filtering features, you can open the related toolbar by clicking on the Filters button, as shown below:

Evaluation can be ordered ascending or descending by filename or evaluation date.

You can also decide to randomize the display of Evaluation.

C. Evaluation filtering

To let you investigate the Evaluation generated and understand the ModelVersion's behavior in depth, Evaluation can be filtered on several criteria.

a. Date

The date picker lets you filter Evaluation on their creation date by clicking on the Select Date button. The date picker will open, letting you select the timeframe to search on.

Once the date range is selected, the associated query is filled in the search bar, letting you complete it directly through the search bar for a more specific query.

b. Metrics

The Evaluation interface also lets you filter on the computed metrics. To do so, click on + Add filter and define, through the modal, the metric and the values to filter on. For instance, let's create a filter that displays only the Evaluation created between 09/19/2023 and 09/22/2023, with an Average Precision (IoU 50-95) between 0.291 and 0.782:

Once applied, the list of active filters (based on date or computed metric) is displayed as shown below:

Any filter can be removed at any time by clicking on it, then on the trash icon.

c. Filter Evaluations by AssetAttribute and/or ShapeAttribute

The + Add filter modal also lets you filter Evaluation on the Attribute values inherited from the ground truth Asset and carried by its Shape, as well as on the Shape themselves, on their type and their Label. This is a direct way to check the ModelVersion behavior on a specific slice of the evaluation bench, for instance only the images whose Asset Attribute marks a difficult acquisition condition. Attribute filtering is detailed here.

D. Search Bar

As with all the image visualization views available on Picsellia, you have access to the search bar powered by our Query Language. This search bar lets you create complex queries leveraging all the properties related to each Evaluation.

The Query Language lets you browse among properties of the image itself (linked to an Asset), of the ground truth Annotation, or of the prediction Annotation. You can rely on auto-completion to see all the properties you can search on.

E. Evaluation deletion

To delete any computed Evaluation, select the Evaluation to delete and click on Delete.

F. Embeddings and visual search

Embeddings can also be computed for the Evaluation of an Experiment, the same way as for the Data of a Datalake (detailed here). On an evaluation bench of a few thousand images, this is how you find the images that resemble the one the ModelVersion failed on, instead of scrolling until you meet them.

The computation is activated from the Data Embeddings page of the Experiment Settings, where you can also follow its progress, Retry the Evaluation that could not be computed, and Deactivate the computation, which deletes the embeddings already computed.

Once the computation is over, you can look for the Evaluation that resemble a selected one, or that match a text prompt.

3. Classification

The case of Classification is handled a bit differently than Object Detection or Segmentation in the Evaluation interface. This is mainly because average recall and average precision metrics don't make sense for Classification Experiment.

The creation of Evaluation for Classification by a training script is detailed here.

Most of the features remain the same as detailed previously.

The visualization of ground truth and prediction on each Evaluation follows the same guideline, with the ground truth Label in green and the predicted Label (and associated confidence score) in red.

The other major difference lies in the computed metrics. For Classification Evaluation, it is the confidence score (referred to as Score in the Evaluation interface) of the prediction that is displayed and usable to create filters.

The ground truth and predicted Label are also displayed as fields in the Table and Details views.

To quickly access the Evaluation where the ground truth Label differs from the predicted one, click on the Show class errors button, available once the Filters bar has been opened.

For instance, in the screenshot above, using the Show class errors button shows only the Evaluation where the ground truth Label differs from the predicted one — in this case, 53 Evaluation out of 279 performed.