Deployments - Prediction Review tool

Picsellia embeds an integrated Prediction Review tool that will allow you to review any Prediction done by a ModelVersion and logged in a Deployment.

The main idea while reviewing a Prediction is to edit its Shape if the human considers this one is not correct. In other words, bring the GroundTruth against the Prediction done by the ModelVersion.

So basically, if the human considers that the Prediction done by the ModelVersion is not fully correct, it means:

• Edit if needed, one or several Shape (Localization, Position, or Label) created by the ModelVersion in the Prediction
• Delete if needed one or several Shape that shouldn't exist
• Create one or several Shape missed by the ModelVersion

This Prediction Review tool is very similar to the Annotation tool. it is compatible with all Detection Types:

• Object Detection
• Classification
• Segmentation
• Line Detection
• Key Point Detection

The Detection Type is automatically set up by inheritance of the Detection Type defined with the deployed ModelVersion.

1. Prediction review process

From any Deployment, in the Prediction tab, you have access through the different views to the PredictedAsset and associated Prediction logged. Each Prediction has a Metadata named Status, this Status reflects the stage of the Prediction in the review process:

• TO REVIEW: The Prediction has not been reviewed by a human yet
• REVIEWED: The Prediction has been reviewed by a human
• SUBMITTED: The Prediction has been reviewed by a human and submitted to the Pipeline through the Feedback Loop

Please note that this process is linear and not retroactive. This means that any Prediction logged in a Deployment is created with the status TO REVIEW, once reviewed it goes in REVIEWED status, and once pushed to the Pipeline it goes in SUBMITTED status. But there is no way to go backward.

2. Simple Review or Prediction Review Campaign

As it is the case for DatasetVersion, you can define and orchestrate modular and personalizable workflows to review your PredictedAsset. This is useful in case you have many PredictedAsset to review with many users involved in this process.

This feature allows to brings traceability, organization, and work progress tracking in your review process.

An exhaustive documentation on this feature is available here.

However, in case you want to straightly forward review the PredictedAsset of your Deployment, you can still use the regular Predictino Review tool, its usage is detailed below.

3. Access the Prediction Review tool

To review a Prediction with TO REVIEW status, you must access the Prediction Review tool.

It can be done from the Details and Grid views on any Deployment through the pen icon when hovering on any PredictedAsset:

When the Prediction Review tool opens, you can see the PredcitedAsset and the Prediction displayed on it.

In the header, you'll retrieve the Deployment name, the PredictedAsset filename. Below is displayed the Detection Type and the position of the current PredictedAsset in the queue:

📘

Visualize Shadow Prediction

In case a Shadow ModelVersion is deployed in addition of the Champion one, it is always the Champion Prediction that is displayed by default but can access the Shadow Prediction as shown below:

The Prediction Review tool allows to visualize the Prediction done by a ModelVersion on a PredictedAsset, each Prediction is composed of one or several Shape (i.e. Classification, Bounding-box, Polygon...). Each Shape is displayed with its associated Label and Confidence score as show below:

To ease the visualization, you can also decide to display only the Shape with a Confidence score above a given threshold. Please note that Shape with Confidence score under this threshold are not deleted, only hidden on the current view.

👍

Where can I retrieve the list of the Shape currently displayed ?

On the right side of the Prediction Review tool, in the Shapes panel, is displayed the list of all the Shape composing the Prediction that are currently displayed. They are grouped by Label.

While hovering over the displayed image in the Prediction Review tool, you can be in two modes:

• Selection to select and edit existing Shape, this mode is the default one when opening the tool (keyboard shortcut: Esc)
• Draw to create a new Shape (keyboard shortcut: D)

4. Navigation

A. Across the current PredictedAsset

The Prediction Review tool offers smooth navigation across the currently displayed PredictedAsset. You can use the scroll wheel of your mouse or trackpad to zoom in/out and press the spacebar to drag and move around on the PredictedAsset.

The initial view of the PredcitedAsset can be retrieved through the Reset Zoom button:

B. Across the PredcitedAsset of the Deployment

At the bottom of the Prediction Review tool, you can open by cliking on Predictions the gallery to visualize all the PredcitedAsset of your Deplomyent with their Review Status. You can click on any of them to open it in the Prediction Review tool.

5. Review a Prediction

As already mentioned, reviewing a Prediction lies in:

• Edit if needed, one or several Shape (Localization, Position, or Label) created by the ModelVersion in the Prediction
• Delete if needed one or several Shape that shouldn't exist
• Create one or several Shape missed by the ModelVersion

A. Edit a predicted Shape

Let's start by editing a Shape that was not properly predicted by the ModelVersion.

First of all, you need to select the to-be-modified Shape. It can be done:

• Directly in Selection mode by clicking on the Shape directly on the PredcitedAsset.
• By clicking on a given Shape in the Labels panel located on the right side.
• By clicking on the circle next to a given Shape in the Shapes panel located on the right side. This will automatically zoom in on the selected Shape.

In all cases, the selected Shape will be highlighted, with all the potential points composing the Shape. Additionally, the selected Shape will also be indicated in the Shapes list with a double circle:

You can change the Label of the selected Shape by:

• Right-clicking on the selected Shape on the PredictedAsset

Selecting the new Label in the dropdown located at the top-left corner

When a Shape is selected, all the points composing this Shape are displayed. With your mouse, you can then click on those points to modify the Shape dimensions. You can also move the Shape anywhere in the frame of the PredictedAsset.

👍

Focus on a Shape

To directly zoom in the selected Shape, you can click on Focus in the right-click menu.

📘

It is not applicable to Classification

Obviously, the modification of dimensions and position of a classification Shape is not possible.

B. Delete a predicted Shape

You can also delete any Shape on an Prediction. Once the Shape is selected, it can be done in several ways as soon as the Shape is selected:

• Press the delete key
• Click on the trash icon

• Right-click on the Shape in the Shapes list or on the image and click on Delete

C. Draw a new Shape

To draw a new Shape in the current Prediction, you just need to select the drawing mode (keyboard shortcut: D), select the Label of the new Shape, and draw it. For instance, let's create a person Shape:

You can abort the drawing of a Shape while doing it using the Esc key.

When the Shape is created, a new line is added to the Shapes list on the right side of the Prediction Review tool, and you can visualize the Shape and its Label on the PredcitedAsset.

D. Save the Review

Once you consider that the Prediction is reviewed, meaning it corresponds to the GroundTruth, you can save the Review by clicking on the Review prediction button or by using the keyboard shortcut Ctrl/Cmd + S.

As a reminder, this operation cannot be undone, so it is highly recommended to be 100% sure of the Review before clicking on Review Prediction .

🚧

Unsaved modification on Shape will be lost

In this case, saving modification on Shape composing a Prediction means cliking on Review prediction

Once reviewed, the Status of the Prediction turns from TO REVIEW to REVIEWED.

The Shape on a Prediction are locked for modification as soon as this one turns in REVIEWED status, it is obviously also the case for Prediction in SUBMITTED status.

The Review is also used against the Prediction to compute Supervised Metrics in the Dashboard as detailed here.

However, once reviewed, even if it is not possible to edit it, it is still possible, as shown below, to visualize the initial Prediction done by the ModelVersion or the Review done by a human on the Prediction:

6. Detection Types

Depending on the Detection Type of the ModelVersion, the Shape composing a Prediction can either be Classification, Bounding-boxes, Polygon, Key Point or Line.

Obviously, the usage of the Prediction Review tool will then differ. For further details on the use of the tool according to the Detection Type, you can refer to the documentation of the Annotation tool as the behavior is similar for Shape creation, edition, or deletion.

7. Addition features

A. Lock and Hide Shape

To facilitate the review process, you can either lock or hide a Shape.

Obviously, a hidden Shape is not deleted, it is only not displayed. Locking a Shape prevents anyone from editing it. To unlock or make a Shape visible, you just need to click again on the Hide or Lock icon.

📘

From the right-click menu

You can lock or hide a Shape from the action menu displyed by right-cliking on the Shape directly on the image or in the Shapes list.

You can also lock or hide all the Shape of an Prediction with the same Label, as the Shape are all grouped together in the Shapes list on the right side.

🔦

You can hide the Labeland associated Confidence Score on the Shape by unchecking the tickbox Show prediction labels.

B. Group Selection

For practical reasons, the Picsellia Prediction Review tool offers the possibility to select several Shape at a time and make modifications to a group of Shape.

Multiple Shape selection can be done:

• On the PredcitedAsset by drawing within the Selection mode a rectangle over the to-be-selected Shape,the selected bunch of Shape will be highlighted in blue in the Shapes list.

• By selecting all the Shape with a specific Label directly from the Shape list, as Shape with the same Label are grouped together. So selecting the Label will automatically select all the Shape with this Label, for instance, head:

  

Once a group of Shape is selected, you can:

• Change the dimension and the position of these Shape
• Change the Label of all the selected Shape with the right-click or the Labels dropdown (located in the top-left corner)
• Delete the selected Shape with the delete key or the trash icon

C. Keyboard shortcut

To make the review process as smooth and efficient as possible on Picsellia, most of the actions related to the Prediction Review tool can be done using buttons on the tool but also through keyboard shortcuts.

The list of available shortcuts can be displayed through the keyboard icon:

You can also define your preferences in the Annotation Tool Settings page: