Whether you framed the Review process with Prediction Review Workflow or not, you will use the Prediction Review Tool to review the Prediction done on your PredictedAsset.
1. Access the Prediction Review tool
To review a Prediction with TO REVIEW status, you must access the Prediction Review tool.
A. In the case of Simple Review
It can be done from the Details and Grid views on any Deployment through the pen icon when hovering on any PredictedAsset:
Open Prediction Review tool
B. In the case of Prediction Review Campaign
It should be done from Tasks on any Deployment that has an ongoing Prediction Review Campaign. For each PredictedAsset that Prediction Review Tool can ben opened by clicking on Go to Task.
Open the Prediction Review Tool from
2. Visualization
When the Prediction Review Tool opens, you can see the PredcitedAsset and the Prediction displayed on it.
In the header, you'll retrieve the Deployment name, the PredictedAsset filename. Below is displayed the Detection Type and the position of the current PredictedAsset in the queue:
Prediction Review tool
📘
Visualize Shadow Prediction
In case a ShadowModelVersion is deployed in addition of the Champion one, it is always the Champion Prediction that is displayed by default but can access the Shadow Prediction as shown below:
Visualization of Champion & Shadow Predictions
The Prediction Review tool allows to visualize the Prediction done by a ModelVersion on a PredictedAsset, each Prediction is composed of one or several Shape(i.e. Classification, Bounding-box, Polygon...). Each Shape is displayed with its associated Label and Confidence score as show below:
A Shape composing a Prediction in Object Detection mode
To ease the visualization, you can also decide to display only the Shape with a Confidence score above a given threshold. Please note that Shape with Confidence score under this threshold are not deleted, only hidden on the current view.
Confidence Threshold for Shapedisplay
👍
Where can I retrieve the list of the Shape currently displayed ?
On the right side of the Prediction Review tool, in the Shapes panel, is displayed the list of all the Shape composing the Prediction that are currently displayed. They are grouped by Label.
While hovering over the displayed image in the Prediction Review tool, you can be in two modes:
Selection to select and edit existing Shape, this mode is the default one when opening the tool (keyboard shortcut: Esc)
Draw to create a new Shape (keyboard shortcut: D)
4. Navigation
A. Across the current PredictedAsset
The Prediction Review tool offers smooth navigation across the currently displayed PredictedAsset. You can use the scroll wheel of your mouse or trackpad to zoom in/out and press the spacebar to drag and move around on the PredictedAsset.
The initial view of the PredcitedAsset can be retrieved through the Reset Zoom button:
B. Across the PredcitedAsset of the Deployment
Reset Zoom
At the bottom of the Prediction Review tool, you can open by cliking on Predictions the gallery to visualize all the PredcitedAsset of your Deplomyent with their Review Status. You can click on any of them to open it in the Prediction Review tool.
PredictedAsset gallery
5. Review a Prediction
As already mentioned, reviewing a Prediction lies in:
Edit if needed, one or several Shape (Localization, Position, or Label) created by the ModelVersion in the Prediction
Delete if needed one or several Shape that shouldn't exist
Create one or several Shape missed by the ModelVersion
This review process can be framed by a Prediction Review Campaign or done straightly forward.
A. Edit a predicted Shape
Let's start by editing a Shape that was not properly predicted by the ModelVersion.
First of all, you need to select the to-be-modified Shape. It can be done:
Directly in Selection mode by clicking on the Shape directly on the PredcitedAsset.
By clicking on a given Shape in the Labels panel located on the right side.
By clicking on the circle next to a given Shape in the Shapes panel located on the right side. This will automatically zoom in on the selected Shape.
Shapeselection
In all cases, the selected Shape will be highlighted, with all the potential points composing the Shape. Additionally, the selected Shape will also be indicated in the Shapes list with a double circle:
Shapeselected
You can change the Label of the selected Shape by:
Right-clicking on the selected Shape on the PredictedAsset
Change Label
Selecting the new Label in the dropdown located at the top-left corner
Change Label
When a Shape is selected, all the points composing this Shape are displayed. With your mouse, you can then click on those points to modify the Shape dimensions. You can also move the Shape anywhere in the frame of the PredictedAsset.
👍
Focus on a Shape
To directly zoom in the selected Shape, you can click on Focus in the right-click menu.
Focus button
📘
It is not applicable to Classification
Obviously, the modification of dimensions and position of a classification Shape is not possible.
B. Delete a predicted Shape
You can also delete any Shape on an Prediction. Once the Shape is selected, it can be done in several ways as soon as the Shape is selected:
Press the delete key
Click on the trash icon
Delete Shape
Right-click on the Shape in the Shapes list or on the image and click on Delete
Delete Shape
C. Draw a new Shape
To draw a new Shape in the current Prediction, you just need to select the drawing mode (keyboard shortcut: D), select the Label of the new Shape, and draw it. For instance, let's create a personShape:
Draw a Shape
Draw a Shape
You can abort the drawing of a Shape while doing it using the Esc key.
When the Shape is created, a new line is added to the Shapes list on the right side of the Prediction Review tool, and you can visualize the Shape and its Label on the PredcitedAsset.
Visualize a Shape
D. Save the Review
Once you consider that the Prediction is reviewed, meaning it corresponds to the GroundTruth, you can save the Review by clicking on the Review prediction button or by using the keyboard shortcut Ctrl/Cmd + S.
As a reminder, this operation cannot be undone, so it is highly recommended to be 100% sure of the Review before clicking on Review Prediction .
Save Review
🚧
Unsaved modification on Shape will be lost
In this case, saving modification on Shape composing a Prediction means cliking on Review prediction
Once reviewed, the Status of the Prediction turns from TO REVIEW to REVIEWED.
The Shape on a Prediction are locked for modification as soon as this one turns in REVIEWED status, it is obviously also the case for Prediction in SUBMITTED status.
The Review is also used against the Prediction to compute Supervised Metrics in the Dashboard as detailed here.
However, once reviewed, even if it is not possible to edit it, it is still possible, as shown below, to visualize the initial Prediction done by the ModelVersion or the Review done by a human on the Prediction:
Visualize Prediction & Review
6. Detection Types
Depending on the Detection Type of the ModelVersion, the Shape composing a Prediction can either be Classification, Bounding-boxes, Polygon, Key Point or Line.
Obviously, the usage of the Prediction Review tool will then differ. For further details on the use of the tool according to the Detection Type, you can refer to the documentation of the Annotation tool as the behavior is similar for Shape creation, edition, or deletion.
7. Additional features
A. Lock and Hide Shape
To facilitate the review process, you can either lock or hide a Shape.
Hide or Lock a Shape
Obviously, a hidden Shape is not deleted, it is only not displayed. Locking a Shape prevents anyone from editing it. To unlock or make a Shape visible, you just need to click again on the Hide or Lock icon.
📘
From the right-click menu
You can lock or hide a Shape from the action menu displyed by right-cliking on the Shape directly on the image or in the Shapes list.
You can also lock or hide all the Shape of an Prediction with the same Label, as the Shape are all grouped together in the Shapes list on the right side.
🔦
You can hide the Labeland associated Confidence Score on the Shape by unchecking the tickbox Show prediction labels.
B. Group Selection
For practical reasons, the Picsellia Prediction Review tool offers the possibility to select several Shape at a time and make modifications to a group of Shape.
Multiple Shape selection can be done:
On the PredcitedAsset by drawing within the Selection mode a rectangle over the to-be-selected Shape, the selected bunch of Shape will be highlighted in blue in the Shapes list.
Manual group selection
By selecting all the Shape with a specific Label directly from the Shape list, as Shape with the same Label are grouped together. So selecting the Label will automatically select all the Shape with this Label, for instance, head:
Group selection by Label
Once a group of Shape is selected, you can:
Change the dimension and the position of these Shape
Change the Label of all the selected Shape with the right-click or the Labels dropdown (located in the top-left corner)
Delete the selected Shape with the delete key or the trash icon
C. Keyboard shortcut
To make the review process as smooth and efficient as possible on Picsellia, most of the actions related to the Prediction Review tool can be done using buttons on the tool but also through keyboard shortcuts.
The list of available shortcuts can be displayed through the keyboard icon:
Keyboard shortcuts
You can also define your preferences in the Annotation Tool Settings page:
Prediction Review Settings
From the Prediction Review Tool Settings you can:
Activate the auto move to the next PredictedAsset once the Prediction of the current one is Reviewed.
Activate the auto-selection of the drawing tool just after having drawn a Shape.
Activate the image caching to load in your Browser cache X images in advance (X being defined by the Cache size value available just below). This setting can be really useful to streamline the reviewing process with big images or if you have a poor internet connection.
Activate the display of the Label for all the Shape displayed in the Prediction Review Tool.
Define the opacity for all the Shape displayed in the Prediction Review Tool.
Define the simplification factor applied when using the Simplify feature a PolygonShape.