Deployments - Prediction Review Campaign

An Prediction Review Campaign allows you to handle properly and efficiently the process of reviewing the PredictedAsset of a Deployment on Picsellia.

By using Prediction Review Campaign on Picsellia, you will be able to:

• Define your Prediction Review Workflow
• Assign Review Tasks
• Check the Reviews done and raise Issues if something is wrong
• Track the progress of the Prediction Review Campaign

1. Create an Prediction Review Campaign

First of all, it is important to note that each Deployment can have up to one Prediction Review Campaign.

To initialize an Prediction Review Campaign on a Deployment, go to the Settings tab, and click on Workflow in the Campaign sub-section.

From the Prediction Review Workflow creation view, you can choose either to create an Empty Workflow (i.e. without Steps already created) or a Default Workflow (i.e. a basic two Steps Workflow that you can for sure modify).

A. Create an Annotation Workflow

Let's consider that you are starting from an Empty Workflow.

Here, you will be able to define the Prediction Review Workflow that defines the Steps your PredictedAsset will follow during the Prediction Review Campaign you are about to launch on this Deployment. The Prediction Review Workflow represents all the Steps the PredictedAsset will go through before being considered properly reviewed. As you can see above, the Workflow is initially free of Steps, so you will now be able to define and configure new Steps before launching the Prediction Review Campaign.

To do so, we just need to click on the + button next to the initial step of your Prediction Review Workflow. A new Step will be added to your Prediction Review Workflow, it is now time to configure it.

Once the new Step is created, its related configuration panel will pop up on the right side.

First, you'll be requested to give a name to your Step and to define if it is a Review or a Quality Check Step.

📘

The only difference between a Review and a Quality Check Step is the ability for Quality Check Step to Accept or Reject what has been done in previous steps.

Then, you need to select the Assignees that will get Tasks attributed as part of this Step. As shown below you can select the Assignees.

📘

Select Assignees

The potential Assignees are the members of the current Organization that are listed are Collaborators in the Deployment.

So if a member is not part of the Assignee list, you should add him as Collaborator in the related Deployment.

Finally, you'll be requested to define the assignation strategy based on two criteria.

• The Asset split strategy allows you to define if the PredictedAsset going through this Step should be equally assigned to all the selected Collaborators (Equally dispatched strategy) or if some Collaborators should get more Tasks assigned then others. In this case, you could set it up using the Custom Weights strategy. For Custom Weights strategy the percentage of Tasks assigned to each Collaborator will be displayed as shown below.
• The Sample rate, allowing you to set the probability an image submitted at the previous step has to actually go through the current Step. This one could be really useful in case of want to perform a review on only a sample of the annotated images.

Now that your Step is fully configured you just need to click on Create to finish the Step creation.

Once created, you can at any time find out the number of Assignees assigned to this Step and the number you can edit the step anytime by clicking on it and accessing the configuration panel.

👍

The Unpriviledged role

If you want some members to only have access to the Tasks they are supposed to do within Prediction Review Campaign running in your Organization and nothing else, you can set them as Unpriviledged members. More details are available here.

Then, you can add as many Steps as you want to create the Prediction Review Workflow that suits your needs.

For instance, a simple and efficient Prediction Review Workflow would be to have an Review Step and then a Quality Check Step with a part of your team assigned to Review and some others to the Quality Check, as shown below:

📘

A Collaborator can be assigned to several Steps of an Prediction Review Workflow

In this case, the Tasks will be assigned to avoid a given PredictedAsset to be annotated and reviewed by the same Collaborator when possible.

B. Configure your Prediction Review Campaign

Now that your Prediction Review Workflow is defined, you can configure the way your Prediction Review Campaign will be initialized by clicking on the settings button. A configuration panel will open allowing you to:

• Add an instruction PDF file for the Collaborators which will be visualizable by all the Assignees of the current Prediction Review Campaign while doing the Tasks they get assigned to. This pdf could, for instance, list all the guidelines and instructions the Collaborators need to be aware of for properly reviewing the PredictedAsset.
• Define a due date for the Prediction Review Campaign
• Make sure the Prediction Review Campaign is closed when all the Tasks are Done

C. Launch the Prediction Review Campaign

At the end of the configuration phase of your Prediction Review Campaign, all the PredictedAsset of your Deployment are at the initial Step named Deployment.

It is now time to start the Prediction Review Campaign. To do so you simply need to click on Sync Manually, this will make all the PredictedAsset present in the Deployment Step transition to the first Step of the Prediction Review Workflow, the related Tasks will then be created.

Now that the Campaign is live, the Campaign badge will be displayed and two new tabs will be created, Tasks and Issues. They will allow you and the other Organization members to go through this ongoing Prediction Review Campaign.

2. Access the Tasks list

As soon as an Prediction Review Campaign is ongoing on a Deployment, you will be able to access a new tab called Tasks.

🚧

Who can access the Tasks tab ?

This Tasks tab is available for every Collaborator in your Deployment except the ones with the Labelerrole (those added as Unprivileged to your Organization).

The members with the role Unpriviledged in the Organization, that have been added in the Collaborators list of the Deployment can only visualize the Tasks that are assigned to them in a dedicated view as detailed here.

The Tasks list allows you to visualize the current Status and Step of the latest Tasks related to any PredictedAsset that is currently going through the Prediction Review Campaign.

This view allows you to filter the Tasks to display. You can filter by the Task Status, by the current Step of the Prediction Review Workflow, or by the current Assignee.

A. Task Status

As shown below each Task created for an PredictedAsset has a current Step and Assignee, those are defined while creating the Prediction Review Workflow as described here. However, we can also notice the presence of a new field named Status for each Task.

This Status field describes the status of the PredcitedAsset journey in the Prediction Review Workflow. The Status of a Task can either be:

• In Progress: Everything is going well until now, the associated PredictedAsset is going through the Prediction Review Workflow normally. However, the PredictedAsset is still at a Review or a Quality Check step reflected by the In Progress status. The Prediction Review Workflow isn't over yet.
• To Rework: A Task with the to Rework status is a Review Task created when a PredictedAsset has been annotated during a first Review Task and submitted as To Rework during the next Quality Check step. Indeed, the Assignee of this Quality Review step didn't accept the proposed Review. So he sent the PredcitedAsset back to the previous Step in the Prediction Review Workflow. Usually, the reason is described in an Issue. As a consequence, a new Task with To Rework status has been re-created at the previous Review step of the Prediction Review Workflow and re-assigned to the Collaborator that created the rejected Review. This Assignee can then modify its first Review by completing this To Rework Task and sending it back to the following Quality Check Step.
• Done: The PredictedAsset has been through the whole Prediction Review Workflow, the PredictedAsset can be considered as reviewed. It will be displayed with the status Reviewed in the Predictions tab and as a consequence can be submitted to the Feedback Loop.
• Skipped: While completing an Review or Quality Check Task, the Assignee has the possibility to Skip the Task, this will kick the associated PredictedAsset out of the Annotation Workflow. There is no Task to be completed on this PredictedAsset anymore but it cannot be considered as properly reviewed. Usually, the reason is described in an Issue.

B. Reassign a Task or a bunch of Tasks

As described in detail here, a Task can only be completed by the Assignee. If for some reason, a Task needs to be completed by someone else, this one can be reassigned for completion to a member that is identified as Collaborator on the Deployment.

If you have sufficient rights, it can be done for a particular Task, by clicking from the Task list on the Assignee, the list of available Collaborators will then be displayed. You just need to click on the new Assignee as shown below:

If you have sufficient rights, it can be done for a bunch of Tasks, by selecting the to-be-reassigned Tasks and clicking on the Reassign button. Then you simply need to select from the list of available Collaborators the one that will get the selected Tasks reassigned.

🚧

A filter on the Steps should be active

For the reassignment of several Tasks, you must ensure that a filter on the Task step is active on the Task list before reassigning.

C. Submit a bunch of Tasks

During a normal functioning process, each Task should be completed by its Assignee by opening the Prediction Review Tool.

But sometimes, Collaborators prefer to first complete their Tasks in the Annotation Studio, save them, and Submit all their Tasks at the same time. In this case, they can use the Submit button available in the Task list.

To bulk submit a bunch of Tasks, you simply need to use the filters or the Query Langage to retrieve and select Tasks to be submitted and click on Submit:

By clicking on Submit, all the selected Tasks will be submitted, meaning that the PredictedAsset will transition through the Annotation Workflow to the next Step. A new Task will be created according to the Task assignation rules of the Step.

Please be aware that only the selected PredcitedAsset with an existing Review will be submitted. If you want to submit also thePredcitedAsset without Review you can use the Move button described here.

In addition, if the next Step in the Workflow is Done, the PredcitedAsset with Issues still open won't be submitted. Once again, if you want to submit also thePredictedAsset open Issues related, you can use the Move button described here.

D. Move a bunch of Tasks

During a normal functioning process, each Task should be completed by its Assignee by opening the Annotation Studio.

But in some cases, you might be forced to move several Tasks at the same time to another Step (including Done or Skipped), whether they are assigned to you or not.

To do so, you simply need to use the filters or the Query Langage to retrieve and select Tasks to be transitioned to another Step. Then click on Move:

A pop-up will then open allowing you to select the destination Step of the selected Tasks:

Once PredictedAsset are move to another Step, obviously new Tasks will be generated at this new Step according the Task assignation rules of this one.

📘

PredictedAsset moving backward in the Workflow

If the selected PredictedAsset are sent backward in the Prediction Review Workflow the new Task created will have the In Rework status, whereas if PredictedAsset are more forward in the Prediction Review Workflow, the new Task created will have the In Progress status.

Please be also aware that unlike the Submit button, using the Move button will transition the selected PredictedAsset even if some Issues are still open or if no Review has been initiated on this PredictedAsset yet.

The Move operation can also be done from the Worflow Settings as shown below. However, from this tab, you cannot granularly select which Tasks from a given step should be moved or not.

E. Redistribute Tasks

At any moment, the distribution of the Tasks at each Step may have changed from the one defined at the Step definition. This can be explained by the manual reassignation of Tasks, by some Assignees that are faster than others or by the update of custom weights.

For all those reasons, you have the possibility for each Step to redistribute among the Assignees the current tasks according the the latest Weights defined , to do so you just need to click on Redistribute as shown below:

As you can see above, for each Step selected, you have an overview of the real and theoric (defined in the Assets split strategy) Tasks repartition among Assignees.

3. Complete a Task

Now that we can visualize, understand, and manipulate the Tasks created in the frame of the current Prediction Review Campaign, it is high time to start completing them to make our PredictedAsset go through the Prediction Review Workflow.

📘

Who can complete a Task?

The only member able to complete a Task is the Assignee.
So if you want to complete a Task without being the Assignee, you need to reassign it first as explained here.

Whatever the Task type (Annotation or Review) to complete it you need to click on the Go to task button as shown below, the Prediction Review Tool will then open allowing you to perform the Annotation or the Review required by the Task. The PredictedAsset can be free of Annotation or in case the Annotation has already been initialized this one will be displayed as submitted at the previous step of the Annotation Workflow.

A. Complete an Review Task

In case you need to complete a Review Task the Prediction Review Tool will open displaying the PredictedAsset to be reviewed. You can rely on the documentation of the Prediction Review Tool here to create the Shape that will compose the Review.

Once you have created or edited the Shape and considered the Review satisfying, you can Submit the Task as shown below.

Submitting a Task will save it and transition it to the next step of the Prediction Review Workflow.
Please note that you can also just Save the Review and submit the Task later on.

However, as you are reviewing within a Prediction Review Campaign, the Prediction Review Tool has some additional features dedicated to Prediction Review Campaign.

Please note that any Collaborators who’s not a Labeler and who has access to the Deployment can click on the Go to task button and open the related Asset in the Annotation Tool, but only the Assignee of the Task can create, edit the Review and submit the Task.

a. Accessing the history of the PredictedAsset

From the Prediction Review Tool you can visualize the history of the PredictedAsset during the Prediction Review Workflow as shown below:

You can see all the steps done or to come of the Prediction Review Workflow. The current step is highlighted in orange. For each step are displayed, the Assignee and the submission date.

In addition by clicking on any step, you can visualize the Review submitted at this step allowing you to see and track how it evolved going through the Prediction Review Workflow.

b. Creating Issues

To ease the communication between the different Collaborators involved at the different steps of the Workflow, Issues can be raised on the whole PredictedAsset or a zone of the PredictedAsset.

Issues can for instance be used when a Collaborator doubts how to annotate an image to request advice from the Collaborator in charge of the Quality Check. It can also be used to track the justification why a PredictedAsset has been skipped and sent back to an Review step. More details on the usage of Issues during an Prediction Review Campaign are available here.

c. Visualizing the Annotation Instructions

As explained here, during the Prediction Review Campaign initialization, a pdf file can be attached.
This can be used to share the annotation guidelines with all the Collaborators who will be involved in the Prediction Review Campaign. During the completion of a Task, a Collaborator can open the pdf file as shown below:

B. Complete a Quality Check Task

In case you need to complete a Quality Check Task, the Prediction Review Tool will open displaying the PredictedAsset, the related Review and the potential Issues created previously during the Prediciton Review Workflow.

During a Quality Check Task, the Assignee has to review the Review done during the previous Review step.

During a Quality Check Task, 4 scenarios can occur:

• The Review is correct according to the Assignee, the Task can be submitted as it is by clicking on the Submit button.
• The Review is not satisfying and the Assignee wants to edit it himself. The Assignee can then use the Prediction Review Tool to edit the Review. Once it is considered as satisfactory, the Assignee can click on Submit to transition the PredictedAsset to the potential next step of the Workflow.
• The Review is not satisfying and the Assignee wants to send the PredictedAsset back to the previous Review step allowing the previous Assignee to rework the Review. Before sending it back, the Assignee will have to create an Issue (Zone or Image Issue) to explain why the PredictedAsset is sent back to the previous step, once done, the Assignee of the Quality Check task can click on To Rework, and a new Task will be created and assigned at the previous step of the Workflow.
• The Assignee of the Quality Check Task considers that the PredictedAsset should not keep going through the Prediction Review Workflow. In this case, the Assignee can click on Skip, then the Assignee will be requested to create an Issue to explain why this one should be skipped.

During the Quality Check step while completing its Task, the Assignee can answer or close the Issues created previously by other Collaborators and create new Issues that will be handled by the next Assignee. (either further in the Workflow if the PredictedAsset is Submitted or upstream if the PredictedAsset is sent to rework).

4. Manage Prediction Review Workflow lifecycle

At any moment, you can update, close or delete your Prediction Review Campaign.

A. Update a Prediction Review Campaign

As explained previously you can manage easily manage your Tasks by transitioning, submitting, or reassigning them.

a. Update a Step

But you can also during a Prediction Review Campaign modify your Workflow. Each step configuration can be updated at any moment by modifying its name, Sample rate, Assignees, or Asset split strategy.

b. Add a Step

You can also add a Step at any moment of an Prediction Review Campaign by clicking on the + at the desired place for your new Step.

The Step configuration panel will then open allowing you to configure this new step the same you already did during Worflow initialization.

As soon as you click on Create, the new Step will enter the Workflow.

This meaning that submitting a PredictedAsset at the previous Step will make it go through it. Additionnaly, any PredictedAsset can be moved to this new Step.

c. Delete a step

You can also delete a Step at any moment of a Prediction Review Campaign by clicking on the Step and Delete.

In case, there are currently no Tasks at the Step to be deleted, then no problems, the Step is actually deleted. However, if there are currently Tasks at this Step, a pop-up will open asking you to select the Step those Tasks should be sent to.

This way you can delete a Step while ensuring there is no impact on its related Tasks.

You can anyway always display the deleted Tasks using the radio button as shown below:

For traceability purposes, even if a Step is deleted, all the work done at this one when it was alive is kept and can be visualized in the History tab of the Prediction Review Tool.

This way you can delete Steps of your Prediciton Review Workflow without losing any information

B. Delete an Prediction Review Campaign

A Collaborator with Admin right on the parent Deployment can decide to Delete an ongoing Prediction Review Campaign.

It can be made from the Deployment Settings page, through the Campaign settings button as shown below:

The deletion of a Prediction Review Campaign is irreversible and for each existing PredictedAsset, only the latest version of the related Review will be kept. In other, words you will lose all the history of the Prediction Review Campaign and won't be able to access the Tasks & Issues tabs anymore.

Therefore we strongly recommend you to be very cautious while deleting an Prediction Review Campaign.

Once the Prediction Review Campaign is deleted, the Deployment is back to its initial state but each PredictedAsset still has the latest version of the Review created through the deleted Prediction Review Campaign.

For sure, from this point you can create brand new Prediction Review Campaign.

5. Issues Management

As already mentioned previously, within a Prediction Review Campaign, any Assignee can create an Issue while completing its Task. The Issue system aims at facilitating communication between the different Assignees for a given PredictedAsset along its transition through the Annotation Workflow.

Once an Issue is created the different Collaborators can chat and comment on this one until it is closed. All the open and closed Issues related to a PredictedAsset are accessible by authorized Collaborators at any step of the Workflow.

Please note that the Issue system is only available for Deployment with a Prediction Review Campaign ongoing.

There are two types of Issues:

• Zone issue: Related to a specific area of the PredictedAsset
• Image Issue: Related to the whole PredictedAsset

A. Create a Zone Issue

This Issue can be related to the whole PredictedAsset or to a specific zone on the PredictedAsset.

When completing a Task on the Prediction Review tool, as an Assignee you can create a Zone Issue by clicking on the Create a zone Issue button as shown below. Then draw on the image a rectangle on the zone affected by the Issue you want to raise.

Please note that any Collaborator who can access the Task (even without being the Assignee) can create a Zone Issue on the PredictedAsset.

Once the zone has been created, the Issues Panel will open in the right panel of the Prediction Review tool. You can then describe your Zone Issue and click on Create Issue.

Once created, your Issue will appear on the Issues panel related to the current PredictedAsset.

B. Create an Image Issue

An Image Issue is not related to a specific area but to the whole PredictedAsset.

An Image Issue can be created directly from the Issues Panel as shown below:

Once the description is prompted the Image Issue will be created after clicking on Create Issue. This new Issue will be also accessible at any time from the Issues panel .

Please note also that the Assignee will be forced to create an Image Issue before Skipping or sending a PredictedAsset for Rework if nor Image or Zone Issue exists on the current PredictedAsset. This is a way to track why this PredictedAsset has been Skipped or sent To Rework.

As it is the case for Zone issues, from this Issue Panel, Collaborators can comment, resolve or re-open the Image Issue.

C. Issues panel

After opening any PredictedAsset from the Tasks list, you can access the Issues panel as shown below on the right side of the Prediction Review Tool.

Here are displayed all the Issues (Zone or Image) related to the current PredictedAsset created during the Prediction Review Campaign. You can switch between the still-open Issues and the closed ones.

By clicking on any Issue (Open or Closed), you will access the conversation thread allowing you to see the conversation related to this issue, add a Reply, Resolve it, or Re-Open it.

Please note that any Collaborator that can access the Task (even without being the Assignee) can interact with an Issue in the Issue Panel.

Please note that any Issue opened will remain until a Collaborator with the necessary rights closes it.

👍

An PredictedAsset cannot turn in Done status with still open Issues

To make a PredictedAsset turn in the Done status, meaning submitting the PredictedAsset at the last step of the Workflow, all the Issues related should be closed. In a few words, you won't be able to submit the very last Task of the Workflow is some Issues are still open.

6. The Labeller role

In case you need to involve some members in your Prediction Review Campaign but willing them only to complete the Tasks they get attributed without letting them visualize or manipulate any other object related to your Organization, you can add them as Unpriviliedged and add them with the Labeler role in the Deployment.

a. Add a member as Labeler in your Deployment

To involve a member in your Prediction Review Campaign and let him complete the assigned Tasks without accessing any other object of your Organization, you first need to add this member with the Unpriviledged role to your Organization.

To do so, you need to be Admin in the current Organization, go to Organization Settings > Members and click on Inviter member.

You can add members using their username if they already have a Picsellia account or by typing their email address (in this case, they will get an email with a link to create their account), then you need to select Unpriviliedged as Default Permission and click on Invite.

Then to make sure you can involve the invited Unprivileged member in an Prediction Review Campaign, you need to add him to the Collaborators list of the Deployment as shown below:

Once done the Unprivildged member is identified as Labeler in the Deployment and can be involved in Prediction Review Campaign for any Deployment related.

B. Access an Organization as an Unprivileged member

The members invited to an Organization with Unprivilideged rights, can access the Organization but by default, they won’t be able to access any object (Datalake, Dataset, Project, Model, Deployment…). They will only be able to visualize the My Tasks view.

This view is only accessible to members with the Unpriviledged role in the current Organization. From this one, the Unpriviledged member will see the list of Tasks assigned to him and nothing else.

The Labeler will see all the Tasks assigned within the current Organization grouped by Prediction Review Campaign. Each Prediction Review Campaign is identified by its Deployment related. The Labeler can visualize all the Tasks that are currently assigned to him grouped by Prediction Review Campaign, the name of the current step that generated this Task, and the number of issues on the related PredictedAsset.

The Labeler can also filter on the Tasks Status or Step name.

To complete a Task as Labeler, you can click on Go to task and you can complete your Task (Review or Quality Check) as detailed here.

The Labeler can also access a detailed view describing all the Issues raised on any PredictedAsset that generates a Task currently assigned to the Labeler.

📘

Labelers can manage Issues, read instructions and access PredictedAsset history

For any PredictedAsset that generate a Task currently assigned to a given Labeler, this one can access all the Issues management features (Open, Reply, Resolve, Re-open).

While completing a Task the Labeler can also access the history of the PredictedAsset int he Prediction Review Workflow and open the Review guidelines file if any.

7. Analytics

Collaborators who are not Labelers can access the Analytics Dashboard related to the current Deployment in the Analytics tab.

If an Prediction Review Campaign has been set, the three following tabs are available:

• Throughput: This tab will through its metrics detail the progress of the Prediction Review Campaign.
• Efficiency: This tab will through its metrics detail the time spent by the Assignees on the Prediction Review Campaign.
• Quality: This tab will detail the Deployment with statistical metrics based on the Review created until now.

📘

Only Quality Analytics are availble out of an Prediction Review Campaign.

A. Throughput

The Throughput tab displays graphs that are related to the progress of the Prediction Review Campaign. From this tab, you will have a clear view of how the PredictedAsset of the Deployment is going through the Prediction Review Workflow in real-time.

In detail, the following metrics are displayed:

• Prediction Review Campaign Progress: A percentage of Prediction Review Campaign completion based on the Tasks already submitted compared to the total number of Tasks to submit to complete the Prediction Review Campaign
• Tasks Distribution: A pie chart showing the total number of active Tasks grouped by step.
• Total Review Assignments: A bar chart showing the total number of Annotation Tasks submitted. The time range depends on the selected filter.
• Total Quality Check Assignments: A bar chart showing the total number of QUality Check Tasks submitted. The time range depends on the selected filter.
• Assignment Count By Collaborator: A bar chart detailing the total number of Tasks currently to each Collaborator involved in the Prediction Review Campaign.
• Issues Quantity: A bar chart detailing the total number of Issues created until now during the Prediction Review Campaign grouped by Creation Date. The time range depends on the selected filter as explained below.
• Issues Type Distribution: A pie chart detailing the total number of Issues created until now during the Prediction Review Campaign grouped by Issues Type(Zone or Image). The time range depends on the selected filter.

For each chart, you can change the Date Range of the data displayed as shown below:

In addition, you can filter the data displayed based on a given step of the Prediction Review Workflow or by Assignee.

B. Efficiency

The Efficiency tab displays graphs that are related to the time dedicated to the different steps of the Prediction Review Campaign in real-time.

In detail, the following metrics are displayed:

• Task Duration: Globalized time information on all the Tasks, the average time to complete a Task, and the minimum and maximum time of Task completion on the whole current Prediction Review Campaign.
• Closed Issues: The total number of Issues closed during the current Prediction Review Campaign.
• Opened Issues: The total number of Issues opened during the current Prediction Review Campaign
• Total Review Time: The total time spent on Review Tasks until now. A bar chart displays this duration on the selected Date Range based on the filter.
• Average Review Time: The average time spent on one Review Task until now. A bar chart displays this duration on the selected Date Range based on the filter.
• Total Quality Check Time: The total time spent on Quality Check Tasks until now. A bar chart displays this duration on the selected Date Range based on the filter.
• Average Quality Check Time: The average time spent on one Quality Check Task until now. A bar chart displays this duration on the selected Date Range based on the filter.

For each chart, you can change the Date Range of the data displayed as shown below:

In addition, you can filter the data displayed based on a given step of the Prediction Review Workflow or by Assignee.