Dataset - Annotation Campaign

An Annotation Campaign allows you to handle properly and efficiently the process of annotating a DatasetVersion on Picsellia.

By using Annotation Campaign on Picsellia, you will be able to:

• Define your Annotation Workflow
• Assign Annotation Tasks
• Review the Annotation done and raise Issues if something is wrong
• Track the progress of the Annotation Campaign

1. Create an Annotation Campaign

First of all, it is important to note that each DatasetVersion can have up to one Annotation Campaign.

To initialize an Annotation Campaign on a DatasetVersion, go to the Settings tab, and click on Workflow in the Campaign sub-section.

From the Annotation Workflow creation view, you can choose either to create an Empty Workflow (i.e. without Steps already created) or a Default Workflow (i.e. a basic two Steps Workflow that you can for sure modify).

A. Create an Annotation Workflow

Let's consider that you are starting from an Empty Workflow.

Here, you will be able to define the Annotation Workflow that defines the Steps your Asset will follow during the Annotation Campaign you are about to launch on this DatasetVersion. The Annotation Workflow represents all the Steps the Asset will go through before being considered properly annotated. As you can see above, the Workflow is initially free of Steps, so you will now be able to define and configure new Steps before launching the Annotation Campaign.

To do so, we just need to click on the + button next to the initial step of your Annotation Workflow. A new Step will be added to your Annotation Workflow, it is now time to configure it.

Once the new Step is created, its related configuration panel will pop up on the right side.

First, you'll be requested to give a name to your Step and to define if it is an Annotation or a Review Step.

📘

The only difference between an Annotation or Review step is the ability in Review to also Accept or Reject what has been done in previous steps.

Then, you need to select the Assignees that will get Tasks attributed as part of this Step. As shown below you can select the Assignees.

📘

Select Assignees

The potential Assignees are the members of the current Organization that are listed are Workers in the Dataset.

So if a member is not part of the Assignee list, you should add him as Worker in the related Dataset.

Finally, you'll be requested to define the assignation strategy based on two criteria.

• The Asset split strategy allows you to define if the Asset going through this Step should be equally assigned to all the selected Workers (Equally dispatched strategy) or if some Workers should get more Tasks assigned then others. In this case, you could set it up using the Custom Weights strategy. For Custom Weights strategy the percentage of Tasks assigned to each Worker will be displayed as shown below.
• The Sample rate, allowing you to set the probability an image submitted at the previous step has to actually go through the current Step. This one could be really useful in case of want to perform a review on only a sample of the annotated images.

Now that your Step is fully configured you just need to click on Create to finish the Step creation.

Once created, you can at any time find out the number of Assignees assigned to this Step and the number you can edit the step anytime by clicking on it and accessing the configuration panel.

👍

The Unpriviledged role

If you want some members to only have access to the Tasks they are supposed to do within Annotation Campaigns running in your Organization and nothing else, you can set them as Unpriviledged members. More details are available here.

Then, you can add as many Steps as you want to create the Annotation Workflow that suits your needs.

For instance, a simple and efficient Annotation Workflow would be to have an Annotation Step and then a Review Step with a part of your team assigned to Annotation and some others to the Review, as shown below:

📘

A Worker can be assigned to several Steps of an Annotation Workflow

In this case, the Tasks will be assigned to avoid a given Asset to be annotated and reviewed by the same Worker when possible.

B. Configure your Annotation Campaign

Now that your Annotation Workflow is defined, you can configure the way your Annotation Campaign will be initialized by clicking on the settings button, a configuration panel will open allowing you to:

• Add an instruction PDF file for the Workers which will be visualizable by all the Assignees of the current Annotation Campaign while doing the Tasks they get assigned to. This pdf could, for instance, list all the guidelines and instructions the Workers need to be aware of for properly annotating the DatasetVersion.
• Define a due date for the Annotation Campaign
• Make sure the Annotation Campaign is closed when all the Tasks are Done

C. Launch the Annotation Campaign

At the end of the configuration phase of your Annotation Campaign, all the Asset of your DatasetVersion are at the initial Step named Dataset Version.

It is now time to start the Annotation Campaign. To do so you simply need to click on Sync Manually, this will make all the Assets present in the DatasetVersion Step transition to the first Step of the Annotation Workflow, the related Tasks will then be created.

Now that the Campaign is live, the Campaign badge will be displayed and two new tabs will be created, Tasks and Issues. They will allow you and the other Organization members to go through this ongoing Annotation Campaign.

2. Access the Tasks list

As soon as an Annotation Campaign is ongoing on a DatasetVersion, you will be able to access a new tab called Tasks.

🚧

Who can access the Tasks tab ?

This Tasks tab is available for every worker in your Dataset except the ones with the Labelerrole (those added as Unprivileged to your Organization).

The members with the role Unpriviledged in the Organization, that have been added in the Workers list of the Dataset can only visualize the Tasks that are assigned to them in a dedicated view as detailed here.

The Tasks list allows you to visualize the current Status and Step of the latest Tasks related to any Asset that is currently going through the Annotation Campaign.

This view allows you to filter the Tasks to display. You can filter by the Task Status, by the current Step of the Annotation Workflow, or by the current Assignee.

A. Task Status

As shown below each Task created for an Asset has a current Step and Assignee, those are defined while creating the Annotation Workflow as described here. However, we can also notice the presence of a new field named Status for each Task.

This Status field describes the status of the Asset journey in the Annotation Workflow. The Status of a Task can either be:

• In Progress: Everything is going well until now, the associated Asset is going through the Annotation Workflow normally. However, the Asset is still at an Annotation or Review step reflected by the In Progress status. The Annotation Workflow isn't over yet.
• To Rework: A Task with the to Rework status is an Annotation Task created when an Asset has been annotated during a first Annotation task and submitted as To Rework during the next Review step. Indeed, the Assignee of this Review step didn't accept the proposed Annotation. So he sent the Asset back to the previous Step in the Annotation Workflow. Usually, the reason is described in an Issue. As a consequence, a new Task with To Rework status has been re-created at the previous Annotation step of the Annotation Workflow and re-assigned to the Worker that created the rejected Annotation. This Assignee can then modify its first Annotation by completing this To Rework task and sending it back for the second time to the following Review Step.
• Done: The Asset has been through the whole Annotation Workflow, the Asset can be considered as annotated.
• Skipped: While completing an Annotation or Review task, the Assignee has the possibility to Skip the Task, this will kick the associated Asset out of the Annotation Workflow. There is no Task to be completed on this Asset anymore but it cannot be considered as properly annotated. Usually, the reason is described in an Issue.

B. Reassign a Task or a bunch of Tasks

As described in detail here, a Task can only be completed by the Assignee. If for some reason, a Task needs to be completed by someone else, this one can be reassigned for completion to a member that is identified as Worker on the Dataset.

If you have sufficient rights, it can be done for a particular Task, by clicking from the Task list on the Assignee, the list of available Workers will then be displayed. You just need to click on the new Assignee as shown below:

If you have sufficient rights, it can be done for a bunch of Tasks, by selecting the to-be-reassigned Tasks and clicking on the Reassign button. Then you simply need to select from the list of available Workers the one that will get the selected Tasks reassigned.

🚧

A filter on the Steps should be active

For the reassignment of several Tasks, you must ensure that a filter on the Task step is active on the Task list before reassigning.

C. Submit a bunch of Tasks

During a normal functioning process, each Task should be completed by its Assignee by opening the Annotation Studio.

But sometimes, Workers prefer to first complete their Tasks in the Annotation Studio, save them, and Submit all their Tasks at the same time. In this case, they can use the Submit button available in the Task list.

To bulk submit a bunch of Tasks, you simply need to use the filters or the Query Langage to retrieve and select Tasks to be submitted and click on Submit:

By clicking on Submit, all the selected Tasks will be submitted, meaning that the Asset will transition through the Annotation Workflow to the next Step. A new Task will be created according to the Task assignation rules of the Step.

Please be aware that only the selected Asset with an existing Annotation will be submitted. If you want to submit also theAsset without Annotation you can use the Move button described here.

In addition, if the next Step in the Workflow is Done, the Asset with Issues still open won't be submitted. Once again, if you want to submit also theAsset open Issues related, you can use the Move button described here.

D. Move a bunch of Tasks

During a normal functioning process, each Task should be completed by its Assignee by opening the Annotation Studio.

But in some cases, you might be forced to move several Tasks at the same time to another Step (including Done or Skipped), whether they are assigned to you or not.

To do so, you simply need to use the filters or the Query Langage to retrieve and select Tasks to be transitioned to another Step. Then click on Move:

A pop-up will then open allowing you to select the destination Step of the selected Tasks:

Once Asset are move to another Step, obviously new Tasks will be generated at this new Step according the Task assignation rules of this one.

📘

Assets moving backward in the Workflow

If the selected Asset are sent backward in the Annotation Workflow the new Task created will have the In Rework status, whereas if Asset are more forward in the Annotation Workflow, the new Task created will have the In Progress status.

Please be also aware that unlike the Submit button, using the Move button will transition the selected Asset even if some Issues are still open or if no Annotation has been initiated on this Asset yet.

The Move operation can also be done from the Worflow Settings as shown below. However, from this tab you cannot granularly select which Tasks from a given step should be moved or not.

3. Complete a Task

Now that we can visualize, understand, and manipulate the Tasks created in the frame of the current Annotation Campaign, it is high time to start completing them to make our Asset go through the Annotation Workflow.

📘

Who can complete a Task?

The only member able to complete a Task is the Assignee.
So if you want to complete a Task without being the Assignee, you need to reassign it first as explained here.

Whatever the Task type (Annotation or Review) to complete it you need to click on the Go to task button as shown below, the Annotation Tool will then open allowing you to perform the Annotation or the Review required by the Task. The Asset can be free of Annotation or in case the Annotation has already been initialized this one will be displayed as submitted at the previous step of the Annotation Workflow.

A. Complete an Annotation task

In case you need to complete an Annotation Task the Annotation Tool will open displaying the Asset to be annotated. You can rely on the documentation of the Annotation Tool here to create the Shape that will compose the Annotation.

Once you have created or edited the Shape and considered the Annotation satisfying, you can Submit the Task as shown below.

Submitting a Task will save it and transition it to the next step of the Annotation Workflow.
Please note that you can also just Save the Annotation and submit the Task later on.

However, as you are annotation within an Annotation Campaign, the Annotation Tool has some additional features dedicated to Annotation Campaigns.

Please note that any Worker who’s not a Labeler and who has access to the DatasetVersion can click on the Go to task button and open the related Asset in the Annotation Tool, but only the Assignee of the Task can create, edit Annotation and submit the Task.

a. Accessing the history of the Asset

From the Annotation Tool you can visualize the history of the Asset during the Annotation Workflow as shown below:

You can see all the steps done or to come of the Annotation Workflow. The current step is highlighted in orange. For each step are displayed, the Assignee and the submission date.

In addition by clicking on any step, you can visualize the Annotation submitted at this step allowing you to see and track how it evolved going through the Annotation Workflow.

b. Creating Issues

To ease the communication between the different workers involved at the different steps of the Workflow, Issues can be raised on the whole Asset or a zone of the Asset.

Issues can for instance be used when a Worker doubts how to annotate an image to request advice from the Worker in charge of the Review. It can also be used to track the justification why an Asset has been skipped and sent back to an Annotation step. More details on the usage of Issues during an Annotation Campaign are available here.

c. Visualizing the Annotation Instructions

As explained here, during the Annotation Campaign initialization, a pdf file can be attached.
This can be used to share the annotation guidelines with all the Workers who will be involved in the Annotation Campaign. During the completion of a Task, a Worker can open the pdf file as shown below:

B. Complete a Review task

In case you need to complete a Review Task, the Annotation Tool will open displaying the Asset, the related Annotation to be reviewed and the potential Issues created previously during the Annotation Workflow.

During a Review Task, the Assignee has to review the Annotation done during the previous Annotation step.

During a Review Task, 4 scenarios can occur:

• The Annotation is correct according to the Assignee, the Task can be submitted as it is by clicking on the Submit button
• The Annotation is not satisfying and the Assignee wants to edit it himself. The Assignee can then use the Annotation Tool to edit the Annotation. Once the Annotation is considered satisfactory, the Assignee can click on Submit to transition the Asset to the potential next step of the Workflow.
• The Annotation is not satisfying and the Assignee wants to send the Asset back to the previous Annotation step allowing the previous Assignee to rework the Annotation. Before sending it back, the Assignee will have to create an Issue (Zone or Image Issue) to explain why the Asset is sent back to the previous step, once done, the Assignee of the Review task can click on To Rework, and a new Task will be created and assigned at the previous step of the Workflow.
• The Assignee of the Review Task considers that the Asset should not keep going through the Annotation Workflow. In this case, the Assignee can click on Skip, then the Assignee will be requested to create an Issue to explain why this one should be skipped.

During the Review step while completing its Task, the Assignee can answer or close the Issues created previously by other Workers and create new Issues that will be handled by the next Assignee (either further in the Workflow if the Asset is Submitted or upstream if the Asset is sent to rework).

4. Manage Annotation Workflow lifecycle

At any moment, you can update, close or delete your Annotation Campaign.

A. Update Annotation Campaign

As explained previously you can manage easily manage your Tasks by transitioning, submitting or reassigning them.

a. Update a Step

But you can also during an Annotation Campaign modify your Workflow. Each step confiugration can be updated at any moment by modifying it's name, Sample rate, Assignees, or Asset split strategy.

b. Add a Step

You can also add a Step at any moment of an Annotation Campaign by clicking on the + at the desired place for your new Step.

The Step configuration panel will then open allowing you to configure this new step the same you already did during Worflow initialization.

As soon as you click on Create, the new Step will enter the Workflow. Meaning Asset submitted at the previous one will go through it and that Asset current at any other Step can be move to this one.

c. Delete a step

You can also delete a Step at any moment of an Annotation Campaign by clicking on the Step and Delete.

In case, there is currently no Tasks at the Step to be deleted, then no problems, the Step is deleted. However, if there are Tasks at this Step, a pop-up will open asking you to select the Step those Tasks should be sent to.

This way you can delete a Step while ensuring there is no impact on its related Tasks.

You can anyway always display the deleted Tasks using the radio button as shown below:

For traceability purposes, even if a Step is deleted, all the work done at this one when it was alive is kept and can be visualized in the History tab of the Annotation Studio.

This way you can delete Steps of your Annotation Workflow without losing any information

B. Delete an Annotation Campaign

A Worker with Admin right on the parent Dataset can decide to Delete an ongoing Annotation Campaign.

It can be made from the DatasetVersion Settings page, through the Campaign settings button as shown below:

Unlike closing an Annotation Campaign, the deletion of an Annotation Campaign is irreversible and for each existing Asset, only the latest version of the related Annotation will be kept. In other, words you will lose all the history of the Annotation Campaign and won't be able to access to the Tasks & Issues tabs anymore.

Therefore we strongly recommend you to be very cautious while deleting an Annotation Campaign.

Once the Annotation Campaign is deleted, the DatasetVersion is back to its initial state but each Asset still has the latest version of the Annotation created through the deleted Annotation Campaign.

For sure, from this point you can create brand new Annotation Campaign.

5. Issues Management

As already mentioned previously, within an Annotation Campaign, any Assignee can create an Issue while completing its Task. The Issue system aims at facilitating communication between the different Assignees for a given Asset along its transition through the Annotation Workflow.

Once an Issue is created the different Workers can chat and comment on this one until it is closed. All the open and closed Issues related to an Asset are accessible by authorized Workers at any step of the Workflow.

Please note that the Issue system is only available for DatasetVersion with an Annotation Campaign ongoing.

There are two types of Issues:

• Zone issue: Related to a specific area of the Asset
• Image Issue: Related to the whole Asset

A. Create a Zone Issue

This Issue can be related to the whole Asset or to a specific zone on the Asset.

When completing a Task on the Annotation tool, as an Assignee you can create a Zone Issue by clicking on the Create a zone Issue button as shown below. Then draw on the image a rectangle on the zone affected by the Issue you want to raise.

Please note that any Worker who can access the Task (even without being the Assignee) can create a Zone Issue on the Asset.

Once the zone has been created, the Issues Panel will open in the right panel of the Annotation Tool. You can then describe your Zone Issue and click on Create Issue.

Once created, your Issue will appear on the Issues panel related to the current Asset.

B. Create an Image Issue

An Image Issue is not related to a specific area but to the whole Asset.

An Image Issue can be created directly from the Issues Panel as shown below:

Once the description is prompted the Image Issue will be created after clicking on Create Issue. This new Issue will be also accessible at any time from the Issues panel .

Please note also that the Assignee will be forced to create an Image Issue before Skipping or sending an Asset for Rework if nor Image or Zone Issue is existing on the current Asset. There is a way to track this Asset has been Skipped or sent To Rework.

As it is the case for Zone issues, from this Issue Panel, Workers can comment, resolve or re-open the Image Issue.

C. Issues panel

After opening any Asset from the Tasks list, you can access the Issues panel as shown below on the right side of the Annotation Tool.

Here are displayed all the Issues (Zone or Image) related to the current Asset created during the Annotation Campaign. You can switch between the still-open Issues and the closed ones.

By clicking on any Issue (Open or Closed), you will access the conversation thread allowing you to see the conversation related to this issue, add a Reply, Resolve it, or Re-Open it.

Please note that any Worker that can access the Task (even without being the Assignee) can interact with an Issue in the Issue Panel.

Please note that any Issue opened will remain until a Worker with the necessary rights closes it.

👍

An Asset cannot turn in Done status with still open Issues

To make an Asset turn in the Done status, meaning submitting the Asset at the last step of the Workflow, all the Issues related should be closed. In a few words, you won't be able to submit the very last Task of the Workflow is some Issues are still open.

6. The Labeller role

In case you need to involve some members in your Annotation Campaign but willing them only to complete the Tasks they get attributed without letting them visualize or manipulate any other object related to your Organization, you can add them as Unpriviliedged and add them with the Labeler role in the Dataset.

a. Add a member as Labeler in your Dataset

To involve a member in your Annotation Campaign and let him complete the assigned Tasks without accessing any other object of your Organization, you first need to add this member with the Unpriviledged role to your Organization.

To do so, you need to be Admin in the current Organization, go to Organization Settings > Members and click on Inviter member.

You can add members using their username if they already have a Picsellia account or by typing their email address (in this case, they will get an email with a link to create their account), then you need to select Unpriviliedged as Default Permission and click on Invite.

Then to make sure you can involve the invited Unprivileged member in an Annotation Campaign, you need to add him to the Workers list of the Dataset as shown below:

Once done the Unprivildged member is identified as Labeler in the Dataset and can be involved in Annotation Campaign for any DatasetVersion related.

B. Access an Organization as an Unprivileged member

The members invited to an Organization with Unprivilideged rights, can access the Organization but by default, they won’t be able to access any object (Datalake, Dataset, Project, Model…). They will only be able to visualize the My Tasks view.

This view is only accessible to members with the Unpriviledged role in the current Organization. From this one, the Unpriviledged member will see the list of Tasks assigned to him and nothing else.

The Labeler will see all the Tasks assigned within the current Organization grouped by Annotation Campaign. Each Annotation Campaign is identified by its DatasetVersion related. The Labeler can visualize all the Tasks that are currently assigned to him grouped by Annotation Campaign, the name of the current step that generated this Task, and the number of issues on the related Asset.

To complete a Task as Labeler, you can click on Go to task and you can complete your Task (Annotation or Review) as detailed here.

The Labeler can also access a detailed view describing all the Issues raised on any Asset that generates a Task currently assigned to the Labeler.

📘

Labelers can manage Issues, read instructions and access Asset history

For any Asset that generate a Task currently assigned to a given Labeler, this one can access all the Issues management features (Open, Reply, Resolve, Re-open).

While completing a Task the Labeler can also access the history of the Asset int he Annotation Workflow and open the Annotation guidelines file if any.

7. Analytics

Workers who are not Labelers can access the Analytics Dashboard related to the current DatasetVersion in the Analytics tab.

If an Annotation Campaign has been set, the three following tabs are available:

• Throughput: This tab will through its metrics detail the progress of the Annotation Campaign.
• Efficiency: This tab will through its metrics detail the time spent by the Assignees on the Annotation Campaign.
• Quality: This tab will detail the DatasetVersion with statistical metrics based on the Annotation created until now.

📘

Only Quality Analytics are availble out of an Annotation Campaign.

A. Throughput

The Throughput tab displays graphs that are related to the progress of the Annotation Campaign. From this tab, you will have a clear view of how the Asset of the DatasetVersion are going through the Annotation Workflow in real-time.

In detail, the following metrics are displayed:

• Annotation Campaign Progress: A percentage of Annotation Campaign completion based on the Tasks already submitted compared to the total number of Tasks to submit to complete the Annotation Campaign
• Tasks Distribution: A pie chart showing the total number of active Tasks grouped by step.
• Total Annotation Assignments: A bar chart showing the total number of Annotation Tasks submitted. The time range depends on the selected filter.
• Total Review Assignments: A bar chart showing the total number of Review Tasks submitted. The time range depends on the selected filter.
• Assignment Count By Worker: A bar chart detailing the total number of Tasks currently to each Worker involved in the Annotation Campaign.
• Issues Quantity: A bar chart detailing the total number of Issues created until now during the Annotation Campaign grouped by Creation Date. The time range depends on the selected filter as explained below.
• Issues Type Distribution: A pie chart detailing the total number of Issues created until now during the Annotation Campaign grouped by Issues Type(Zone or Image). The time range depends on the selected filter.

For each chart, you can change the Date Range of the data displayed as shown below:

In addition, you can filter the data displayed based on a given step of the Annotation Workflow or by Assignee.

B. Efficiency

The Efficiency tab displays graphs that are related to the time dedicated to the different steps of the Annotation Campaign in real-time.

In detail, the following metrics are displayed:

• Task Duration: Globalized time information on all the Tasks, the average time to complete a Task, and the minimum and maximum time of Task completion on the whole current Annotation Campaign.
• Closed Issues: The total number of Issues closed during the current Annotation Campaign.
• Opened Issues: The total number of Issues opened during the current Annotation Campaign
• Total Annotation Time: The total time spent on Annotation Tasks until now. A bar chart displays this duration on the selected Date Range based on the filter.
• Average Annotation Time: The average time spent on one Annotation Task until now. A bar chart displays this duration on the selected Date Range based on the filter.
• Total Review Time: The total time spent on Review Tasks until now. A bar chart displays this duration on the selected Date Range based on the filter.
• Average Review Time: The average time spent on one Review Task until now. A bar chart displays this duration on the selected Date Range based on the filter.

For each chart, you can change the Date Range of the data displayed as shown below:

In addition, you can filter the data displayed based on a given step of the Annotation Workflow or by Assignee.

C. Quality

The Quality tab displays graphs that are mainly related to the Annotation of the Asset of the DatasetVersion. They are useful for assessing in real-time, the quality, diversity, and quantity of the current DatasetVersion.

In detail, the following metrics are displayed:

• Data Distribution: This bar chart displays the repartition of Shape per Label.
• Objects by Image Distribution: This bar chart displays the repartition of Asset per number of Shape in the related Annotation.
• Object areas Distribution: This bar chart displays the repartition of Shape per area of Shape.
• Image areas Distribution: This bar chart displays the repartition of Asset per area of Asset.