Dataset - Annotation Campaign

An Annotation Campaign allows you to handle properly and efficiently the process of annotating a DatasetVersion on Picsellia.

By using Annotation Campaign on Picsellia, you will be able to:

• Define your Annotation Workflow
• Assign Annotation Tasks
• Review the Annotation done and raise issues if something is wrong
• Track the progress of the Annotation Campaign

1. Create an Annotation Campaign

First of all, it is important to note that each DatasetVersion can have up to one Annotation Campaign.

To initialize an Annotation Campaign on a DatasetVersion, go to the Settings tab, and click on Workflow in the Campaign sub-section.

A. Create an Annotation Workflow

Here, you will be able to define the Annotation Workflow that defines the steps your Asset will follow during the Annotation Campaign you are about to launch on this DatasetVersion. The Annotation Workflow represents all the steps the Asset will go through before being considered properly annotated. As you can see above, the Workflow is initially free of steps, so you will now be able to define and configure new steps before launching the Annotation Campaign.

To do so, we just need to click on the + button next to the initial step of your Annotation Workflow. A new step will be added to your Annotation Workflow, it is now time to configure it.

Once the new step is created, its related configuration panel will pop up on the right side, allowing you to give it a name, define its type among Annotation or Review, and select the Assignees that will get Tasks attributed as part of of this step. By clicking on Save step, this one will be updated and added to the Annotation Workflow you are creating.

📘

Select Assignees

The potential Assignees are the members of the current Organization that are listed are Workers in the Dataset.

So if a member is not part of the Assignee list, you should add him as Worker in the related Dataset.

Once created, you can edit the step anytime by clicking on it and accessing the configuration panel.

👍

The Unpriviledged role

If you want some members to only have access to the Tasks they are supposed to do within Annotation Campaigns running in your Organization and nothing else, you can set them as Unpriviledged members. More details are available here.

Then, you can add as many steps as you want to create the Annotation Workflow that suits your needs.

For instance, a simple and efficient Annotation Workflow would be to have an Annotation step and then a Review step with a part of your team assigned to Annotation and some others to the Review, as shown below:

📘

A Worker can be assigned to several steps of an Annotation Workflow

B. Configure your Annotation Campaign

Now that your Annotation Workflow is defined, you can configure the way your Annotation Campaign will be initialized by clicking on the initial step named Dataset Version, a configuration panel will open allowing you to:

• Define the expected behavior if new Asset are added to the DatasetVersion while the Annotation Campaign is ongoing
• The attribution policy among Assignees at each step
• Define what needs to be done with the potentially already existing Annotation in the current DatasetVersion.

Regarding the Pre-existing annotation strategy, you'll have 3 options:

• Back to annotation, it means that the Asset will be sent to the first Annotation step without any Annotation. At the end of the Annotation Workflow, the Annotation created during the Annotation Campaign will override the pre-existing one.
• First Quality check, it means that the Asset will directly be sent to the first Review step of the Annotation Workflow but with the pre-existing Annotation considered as the Annotation to be reviewed by the Assignees. The idea here is to skip the first Annotation step for already annotated Asset.
• Done, it means that the already annotated Asset are directly sent to the end of the Annotation Workflow, as soon as the Annotation Campaign is launched, the already annotated Asset will be considered as annotated during this AnnotationCampaign.

The last configuration step lies in the panel accessible through the settings button as shown below:

From this panel, you can first upload in the Instructions section a pdf file which will be visualizable by all the Assignees of the current Annotation Campaign while doing the Tasks they get assigned to. This pdf could, for instance, list all the guidelines and instructions the Workers need to be aware of for properly annotating the DatasetVersion.

This settings panel also allows you to:

• Define a due date for the Annotation Campaign which will allow us to notify you when the end of the Campaign is near or even out of date
• Choose to disable new Asset from being added to the Annotation Campaign when all the initial Tasks are done.

C. Launch the Annotation Campaign

At the end of the configuration phase of your Annotation Campaign, all the Asset of your DatasetVersion are at the initial step, by clicking on Launch campaign , the Annotation Campaign will start and the Asset will start going through the Annotation Workflow according to the policies defined previously.

When an Annotation Campaign is launched, the Asset will transition to the first step and the related Tasks will be generated for the Assignees. In addition, two new tabs will be created, Tasks and Issues they will allow you and the other Organization members to go through this ongoing Annotation Campaign.

2. Access the Tasks list

As soon as an Annotation Campaign is ongoing on a DatasetVersion, you will be able to access a new tab called Tasks.

🚧

Who can access the Tasks tab ?

This Tasks tab is available for every worker in your Dataset except the ones with the Labelerrole (those added as Unprivileged to your Organization).

The members with the role Unpriviledged in the Organization, that have been added in the Workers list of the Dataset can only visualize the Tasks that are assigned to them in a dedicated view as detailed here.

The Tasks list allows you to visualize the current status and step of any Asset that is currently going through the Annotation Campaign.

This view allows you to filter the Tasks to display. You can filter by the Task status, by the current step of the Annotation Workflow, or by the current Assignee.

A. Task Status

As shown below each Task created has a current step and Assignee, those are defined while creating the Annotation Workflow as described here. However, we can also notice the presence of a new field named Status for each Task.

This Status field describes the status of the Asset journey in the Annotation Workflow. The Status of a Task can either be:

• In Progress: Everything is going well until now, the associated Asset is going through the Annotation Workflow normally. However, the Asset is still at an Annotation or Review step reflected by the In Progress status. The Annotation Workflow isn't over yet.
• To Rework: A Task with the to Rework status is an Annotation Task created when an Asset has been annotated during a first Annotation task and submitted as To Rework during the next Review step. Indeed, the Assignee of this Review step didn't accept the proposed Annotation. So he sent the Asset back to the previous step in the Annotation Workflow. Usually, the reason is described in an Issue. As a consequence, a new Task with To Rework status has been re-created at the previous Annotation step of the Annotation Workflow and re-assigned to the Worker that created the rejected Annotation. This Assignee can then modify it's first Annotation by completing this To Rework task and sending it back for the second time to the following Review step.
• Done: The Asset has been through the whole Annotation Workflow, the Asset can be considered as annotated
• Skipped: While completing an Annotation or Review task, the Assignee has the possibility to Skip the Task, this will kick the associated Asset out of the Annotation Workflow. There is no Task to be completed on this Asset anymore but it cannot be considered as properly annotated. Usually, the reason is described in an Issue.

B. Reassign a Task

As described in detail here, a Task can only be completed by the Assignee. If for some reason, a Task needs to be completed by someone else, this one can be reassigned for completion to a member that is identified as Worker on the Dataset.

If you have sufficient rights, it can be done for a particular Task, by clicking from the Task list on the Assignee, the list of available Workers will then be displayed. You just need to click on the new Assignee as shown below:

If you have sufficient rights, it can be done for a bunch of Tasks, by selecting the to-be-reassigned Tasks and clicking on the Reassign button. Then you simply need to select from the list of available Workers the one that will get the selected Tasks reassigned.

🚧

A filter on the steps should be active

For the reassignment of several Tasks, you must ensure that a filter on the Task step is active on the Task list before reassigning.

3. Complete a Task

Now that we can visualize, understand, and manipulate the Tasks created in the frame of the current Annotation Campaign, it is high time to start completing them to make our Asset go through the Annotation Workflow.

📘

Who can complete a Task?

The only member able to complete a Task is the Assignee.
So if you want to complete a Task without being the Assignee, you need to reassign it first as explained here.

Whatever the Task type (Annotation or Review) to complete it you need to click on the Go to task button as shown below, the Annotation Tool will then open allowing you to perform the Annotation or the Review required by the Task. The Asset can be free of Annotation or in case the Annotation has already been initialized this one will be displayed as submitted at the previous step of the Annotation Workflow.

A. Complete an Annotation task

In case you need to complete an Annotation Task the Annotation Tool will open displaying the Asset to be annotated. You can rely on the documentation of the Annotation Tool here to create the Shape that will compose the Annotation.

Once you have created or edited the Shape and considered the Annotation satisfying, you can Submit the Task as shown below.

Submitting a Task will save it and transition it to the next step of the Annotation Workflow.
Please note that you can also just Save the Annotation and submit the Task later on.

However, as you are annotation within an Annotation Campaign, the Annotation Tool has some additional features dedicated to Annotation Campaigns.

Please note that any Worker who’s not a Labeler and who has access to the DatasetVersion can click on the Go to task button and open the related Asset in the Annotation Tool, but only the Assignee of the Task can create, edit Annotation and submit the Task.

a. Accessing the history of the Asset

From the Annotation Tool you can visualize the history of the Asset during the Annotation Workflow as shown below:

You can see all the steps done or to come of the Annotation Workflow. The current step is highlighted in orange. For each step are displayed, the Assignee and the submission date.

In addition by clicking on any step, you can visualize the Annotation submitted at this step allowing you to see and track how it evolved going through the Annotation Workflow.

b. Creating Issues

To ease the communication between the different workers involved at the different steps of the Workflow, Issues can be raised on the whole Asset or a zone of the Asset.

Issues can for instance be used when a Worker doubts how to annotate an image to request advice from the Worker in charge of the Review. It can also be used to track the justification why an Asset has been skipped and sent back to an Annotation step. More details on the usage of Issues during an Annotation Campaign are available here.

c. Visualizing the Annotation Instructions

As explained here, during the Annotation Campaign initialization, a pdf file can be attached.
This can be used to share the annotation guidelines with all the Workers who will be involved in the Annotation Campaign. During the completion of a Task, a Worker can open the pdf file as shown below:

B. Complete a Review task

In case you need to complete a Review Task, the Annotation Tool will open displaying the Asset, the related Annotation to be reviewed and the potential Issues created previously during the Annotation Workflow.

During a Review Task, the Assignee has to review the Annotation done during the previous Annotation step.

During a Review Task, 4 scenarios can occur:

• The Annotation is correct according to the Assignee, the Task can be submitted as it is by clicking on the Submit button
• The Annotation is not satisfying and the Assignee wants to edit it himself. The Assignee can then use the Annotation Tool to edit the Annotation. Once the Annotation is considered satisfactory, the Assignee can click on Submit to transition the Asset to the potential next step of the Workflow.
• The Annotation is not satisfying and the Assignee wants to send the Asset back to the previous Annotation step allowing the previous Assignee to rework the Annotation. Before sending it back, the Assignee will have to create an Issue (Zone or Image Issue) to explain why the Asset is sent back to the previous step, once done, the Assignee of the Review task can click on To Rework, and a new Task will be created and assigned at the previous step of the Workflow.
• The Assignee of the Review Task considers that the Asset should not keep going through the Annotation Workflow. In this case, the Assignee can click on Skip, then the Assignee will be requested to create an Issue to explain why this one should be skipped.

During the Review step while completing its Task, the Assignee can answer or close the Issues created previously by other Workers and create new Issues that will be handled by the next Assignee (either further in the Workflow if the Asset is Submitted or upstream if the Asset is sent to rework).

4. End of the Annotation Workflow

Once an Asset has been through all the steps of the Annotation Workflow, the Asset will be displayed in the Tasks list with the Done status as shown below:

You can also retrieve all the Asset that have been skipped with the Skipped status:

The Annotation Campaign is considered as over once the target date is reached or once all the Tasks are in Done or Skipped status.

5. Issues Management

As already mentioned previously, within an Annotation Campaign, any Assignee can create an Issue while completing its Task. The Issue system aims at facilitating communication between the different Assignees for a given Asset along its transition through the Annotation Workflow.

Once an Issue is created the different Workers can chat and comment on this one until it is closed. All the open and closed Issues related to an Asset are accessible by authorized Workers at any step of the Workflow.

Please note that the Issue system is only available for DatasetVersion with an Annotation Campaign ongoing.

There are two types of Issues:

• Zone issue: Related to a specific area of the Asset
• Image Issue: Related to the whole Asset

A. Create a Zone Issue

This Issue can be related to the whole Asset or to a specific zone on the Asset.

When completing a Task on the Annotation tool, as an Assignee you can create a Zone Issue by clicking on the Create a zone Issue button as shown below. Then draw on the image a rectangle on the zone affected by the Issue you want to raise.

Please note that any Worker who can access the Task (even without being the Assignee) can create a Zone Issue on the Asset.

Once the zone has been created, the Issues Panel will open in the right panel of the Annotation Tool. You can then describe your Zone Issue and click on Create Issue.

Once created, your Issue will appear on the Issues panel related to the current Asset.

B. Create an Image Issue

An Image Issue is not related to a specific area but to the whole Asset.

An Image Issue can be created directly from the Issues Panel as shown below:

Once the description is prompted the Image Issue will be created after clicking on Create Issue. This new Issue will be also accessible at any time from the Issues panel .

Please note also that the Assignee will be forced to create an Image Issue before Skipping or sending an Asset for Rework if nor Image or Zone Issue is existing on the current Asset. There is a way to track this Asset has been Skipped or sent To Rework.

As it is the case for Zone issues, from this Issue Panel, Workers can comment, resolve or re-open the Image Issue.

C. Issues panel

After opening any Asset from the Tasks list, you can access the Issues panel as shown below on the right side of the Annotation Tool.

Here are displayed all the Issues (Zone or Image) related to the current Asset created during the Annotation Campaign. You can switch between the still-open Issues and the closed ones.

By clicking on any Issue (Open or Closed), you will access the conversation thread allowing you to see the conversation related to this issue, add a Reply, Resolve it, or Re-Open it.

Please note that any Worker that can access the Task (even without being the Assignee) can interact with an Issue in the Issue Panel.

Please note that any Issue opened will remain until a Worker with the necessary rights closes it.

👍

An Asset cannot turn in Done status with still open Issues

To make an Asset turn in the Done status, meaning submitting the Asset at the last step of the Workflow, all the Issues related should be closed. In a few words, you won't be able to submit the very last Task of the Workflow is some Issues are still open.

6. The Labeller role

In case you need to involve some members in your Annotation Campaign but willing them only to complete the Tasks they get attributed without letting them visualize or manipulate any other object related to your Organization, you can add them as Unpriviliedged and add them with the Labeler role in the Dataset.

a. Add a member as Labeler in your Dataset

To involve a member in your Annotation Campaign and let him complete the assigned Tasks without accessing any other object of your Organization, you first need to add this member with the Unpriviledged role to your Organization.

To do so, you need to be Admin in the current Organization, go to Organization Settings > Members and click on Inviter member.

You can add members using their username if they already have a Picsellia account or by typing their email address (in this case, they will get an email with a link to create their account), then you need to select Unpriviliedged as Default Permission and click on Invite.

Then to make sure you can involve the invited Unprivileged member in an Annotation Campaign, you need to add him to the Workers list of the Dataset as shown below:

Once done the Unprivildged member is identified as Labeler in the Dataset and can be involved in Annotation Campaign for any DatasetVersion related.

B. Access an Organization as an Unprivileged member

The members invited to an Organization with Unprivilideged rights, can access the Organization but by default, they won’t be able to access any object (Datalake, Dataset, Project, Model…). They will only be able to visualize the My Tasks view.

This view is only accessible to members with the Unpriviledged role in the current Organization. From this one, the Unpriviledged member will see the list of Tasks assigned to him and nothing else.

The Labeler will see all the Tasks assigned within the current Organization grouped by Annotation Campaign. Each Annotation Campaign is identified by its DatasetVersion related. The Labeler can visualize all the Tasks that are currently assigned to him grouped by Annotation Campaign, the name of the current step that generated this Task, and the number of issues on the related Asset.

To complete a Task as Labeler, you can click on Go to task and you can complete your Task (Annotation or Review) as detailed here.

The Labeler can also access a detailed view describing all the Issues raised on any Asset that generates a Task currently assigned to the Labeler.

📘

Labelers can manage Issues, read instructions and access Asset history

For any Asset that generate a Task currently assigned to a given Labeler, this one can access all the Issues management features (Open, Reply, Resolve, Re-open).

While completing a Task the Labeler can also access the history of the Asset int he Annotation Workflow and open the Annotation guidelines file if any.

7. Analytics

Workers who are not Labelers can access the Analytics Dashboard related to the current DatasetVersion in the Analytics tab.

If an Annotation Campaign has been set, the three following tabs are available:

• Throughput: This tab will through its metrics detail the progress of the Annotation Campaign.
• Efficiency: This tab will through its metrics detail the time spent by the Assignees on the Annotation Campaign.
• Quality: This tab will detail the DatasetVersion with statistical metrics based on the Annotation created until now.

📘

Only Quality Analytics are availble out of an Annotation Campaign.

A. Throughput

The Throughput tab displays graphs that are related to the progress of the Annotation Campaign. From this tab, you will have a clear view of how the Asset of the DatasetVersion are going through the Annotation Workflow in real-time.

In detail, the following metrics are displayed:

• Annotation Campaign Progress: A percentage of Annotation Campaign completion based on the Tasks already submitted compared to the total number of Tasks to submit to complete the Annotation Campaign
• Tasks Distribution: A pie chart showing the total number of active Tasks grouped by step.
• Total Annotation Assignments: A bar chart showing the total number of Annotation Tasks submitted. The time range depends on the selected filter.
• Total Review Assignments: A bar chart showing the total number of Review Tasks submitted. The time range depends on the selected filter.
• Assignment Count By Worker: A bar chart detailing the total number of Tasks currently to each Worker involved in the Annotation Campaign.
• Issues Quantity: A bar chart detailing the total number of Issues created until now during the Annotation Campaign grouped by Creation Date. The time range depends on the selected filter as explained below.
• Issues Type Distribution: A pie chart detailing the total number of Issues created until now during the Annotation Campaign grouped by Issues Type(Zone or Image). The time range depends on the selected filter.

For each chart, you can change the Date Range of the data displayed as shown below:

In addition, you can filter the data displayed based on a given step of the Annotation Workflow or by Assignee.

B. Efficiency

The Efficiency tab displays graphs that are related to the time dedicated to the different steps of the Annotation Campaign in real-time.

In detail, the following metrics are displayed:

• Task Duration: Globalized time information on all the Tasks, the average time to complete a Task, and the minimum and maximum time of Task completion on the whole current Annotation Campaign.
• Closed Issues: The total number of Issues closed during the current Annotation Campaign.
• Opened Issues: The total number of Issues opened during the current Annotation Campaign
• Total Annotation Time: The total time spent on Annotation Tasks until now. A bar chart displays this duration on the selected Date Range based on the filter.
• Average Annotation Time: The average time spent on one Annotation Task until now. A bar chart displays this duration on the selected Date Range based on the filter.
• Total Review Time: The total time spent on Review Tasks until now. A bar chart displays this duration on the selected Date Range based on the filter.
• Average Review Time: The average time spent on one Review Task until now. A bar chart displays this duration on the selected Date Range based on the filter.

For each chart, you can change the Date Range of the data displayed as shown below:

In addition, you can filter the data displayed based on a given step of the Annotation Workflow or by Assignee.

C. Quality

The Quality tab displays graphs that are mainly related to the Annotation of the Asset of the DatasetVersion. They are useful for assessing in real-time, the quality, diversity, and quantity of the current DatasetVersion.

In detail, the following metrics are displayed:

• Data Distribution: This bar chart displays the repartition of Shape per Label.
• Objects by Image Distribution: This bar chart displays the repartition of Asset per number of Shape in the related Annotation.
• Object areas Distribution: This bar chart displays the repartition of Shape per area of Shape.
• Image areas Distribution: This bar chart displays the repartition of Asset per area of Asset.