Dataset - Annotation Campaign
An Annotation Campaign allows you to handle properly and efficiently the process of annotating a DatasetVersion
on Picsellia.
By using Annotation Campaign on Picsellia, you will be able to:
- Define your Annotation Workflow
- Assign Annotation Tasks
- Review the
Annotation
done and raise issues if something is wrong - Track the progress of the Annotation Campaign
1. Create an Annotation Campaign
First of all, it is important to note that each DatasetVersion
can have up to one Annotation Campaign.
To initialize an Annotation Campaign on a DatasetVersion
, go to the Settings tab, and click on Workflow in the Campaign sub-section.
A. Create an Annotation Workflow
Here, you will be able to define the Annotation Workflow that defines the steps your Asset
will follow during the Annotation Campaign you are about to launch on this DatasetVersion
. The Annotation Workflow represents all the steps the Asset
will go through before being considered properly annotated. As you can see above, the Workflow is initially free of steps, so you will now be able to define and configure new steps before launching the Annotation Campaign.
To do so, we just need to click on the + button next to the initial step of your Annotation Workflow. A new step will be added to your Annotation Workflow, it is now time to configure it.
Once the new step is created, its related configuration panel will pop up on the right side, allowing you to give it a name, define its type among Annotation or Review, and select the Assignees that will get Tasks attributed as part of of this step. By clicking on Save step, this one will be updated and added to the Annotation Workflow you are creating.
Select Assignees
The potential Assignees are the members of the current Organization that are listed are Workers in the
Dataset
.So if a member is not part of the Assignee list, you should add him as Worker in the related
Dataset
.
Once created, you can edit the step anytime by clicking on it and accessing the configuration panel.
The Unpriviledged role
If you want some members to only have access to the Tasks they are supposed to do within Annotation Campaigns running in your Organization and nothing else, you can set them as Unpriviledged members. More details are available here.
Then, you can add as many steps as you want to create the Annotation Workflow that suits your needs.
For instance, a simple and efficient Annotation Workflow would be to have an Annotation step and then a Review step with a part of your team assigned to Annotation and some others to the Review, as shown below:
A Worker can be assigned to several steps of an Annotation Workflow
B. Configure your Annotation Campaign
Now that your Annotation Workflow is defined, you can configure the way your Annotation Campaign will be initialized by clicking on the initial step named Dataset Version, a configuration panel will open allowing you to:
- Define the expected behavior if new
Asset
are added to theDatasetVersion
while the Annotation Campaign is ongoing - The attribution policy among Assignees at each step
- Define what needs to be done with the potentially already existing
Annotation
in the currentDatasetVersion
.
Regarding the Pre-existing annotation strategy, you'll have 3 options:
- Back to annotation, it means that the
Asset
will be sent to the first Annotation step without anyAnnotation
. At the end of the Annotation Workflow, theAnnotation
created during the Annotation Campaign will override the pre-existing one. - First Quality check, it means that the
Asset
will directly be sent to the first Review step of the Annotation Workflow but with the pre-existingAnnotation
considered as theAnnotation
to be reviewed by the Assignees. The idea here is to skip the first Annotation step for already annotatedAsset
. - Done, it means that the already annotated
Asset
are directly sent to the end of the Annotation Workflow, as soon as the Annotation Campaign is launched, the already annotatedAsset
will be considered as annotated during this AnnotationCampaign.
The last configuration step lies in the panel accessible through the settings button as shown below:
From this panel, you can first upload in the Instructions section a pdf file which will be visualizable by all the Assignees of the current Annotation Campaign while doing the Tasks they get assigned to. This pdf could, for instance, list all the guidelines and instructions the Workers need to be aware of for properly annotating the DatasetVersion
.
This settings panel also allows you to:
- Define a due date for the Annotation Campaign which will allow us to notify you when the end of the Campaign is near or even out of date
- Choose to disable new
Asset
from being added to the Annotation Campaign when all the initial Tasks are done.
C. Launch the Annotation Campaign
At the end of the configuration phase of your Annotation Campaign, all the Asset
of your DatasetVersion
are at the initial step, by clicking on Launch campaign , the Annotation Campaign will start and the Asset
will start going through the Annotation Workflow according to the policies defined previously.
When an Annotation Campaign is launched, the Asset
will transition to the first step and the related Tasks will be generated for the Assignees. In addition, two new tabs will be created, Tasks and Issues they will allow you and the other Organization members to go through this ongoing Annotation Campaign.
2. Access the Tasks list
As soon as an Annotation Campaign is ongoing on a DatasetVersion
, you will be able to access a new tab called Tasks.
Who can access the Tasks tab ?
This Tasks tab is available for every worker in your
Dataset
except the ones with theLabeler
role (those added as Unprivileged to your Organization).The members with the role Unpriviledged in the Organization, that have been added in the Workers list of the
Dataset
can only visualize the Tasks that are assigned to them in a dedicated view as detailed here.
The Tasks list allows you to visualize the current status and step of any Asset
that is currently going through the Annotation Campaign.
This view allows you to filter the Tasks to display. You can filter by the Task status, by the current step of the Annotation Workflow, or by the current Assignee.
A. Task Status
As shown below each Task created has a current step and Assignee, those are defined while creating the Annotation Workflow as described here. However, we can also notice the presence of a new field named Status for each Task.
This Status field describes the status of the Asset
journey in the Annotation Workflow. The Status of a Task can either be:
- In Progress: Everything is going well until now, the associated
Asset
is going through the Annotation Workflow normally. However, the Asset is still at an Annotation or Review step reflected by the In Progress status. The Annotation Workflow isn't over yet. - To Rework: A Task with the to Rework status is an Annotation Task created when an
Asset
has been annotated during a first Annotation task and submitted as To Rework during the next Review step. Indeed, the Assignee of this Review step didn't accept the proposedAnnotation
. So he sent theAsset
back to the previous step in the Annotation Workflow. Usually, the reason is described in an Issue. As a consequence, a new Task with To Rework status has been re-created at the previous Annotation step of the Annotation Workflow and re-assigned to the Worker that created the rejectedAnnotation
. This Assignee can then modify it's firstAnnotation
by completing this To Rework task and sending it back for the second time to the following Review step. - Done: The
Asset
has been through the whole Annotation Workflow, theAsset
can be considered as annotated - Skipped: While completing an Annotation or Review task, the Assignee has the possibility to Skip the Task, this will kick the associated
Asset
out of the Annotation Workflow. There is no Task to be completed on thisAsset
anymore but it cannot be considered as properly annotated. Usually, the reason is described in an Issue.
B. Reassign a Task
As described in detail here, a Task can only be completed by the Assignee. If for some reason, a Task needs to be completed by someone else, this one can be reassigned for completion to a member that is identified as Worker on the Dataset
.
If you have sufficient rights, it can be done for a particular Task, by clicking from the Task list on the Assignee, the list of available Workers will then be displayed. You just need to click on the new Assignee as shown below:
If you have sufficient rights, it can be done for a bunch of Tasks, by selecting the to-be-reassigned Tasks and clicking on the Reassign button. Then you simply need to select from the list of available Workers the one that will get the selected Tasks reassigned.
A filter on the steps should be active
For the reassignment of several Tasks, you must ensure that a filter on the Task step is active on the Task list before reassigning.
3. Complete a Task
Now that we can visualize, understand, and manipulate the Tasks created in the frame of the current Annotation Campaign, it is high time to start completing them to make our Asset
go through the Annotation Workflow.
Who can complete a Task?
The only member able to complete a Task is the Assignee.
So if you want to complete a Task without being the Assignee, you need to reassign it first as explained here.
Whatever the Task type (Annotation or Review) to complete it you need to click on the Go to task button as shown below, the Annotation Tool will then open allowing you to perform the Annotation or the Review required by the Task. The Asset
can be free of Annotation
or in case the Annotation
has already been initialized this one will be displayed as submitted at the previous step of the Annotation Workflow.
A. Complete an Annotation task
In case you need to complete an Annotation Task the Annotation Tool will open displaying the Asset
to be annotated. You can rely on the documentation of the Annotation Tool here to create the Shape
that will compose the Annotation
.
Once you have created or edited the Shape
and considered the Annotation
satisfying, you can Submit the Task as shown below.
Submitting a Task will save it and transition it to the next step of the Annotation Workflow.
Please note that you can also just Save the Annotation
and submit the Task later on.
However, as you are annotation within an Annotation Campaign, the Annotation Tool has some additional features dedicated to Annotation Campaigns.
Please note that any Worker whoβs not a Labeler and who has access to the DatasetVersion
can click on the Go to task button and open the related Asset
in the Annotation Tool, but only the Assignee of the Task can create, edit Annotation
and submit the Task.
a. Accessing the history of the Asset
Asset
From the Annotation Tool you can visualize the history of the Asset
during the Annotation Workflow as shown below:
You can see all the steps done or to come of the Annotation Workflow. The current step is highlighted in orange. For each step are displayed, the Assignee and the submission date.
In addition by clicking on any step, you can visualize the Annotation
submitted at this step allowing you to see and track how it evolved going through the Annotation Workflow.
b. Creating Issues
To ease the communication between the different workers involved at the different steps of the Workflow, Issues can be raised on the whole Asset
or a zone of the Asset
.
Issues can for instance be used when a Worker doubts how to annotate an image to request advice from the Worker in charge of the Review. It can also be used to track the justification why an Asset
has been skipped and sent back to an Annotation step. More details on the usage of Issues during an Annotation Campaign are available here.
c. Visualizing the Annotation Instructions
As explained here, during the Annotation Campaign initialization, a pdf file can be attached.
This can be used to share the annotation guidelines with all the Workers who will be involved in the Annotation Campaign. During the completion of a Task, a Worker can open the pdf file as shown below:
B. Complete a Review task
In case you need to complete a Review Task, the Annotation Tool will open displaying the Asset
, the related Annotation
to be reviewed and the potential Issues created previously during the Annotation Workflow.
During a Review Task, the Assignee has to review the Annotation
done during the previous Annotation step.
During a Review Task, 4 scenarios can occur:
- The
Annotation
is correct according to the Assignee, the Task can be submitted as it is by clicking on the Submit button - The
Annotation
is not satisfying and the Assignee wants to edit it himself. The Assignee can then use the Annotation Tool to edit the Annotation. Once theAnnotation
is considered satisfactory, the Assignee can click on Submit to transition theAsset
to the potential next step of the Workflow. - The
Annotation
is not satisfying and the Assignee wants to send theAsset
back to the previous Annotation step allowing the previous Assignee to rework theAnnotation
. Before sending it back, the Assignee will have to create an Issue (Zone or Image Issue) to explain why theAsset
is sent back to the previous step, once done, the Assignee of the Review task can click on To Rework, and a new Task will be created and assigned at the previous step of the Workflow. - The Assignee of the Review Task considers that the
Asset
should not keep going through the Annotation Workflow. In this case, the Assignee can click on Skip, then the Assignee will be requested to create an Issue to explain why this one should be skipped.
During the Review step while completing its Task, the Assignee can answer or close the Issues created previously by other Workers and create new Issues that will be handled by the next Assignee (either further in the Workflow if the Asset
is Submitted or upstream if the Asset
is sent to rework).
4. End of the Annotation Workflow
Once an Asset
has been through all the steps of the Annotation Workflow, the Asset
will be displayed in the Tasks list with the Done status as shown below:
You can also retrieve all the Asset
that have been skipped with the Skipped status:
The Annotation Campaign is considered as over once the target date is reached or once all the Tasks are in Done or Skipped status.
5. Issues Management
As already mentioned previously, within an Annotation Campaign, any Assignee can create an Issue while completing its Task. The Issue system aims at facilitating communication between the different Assignees for a given Asset
along its transition through the Annotation Workflow.
Once an Issue is created the different Workers can chat and comment on this one until it is closed. All the open and closed Issues related to an Asset
are accessible by authorized Workers at any step of the Workflow.
Please note that the Issue system is only available for DatasetVersion
with an Annotation Campaign ongoing.
There are two types of Issues:
- Zone issue: Related to a specific area of the
Asset
- Image Issue: Related to the whole
Asset
A. Create a Zone Issue
This Issue can be related to the whole Asset
or to a specific zone on the Asset
.
When completing a Task on the Annotation tool, as an Assignee you can create a Zone Issue by clicking on the Create a zone Issue button as shown below. Then draw on the image a rectangle on the zone affected by the Issue you want to raise.
Please note that any Worker who can access the Task (even without being the Assignee) can create a Zone Issue on the Asset
.
Once the zone has been created, the Issues Panel will open in the right panel of the Annotation Tool. You can then describe your Zone Issue and click on Create Issue.
Once created, your Issue will appear on the Issues panel related to the current Asset
.
B. Create an Image Issue
An Image Issue is not related to a specific area but to the whole Asset
.
An Image Issue can be created directly from the Issues Panel as shown below:
Once the description is prompted the Image Issue will be created after clicking on Create Issue. This new Issue will be also accessible at any time from the Issues panel .
Please note also that the Assignee will be forced to create an Image Issue before Skipping or sending an Asset
for Rework if nor Image or Zone Issue is existing on the current Asset
. There is a way to track this Asset
has been Skipped or sent To Rework.
As it is the case for Zone issues, from this Issue Panel, Workers can comment, resolve or re-open the Image Issue.
C. Issues panel
After opening any Asset
from the Tasks list, you can access the Issues panel as shown below on the right side of the Annotation Tool.
Here are displayed all the Issues (Zone or Image) related to the current Asset
created during the Annotation Campaign. You can switch between the still-open Issues and the closed ones.
By clicking on any Issue (Open or Closed), you will access the conversation thread allowing you to see the conversation related to this issue, add a Reply, Resolve it, or Re-Open it.
Please note that any Worker that can access the Task (even without being the Assignee) can interact with an Issue in the Issue Panel.
Please note that any Issue opened will remain until a Worker with the necessary rights closes it.
An
Asset
cannot turn in Done status with still open IssuesTo make an
Asset
turn in the Done status, meaning submitting theAsset
at the last step of the Workflow, all the Issues related should be closed. In a few words, you won't be able to submit the very last Task of the Workflow is some Issues are still open.
6. The Labeller role
In case you need to involve some members in your Annotation Campaign but willing them only to complete the Tasks they get attributed without letting them visualize or manipulate any other object related to your Organization, you can add them as Unpriviliedged and add them with the Labeler role in the Dataset
.
a. Add a member as Labeler in your Dataset
Dataset
To involve a member in your Annotation Campaign and let him complete the assigned Tasks without accessing any other object of your Organization, you first need to add this member with the Unpriviledged role to your Organization.
To do so, you need to be Admin in the current Organization, go to Organization Settings > Members and click on Inviter member.
You can add members using their username if they already have a Picsellia account or by typing their email address (in this case, they will get an email with a link to create their account), then you need to select Unpriviliedged as Default Permission and click on Invite.
Then to make sure you can involve the invited Unprivileged member in an Annotation Campaign, you need to add him to the Workers list of the Dataset
as shown below:
Once done the Unprivildged member is identified as Labeler in the Dataset
and can be involved in Annotation Campaign for any DatasetVersion
related.
B. Access an Organization as an Unprivileged member
The members invited to an Organization with Unprivilideged rights, can access the Organization but by default, they wonβt be able to access any object (Datalake
, Dataset
, Project
, Model
β¦). They will only be able to visualize the My Tasks view.
This view is only accessible to members with the Unpriviledged role in the current Organization. From this one, the Unpriviledged member will see the list of Tasks assigned to him and nothing else.
The Labeler will see all the Tasks assigned within the current Organization grouped by Annotation Campaign. Each Annotation Campaign is identified by its DatasetVersion
related. The Labeler can visualize all the Tasks that are currently assigned to him grouped by Annotation Campaign, the name of the current step that generated this Task, and the number of issues on the related Asset
.
To complete a Task as Labeler, you can click on Go to task and you can complete your Task (Annotation or Review) as detailed here.
The Labeler can also access a detailed view describing all the Issues raised on any Asset
that generates a Task currently assigned to the Labeler.
Labelers can manage Issues, read instructions and access
Asset
historyFor any
Asset
that generate a Task currently assigned to a given Labeler, this one can access all the Issues management features (Open, Reply, Resolve, Re-open).While completing a Task the Labeler can also access the history of the
Asset
int he Annotation Workflow and open the Annotation guidelines file if any.
7. Analytics
Workers who are not Labelers can access the Analytics Dashboard related to the current DatasetVersion
in the Analytics tab.
If an Annotation Campaign has been set, the three following tabs are available:
- Throughput: This tab will through its metrics detail the progress of the Annotation Campaign.
- Efficiency: This tab will through its metrics detail the time spent by the Assignees on the Annotation Campaign.
- Quality: This tab will detail the
DatasetVersion
with statistical metrics based on the Annotation created until now.
Only Quality Analytics are availble out of an Annotation Campaign.
A. Throughput
The Throughput tab displays graphs that are related to the progress of the Annotation Campaign. From this tab, you will have a clear view of how the Asset of the DatasetVersion are going through the Annotation Workflow in real-time.
In detail, the following metrics are displayed:
- Annotation Campaign Progress: A percentage of Annotation Campaign completion based on the Tasks already submitted compared to the total number of Tasks to submit to complete the Annotation Campaign
- Tasks Distribution: A pie chart showing the total number of active Tasks grouped by step.
- Total Annotation Assignments: A bar chart showing the total number of Annotation Tasks submitted. The time range depends on the selected filter.
- Total Review Assignments: A bar chart showing the total number of Review Tasks submitted. The time range depends on the selected filter.
- Assignment Count By Worker: A bar chart detailing the total number of Tasks currently to each Worker involved in the Annotation Campaign.
- Issues Quantity: A bar chart detailing the total number of Issues created until now during the Annotation Campaign grouped by Creation Date. The time range depends on the selected filter as explained below.
- Issues Type Distribution: A pie chart detailing the total number of Issues created until now during the Annotation Campaign grouped by Issues Type(Zone or Image). The time range depends on the selected filter.
For each chart, you can change the Date Range of the data displayed as shown below:
In addition, you can filter the data displayed based on a given step of the Annotation Workflow or by Assignee.
B. Efficiency
The Efficiency tab displays graphs that are related to the time dedicated to the different steps of the Annotation Campaign in real-time.
In detail, the following metrics are displayed:
- Task Duration: Globalized time information on all the Tasks, the average time to complete a Task, and the minimum and maximum time of Task completion on the whole current Annotation Campaign.
- Closed Issues: The total number of Issues closed during the current Annotation Campaign.
- Opened Issues: The total number of Issues opened during the current Annotation Campaign
- Total Annotation Time: The total time spent on Annotation Tasks until now. A bar chart displays this duration on the selected Date Range based on the filter.
- Average Annotation Time: The average time spent on one Annotation Task until now. A bar chart displays this duration on the selected Date Range based on the filter.
- Total Review Time: The total time spent on Review Tasks until now. A bar chart displays this duration on the selected Date Range based on the filter.
- Average Review Time: The average time spent on one Review Task until now. A bar chart displays this duration on the selected Date Range based on the filter.
For each chart, you can change the Date Range of the data displayed as shown below:
In addition, you can filter the data displayed based on a given step of the Annotation Workflow or by Assignee.
C. Quality
The Quality tab displays graphs that are mainly related to the Annotation
of the Asset
of the DatasetVersion
. They are useful for assessing in real-time, the quality, diversity, and quantity of the current DatasetVersion
.
In detail, the following metrics are displayed:
- Data Distribution: This bar chart displays the repartition of
Shape
per Label. - Objects by Image Distribution: This bar chart displays the repartition of
Asset
per number ofShape
in the relatedAnnotation
. - Object areas Distribution: This bar chart displays the repartition of
Shape
per area ofShape
. - Image areas Distribution: This bar chart displays the repartition of
Asset
per area ofAsset
.
Updated 7 months ago