Annotation tool for video

A few seconds of video already represent hundreds of frames, and annotating each one by hand is out of reach for any Computer Vision team. When a DatasetVersion contains video Asset, the Annotation tool switches to a dedicated video interface: instead of a single static image, you work on a timeline of frames, and a Shape follows an object across many frames as a Track. You only place the Shape where the object visibly changes position, and Picsellia fills in the frames in between.

1. Access the video Annotation tool

Access works the same way as for images: from the Grid view or the Details view of the DatasetVersion, click the pen icon displayed when you hover a video Asset. The tool opens on the video, and the header shows its file name.

Before annotating, make sure the DatasetVersion has a Version Type and at least one Label, as detailed in Set up the dataset.

2. Interface overview

The video Annotation tool reuses the same left toolbar (Select mode and the drawing tools matching the Version Type of the DatasetVersion) and the same right-side icon rail (Shapes, Selection, Data, Details, Settings, Shortcuts) as the image tool.

Two elements are specific to video:

• At the bottom-left, two toggles control the footer, independently from each other: Timeline shows or hides the timeline (the grid of the Track) of the current video, and Assets shows or hides the strip with every Asset of the DatasetVersion, to jump to another video. The strip can therefore stay open while the timeline is hidden.
• Below the video, a footer made of two parts replaces the simple Assets footer of the image tool: the player, which plays the video and moves along its frames, and the timeline, the grid that represents the Track of the video along its frames.

Hiding the timeline does not hide the player: it keeps its compact form, which gathers the frame information, the playback controls, the seek bar, and the Step and Speed selectors on a single row. Use it when you want the maximum room for the video itself and no longer need to see where the Track sit along it. The compact player is displayed alone, or above the strip of Asset when Assets is active.

A. Playback controls

The player is the part of the footer that lets you play the video and navigate in it. Above the timeline, the current Frame number (out of the total number of frames) and its timecode are displayed, together with the playback controls:

• Move to first frame and Move to last frame, to jump to the start or the end of the video.
• Move to previous frame and Move to next frame, to step one frame at a time.
• Move N frames before and Move N frames after, to jump by a bigger increment. N is the Step value displayed on the right of the player, which you can change from its dropdown.
• Play / Pause, and a Speed selector (x 1 by default) to control the playback rate.

These controls also have keyboard shortcuts, detailed in Shortcuts.

B. The timeline

The timeline is the grid that represents the Track of the video along its frames, below the player. It lists the Track of the video, grouped by Label. As soon as a Label has several Track, they are gathered in a single row showing the number of Track it contains (for instance circle (2)). Expanding this row with the chevron on the left reveals the individual Track (circle #1, circle #2...). Each Track is displayed as a horizontal bar spanning the frames it covers, with a diamond marker on every keyframe. On a collapsed row, the keyframes of all its Track are displayed as circles. The green vertical line is the playback cursor, positioned on the current frame.

The Shapes panel of the right-side rail groups the Track the same way, which gives you a second way to select and manage them.

A Track does not have to span the whole video: it can start and end on any frame, for instance when an object enters or leaves the frame partway through.

The timeline opens zoomed out on about 100 frames, so that you see where the Track of the Asset sit before working on them, and you can zoom out further until every frame of the video fits in it. Zoom in and out with Cmd/Ctrl + the scroll wheel (or a pinch on a trackpad) or with the zoom slider, and scroll the timeline horizontally with Shift + the scroll wheel. The grid adapts to the zoom level, showing one line every 1, 10, or 100 frames depending on how far out you are, and the frame numbers are displayed when you hover its header.

A single click on the timeline moves the playback cursor to that frame; a double click on an empty area, on the header, or on the cursor head clears the selection.

3. What is a Track?

On video, a Shape is not a single, static object but a Track: the same Shape followed across a range of frames. A Track has one Label, and, as detailed below, one Text and one value per Attribute for its whole duration. What changes from one frame to the next is its geometry, which is described by keyframes and interpolation.

A. Keyframes

A keyframe is a frame on which you have set the geometry of the Shape yourself: the position and dimensions of a rectangle, the vertices of a polygon, or the vertices of a Skeleton. A Track always has at least one keyframe, created when you draw the Shape. Each keyframe is displayed as a diamond on the bar of the Track in the timeline.

For a Computer Vision team, keyframes are what makes video annotation affordable: you describe the motion of an object with a handful of keyframes, placed where its motion changes (it starts moving, changes direction, speeds up, stops), instead of annotating hundreds of frames one by one.

You create a keyframe by moving, resizing, or reshaping the Shape on a frame of the Track that is not a keyframe yet. You can also use the keyframe button, displayed above the timeline when a Track is selected:

• The diamond is outlined on a frame that is interpolated, and filled green on a keyframe. Click on it to turn the current frame into a keyframe, or to remove the keyframe.
• The two arrows on each side jump to the previous and to the next keyframe of the selected Track.

B. Linear interpolation

Between two consecutive keyframes, Picsellia computes the geometry of the Shape on every frame by linear interpolation: the Shape moves in a straight line at constant speed from one keyframe to the next, and its dimensions change at a constant pace. For instance, if the left edge of a box is at x = 100 on a keyframe and at x = 140 four frames later, it is at x = 110, 120, and 130 on the frames in between. For a polygon or a Skeleton, each vertex follows its own straight path the same way.

Two consequences are worth keeping in mind when you annotate:

• Interpolation is linear, so an object that accelerates, slows down, or changes direction between two keyframes drifts away from the straight path the Shape follows. Add a keyframe where the Shape starts to drift: the more irregular the motion, the more keyframes you need.
• After the last keyframe of a Track that has been extended (see Create a Track), the Shape stays where the last keyframe placed it.

Interpolation can be turned off with Enable Interpolation in the Settings panel.

4. Create a Track

Creating a Track means drawing a Shape on the frame where an object first appears, exactly like on an image Asset, then extending it and adding keyframes until it follows the object. You do not need to draw the same Shape again on every frame.

A. Choose a Label

The Label selector, at the top left of the Annotation tool, sets the Label given to the next Shape you draw. Click on it to open the list of the Label of the DatasetVersion, which you can filter with the search field. The number displayed next to each Label is the key ("0" to "9" on your keyboard) that switches to it. You can still change the Label of a Track afterwards, as detailed below.

B. Choose the drawing tool

The drawing tools displayed in the left toolbar depend on the Version Type of the DatasetVersion. Hover a tool to display its name and its keyboard shortcut. On video, you can create a Track with:

The SAM-assisted tools are not available on video either, as mentioned above.

The drawing gestures are the ones of images:

• Rectangle: press at one corner, drag, and release at the opposite corner.
• Polygon: click on each vertex, then click on the first vertex again to close the polygon.
• Skeleton: click on each vertex, in the order defined by the Skeleton of the Label. The Shape is completed once all the vertices are placed.

C. Draw the first Shape

1. Move to the frame where the object first appears, with the playback controls described above.
2. Select the Label and the drawing tool.
3. Draw the Shape on the object.

A new Track is created and a row appears for it in the timeline and in the Shapes panel. Its bar starts on the frame you drew the Shape on, and the first keyframe is displayed as a diamond at its start.

D. Extend the Track

The new Track covers a limited number of frames starting from the current one: 5 by default, as defined by Annotation Default Duration when created in the Settings panel. Beyond its last frame, the Shape is no longer displayed, so the Track must be extended to follow the object further.

The quickest way is to right-click the bar of the Track in the timeline and choose Expand to Last Frame, or Expand to Cursor after moving the playback cursor to the frame you want to reach (see Edit a Track). If most of your objects stay on screen for a long time, you can also raise the default duration in the Settings panel.

An extended Track has no new keyframe: the Shape stays where you drew it, even though the object keeps moving.

E. Follow the object with keyframes

1. Move to a frame where the object has moved, for instance the last frame where it is still visible.
2. Move, resize, or reshape the Shape so that it fits the object again. This creates a keyframe on this frame.

The frames in between are interpolated, as detailed above. Scrub through the frames to check the result, and add a keyframe wherever the Shape drifts away from the object.

F. Annotate the other objects

Draw another Shape to create another Track: each object of the video has its own Track, with its own keyframes. Track that share a Label are grouped in the timeline, as described above.

G. Save your work

Click on Save at the top right of the Annotation tool to save the Annotation. A notification confirms it. What happens next depends on whether an Annotation Campaign is ongoing on the DatasetVersion:

• Without an active Annotation Campaign: the Accept and Reject buttons appear in the header once the Annotation is saved, as for image Annotation. They let you review the Annotation, as detailed in Quality review.
• With an active Annotation Campaign: the Annotation tool works the same way on video, with the additional features dedicated to an Annotation Campaign, such as submitting the Task, the history of the Asset in the Annotation Workflow, Issues, and the Annotation Instructions. The Accept and Reject buttons are not displayed. See Complete an Annotation task.

5. Edit a Track

Selecting Track in the timeline works like selecting files in a file explorer: a click selects only the item you clicked, Cmd/Ctrl + click adds it to or removes it from the selection, and Shift + click extends the selection to every item between the last one you clicked and this one, keeping what you had already picked. A right-click keeps the current selection when the item you right-click is part of it, and otherwise selects that item alone, so the menu that opens always acts on what you see highlighted.

Two behaviors are specific to the rows that group the Track of a Label: a range that crosses one of them selects every Track of that group, and clicking the row of a group whose Track are all selected unselects them instead of selecting them again. Selecting a Shape directly on the video also expands the group of its Label, in the timeline and in the Shapes panel, so the Track you are working on is never hidden under a collapsed row.

A. Common actions

Using the … option menu of a Track row, in the timeline or in the Shapes panel, gives access to: Hide, Lock, Focus, Opacity, Change Label, Duplicate, Select all Keyframes, Go to track, and Delete.

B. Actions on the timeline

To change the frames a Track covers, right-click its bar in the timeline. The menu offers:

• Split (Shift + S): cut the Track into two parts at the current frame. Both parts keep the position that interpolation gave the Shape before the split.
• Cut to Cursor (Shift + C): trim the Track so that it ends at the current frame.
• Expand to Cursor (Shift + K): extend the Track so that it reaches the current frame.
• Expand to Last Frame (Shift + L) / Expand to First Frame (Shift + J): extend the Track all the way to the end or to the start of the video.
• Join (Shift + M): when a Track has been split into two distinct sub-tracks, join them back into a single Track. The action is only displayed in the menu in that case.
• Go to cursor: scroll the timeline to the playback cursor.
• Go to track: the other way around, move the playback cursor to the first keyframe of the selected Track and scroll the timeline to it, which is the quickest way to get back to a Track you spotted in a long video.
• Copy (Cmd/Ctrl + C), Paste (Cmd/Ctrl + V), and Duplicate (Cmd/Ctrl + B), to reuse a Track instead of drawing it again.
• Remove, and Change Label.

Expanding a Track is the quickest way to cover an object that stays on screen for a long time: extend the Track, then add keyframes only where the object moves.

6. Shape details and Attribute

Selecting a Track opens the same Selection panel as on images. It shows the values of the Shape on the current frame, and lets you edit what describes the Track:

• The Label, to change the class of the Track (see also Change Label in the menus above).
• The free-text Text field, mainly used for OCR use cases. See Fill Shape Attributes and Asset Attributes.
• For a rectangle, its position and dimensions (X, Y, Width, and Height) on the current frame.
• The Shape attributes section, which lists one input per Attribute configured on the Label. To configure them, see Attributes and Add Attribute to a Label.
• For a Skeleton, the list of its vertices, each with its visibility state.

The Text and the Attribute values belong to the whole Track, not to a keyframe: filling or changing them on one frame applies them to every frame of the Track. The Text is also displayed next to the Label name on the Shape, for instance Objet_a_risque (zone bleue).

7. Settings

The Settings panel, available from the right-side icon rail, holds your personal preferences for the Annotation tool. They apply to your own account and persist across DatasetVersion: once you have changed them, click on Update to save them.

A. Generic settings

Most of the settings are the same as on images: the General ones (such as the default zoom), the Performances ones (cache), the Shapes ones (such as the Shape opacity), and the Labels ones (such as the display of the Label name on each Shape). They are detailed in Settings & Shortcuts.

B. Player settings

On a video DatasetVersion, the panel has an additional Player section, dedicated to the video player and to the creation of Track:

• Annotation Default Duration when created: the number of frames covered by a new Track, starting from the frame you draw it on. It is 5 by default. Beyond its last frame, a Track is no longer displayed until you extend it (see Extend the Track). If the objects you annotate usually stay on screen for a long time, raising this value saves you from extending each Track; keep it low if they appear for a few frames only, so that you do not have to trim them.
• Display Skip to End / Beginning control: shows or hides the buttons that jump to the first and to the last frame of the video in the playback controls. It is turned on by default.
• Display Frame Step controls: shows or hides the frame-by-frame navigation buttons of the playback controls. It is turned on by default. You can turn it off if you navigate with the keyboard shortcuts and want a lighter footer.
• Enable Interpolation: when it is turned on (default), the Shape of a Track is interpolated linearly between two keyframes, as detailed in Linear interpolation. This is the setting that lets you place a handful of keyframes instead of annotating every frame.
• Cursor out of Timeline behavior: how the visible part of the timeline follows the playback cursor when the cursor reaches its edge, for instance while you play the video or step through it. The options are Stick to the Right, Stick to the Middle (default), and Moves back to the left. Choose the one that keeps the part of the timeline you are working on in view.

8. Shortcuts

Most of the actions of the Annotation tool have a keyboard shortcut, which saves a lot of time on video where you move back and forth between frames all the time. The Shortcuts panel, at the bottom of the right-side icon rail, recaps the shortcuts currently active for the DatasetVersion you are annotating. On video, it contains an additional group, named Timeline in the panel, which gathers the shortcuts of the player (to play the video and move along its frames) and the ones of the timeline (to zoom it and scroll it):

The number of frames of a jump is the Step value displayed on the right of the player. The Alt key is the Option key on a Mac. The two wheel gestures are listed in the Shortcuts panel with a + Wheel suffix, and the modifier they use can be personalized like any other shortcut.

A few more shortcuts are useful when you work on Track:

The actions that change the frames a Track covers, described in Actions on the timeline, have their own shortcuts. They only exist on video, and they are listed with the other video shortcuts in the Shortcuts panel, where you can personalize them:

Join (Shift + M), displayed as Join tracks in the Shortcuts panel, is the opposite of Split. The action is only offered when a Track has been split into two distinct sub-tracks, and it joins them back into a single Track.

The three Expand shortcuts mirror the J / K / L keys of the player, so the hand you use to move along the video is the one you use to stretch a Track over it. Each of these shortcuts is displayed next to its action in the context menu of a Track, and the … option menu recalls the shortcut of the actions it offers as well.

The other shortcuts, such as the ones to draw, zoom, undo, save, or review an Annotation, are the same as on images. They are listed in Settings & Shortcuts, together with how to personalize your shortcuts.

9. Export video Annotation

Once your video Track are annotated, you will probably want to use them to train a model or to move them to another tool. Video Annotation can be exported to a COCO annotation file, but for now this is only possible through the Python SDK: the Annotations export of the DatasetVersion described in Annotation import and export does not export them yet.

With the SDK, call the export_annotation_file method of the DatasetVersion with the export_video parameter set to True:

dataset_version.export_annotation_file("COCO", "./", export_video=True)

By default, export_video is False. When it is True, the file describes the video Asset of the DatasetVersion instead of its image Asset. The file is downloaded in the directory given as the second argument.