Experiment Tracking

As it is a whole discipline in itself, Picsellia proposes a fully functional Experiment Tracking system that allows you to elevate your workflow and finally achieve top performances. When training AI models, you will perform a lot of different Experiment with different pre-trained ModelVersion, DatasetVersion, sets of parameters, evaluation techniques, etc.

This is a very iterative process that is time-intensive. Once your training script is ready, you will spend a lot of time iterating on all the different variables your final ModelVersion depends on, and launching your script over those variables until you are satisfied with its performance.

One big challenge you will face as a data scientist, AI researcher, or engineer is being able to store all the important Metrics for each Experiment, store the principal files as Artifacts, and finally compare your Experiment to find the best one.

This whole process is called Experiment Tracking, and you can do it seamlessly on Picsellia! In this section, you will learn how 😊.

1. The Experiment Tracking dashboard

As mentioned in the Experiment overview page, each Experiment has its own Experiment Tracking dashboard, which is basically a placeholder to be filled with Metrics computed before, during, or after the execution of the training script.

This means the Experiment Tracking dashboard is fully personalizable and lets the author of the script log only the metrics that are relevant to them and their team. The creation of callbacks in the training script to log Metrics on the Experiment Tracking dashboard is detailed on this page of our guide.

If you use, as Base architecture, a ModelVersion from the Public Registry, the Metrics displayed in the Experiment Tracking dashboard will be the ones defined by the Picsellia team in the training scripts associated with that ModelVersion.

As a reminder, some Metrics can be created during the Experiment creation process. For instance, the training parameters will be logged in the Experiment Tracking dashboard as a Table. The initial LabelMap inherited from the Base architecture will also generate a Table in the created Experiment; the LabelMap will then be updated with the Label of the DatasetVersion attached to the Experiment as soon as the training script execution is launched.

Below, you will find an example of how to log a new Metric and an exhaustive list of the supported Metric types.

2. The Log method

Logging a new Metric to Picsellia is as simple as this:

from picsellia import Client
from picsellia.types.enums import LogType

api_token = 'YOUR TOKEN'
project_name = 'My awesome Project'

client = Client(api_token)
project = client.get_project(project_name)

experiment = project.get_experiment(
    name='my_new_experiment'
    )

data = [3, 1.25, 1.4, 0.35, 0.95]
experiment.log('TotalLoss', data, LogType.LINE)

A new Metric has been created in the Experiment Tracking dashboard of your Experiment:

What you just created is what we call a Log asset. Now, let's dive into it to see how to use it properly and how far we can go.

3. The Log object

To store something related to an Experiment and visualize it in Picsellia, you have to store it in what we call a Log, composed of the following properties:

• name: the name of the tab you will see in your Experiment; it can group several data assets.
• data: the value, dictionary, array, or image you want to store.
• type: the type of visualization you want to render your asset in (e.g. array, line chart, bar chart...).

from picsellia.types.enums import LogType

experiment.log(name='logs', data=data, type=LogType.LINE)

Once created, you can retrieve a Log at any time with experiment.get_log(name=...), then call update on it to change its data (or name) or delete it. A Line Log additionally supports append, to add values to it without touching the ones already stored.

4. Metric types

Here is the list of Metric types supported for now:

class LogType(StrEnum):
    VALUE = "VALUE"
    IMAGE = "IMAGE"
    LINE = "LINE"
    TABLE = "TABLE"
    BAR = "BAR"
    HEATMAP = "HEATMAP"
    LABELMAP = "LABELMAP"
    EVALUATION = "EVALUATION"

A. Single Value

Store one value to be displayed alone. It can be of any type (int, float, str...).

experiment.log(name='accuracy', type=LogType.VALUE, data=0.95)

Single values will appear in your Experiment dashboard, in the summary at the top of the page, like this:

B. Line

A basic line chart:

data = [3, 8, 2, 7, 9]
experiment.log(name='logs', type=LogType.LINE, data=data)

If you hover over a Line or Multi-Line Metric, the associated values will be displayed for the Metric. You can also draw a square on the chart area to zoom in on the selected area.

The format of your data to display a line chart must be one of the following.

a. List format

If you don't specify it, we automatically increment the steps (x_axis) for you, so you can just send their values like this:

experiment.log(name='logs', type=LogType.LINE, data=[5, 8, 2, 9, 10])

For a Line Log specifically, the replace parameter doesn't work the way its name suggests: replace=True (the default) still appends to the existing values, exactly like logging without specifying it at all. To fully replace a Line Log's data instead of appending to it, retrieve it and update it directly:

log = experiment.get_log('logs')
log.update(data=[5, 8, 2, 9, 10])

b. Custom Axis

If you want to specify your own x_axis steps, you can send the data this way:

data = {
    'steps': [0, 2, 4, 8, 10],
    'values': [0.7, 0.8, 0.9, 0.3, 0.12]
}
experiment.log(name='logs', type=LogType.LINE, data=data)

C. Multi Lines

You can also visualize multiple lines on the same chart by adding keys to your dictionary (with or without steps):

data = {
    'loss1': [0.45, 0.2, 0.98, 0.47, 0.28],
    'loss2': [0.7, 0.8, 0.9, 0.3, 0.12],
    'loss3': [0.21, 0.47, 0.74, 0.56, 0.86]
}
experiment.log(name='logs', type=LogType.LINE, data=data)

D. Bar

A basic bar chart:

data = {
    'x': ['car', 'person', 'bird'],
    'y': [5, 8, 2],
}
experiment.log(name='labels', type=LogType.BAR, data=data)

E. Table

data = {
    "Loss/total_loss": 1.202446,
    "Loss/localization_loss": 0.022069892,
    "Loss/classification_loss": 1.1457626,
    "Loss/regularization_loss": 0.034613654,
    "DetectionBoxes_Recall/AR@1": 0.0
}
experiment.log(name='metrics', type=LogType.TABLE, data=data)

F. Image

path = "path/to/my/image.jpg"
experiment.log("img3", type=LogType.IMAGE, data=path)

G. Heatmap

conf = [[33,  0, 13],
       [ 2,  7, 16],
       [10,  0,  0]]

confusion = {
    'categories': ['car', 'pedestrian'],
    'values': conf
}
exp.log(name='confusion', data=confusion, type=LogType.HEATMAP)

H. Remarks on LabelMap and Evaluation

a. LabelMap

Unlike the other Metric types, LABELMAP isn't meant to be logged directly by your training script. It's the type of the labelmap Log that Picsellia automatically creates and maintains for you, as introduced in section 1: inherited from the Base architecture on Experiment creation, then kept up to date with the Label of the DatasetVersion you attach.

b. Evaluation

Similarly, EVALUATION isn't a type you should pass to experiment.log(). Attaching predictions to your Experiment to build Evaluation goes through a dedicated set of methods instead — add_evaluation, add_evaluations, and compute_evaluations_metrics — detailed in the Evaluations interface page and this tutorial.