Configure and use a Pipeline

1. What is a Pipeline on Picsellia?

Pipelines on Picsellia implement and automate the Continuous Learning philosophy on a ModelVersion deployed in a given Deployment.

The idea is to add PredictedAsset, with their Prediction reviewed by a human, to the DatasetVersion that was used to train the currently deployed ModelVersion. With these additional Asset, the DatasetVersion becomes more representative of what the model meets in production: if you retrain the ModelVersion on this improved DatasetVersion, the new ModelVersion is likely to perform better.

This philosophy leverages the versioning systems offered by Picsellia on Model and Dataset, so that you keep a clear and precise trace of the DatasetVersion and the ModelVersion after each retraining.

To automate this procedure, you configure a Pipeline in the Settings tab of the Deployment, in its Pipelines section. A Pipeline is composed of four steps:

• Training Data: indicates to the Deployment the DatasetVersion used to train the deployed ModelVersion. It is used, for instance, to compute the Outlier score of the PredictedAsset.
• Feedback Loop: defines the DatasetVersion that will be enriched by the PredictedAsset and reviewed Prediction that you push to the Pipeline.
• Continuous Training: defines when and how the retraining of the ModelVersion is launched.
• Continuous Deployment: defines the deployment strategy to adopt with the retrained ModelVersion.

Each step you configure displays a badge in the header of the Deployment. The Feedback Loop, the Continuous Training, and the Continuous Deployment also have an Enabled switch, which lets you activate or disable them whenever you want.

Once the Pipeline is fully configured, the only manual action needed to retrain a ModelVersion is the review of the Prediction (detailed here) and their submission to the Feedback Loop.

2. Setup Training Data

The Training Data indicates, for the current Deployment, which DatasetVersion was used as training dataset to train the deployed ModelVersion.

This way, Picsellia compares any further PredictedAsset logged on the Deployment with the DatasetVersion defined as Training Data, and computes metrics such as the Outlier score and the KS Drift. Picsellia uses up to 1000 images of this DatasetVersion to compute the data distribution that detects drift and outliers.

When a Deployment is created, the Training Data is not initialized, and until it is, no Outlier score is computed for the Deployment: the Dashboard displays Training Data status is disabled on the corresponding charts. This computation is not retroactive, which is why it is highly recommended to set it up as soon as the Deployment is created.

To define the Training Data, go to the Settings tab of the Deployment, open Training Data in the Pipelines section, and click on Select a dataset version to choose the Dataset and the DatasetVersion used to train the deployed ModelVersion. Then click on Update to save.

Once configured, the section displays a card with the name of the DatasetVersion, its Dataset, its number of Asset, and its last modification. To use another DatasetVersion, click on the pen icon of the card, select another one, then click on Update.

Picsellia processes the DatasetVersion designated as Training Data to be able to compare any further PredictedAsset with it, and as a consequence to provide the related Outlier score.

Once the Training Data is properly initialized, a Training Data badge appears in the header of the Deployment.

3. Setup Feedback Loop

The Feedback Loop is initialized by defining the DatasetVersion that receives, as Asset, the PredictedAsset that you push through the Pipeline.

To configure it, open Feedback Loop in the Pipelines section of the Settings tab and click on Attach dataset versions to select the DatasetVersion. Then click on Set up and activate.

Once configured, the DatasetVersion are displayed as cards and a Feedback Loop badge appears in the header. To add or remove a DatasetVersion, click on the pen icon of a card, then on Update. The Enabled switch lets you activate or disable the Feedback Loop whenever you want.

Consistency is key when defining such a mechanism between a ModelVersion, a Deployment, and a DatasetVersion. Picsellia makes some verifications, for instance that the LabelMap or the Detection Type of the deployed ModelVersion and of the DatasetVersion defined in the Feedback Loop are the same. As a user, you are also in charge of ensuring consistency at each step of a Computer Vision project.

Once the Feedback Loop is initialized, PredictedAsset can be submitted to the Pipeline and added as Asset in the defined DatasetVersion, as detailed here.

4. Setup Continuous Training

The Continuous Training is initialized by defining when and how the retraining of the deployed ModelVersion is launched.

You define the Experiment that will be created and executed once the trigger is reached. This Experiment takes the ModelVersion deployed on the current Deployment as Base architecture.

From the Continuous Training section, you define:

• The Project in which the Experiment is created.
• The DatasetVersion to attach to the Experiment.
• The Training Parameters of the Experiment.
• The trigger.

This section looks like the Experiment creation form.

A. Project

Select the Project in which the Experiment is created, by clicking on Attach a project. A good practice is to select the same Project as the one containing the source Experiment of the deployed ModelVersion. The Project is then displayed as a card with its number of Experiment and Dataset, and you can change it with its pen icon.

B. DatasetVersion

Then select the DatasetVersion to attach to the Experiment that the Continuous Training creates once the trigger is reached. You choose them among the ones attached to the Project selected before.

As when you create an Experiment manually, you select the DatasetVersion to attach and give an Alias to each of them. If the deployed ModelVersion owns some Dataset Constraints, the selection adapts accordingly. Please refer to this page for further details, as it is almost the same as in the Experiment creation form. An attached DatasetVersion can be changed with its pen icon.

C. Training Parameters

In the Configuration section, you define the type of Training executed once the trigger is reached. For the moment, the only available value is Experiment.

The Parameters are the Training Parameters of the Experiment that will be created. As when you create an Experiment manually, they are inherited, with their default value, from the Base architecture, here the deployed ModelVersion. Each one is displayed with its value and its type. Use the pen icon to edit a parameter and the trash icon to delete it. To add a new one, fill the Name and Value fields and click on the blue + button.

When the Continuous Training is triggered, the Training Script associated with the deployed ModelVersion is executed in a new Experiment. If it does not receive the Training Parameters it expects, it can fail, so be careful when adding or removing a Training Parameter.

D. Trigger

The trigger defines when the Continuous Training is launched. For now, the only available Trigger type is the Feedback Loop: the Experiment is created and launched once the number of reviewed Prediction sent to the Pipeline through the Feedback Loop reaches the Threshold that you define in this section.

---

All the required elements are now defined, so the Continuous Training is ready to be triggered, to create and execute an Experiment that retrains the currently deployed ModelVersion. Save the configuration with the button at the bottom of the page. Once it is configured, a Continuous Training badge appears in the header of the Deployment.

The configuration can be modified at any moment: modifications are taken into account once you click on the button at the bottom of the page. The Continuous Training can also be put on hold at any moment with the Enabled switch.

5. Setup Continuous Deployment

The Continuous Deployment is initialized by defining what to do when the Experiment created and launched by the Continuous Training is over.

The Continuous Training must be configured before you can set up the Continuous Deployment: until then, the section displays In order to activate the Continuous Deployment, first you need to setup your Continuous training pipeline, with a link to the Continuous Training section.

The Policy dropdown offers three policies:

• Deploy Champion: when the Experiment is over, it is exported automatically as a new ModelVersion of the deployed one. This new ModelVersion is then deployed on the current Deployment and replaces the previously deployed one. Further inferences are done by the newly created ModelVersion.
• Deploy Shadow: the Experiment is exported automatically as a new ModelVersion of the deployed one, then deployed as Shadow Model on the current Deployment. The Champion and Shadow ModelVersion both make Prediction for further inferences. It is the best way to assess that the new ModelVersion outperforms the previous one before turning it into the champion.
• Deploy Manual: the Experiment is exported automatically as a new ModelVersion of the deployed one, but it is not deployed by the Continuous Deployment: a user has to do it manually.

Once you have made your choice, save the configuration with the button at the bottom of the page. When the Continuous Deployment is activated, a Continuous Deployment badge appears in the header of the Deployment.

The configuration can be modified at any moment, and the Continuous Deployment can be put on hold with the Enabled switch.

---

Your Pipeline is now fully configured and operational. The next sections show how to push PredictedAsset and reviewed Prediction into it to perform Continuous Learning on a ModelVersion.

6. Submit a reviewed Prediction to the Feedback Loop

For this part, consider a Deployment whose Pipeline is configured as explained above, and on which PredictedAsset and their Prediction have been logged since.

By default, all the Prediction logged have the status TO REVIEW, which means that the Prediction is displayed as it was initially predicted by the deployed ModelVersion.

Picsellia lets you review any Prediction logged on the Deployment with the Prediction Review tool, detailed here. Once reviewed, a Prediction turns into the REVIEWED status, which means it can now be submitted to the Pipeline.

To find the reviewed Prediction, you can filter the Predictions tab with the Query Language, for instance with predictions.status = "Reviewed".

Select the PredictedAsset with a Prediction in REVIEWED status: the bar above the list displays the number of selected Prediction and the Submit to Feedback Loop button.

Click on Submit to Feedback Loop. A window opens and lets you choose the DatasetVersion to which the PredictedAsset are sent, among the ones defined in the Feedback Loop configuration. You can filter them by Version Type or search them by name. Adding the PredictedAsset to the DatasetVersion also adds all their reviews. Please note that only one DatasetVersion can be selected, meaning that the PredictedAsset and their reviewed Prediction are added as Asset in the selected DatasetVersion.

Once submitted, the PredictedAsset and their reviewed Prediction are added as Asset to the selected DatasetVersion, and the Prediction turns into the SUBMITTED status.

7. Trigger the Continuous Training and Continuous Deployment

By submitting PredictedAsset through the Feedback Loop, you enrich the DatasetVersion of your Pipeline. As soon as the number of PredictedAsset and reviewed Prediction submitted to the Feedback Loop reaches the Threshold defined here, a new Experiment is created according to the configuration of the Continuous Training.

This Experiment is then launched, which means that the Training Script is executed, on the Picsellia Training Engine.

As soon as the training is over, the Experiment is exported as a new version of the deployed ModelVersion, and potentially deployed on the current Deployment according to the Continuous Deployment policy you selected.

---

You have now performed the complete Continuous Learning in a structured way: you reviewed Prediction made by the initial ModelVersion, enriched your training DatasetVersion, retrained, and redeployed a new ModelVersion. This new version is supposed to outperform the initial one: to make sure of it, you can still refer to the Dashboard.