Processings

A Processing is a piece of code (like a Python script) executed on Picsellia infrastructure that can interact with any object related to your Organization (Datalake, DatasetVersion, ModelVersion...).

This page details how to use a Processing on a ModelVersion. The use of a Processing on a Datalake and on a DatasetVersion is detailed here and here.

1. Why use a Processing on a ModelVersion?

Imagine you want to convert a ModelVersion to another format so that you can deploy it on a given device. Normally, the steps to achieve this would be:

• Downloading your Model files locally.
• Running a conversion script on them, in an environment that has the right dependencies.
• Uploading the converted Model files to Picsellia.

Running a script can be an automatic task, but this process is fully manual, and it requires a computer able to run the code. A Processing lets you automate it and launch it whenever you want, on the ModelVersion you want, directly from the platform.

A Processing can be run on a ModelVersion to perform tasks like:

• Model framework conversion.
• Model quantization or pruning.
• Model files adaptation for edge deployment.
• Anything you can think of regarding your Model files.

For a Computer Vision team, this also means these operations are reproducible and traceable: every run is recorded as a Job, so you can retrieve which Processing was executed on a ModelVersion, when, and by whom.

2. Access the Processing list

You can reach the list of available Processing at any moment by clicking on Processings in the left navigation bar.

This page lists all the Processing available for the current Organization. You can switch between:

• The Public and Private Processing: the Public ones are created by the Picsellia team and available to everyone, the Private ones are the ones you created.
• The type of object they can be used on: Datalake, Dataset, or Model.

For each Processing, the list displays its Name, Description, and Created at date, and a search bar lets you find one by name.

It is also from this page that you can create your own Private Processing, with the + Add Processing button, as detailed in this guide.

3. Use a Processing on a ModelVersion

Take a concrete example: converting a YOLOX ModelVersion to a Core ML one, so that the converted model can be deployed on iOS devices.

First, access the Model and the ModelVersion the Processing should be applied to, and click on Process in the header of the ModelVersion overview.

A Launch processing modal opens, in which you pick the Processing to execute among the Public and Private ones. Each one is displayed as a card with its name, the beginning of its description, and its task (for instance Model compression). Here, select the yolox-to-coreml-converter Processing, available to everyone as a Public Processing, then click on Next.

The modal then displays the description of the Processing and lets you provide what it needs to run, in two parts:

• Inputs: depending on the Processing, you might be requested to provide some inputs. An input can be a Picsellia object (for instance a ModelVersion or a DatasetVersion), in which case a selector lets you choose among the ones available in the current Organization, or a text or a number, in which case a field is displayed to enter its value.
• Parameters: a Processing also has a list of parameters that already have default values, which let you fine-tune the behavior of the script behind it. The default values are usually defined to make it work in a standard configuration. Here, they are export_format and input_model_file_name. You can edit or remove an existing parameter, or add a new one by filling its Name and Value and clicking on the add button.

Once everything is set, click on Launch: the Docker image associated with the Processing is executed on the default infrastructure. If several compute infrastructures are integrated with your Organization, you can select the one to run the Processing on. To integrate a new compute infrastructure, you need to define a Compute Connector, as detailed here.

4. Track the Processing progress

Launching a Processing creates a Job that runs in the background. You can access its status and much more information about it from the Jobs page, available in the left navigation bar. This page shows the history of all the Job that ran or are currently running, which can be Processing, Training, or Internal Job: here, select the Processings tab.

If you just launched a Processing, you should see its Job at the top of the list. Clicking on its name opens its detailed view, with the user who triggered it, the object it was applied to, its status, progression, and duration, and the history of its runs with their logs.

This way, you can track the progress and the status of your Job and check that everything is going well. It is also a way to keep track of any action performed on a ModelVersion, such as which Processing was executed, when, and by whom.

Once your Job is over, you have access to the full history of its logs and its total running time, and its status switches to Succeeded, or Failed if there were issues at runtime. If it fails, you can find the issue thanks to the stack trace in the logs.

Once you have detected the issue, fixed it, and updated the Docker image of your Processing, you can click on the Re-run button at the top right of the Job. This creates and launches a new run with the same configuration (inputs and parameters). The Job page is detailed here.

When the Job has succeeded, check the result on your ModelVersion: what a Processing produces depends on the Processing, as described in its description. In this example, you retrieve the ModelVersion converted to the expected format.