Datalake - Upload and Import Data

1. Accessing the Datalake

To access the proper Organization's Datalake, ensure that you're within the right Organization. Each Organization can have several Datalake that are independent one from another and potentially associated with different Cloud Storage Integrations. Accessing and switching between Datalake is detailed here.

Now, it's time to add Data to it!

2. Two ways to add Data to a Datalake

There are two ways to create Data into a Picsellia Datalake:

1. Upload Data from a local drive. In this case, the uploaded Data will be physically stored by Picsellia on the bucket linked to the Datalake through a Cloud Storage Integration.
2. Import Data already stored on your own Cloud Object Storage. In this case, you will be able to visualize and explore Data already physically stored on your bucket through your Picsellia Datalake.

Please note that depending on the Datalake you are using, only one or both methods are available:

• If you are using the Datalake named default, connected to the Cloud Storage Integration created by default for your Organization (as detailed here), you will only be able to upload Data from your local drive.
• If you created a new Datalake connected to your own bucket following this tutorial, you will be able to either upload Data from a local drive, or import Data already stored on your bucket.

Uploading from a local drive is the fastest way to get a Computer Vision project started, while importing from your own Cloud Object Storage lets you plug Picsellia directly into an existing data pipeline: your images and videos stay stored where they already are, avoiding duplicated storage costs, and any new file landing in your bucket can be picked up without a manual re-upload.

3. Upload Data from a local drive

After having accessed the Datalake you want to upload Data to, click on the Upload data button.

A modal will appear. From here, click on Select files, which will open your OS's file browser, allowing you to choose the images or videos you wish to upload into your Picsellia Datalake.

Once you've selected files from your local device, you can create or select the appropriate DataTag to associate with the batch of Data you're about to upload to your Datalake. To learn more about tags and their usage, refer to this documentation page.

After selecting the necessary files and DataTags, click on Upload.

During the upload, a progress bar will display. Depending on the amount of Data being uploaded, you may need to be patient, or consider using our SDK to upload Data which is more reliable for large volumes.

4. Import Data from your Cloud-based Object Storage

If the Datalake you're accessing is already connected to your own bucket, the only remaining action to import Data from it is to select, from Picsellia, the images or videos stored on your bucket that have to be visualized as Data in your Datalake.

To proceed, click on the Import bucket objects button.

A modal will open, displaying the file structure of the connected bucket and allowing you to choose the images you wish to import into your Picsellia Datalake.

You can then select the files or folder to import from your bucket to your Datalake as shown below:

Once the import is launched, you can track its completion from the Jobs panel.

Depending on the amount of Data being imported, you may need to be patient or consider the SDK, which is more reliable for large volumes; it also comes with more customization options.

5. Supported formats

Picsellia supports virtually every image and video format. For more exotic formats that may not display properly, or for very large images, the Tiling feature and the Projections feature both provide workarounds to facilitate visualization.

6. Visualize and sort your Data

Once your Data has been uploaded or imported, you can see the total number of Data in your Datalake and sort it, as detailed here:

As soon as your Data has been uploaded or imported, you can start structuring it using DataTag or Metadata.