Philosophy and infrastructure

A Datalake is a place shared by all the members of an Organization to gather all the images and videos (called Data) in the frame of your Computer Vision projects.

The Datalake feature mainly aims at having all your Data available for visualization, structuring, and exploration.

For a Computer Vision team, this means having a single source of truth for your raw images and videos instead of scattered local folders or bucket copies: the same pool of Data can then be reused to build and version as many Dataset as needed for different models or use cases, without duplicating storage or losing track of where your Data comes from.

First of all, it is important to note that an Organization can have several Datalake.

Each Datalake is connected through a Cloud Storage Integration to a bucket on an Object Storage (hosted by a Cloud provider or locally) where the Data visualized on the Datalake is physically stored.

When creating a new Picsellia Organization, a new dedicated bucket is created on the Picsellia Object Storage (hosted by AWS). The Datalake of the freshly created Organization is called Default and is connected to this bucket, and all Data uploaded to this Datalake will be physically stored on this Picsellia Object Storage.

However, you can also decide to create a new Datalake for your Organization and connect it to your own bucket hosted by your Cloud provider. To do so, please refer to this tutorial.

You can easily switch from a Datalake to another, as detailed here.

Every machine learning project begins with data, and in our case of Computer Vision, it starts with images and videos.

There are two ways to upload your Data using Picsellia:

1. Upload locally stored Data directly to Picsellia. In this scenario, your Data will be physically stored by Picsellia in the Object Storage linked to the current Datalake.
2. Import Data already stored on your own Cloud Object Storage to Picsellia's Datalake and access them through Picsellia.

All users with Admin rights in a given Organization can access the Organization Settings, particularly the Storages and Datalakes tab, detailed here. From this one, you can manage the existing Datalake and Cloud Storage Integrations.

Please note that if you are accessing the Datalake connected to the bucket created for you on the Picsellia Object Storage, which is the Datalake called Default and created for you when initializing a new Organization, you will only be able to upload Data from your local drive. The uploaded Data will be physically stored on Picsellia's Object Storage and visualized on the Default Datalake using the Cloud Storage Integration created by default for your Organization named hinokuni-storage-production.

If you create a new Datalake following this tutorial, which will use a new Cloud Storage Integration linked to your own bucket, then you will be able to either:

• Upload Data from a local drive. In this case, the uploaded Data will be physically stored by Picsellia in your bucket
• Import Data already stored on your bucket. In this case, you will be able to visualize and explore Data already physically stored on your bucket through your Picsellia Datalake.