Getting Folder data from Azure data lake gen2 in AzureML

Ghazi Dhouafli 0

I'm currently working on a machine learning project, I do have a folder stored in azure data lake gen2 that has all the data I need to train the model. I created a datastore and I created a data asset in which the type is a folder(uri folder). So far everything is fine but I do fail to understand what I shall do next. The folder that I have has json files and npy files and they are all needed to train the model.

romungi-MSFT 43,696 Reputation points Microsoft Employee

2023-04-10T06:47:25.04+00:00
@Ghazi Dhouafli As a datastore you should be able to use the reference of the datastore and the relative paths from original data source should be available in the experiment from the datastore. For example:

adlsgen2_data_ref is a reference to your datastore that registered your data lake v2 source and path_on_datastore the relative path in the backing storage for the data reference.

adlsgen2_data_ref = DataReference( datastore=adlsgen2_datastore, data_reference_name='adlsgen2_test_data', path_on_datastore='testdata')

This notebook can be used as a reference on using the different datastores that are supported. For a data asset you should be able to use the path azureml:<my_data>:<version> where my_data should be the data asset name that was registered for the data lake v2 data source. This section on accessing data from the documentation can be used as reference.
romungi-MSFT 43,696 Reputation points Microsoft Employee

2023-04-13T08:26:34.34+00:00

@Ghazi Dhouafli Did the above suggestion help to check if you are able to use ADLS2 with your experiments?

Share via

Getting Folder data from Azure data lake gen2 in AzureML