Accessing files with the URI_FOLDER type data-assets

Question

Accessing files with the URI_FOLDER type data-assets

Vishal Mahajan 35

I have come across the following example at https://learn.microsoft.com/en-us/training/modules/make-data-available-azure-machine-learning/4-create-data-asset

import argparse 
import glob 
import pandas as pd 
parser = argparse.ArgumentParser() 
parser.add_argument("--input_data", type=str) 
args = parser.parse_args() 
data_path = args.input_data 
all_files = glob.glob(data_path + "/*.csv") 
df = pd.concat((pd.read_csv(f) for f in all_files), sort=False)

For example, if I create the data-asset and try to list the files in the directory using os, glob package:

my_path = "../data/"
# set the version number of the data asset
v1 = "initial"

			   version=v1,    
			   description="Credit card data",    
 			   path=my_path,    
			   type=AssetTypes.URI_FOLDER)

my_data.path
# output: azureml://datastores/workspaceblobstore/paths/LocalUpload/SOMETHING/data

os.listdir(my_data.path)
# FileNotFoundError: [Errno 2] No such file or directory:

# copying and using complete path from the data-stores 
os.listdir("azureml://subscriptions/<sid>/resourcegroups/<rid>/workspaces/<wid>/datastores/workspaceblobstore/paths/LocalUpload/<cid>/data"))
# FileNotFoundError: [Errno 2] No such file or directory:

I have been able to make it work only with the following azureml.fsspec I think the tutorials and documentation should be clear about that. Even the applied skills labs have code that tries to access the files without the following package. Please correct me if I am doing something wrong.

from azureml.fsspec import AzureMachineLearningFileSystem
fs = AzureMachineLearningFileSystem(my_data.path)
fs.ls()

Rakesh Gurram 15,715 Reputation points Microsoft External Staff Moderator

2024-02-06T05:23:31.35+00:00

Hi Vishal Mahajan,

Thanks for reaching out to us on Microsoft Q&A forum.

The error message "FileNotFoundError: [Errno 2] No such file or directory" indicates that the path to the file or directory is incorrect.
Please double-check the path to the file or directory and ensure that it is correct. The azureml.fsspec is a python package to access the files in the directory.

Please find the document for your reference: Create datastores.

Data concepts in Azure Machine Learning

Thank you.

Vishal Mahajan 35

I checked the paths and they seem to be working fine with AzureMachineLearningFileSystem but not with os or glob. I have provided a reproducible example for Azure ML Notebook


# create a data asset
# set the version number of the data asset
v1 = "initial"
my_data = Data(name="credit-card-data-folder",
			   version=v1,    
			   description="Credit card data",
			   path=my_path,    
			   type=AssetTypes.URI_FOLDER,)
ml_client.data.create_or_update(my_data)
print(f"Data asset created. Name: {my_data.name}, version: {my_data.version}")

### test the paths with os and AzureMachineLearningFileSystem 

# This does not works
import os
os.listdir(data_asset.path)


# This works
from azureml.fsspec import AzureMachineLearningFileSystem
fs = AzureMachineLearningFileSystem(data_asset.path)
fs.ls()

Rakesh Gurram 15,715 Reputation points Microsoft External Staff Moderator

2024-02-08T09:27:28.8233333+00:00

Hi Vishal Mahajan,

Hope you are doing well.

I wanted to check if you had the opportunity to review the information which was provided in our previous comment.

Please feel free to contact us if you have any more quires.

If you feel that your quires have been resolved, please accept the answer by clicking the "Upvote" and "Accept Answer" on the post.

Thank you.
Vishal Mahajan 35 Reputation points

2024-02-08T10:48:38.2+00:00

thanks Rakesh. Yes, i can understand now and this is what i also suspected after trying it. I just feel that this aspect should be also mentioned in the documentation or in relevant modules.
Rakesh Gurram 15,715 Reputation points Microsoft External Staff Moderator

2024-02-08T14:02:11.0166667+00:00

Hi Vishal Mahajan,

Yes, we will convey your suggestion internally.

Please Accept Answer the answer if the information helps you.

Thank you.

Accepted answer

0 additional answers

Your answer

Rakesh Gurram 15,715 Reputation points Microsoft External Staff Moderator

2024-02-06T05:23:31.35+00:00

Hi Vishal Mahajan,

Thanks for reaching out to us on Microsoft Q&A forum.

The error message "FileNotFoundError: [Errno 2] No such file or directory" indicates that the path to the file or directory is incorrect.
Please double-check the path to the file or directory and ensure that it is correct. The azureml.fsspec is a python package to access the files in the directory.

Please find the document for your reference: Create datastores.

Data concepts in Azure Machine Learning

Thank you.
Vishal Mahajan 35 Reputation points

2024-02-06T14:19:51.37+00:00

I checked the paths and they seem to be working fine with AzureMachineLearningFileSystem but not with os or glob. I have provided a reproducible example for Azure ML Notebook

# create a data asset # set the version number of the data asset v1 = "initial" my_data = Data(name="credit-card-data-folder", version=v1, description="Credit card data", path=my_path, type=AssetTypes.URI_FOLDER,) ml_client.data.create_or_update(my_data) print(f"Data asset created. Name: {my_data.name}, version: {my_data.version}") ### test the paths with os and AzureMachineLearningFileSystem # This does not works import os os.listdir(data_asset.path) # This works from azureml.fsspec import AzureMachineLearningFileSystem fs = AzureMachineLearningFileSystem(data_asset.path) fs.ls()
Rakesh Gurram 15,715 Reputation points Microsoft External Staff Moderator

2024-02-08T09:27:28.8233333+00:00

Hi Vishal Mahajan,

Hope you are doing well.

I wanted to check if you had the opportunity to review the information which was provided in our previous comment.

Please feel free to contact us if you have any more quires.

If you feel that your quires have been resolved, please accept the answer by clicking the "Upvote" and "Accept Answer" on the post.

Thank you.
Vishal Mahajan 35 Reputation points

2024-02-08T10:48:38.2+00:00

thanks Rakesh. Yes, i can understand now and this is what i also suspected after trying it. I just feel that this aspect should be also mentioned in the documentation or in relevant modules.
Rakesh Gurram 15,715 Reputation points Microsoft External Staff Moderator

2024-02-08T14:02:11.0166667+00:00

Hi Vishal Mahajan,

Yes, we will convey your suggestion internally.

Please Accept Answer the answer if the information helps you.

Thank you.

Answer 1

Rakesh Gurram 15,715 Microsoft External Staff Moderator

Hi Vishal Mahajan,

Sorry for the inconvenience.

Based on your code, it seems that you are trying to access the files in the Azure data asset, using the os module, which is not supported in Azure Machine Learning. Instead, you should use the AzureMachineLearningFileSystem class from the azureml.fsspec module to access the files in the Azure data asset.

If you are still facing any issue, please let us know in the comments. We are glad to help you.

Thank you.

Share via

Accessing files with the URI_FOLDER type data-assets

0 additional answers

Your answer