How to mount the azure dataset in jupyter notebook ?

Question

How to mount the azure dataset in jupyter notebook ?

V JEEVA 46

I have created dataset which points to two datastores. I have registered it as well.
But when I mount the dataset, it is showing some error.

romungi-MSFT 29,536 Microsoft Employee

@V JEEVA What is the error that you are seeing and are you using Azure ML registered dataset or Azure opendatasets?
If you are using Azure ML then you can mount or download for a linux based compute. Something like below:

import tempfile  
mounted_path = tempfile.mkdtemp()  
  
# mount dataset onto the mounted_path of a Linux-based compute  
mount_context = dataset.mount(mounted_path)  
  
mount_context.start()  
  
import os  
print(os.listdir(mounted_path))  
print (mounted_path)

Where dataset refers to your registered workspace dataset reference.

If you are using the Azure Open Datasets then you can use them with Jupyter notebook after you install the package azureml-opendatasets

pip install azureml-opendatasets

For access use below snippet.

# This is a package in preview.  
from azureml.opendatasets import NycTlcYellow  
  
from datetime import datetime  
from dateutil import parser  
  
  
end_date = parser.parse('2018-06-06')  
start_date = parser.parse('2018-05-01')  
nyc_tlc = NycTlcYellow(start_date=start_date, end_date=end_date)  
nyc_tlc_df = nyc_tlc.to_pandas_dataframe()  
  
nyc_tlc_df.info()

V JEEVA 46 Reputation points

2022-08-26T09:23:57.25+00:00

@romungi-MSFT Thanks a lot for answering the question. Your method worked perfectly.

Also can I mount the dataset to a specific folder ?#1
There is some warning while I mounted the dataset that points to two different datastores but worked fine#2 - Is it correct , can I ignore the warning or is it some kind of error ?

Access datastores

az_datastore = Datastore.get(ws,'external')
meta_datastore = Datastore.get(ws,'meta')

1 Wanted to mount dataset to a specific folder that I created

dataset_one = Dataset.File.from_files([(az_datastore,r"all_img/**/*")])
dataset_one.register(workspace=ws,name='test_d2_1',description='Testing purpose only')

ML Workspace/studio:

os.mkdir('data')
DATA_DIR = os.path.abspath("data")
mount_context = dataset.mount(mount_point=DATA_DIR)
mount_context.start()

output:
fusermount: mounting over filesystem type 0xfe534d42 is forbidden
PyFuseError: MountPointCreationError(Some(1))

2 Created dataset pointing to two different datastores : az_datastore,meta_datastore

dataset_two = Dataset.File.from_files([(az_datastore,r"all_img/day_one_image/**/*"),(meta_datastore,r"img_label.csv")])
dataset_two.register(workspace=ws,name='test_d2_2',description='Testing purpose only')

ML Workspace:

mounted_path = tempfile.mkdtemp()
mount_context = dataset.mount(mounted_path)
mount_context.start()
print(os.listdir(mounted_path))
print (mounted_path)

output:
Not mounting as a volume: Volume mount only supports datasets with exactly one datastore path, actual: 2.
Falling back to dataflow mount.
['external', 'meta']
/tmp/tmpojzekdud

romungi-MSFT 29,536 Reputation points Microsoft Employee

2022-08-25T11:57:20.217+00:00

@V JEEVA What is the error that you are seeing and are you using Azure ML registered dataset or Azure opendatasets?
If you are using Azure ML then you can mount or download for a linux based compute. Something like below:

import tempfile mounted_path = tempfile.mkdtemp() # mount dataset onto the mounted_path of a Linux-based compute mount_context = dataset.mount(mounted_path) mount_context.start() import os print(os.listdir(mounted_path)) print (mounted_path)

Where dataset refers to your registered workspace dataset reference.

If you are using the Azure Open Datasets then you can use them with Jupyter notebook after you install the package azureml-opendatasets

pip install azureml-opendatasets

For access use below snippet.

# This is a package in preview. from azureml.opendatasets import NycTlcYellow from datetime import datetime from dateutil import parser end_date = parser.parse('2018-06-06') start_date = parser.parse('2018-05-01') nyc_tlc = NycTlcYellow(start_date=start_date, end_date=end_date) nyc_tlc_df = nyc_tlc.to_pandas_dataframe() nyc_tlc_df.info()
V JEEVA 46 Reputation points

2022-08-26T09:23:57.25+00:00

@romungi-MSFT Thanks a lot for answering the question. Your method worked perfectly.

Also can I mount the dataset to a specific folder ?#1
There is some warning while I mounted the dataset that points to two different datastores but worked fine#2 - Is it correct , can I ignore the warning or is it some kind of error ?

Access datastores

az_datastore = Datastore.get(ws,'external')
meta_datastore = Datastore.get(ws,'meta')

1 Wanted to mount dataset to a specific folder that I created

dataset_one = Dataset.File.from_files([(az_datastore,r"all_img/**/*")])
dataset_one.register(workspace=ws,name='test_d2_1',description='Testing purpose only')

ML Workspace/studio:

os.mkdir('data')
DATA_DIR = os.path.abspath("data")
mount_context = dataset.mount(mount_point=DATA_DIR)
mount_context.start()

output:
fusermount: mounting over filesystem type 0xfe534d42 is forbidden
PyFuseError: MountPointCreationError(Some(1))

2 Created dataset pointing to two different datastores : az_datastore,meta_datastore

dataset_two = Dataset.File.from_files([(az_datastore,r"all_img/day_one_image/**/*"),(meta_datastore,r"img_label.csv")])
dataset_two.register(workspace=ws,name='test_d2_2',description='Testing purpose only')

ML Workspace:

mounted_path = tempfile.mkdtemp()
mount_context = dataset.mount(mounted_path)
mount_context.start()
print(os.listdir(mounted_path))
print (mounted_path)

output:
Not mounting as a volume: Volume mount only supports datasets with exactly one datastore path, actual: 2.
Falling back to dataflow mount.
['external', 'meta']
/tmp/tmpojzekdud

Answer 1

romungi-MSFT 29,536 Microsoft Employee

For the first scenario, I see that error is a known issue with a particular version of fuse. Please see this bug report.
You can try to install the latest version of fuse as mentioned in the note of the documentation.

For the second scenario, I can see only one dataset is supported with relative path to the source files. Please see the reference and samples here. I think in your case the first datastore was mounted.

If an answer is helpful, please click on or upvote which might help other community members reading this thread.

V JEEVA 46 Reputation points

2022-08-29T10:31:13.333+00:00

Thanks for the info !

How to mount the azure dataset in jupyter notebook ?

Access datastores

1 Wanted to mount dataset to a specific folder that I created

2 Created dataset pointing to two different datastores : az_datastore,meta_datastore

0 additional answers

Activity