Azure ML Datastore\Datasets

Question

Hello:

I want to know that if it is possible automate copy file from azure storage to Azure ML folder.

I understand that it is duplication of data, but I want to know if yes, how I can do that.

Any pointer is greatly appreciated.

Thanks

Answer 1

Depending on the frequency at which you would like to move data you can create scripts that could run on crontab to move the data between source storage account to your workspace blob store. For example, use azcopy to perform this activity.

A very comprehensive method to move storage between storage accounts is available as a Microsoft learn module that you could take to understand the possibilities and attain this from code to automate in your application.

I would ideally assume that you would like to pull data when your experiment kicks off because you cannot move data to an experiments run id folder unless the experiment has started, In this case you could use the first option to place the data in your workspace blob store and then use it in your experiment without moving it to any other storage. I hope this helps.