question

keonabut avatar image
0 Votes"
keonabut asked azure-cxp-api edited

Azure ML Dataset and Snapshot

Hi experts,

My customer want to snapshot datasets for reproducibility. I found method "create_snapshot", but found that it is deprecated. Is there any alternative way for dataset snapshot ?


Thanks,
Keita


azure-machine-learning
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

GiftA-MSFT avatar image
0 Votes"
GiftA-MSFT answered GiftA-MSFT edited

Currently datasets don't have snapshot capabilities. However, you can develop a heuristic where you create a snapshot of your data via blob (i.e if they are using blob). With the new dataset API, you are able to version and track datasets. A version will refer to your data but won't create a point in time snapshot. Hence, we recommend that you format your data to be in folders, so that when new data is added, it creates a folder for it, then the version will refer to old data (old folder) plus the new data (new folder). Please check out this document on how to version and track Azure Machine Learning datasets for reproducibility.



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Snapshot capability is necessary because Azure ML Dataset is just a reference to Azure Storage and Database. Is there any alternative way for snapshot dataset in Azure ML ?

0 Votes 0 ·

Please review updated response. The product team would be glad to discuss further to understand your scenario. Please let me know if you would like to discuss further and I can help facilitate. Thanks.

0 Votes 0 ·