how to retreive a dataset in ml studio published pipeline which is sent as a parameter from Azure data factory pipeline

Question

how to retreive a dataset in ml studio published pipeline which is sent as a parameter from Azure data factory pipeline

Shivan Pulugujja 0

I am passing triggered file as a parameter to ml studion pipeline in azure data factory

ml parameters

then i am able get the file as well in the input for the activity

ml input

Now my question is how to parse or retrieve this data and send it to next steps in pipeline

ml designer

by searching i found out we can use import data or execute python script components but didn't find exact answer. Can someone help me with the best approach. Thanks

romungi-MSFT 48,916 Reputation points Microsoft Employee Moderator

2023-06-12T10:18:49.1766667+00:00

@Shivan Pulugujja With import data you can use registered datasets and the corresponding paths in the pipeline. I think in this case you are using the pipeline which uses import data module with datastore as source but the file in question is not registered with the datastore. I think you need to first transfer or register the data in Azure ML datastore and then call the pipeline to use the file. Please see this notebook on how the different data stores are used to import data.
Shivan Pulugujja 0 Reputation points

2023-06-12T10:40:18.8033333+00:00

@romungi-MSFT i am getting the dataset from storage account directly. Whenever the data in the storage account is uploaded the ADF pipeline is triggered, i want to send that triggered file to the ML pipeline, as show in the above images i am able to get the file as well in ADF but i don't how to receive it in Azure ML
Shivan Pulugujja 0 Reputation points

2023-06-12T10:41:08.0766667+00:00

I don't know how to receive it in azure ml
Shivan Pulugujja 0 Reputation points

2023-06-12T10:41:52.1833333+00:00

I don't know how to receive the data set and send it to next pipeline steps
romungi-MSFT 48,916 Reputation points Microsoft Employee Moderator

2023-06-13T10:46:31.6166667+00:00

I think you can use the web service input in your designer pipeline which can use the storage location of your file along with import data module to download the file using HTTP URI. By passing the storage URI as input to the pipeline endpoint by configuring the pipeline endpoint id in Machine Learning Execute Pipeline activity the file location is available can be made available for download for import data module.

Your answer

romungi-MSFT 48,916 Reputation points Microsoft Employee Moderator

2023-06-12T10:18:49.1766667+00:00

@Shivan Pulugujja With import data you can use registered datasets and the corresponding paths in the pipeline. I think in this case you are using the pipeline which uses import data module with datastore as source but the file in question is not registered with the datastore. I think you need to first transfer or register the data in Azure ML datastore and then call the pipeline to use the file. Please see this notebook on how the different data stores are used to import data.
Shivan Pulugujja 0 Reputation points

2023-06-12T10:40:18.8033333+00:00

@romungi-MSFT i am getting the dataset from storage account directly. Whenever the data in the storage account is uploaded the ADF pipeline is triggered, i want to send that triggered file to the ML pipeline, as show in the above images i am able to get the file as well in ADF but i don't how to receive it in Azure ML
Shivan Pulugujja 0 Reputation points

2023-06-12T10:41:08.0766667+00:00

I don't know how to receive it in azure ml
Shivan Pulugujja 0 Reputation points

2023-06-12T10:41:52.1833333+00:00

I don't know how to receive the data set and send it to next pipeline steps
romungi-MSFT 48,916 Reputation points Microsoft Employee Moderator

2023-06-13T10:46:31.6166667+00:00

I think you can use the web service input in your designer pipeline which can use the storage location of your file along with import data module to download the file using HTTP URI. By passing the storage URI as input to the pipeline endpoint by configuring the pipeline endpoint id in Machine Learning Execute Pipeline activity the file location is available can be made available for download for import data module.

Share via

how to retreive a dataset in ml studio published pipeline which is sent as a parameter from Azure data factory pipeline

Your answer