how to retrieve parameterized dataset of datafactory by rest api

Zhuozhen Zhao 1

In azure data factory, we define some parameterized datasets, for example, we define a common parquet dataset with the parquet file path parameterized.
What I wanna do is to retrieve the dataset with an exact parameter value by rest API.
But with the document(https://learn.microsoft.com/en-us/rest/api/datafactory/datasets/get), it seems not able to retrieve the exact dataset with a specific parquet path.
So is there any other rest API can work it out?

MartinJaffer-MSFT 26,081 Reputation points

2021-06-07T20:04:04.02+00:00

Hello @Zhuozhen Zhao and welcome to Microsoft Q&A.

Are you trying to retrieve the data in the file, or the dataset definition?

The document you pointed to is for fetching the dataset definition. Since it is retrieving the definition, specifying the parameter value doesn't make sense. I think I am misunderstanding your ask.

Could you please clarify?
Zhuozhen Zhao 1 Reputation point

2021-06-08T06:13:22.883+00:00

Hi Martin, thanks for your reply.
We are building a metadata service that includes data lineage as well. So schema of the dataset is required. But when the dataset is defined as parameterized, the structure returned by the rest API(https://learn.microsoft.com/en-us/rest/api/datafactory/datasets/get) always return an empty schema. So my question is how to specify the parameter values of the parameterized dataset, so that the schema of the specific dataset could be returned by the rest api.
MartinJaffer-MSFT 26,081 Reputation points

2021-06-16T03:30:06.5+00:00

@Zhuozhen Zhao

In some cases, the dataset schema was never defined.

If the schema was not saved as part of the dataset, then there is nothing to retrieve, it will come back empty.

First try opening the data factory, and going to the schema and importing the dataset. If parameterized it will ask for a value.

Some dataset types like REST do not even keep the schema. I noticed for the REST dataset type, "schema":[] while in a JSON dataset it was "schema":{} . The JSON had option to import schema while REST did not.

Also, be aware this call does not work with Git. This call queries the published items.
Zhuozhen Zhao 1 Reputation point

2021-06-16T06:13:38.217+00:00

Hi,
So the key point to let the rest API of DataSet return schema is that the schema should be explicitly defined in the DataSet's schema tab UI.
Am I correct?

Unfortunately, in our project, some datasets are defined as "template dataset". For instance, we defined a MySQL db dataset, whose table name is a parameter, and it's specified in a certain pipeline. So the schema of the dataset is quite dynamic depending on the value of the parameter.
Is there any suggestion on how to retrieve the schema of the "template dataset" with a given value of the parameter?
MartinJaffer-MSFT 26,081 Reputation points

2021-07-12T16:19:59.627+00:00

You are correct. The API is not fit for this purpose.

It maybe possible to get the table schema with a get metadata activity.

Share via

how to retrieve parameterized dataset of datafactory by rest api

Your answer