Creating a dataset with a parameterized schema

Question

Hi,

in order to import csv files or SQL tables with the goal to get a specified data set, not all data, for each csv file or SQL table, is it possible to create a dataset parameterizing the data schema?

Thanks

Answer 1

Yes, it is possible.

First define the schema that you want to apply where you specify the structure of the dataset, including the data types and columns you want to include.

You can parameterize the schema in a way that allows dynamic adaptation based on the source data.

This could be achieved through scripting or by using Azure Synapse's built-in tools.

You can later create a mapping data flow where you can specify the source, the sink (where the data will be loaded), and the transformations, which include applying your schema.

Here is more in the documentation :

https://learn.microsoft.com/en-us/azure/data-factory/parameters-data-flow

Answer 2

Hi pmscorca ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

As per my understanding you are trying to ask if there is any option to parameterize the dataset to have a schema that represents a subset of the entire set of fields. Kindly confirm if you are asking about having dynamic mapping of schema while copying the data using ADF/Synapse pipeline.

You can create customized mapping having desired schema in a json format and use the same in mapping tab of copy activity by removing default auto-mapping option.

Kindly checkout the following video for implementation purpose:

Dynamic Column mapping in Copy Activity in Azure Data Factory

In case you are searching for the way to parameterize dataset , kindly watch out :

Parameterize the dataset and pipeline in ADF

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou