Hi All, I have parquet files year, month wise in ADLS. Now, I need to copy the data from these parquet files and push it to corresponding customer tables in Azure SQL Managed instance. In the destination we don't have these customer tables already create. So, I want to dynamically check if the table exists in destination location before copying the data and if the table is not there then I need to dynamically create those individual customer tables and then push the data. Is it possible? Regards, Mohanraj

Hi You can use copy activity and auto create table option in the sink settings to achieve what you want. Please go through this documentation for more details. For you to use this feature, make sure to pass the table name to data set dynamically. Here is a video reference for the same. Hope it helps. Thanks.

Dynamic table creation in Pre-Copy script through ADF

Accepted answer

Subashri Vasudevan 11,206 Reputation points

2023-08-27T07:36:51.8533333+00:00

Hi

You can use copy activity and auto create table option in the sink settings to achieve what you want. Please go through this documentation for more details. For you to use this feature, make sure to pass the table name to data set dynamically.

Here is a video reference for the same.

Hope it helps.

Thanks.
Please sign in to rate this answer.
Mohanraj Ramalingam 150 Reputation points

2023-08-27T14:25:53.88+00:00

Thanks @Subashri Vasudevan for the suggestion. I am aware of "Auto create table" option. The issue is first time if I load the data then "Auto create table" option is fine. But, In my case every month I need to insert data into the same table and every month I don't to create new table in the destination system after my pipeline is run for each month. So, I thought of include a script in Pre-Copy option to check if table is not there then to create a table. But, the Pre-Copy script doesn't accept the dynamic values and same goes for "Execute script" option as well.

Regards,

Mohanraj

Subashri Vasudevan 11,206 Reputation points

2023-08-27T14:28:31.67+00:00

Hi, thanks for the details. Auto create option works like this. If table is there, it won't create. Otherwise it will create. So i think you can still use auto create table option.

Subashri Vasudevan 11,206 Reputation points

2023-08-29T04:21:29.65+00:00

Hi Mohanraj

Did you get a chance to check the comment? If the answer is helpful, request you to Accept it as answer. Or if you have questions, feel free to let us know.

Thanks.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Dynamic table creation in Pre-Copy script through ADF

0 additional answers

Your answer