How to create single dataset for multiple azure data flows?

Question

How to create single dataset for multiple azure data flows?

Pankaj Mandania 61

I have about 50 files that i need to transform using a Azure Data Flows. Is there a way i can create a single data set for multiple data flows? I want to avoid creating a dataset for every single file.

Nandan Hegde 36,716 Reputation points MVP Volunteer Moderator

2023-09-19T14:10:32.5033333+00:00

Hey, You can create a dynamic dataset wherein a single dataset can be created for all files by passing the values of the file path and name at run time https://sqlkover.com/dynamic-datasets-in-azure-data-factory/amp/
Harishga 6,005 Reputation points Microsoft External Staff

2023-09-21T11:07:24.9566667+00:00

@Pankaj Mandania We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Nandan Hegde 36,716 Reputation points MVP Volunteer Moderator

2023-09-19T14:10:32.5033333+00:00

Hey, You can create a dynamic dataset wherein a single dataset can be created for all files by passing the values of the file path and name at run time https://sqlkover.com/dynamic-datasets-in-azure-data-factory/amp/
Harishga 6,005 Reputation points Microsoft External Staff

2023-09-21T11:07:24.9566667+00:00

@Pankaj Mandania We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Harishga 6,005 Microsoft External Staff

@Pankaj Mandania

Welcome to Microsoft Q&A platform and thanks for posting your question here.

There are various ways to define source files in Data Flow.

One file could contain a list of the sources you want to use. (Single dataset; not inline)

You can choose to use a folder's entire contents.

A wildcard expression is possible.

These all would eliminate the need for 50 datasets. Personally, I like to use inline in Data Flow rather than datasets. So, I won't have cluttered my workspace.

I hope this information helps you resolve the issue. Let me know if you have any further questions or concerns.

Harishga 6,005 Reputation points Microsoft External Staff

2023-09-22T11:43:06.1133333+00:00

Pankaj Mandania
Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

How to create single dataset for multiple azure data flows?

1 answer

Your answer