Data flow error

Question

Hi Team,

I have a data flow which i want to loop through multiple sub folders.

Folder structure:

X-->1A

X-->2A

Y-->1A

Y-->2A

where, X and Y are root folders in ADLS containter (container1) and 1A, 2A are sub folders.

Inside 1A, 2A, i have a csv file of same structure.

During runtime, i want to pass X or Y and then 1A and 2A.

I tried to use wildcard file setting in path by passing folder hierarchy as X/1A/*.csv

But i am getting below error during run time,

{"message":"Job failed due to reason: at Sink 'sink1': Only file formats are allowed in the file store. Details:","failureType":"UserError","target":"Data flow1","errorCode":"DFExecutorUserError"}

Please help in this regard, if it is allowed to use the folder structure as below,

TIA

Answer 1

@Subashri Vasudevan

Thanks for using Microsoft Q&A forum and posting your query.

Firstly, the error message seems wired as it is complaining about the sink instead of source. Are you using dynamic expression and the same dataset being used in source and sink transformations?

I just tried quickly with hard coded path for wild card path and everything seem to work as usual.

Here is my source folder structure looks like in ADLS Gen2.

Below is how I configured in my Data flow source:

In dataset point it to container. And then in data flow source options settings, configure the path as below:

wild card paths = input/TextFiles/2020/*/*/*.txt

Was able to successfully preview the data of all subfolders based on the wild card path configuration:

Were you able to configure the same wild card path in copy activity and able to preview the data without any issues? I would suggest to try it out and see if something is missing in your configuration.

Hope this helps. Let us know how it goes

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.