Data flow error

Suba Balaji 11,206 Reputation points
2023-05-28T12:39:54.31+00:00

Hi Team,

I have a data flow which i want to loop through multiple sub folders.

Folder structure:

X-->1A

X-->2A

Y-->1A

Y-->2A

where, X and Y are root folders in ADLS containter (container1) and 1A, 2A are sub folders.

Inside 1A, 2A, i have a csv file of same structure.

During runtime, i want to pass X or Y and then 1A and 2A.

I tried to use wildcard file setting in path by passing folder hierarchy as X/1A/*.csv

But i am getting below error during run time,

{"message":"Job failed due to reason: at Sink 'sink1': Only file formats are allowed in the file store. Details:","failureType":"UserError","target":"Data flow1","errorCode":"DFExecutorUserError"}

Please help in this regard, if it is allowed to use the folder structure as below,

TIA

Screenshot 2023-05-28 at 6.08.09 PM

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,122 questions
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee
    2023-05-30T23:33:54.74+00:00

    @Suba Balaji

    Thanks for using Microsoft Q&A forum and posting your query.

    Firstly, the error message seems wired as it is complaining about the sink instead of source. Are you using dynamic expression and the same dataset being used in source and sink transformations?

    I just tried quickly with hard coded path for wild card path and everything seem to work as usual.

    Here is my source folder structure looks like in ADLS Gen2.

    User's image

    Below is how I configured in my Data flow source:
    User's image

    In dataset point it to container. And then in data flow source options settings, configure the path as below:

    wild card paths = input/TextFiles/2020/*/*/*.txt

    User's image

    Was able to successfully preview the data of all subfolders based on the wild card path configuration:

    User's image

    Were you able to configure the same wild card path in copy activity and able to preview the data without any issues? I would suggest to try it out and see if something is missing in your configuration.

    Hope this helps. Let us know how it goes


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    1 person found this answer helpful.
    0 comments No comments

0 additional answers

Sort by: Most helpful