Access all files from folder and subfolders in blob storage by pipeline/data-flow

Hi @Gerald Rupp ,

Welcome to Microsoft Q&A forum and thanks for reaching out here.

As per my understanding you have json files located in your storage account in the below folder structure and sample file looks like below, please correct me if I'm wrong.

Folder Structure:

Container

 Root

     Year

        Month

           Date

                 File1.json

                 ....

                 ....

                FileN.json

Assuming your sample file looks like below:

[{
    "Type": "A",
    "Data": {
        "Attribute11" : "Value11",
        "Attribute21" : "Value21"
    }
}, 
{
    "Type": "B",
    "Data": {
        "Attribute31" : "Value31",
        "Attribute41" : "Value41"
    }
}, 
{
    "Type": "C",
    "Data": {
        "Attribute51" : "Value51",
        "Attribute61" : "Value61"
    }
}
]

And you would like to seperate the data based on the Type Attribute (which could be A, B or C) and create a folder with that name and save them in that folder. Please correct me if I'm wrong anywhere.

To achieve the above requirement, the best way is to use Mapping data flow as it will reduce the complexity and also you can transform the data as per your custom requirement.

Steps to be followed: In dataset configuration, just provide the container name and leave directory and filename empty as you will configure them in data flow using wild card path.

Then in Mapping data flow source settings, please select source options as below using wild card paths and provide Partition root path and select the document form of your source.

Next, have a conditional split transformation, which will be used to split the data in all your source files based on each Type Attribute and write them to respective folders in your storage account.

Then have Select transformation to select only the columns that you would like to copy to each sink folder based on the Type Attribute as shown above.

Hope this helps. Kindly let us know if your requirement is different than my understanding. If that's case, please share few additional details about the requirement and we would be happy to assist accordingly.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.