ADF copy source file path with wildcard in dynamic parameter

Unknown-2795 51 Reputation points


We have ADF copy and the source file path which has date in it. The date is getting resolved with an input pipeline parameter. The problem is input parameter date has yyyyMMddHH and the actual path has yyyyMMddHHmmss. So in input we don't get minutes and seconds.

for example, in input parameter we get 10/20/2022 05:00:00, which will be get formatted to 2022102005. And the actual paths look like these

a/b/c/20221020051356/1.parquet, 2.parquet...
a/b/c/20221020065434/1.parquet, 2.parquet...
a/b/c/20221020072241/1.parquet, 2.parquet...

How can I create a dataset with wild card, which can resolve the above folders ?

Here is our dataset example. Can you please give me an example of how the wildcard file path can be used here?



Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
8,447 questions
{count} vote

1 answer

Sort by: Most helpful
  1. HimanshuSinha-msft 19,246 Reputation points Microsoft Employee

    Hello @Unknown-2795 ,
    Thanks for the question and using MS Q&A platform.
    As we understand the ask here is how to create a path like YYYYMMDDHH* ( eg ( 2022201005* ) , please do let us know if its not accurate.
    I see that since you are already paramterizing the dataet , i think you are going in the right direction .

    I was able to male it will in two steps .

    Step 1 . Use the split function to read the parameter and split the string at '/' and then also replace the black which i see after the YYYY ( eg /2022 05 )

    @split(replace(substring(pipeline().parameters.DateAsParameter,0,13),' ',''),'/')

    Note the Split function always writes back to a variable type array ( see the screenshot below ) .

    Step 2 .
    Use the concatenate function (substring(variables('var1')2,0,4),variables('var1')1,variables('var1')[0],substring(variables('var1')2,4,2),'*')

    Note : I am using the variables('var1')2 twice to get the YYYY and also HH .

    I did tested the solution for one of the date which you posted .

    "name": "NewPath",
    "value": "2022201005*"
    Your path is


    Please do let me if you have any queries.

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.
    0 comments No comments