Share via

Azure Blob Storage Dataset Wild card file name

jigsm 236 Reputation points
2020-10-08T01:10:04.117+00:00

Hello,

I have a requirement where in the user will upload delimited file in the Azure Blob Storage and the Azure Data Factory pipeline will copy the file from Azure Blob Storage

and dump it into Azure SQL DB.

To achieve this, I have created a Container by the name-Import and in that I have created folder by the name Incoming.

so, when the user uploads the file, the file will be added in Import/Incoming location of the Blob Storage.

The user will upload files of any file name like import.csv or Import_10072020.csv etc.

I have configured an Event based trigger which will trigger when the blob is created.

No my difficulty is that how to set the wild card file name in the Event Trigger and Azure Blob Storage Dataset and Copy Data sSource

If I hard code the file name then everything works fine, but as the file names will not be constants, how to set the wild card file name in Event Trigger, Dataset and in

Copy Data source? (Screen shots attached for reference)

30764-eventtrigger.png

30765-datasetwildcardfilename.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,182 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee
    2020-10-10T00:44:41.287+00:00

    Hi @jigsm ,

    Thanks for your response and confirmation. Please follow below steps to achieve your requirement
    Note: Please make sure that your subscription is registered with the Event Grid resource provider as event triggers are dependent on Azure Event Grid

    1. Create pipeline parameters as below and do not input any values to them, since you will be passing those values from trigger parameters in the next steps.
      31289-pipelineparameterseventtriggere.png
    2. Then create your event trigger from pipeline as shown below (here my container name is copyfromsqlserver please update with your container name)
      31197-image.png 31334-image.png
    3. Then create even trigger parameters which maps to pipeline parameters created earlier.
      31299-image.png
    4. Then create your copy source dataset and create dataset parameters as shown below.
      31374-image.png
    5. Then use those dataset parameters inside dataset connection settings
      31258-image.png
    6. Then map these dataset parameters to pipeline parameters as shown below under Copy Source settings.
      31259-image.png
    7. Now configure your sink accordingly and publish your pipeline and drop a blob/file in source location which would eventually trigger your pipeline.

    Hope this helps. If you have further query, do let me know.

    Thank you

    ----------

    Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.