Azure Data Factroy - select files from a folder based on a wildcard filename

Ja 1 Reputation point
2020-09-11T10:51:18.72+00:00

I'm new to ADF and thought I'd start with something which I thought was easy and is turning into a nightmare!
I'm trying to do the following.

Copy files from a ftp folder based on a wildcard e.g. 'PN'.csv and sink into another ftp folder. (wildcard* in the 'wildcardPNwildcard.csv' have been removed in post)

I have ftp linked servers setup and a copy task which works if I put the filename, all good.

I need to send multiple files so thought I'd use a Metadata to get file names, but looks like this doesn't accept wildcard...

Can this be done in ADF, must be me as I would have thought what I'm trying to do is bread and butter stuff for Azure.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,341 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. HarithaMaddi-MSFT 10,126 Reputation points
    2020-09-11T15:29:18.253+00:00

    Hi @Ja ,

    Welcome to Microsoft Q&A Platform. Thanks for posting the query.

    Azure Data Factory enabled wildcard for folder and filenames for supported data sources as in this link and it includes ftp and sftp. Please click on advanced option in dataset as below in first snap or refer to wild card option from source in "Copy Activity" as below and it can recursively copy files from one folder to another folder as well.
    24060-wildcardexpression.png
    24211-advancedoptiondataset.png

    Please suggest if this does not align with your requirement and we can assist further.