Adf - filenames from folder based on wildcard.

Question

Adf - filenames from folder based on wildcard.

Pravalika-randstad 240

I'm new to ADF, and I started with what I thought would be easy, but it's becoming quite challenging! Here's what I'm trying to achieve:

I want to copy files from an FT folder using a wildcard, like 'PN*csv,' and send them to another FTP folder. I have FTP linked servers set up, and it works well when I specify a specific filename.

Now, I need to transfer multiple files, so I thought of using Metadata to get the file name: it seems that Metadata doesn't accept wildcards.

Is it possible to do this in ADF? I assume it should be straightforward in Azure, so it migh something I'm missing."

Subashri Vasudevan 11,231 Reputation points

2023-10-17T05:35:19.2133333+00:00

Hi Pravalika

There is no need for getmetadata activity or looping mechanism, as the copy activity itself supports wildcard based file names.

As mentioned by @Smaran Thoomu you can point to any folder from (in my case output) and in the copy activity, choose wildcard file name and give * in the file name portion as below

In the sink, simply mention a folder name where you want to copy files to (wildcardcopy in my case) and leave the filename part empty as below -

This is for ADLS to ADLS but I hope it works for SFTP\FTP based source sink too.

Please let us know if thats of help.

Accepted answer

0 additional answers

Your answer

Subashri Vasudevan 11,231 Reputation points

2023-10-17T05:35:19.2133333+00:00

Hi Pravalika

There is no need for getmetadata activity or looping mechanism, as the copy activity itself supports wildcard based file names.

As mentioned by @Smaran Thoomu you can point to any folder from (in my case output) and in the copy activity, choose wildcard file name and give * in the file name portion as below

In the sink, simply mention a folder name where you want to copy files to (wildcardcopy in my case) and leave the filename part empty as below -

This is for ADLS to ADLS but I hope it works for SFTP\FTP based source sink too.

Please let us know if thats of help.

Answer 1

Smaran Thoomu 24,575 Microsoft External Staff Moderator

@Pravalika-randstad Thanks for posting your Query. As I understand you're trying to copy files from an FTP folder using a wildcard and send them to another FTP folder in Azure Data Factory (ADF).
Azure Data Factory now supports wildcards for both folder and filenames in various data sources, including FTP and SFTP. To utilize this feature, you can click on the 'Advanced' option in the dataset, as shown in the first screenshot. Alternatively, you can explore the wildcard option in the 'Copy Activity,' allowing you to efficiently copy files between folders, including recursive copying.
User's image

User's image

Relevant links:

Please let us know if this doesn't meet your needs, and we'll be happy to provide additional assistance.

Smaran Thoomu 24,575 Reputation points Microsoft External Staff Moderator

2023-10-18T03:40:52.1366667+00:00

@Pravalika-randstad Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Adf - filenames from folder based on wildcard.

0 additional answers

Your answer