question

FaizanArefin-2979 avatar image
0 Votes"
FaizanArefin-2979 asked FaizanArefin-2979 commented

Copy multiple files from Storage

Dear All,

I an new to Data Engineer and need your suggestion here.
I have a pipeline with defaulted File Path and File Name as shown below.

202653-adf1.jpg

Now we have multiple files available in the same File Path.So what changes should i do so that all the files are copied from the source.

Also this a one time activity where i have to copy multiple files.How can i achieve this without publishing the Pipeline.

Thanks
Faizan


azure-data-factory
adf1.jpg (52.3 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered FaizanArefin-2979 commented

Hello @FaizanArefin-2979,

Thanks for the question and using MS Q&A platform.

If it is a onetime action and if you don't want to publish the changes, then you can clone the pipeline and just update the copy activity settings as shown in below image using wild card file name and folder path pipeline parameter for your folder path and then click on debug button (click only once) highlighted below. Since you will be using wild card naming (*) for files, all files in that folder path will be copied to your sink data store. Once you are done with this action you can undo the changes.

202952-image.png


202920-image.png

For more information please refer to this documentation: Folder and file filter examples

202858-image.png


Also here is a demonstration video by one of the community volunteers - ADF Wildcard Explained in Detail | Copy activity


Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (23.5 KiB)
image.png (53.2 KiB)
image.png (29.2 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks a lot KranthiPakala-MSFT.I understood now.

1 Vote 1 ·