How to copy folders+files as per the folder names passed as parameter.

Question

Hi,

I have an on-prem file storage, there I have a folder named 'India'. Inside 'India', there are 36 folders with all the state/union teritory names. I want to build a copy data pipeline that takes a parameter for state names, and while copying it will exclude the folder names given in the parameter.

for example, if I give 'Kerala', it will exclude Kerala. If I give 'Kerala','Karnataka','Odisha' it will exclude them while copying.

Thanks.

Answer 1

Hi Prasenjit Patnaik ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As I understand your query, you want to exclude processing the folder whose foldername is supplied by the pipeline parameters. Please let me know if that is not the requirement.

You can use filter activity after get metadata activity to filterout the folder which is supplied by the pipeline parameter. However, your scenario is different as you want to pass multiple state names in the parameter and want to exclude all of them from the execution which would be complex as it would require multiple loops in ADF .

In dataflow, you can use exceptfunction to which returns a difference set of one array from another . However, similar function is not present in ADF directly.

I would suggest you to load all the foldernames into a SQL table and perform relevant join to find out the non matching rows.

For implementation part for loading get metadata output into sql, please checkout this video: How to copy Top N last modified files from one ADLS folder to another using ADF pipeline

For fetching out the non matching rows, please check out this post:

Sql Query help to get non matching records from two tables

Hope it helps. Kindly accept the answer if it's helpful. Let me know if you want me to provide complete solution in case you find any difficulty. Thanks