copy from fileshare to blob container

Question

copy from fileshare to blob container

Niren Adhikary 96

Hello,

I need to copy file from source path in fileshare to destination blob container in ADLS2 using an automated process. Basically when a file is modified/updated in the file share, the file should get copied to destination blob container and the file should be renamed.

What is the best possible way to do that ? i checked using login app but could not find a connector for ADLS

Thanks

MartinJaffer-MSFT 26,236 Reputation points

2022-12-12T21:42:00.38+00:00

Hello and welcome to Microsoft Q&A @Niren Adhikary

I understand you want to copy / move files from Azure Storage files (?) to Azure Storage Data Lage Gen2. However I am not sure what service you are using or what "login app" is. Do you have a service / product / method for moving, or are you looking for one? There are many ways to copy, but fewer ways to detect modify/upload. Usually that would involve Event Grid, if supported. I'm not seeing an obvious connector for fileshares. Looking from the fileshare docs, I'm not seeing much, but there are the logs we could piggyback off of, thought that would be slower I expect.
Niren Adhikary 96 Reputation points

2022-12-13T02:16:09.49+00:00

I am simply looking for whenever a file is modified/created in azure file share directory then it should move the file from source file share location to destination blob container. In logic app, the connector is missing for fileshare but it has for blob container. How can I achieve it.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-12-15T19:20:04.813+00:00

Hello @Niren Adhikary ,

We still have not heard back from you. Just wanted to check if any of the below workarounds were helpful? And, if you have any further query do let us know.

Thank you

1 answer

Your answer

MartinJaffer-MSFT 26,236 Reputation points

2022-12-12T21:42:00.38+00:00

Hello and welcome to Microsoft Q&A @Niren Adhikary

I understand you want to copy / move files from Azure Storage files (?) to Azure Storage Data Lage Gen2. However I am not sure what service you are using or what "login app" is. Do you have a service / product / method for moving, or are you looking for one? There are many ways to copy, but fewer ways to detect modify/upload. Usually that would involve Event Grid, if supported. I'm not seeing an obvious connector for fileshares. Looking from the fileshare docs, I'm not seeing much, but there are the logs we could piggyback off of, thought that would be slower I expect.
Niren Adhikary 96 Reputation points

2022-12-13T02:16:09.49+00:00

I am simply looking for whenever a file is modified/created in azure file share directory then it should move the file from source file share location to destination blob container. In logic app, the connector is missing for fileshare but it has for blob container. How can I achieve it.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-12-15T19:20:04.813+00:00

Hello @Niren Adhikary ,

We still have not heard back from you. Just wanted to check if any of the below workarounds were helpful? And, if you have any further query do let us know.

Thank you

Answer 1

KranthiPakala-MSFT 46,642 Microsoft Employee Moderator

Hello @Niren Adhikary ,

Welcome to the Microsoft Q&A platform and thanks for posting your query.

Could you please elaborate a bit on your source (fileshare). Is it an onPrem file share path? or something else?
In case if it is On prem file share path, then Azure data factory would be ideal for your requirement. You can use copy activity in ADF to copy files from your file share to ADLS using windows authentication.

For more info about the FileSystem connector in ADF please refer to this document: Copy data to or from a file system by using Azure Data Factory or Azure Synapse Analytics

In order to copy files to your destination ADLS you can use ADLS connector in ADF. Please refer to this document for more info about this connector: Copy and transform data in Azure Data Lake Storage Gen2 using Azure Data Factory or Azure Synapse Analytics

Using copy activity in ADF you can also rename the file name before copying it to your desired destination.

Since you have a requirement to copy when the files are updated/modified, you will have to write an application or web function to trigger your ADF pipeline. Or you can also create a workflow in Azure Logic apps and trigger it when a file is created or added to a file share in On prem file system and in the same workflow, you can trigger the ADF pipeline when the file is being added or created in the file share.

For more info about this approach please refer to below documents:

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Niren Adhikary 96 Reputation points

2022-12-13T02:14:43.623+00:00

I am simply looking for whenever a file is modified/created in azure file share directory then it should move the file from source file share location to destination blob container. In logic app, the connector is missing for fileshare but it has for blob container. How can I achieve it.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-12-13T14:00:04.44+00:00
Hello @Niren Adhikary ,

Thanks for clarifying the source details. If your source is Azure file share (Azure file storage) then ADF has a built-in connector to copy files from Azure File share, but ADF do not have any existing feature that could trigger your pipeline based on new events (new file added/modifiled) in Azure File Share. We have event triggers in ADF but that doesn't support Azure File Share (It only works with Azure Data Lake Storage Gen2 and General-purpose version 2 storage accounts). Neither Azure function nor Logic App supports triggers on Azure file share.

Couple of workarounds I could think of are:

A closest workaround could be to use ADF schedule trigger to trigger your ADF pipeline that copies files from Azure File Share to your ADLS Gen2 on scheduled (periodic) intervals.

If Schedule trigger doesn't fits your requirement, then you will have to write a custom code or an application which has to monitor azure file for a new file arrival and call a REST API to trigger the ADF pipeline (Pipelines - Create Run) upon new file arrival.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-12-13T14:03:57.94+00:00
Continuation to above comment:

Below are few similar feedbacks logged by other users in Data factory IDEAS forum. We recommend you please up-vote and comment on them to help increase the priority of the feature requests. All the feedback shared in this forum are actively monitored and reviewed by respective product owners and priority the feature requests accordingly:

Trigger for events in Azure File Share

Event triggers for files and folder in Azure File Share

Storage event triggers from a file share within Data Factory

Hope this info helps. Do let us know if you have further query.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-12-13T19:05:57.9+00:00

Hello @Niren Adhikary ,

To elaborate a bit on schedule trigger workaround implementation, First, you will have to create a Schedule trigger which will trigger the pipeline every one hour. And in the Pipeline copy activity source settings, you will have to configure Filter by last modified with reference to your pipeline trigger time and look for 1 hour back window and if any new files added or modified in that 1-hour window it will get copied to your sink location.

Schedule Trigger Configuration:

Copy Activity Source configuration to look back for past 1 hour window from the trigger execution time.

Hope these details help. Do let us know if you have further query.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-12-14T18:16:41.613+00:00

Hello @Niren Adhikary ,

Just checking in to see if you have got a chance to see my previous response with workaround details. And, if you have any further query do let us know.

Thank you

Share via

copy from fileshare to blob container

1 answer

Your answer