Moving and manipulating on prem files from ADF

AzeemK 516 Reputation points
2021-08-19T22:03:26.02+00:00

I have few .csv files on onprem share , I want to use my ADF pipeline to move these files to archive folder also open these files and replace delimiter before ingestion to ADLS

Batch processing ?

Azure Function ?

any ideas

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,833 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. KranthiPakala-MSFT 46,512 Reputation points Microsoft Employee
    2021-08-20T13:35:46.04+00:00

    Hi @AzeemK ,

    Thanks for using Microsoft Q&A forum and posting your query.

    You can use Azure Data Factory pipeline with Copy activity to achieve this requirement.

    If you want to just copy the file with a change in delimiter, then you can use copy activity and in sink dataset, you can change the delimiter that you want to create files with.

    125035-image.png

    If your ask is to MOVE (copy source file to target and delete on source location) the file, then you will have to use COPY activity (having sink dataset edited to have your desired delimiter) followed by a DELETE activity to delete the file from source after copying to sink target location.

    Here is a related blog (Disclaimer: Not an MSFT owned) by a community volunteer: Move Files With Azure Data Factory- Part I

    Hope this info helps. Do let us know if you have further query.

    ----------

    • Please accept an answer and up-vote if it helps. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.