Passing the output of copy activity to a parameter in Azure Data Factory

Question

Hi, I'm using Copy Activity to call a HTTP server for fetching the data from a URL and storing the data in SQL database. At the same time I must also use this fetched data to perform further operations in same pipeline.

So right now I'm storing data in SQL database using copy activity and again fetching the data from SQL database using lookup activity and then using it in pipeline. So I'm connecting to SQL database two times which reduces performance.

So is there a way to store the fetched data from copy activity to a parameter and then use this parameter to do further operations in pipeline?

Answer 1

Hello @Madhan ,

Thanks for the question and using MS Q&A platform.

In Mapping Data Flow , you have cache sink that can meet your requirement.

Reference : https://www.youtube.com/watch?v=9d39Bw2uFLw

However, this feature is available inside the data flow. If you are looking outside - Copy activity - this may not be feasible out of box. I would suggest you to stage the data temporarily in a blob storage.

This blob data can be written to SQL. The same can be utilized accross the pipeline.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators