Data is not selected for copying from source to sink by ADF when Logic App runs at same time as Copy Data Activity

Jun Ting Ho 141 Reputation points
2021-12-10T09:51:21.12+00:00

Hi all,

My team and I have a workflow that is currently running as follows:

156570-20211210-microsoftteams-image.png

However, recently we’ve found that if the Logic App is running at the same time as the “Copy Data” activity, what would occur sometimes is that the data would be updated by the Stored Procedure, but would not be selected for copying from source to sink (i.e. we find the data updated within the DB but are unable to find it within the sink .txt file). However, data present prior to the commencement of the “Copy Data” activity would be unaffected.

Could we ask why such a situation would occur, and what a rectification would be?

Azure SQL Database
Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
2,839 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,526 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2021-12-14T01:14:38.487+00:00

    Hello @Jun Ting Ho ,
    Thanks for the ask and using Microsoft Q&A platform .
    If I understand this correct , we have two different processes ( Logic apps & ADF ) which are running on on the same table and due to timing issues , i think you are running into issues .
    In case if my understanding is correct , i think you can do this two ways .

    1. Call the logic app from the ADF first and then Copy activity . This way lineage will be set . You can look into this example . https://stackoverflow.com/questions/67585624/how-to-securely-invoke-a-azure-logic-app-from-azure-data-factory
    2. This option is very simple , in the existing ADF pipeline , add a lookup and try to check if the data is inserted by the logic app ( may be querying the timestamp ) , add an if statement and inside the if add the copyting activity . The logic should to check if there is any latetst record using lookup , if its then copy the data else do nothing .

    Please do let me know how it goes .
    Thanks
    Himanshu

    -------------------------------------------------------------------------------------------------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators