Share via

weird behaviour of copy data activity in azure data factory

n k 106 Reputation points
2020-09-22T09:03:04.213+00:00

Hi Experts,
i have 2 activity which copies the data from table in azure sql db to other azure sql db which is in VNET.
config of source sql db is S7800 DTU and that of destination db is S6400 DTU.
the amount the data which is transferred in both the tables is around 400K.
we are using self hosted integration runtime for transferring the Data . Since the destination is on VNET.
VM Specs are Standard D8s v3 (8 vcpus, 32 GiB memory).
am facing a weird issue
out of 2 copy activity one completed quickly in secs and throughput is in mbs 7.8mb/sec
however other activity takes around 18 mins to complete and throughput is 320Kb/sec
Both the activities are in one pipeline and parallel 26442-copy1.jpg26443-copy2.jpg
can any one please advise what can be issue

Thanks,

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,623 questions
0 comments
Accepted answer
  1. n k 106 Reputation points
    2020-09-28T09:12:06.8+00:00

    Hi ,
    We Managed to solve this internally . Just in case if this can be help to some one.
    there was type conversion properties in mapping section of slower task . we went in Json of pipeline and made that property to false which was true and transfer started happening quickly again .
    28599-image.png

    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator
    2020-09-22T20:02:42.807+00:00

    Hi @n k ,

    Welcome to Microsoft Q&A platform and thanks for your query.

    One thing I could think of is, CPU usage might be high on the Self-hosted IR node or available memory is low on your SHIR machine while running the copy which took 18 minutes duration.

    In order to troubleshoot the Integration runtime performance, please check the IR logs for more information.

    26622-image.png

    There could be others reasons as well. I would recommend you to please refer to this doc to troubleshoot your copy activity on your SHIR.
    Please refer to Understand copy activity execution details section as well as Troubleshoot copy activity on Self-hosted IR section to understand which stage has the longest duration, and apply the guidance called out in the doc to boost copy performance.

    Note:
    26613-image.png

    Hope this info helps. If you need further assistance on this, please do share both the pipeline run ID's for deeper analysis, we will be happy to take a deeper look and assist further.

    Thank you

    ----------

    Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.