Why is Sink processing time take too long to complete in my data flow activity

Ovindu Wijethunge 40 Reputation points
2023-12-13T02:54:31.1133333+00:00

Hi Team,

I created a pipeline by using a data flow activity which is going to transfer json file into data warehouse table as a row. So I used event based trigger to trigger pipeline. So whenever json file is inserted into blob storage trigger will be triggered and pipeline will be executed.

inside data flow activity I used 3 activities which is PickUp json -> Flatten Json -> Sink to Data warehouse table.

and also this has taken 1.3 vCore hours to complete and this is a simple transformation.

I will post in below a screen shots of the data flow.

Could you take a quick look at the screenshots and let me know if you have any advice on how to fix this? Your insights would be really helpful.

Thanks a lot!

Screenshot 2023-12-13 075938

Screenshot 2023-12-13 080049

Screenshot 2023-12-13 081846

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,640 questions
0 comments No comments
{count} votes

Accepted answer
  1. Smaran Thoomu 24,260 Reputation points Microsoft External Staff Moderator
    2023-12-13T11:00:01.9066667+00:00

    Hi @Ovindu Wijethunge ,

    Welcome to Microsoft Q&A Platform. Thank you for posting your query.

    Based on your description, it seems like the Sink processing time is taking long to complete in your data flow activity. There could be several reasons why this is happening, but I'll try to provide some guidance to help you resolve this issue.

    Firstly, it's important to note that Sink processing time can be affected by several factors, including the size of the data, the complexity of the transformation, and the resources allocated to the data flow activity. In your case, you mentioned that the transformation is simple, but it still took 1.3 vCore hours to complete. This suggests that there may be some inefficiencies in your data flow activity that are causing the Sink processing time to take longer than expected.

    Here are some steps you can take to troubleshoot this issue:

    1. Check the data flow activity settings: Make sure that the settings for your data flow activity are optimized for performance. For example, you can try increasing the number of partitions to improve parallelism or reducing the batch size to reduce memory usage.
    2. Check the data flow activity logs: Azure provides detailed logs for data flow activities, which can help you identify any errors or performance issues. You can access these logs from the Azure portal or using Azure PowerShell.
    3. Check the data flow activity resources: Make sure that your data flow activity has sufficient resources allocated to it. You can try increasing the number of vCores or memory allocated to the activity to improve performance.
    4. Check the data flow activity dependencies: Make sure that all the dependencies for your data flow activity are up-to-date and functioning correctly. For example, if you're using a linked service to connect to your data warehouse, make sure that the connection is working correctly. Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.