Hi @Ovindu Wijethunge ,
Welcome to Microsoft Q&A Platform. Thank you for posting your query.
Based on your description, it seems like the Sink processing time is taking long to complete in your data flow activity. There could be several reasons why this is happening, but I'll try to provide some guidance to help you resolve this issue.
Firstly, it's important to note that Sink processing time can be affected by several factors, including the size of the data, the complexity of the transformation, and the resources allocated to the data flow activity. In your case, you mentioned that the transformation is simple, but it still took 1.3 vCore hours to complete. This suggests that there may be some inefficiencies in your data flow activity that are causing the Sink processing time to take longer than expected.
Here are some steps you can take to troubleshoot this issue:
- Check the data flow activity settings: Make sure that the settings for your data flow activity are optimized for performance. For example, you can try increasing the number of partitions to improve parallelism or reducing the batch size to reduce memory usage.
- Check the data flow activity logs: Azure provides detailed logs for data flow activities, which can help you identify any errors or performance issues. You can access these logs from the Azure portal or using Azure PowerShell.
- Check the data flow activity resources: Make sure that your data flow activity has sufficient resources allocated to it. You can try increasing the number of vCores or memory allocated to the activity to improve performance.
- Check the data flow activity dependencies: Make sure that all the dependencies for your data flow activity are up-to-date and functioning correctly. For example, if you're using a linked service to connect to your data warehouse, make sure that the connection is working correctly. Hope this helps. Do let us know if you any further queries.
If this answers your query, do click Accept Answer
and Yes
for was this answer helpful. And, if you have any further query do let us know.