Connection reset error for dataflow activity in Azure Data Factory

Question

I keep getting error "Connection reset" once a while for dataflow activity used in Data Factory pipeline with the triggers. It is incredibly stressful as the error would be gone if I run it again in the morning when I find out.

At first I thought maybe my VM was doing automatic updates and was having restarts as part of it, since the VM is turned off every night and turned on again for the morning trigger to run the daily morning load. However, that is not the case as I have turned off all automatic updates, and the logs also confirm there is no restarting of the machine while the trigger is running.

Other copy activities in the pipelines using the same VM works fine at the same time, only the dataflow is unstable and showing this error once a while. Does anyone also have the same issue and is there anyway to fix this?

Answer 1

Hello @Quyen Dang How are you doing!

I understand that you may be feeling frustrated or concerned about the issue you are experiencing. This forum here is to help and support you in any way that is possible!

Check the network connectivity between the Data Factory and the data source. You can also try increasing the timeout value for the dataflow activity.
You can also try running the dataflow activity in debug mode to identify any issues.
You can try increasing the size of the VM or optimizing the dataflow to reduce the resource utilization.

If the issue persists, you can let us know and we will help offline!