Broadcast join timeout error

Question

The pipelines are scheduled to refresh everyday and they used to refresh & execute successfully. But from past 2 days I am getting an error - "Broadcast join timeout error". The pipelines are taking more time than usual to execute and failing eventually.

Can anyone help me with this please.

Answer 1

Hi T Venkat Anurag ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As I understand your issue, you are getting Broadcast timeout error in the mapping dataflow. Please let me know if that is not the correct understanding.

If the size of the broadcasted data is too large for the Spark node, you may get an out of memory error. To avoid out of memory errors, use memory optimized clusters. If you experience broadcast timeouts during data flow executions, you can switch off the broadcast optimization. However, this will result in slower performing data flows.

When working with data sources that can take longer to query, like large database queries, it is recommended to turn broadcast off for joins. Source with long query times can cause Spark timeouts when the cluster attempts to broadcast to compute nodes.

Another good choice for turning off broadcast is when you have a stream in your data flow that is aggregating values for use in a lookup transformation later. This pattern can confuse the Spark optimizer and cause timeouts.

For more details, kindly check out the below documentation:

Optimizing Joins, Exists, and Lookups via Broadcasting

Hope it helps. Kindly accept the answer if your found it helpful. Thankyou