datafactory broadcast error

Question

hello,

we have several environments. dev, test, ..., prod

In ADF, in environment preprod, (but not in any other environment), one of the pipelines fails as follows:

when I run the dataflow in debug, the sink delta succeeds.

when I run the pipeline which runs the dataflow then it fails with the following error message as shown below.

please note, the Broadcast setting is left as default which is set to Auto for each of the joins used.

please note that I then changed the broadcast to fixed and still get the same error

ERROR message:

Details

Job failed due to reason: at Sink 'sinkPresentation': Broadcast join timeout error, you can choose 'Off' of broadcast option in join/exists/lookup transformation to avoid this issue. If you intend to broadcast join option to improve performance then make sure broadcast stream can produce data within 60 secs in debug runs and 300 secs in job runs.

Source

Pipeline

pl_xyzPresentation_delta_xxx

Data flow

df_xyzPresentation_delta_xxx

Answer 1

Hello @arkiboys

Request you to check this thread of your earlier question in the same lines and let us know if you have any further question on this.

https://learn.microsoft.com/en-us/answers/questions/1274783/data-flow-broadcast-join-timeout-error

Thanks