How to fix Job failed due to reason: at Sink 'SinkUpsert': Cannot broadcast the table that is larger than 8GB: 24 GB

Question

How to fix Job failed due to reason: at Sink 'SinkUpsert': Cannot broadcast the table that is larger than 8GB: 24 GB

Dhruva B 50

Getting error as Job failed due to reason: at Sink 'SinkUpsert': Cannot broadcast the table that is larger than 8GB: 24 GB Actually I am not the exact code developer for this and that is why unable to understand the 8:24 logic kindly someone explain me or guide me towards some documentation to prevent this type of failures

PRADEEPCHEEKATLA 90,601 Reputation points

2024-04-17T07:48:17.0733333+00:00

@Dhruva B - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

0 additional answers

Your answer

PRADEEPCHEEKATLA 90,601 Reputation points

2024-04-17T07:48:17.0733333+00:00

@Dhruva B - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

PRADEEPCHEEKATLA 90,601

@Dhruva B - Thanks for the question and using MS Q&A platform.

The error message you are seeing indicates that the table you are trying to broadcast is larger than the maximum size allowed, which is 8GB. In your case, the table size is 24GB, which is why the job is failing.

To prevent this type of failure, you can try the following:

Split the table into smaller chunks: You can split the table into smaller chunks and then broadcast each chunk separately. This will ensure that the size of each broadcasted table is within the allowed limit.
Use a different sink: If splitting the table is not an option, you can try using a different sink that does not have the same size limitations.
Increase the maximum broadcast size: If you need to broadcast large tables frequently, you can contact Azure support to request an increase in the maximum broadcast size limit.

Wants to know more details on what could cause this error message, checkout this article Spark exception: Cannot broadcast the table larger than 8 GB: 10 GB which explains the root cause and how to resolve the issue.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Dhruva B 50 Reputation points

2024-04-05T12:30:50.16+00:00

Thanks for the guidence @PRADEEPCHEEKATLA I have one more question here, the data which are being picked and processed is less then 500 MB files, how can it show as 24GB/ cross the limit of 8GB.

Any idea where we can exactly pin point the size being used.
PRADEEPCHEEKATLA 90,601 Reputation points

2024-04-08T05:01:51.2566667+00:00

@Dhruva B - It's possible that the size of the data being processed is less than 500 MB, but the size of the table being broadcasted is larger than 8GB. This can happen if the table is being created by aggregating or joining multiple smaller tables.

To pinpoint the exact size being used, you can try the following:

Check the size of the source data: Check the size of the source data being used in the pipeline. This will give you an idea of the maximum size of the table that can be created.

Check the size of the intermediate tables: If you are aggregating or joining multiple tables to create the final table, check the size of the intermediate tables being created. This will help you identify which table is causing the size limit to be exceeded.

Use the Azure Data Factory Monitor: Use the Azure Data Factory Monitor to monitor the pipeline and identify the exact location where the error is occurring. The monitor provides detailed information about the pipeline execution, including the size of the data being processed at each stage.

I hope this helps! Let me know if you have any further questions.
Dhruva B 50 Reputation points

2024-04-08T05:09:58.1033333+00:00

@PRADEEPCHEEKATLA if I am not wrong we are able to see only the no. If rows being read in a particular transformation or stage, how can we convert it into the size? When I checked in the monitor I was only able to see the rows read/written. Thanks for the support. Is there any other Matrix/tool which could help us track down the size of source data? Thanks
PRADEEPCHEEKATLA 90,601 Reputation points

2024-04-10T03:42:49.4433333+00:00

@Dhruva B - You are correct that the Azure Data Factory Monitor only displays the number of rows being read/written in each transformation or stage, and not the size of the data. However, you can estimate the size of the data based on the number of rows and the size of each row.

For example, if you know the average size of each row in your data, you can multiply the number of rows by the average row size to estimate the total size of the data. You can also use tools like Excel or Python to calculate the size of the data based on the number of rows and the schema of the data.

Another option is to use Azure Storage Explorer to view the size of the source data stored in Azure Blob Storage or Azure Data Lake Storage. You can navigate to the container or folder where the data is stored and view the size of each file. This will give you an idea of the total size of the data being processed.

I hope this helps! Let me know if you have any further questions.

Share via

How to fix Job failed due to reason: at Sink 'SinkUpsert': Cannot broadcast the table that is larger than 8GB: 24 GB

0 additional answers

Your answer