Job aborted due to stage failure: Task 0 in stage 1486.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1486.0

Question

Job aborted due to stage failure: Task 0 in stage 1486.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1486.0

Abhishek Dutta (abhishek.dutta3@tcs.com) 0

While doing the merge load from bronze layer to silver below error is coming

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1486.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1486.0 (TID 1665) (10.116.129.142 executor 0): org.apache.spark.SparkException: Failed to store executor broadcast spark_join_relation_469_-315473829 in BlockManager.

Please suggest what are the checks that need to be done for this error.

KranthiPakala-MSFT 46,612 Reputation points Microsoft Employee

2023-02-14T00:44:13.0166667+00:00

Hi Abhishek Dutta (******@tcs.com),

Just checking in to see if my previous suggestion was helpful to resolve your issue. And, if you have any further query do let us know.

Thank you
KranthiPakala-MSFT 46,612 Reputation points Microsoft Employee

2023-02-23T09:04:07.3333333+00:00

@Abhishek Dutta (abhishek.dutta3@tcs.com) We still have not heard back from you. Just wanted to check if the below information was helpful? If it answers your query, please do click Accept Answer and Yes for "was this answer helpful", as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thank you

1 answer

Your answer

KranthiPakala-MSFT 46,612 Reputation points Microsoft Employee

2023-02-14T00:44:13.0166667+00:00

Hi Abhishek Dutta (******@tcs.com),

Just checking in to see if my previous suggestion was helpful to resolve your issue. And, if you have any further query do let us know.

Thank you
KranthiPakala-MSFT 46,612 Reputation points Microsoft Employee

2023-02-23T09:04:07.3333333+00:00

@Abhishek Dutta (abhishek.dutta3@tcs.com) We still have not heard back from you. Just wanted to check if the below information was helpful? If it answers your query, please do click Accept Answer and Yes for "was this answer helpful", as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thank you

Answer 1

Hi Abhishek Dutta (******@tcs.com),

Welcome to Microsoft Q&A forum and thanks for posting your query.

That error message comes from Spark's TorrentBroadcast class. Fortunately, the source code has a handy comment that explains how it works. Based on how the broadcast works, I suspect that this error is caused by the driver running out of memory. To overcome this issue, please try increasing the size of the driver node and see if that resolves the error.

User's image

If increasing the driver node didn't help to resolve the issue, then I would recommend filing a support ticket for deeper investigation.

Hope this info helps. Let us know how it goes.

Thank you

Share via

Job aborted due to stage failure: Task 0 in stage 1486.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1486.0

1 answer

Your answer