![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 98%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAA%3C/text%3E%3C/svg%3E)
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,514 questions
@Agarwal, Abhishek - Thanks for the question and using MS Q&A platform.
This error message indicates that the job failed due to a stage failure caused by a lost task. The most recent failure was due to an ExecutorLostFailure, which means that executor 36 exited because of one of the running tasks. The reason for this failure is likely due to containers exceeding thresholds or network issues**.**
To solve this problem, you can use the following options:
Option-1: Use a powerful cluster (both drive and executor nodes have enough memory to handle big data) to run data flow pipelines with setting "Compute type" to "Memory optimized".
Option-2: Use larger cluster size (for example, 48 cores) to run your job.
Option-3: Check the driver logs for WARN messages to identify any potential issues with the driver.
If none of these options work, please provide more information about your specific scenario so that I can assist you better.
Here are some suggestions that could help you reduce your execution time:
Increase the number of worker nodes: Adding more worker nodes to your cluster can help you process data in parallel and reduce the execution time of your Spark jobs. You can choose the worker type based on the amount of memory and CPU required for your workload.
Increase the number of cores per worker node: Increasing the number of cores per worker node can help you process more data in parallel and reduce the execution time of your Spark jobs. However, keep in mind that increasing the number of cores per worker node can also increase the cost of your cluster.
Increase the amount of memory per worker node: Increasing the amount of memory per worker node can help you process larger datasets and reduce the number of shuffles required during the execution of your Spark jobs. This can also help you reduce the execution time of your Spark jobs.
Use a larger driver node: If your PySpark code requires a lot of memory, you may want to consider using a larger driver node. This can help you avoid out-of-memory errors and improve the performance of your Spark jobs.
Use autoscaling: Databricks provides an autoscaling feature that can automatically add or remove worker nodes based on the workload. This can help you optimize the cost of your cluster while ensuring that you have enough resources to process your data.
Regarding the worker types and driver types, the main difference between them is the amount of memory and CPU available. The more memory and CPU available, the more data you can process in parallel, which can help you reduce the execution time of your Spark jobs. You can choose the worker and driver types based on the requirements of your workload.
For more details, refer to Best practices: Cluster configuration.
Hope this helps. Do let us know if you any further queries.
If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.