Databricks Cluster (GC issue)

Imran Mondal 241 Reputation points
2021-04-14T09:25:38.62+00:00

I have the below cluster setting. Currently, I am getting data every 30 Milliseconds. So for 1 hr, I am getting around 1 GB of Data. I am running this job every 5 hrs but it is throwing me GC error, Please help me.

87587-cluster.png

Also attached my current cluster setting. Any help would be great.

87701-ezgifcom-gif-maker-8.gif

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,214 questions
{count} votes