Databricks Cluster (GC issue)

Imran Mondal 241 Reputation points

I have the below cluster setting. Currently, I am getting data every 30 Milliseconds. So for 1 hr, I am getting around 1 GB of Data. I am running this job every 5 hrs but it is throwing me GC error, Please help me.


Also attached my current cluster setting. Any help would be great.


Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,214 questions
{count} votes