Append in Liquid Cluster enabled table is not completing on DBR 15.3 version

Question

I am trying do analysis with a Partition Table and Liquid Clustered table. As per Azure Databricks recommendation, I am using DBR 15.2 to execute the code.

I have created a clustered table as and using an append operation which is specified below.

Few Stats: Size of df is around 950 GB with 9Bn row.

When I run this piece of code in DBR 14.3, it executes in around 12 minutes. But when I am using DBR 15.2 or DBR 15.3, it is not completing. I have left the process to run for around 2 hrs also.

Can you please advice why this is not running in the latest DBR runtime.

CREATE TABLE ext_IOT_event_clust ( 
	id BIGINT, 
	device_id STRING, 
	country STRING, 
	manufacturer STRING, 
	model_line STRING, 
	event_type STRING, 
	event_ts TIMESTAMP) 
USING delta CLUSTER BY (country) 
LOCATION 'abfss://adls-location' 


# Writing to the cluster table

df_raw_event.write.format("delta").option("path","abfss://adls-location2") \
    .mode("append") \
    .saveAsTable("ext_IOT_event_clust")

Answer 1

Your issue could be due to several factors.

First thing I think about is newer runtime versions might introduce changes or optimizations. So ypu may need to review and potentially optimize your cluster configuration to ensure it has sufficient resources for the 950 GB, 9 billion row dataset.

Check for any specific performance tuning parameters for Delta Lake or Spark SQL that might help. If you didn't enable any detailed logging and monitoring, I think you should at least to capture more information about the write operation and identify performance bottlenecks.

Review your dataframe transformations and actions for efficiency, and verify that your clustering strategy is appropriate for your query patterns.

Steps to you can follow to diagnose :

• Check the Databricks cluster logs for any error messages or warnings related to the write operation. Look for any exceptions or stack traces that might indicate the cause of the problem.
• Use the Spark SQL EXPLAIN command to generate the execution plan for your write operation. This can help you understand how Spark is processing the write and identify any potential bottlenecks.
• Use Databricks' built-in performance profiling tools to analyze the execution of your job. This can provide insights into where the time is being spent during the write operation.