Datalake table update slower

Question

Datalake table update slower

Abhishek Gaikwad 196

We have a datalake table which has huge billion of records in parquet format. When we run any SQL queries against this table the queries are slower.
We are facing an issue when update certain colums to null values for this table. However when we copy this table to a new parquet file and then run update against the new parquet file the updates are faster. Can you please confirm what could be the reason the updates are faster when we copy data to a new file and the old file upates takes long.

PRADEEPCHEEKATLA 91,866 Reputation points

2021-11-18T05:09:20.76+00:00
Hi @Abhishek Gaikwad ,

Thanks for the question and using MS Q&A platform.

Could you please help me "How you are running SQL queries against this tables"?

Are you using Azure Databricks or Azure Synapse Analytics or any other services?
Abhishek Gaikwad 196 Reputation points

2021-11-18T12:53:53.6+00:00

Using databricks to update the datalake table
PRADEEPCHEEKATLA 91,866 Reputation points

2021-11-22T04:24:39.617+00:00
Hi @Abhishek Gaikwad ,

Could you please share more details below details:

Databricks runtime version?

Query which you are running?

Time taken to update the data lake table screenshot preferable?
PRADEEPCHEEKATLA 91,866 Reputation points

2021-11-24T10:58:29.253+00:00

Hi @Abhishek Gaikwad ,

Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.