Azrue Table - Batch delete or Multi Delete

ash 66

Is it still possible to perform a batch delete or multi delete operation in a single execution?

As per pypi library of Azure.data.table- release update 12.0.0b7 (2021-05-11) it stats the following

Removed the TableClient.create_batch method along with the TableBatchOperations object. The transactional batching is now supported via a simple Python list of tuples.

So wondering if batch or multi delete is still possible using python.

GeethaThatipatri-MSFT 29,017 Reputation points Microsoft Employee

2022-10-27T19:45:44.693+00:00

Hi, @ash Thanks for posting your question in the Microsoft Q&A forum.
Currently, I am checking with the internal team to provide you with more details.

Regards
Geetha
ash 66 Reputation points

2022-10-28T04:36:16.003+00:00

@GeethaThatipatri-MSFT , Thanks for checking this for me.

Some additional reasons why I am looking for batch delete:
Currently if I want to delete entries on Azure table, for 300k entries its taking way to longer to complete the delete operations. Wondering whether batch delete may reduce the time to perform this operation.
GeethaThatipatri-MSFT 29,017 Reputation points Microsoft Employee

2022-10-31T23:57:20.783+00:00

@ash Have you looked at the following git repo?

Regards
Geetha
ash 66 Reputation points

2022-11-01T01:51:27.143+00:00

Thanks @GeethaThatipatri-MSFT , this _batch_delete_entity only deletes a single entity using partition and row key as these key parameters accept single value only.

What I am looking for is to delete a range of records based on partition key or using query. Is this possible?

ash 66

I found an async version of delete and did some experiment and it has the same performance as normal delete operation.

following the code form this location, I did some changes as below

async def delete_entity(self,customer_list):  
        credential = AzureNamedKeyCredential(self.account_name, self.access_key)  
        table_client = TableClient(endpoint=self.endpoint, table_name=self.table_name, credential=credential)  
  
        # [START delete_entity]  
        async with table_client:  
            start_time = datetime.datetime.today()  
            for position , customer_number in enumerate(customer_list):  
                await table_client.delete_entity(row_key=customer_number, partition_key=str(0))  
                   if position %100 == 0 and position != 0:  
                       print("time taken to delete 100 records is {0}".format(datetime.datetime.today() - start_time ))  
                       start_time = datetime.datetime.today()  
                       print("Customer number at  position number {1} is  {0},".format(customer_number, position ))

For deleting 100 records it takes between 30-35 seconds using both normal and async operations. I do not see any performance difference with async version or am I doing anything wrong in this async version of the code?

My table has 30k records with unique partition key.

ash 66 Reputation points

2022-11-09T06:22:25.583+00:00

@GeethaThatipatri-MSFT Do you have suggestions on how the delete query performance can be improved?
GeethaThatipatri-MSFT 29,017 Reputation points Microsoft Employee

2022-11-10T01:40:12.757+00:00

@ash sorry for the delay in my response I am checking with the internal team and provide you with more details.

Regards
Geetha

Accepted answer

GeethaThatipatri-MSFT 29,017 Reputation points Microsoft Employee

2022-11-10T21:22:05.75+00:00

Hi, @ash batch delete is still supported - here's a sample:
The sample shows a batch with various upsert/update operations, but this could be all 'delete' operations and yes, you could more efficiently delete asynchronously by doing the operations concurrently rather than one by one
Something like this:
delete_tasks = [client.delete_entity(partition_key=str(0), row_key=c) for c in customer_list]
await asyncio.wait(delete_tasks)
there are various other tools in the asyncio module for running tasks concurrently:
https://docs.python.org/3/library/asyncio-task.html#waiting-primitives

I hope this information helps

Regards
Geetha
Please sign in to rate this answer.
GeethaThatipatri-MSFT 29,017 Reputation points Microsoft Employee

2022-11-11T23:07:08.657+00:00

Hi, @ash Did you have the chance to check the above reply, please let us know if you find the above reply useful. If yes, do click on the 'Mark as answer'in the above reply. This will help other community members facing similar queries to refer to this solution

Regards
Geetha

ash 66 Reputation points

2022-11-14T14:02:50.457+00:00

Hello @GeethaThatipatri-MSFT ,

Thanks for the response.
I will try it and get back to you in this week.

GeethaThatipatri-MSFT 29,017 Reputation points Microsoft Employee

2022-11-14T14:18:40.483+00:00

Thanks, @ash for your response, sure please let us know once you test from your end.

Regards
Geetha

ash 66 Reputation points

2022-11-18T07:45:31.123+00:00

Hello @GeethaThatipatri-MSFT ,

I tried and it worked. I am able to delete more than 500 entities under a minute on a single table locally.
This is good news as the code successfully deletes the entries.

GeethaThatipatri-MSFT 29,017 Reputation points Microsoft Employee

2022-11-18T14:56:10.68+00:00

@ash Good to know it worked for you
Thank you for visiting the Microsoft QA forums! Have a great day!

Regards
Geetha
Sign in to comment

Share via

Azrue Table - Batch delete or Multi Delete

0 additional answers