How to fix "LeaseIdMissing" or "LeaseNotPresent" Error in Synapse pipeline

Kiran 26

Hello,

In past couple of weeks we are facing issues with data lake loads. As a part of the process we delete historical data from the data lake and reload the data into the data lake through synapse pipelines. When trying to delete the historical data folder we run into following error. We have been running this process for nearly 2 years and never ran into this issue before. Nothing has changed on the process or pipeline's logic.

If the error is not visible from screenshot below is the YAML version of the error:

"errorCode": "BadRequest",
"message": "Operation on target DeleteFolder failed: Failed to execute delete activity with data source 'AzureBlobFS' and error 'ADLS Gen2 operation failed for: Operation returned an invalid status code 'PreconditionFailed'. Account: ''. FileSystem: 'data'. Path: 'landing/boss/tworkorderheader/HistoricalLoad/tworkorderheader-2012-3_00000.parquet'. ErrorCode: 'LeaseIdMissing'. Message: 'There is currently a lease on the resource and no lease ID was specified in the request.'. RequestId: 'c13f663a-a01f-008b-2228-09e493000000'.

Checked the data lake for the parquet file in the question and the "Lease State" is not acquired as shown in the second image.

I have tried limiting the concurrency on the delete activity to make sure that the same file is not being accessed by some other process but it didn't worked either.

Any help is much appreciated. Thanks!

User's image

QuantumCache 20,266 Reputation points

2023-10-30T20:53:29.4666667+00:00
Hello @Kiran

Looks like you have done some troubleshooting steps in this matter,

Are any other processes or applications that might be accessing the same file. If there are, can we stop those processes or applications and try deleting the file again?

Did you try to increase the retry count in the delete activity to allow for more attempts to delete the file or folder. This can help if the lease is released shortly after the initial attempt to delete the file or folder.

You can try deleting the file or folder manually using the Azure Storage Explorer or Azure Portal, is that successful?

We have been running this process for nearly 2 years and never ran into this issue before. Nothing has changed on the process or pipeline's logic.

So how are you now dealing with this situation? Did you try any manual steps (scripts) to remove the lease?

# Parameters for storage account resources $resourceGroupName = "RGName" $storageAccountName = "storageAccountName" $fileShareName = "FSName" # Get reference to storage account $storageAccount = Get-AzStorageAccount ` -ResourceGroupName $resourceGroupName ` -Name $storageAccountName # Remove share and share snapshot leases Get-AzStorageShare -Context $storageAccount.Context | ` Where-Object { $_.Name -eq $fileShareName } | ` ForEach-Object { try { $leaseClient = [Azure.Storage.Files.Shares.Specialized.ShareLeaseClient]::new($_.ShareClient) $leaseClient.Break() | Out-Null } catch { } }
Kiran 26 Reputation points

2023-10-31T15:28:07.0766667+00:00

@SatishBoddu-MSFT

Are any other processes or applications that might be accessing the same file. If there are, can we stop those processes or applications and try deleting the file again?

There aren't any other processes running during the datalake load. The delete activity resides inside the ForEach loop and processes multiple tables through iterations. When I run the process for one particular table it executes fine but when I run the process for multiple tables that is when we are running into the LeaseID missing error.

Did you try to increase the retry count in the delete activity to allow for more attempts to delete the file or folder. This can help if the lease is released shortly after the initial attempt to delete the file or folder.

Per your suggestion, I did increased the retry attempts to 3 and retry intervals to 30 seconds in the delete activity. Ran the process and still ran into the same error "ErrorCode: 'LeaseIdMissing'".

You can try deleting the file or folder manually using the Azure Storage Explorer or Azure Portal, is that successful?

I did tried manually deleting the file and folder using Azure Storage Explorer and I was successful in deleting both.

So how are you now dealing with this situation? Did you try any manual steps (scripts) to remove the lease?

As mentioned, this situation surfaced couple of weeks back, tried few things such as to tweak the concurrency of the delete activity, run the process for smaller batch, run the process in debug mode to monitor if there are any overlapping issues.

This particular error occurs randomly on different files, after the issue has occurred if we navigate to that particular file/folder there wouldn't be a Lease acquired on the file/folder for us to do anything about it.

We haven't tried any manual steps to remove the lease on a file/folder because the error is random, not always on one single file/folder and lease acquisition is not visible at the surface of the application, probably something in the background that is applying the lease and not getting released when needed.
QuantumCache 20,266 Reputation points

2023-10-31T16:50:49.01+00:00

Hello @Kiran,

Thanks for providing the more info on this issue.

Please send an email to us.

1 answer

Vijayan, Uvaneswaran 20 Reputation points

2024-04-24T04:10:25.4633333+00:00

Enabling all the networks and disabling the lease through cloud Shell xi earned the trick.

However if i use firewall networks due to the our hybrid regulation that every traffic needs to be monitored why i am unable to delete. Seems doesn't make sense. What shall we do to whitelist or enable for us to delete while under firewall networks.

While the environment is strapped, now when i run the cloud shell command to delete a snapshot to test the vault.

What can we do without enabling the storage account configuration " enabled from all networks"
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to fix "LeaseIdMissing" or "LeaseNotPresent" Error in Synapse pipeline

1 answer

Your answer