In past couple of weeks we are facing issues with data lake loads. As a part of the process we delete historical data from the data lake and reload the data into the data lake through synapse pipelines. When trying to delete the historical data folder we run into following error. We have been running this process for nearly 2 years and never ran into this issue before. Nothing has changed on the process or pipeline's logic.
If the error is not visible from screenshot below is the YAML version of the error:
"message": "Operation on target DeleteFolder failed: Failed to execute delete activity with data source 'AzureBlobFS' and error 'ADLS Gen2 operation failed for: Operation returned an invalid status code 'PreconditionFailed'. Account: ''. FileSystem: 'data'. Path: 'landing/boss/tworkorderheader/HistoricalLoad/tworkorderheader-2012-3_00000.parquet'. ErrorCode: 'LeaseIdMissing'. Message: 'There is currently a lease on the resource and no lease ID was specified in the request.'. RequestId: 'c13f663a-a01f-008b-2228-09e493000000'.
Checked the data lake for the parquet file in the question and the "Lease State" is not acquired as shown in the second image.
I have tried limiting the concurrency on the delete activity to make sure that the same file is not being accessed by some other process but it didn't worked either.
Any help is much appreciated. Thanks!