How to fix "The specified path already exists" issue raised by Databricks Delta Live Table pipeline executions

Slim MISSAOUI 35 Reputation points
2024-08-28T09:26:38.4266667+00:00

Hello,

I have several DLT pipelines that are generating an exception "The specified path already exists". The exception indicates that there is an issue with the internal checkpoint file of the DLT streaming tables.

org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed: "The specified path already exists.", 409, PUT, https://xxx.dfs.core.windows.net/managed/__unitystorage/schemas/43a-ae09-4e97-a2eb-324fd2e84f2e/tables/ee9d67c-2043-4250-af53-c1951aad6/_dlt_metadata/checkpoints/configuration__ids/24?resource=directory&timeout=90&st=2024-08-28T07:27:49Z&sv=2020-02-10&ske=2024-08-28T09:27:49Z&sig=XXXXX&sktid=0652c929-6106-451-ba96-0ebb59a37670&se=2024-08-28T08:47:09Z&sdd=5&skoid=5e4b8638-3c04-45fXXXXXXXXXXXXXXXXX&spr=https&sks=b&skt=2024-08-28T07:27:49Z&sp=racwdm&skv=2021-08-06&sr=d, PathAlreadyExists, "The specified path already exists.

Moreover, the problem is random (it doesn't always occur at the same stage of the workflows) and affects all my Databricks environments (3 environments).

Could you please help me ?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,175 questions
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee
    2024-09-18T02:14:53.82+00:00

    @Slim MISSAOUI - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

    Ask: How to fix "The specified path already exists" issue raised by Databricks Delta Live Table pipeline executions

    Solution: The issue is resolved. The root cause is a build deployed by Microsoft in their Azure environment.

    The issue has been resolved following the deployment of a hotfix by Microsoft.

    If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

    If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    1 person found this answer helpful.
    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Slim MISSAOUI 35 Reputation points
    2024-09-17T18:02:01.5966667+00:00

    The root cause is a build deployed by Microsoft in their Azure environment.

    The issue has been resolved following the deployment of a hotfix by Microsoft.

    1 person found this answer helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.