Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,216 questions
Parallel Writing of Data to ADLS Delta Storage Causes "The specified path already exists.", 409, PUT" Error
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 7.000000000000001%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOA%3C/text%3E%3C/svg%3E)
Orowole, Ayebogbon-XT
0
Reputation points
I have a Spark (Spark 3.1.2, Scala 2.12) application that reads json records from a table. The records are distributed across cluster executors.
In the application, I have used foreach function to loop through the records in the table. For each record, different transformations take place. After the transformations, the resulted dataframe is being writing to an ADLS storage.
It's possible during the for each loop two executors can write data to the same location at the same time in the ADLS storage (Parallel PUT operation). The application works fine in DEV and PRE-PROD. However, once in a while, I get the following error from the Log4j output. Sometimes, the job succeeded but still output the error in the log4j. Sometimes, the job failed
Error
ERROR AbfsClient: HttpRequest: 409,err=PathAlreadyExists,appendpos=,cid=9f83f144-94b8-4108-8e41-4b753eab3575,rid=e46cbae2-101f-0014-6286-104eea000000,connMs=0,sendMs=0,recvMs=48,sent=0,recv=168,method=PUT,url=https://storageAccount.dfs.core.windows.net/container/parentFolder/childFolder/grandChildFolder/_delta_log/00000000000000007568.json?timeout=90
ERROR AzureBlobFileSystem:V3: FS_OP_RENAME SRC[abfss://container@storageAccount.dfs.core.windows.net/container/parentFolder/childFolder/grandChildFolder/_delta_log/__tmp_path_dir/.00000000000000007568.json.8b7fd480-6154-443d-8ae9-b1357cec4e7b.tmp] DST[abfss://container@storageAccount.dfs.core.windows.net/container/parentFolder/childFolder/grandChildFolder/_delta_log/00000000000000007568.json] Rename failed. AbfsRestOperationException: Operation failed: "The specified path already exists.", 409, PUT, https://container@storageAccount.dfs.core.windows.net/container/parentFolder/childFolder/grandChildFolder/_delta_log/00000000000000007568.json?timeout=90, PathAlreadyExists, "The specified path already exists. RequestId:e46cbae2-101f-0014-6286-104eea000000 Time:2023-11-06T07:57:24.7258168Z"
Operation failed: "The specified path already exists.", 409, PUT, https://container@storageAccount.dfs.core.windows.net/container/parentFolder/childFolder/grandChildFolder/_delta_log/00000000000000007568.json?timeout=90, PathAlreadyExists, "The specified path already exists. RequestId:e46cbae2-101f-0014-6286-104eea000000 Time:2023-11-06T07:57:24.7258168Z"
at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:261)
at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsClient.renamePath(AbfsClient.java:355)
at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.rename(AzureBlobFileSystemStore.java:766)
at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.renameWithInstrumentation(AzureBlobFileSystem.java:381
Similar question is asked in this link but no solution provided: https://learn.microsoft.com/en-ie/answers/questions/185752/streaming-upserts-constantly-report-mysterious-log
{count} votes
-
PRADEEPCHEEKATLA-MSFT 90,146 Reputation points Microsoft Employee2023-11-07T08:46:09.23+00:00 @Orowole, Ayebogbon-XT - Thanks for the question and using MS Q&A platform.
It seems like you are facing an issue while writing data to ADLS storage in parallel. The error message "The specified path already exists." indicates that two executors are trying to write data to the same location at the same time in the ADLS storage. This can happen when multiple executors are writing to the same file or folder in parallel.
To avoid this issue, you can try the following solutions:
Use a unique file name for each write operation: You can generate a unique file name for each write operation to avoid conflicts. You can use a timestamp or a UUID to generate a unique file name.
Use a partitioned output: You can partition the output data based on a column value to avoid conflicts. This will ensure that each partition is written to a separate file or folder.
Use a single writer: You can use a single writer to write data to the ADLS storage. This will ensure that only one executor is writing to the storage at a time.
Increase the retry count: You can increase the retry count for the write operation. This will allow the operation to retry in case of a conflict.
I hope this helps. Let me know if you have any further questions.
-
PRADEEPCHEEKATLA-MSFT 90,146 Reputation points Microsoft Employee
2023-11-15T07:00:17.68+00:00 @Orowole, Ayebogbon-XT - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
-
Orowole, Ayebogbon-XT 0 Reputation points
2023-11-17T09:34:11.47+00:00 Base on the structure of our application, we need the parallel writing. The only viable solution from the ones you have mentioned above is the last one, which is increase the retry count of the write operation.
I have tried looking for a way to increase spark write operation retry count but no luck so far.
Do you have any resources you can provide on that?
-
Orowole, Ayebogbon-XT 0 Reputation points
2023-11-19T05:27:49.3566667+00:00 Still looking for answer
-
Orowole, Ayebogbon-XT 0 Reputation points
2023-11-19T05:29:35.4566667+00:00 Could I have got the same error if I had used parquet instead of delta dataset type?
-
PRADEEPCHEEKATLA-MSFT 90,146 Reputation points Microsoft Employee
2023-11-20T08:49:42.0133333+00:00 @Orowole, Ayebogbon-XT - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR#
-
PRADEEPCHEEKATLA-MSFT 90,146 Reputation points Microsoft Employee
2023-11-27T09:27:45.6+00:00 @Orowole, Ayebogbon-XT - Did you get a chance to open a support ticket?
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 38%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
Lakhs 0 Reputation points2024-09-27T09:08:42.7633333+00:00 Is there a solution to this problem? Can anyone pls share?
Sign in to comment
Sign in to answer