Data flow run fails with InvalidAbfsRestOperationExceptionjava.net.SocketTimeout Exception: connect timed out\n\tat org.apache.hadoop.fs.azurebfs.services

Question

I am running dataflow activity in a pipeline that copies data from blob storage to adls, its throwing error like :

{"message":"Job failed due to reason: at Source 'SrcLocationConcerto': Status code: -1 error code: null error message: InvalidAbfsRestOperationException. Details:Status code: -1 error code: null error message: InvalidAbfsRestOperationExceptionjava.net.SocketTimeoutException: connect timed out\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:285)\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:194)\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:783)\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:765)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:294)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:785)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:517)\n\tat org.apache.hadoop.fs.Globber2.getFileStatus(Globber2.java:43)\n\tat org.apache.hadoop.fs.Globber2.doGlob(Globber2.java:261)\n\tat org.apache.hadoop.fs.Globber2.glob(Globber2.java:129)\n\tat","failureType":"UserError","target":"DF_LocationData_LS","errorCode":"DFExecutorUserError"}

I have checked - the linked service , integration private endpoint are fine and no issue with them.

When i do the same task with copy activity instead of dataflow it is able to read and write data in blob as well as adls.

This issue is in UAT, i have same pipelines in dev, it has same configuration and its running in dev,its failing with this error in uat.

Please assist resolve this ,thanks in adv.

Answer 1

@Wasim

Welcome to Microsoft Q&A platform and thanks for posting your question.

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Ask: I am running dataflow activity in a pipeline that copies data from blob storage to adls, its throwing error like :{"message":"Job failed due to reason: at Source 'SrcLocationConcerto': Status code: -1 error code: null error message: InvalidAbfsRestOperationException. Details:Status code: -1 error code: null error message: InvalidAbfsRestOperationExceptionjava.net.SocketTimeoutException: connect timed out\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:285)\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:194)\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:783)\n\tat org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:765)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:294)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:785)\n\tat org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:517)\n\tat org.apache.hadoop.fs.Globber2.getFileStatus(Globber2.java:43)\n\tat org.apache.hadoop.fs.Globber2.doGlob(Globber2.java:261)\n\tat org.apache.hadoop.fs.Globber2.glob(Globber2.java:129)\n\tat","failureType":"UserError","target":"DF_LocationData_LS","errorCode":"DFExecutorUserError"}

Solution: the issue got resolved changing the linked service authentication type from Managed Identity to Account key.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.