In ADF using HDFS linked service my copy file activity throws the following error

Ács Dániel 25

Hi,

I have an issue using ADF with HDFS linked service.

I created a HDFS connection then a copy acitivity from HDFS to Azure Data Lake gen2. The source is a CSV file and the copy format is binary. When I run the pipeline I get the following error:

Operation on target Copy_cf8 failed: ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: 'FOLDER1/hdfs/TEST_DATA0.csv'.,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read a 'Hdfs' file. File path: 'PATH/TO_THE_FOLDER/TEST_DATA0.csv'. Response details: '{}'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.ArgumentNullException,Message=Value cannot be null.

Parameter name: uriString,Source=System,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read a 'Hdfs' file. File path: 'PATH/TO_THE_FOLDER/TEST_DATA0.csv'. Response details: '{}'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.ArgumentNullException,Message=Value cannot be null.

Parameter name: uriString,Source=System,'

Thanks in advance,

Daniel

phemanth 5,735 Reputation points Microsoft Vendor

2024-03-29T07:14:29.97+00:00
@Ács Dániel
Thanks for reaching out to Microsoft Q&A

The error message you’re seeing suggests that there might be an issue with the URI string of your HDFS file. Here are a few things you could check:

File Path: Ensure that the file path ‘PATH/TO_THE_FOLDER/TEST_DATA0.csv’ is correct and the file ‘TEST_DATA0.csv’ exists at that location in your HDFS.

URI String: The error message indicates that the value of ‘uriString’ cannot be null. Make sure that the URI string in your linked service or dataset is correctly defined and not null.

Linked Service Configuration: Verify the configuration of your HDFS linked service in ADF. The linked service should have the correct connection information needed for ADF to connect to your HDFS.

Integration Runtime: If your HDFS is located inside an on-premises network, an Azure virtual network, or Amazon Virtual Private Cloud, you need to configure a self-hosted integration runtime to connect to it. Make sure that the integration runtime can access all the [name node server]: [name node port] and [data node servers]: [data node port] of the Hadoop cluster.

Permissions: Check if ADF has the necessary permissions to read the file from your HDFS and write to your Azure Data Lake Storage Gen2.

Hope this helps. Do let us know if you any further queries.
phemanth 5,735 Reputation points Microsoft Vendor

2024-04-01T05:04:13.3633333+00:00

@Ács Dániel We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Ács Dániel 25 Reputation points

2024-04-02T07:35:55.2166667+00:00

Hello, Thanks for the advice the things that I tried and done since I experienced this issue:

-The linked service succesfully reached the HDFS folder because I can select manuali the file in ADF from its HDFS folder.

-I also successfully tested connection when created linked service and it was successfull.

-We use self hosted integration runtime to reach HDFS.

-We also discovered that the HDFS side permissions are not the problem, because we tried with the admin user and it did not helped.

What do you mean by this point: URI String: The error message indicates that the value of ‘uriString’ cannot be null. Make sure that the URI string in your linked service or dataset is correctly defined and not null.

Can you decribe it a bit more detailed?

Thank you for your help.

Daniel
phemanth 5,735 Reputation points Microsoft Vendor

2024-04-03T10:11:23.3533333+00:00
@Ács Dániel

URI String Explained:

A URI (Uniform Resource Identifier) acts like an address that specifies the location of a resource on a network, similar to a web address.

In ADF, when you configure a linked service to connect to an external data source like HDFS, you define a URI string within the linked service properties.

This URI string tells ADF exactly how to construct the complete path to access the HDFS file system and the specific file you want to copy.

Why the Error Points to URI String:

The error message "Value cannot be null for uriString" suggests that the URI string defined in your HDFS linked service might be missing or incorrect.

It could be entirely empty, contain typos, or have the wrong format for your HDFS setup.

How to Check and Fix the URI String:

Review Linked Service Configuration: Go to your ADF workspace and navigate to the linked service you created for HDFS. Look for the section where URI details are specified. This might be named differently depending on the specific HDFS connector you're using, but it could be labeled as "URI," "HDFS Cluster URI," or something similar.

Verify the URI String: Check if the URI string has any value defined. If it's empty, you'll need to enter the correct URI based on your HDFS configuration. Ensure there are no typos or syntax errors in the existing URI string.

Constructing the URI String: The exact format of the URI string depends on your HDFS implementation. However, it typically follows a pattern like this:
<scheme>://<namenode_hostname>:<port>/<path/to/your/file.csv>

Replace the placeholders with your specific details:

<scheme>: Typically "hdfs" for standard HDFS.

<namenode_hostname>: The hostname or IP address of your HDFS Namenode server.

<port>: The port number used by the Namenode (usually 9000 by default).

<path/to/your/file.csv>: The complete path to the file you want to copy, including the directory structure within HDFS.

Additional Tips:

Refer to the documentation for your specific HDFS connector in ADF to find the exact URI format requirements. You can find it here: https://learn.microsoft.com/en-us/purview/register-scan-hdfs

Consider simplifying the path in the URI string for testing purposes. Try using a file located in the root directory of your HDFS to isolate potential path-related issues.

Ács Dániel 25

I did not find the URI, when I configured the linked service there were no URI input as u refer to it. On the otherhand I can show you my linked service config:

{
    "name": "hdfs_connection",
    "type": "Microsoft.DataFactory/factories/linkedservices",
    "properties": {
        "annotations": [],
        "type": "Hdfs",
        "typeProperties": {
            "url": "https://name.ofthe.service.com:15000/webhdfs/v1/",
            "authenticationType": "Windows",
            "userName": "hdfs_user@service.name.com",
            "encryptedCredential": "ZjQwNmQ1ZjFjMDAiLCJWZXJzaW9uIjoiMi4wIiwiQ2xhc3NUeXDIOASNDf89wew9jfwe9jfwefjewfjejfwe9j"
        },
        "connectVia": {
            "referenceName": "shir",
            "type": "IntegrationRuntimeReference"
        }
    }
}

phemanth 5,735 Reputation points Microsoft Vendor

2024-04-04T06:13:10.9133333+00:00
@Ács Dániel It appears the HDFS connector you're using might not require an explicit URI definition within the linked service configuration. Here's what we can analyze based on your provided JSON:

Configuration Properties: The key element seems to be "url" under "typeProperties". This likely constructs the base URL for accessing your HDFS along with the path you provide during copy activity configuration. "authenticationType", "userName", and "encryptedCredential" define the authentication method (Windows in this case) and credentials for connecting to HDFS.

Possible Reasons for Missing URI Field: The connector might internally construct the URI based on the provided URL aEditnd path during data transfer. It could be a pre-built connector with a specific format that doesn't require a separate URI field.

Recommendations:

Test with a Simple Path: In your copy activity configuration, try specifying a simple path to a file located in the root directory of your HDFS. This can help isolate any potential path-related issues.

Refer to the documentation for your specific HDFS connector in ADF to find the exact URI format requirements. You can find it here: https://learn.microsoft.com/en-us/purview/register-scan-hdfs
Ács Dániel 25 Reputation points

2024-04-04T07:21:49.8266667+00:00

@phemanth

I've tried it with a simple path the problem still remains the same.

Daniel
Ács Dániel 25 Reputation points

2024-04-04T09:02:26.5633333+00:00

@phemanth

I've tried it with a simple path the problem still remains the same.

Daniel
phemanth 5,735 Reputation points Microsoft Vendor

2024-04-05T06:15:53.98+00:00
@Ács Dániel Here are some further steps you can take to troubleshoot the problem:

1. Verify Integration Runtime Access:

Double-check that your self-hosted integration runtime has permissions to access the HDFS Namenode and DataNode servers. This includes ensuring it can reach the specific ports used by these services (typically Namenode port 9000 and DataNode port 50070). You can consult your network security team or review firewall configurations if necessary.

2. Scrutinize the URL:

While the linked service configuration doesn't explicitly define a URI, the "url" property under "typeProperties" likely plays a role. Examine this URL closely for any errors: Typos: Ensure there are no typos in the hostname, port number, or path segments. Format: Verify the format matches your HDFS implementation. It should typically resemble "<scheme>://<namenode_hostname>:<port>/" (e.g., https://namenode.contoso.com:15000/webhdfs/v1/).

3. ADF Logs and Diagnostics:

Enable diagnostic logging for the ADF pipeline execution. This can provide more detailed information about the issue during the copy activity. Refer to the ADF documentation for enabling diagnostics: https://learn.microsoft.com/en-us/azure/data-factory/

Review the ADF logs for any specific errors or warnings related to the HDFS connection or file access.
Ács Dániel 25 Reputation points

2024-04-09T08:13:52.13+00:00

@phemanth

We have tried to open the suggested ports but, ended up with the same error.

I also double checked, the webhdfs format, and it's also the sugested fomat.

The problem is still persists.
phemanth 5,735 Reputation points Microsoft Vendor

2024-04-10T08:52:12.11+00:00

@Ács Dánielcould you please share Linked services, datasets and copy activiity configuration details to understand the issue better.

Ács Dániel 25

{
    "name": "hdfs_connection_service",
    "type": "Microsoft.DataFactory/factories/linkedservices",
    "properties": {
        "annotations": [],
        "type": "Hdfs",
        "typeProperties": {
            "url": "https://hdfsnode.domain.com:14000/webhdfs/v1/",
            "authenticationType": "Windows",
            "userName": "user@corp.domain.com",
            "encryptedCredential": "eyJDcmVkZW50aWFsSWQiOiJmYjJjODk3Yi03NDNlLTQ0NzAtOGFiNy1iYTg0ZTk1MzBmZmEiLCJWZXJzaW9uIjoiMi4wIiw"
        },
        "connectVia": {
            "referenceName": "shir",
            "type": "IntegrationRuntimeReference"
        }
    }
}

@phemanth Here is the requested data.

{
    "name": "teszt_problem",
    "properties": {
        "activities": [
            {
                "name": "Copy_cf8",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [
                    {
                        "name": "Source",
                        "value": "/
                    },
                    {
                        "name": "Destination",
                        "value": "/DEST/FOLDER/"
                    }
                ],
                "typeProperties": {
                    "source": {
                        "type": "BinarySource",
                        "storeSettings": {
                            "type": "AzureBlobFSReadSettings",
                            "recursive": true
                        },
                        "formatSettings": {
                            "type": "BinaryReadSettings"
                        }
                    },
                    "sink": {
                        "type": "BinarySink",
                        "storeSettings": {
                            "type": "AzureBlobFSWriteSettings"
                        }
                    },
                    "enableStaging": false,
                    "validateDataConsistency": false
                },
                "inputs": [
                    {
                        "referenceName": "Binary15",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "Binary16",
                        "type": "DatasetReference"
                    }
                ]
            }
        ],
        "annotations": [],
        "lastPublishTime": "2024-04-09T12:15:45Z"
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}

User's image

Accepted answer

phemanth 5,735 Reputation points Microsoft Vendor

2024-04-15T17:09:42.9266667+00:00
@Ács Dániel Welcome to Microsoft Q&A platform and thanks for posting your question.

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Ask: I have an issue using ADF with HDFS linked service.

I created a HDFS connection then a copy acitivity from HDFS to Azure Data Lake gen2. The source is a CSV file and the copy format is binary. When I run the pipeline I get the following error:

Operation on target Copy_cf8 failed: ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: 'FOLDER1/hdfs/TEST_DATA0.csv'.,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read a 'Hdfs' file. File path: 'PATH/TO_THE_FOLDER/TEST_DATA0.csv'. Response details: '{}'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.ArgumentNullException,Message=Value cannot be null.

Parameter name: uriString,Source=System,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read a 'Hdfs' file. File path: 'PATH/TO_THE_FOLDER/TEST_DATA0.csv'. Response details: '{}'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.ArgumentNullException,Message=Value cannot be null.

Parameter name: uriString,Source=System,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read a 'Hdfs' file. File path: 'PATH/TO_THE_FOLDER/TEST_DATA0.csv'. Response details: '{}'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.ArgumentNullException,Message=Value cannot be null.

Parameter name: uriString,Source=System,'

Solution: Hello, I just wanted to say thank you for your effort, we have successfully solved the problem with the following fix:

{ "name": "hdfs_connection_service", "type": "Microsoft.DataFactory/factories/linkedservices", "properties": { "annotations": [], "type": "Hdfs", "typeProperties": { "url": "https://hdfsnode.domain.com:14000/webhdfs/v1/", "authenticationType": "Windows", "userName": "user@corp.domain.com", "encryptedCredential": "eyJDcmVkZW50aWFsSWQiOiJmYjJjODk3Yi03NDNlLTQ0NzAtOGFiNy1iYTg0ZTk1MzBmZmEiLCJWZXJzaW9uIjoiMi4wIiw" "avoidRedirect": "true" #<-This line here fixed our problem }, "connectVia": { "referenceName": "shir", "type": "IntegrationRuntimeReference" } } }

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

1 additional answer

Ács Dániel 25 Reputation points

2024-04-15T13:32:07.8033333+00:00
@phemanth

Hello, I just wanted to say thank you for your effort, we have successfully solved the problem with the following fix:

{

"name": "hdfs_connection_service", "type": "Microsoft.DataFactory/factories/linkedservices", "properties": { "annotations": [], "type": "Hdfs", "typeProperties": { "url": "https://hdfsnode.domain.com:14000/webhdfs/v1/", "authenticationType": "Windows", "userName": "user@corp.domain.com", "encryptedCredential": "eyJDcmVkZW50aWFsSWQiOiJmYjJjODk3Yi03NDNlLTQ0NzAtOGFiNy1iYTg0ZTk1MzBmZmEiLCJWZXJzaW9uIjoiMi4wIiw" "avoidRedirect": "true" #<-This line here fixed our problem }, "connectVia": { "referenceName": "shir", "type": "IntegrationRuntimeReference" } }

}

Best Regards,

Daniel
Please sign in to rate this answer.
phemanth 5,735 Reputation points Microsoft Vendor

2024-04-15T17:11:14.8833333+00:00

@Ács Dániel Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.
Sign in to comment