How can i copy data/files from ADLS to HDFS? I have few approaches like: Using ADF activity Using DistCp Using Python Libraries Please provide the detail explanation if any one of these will work?

Hi @Kajol Patira , Welcome to Microsoft Q&A forum and thanks for reaching out. You cannot achieve this directly using Copy activity in ADF as HDFS is not supported as sink. Current HDS if only supported as Source. One way to achieve this is by writing your own custom code and using it in Azure Function activity or Customer activity in ADF Or using DistCP tool you may try as explained in this document - https://learn.microsoft.com/azure/storage/blobs/data-lake-storage-use-distcp . Hope this info helps. ---------- Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

How to copy data/files from ADLS to HDFS?

Kajol Patira 1

How can i copy data/files from ADLS to HDFS?
I have few approaches like:

Using ADF activity
Using DistCp
Using Python Libraries
Please provide the detail explanation if any one of these will work?

Praveen Nair 1 Reputation point

2021-05-21T17:07:06.13+00:00
Create an Azure Data Factory pipeline (may be copy-data template) with Azure Data Lake store as the source connection, and HDFS as the destination.

Ref:

https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-store

Sorry, I do not have a detailed explanation at this point.

1 answer

KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2021-05-21T20:16:22.117+00:00

Hi @Kajol Patira ,

Welcome to Microsoft Q&A forum and thanks for reaching out.

You cannot achieve this directly using Copy activity in ADF as HDFS is not supported as sink. Current HDS if only supported as Source. One way to achieve this is by writing your own custom code and using it in Azure Function activity or Customer activity in ADF

Or using DistCP tool you may try as explained in this document - https://learn.microsoft.com/azure/storage/blobs/data-lake-storage-use-distcp.

Hope this info helps.

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2021-05-25T00:04:52.133+00:00

Hi @Kajol Patira ,

Just checking in to see if the above suggestion was helpful. If this answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2021-05-26T06:20:31.477+00:00

Hi @Kajol Patira ,

We still have not heard back from you. Just wanted to check if the above suggestion was helpful? If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread.

And, if you have any further query do let us know.

Abhishek Singh 0 Reputation points

2023-09-11T07:11:33.92+00:00

Hi I have the similar requirement to copy data from ADLS to HDFS. I see this answer was 2 years back, So is it possible now to copy activity in ADF using HDFS as sink or still not possible to use HDFS as sink?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to copy data/files from ADLS to HDFS?

1 answer

Your answer