How to send zip file to webhdfs path using webactivity.

Question

I have csv.gz format files in blob storage. I have to copy the files and load it to webhdfs path. Right now I am reading files(csv.gz) from blob which generate .csv format ouput and create set variable to hold the content of dataset and then passing the content in body of webhdfs path.
Could you please help on this.

Answer 1

Based on the references

https://learn.microsoft.com/en-us/azure/data-factory/quickstart-hello-world-copy-data-tool

https://learn.microsoft.com/en-us/azure/data-factory/connector-hdfs

Sure, let's summarize what we've found so far.

To copy gzipped CSV files from Azure Blob Storage to a webhdfs path, you can use the Azure Data Factory, which provides an interface for data transfer between different storage services.

• Prepare Source Data in Azure Blob Storage
• Create a Data Factory
• Use the Copy Data Tool to Copy Data
• Create a Linked Service to HDFS
• Configure the Linked Service Properties
• Specify Dataset Properties You can follow whole steps from the references