Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,562 questions
SFTP on ADLS
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 90%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
Kman
41
Reputation points
We have a third-party company that would like to provide us with their CSV files. They would like us to retrieve the files within their SFTP from their ADLS.
I would like to know what is the best way to get the CSV files from a third party into our ADLS.
Both platforms have no network connection.
I would like to use ADF to retrieve the CSV files from their Azure Platform into our Azure Platform but what else do I require? Would I need a Self Hosted Integration Runtime?
Azure Data Lake Storage
Azure Storage
Azure Storage
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,542 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,202 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator2023-06-30T20:20:54.3166667+00:00 To retrieve CSV files from a third-party SFTP server and load them into your Azure Data Lake Storage, you can use Azure Data Factory with a Self-Hosted Integration Runtime.
Here you need to install SHIR on the third party SFTP server and then create SFTP linked service that points to the SFTP server.
Create ADLS linked service.
using the copy activity, Configure the source dataset to point to the SFTP dataset you created earlier, and configure the sink dataset to point to your ADLS.
I hope this helps. Please let me know if you have any further questions.
-
Kman 41 Reputation points
2023-07-02T10:11:30.7233333+00:00 The SFTP is in the Azure Blob Storage, is the process the same as you outlined? Because this isn't a third-party SFTP.
https://learn.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-support
-
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator
2023-07-04T00:33:46.9766667+00:00 Thanks for the additional details.
Here, you do not need a Self-Hosted Integration Runtime in this scenario. Since both the SFTP server is in Azure Blob Storage and your ADLS is in Azure, you can use Azure Data Factory to copy the CSV files from the SFTP server to your ADLS without the need for a SHIR.
- In your Azure portal, create a new pipeline in ADF. Add a Copy Data activity to the pipeline.
- In the Copy Data activity, configure the source dataset to point to the SFTP server where the CSV files are exported. Use the SFTP connector to access the SFTP server.
- Configure the sink dataset to point to your ADLS. This is where the CSV files will be copied to.
- Run the pipeline to move the CSV files from the SFTP server to your ADLS.
I hope this helps. Please let me know if you have any further questions.
-
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator
2023-07-11T21:15:07.1966667+00:00 I am checking to see if you got a chance to look into my above response. Please let me know if you have any further questions.
Sign in to comment
Sign in to answer