Website data ingest

Sourav 130 Reputation points
2024-01-22T12:10:39.9766667+00:00

Hi - we have a public website and we want to ingest the data in CSV files from this site into azure datalake via ADF. We want to ensure data security and governance while integration and strict about secure authentication practices which should not follow basic authentication type.The data in this public website can be accessible via API.I would like to know how we should be bring tha data and what adf connector to be used followed by the steps or what can be the best possible approach to ingest the data keeping in mind data security practices and governance. Encryption of data in transit and at rest. Thank you

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,540 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,253 questions
{count} votes

Accepted answer
  1. Bastea Bogdan 160 Reputation points Microsoft Employee
    2024-01-23T11:58:03.6566667+00:00

    Hi Sourav,

    If your WebSite is reachable using HTTPS (port 443), then data will be encrypted.

    According to this doc : https://learn.microsoft.com/en-us/azure/data-factory/data-movement-security-considerations#encryption-in-transit

    In hybrid scenario (non-Cloud -> Cloud) "All data transfers are via secure channel HTTPS and TLS over TCP to prevent man-in-the-middle attacks during communication with Azure services."

    Therefore, even if in ADF you configure a pipeline using HTTP connector, this connector can read from both HTTP and HTTPS : https://learn.microsoft.com/en-gb/azure/data-factory/connector-http?tabs=data-factory#supported-capabilities

    "You can use this HTTP connector to:

    • Retrieve data from an HTTP/S endpoint by using the HTTP GET or POST methods."

    Please let me know if this clarify your ask.

    Thank you!


0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.