Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,562 questions
How to bring data from a website and ingest into data lake storage and compare the data with other datasets
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 35%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sourav
130
Reputation points
Hello,
We have a requirement to bring data from a publicly available source website, the data can be accessed programmatically over API, we need to bring these datasets and compare it with other data within Azure. We need to ensure data protection and encryption.
I could think of couple of options-
- We manually keep the file in a shared folder on-prem location and ADF pipeline to pick these files, write some python scripts within notebook to compare the data.
- We manually upload these files in share point site and and ADF pipeline to pick these files, write some python scripts within notebook to compare the data.
3.Use ADF pipeline Web or API connector for linked service and bring the data into data lake ,write some python scripts within notebook to compare the data.
These are assumptions hence I am looking for an expert advise and more detail level solution approach, keeping in mind we need to protect these data while in-transit and while we store the data in azure. Some sort of encryption techniques and capability to ensure data is protected through out the process.
Thanks in advance.
Regards,
Sourav
Azure Data Lake Storage
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,646 questions
Microsoft 365 and Office | SharePoint | For business | Windows
11,705 questions
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu 24,260 Reputation points Microsoft External Staff Moderator2023-10-30T08:52:43.4166667+00:00 Hi @Sourav Thank you for reaching out to us with your query.
I have reviewed your options and I recommend using the ADF pipeline Web or API connector to bring the data into data lake. This is the most scalable and secure option.
To compare the data with other datasets, you can write some Python scripts within a notebook. You can use Azure Databricks or Azure Synapse Analytics to create a notebook and write the Python scripts.
Here is a summary of the solution approach to protect data in transit and at rest.
- Protect data in transit by using HTTPS or SSL/TLS encryption and Azure ExpressRoute.
- Protect data at rest by encrypting the data in Azure Data Lake using Azure Storage Service Encryption (SSE) with customer-managed keys stored in Azure Key Vault.
- Use an ADF pipeline with the following activities:
- A Web Activity to connect to the publicly available source website and download the data.
- A Copy Activity to copy the data from the Web Activity to Azure Data Lake with SSE-KMS encryption.
- A Data Flow Activity to compare the data in Azure Data Lake with the other data in Azure.
- A Copy Activity to copy the data from the Web Activity to Azure Data Lake with SSE-KMS encryption.
- A Web Activity to connect to the publicly available source website and download the data.
References:
- https://azure.microsoft.com/en-us/explore/trusted-cloud/privacy
- https://azure.microsoft.com/en-us/blog/transparent-data-encryption-or-always-encrypted/
- https://learn.microsoft.com/en-us/azure/security/fundamentals/data-encryption-best-practices
I hope this helps! Please let me know if you have any other questions.
-
Sourav 130 Reputation points
2023-10-30T14:56:52.8833333+00:00 Hello -
I am not clear with the answer. How do you back the recommendation option web API connector that it is more secure ?
Please provide more detail solution approach on how to ingest the data from the Public web site.
How to encrypt the data at rest and in transit - In Azure Datalake how do we encrypt data ? We cannot use express route for a public website and express route is not a cost effective solution.
Regards,
Sourav
-
Smaran Thoomu 24,260 Reputation points Microsoft External Staff Moderator
2023-10-31T12:38:22.6733333+00:00 @Sourav
I apologize for the confusion in my previous response. Let me provide a more detailed solution approach to your query.Ingest data from a publicly available source website using ADF pipeline with a Web or API connector:
Create an ADF pipeline with the following activities:
- Web Activity: This activity connects to the publicly available source website and downloads the data.
- Copy Activity: This activity copies the data from the Web Activity to Azure Data Lake Storage (ADLS) Gen2 with SSE-KMS encryption.
To configure the Web Activity to connect to the publicly available source website, you will need to provide the following information:
- The URL of the website
- The HTTP method to use (GET, POST, PUT, etc.)
- The request headers (if needed)
- The request body (if needed)
To configure the Copy Activity to copy the data to ADLS Gen2 with SSE-KMS encryption, you will need to provide the following information:
- The source dataset (the Web Activity output)
- The destination dataset (the ADLS Gen2 container and file path)
- The Azure Key Vault linked service and customer-managed key to use for encryption
Once you have configured the ADF pipeline, you can deploy and run it to start ingesting the data from the publicly available source website to ADLS Gen2.
Encrypt the data at rest and in transit:
- Encrypt the data at rest in ADLS Gen2:
- ADLS Gen2 provides encryption at rest by default using Microsoft-managed keys. You can also bring your own keys (BYOK) for encryption at rest. To enable BYOK, you will need to create a key vault in Azure Key Vault and create a customer-managed key. Then, you will need to configure the ADLS Gen2 account to use the customer-managed key for encryption at rest.
- Encrypt the data in transit between the publicly available source website and ADF:
- To encrypt the data in transit, you will need to configure the Web Activity to use HTTPS. You can also use Azure ExpressRoute to create a private connection between your on-premises network and Azure. This will encrypt the data in transit between your on-premises network and ADF.
Note: ExpressRoute is not required to encrypt the data in transit. You can use a VPN to create a secure connection between your on-premises network and Azure. However, ExpressRoute provides a more reliable and secure connection.
I hope this information is helpful. Please let me know if you have any other questions.
-
Sourav 130 Reputation points
2023-10-31T18:32:09.8733333+00:00 @Smaran Thoomu Thanks for the above comment. How can we say web activity method is secure ? Is there a way to enable encryption in transit while using this method ?
-
Smaran Thoomu 24,260 Reputation points Microsoft External Staff Moderator
2023-11-01T12:41:55.6633333+00:00 Hi Sourav,
Yes, there is a way to enable encryption in transit while using the Web Activity in Azure Data Factory (ADF). You can do this by using HTTPS.To configure the Web Activity to use HTTPS, you will need to provide the following information:
- The URL of the website
- The HTTP method to use (GET, POST, PUT, etc.)
- The request headers (if needed)
- The request body (if needed)
- The HTTPS certificate of the website (if needed)
If the website does not use HTTPS, you can use a proxy server to encrypt the traffic.
Here are some of the benefits of using the Web Activity method to ingest data from a publicly available source website:
- It is a scalable and secure method.
- It is easy to use and configure.
- It is relatively inexpensive.
Drawbacks of using the Web Activity method:
- It is not as flexible as some other methods, such as using a custom connector.
- It may not be able to ingest data from all websites.
Overall, the Web Activity method is a good option for ingesting data from a publicly available source website, especially if you need a secure and scalable solution.
I hope this information is helpful. Please let me know if you have any other questions.
Sign in to comment
Sign in to answer