Data Ingestion

Question

Hi,

We need to ingest on-prem oracle data into our azure storage for data analytics purpose.

Source > On-prem Oracle

Destination> Azure Storage

We have express route connect with on-prem. My understanding we can use ADF SHIR and create the pipeline to ingest the raw data into azure.

How do we enable security and encryption like TLS and any other data security methods to ensure the data is secure as we have sensitive information.

Could you please provide a clear step explaining the high level solution concepts we can implement in this case.

Thank you

Sourav

Answer 1

@Sourav

Yes, you can use Azure Data Factory (ADF) to ingest data from an on-premises Oracle database into Azure Storage.

When you move data between on-premises and the cloud, the activity uses a self-hosted integration runtime to transfer the data between an on-premises data source and the cloud.

Here is a high-level summary of the data-flow steps for copying with a self-hosted IR:

To ensure the security of your data, you can use the following methods:

Use Azure Data Factory Self-Hosted Integration Runtime (SHIR) to securely connect to your on-premises Oracle database. The SHIR is a lightweight agent that you install on a local machine or a virtual machine in your on-premises environment. It provides a secure communication channel between your on-premises environment and Azure. You can configure the SHIR to use Transport Layer Security (TLS) to encrypt the data in transit.

Use Azure Storage Service Encryption (SSE) to encrypt your data at rest. SSE automatically encrypts your data before persisting it to Azure Storage. SSE uses 256-bit Advanced Encryption Standard (AES) encryption to protect your data.

Use Azure Key Vault to manage your encryption keys. Azure Key Vault is a cloud service that provides secure storage of keys, secrets, and certificates. You can use Azure Key Vault to store and manage the encryption keys for your data.

Here are the high-level steps to implement this solution:

Create an Azure Data Factory instance in your Azure subscription.

Create a Self-Hosted Integration Runtime and install it on a local machine or a virtual machine in your on-premises environment.

Create a linked service for your on-premises Oracle database in Azure Data Factory.

Create a linked service for your Azure Storage account in Azure Data Factory.

Create a pipeline in Azure Data Factory to copy data from your on-premises Oracle database to Azure Storage.

Configure the pipeline to use the Self-Hosted Integration Runtime and the linked services for your Oracle database and Azure Storage.

Enable Transport Layer Security (TLS) on the Self-Hosted Integration Runtime to encrypt the data in transit.

Enable Azure Storage Service Encryption (SSE) to encrypt your data at rest.

Use Azure Key Vault to manage your encryption keys.

For more details, refer to the below articles:

For more details, refer to the below articles:

Integration runtime in Azure Data Factory Azure security baseline for Data Factory Security considerations for data movement in Azure Data Factory

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.