Bring data into azure datalake.

Question

Bring data into azure datalake.

Niren Adhikary 96

Hello - we would like to know how we can bring dynamics/dataverse data into azure DATALAKE. Synapse link from dataverse to datalake will not work as datalake will be private access only and Synapse link requires public endpoints. Please provide a high level and secure method to bring the data from dynamics to data lake.

Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-07-13T22:35:44.95+00:00

Hello Niren Adhikary,

You can use ADF or Synapse to bring dynamics/dataverse data into datalake.

You can use Dynamic Connector as the source and Datalake as the sink here.

If your Dynamics/Dataverse instance is not accessible over the public internet, you need to use a self-hosted integration runtime to connect to it securely(as you mentioned, Synapse Link may not be an option for you due to private access.)

SHIR can be installed on a virtual machine or on-premises environment that can be used to securely connect to your Dynamics/Dataverse instance over a private network.

https://learn.microsoft.com/en-us/azure/data-factory/connector-dynamics-crm-office-365?tabs=data-factory

I hope this helps. Please let me know if you have any further questions.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-07-14T17:29:31.2466667+00:00

Hello Niren Adhikary,

I am checking to see if you got a chance to look into the above response. Please let me know if you have any further questions.
Niren Adhikary 96 Reputation points

2023-07-17T07:50:23.13+00:00

@Bhargava Gunnam I believe the above reply is not valid Dynamics 365 on cloud is hosted separately right? So how to connect to Dynamics to bring the data into Datalake provided the connection on Azure is private and how can we connect to dynamics which is hosted outside? How do we ensure data governance(Microsoft Purview) , encryption and data security etc. Please could you provide a detail approach about the solution we need to consider.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-07-17T19:27:25.0333333+00:00

To connect Dynamics 365 over a private network, you can either use a VPN gateway or express route to establish the connection between your Vnet and Dynamic 365 instance. And then, install SHIR on a VM or on-prem that can connect to Dynamic 365 in a private network.

Purview data Governance: https://learn.microsoft.com/en-us/azure/purview/catalog-permissions#understand-how-to-use-the-microsoft-purview-governance-portals-roles-and-collections

data security and encryption: https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/security/fundamentals/data-encryption-best-practices.md#azure-data-security-and-encryption-best-practices
Niren Adhikary 96 Reputation points

2023-07-17T19:42:31.9066667+00:00

We have express route on Azure and on-prem. I am clear with your response. Could you explain clearly and the steps. I think you are also mixing up ADF and Synapse link. Can we use Synapse link to bring the data into azure datalake. The datalake will be private access only and Synapse link requires public endpoints. How can we connect using express route to dynamics? Dynamics is not sitting on-Prem or on Azure but hosted separately right? What steps we need to follow.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-07-19T19:46:36.3333333+00:00

As per the documentation,

You can select "Enabled from selected virtual networks and IP addresses" for linked storage account and workspace to grant access only from selected virtual networks and IP addresses or to use private endpoints, you must create a Synapse Link with managed identities.Use managed identities for Azure with your Azure data lake storage

Please note: you must enable public network access for Azure resources for both initial setup and delta sync.

https://learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-synapse