Azure Storage Account - Network Security when using Microsoft Dynamics 365 F/O Export to Data Lake

Question

Hi,

 

I am a consultant trying to help a client setup ingestion from Dynamics 365 Finance and Operations into Databricks using Fivetran. We have it working, but we're not happy with the security and wondering if there's anything we can do to improve it.

They have set up the "Export to Azure Data Lake" add-in, but the only way this is working is when the networking for the Storage Account is set to Public network access: Enabled from all networks. We could not find any way to use private endpoints, nor whitelisting a set of known IP ranges, and the best we could find was this blog post: https://simplyfies.com/2022/10/12/disable-public-access-on-d365-export-to-data-lake/ which requires you to put the storage account in a different region to force it to go through public internet just so the IP addresses could be known.

Also see https://learn.microsoft.com/en-gb/azure/storage/common/storage-network-security?tabs=azure-portal#grant-access-from-an-internet-ip-range which says:

You can't use IP network rules in the following cases:

• To restrict access to clients in same Azure region as the storage account. IP network rules have no effect on requests that originate from the same Azure region as the storage account.

Furthermore, from the Storage Account to Databricks through Fivetran has the same problem, Fivetran asks for a SAS URL, which allows you to restrict the IP range, but because they're all within the same region it has the same problem. Which makes the SAS URL even more dangerous doesn't it, if someone mishandles it? 

Of course one solution is to move the storage account to a different region, but is that really an improvement? Move the resources to different regions, so it goes through public internet just so you know the IP addresses to restrict by? Will that also increase egress/ingest costs?

Surely there should be a better way to restrict these things and keep the SA secure when going through same region Azure private network.

So far, the only way we have successfully gotten this end-to-end process to work is to open the storage account networking to full public internet even though both services are going through private network within the same region.

 

Any advice or suggestions on this are greatly appreciated!!

Thank you!

Answer 1

Hello @Terry Munro,

I'm glad that you were able to share your ideas and thank you for posting your suggested solution on this forum so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Below is the suggestion from OP: Terry Munro,

It's important to note that moving the storage account to a different region just to restrict access by IP address is not a recommended solution. This can increase egress/ingress costs and may not provide the level of security that you need. Instead, you should consider using one or more of the below solutions to secure your storage account!!!

Thank you for confirming that. As I've outlined in my initial post, Microsoft Dynamics 365 F/O does not appear to offer the option of using any* of these options.

Private Endpoints

This of-course would be our ideal, but neither service currently offers it, and that is outside of our control. Plus I assume there'd be no point in only one of the two services supporting it, because the Storage Account would still require being set to allow public network traffic for the other one.

Service Endpoints

They are already using this, but like I was trying to explain in the initial post, the problem does not lie in how the services are able to connect. The problem lies in the lack of any way to deny all other network access.

Virtual Network Service Endpoints

We do not have control/access to the Virtual Networks used by Microsoft Dynamics 365 F/O nor do we have it for Fivetran, these are third party applications/SaaS.

Use Firewalls and Virtual Networks

This is basically what the whole post has been about, we'd love to use this to limit access to only allowed IP ranges, however due to the following:

You can't use IP network rules in the following cases:

• To restrict access to clients in same Azure region as the storage account. IP network rules have no effect on requests that originate from the same Azure region as the storage account. Use Virtual network rules to allow same-region requests.;

source: https://learn.microsoft.com/en-gb/azure/storage/common/storage-network-security?tabs=azure-portal#grant-access-from-an-internet-ip-range

It's not possible, furthermore we cannot use Virtual Network Rules for these services as they are not in our control, as I explained earlier.

Use SAS Tokens

We are using SAS tokens for Fivetran at-least, but this doesn't address the problem of locking down the storage account itself does it? Are we able to disable public access to a storage account while using SAS Tokens? It's my understanding that this would not work, in every experiment I've tried this results in no connectivity.

This concern could be fixed in multiple ways, but none of those ways are available to the user as far as I can tell.

Microsoft Dynamics 365 F/O and Fivetran could allow you to set up private endpoints to your storage accounts. That would be ideal, but we can't control that. Also note that even if we can work with Fivetran to arrange such a setup it would be pointless unless Microsoft Dynamics 365 F/O could also do it, since the Storage Account still must be publicly addressable, for that.

Alternatively, it would be nice if the mentioned limitation earlier didn't prevent you from blocking public traffic, while allowing the private traffic from these sources.

I am still hoping there is a solution that I've missed, or I've misunderstood something and one of my tests has been flawed, but currently I have not been able to find a reasonable solution.

Regardless, thank you for reaching out and providing this information, I hope it helps someone.

Please click "Accept Answer". So that we can close this thread.

Answer 2

Hi @SatishBoddu-MSFT,

It's important to note that moving the storage account to a different region just to restrict access by IP address is not a recommended solution. This can increase egress/ingress costs and may not provide the level of security that you need. Instead, you should consider using one or more of the below solutions to secure your storage account!!!

Thank you for confirming that. As I've outlined in my initial post, Microsoft Dynamics 365 F/O does not appear to offer the option of using any* of these options.

Private Endpoints

This of-course would be our ideal, but neither service currently offers it, and that is outside of our control. Plus I assume there'd be no point in only one of the two services supporting it, because the Storage Account would still require being set to allow public network traffic for the other one.

Service Endpoints

They are already using this, but like I was trying to explain in the initial post, the problem does not lie in how the services are able to connect. The problem lies in the lack of any way to deny all other network access.

Virtual Network Service Endpoints

We do not have control/access to the Virtual Networks used by Microsoft Dynamics 365 F/O nor do we have it for Fivetran, these are third party applications/SaaS.

Use Firewalls and Virtual Networks

This is basically what the whole post has been about, we'd love to use this to limit access to only allowed IP ranges, however due to the following:

You can't use IP network rules in the following cases:

• To restrict access to clients in same Azure region as the storage account. IP network rules have no effect on requests that originate from the same Azure region as the storage account. Use Virtual network rules to allow same-region requests.;

source: https://learn.microsoft.com/en-gb/azure/storage/common/storage-network-security?tabs=azure-portal#grant-access-from-an-internet-ip-range

It's not possible, furthermore we cannot use Virtual Network Rules for these services as they are not in our control, as I explained earlier.

Use SAS Tokens

We are using SAS tokens for Fivetran at-least, but this doesn't address the problem of locking down the storage account itself does it? Are we able to disable public access to a storage account while using SAS Tokens? It's my understanding that this would not work, in every experiment I've tried this results in no connectivity.

This concern could be fixed in multiple ways, but none of those ways are available to the user as far as I can tell.

Microsoft Dynamics 365 F/O and Fivetran could allow you to set up private endpoints to your storage accounts. That would be ideal, but we can't control that. Also note that even if we can work with Fivetran to arrange such a setup it would be pointless unless Microsoft Dynamics 365 F/O could also do it, since the Storage Account still must be publicly addressable, for that.

Alternatively, it would be nice if the mentioned limitation earlier didn't prevent you from blocking public traffic, while allowing the private traffic from these sources.

I am still hoping there is a solution that I've missed, or I've misunderstood something and one of my tests has been flawed, but currently I have not been able to find a reasonable solution.

Regardless, thank you for reaching out and providing this information, I hope it helps someone.