How to configure firewall rule for access from Databricks to Postgres flexible server

Question

We are migrating our databases from Postgres single server to Postgres flexible server ahead of end of support for single servers. We are using Public Access as opposed to Private access in Networking tab of flexible server. This is because we want to allow customers to deploy our application from a Windows PC rather than Virtual Machine provisioned in Azure. So public access allows us to access this Postgres server from Windows PC and from AppService. To allow this we add firewall rules to for Windows PC IP address and for all AppService outbound IP addresses. This works. However, we also want to access this flexible server from a Databricks cluster. We would prefer not to enable 'Allow public access from any Azure service within Azure to this server' checkbox to allow Databricks access, because it opens it up to access from any Azure service. We did enable this checkbox temporarily and our Databricks jobs were able to access the flexible server, but would prefer not to do it if there is a more secure way of accessing it from Databricks. So the question is: Can a firewall be configured to allow Databricks access to this Postgres flexible server? If yes then how do we configure it? I looked through many articles on configuring Public and Private Networking but none of them address this specific issue. We also would also like to know in case we decided to use 'Private Access (VNet Integration), how we would need to configure Postgres flexible server to allow access from Databricks cluster. We configured Databricks public and private subnets in addition to the public subnet and we use Vnet injection during Databricks cluster creation. On Postgres single server there was an option to specify VNet subnets to allow connectivity from Databricks, but on flexible server there is no such option so we don't know if this is possible. If we go this route, then we probably must use a VM configured inside Azure for application deployment and lose the option of deployment from a Windows PC. Am I right?

Answer 1

Hi Swavek Lorenc Thank you for posting query in Microsoft Q&A Platform. To allow Databricks access to your Postgres flexible server, you can create a firewall rule to allow traffic from the outbound IP addresses of your Databricks cluster. You can find the outbound IP addresses of your Databricks cluster in the Azure portal. To create a firewall rule, you can follow the steps mentioned in the document "Manage firewall rules - Azure portal - Azure Database for PostgreSQL - Flexible Server". If you decide to use 'Private Access (VNet Integration)', you would need to configure Postgres flexible server to allow access from Databricks cluster. You can do this by creating a virtual network (VNet) and a subnet for your Postgres flexible server and another subnet for your Databricks cluster. You can then configure VNet peering between the two subnets to allow communication between them. You can find more information on how to configure VNet peering in the document "Networking in Azure Database for PostgreSQL - Flexible Server". However, if you go this route, you would need to use a VM configured inside Azure for application deployment and lose the option of deployment from a Windows PC.