Is it possible to whitelist/get IP range for a Spark Pool in a Synapse with managed VNET?

asciibscii 50

We have a Synapse Workspace that has managed VNET enabled and have set up all required private endpoints to our other Azure resources. In this Synapse we want to get data from API calls that requires the requester to get their IP address whitelisted with the provider. Currently we're making requests using Copy Activities with a self hosted integration runtime, where the IP address for the server hosting the SHIR is whitelisted by the API provider.

We would prefer to make API calls using a notebook in Synapse if possible but are running into the problem that we don't know what IP addresses to send to the API provider for whitelisting or if it is even possible to do so.

1 answer

Luis Arias 5,136 Reputation points

2023-10-23T21:01:04.4366667+00:00

Hi,

I assume that this APIs is outside of Microsoft Azure platform if that the case you need to withelist the public Ips of Azure managed services (In this case Synapse).

https://learn.microsoft.com/en-us/azure/synapse-analytics/security/gateway-ip-addresses

To update the ipaddresses automatically you can get the lastest list from :

https://www.microsoft.com/en-us/download/details.aspx?id=56519

On the other hand If these APIs are within Azure services ,you can use a network service like private endpoint or peering to make the communication internally.

Cheers,

Luis Arias

If the information helped address your question, please Accept the answer.
Please sign in to rate this answer.
QuantumCache 20,031 Reputation points

2023-10-31T05:40:32.7366667+00:00

Hello @asciibscii

Just checking if we are still connected on this discussion? Please let us know if you need to add more info so that we better assist you!

Check if the API provider allows whitelisting of IP ranges instead of individual IP addresses. If so, they can whitelist the IP range of the Synapse workspace's managed VNET.

If the API provider only allows individual IP addresses to be whitelisted, you may try to find the IP address of the Synapse workspace's managed VNET. You can do this by checking the private IP address of the Synapse workspace's virtual network interface card (NIC) in the Azure portal.

If you are unable to find the IP address of the Synapse workspace's managed VNET, you can try to create a self-hosted integration runtime (SHIR) in the Synapse workspace's managed VNET. This will allow you to use the same IP address that is currently whitelisted by the API provider.

If none of the above steps work, we suggest to contact the API provider's support team to see if there are any other options available for whitelisting IP addresses.

Iglesias Fernandez, Carlos 0 Reputation points

2023-11-03T07:48:35.17+00:00

Hello Luis,

Would you know the ranges for Synapse Spark notebooks?

I think this is what asciibscii is asking and I have the same question, as I'd also like to use notebooks for calling APIs.

Thank you!

Luis Arias 5,136 Reputation points

2023-11-03T09:17:23.1866667+00:00

Hi Carlos,

Yes, You have that two option, use this list of range space in your API withelist that I share above or as mention Satish configure workspace's managed VNET and try to find the IP address of the Synapse.

Regards, Luis

Luis Arias 5,136 Reputation points

2023-11-06T11:03:12.5833333+00:00

Hi @Iglesias Fernandez, Carlos just to check if everything is good, accept the answer.

René Burdack 0 Reputation points

2024-05-13T07:08:20.25+00:00

Hi guys,

can we push this topic again:

We have a similar question: A graphQL API with pagination via body cursor has to be connected via a pySpark notebook in the Synapse Workspace. The reason is obvious, Copy-Pipelines do not support paging via cursor in the Http-Body, thus linked-services private endpoints and integration runtime were ruled out as a solution.

At least from what we know so far.

We have now treated our workspace to a managed VNET and included this in the firewall rules of the graphQL API.

Unfortunately, this does not seem to be sufficient, if I interpret the original answer correctly, one must also unlock the public IPs of Azure managed Services in the graphQL API?

This is difficult to explain to our compliance department in our scenario, as both the graphQL API and our Synapse Workspace are services in the internal network with internal IP ranges.

According to our company guidelines, there is no possibility to release external IPs, especially not those that are then activated in almost every other company. This would be an invitation to attackers... What other option am I missing to arrive at a clean solution here?
Sign in to comment

Share via

Is it possible to whitelist/get IP range for a Spark Pool in a Synapse with managed VNET?

1 answer