Notebook IP address blocked by workspace IP ACL

Repetti Pierangelo 20

Hello,

I have a databricks workspace for which I have defined an IP address ACL so that it can be reached by selected addresses (our corporate network).

I am executing some python code on a notebook in the workspace. The code is trying to connect to a serving endpoint in the same workspace, but it gets a 403 error

'{"error_code":"403","message":"Source IP address: x.x.x.x is blocked by Databricks IP ACL for workspace: NNNNNNNNNNNN"}'

Why is this happening ? Does this traffic go through the general internet ? Why is it not confined inside the workspace ?

Thanks.

Amira Bedhiafi 25,261 Reputation points

2024-09-25T20:46:02.21+00:00
The issue you're encountering, where your Python code running in a Databricks notebook is being blocked by the Databricks IP ACL, likely happens because of how network traffic flows within the workspace.

In Azure Databricks, the networking rules (such as IP ACLs) are applied at the workspace level to control the traffic that enters and leaves the workspace. However, even though your notebook and the serving endpoint are part of the same workspace, the communication between them may still pass through external or internal networks that are subject to the workspace's IP ACL.

Here are some potential reasons for the issue:

Serving endpoints may route traffic through a public internet-facing interface or an internal Databricks-managed VPC, depending on your workspace's network configuration. If the traffic between the notebook and the endpoint is leaving the workspace and returning (even internally), the IP ACL could block the request, as it's being treated like any other external traffic.

The IP address of the notebook or the cluster you're using may not be included in the IP ACL rules you configured. While you might expect internal traffic to bypass these rules, it still needs to comply with the ACL unless specific allowances have been made.

If your workspace uses Azure Virtual Network (VNet) or private link, and IP ACLs are enforced, ensure that the subnet and the private endpoint configurations are properly set up to allow communication within the workspace. Any misconfigurations could cause traffic between different services (like notebooks and serving endpoints) to be blocked.

How to resolve this issue:

Check the IP address of the cluster running the notebook and ensure it is allowed in your Databricks IP ACL list. You can get the cluster’s public IP from the cluster details in Databricks.

Ensure that your network setup allows intra-workspace communication between the notebook cluster and the serving endpoint. If you're using a VNet, private link, or other networking features, verify that the subnets and network security groups allow traffic between these components.

If the serving endpoint is exposed to public internet traffic or relies on a different network configuration, consider adjusting the IP ACL rules or network security settings to ensure that traffic from within the workspace is allowed.

If you’re using Azure Private Link or VNet injection, ensure that the network rules and ACLs within those setups are configured to allow the necessary traffic between services within the workspace.
phemanth 10,740 Reputation points Microsoft Vendor

2024-09-27T13:00:48.97+00:00

@Repetti Pierangelo Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Repetti Pierangelo 20 Reputation points

2024-09-30T06:42:10.1866667+00:00

Hello,

unfortunately your suggestions were not helpful in this case.

The users changed their python code and it worked.

All the same, thanks for your comment.
phemanth 10,740 Reputation points Microsoft Vendor

2024-10-01T06:39:09.2733333+00:00

@Repetti Pierangelo

Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.
phemanth 10,740 Reputation points Microsoft Vendor

2024-10-04T03:38:30.89+00:00

@Repetti Pierangelo Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

phemanth 10,740 Reputation points Microsoft Vendor

2024-10-01T06:38:41.8633333+00:00

@Repetti Pierangelo

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

**Ask:**I have a databricks workspace for which I have defined an IP address ACL so that it can be reached by selected addresses (our corporate network).

I am executing some python code on a notebook in the workspace. The code is trying to connect to a serving endpoint in the same workspace, but it gets a 403 error

'{"error_code":"403","message":"Source IP address: x.x.x.x is blocked by Databricks IP ACL for workspace: NNNNNNNNNNNN"}'

Why is this happening ? Does this traffic go through the general internet ? Why is it not confined inside the workspace ?

Solution: The users changed their python code and it worked.

All the same, thanks for your comment.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
phemanth 10,740 Reputation points Microsoft Vendor

2024-10-03T04:25:24.3066667+00:00

@Repetti Pierangelo Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Notebook IP address blocked by workspace IP ACL

How to resolve this issue:

1 answer

Your answer