ADB is Azure service so you should ideally connect with Azure IR only.
You can take a look at below document where it says we can enable IP level security. Databricks will be accessible only from allowed IP addresses.
https://learn.microsoft.com/en-us/azure/databricks/security/network/ip-access-list
----------
Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav