Databricks Libraries Installation SSL Error

Question

Databricks Libraries Installation SSL Error

Balwant Singh 55

HELP ME TO SOLVE THIS ERROR- PLEASE ELABORATE IN DETAIL...

Library installation attempted on the driver node of cluster 0606-104958-63c5i7so and failed. Please refer to the following error message to fix the library or contact Databricks support. Error code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'great-expectations' --disable-pip-version-check) exited with code 1. WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)'))': /packages/05/f8/beb300ec886bc4e7ff1fe50272b49f37dda9d21747be13b55893062ae1f7/great_expectations-1.1.0-py3-none-any.whl WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)'))': /packages/05/f8/beb300ec886bc4e7ff1fe50272b49f37dda9d21747be13b55893062ae1f7/great_expectations-1.1.0-py3-none-any.whl WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)'))': /packages/05/f8/beb300ec886bc4e7ff1fe50272b49f37dda9d21747be13b55893062ae1f7/great_expectations-1.1.0-py3-none-any.whl WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)'))': /packages/05/f8/beb300ec886bc4e7ff1fe50272b49f37dda9d21747be13b55893062ae1f7/great_expectations-1.1.0-py3-none-any.whl WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)'))': /packages/05/f8/beb300ec886bc4e7ff1fe50272b49f37dda9d21747be13b55893062ae1f7/great_expectations-1.1.0-py3-none-any.whl ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/05/f8/beb300ec886bc4e7ff1fe50272b49f37dda9d21747be13b55893062ae1f7/great_expectations-1.1.0-py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)'))) i want to look into this error when we face this error i want to troubleshoot the environment and run them clear

Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2024-10-09T09:51:47.8033333+00:00
Hi @Balwant Singh Chouhan
Welcome to Microsoft Q&A platform and thank you for reaching out.
Based on the error you're encountering, it looks like there may be a connectivity issue between your Databricks cluster and the source where you're trying to install the library. This could be caused by network restrictions, such as firewalls or DNS settings, preventing your cluster from reaching the Python Package Index (PyPI) or other sources.

Here are some steps to troubleshoot the issue:

Network Configuration:

Please check if any firewalls, Network Security Groups (NSGs), or route tables are blocking outbound connectivity from the cluster to the following domains:

pypi.org

files.pythonhosted.org

You may need to allow outbound traffic to these sources to successfully download the libraries.

DNS Configuration:

If you're seeing "Unknown host" or other DNS-related errors, there could be an issue with the DNS server configuration on your network. Ensure that the DNS servers used by your virtual network can resolve the required domain names.

Retry Installation:

After verifying the network and DNS settings, please retry the installation.

I hope it helps. please let me know if you have any further questions.
Balwant Singh 55 Reputation points

2024-10-09T11:03:41.01+00:00

Thank you so much.

Do any chance I would replicate this into my environment, or we connect over teams. Please let me know if it possible.
Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2024-10-10T08:22:30.68+00:00

@Balwant Singh Chouhan Thank you for your response. I'm glad to hear that my suggestions were helpful. Regarding your question about replicating the solution in your environment or connecting over Teams, I'm happy to help you troubleshoot the issue further.

If you would like to replicate the solution in your environment, you can follow the steps I provided in my previous response to troubleshoot the issue. If you encounter any issues or have any questions during the process, feel free to ask and I'll do my best to assist you here.
Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2024-10-11T18:41:38.0633333+00:00

@Balwant Singh Chouhan Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Balwant Singh 55 Reputation points

2024-10-11T18:46:27.65+00:00

Sorry to type i cannot create the architecture 7little confused, how i replicate this could you help me to do so in this way, pasting image here
Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2024-10-14T14:15:36.98+00:00
Hi @Balwant Singh Chouhan

Thank you for providing more clarity with the diagram.

Based on the architecture you shared and the SSL error you're encountering during the library installation in Databricks, it seems the issue is related to your environment’s network configuration, particularly involving SSL verification when trying to install libraries from external sources (like PyPI). Here's a more detailed guide to help you replicate the architecture and troubleshoot the error:

Network Configuration (Firewall and Private Endpoint):

In your architecture, the SHIR-VM on your on-premises setup communicates with Azure Databricks through a Virtual Network Gateway (VNG), firewall, and private endpoint.

You need to ensure that outbound traffic from your Databricks cluster can reach external domains such as pypi.org and files.pythonhosted.org for library installations.

Check if any firewall rules are blocking outbound HTTPS traffic. You may need to whitelist these domains in your firewall to allow access for downloading libraries.

SSL Certificate Error Resolution:

The error you are encountering indicates that the SSL certificate verification failed, likely due to your cluster not being able to verify the source's certificate.

To resolve this, ensure your cluster has access to the necessary Certificate Authority (CA) bundle used to validate these certificates. This can sometimes be resolved by ensuring the virtual network allows the necessary internet access or installing a valid CA certificate on the cluster.

DNS and Private Endpoint Setup:

Since you are using a private endpoint, it's possible that DNS resolution is restricted within your virtual network. Verify that the DNS settings in your environment can resolve external domain names like files.pythonhosted.org.

If DNS resolution is failing, you might need to configure your VNet to use public DNS servers or set up conditional forwarding for these domains.

Retry Installation:

Once the network and DNS configurations are corrected, retry the library installation on your Databricks cluster by running the following command:

pip install great-expectations --trusted-host pypi.org --trusted-host files.pythonhosted.org

This command bypasses SSL verification but should only be used after you've verified that network and firewall settings are correctly configured.

I hope it helps. please let me know if you have any further questions.

1 answer

Your answer

Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2024-10-09T09:51:47.8033333+00:00

Hi @Balwant Singh Chouhan
Welcome to Microsoft Q&A platform and thank you for reaching out.
Based on the error you're encountering, it looks like there may be a connectivity issue between your Databricks cluster and the source where you're trying to install the library. This could be caused by network restrictions, such as firewalls or DNS settings, preventing your cluster from reaching the Python Package Index (PyPI) or other sources.

Here are some steps to troubleshoot the issue:

Network Configuration:

Please check if any firewalls, Network Security Groups (NSGs), or route tables are blocking outbound connectivity from the cluster to the following domains:

pypi.org

files.pythonhosted.org

You may need to allow outbound traffic to these sources to successfully download the libraries.

DNS Configuration:

If you're seeing "Unknown host" or other DNS-related errors, there could be an issue with the DNS server configuration on your network. Ensure that the DNS servers used by your virtual network can resolve the required domain names.

Retry Installation:

After verifying the network and DNS settings, please retry the installation.

I hope it helps. please let me know if you have any further questions.
Balwant Singh 55 Reputation points

2024-10-09T11:03:41.01+00:00

Thank you so much.

Do any chance I would replicate this into my environment, or we connect over teams. Please let me know if it possible.
Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2024-10-10T08:22:30.68+00:00

@Balwant Singh Chouhan Thank you for your response. I'm glad to hear that my suggestions were helpful. Regarding your question about replicating the solution in your environment or connecting over Teams, I'm happy to help you troubleshoot the issue further.

If you would like to replicate the solution in your environment, you can follow the steps I provided in my previous response to troubleshoot the issue. If you encounter any issues or have any questions during the process, feel free to ask and I'll do my best to assist you here.
Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2024-10-11T18:41:38.0633333+00:00

@Balwant Singh Chouhan Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Balwant Singh 55 Reputation points

2024-10-11T18:46:27.65+00:00

Sorry to type i cannot create the architecture 7little confused, how i replicate this could you help me to do so in this way, pasting image here

Answer 1

@Anonymous I have seen the same in different customers. All these customers have onething in common , where they had an appliance in between Databricks and Internet which intercepts the traffic to do its job. When that happened the actual SSL certificate thumbprint is lost resulting in similar error. I'm not sure how secure your environment and what level of control you have. We can try couple of things.

Talk to your networking team and ask them to whitelist these URL's in their appliance so that its not intercepted
Add a custom route to your subnet to route this traffic directly to the Azure internet instead of the appliance.
Last and fast approach is to download manually from the URL and upload to the workspace storage. You can then install the library from the workspace storage.

Share via

Databricks Libraries Installation SSL Error

1 answer

Your answer