How to run pysftp connection using serverless compute in Azure Databricks

Question

How to run pysftp connection using serverless compute in Azure Databricks

Anonymous

Hello Community,

The code below ran successfully using all purpose cluster, but when I switched to serverless compute, I ran into exception: connection timed out. Usually it would take around 2 mins and then failed. User's image

I searched online about this issue and but I was not able to find one related to running pysftp using serverless compute in Databricks. What're the potential root causes? How to fix this issue?

Thanks.

Anonymous

2025-04-30T05:12:34.54+00:00
please check the below troubleshooting steps and confirm us

Serverless compute might have different network configurations compared to all-purpose clusters. This could lead to connectivity issues with external servers.

The serverless environment might have stricter firewall rules, blocking outbound connections to your SFTP server.

Serverless compute might have different timeout settings, causing the connection to fail if it takes too long to establish.

Solutions:

Ensure that the serverless compute environment has the necessary permissions to access the SFTP server. You might need to configure network security groups (NSGs) or firewall rules to allow outbound traffic to the SFTP server's IP and port.

As an alternative, you can use Azure Data Factory (ADF) to handle the SFTP connection and data transfer. ADF can copy data from the SFTP server to Azure Data Lake Storage (ADLS), and then you can trigger a

If possible, try increasing the timeout settings in your pysftp connection configuration. This might help if the connection is taking longer to establish due to network latency.

Add more detailed logging to your script to capture the exact point of failure. This can help identify whether the issue is with the network, authentication, or something else.

import pysftp import logging logging.basicConfig(level=logging.DEBUG) def get_pysftp_connection(): username = 'abc' password = 'xyz' hostname = 'efg.com' port = 22 cnopts = pysftp.CnOpts() cnopts.hostkeys = None # Disable host key checking try: connection = pysftp.Connection(host=hostname, username=username, password=password, port=port, cnopts=cnopts) logging.info("Connection established successfully") return connection except Exception as e: logging.error(f"Failed to establish connection: {e}") raise # Establish SSH connection connection = get_pysftp_connection() with connection as sftp: # Perform operations here files = sftp.listdir() print(files)
Anonymous

2025-04-30T13:39:03.17+00:00
Hi phemanth, thanks for your answer.

We didn't do any networking related configurations, just by default which means serverless compute's network policy is ALL allowing any outbound connection. Also, since all purpose cluster can connect SFTP, so nsg and firewall should be okay.

We are not allowed to use ADF.

Increased timeout to 10 mins, but still fail in 2 mins.

Logging does nothing but just log the same exception information. Not very helpful for debugging.

I notice that even we use dummy information (i.e. fabricated username, password, hostname), we will get the same exception information. Looks like the outbound connection is not allowed, but the network policy for serverless is ALL, and also in another job, we can run RestAPI to connect to external systerm using serverless compute. Really strange.

Do you know any other possible solutions?

Thanks.
Anonymous

2025-05-02T04:20:23.79+00:00
@Quan Sun Given that the serverless compute environment's network policy is set to allow all outbound connections, and you've confirmed that other jobs can connect to external systems, the issue might be more specific to the pysftp library or the way it interacts with the serverless environment.

Here are additional suggestions

Check for Library Compatibility: Ensure that the versions of pysftp and its dependencies (like paramiko) are compatible with the serverless environment. Sometimes, specific versions might have issues that are resolved in later releases.

Use Paramiko Directly: Since pysftp is built on top of paramiko, you might try using paramiko directly to see if it provides more detailed error messages or better compatibility.

import paramiko import logging logging.basicConfig(level=logging.DEBUG) def get_paramiko_connection(): username = 'abc' password = 'xyz' hostname = 'efg.com' port = 22 try: transport = paramiko.Transport((hostname, port)) transport.connect(username=username, password=password) sftp = paramiko.SFTPClient.from_transport(transport) logging.info("Connection established successfully") return sftp except Exception as e: logging.error(f"Failed to establish connection: {e}") raise # Establish SSH connection sftp = get_paramiko_connection() with sftp as sftp_client: # Perform operations here files = sftp_client.listdir() print(files)

Network Diagnostics: Run network diagnostics from within the serverless environment to ensure there are no hidden network issues. You can use tools like ping, telnet, or nc (netcat) to test connectivity to the SFTP server.

import os hostname = 'efg.com' response = os.system(f"ping -c 4 {hostname}") if response == 0: print(f"{hostname} is reachable") else: print(f"{hostname} is not reachable")

Custom DNS Resolution: Sometimes, DNS resolution issues can cause connectivity problems. Ensure that the serverless environment can resolve the hostname correctly. You might try using the IP address directly instead of the hostname.
Anonymous

2025-05-05T04:26:56.85+00:00

@Quan Sun We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Anonymous

2025-05-23T00:39:39.1366667+00:00

@Anonymous Sorry for late reply and thank you so much for the suggestions. We tried all of the suggestions above, all failed. So, we decided to use all-purpose cluster instead of serverless compute.

Your answer

Anonymous

2025-04-30T05:12:34.54+00:00

please check the below troubleshooting steps and confirm us

Serverless compute might have different network configurations compared to all-purpose clusters. This could lead to connectivity issues with external servers.

The serverless environment might have stricter firewall rules, blocking outbound connections to your SFTP server.

Serverless compute might have different timeout settings, causing the connection to fail if it takes too long to establish.

Solutions:

Ensure that the serverless compute environment has the necessary permissions to access the SFTP server. You might need to configure network security groups (NSGs) or firewall rules to allow outbound traffic to the SFTP server's IP and port.

As an alternative, you can use Azure Data Factory (ADF) to handle the SFTP connection and data transfer. ADF can copy data from the SFTP server to Azure Data Lake Storage (ADLS), and then you can trigger a

If possible, try increasing the timeout settings in your pysftp connection configuration. This might help if the connection is taking longer to establish due to network latency.

Add more detailed logging to your script to capture the exact point of failure. This can help identify whether the issue is with the network, authentication, or something else.

import pysftp import logging logging.basicConfig(level=logging.DEBUG) def get_pysftp_connection(): username = 'abc' password = 'xyz' hostname = 'efg.com' port = 22 cnopts = pysftp.CnOpts() cnopts.hostkeys = None # Disable host key checking try: connection = pysftp.Connection(host=hostname, username=username, password=password, port=port, cnopts=cnopts) logging.info("Connection established successfully") return connection except Exception as e: logging.error(f"Failed to establish connection: {e}") raise # Establish SSH connection connection = get_pysftp_connection() with connection as sftp: # Perform operations here files = sftp.listdir() print(files)
Anonymous

2025-04-30T13:39:03.17+00:00

Hi phemanth, thanks for your answer.

We didn't do any networking related configurations, just by default which means serverless compute's network policy is ALL allowing any outbound connection. Also, since all purpose cluster can connect SFTP, so nsg and firewall should be okay.

We are not allowed to use ADF.

Increased timeout to 10 mins, but still fail in 2 mins.

Logging does nothing but just log the same exception information. Not very helpful for debugging.

I notice that even we use dummy information (i.e. fabricated username, password, hostname), we will get the same exception information. Looks like the outbound connection is not allowed, but the network policy for serverless is ALL, and also in another job, we can run RestAPI to connect to external systerm using serverless compute. Really strange.

Do you know any other possible solutions?

Thanks.
Anonymous

2025-05-05T04:26:56.85+00:00

@Quan Sun We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Anonymous

2025-05-23T00:39:39.1366667+00:00

@Anonymous Sorry for late reply and thank you so much for the suggestions. We tried all of the suggestions above, all failed. So, we decided to use all-purpose cluster instead of serverless compute.

Share via

How to run pysftp connection using serverless compute in Azure Databricks

Your answer