Error when using pandas to write data to ADLS in Azure Synapse Analytics

Question

Error when using pandas to write data to ADLS in Azure Synapse Analytics

Viet Tran 50

Hi Mr/Mrs,

I want to use pandas to write dataframe to .csv file into ADLS, but I got this issue although I put the access key, it didn't work.

User's image

How can I fix this issue?

Thanks!

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-04-29T11:17:14.6433333+00:00
@Viet Tran

Thanks for reaching out to Microsoft Q&A

I see you’re having trouble connecting to Azure Data Lake Storage (ADLS) from your Python script. The error message “Cannot connect to host <URL>:443 ssl:True [Name or service not known]” suggests a connection issue.

Here are a few things you could check:

Correct URL and Access Keys: Make sure the URL and access keys are correct. Typos or incorrect keys can lead to connection issues.

Network Permissions: Ensure that your network allows connections to the ADLS. Some networks may have restrictions or firewalls that block certain connections.

Firewall or Network Security Group Rules: There might be firewall or network security group rules that are blocking your connection. You might need to update these rules to allow your script to connect to ADLS.

Hope this helps. Do let us know if you any further queries.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-01T14:06:11.2433333+00:00

@Viet Tran We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Viet Tran 50 Reputation points

2024-05-07T07:56:29.2366667+00:00

Hi @phemanth,
Actually, I follow the instruction from this link to try to write .json file to storage account.

https://stackoverflow.com/questions/76510602/writing-json-data-to-file-in-azure-synapse-pyspark-notebook

I checked Correct URL and Access Keys and sured that this is right.

How can I check the Network Permissions and Firewall or Network Security Group Rules? Which permission and rule can I use for using pandas to write .json file into storage account?

Many thanks!
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-08T07:19:13.6533333+00:00
@Viet Tran While the provided Stack Overflow link might not directly address checking Network Permissions and Firewall or Network Security Group Rules, here's a general approach you can follow:

Network Permissions:

Azure Synapse Analytics workspaces typically inherit network permissions from the underlying Azure virtual network.

To check network permissions, you'll need access to Azure networking resources and familiarity with Azure virtual networks.

Consult your Azure administrator or network security team for details on network permissions configured for your workspace.

Firewall/Network Security Group (NSG) Rules:

NSGs are used to control inbound and outbound traffic to Azure resources.

You can check NSG rules associated with your Synapse workspace or storage account in the Azure portal.

Here's a step-by-step guide to check NSG rules in the Azure portal:

Sign in to the Azure portal and navigate to your Synapse workspace.

Go to the Settings section of your workspace.

Under Security, click on Network security groups.

This will list any NSGs associated with your workspace. You can also check NSGs associated with your storage account by navigating to the storage account settings and following similar steps.

Click on an NSG to view its detailed configuration, including inbound and outbound security rules.

Rules for Writing JSON to ADLS using Pandas:

To write data to ADLS using pandas, you'll typically need an NSG rule that allows outbound traffic on port 443 (HTTPS) for the destination address of your ADLS storage account.

The NSG rule should also specify the appropriate protocol (TCP) and source (e.g., IP address or subnet of your Synapse workspace)

1 answer

Your answer

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-04-29T11:17:14.6433333+00:00

@Viet Tran

Thanks for reaching out to Microsoft Q&A

I see you’re having trouble connecting to Azure Data Lake Storage (ADLS) from your Python script. The error message “Cannot connect to host <URL>:443 ssl:True [Name or service not known]” suggests a connection issue.

Here are a few things you could check:

Correct URL and Access Keys: Make sure the URL and access keys are correct. Typos or incorrect keys can lead to connection issues.

Network Permissions: Ensure that your network allows connections to the ADLS. Some networks may have restrictions or firewalls that block certain connections.

Firewall or Network Security Group Rules: There might be firewall or network security group rules that are blocking your connection. You might need to update these rules to allow your script to connect to ADLS.

Hope this helps. Do let us know if you any further queries.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-01T14:06:11.2433333+00:00

@Viet Tran We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Viet Tran 50 Reputation points

2024-05-07T07:56:29.2366667+00:00

Hi @phemanth,
Actually, I follow the instruction from this link to try to write .json file to storage account.

https://stackoverflow.com/questions/76510602/writing-json-data-to-file-in-azure-synapse-pyspark-notebook

I checked Correct URL and Access Keys and sured that this is right.

How can I check the Network Permissions and Firewall or Network Security Group Rules? Which permission and rule can I use for using pandas to write .json file into storage account?

Many thanks!
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-08T07:19:13.6533333+00:00

@Viet Tran While the provided Stack Overflow link might not directly address checking Network Permissions and Firewall or Network Security Group Rules, here's a general approach you can follow:

Network Permissions:

Azure Synapse Analytics workspaces typically inherit network permissions from the underlying Azure virtual network.

To check network permissions, you'll need access to Azure networking resources and familiarity with Azure virtual networks.

Consult your Azure administrator or network security team for details on network permissions configured for your workspace.

Firewall/Network Security Group (NSG) Rules:

NSGs are used to control inbound and outbound traffic to Azure resources.

You can check NSG rules associated with your Synapse workspace or storage account in the Azure portal.

Here's a step-by-step guide to check NSG rules in the Azure portal:

Sign in to the Azure portal and navigate to your Synapse workspace.

Go to the Settings section of your workspace.

Under Security, click on Network security groups.

This will list any NSGs associated with your workspace. You can also check NSGs associated with your storage account by navigating to the storage account settings and following similar steps.

Click on an NSG to view its detailed configuration, including inbound and outbound security rules.

Rules for Writing JSON to ADLS using Pandas:

To write data to ADLS using pandas, you'll typically need an NSG rule that allows outbound traffic on port 443 (HTTPS) for the destination address of your ADLS storage account.

The NSG rule should also specify the appropriate protocol (TCP) and source (e.g., IP address or subnet of your Synapse workspace)

Answer 1

Amira Bedhiafi 33,071 Volunteer Moderator

I think your issue might be with the path : Your storage account path should be in abfss://<containerName>@<storageaccountName>.dfs.core.windows.net/<path>

Please note, the authentication identity which you are using inside linked service of adls gen2 should have Storage Blob data contributor role.

Share via

Error when using pandas to write data to ADLS in Azure Synapse Analytics

1 answer

Your answer