Unable to write data in ADLS

Question

I used the code provided in the link (https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/tutorial-spark-pool-filesystem-spec#readwrite-data-using-linked-service) for writing data from synapse workspace to ADLS Gen2.
I am getting an error with message "ValueError: Protocol not known: abfss" .Below is the code

import fsspec
import pandas

adls_account_name = '<<adlsName>>' #Provide exact ADLS account name

data = pandas.DataFrame({'Name':['Tom', 'nick', 'krish', 'jack'], 'Age':[20, 21, 19, 18]})
sas_key = TokenLibrary.getConnectionString('<<linkedServicePath>>')

fsspec_handle = fsspec.open('abfss://<<containerName>>/<<path>>', account_name =adls_account_name, sas_token=sas_key, mode="wt")

with fsspec_handle.open() as f:
data.to_csv(f)

Note: I am using FSSPEC to write the data as my spark pool has python version of 3.6.

Answer 1

Hello @Sukumar Vinnakota ,

Thanks for the question and using MS Q&A platform.

This is known limitation with pandas dataframe does not support the protocol - abfss.

As per the repro, when I tried to pass the path as abfss - experiencing the same error message as shown above:

When I tried to pass the path as abfs it's worked as excpeted.

For more details, refer to the SO thread which addressing similar issue.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators