Accessing Azure ADLS gen2 with Pyspark on Windows

alexander tikhomirov 31 Reputation points

Maybe it not corrected tag about azure synapse, but my team is checking to have local dev experience to write code for Synapse Job definition. What the best practise in this case?
I have working local dev environment VSCode, PySpark and I could execute PySpark code and theoretically use the same scripts in Synapse Job definition, but I would like to use command'abfss://XXX@X', format='csv')

to read file from adls gen2 from both environment, locally and in remote spark cluster, is it possible? Someone has such experience?

locally doesn't work

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,553 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 38,301 Reputation points Microsoft Employee

    Hi @alexander tikhomirov ,

    Thank you for posting query in Microsoft Q&A Platform.

    To access ADLS gen2 from windows machine. You need to perform below high level steps.

    • Set up the environment
    • Configure your storage account in Hadoop
    • Connect to your storage account.

    All above steps are detailed documented in below link. Kindly follow the steps and see if that helps. Please let us know how it goes.