Not able to connect to Blob Storage using Spark in Azure General pupose storage account in AKS

Goel, Akanksha 66 Reputation points
2022-03-09T03:34:19.647+00:00

I am getting following error :

Exception occurred in driver code.Message: :Wrong FS: abfss://yyy@X .dfs.core.windows.net/etl3, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: abfss://yyy@X .dfs.core.windows.net/etl3, expected: file://

I am using the following hadoop configuration to connect to Blob:
sparkSession.sparkContext.hadoopConfiguration.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")

Properties to connect to Blob store :
BLOBSTORAGE_ACCNAME = "fs.azure.account.key.xxx.blob.core.windows.net"
BLOBSTORAGE_ACCKEY = "xxx"
OFFSET_PATH = "abfss://sdr@X .dfs.core.windows.net/etl3/"
DEFAULT_FS = "abfss://sdr@X .dfs.core.windows.net/"

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,427 questions
Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,855 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Prrudram-MSFT 21,966 Reputation points
    2022-03-19T11:52:46.76+00:00

    Hello @Goel, Akanksha ,

    Thank you for reaching out to the Microsoft Q&A platform. Happy to answer your question. I understand you are running Apache Spark jobs on AKS. Were you able to create a storage account and upload the jar files as described in here spark-job.
    Also, i found an article online with guided approach to run Apache Spark on AKS with Azure Storage integration here apache-spark-on-azure-kubernetes-service-aks
    Please go through this and see if it helps.

    I would need to check with internal teams as these errors doesn't have any TSGs in the documentation. Could you please share more details about the exact configuration you are using?

    Please "Accept as Answer" and Upvote if any of the above helped to help others in the community looking for remediation for similar issues.

    0 comments