question

GoelAkanksha-9319 avatar image
0 Votes"
GoelAkanksha-9319 asked RamSunkara-0818 commented

Not able to connect to Blob Storage using Spark in Azure General pupose storage account in AKS

I am getting following error :

Exception occurred in driver code.Message: :Wrong FS: abfss://yyy@xxx.dfs.core.windows.net/etl3, expected: file:/// java.lang.IllegalArgumentException: Wrong FS: abfss://yyy@xxx.dfs.core.windows.net/etl3, expected: file://

I am using the following hadoop configuration to connect to Blob:
sparkSession.sparkContext.hadoopConfiguration.set("fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")

Properties to connect to Blob store :
BLOBSTORAGE_ACCNAME = "fs.azure.account.key.xxx.blob.core.windows.net"
BLOBSTORAGE_ACCKEY = "xxx"
OFFSET_PATH = "abfss://sdr@xxx.dfs.core.windows.net/etl3/"
DEFAULT_FS = "abfss://sdr@xxx.dfs.core.windows.net/"


azure-blob-storageazure-kubernetes-service
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@Prrudram-MSFT

  • We hit the same error today

  • We followed the link you suggested, it did not help either

  • Here is my docker file I used to make spark image and jar files copied
    193005-dockerfilespark.txt

Any help is really appreciated.



0 Votes 0 ·

1 Answer

Prrudram-MSFT avatar image
0 Votes"
Prrudram-MSFT answered

Hello @GoelAkanksha-9319,

Thank you for reaching out to the Microsoft Q&A platform. Happy to answer your question. I understand you are running Apache Spark jobs on AKS. Were you able to create a storage account and upload the jar files as described in here spark-job.
Also, i found an article online with guided approach to run Apache Spark on AKS with Azure Storage integration here apache-spark-on-azure-kubernetes-service-aks
Please go through this and see if it helps.

I would need to check with internal teams as these errors doesn't have any TSGs in the documentation. Could you please share more details about the exact configuration you are using?

Please "Accept as Answer" and Upvote if any of the above helped to help others in the community looking for remediation for similar issues.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.