Access secret key from Azure Keyvault to read and write to blob containers

Anisha Rayapati 11 Reputation points
2021-08-27T21:21:49.397+00:00

I am getting a 401 error while using the below code

Error: NoAuthenticationInformation, "Server failed to authenticate the request. Please refer to the information in the www-authenticate header.

%%pyspark
spark.conf.set("fs.azure.account.auth.type", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type", "com.microsoft.azure.synapse.tokenlibrary.AkvBasedSASProvider")
spark.conf.set("spark.storage.synapse.akv", "<AZURE KEY VAULT NAME>")
spark.conf.set("spark.storage.akv.secret", "<SECRET KEY>")

df = spark.read.csv('abfss://<CONTAINER>@<ACCOUNT>.dfs.core.windows.net/<FILE PATH>')

display(df.limit(10))

Azure Key Vault
Azure Key Vault
An Azure service that is used to manage and protect cryptographic keys and other secrets used by cloud apps and services.
1,194 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,696 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Gareth Western 11 Reputation points
    2021-11-25T14:22:10.83+00:00

    Use Microsoft Spark Utilities:

    secret= mssparkutils.credentials.getSecret(KEYVAULT_NAME, SECRET_NAME)  
    
    0 comments No comments

  2. PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee
    2021-12-08T07:20:30.237+00:00

    Hello @Arnfried Rixen , @Anisha Rayapati , @Sivagnana Sundaram, Krithiga ,

    Thanks for the question and using MS Q&A platform.

    This is a known issue with product. Our product team currently working on the solution. I will update this thread once it’s available.

    Meanwhile workaround is to create a linked service against the Azure key vault.
    Use get secret with linked service using the below documentation:
    https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-secure-credentials-with-tokenlibrary?pivots=programming-language-csharp#getsecret

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments