Issue when running notebook from pipeline

Question

Issue when running notebook from pipeline

Shao Peng Sun 91

I have a notebook. And in this notebook, I update in spark session configuration to use service principle to make connection to storage account because I am only allowed to write into the storage account via the service principle.
And to make it work, I also need to change the spark session configuration to enable "Run as managed identity".
If I just run this notebook, it works very well. The problem is if I create a pipeline and run this notebook from pipeline, then it will be failed, and get error "

Operation failed: "This request is not authorized to perform this operation using this permission.", 403

" Could you please help on this? What should I do to make it also work by running this notebook from pipeline?

spark.conf.set("fs.azure.account.auth.type.kynsightprodweudlsa01.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.kynsightprodweudlsa01.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.kynsightprodweudlsa01.dfs.core.windows.net", sp_clientid)
spark.conf.set("fs.azure.account.oauth2.client.secret.kynsightprodweudlsa01.dfs.core.windows.net", sp_secretvalue)
spark.conf.set("fs.azure.account.oauth2.client.endpoint.kynsightprodweudlsa01.dfs.core.windows.net", client_endpoint)

Shao Peng Sun 91 Reputation points

2023-04-21T11:59:42.68+00:00

Thank you Bhargava! So that means if we only have write access to storage account with service principle and don't have Storage Blob Data Contributor role, then we cannot run the notebook via pipeline to write into the storage account? Is there a solution that we can specify to use that service principle in pipeline?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-04-25T22:41:17.4466667+00:00

Hello Shao Peng Sun, Sorry for the delayed response.

If you only have write access to the storage account using a service principal, and you do not have the "Storage Blob Data Contributor" role, you will not be able to run the notebook via pipeline to write into the storage account.

This is because the Synapse workspace's MSI, which is used by the pipeline to access resources, does not have the necessary permissions to write to the storage account.

To use the service principal in the pipeline, you'll need to specify the service principal's credentials in the pipeline's linked service definition. You can do this by creating a new linked service for the service principal and specifying the credentials in the linked service definition.

Please follow the below blog. Here You can use the service principle instead of the managed identity. https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/using-the-workspace-msi-to-authenticate-a-synapse-notebook-when/ba-p/2330029 I hope this helps. Please let me know if you have any further questions.
Shao Peng Sun 91 Reputation points

2023-04-26T05:09:10.75+00:00

Thank you @Bhargava-MSFT ! But in the pipeline I am using "notebook activity" to invoke the notebook script. And I didn't find a place to use linked service when using notebook activity in pipeline. Do you know how to do that?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-04-26T20:35:27.07+00:00

Hello Shao Peng Sun,

You need to update the notebook code to use the linked service configuration.

Microsoft engineer explained this in this blow tech article. Please go through it and let me know for any additional questions.

https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/using-the-workspace-msi-to-authenticate-a-synapse-notebook-when/ba-p/2330029

Notebook code:

val linked_service_name = “LinkedServerName”

// replace with your linked service name

%%spark

// Allow SPARK to access from Blob remotely

val sc = spark.sparkContext

spark.conf.set(“spark.storage.synapse.linkedServiceName”, linked_service_name)

spark.conf.set(“fs.azure.account.oauth.provider.type”, “com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider”)

//replace the container and storage account names

val df = “abfss://******@StorageAccount.dfs.core.windows.net/”

print(“Remote blob path: ” + df)

mssparkutils.fs.ls(df)
Shao Peng Sun 91 Reputation points

2023-04-27T08:23:18.27+00:00

Thank you @Bhargava-MSFT ! In that article, it mentioned when you config the linked service, the Linked Service need to use the workspace MSI. But what we have is only Service Principle, so if I use the Service Principle to create the linked service, and use this in notebook, and then create a pipeline to use this notebook, would it work?
Shao Peng Sun 91 Reputation points

2023-04-27T13:22:22.5933333+00:00

I just tried to create a linked service with service principle, and use that in notebook. The linked service works but the notebook failed with error below
: Status code: -1 error code: null error message: Auth failure: HTTP Error -1CustomTokenProvider getAccessToken threw java.io.IOException : POST failed with 'Bad Request' (400) and message: {"result":"DependencyError","errorId":"BadRequest","errorMessage":"[Code=, Target=, Message=]. TraceId : 039fda77-a75c-41b2-afd7-34a3afd6a9bb | client-request-id : 0c9d1158-ab88-457a-9581-d01299b8618f. Error Component : LSR"}org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator$HttpException: HTTP Error -1CustomTokenProvider getAccessToken threw java.io.IOException : POST failed with 'Bad Request' (400) and message: {"result":"DependencyError","errorId":"BadRequest","errorMessage":"[Code=, Target=, Message=]. TraceId : 039fda77-a75c-41b2-afd7-34a3afd6a9bb | client-request-id : 0c9d1158-ab88-457a-9581-d01299b8618f. Error Component : LSR"}
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-04-27T21:07:19.6666667+00:00

Hello Shao Peng Sun,

As per the error message seems like there was an authentication failure when trying to access a resource.

Did you provide the storage blob data contributor access?

Following that document, you can run the code with the service principal.

From my end, I successfully ran the code using the notebook.

And yes, instead of the managed identity, you can use the service principal( I have mentioned this in my previous comment)

Here is my linked service with the service principal connection:

My notebook code:

I hope this helps. Please let me know if you have any further questions.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-05-01T15:43:23.59+00:00

Hello Shao Peng Sun,

I am checking to see if you have any further questions here.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.
Shao Peng Sun 91 Reputation points

2023-05-04T04:48:42.82+00:00
Sorry I just back from my holiday.

I am not sure why. I could see the solution works for you. But it does not work for my.
What I have are: 1) SP which has write access to Storage Account. 2) A linked service created by using the SP. 3) My own id which does not have write access to Storage Account.

What happen in my side is:

If I add the SP into spark session by scripts below and execute the script in notebook, then it works and I can write into storage account.

spark.conf.set("fs.azure.account.auth.type.kynsightprodweudlsa01.dfs.core.windows.net", "OAuth") spark.conf.set("fs.azure.account.oauth.provider.type.kynsightprodweudlsa01.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider") spark.conf.set("fs.azure.account.oauth2.client.id.kynsightprodweudlsa01.dfs.core.windows.net", sp_clientid) spark.conf.set("fs.azure.account.oauth2.client.secret.kynsightprodweudlsa01.dfs.core.windows.net", sp_secretvalue) spark.conf.set("fs.azure.account.oauth2.client.endpoint.kynsightprodweudlsa01.dfs.core.windows.net", client_endpoint)

But if I create a pipeline to invoke the notebook which could run by itself, then it got failed.

If I add the linked service into spark session and execute the script in notebook, it failed to write into storage account. But when I test the linked service, it works.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-05-04T18:15:06.5266667+00:00

Hello Shao Peng Sun,

looks like the issue is with Access.

Can you provide "storage blob data contributor" access to the synpase workspace managed service identity

as you are running the notebook via the pipeline?

1 answer

Your answer

Shao Peng Sun 91 Reputation points

2023-04-21T11:59:42.68+00:00

Thank you Bhargava! So that means if we only have write access to storage account with service principle and don't have Storage Blob Data Contributor role, then we cannot run the notebook via pipeline to write into the storage account? Is there a solution that we can specify to use that service principle in pipeline?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-04-25T22:41:17.4466667+00:00

Hello Shao Peng Sun, Sorry for the delayed response.

If you only have write access to the storage account using a service principal, and you do not have the "Storage Blob Data Contributor" role, you will not be able to run the notebook via pipeline to write into the storage account.

This is because the Synapse workspace's MSI, which is used by the pipeline to access resources, does not have the necessary permissions to write to the storage account.

To use the service principal in the pipeline, you'll need to specify the service principal's credentials in the pipeline's linked service definition. You can do this by creating a new linked service for the service principal and specifying the credentials in the linked service definition.

Please follow the below blog. Here You can use the service principle instead of the managed identity. https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/using-the-workspace-msi-to-authenticate-a-synapse-notebook-when/ba-p/2330029 I hope this helps. Please let me know if you have any further questions.
Shao Peng Sun 91 Reputation points

2023-04-26T05:09:10.75+00:00

Thank you @Bhargava-MSFT ! But in the pipeline I am using "notebook activity" to invoke the notebook script. And I didn't find a place to use linked service when using notebook activity in pipeline. Do you know how to do that?
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-04-26T20:35:27.07+00:00

Hello Shao Peng Sun,

You need to update the notebook code to use the linked service configuration.

Microsoft engineer explained this in this blow tech article. Please go through it and let me know for any additional questions.

https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/using-the-workspace-msi-to-authenticate-a-synapse-notebook-when/ba-p/2330029

Notebook code:

val linked_service_name = “LinkedServerName”

// replace with your linked service name

%%spark

// Allow SPARK to access from Blob remotely

val sc = spark.sparkContext

spark.conf.set(“spark.storage.synapse.linkedServiceName”, linked_service_name)

spark.conf.set(“fs.azure.account.oauth.provider.type”, “com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider”)

//replace the container and storage account names

val df = “abfss://******@StorageAccount.dfs.core.windows.net/”

print(“Remote blob path: ” + df)

mssparkutils.fs.ls(df)
Shao Peng Sun 91 Reputation points

2023-04-27T08:23:18.27+00:00

Thank you @Bhargava-MSFT ! In that article, it mentioned when you config the linked service, the Linked Service need to use the workspace MSI. But what we have is only Service Principle, so if I use the Service Principle to create the linked service, and use this in notebook, and then create a pipeline to use this notebook, would it work?
Shao Peng Sun 91 Reputation points

2023-04-27T13:22:22.5933333+00:00

I just tried to create a linked service with service principle, and use that in notebook. The linked service works but the notebook failed with error below
: Status code: -1 error code: null error message: Auth failure: HTTP Error -1CustomTokenProvider getAccessToken threw java.io.IOException : POST failed with 'Bad Request' (400) and message: {"result":"DependencyError","errorId":"BadRequest","errorMessage":"[Code=, Target=, Message=]. TraceId : 039fda77-a75c-41b2-afd7-34a3afd6a9bb | client-request-id : 0c9d1158-ab88-457a-9581-d01299b8618f. Error Component : LSR"}org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator$HttpException: HTTP Error -1CustomTokenProvider getAccessToken threw java.io.IOException : POST failed with 'Bad Request' (400) and message: {"result":"DependencyError","errorId":"BadRequest","errorMessage":"[Code=, Target=, Message=]. TraceId : 039fda77-a75c-41b2-afd7-34a3afd6a9bb | client-request-id : 0c9d1158-ab88-457a-9581-d01299b8618f. Error Component : LSR"}
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-04-27T21:07:19.6666667+00:00

Hello Shao Peng Sun,

As per the error message seems like there was an authentication failure when trying to access a resource.

Did you provide the storage blob data contributor access?

Following that document, you can run the code with the service principal.

From my end, I successfully ran the code using the notebook.

And yes, instead of the managed identity, you can use the service principal( I have mentioned this in my previous comment)

Here is my linked service with the service principal connection:

My notebook code:

I hope this helps. Please let me know if you have any further questions.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-05-01T15:43:23.59+00:00

Hello Shao Peng Sun,

I am checking to see if you have any further questions here.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.
Shao Peng Sun 91 Reputation points

2023-05-04T04:48:42.82+00:00

Sorry I just back from my holiday.

I am not sure why. I could see the solution works for you. But it does not work for my.
What I have are: 1) SP which has write access to Storage Account. 2) A linked service created by using the SP. 3) My own id which does not have write access to Storage Account.

What happen in my side is:

If I add the SP into spark session by scripts below and execute the script in notebook, then it works and I can write into storage account.

spark.conf.set("fs.azure.account.auth.type.kynsightprodweudlsa01.dfs.core.windows.net", "OAuth") spark.conf.set("fs.azure.account.oauth.provider.type.kynsightprodweudlsa01.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider") spark.conf.set("fs.azure.account.oauth2.client.id.kynsightprodweudlsa01.dfs.core.windows.net", sp_clientid) spark.conf.set("fs.azure.account.oauth2.client.secret.kynsightprodweudlsa01.dfs.core.windows.net", sp_secretvalue) spark.conf.set("fs.azure.account.oauth2.client.endpoint.kynsightprodweudlsa01.dfs.core.windows.net", client_endpoint)

But if I create a pipeline to invoke the notebook which could run by itself, then it got failed.

If I add the linked service into spark session and execute the script in notebook, it failed to write into storage account. But when I test the linked service, it works.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-05-04T18:15:06.5266667+00:00

Hello Shao Peng Sun,

looks like the issue is with Access.

Can you provide "storage blob data contributor" access to the synpase workspace managed service identity

as you are running the notebook via the pipeline?

Answer 1

Hello Shao Peng Sun, Please correct me if my understanding is wrong. You were having issues when running the notebook via the pipeline. But it's running fine, when you run it from the synapse studio.

Synapse notebooks use Azure Active Directory (Azure AD) pass-through to access the ADLS Gen2 accounts. If you are running the notebook directly on the synapse then your account needs to have Storage Blob Data Contributor to access the ADLS Gen2 account (or folder).

If you are running the notebook via the pipeline, then synpase workspace managed service identity needs to have Storage Blob Data Contributor to access the ADLS Gen2 account (or folder).

I hope this helps. Please let me know if you have any further questions.

Share via

Issue when running notebook from pipeline

1 answer

Your answer