Self Hosted Integration Runtime - Access Key Vault

Radhika Ganapathi Ndq 6

Hi,

I am using a setup of Data Factory - Self Hosted Integration Runtime for moving data from on-prem to azure blob storage.

Recently I created an Azure Key Vault service, and gave the managed identity of the Data Factory the necessary access permissions. The linked service connecting to the on-prem sql server via the integration runtime does not work correctly with retrieving the secrets from Azure Key vault. I am pretty sure I have followed the documentation to set it up correctly.

Failed to get the secret from key vault, secretName: onprem, secretVersion: , vaultBaseUrl: https://crskpi-use2-dev-kv-fvrr.vault.azure.net/. The error message is: An error occurred while sending the request.
The remote server returned an error: (403) Forbidden..
Activity ID: 753732ed-9770-4242-9afa-bc560de5fa4d.

HarithaMaddi-MSFT 10,136 Reputation points

2020-11-16T07:32:03.67+00:00

Hi @Radhika Ganapathi Ndq ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

As you are using Self-Hosted Integration Runtime, please suggest if corresponding host IP address is added to the Key Vault under networking if "All networks" is not selected.

Hope this helps! Please let us know if issue persists after adding the same and share the data factory name, region and pipeline run id for further investigation.
Radhika Ganapathi Ndq 6 Reputation points

2020-11-16T18:35:58.473+00:00

Hi @HarithaMaddi-MSFT
In answer to your question - yes, I am using the self-hosted IR, it does not work even when Allowed access from all networks.
I have also tried with allowing access to only restricted networks and added the IP Address of the Integrated Runtime, allow trusted Microsoft Services to bypass this firewall to true.
The problem is still occurring.
HarithaMaddi-MSFT 10,136 Reputation points

2020-11-17T08:33:54.863+00:00

Hi @Radhika Ganapathi Ndq ,

Thanks for confirming the details. Can you please share data factory name, region and report id using below steps so that our team can access SHIR logs for further investigation.

In the integration runtime configuration manager, please click on diagnostics icon, reproduce the error and then click on ‘Send logs’.

You will be able to see report id, please share that with us to enable viewing logs.

Looking forward for your response!
HarithaMaddi-MSFT 10,136 Reputation points

2020-11-20T08:28:34.817+00:00

Hi @Radhika Ganapathi Ndq ,

We have not received a response from you. Please share the above details as requested for additional investigation and we will continue to engage with you on the query.
Radhika Ganapathi Ndq 6 Reputation points

2020-11-20T15:14:37.703+00:00

@HarithaMaddi-MSFT , I reproduced the error and sent the logs yesterday. Could you please check at your end?

Thanks
HarithaMaddi-MSFT 10,136 Reputation points

2020-11-24T14:14:18.537+00:00

Hi @Radhika Ganapathi Ndq ,

Apologies for the delay as I was on leave for couple of days. Can you please share the report id with us to access the logs.

Thanks!
Radhika Ganapathi Ndq 6 Reputation points

2020-11-24T18:19:06.507+00:00

@HarithaMaddi-MSFT , this is the reportid: e6fd732f-0cc6-43aa-9ea0-31fbb841bb33
HarithaMaddi-MSFT 10,136 Reputation points

2020-11-25T14:24:35.263+00:00

Thanks @Radhika Ganapathi Ndq for sharing the reportid with us. I will work with the product team and will get back to you with more insights.

Thanks for your patience!
HarithaMaddi-MSFT 10,136 Reputation points

2020-11-25T14:53:52.773+00:00

Hi @Radhika Ganapathi Ndq ,

I missed checking one more point from the document as below, if I follow this, I am able to access the secret from the ADF to connect to on-premise SQL server. Please confirm to proceed with further investigation.

Appreciate your support and looking forward for your response!
HarithaMaddi-MSFT 10,136 Reputation points

2020-11-26T15:45:00.827+00:00

Hi @Radhika Ganapathi Ndq ,

We have not received a response from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, let us know if above permission is granted and if issue persists, we will continue to engage with you on the issue.
Radhika Ganapathi Ndq 6 Reputation points

2020-11-27T04:58:14.587+00:00

Yes, confirming that the managed identity has been granted access (Get + List).
Issue still persists.
HarithaMaddi-MSFT 10,136 Reputation points

2020-11-27T06:38:50.497+00:00

Thanks @Radhika Ganapathi Ndq for confirming the details. I will proceed with further investigation with product team and will reach to you with further details.

Thanks for your patience!
HarithaMaddi-MSFT 10,136 Reputation points

2020-11-30T08:51:22.36+00:00

Hi @Radhika Ganapathi Ndq ,

Product team asked us to double check with you on the access permissions of managed identity of data factory on key vault and also if possible please reproduce error again and click on send logs to generate new report id as logs they are accessing seem to be little older ones.

Please let us know your thoughts.
Julio Arruda 1 Reputation point MVP

2020-12-01T18:45:42.99+00:00

I have the same problem.
HarithaMaddi-MSFT 10,136 Reputation points

2020-12-02T08:01:34.883+00:00

Hi @Radhika Ganapathi Ndq and @Julio Arruda ,

Can you double confirm that if using command line to get the secret from key vault on the machine where SHIR is installed is successful. Please use showazkeyvalutsecret and getazkeyvaultsecret on the approach.

Thanks for your support!
Julio Arruda 1 Reputation point MVP

2020-12-02T10:16:51.487+00:00

In my scenario, the configuration probably is correct, because if i try to use the default integration runtime, created from data factory, the process working. It's only fail if i try to use my own self hosted one.
HarithaMaddi-MSFT 10,136 Reputation points

2020-12-02T11:06:23.997+00:00

Hi @Julio Arruda ,

Thanks for sharing the update. That clearly explains the access issue from host network to the key vault, kindly suggest if host IP addresses are added to the access in Key Vault. To check the issue is in host network access, it was requested to check access from host machine with SHIR using CLI commands as above.

Please let us know your thoughts.
Radhika Ganapathi 1 Reputation point

2020-12-03T21:32:05.153+00:00

@HarithaMaddi-MSFT , the az commands came back with forbidden error - we are working internally with our network/firewall team to whitelist the key vault urls. Thanks.
KalyanChanumolu-MSFT 8,336 Reputation points

2020-12-04T11:50:58.06+00:00

@Radhika Ganapathi Thank you for the update.
Please let us know if you need any further assistance.

1 answer

José Faustino 1 Reputation point

2020-12-18T19:12:27.22+00:00

Hello.
We have faced exactly the same issue, and the solution was to whitelist the access (url) to the keyvault in our proxy.

In the documentation there is this note: create-self-hosted-integration-runtime

Tasks might fail in a self-hosted integration runtime that you installed on a Windows server for which FIPS-compliant encryption is enabled. To work around this problem, you have two options: store credentials/secret values in an Azure Key Vault or disable FIPS-compliant encryption on the server. To disable FIPS-compliant encryption, change the following registry subkey's value from 1 (enabled) to 0 (disabled): HKLM\System\CurrentControlSet\Control\Lsa\FIPSAlgorithmPolicy\Enabled. If you use the self-hosted integration runtime as a proxy for SSIS integration runtime, FIPS-compliant encryption can be enabled and will be used when moving data from on premises to Azure Blob Storage as a staging area
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Self Hosted Integration Runtime - Access Key Vault

1 answer

Your answer