Events
31 Mar, 11 pm - 2 Apr, 11 pm
The biggest Fabric, Power BI, and SQL learning event. March 31 – April 2. Use code FABINSIDER to save $400.
Register todayTroubleshoot Azure Data Factory and Synapse Analytics security and access control issues
APPLIES TO: 
Azure Data Factory
Azure Synapse Analytics
Tip
Try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting. Learn how to start a new trial for free!
This article explores common troubleshooting methods for security and access control in Azure Data Factory and Synapse Analytics pipelines.
Various error messages might be returned when connectivity issues occur in the source or sink datastore.
The problem is usually caused by one of the following factors:
The proxy setting in the self-hosted integration runtime (IR) node, if you're using a self-hosted IR.
The firewall setting in the self-hosted IR node, if you're using a self-hosted IR.
The firewall setting in the cloud datastore.
To ensure that this is a connectivity issue, check the following points:
- The error is thrown from the source or sink connectors.
- The failure is at the start of the copy activity.
- The failure is consistent for Azure IR or the self-hosted IR with one node, because it could be a random failure in a multiple-node self-hosted IR if only some of the nodes have the issue.
If you're using a self-hosted IR, check your proxy, firewall, and network settings, because connecting to the same datastore could succeed if you're using an Azure IR. To troubleshoot this scenario, see:
If you're using an Azure IR, try to disable the firewall setting of the datastore. This approach can resolve the issues in the following two situations:
- Azure IR IP addresses aren't in the allowlist.
- The Allow trusted Microsoft services to access this storage account feature is turned off for Azure Blob Storage and Azure Data Lake Storage Gen 2.
- The Allow access to Azure services setting isn't enabled for Azure Data Lake Storage Gen1.
If none of the preceding methods works, contact Microsoft for help.
You created managed private endpoint from ADF and obtained an approved private endpoint. But, after deleting or rejecting the private endpoint later, the managed private endpoint in ADF still persists to exist and shows "Approved".
Currently, ADF stops pulling private end point status after it's approved. Hence the status shown in ADF is stale.
You should delete the managed private end point in ADF once existing private endpoints are rejected/deleted from source/sink datasets.
After you disable public network access for the service, the self-hosted integration runtime throws following errors: The Authentication key is invalid or empty. or Cannot connect to the data factory. Please check whether the factory has enabled public network access or the machine is hosted in a approved private endpoint Virtual Network.
The problem is most likely caused by a Domain Name System (DNS) resolution issue, because disabling public connectivity and establishing a private endpoint prevents reconnection.
To verify whether the service's fully qualified domain name (FQDN) is resolved to the public IP address, do the following:
Confirm that you've created the Azure virtual machine (VM) in the same virtual network as the service's private endpoint.
Run PsPing and Ping from the Azure VM to the service FQDN:
psping.exe <dataFactoryName>.<region>.datafactory.azure.net:443ping <dataFactoryName>.<region>.datafactory.azure.netNote
You must specify a port for the PsPing command. Port 443 is shown here but is not required.
Check to see whether both commands resolve to an Azure Data Factory public IP that's based on a specified region. The IP should be in the following format:
xxx.xxx.xxx.0
To resolve the issue, do the following:
As option, we would like to suggest you to manually add a "Virtual Network link" under the "Private link DNS Zone" for the service. For details, refer to the Azure Private Link article. The instruction is for configuring the private DNS zone or custom DNS server to resolve the service FQDN to a private IP address.
However, if you don't want to configure the private DNS zone or custom DNS server, try the following temporary solution:
Change the host file in Windows, and map the private IP (the service's private endpoint) to the service FQDN.
In the Azure VM, go to
C:\Windows\System32\drivers\etc, and then open the host file in Notepad. Add the line that maps the private IP to the FQDN at the end of the file, and save the change.
Rerun the same commands as in the preceding verification steps to check the response, which should contain the private IP.
Re-register the self-hosted integration runtime, and the issue should be resolved.
You're unable to register the IR authentication key on the self-hosted VM because the private link is enabled. You receive the following error message:
"Failed to get service token from ADF service with key *************** and time cost is: 0.1250079 second, the error code is: InvalidGatewayKey, activityId is: XXXXXXX and detailed error message is Client IP address isn't valid private ip Cause Data factory couldn't access the public network thereby not able to reach out to the cloud to make the successful connection."
The issue could be caused by the VM in which you're trying to install the self-hosted IR. To connect to the cloud, ensure that public network access is enabled.
Solution 1
To resolve the issue, do the following:
Go to the Factories - Update page.
At the upper right, select the Try it button.
Under Parameters, complete the required information.
Under Body, paste the following property:
{ "tags": { "publicNetworkAccess":"Enabled" } }Select Run to run the function.
Under Parameters, complete the required information.
Under Body, paste the following property:
{ "tags": { "publicNetworkAccess":"Enabled" } }Select Run to run the function.
Confirm that Response Code: 200 is displayed. The property you pasted should be displayed in the JSON definition as well.
Add the IR authentication key again in the integration runtime.
Solution 2
To resolve the issue, go to Azure Private Link.
Try to enable public network access on the user interface, as shown in the following screenshot:
Both Azure Resource Manager and the service are using the same private zone creating a potential conflict on customer's private DNS with a scenario where the Azure Resource Manager records won't be found.
- Find Private DNS zones privatelink.azure.com in Azure portal.
- Check if there's an A record adf.
- Go to Virtual network links, delete all records.
- Navigate to your service in Azure portal and recreate the private endpoint for the portal.
- Go back to Private DNS zones, and check if there's a new private DNS zone privatelink.adf.azure.com.
When copying data with Azure Blob Storage account public access, pipeline runs randomly fail with following error.
For example: The Azure Blob Storage sink was using Azure IR (public, not Managed virtual network) and the Azure SQL Database source was using the Managed virtual network IR. Or source/sink use Managed virtual network IR only with storage public access.
<LogProperties><Text>Invoke callback url with req: "ErrorCode=AzureBlobFailedToCreateContainer,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Unable to create Azure Blob container. Endpoint: XXXXXXX/, Container Name: test.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.WindowsAzure.Storage.StorageException,Message=Unable to connect to the remote server,Source=Microsoft.WindowsAzure.Storage,''Type=System.Net.WebException,Message=Unable to connect to the remote server,Source=System,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond public ip:443,Source=System,'","Details":null}}</Text></LogProperties>.
The service might still use Managed virtual network IR, but you could encounter such error because the public endpoint to Azure Blob Storage in Managed virtual network isn't reliable based on the testing result, and Azure Blob Storage and Azure Data Lake Gen2 aren't supported to be connected through public endpoint from the service's Managed Virtual Network according to Managed virtual network & managed private endpoints.
- Having private endpoint enabled on the source and also the sink side when using the Managed virtual network IR.
- If you still want to use the public endpoint, you can switch to public IR only instead of using the Managed virtual network IR for the source and the sink. Even if you switch back to public IR, the service may still use the Managed virtual network IR if the Managed virtual network IR is still there.
Internal error while trying to Delete a data factory or Synapse workspace with Customer Managed Key (CMK) and User Assigned Managed Identity (UA-MI)
{\"error\":{\"code\":\"InternalError\",\"message\":\"Internal error has occurred.\"}}
If you're performing any operations related to CMK, you should complete all operations related to the service first, and then external operations (like Managed Identities or Key Vault operations). For example, if you want to delete all resources, you need to delete the service instance first, and then delete the key vault. If you delete the key vault first, this error occurs since the service can't read the required objects anymore, and it won't be able to validate if deletion is possible or not.
There are three possible ways to solve the issue. They are as follows:
You revoked the service's access to Key vault where the CMK key was stored. You can reassign access to the following permissions: Get, Unwrap Key, and Wrap Key. These permissions are required to enable customer-managed keys. Refer to Grant access to customer-managed keys. Once the permission is provided, you should be able to delete the service.
Customer deleted Key Vault / CMK before deleting the service. CMK in the service should have "Soft Delete" enabled and "Purge Protect" enabled which has default retention policy of 90 days. You can restore the deleted key.
Review Recover deleted Key and Deleted Key ValueUser Assigned Managed Identity (UA-MI) was deleted before the service. You can recover from this by using REST API calls. You can do this in an http client of your choice in any programming language. If you have not anything already set up for REST API calls with Azure authentication, the easiest way to do this 'd be by using Fiddler. Follow following steps.
Make a GET call using Method: GET Url like
https://management.azure.com/subscriptions/YourSubscription/resourcegroups/YourResourceGroup/providers/Microsoft.DataFactory/factories/YourFactoryName?api-version=2018-06-01You need to create a new User Managed Identity with a different Name (the same name might work, but just to be sure, it's safer to use a different name than the one in the GET response)
Modify the encryption.identity property and identity.userassignedidentities to point to the newly created managed identity. Remove the clientId and principalId from the userAssignedIdentity object.
Make a PUT call to the same url passing the new body. It's important that you're passing whatever you got in the GET response, and only modify the identity. Otherwise they would override other settings unintentionally.
After the call succeeds, you'll be able to see the entities again and retry deleting.
You might notice other data factories (on different tenants) as you're attempting to share the self-hosted IR from the UI, but you can't share it across data factories that are on different tenants.
The self-hosted IR can't be shared across tenants.
For more help with troubleshooting, try the following resources:
Additional resources
Training
Learning path
Use advance techniques in canvas apps to perform custom updates and optimization - Training
Use advance techniques in canvas apps to perform custom updates and optimization
Certification
Microsoft Certified: Azure Data Engineer Associate - Certifications
Demonstrate understanding of common data engineering tasks to implement and manage data engineering workloads on Microsoft Azure, using a number of Azure services.