ClientIPNotAuthorized Error When Deploying Cosmos Lens Jobs via ARM Template on Azure DevOps

Question

I’m working on building an agent to migrate Lens jobs to Azure Data Factory by deploying ARM templates through Azure DevOps pipelines.

The deployment works successfully for Kusto-type Lens jobs, but for Cosmos-type Lens jobs, I encounter this error:

ClientIPNotAuthorized: Client IP not authorized to access the API. 
Please ensure you are on corpnet, or that your IP is on an allowlist for the activities in your pipeline.

To address this, I am set up a self-hosted agent VM (with a static public IP) to run my pipelines so that I can allowlist that IP in my ADLS Gen2 Account.

I also followed the steps provided on the Q&A platform: ClientIPNotAuthorized error when deploying Cosmos Lens jobs to ADF via ARM template - Microsoft Q&A. However, this did not help resolve the issue.

Could you please help me identify what might be missing or incorrectly configured?

Answer 1

We set up a self-hosted agent VM (with a static public IP) to run my pipelines. The static IP was allowlisted in all data centers in Azure Public and this resolved the issue.

Answer 2

Hello Rithika Shankar, I am assuming that this cosmos-type lens jobs contains scope activities which eventually being called via ADF and the same pipeline is failing? Also, have you tried connecting MSFT_AzVPN first, before running the pipeline?

The core issue which I think is, your self-hosted agent, even with a static public IP, is running on the public internet, not on Corpnet. Therefore, when your Azure DevOps pipeline attempts to deploy the ARM template, the ADF management endpoint correctly identifies the call as non-Corpnet and blocks it, returning the ClientIPNotAuthorized error.

To resolve this, your deployment agent must run from a machine that is on the Microsoft corporate network.

1. Use a Corpnet-Hosted Agent: The most reliable solution is to configure your Azure DevOps pipeline to use a self-hosted agent that is running on a virtual machine explicitly provisioned within the Microsoft Corpnet.
2. Please refer to Microsoft docs for guidance (Need to connect AzVPN for this) on setting up an Azure DevOps agent on a Corpnet virtual machine for CI/CD purposes.
3. Use a Remote Desktop connection within Microsoft's network (as mentioned in the documentation) and execute the deployment from there

Whitelisting the IP in other services like Cosmos DB or ADLS Gen2 will not help because the block is happening at the Azure Data Factory control plane before any connection to those data services is even attempted.

Please note that VPN connections are NOT considered to be within Corpnet. If you need to access IP restricted data factories remotely, you may be able to route traffic to management.azure.com over your corpnet VPN connection. Keep in mind this can fix ADF access but cause other issues so proceed with caution: Open an administrative command prompt Run 'nslookup management.azure.com ' and make note of the IP address listed Run 'ipconfig' to get your MSFTVPN IP address Add a route rule to force all traffic to management.azure.com via VPN with 'route -p add <<nslookupip>> mask 255.255.255.255 <<Your MSFTVP IPv4 address>>' Test access to Data Factory in an InPrivate window to avoid caching issues If this causes any issues, you can undo this by running 'route -p delete <<nslookupip>>' Alternatively, you can get reliable access to ADF via a Remote Desktop. You can request one here: https://microsoft.sharepoint.com/sites/Security_Tools_Services/SitePages/WindowsVirtualDesktop/CSEO-Windows-Virtual-Desktop-Pilot.aspx

Let me know if this works for you.

Thanks again for reaching us out.

-Pratyush