Why does updating the firewall IP configuration in Azure Cosmos DB result in slow performance?

Vlada Jerkovic 15 Reputation points
2024-02-04T10:36:30.9933333+00:00

When utilizing GitHub Actions to update the networking firewall settings of CosmosDB, the process of adding a single IP address requires approximately 8 minutes to complete. Similarly, the removal of an IP address from the firewall settings also takes an equivalent duration of 8 minutes. I'm expecting much faster execution. The same approach for MSSQL server it takes seconds to update firewall IPs. GitHubActionCosmosDb

It just executes following command: Example CURRENT_IPS: 0.0.0.0,1.2.3.4,2.3.4.5 Example DESIRED_IPS: 0.0.0.0,1.2.3.4,2.3.4.5,9.8.7.6

- name: Azure CLI Command - Adding Worker IP into CosmosDB
  run: |
      # Adding current IP from worker to CosmoDB
      echo "workerIP=$workerIP"
      DESIRED_IP=$workerIP
      export CURRENT_IPS=$(az cosmosdb list | jq -r '.[0].ipRules | .[] | .ipAddressOrRange' | paste -sd "," -)
      echo "CURRENT_IPS=$CURRENT_IPS" >> $GITHUB_ENV
      DESIRED_IPS=$CURRENT_IPS,$DESIRED_IP
      echo "DESIRED_IPS: $DESIRED_IPS"
      az cosmosdb update -n $VAR_AZURE_COSMOS_DB_ACCOUNT_NAME -g $VAR_AZURE_RESOURCE_GROUP --ip-range-filter "$DESIRED_IPS"

- name: Azure CLI Command - Removing Worker IP from CosmosDB
  run: |
      # Rollback old/original IP list to CosmoDB
      echo "CURRENT_IPS: ${{ env.CURRENT_IPS }}"
      az cosmosdb update -n $VAR_AZURE_COSMOS_DB_ACCOUNT_NAME -g $VAR_AZURE_RESOURCE_GROUP --ip-range-filter "${{ env.CURRENT_IPS }}"
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,672 questions
{count} votes

4 answers

Sort by: Most helpful
  1. Vlada Jerkovic 15 Reputation points
    2024-02-14T20:10:53.1833333+00:00

    1). In CLI, you have used az cosmos db list, are you trying to perform firewall change for all Cosmos DBs together? If the number of DBs are more, then it might be slowed. *Try to apply filter in Command for lesser DBs and see if this improves.
    *
    I have defined inside teraform some IPs we need to have access into for CosmosDB (just one DB) with 3 Containers. Since every GitHub worker is assigned a different public IP upon execution, we need to include the worker's IP and remove it afterward, leaving the previous ones intact. This is the reason why we are using following command to get a list of configured IPs.

    CURRENT_IPS=$(az cosmosdb list | jq -r '.[0].ipRules | .[] | .ipAddressOrRange' | paste -sd "," -)
    

    Then we concatenate with configured CURRENT_IPs + WorkerIP and do update of Firewall policy. After task is finished we are removing WorkerIp.

    ---2). Please try executing same command from Azure CLI Bash and see if takes same time?
    I can confirm that it takes also 8-9 minutes from Azure CLI Bash and 8-9 minutes more to remove IP

    Adding IP: 8-9 minutes

    az cosmosdb update -n "AzureCosmosDBaccount" -g "ResourceGroup" --ip-range-filter "0.0.0.0,79.XXX.XXX.98"
    

    Removing IP: 8-9 minutes

    az cosmosdb update -n "AzureCosmosDBaccount" -g "ResourceGroup" --ip-range-filter "0.0.0.0"
    
    
    1 person found this answer helpful.

  2. Brian Williams 5 Reputation points
    2024-04-26T01:30:18.3533333+00:00

    I am experiencing the same when building/managing a Cosmos DB account in Pulumi. It seems to be more concentrated to either an issue with the Azure Cosmos DB Resource Provider REST API or something behind it on Microsoft's side that causes this issue no matter what. For me, it is a new implementation. I have not added any containers or data yet.

    I agree with the security implications previously mentioned. I could not imagine doing this in a pipeline like GH Actions where I may be paying for pipeline minutes.

    1 person found this answer helpful.

  3. ShaktiSingh-MSFT 15,421 Reputation points
    2024-02-05T09:34:37.8+00:00

    Hi Vlada Jerkovic •,

    Welcome to Microsoft Q&A forum.

    As I understand, you are updating the firewall IP configuration in Azure Cosmos DB result via CLI command from GitHub actions resulting in slowness.

    Thanks for sharing the CLI snippet.

    Could you please help us letting know below pointers:

    1). In CLI, you have used az cosmos db list, are you trying to perform firewall change for all Cosmos DBs together? If the number of DBs are more, then it might be slowed.

    Try to apply filter in Command for lesser DBs and see if this improves.

    2). Please try executing same command from Azure CLI Bash and see if takes same time? User's image

    Awaiting your reply. Thanks

    0 comments No comments

  4. Vlada Jerkovic 15 Reputation points
    2024-02-21T09:57:13.2133333+00:00

    Also to add that this issue represents a significant security vulnerability with CosmosDb: if a permitted IP is compromised, attackers can exploit this to exfiltrate data. The process to block and update an IP takes about 8 minutes—a window during which considerable harm can occur. Many organizations employ detection systems and scripts to immediately quarantine suspicious IPs, highlighting the urgency of a more prompt response mechanism. It's crucial that firewall adjustments have the capability for instant implementation.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.