TimeoutException when copying from Cosmod DB mongo to azure sql

Question

Hi,
We have an ADF pipeline that simply copies data from a CosmosDB (Mongo API) to an Azure SQL database.
The CosmosDB linked service is set up with Azure Key vault, and it works fine.
Most of the times, the pipeline runs smoothly.
But recently, sometimes the pipeline fails due to timeout exception. This occurs maybe 1/20 times - even in the middle of the night when the database is basically idle. I've checked the throughput around the time when the pipeline fails, and we are not exceeding the resource limit.

This is the error message:
"Message": "Failure happened on 'Source' side. ErrorCode=MongoDbFailedWithMongoDbServerError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=>
The operation failed with server error. ConnectionId: '{ ServerId : { ClusterId : 1, EndPoint : \"Unspecified/xxxx.mongo.cosmos.azure.com:yyyy\" }, LocalValue : 3 }'.,Source=Microsoft.DataTransfer.Runtime.MongoDbV2Connector,''Type=MongoDB.Driver.MongoExecutionTimeoutException,Message=Operation exceeded time limit.,Source=MongoDB.Driver.Core,'"

Is there a way to extend the timeout maybe?

Answer 1

Hi,
I ended up contacting the Cosmos support team.
The issue was due to over-consumption of resources in their Cosmos DB clusters:

"The issue is caused by resource over-consumption of system activities in one of the Cosmos DB clusters in the West Europe region.
This particular cluster was built with a custom topology that is designed to scale to handle the maximum load required by a particular customer workload.
There is also a configuration that restricts the placement of new database accounts in this cluster. In this case, the placement constraint configuration was set incorrectly, which resulted in large number of database accounts landing in this cluster.
As the number of database accounts in the cluster increases overtime, it reached a tipping point where system activities involved in routine management of database accounts spiked to consume 100% of the CPU cycles in a subset of the nodes in the cluster. As result, customers requests that reached those nodes either failed or experienced high latency in responses."

We have not experienced the timeout exception since this issue was resolved.

Sorry for my late reply, and thank you for taking your time with this.

Answer 2

Hi @LinLinea - Although this specific ADF error is not detailed in the Azure Cosmos DB section of the Troubleshooting Guide for Azure Data Factory Connectors, it is indirectly discussed in the Troubleshoot common issues in Azure Cosmos DB's API for MongoDB document as a specific error: ExceededTimeLimit with the description: The request has exceeded the timeout of 60 seconds of execution.

To resolve this, please increase your RU/s or narrow the scope of your query down to smaller (logical) blocks or set of records as per the recommended solution for this error code (50):

There can be many causes for this error. One of the causes is when the current allocated request units capacity is not sufficient to complete the request. This can be solved by increasing the request units of that collection or database. In other cases, this error can be worked-around by splitting a large request into smaller ones.

In addition to the above, if you could please provide the failed pipeline run ID to us we can then provide a deeper analysis to identify root cause in your specific instance. (Note: There is now a Private Chat feature available to you to exchange information in this forum thread if you do not wish to share information publicly).