Comsos DB Mongo - error copying from one collection to another

Question

error while copying - updated the collection throughput and made other changes to pipeline, reduced batch size, changed to upsert from insert but some data gets copied and fails for some data


Operation on target Copy_ntz failed: Failure happened on 'Sink' side. ErrorCode=MongoDbFailedWithMongoDbServerError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=>The operation failed with server error. ConnectionId: '{ ServerId : { ClusterId : 1, EndPoint : "Unspecified/dscm-ppe2-northcentralus.mongo.cosmos.azure.com:10255" }, LocalValue : 4, ServerValue : "1568641427" }'.,Source=Microsoft.DataTransfer.Runtime.MongoDbAtlasConnector,''Type=MongoDB.Driver.MongoBulkWriteException`1[[MongoDB.Bson.BsonDocument, MongoDB.Bson, Version=2.10.4.0, Culture=neutral, PublicKeyToken=15b1115599983c50]],Message=A bulk write operation resulted in one or more errors.
  Performing an update would modify the immutable field 'ShardKey',Source=MongoDB.Driver,'
ErrorCode=MongoDbFailedWithMongoDbServerError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=>The operation failed with server error. ConnectionId: '{ ServerId : { ClusterId : 1, EndPoint : "Unspecified/dscm-ppe2-northcentralus.mongo.cosmos.azure.com:10255" }, LocalValue : 4, ServerValue : "625745021" }'.,Source=Microsoft.DataTransfer.Runtime.MongoDbAtlasConnector,''Type=MongoDB.Driver.MongoBulkWriteException`1[[MongoDB.Bson.BsonDocument, MongoDB.Bson, Version=2.10.4.0, Culture=neutral, PublicKeyToken=15b1115599983c50]],Message=A bulk write operation resulted in one or more errors.
  E11000 duplicate key error collection: JitResourcesV1.JitPolicySetV1. Failed _id or unique index constraint.
  E11000 duplicate key error collection: JitResourcesV1.JitPolicySetV1. Failed _id or unique index constraint.
  E11000 duplicate key error collection: JitResourcesV1.JitPolicySetV1. Failed _id or unique index constraint.
  E11000 duplicate key error collection: 

Answer 1

Thanks for your question. 

Kindly review the responses provided in the two threads below to see if they are applicable to your situation. Additionally, please take note of the comments as they may offer further insights.

 https://learn.microsoft.com/en-us/answers/questions/845141/azure-cosmos-api-for-mongodb-to-other-using-data-f

 https://learn.microsoft.com/en-us/answers/questions/1163177/adf-copy-data-to-cosmosdb-producing-errors

 Thank you!

Answer 2

Thanks, GeethaThatipatri-MSFT, Rahul Randive

Hello @Bhagyashri Nivdunge

Can you please check if you are inserting or creating a document with null values for the Unique key Properties. The insert would fail with "duplicate key error collection" if multiple documents are passed with null.

Example: Unique Key on FirstName and LastName Property

{

'firstName' : 'My first name',

'lastName': 'My lastName',

}

The null is passed if the document does not have any property defined and would lead to Duplicate Key Error

Work arounds: The workaround is to pass a unique non meaningful values for Unique Properties. You can use GUID to avoid the duplicate error.

https://learn.microsoft.com/en-us/azure/cosmos-db/unique-keys#define-a-unique-key