Comsos DB Mongo - error copying from one collection to another

Bhagyashri Nivdunge 1 Reputation point Microsoft Employee
2023-05-05T16:55:35.1333333+00:00 
error while copying - updated the collection throughput and made other changes to pipeline, reduced batch size, changed to upsert from insert but some data gets copied and fails for some data


Operation on target Copy_ntz failed: Failure happened on 'Sink' side. ErrorCode=MongoDbFailedWithMongoDbServerError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=>The operation failed with server error. ConnectionId: '{ ServerId : { ClusterId : 1, EndPoint : "Unspecified/dscm-ppe2-northcentralus.mongo.cosmos.azure.com:10255" }, LocalValue : 4, ServerValue : "1568641427" }'.,Source=Microsoft.DataTransfer.Runtime.MongoDbAtlasConnector,''Type=MongoDB.Driver.MongoBulkWriteException`1[[MongoDB.Bson.BsonDocument, MongoDB.Bson, Version=2.10.4.0, Culture=neutral, PublicKeyToken=15b1115599983c50]],Message=A bulk write operation resulted in one or more errors.
  Performing an update would modify the immutable field 'ShardKey',Source=MongoDB.Driver,'
ErrorCode=MongoDbFailedWithMongoDbServerError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=>The operation failed with server error. ConnectionId: '{ ServerId : { ClusterId : 1, EndPoint : "Unspecified/dscm-ppe2-northcentralus.mongo.cosmos.azure.com:10255" }, LocalValue : 4, ServerValue : "625745021" }'.,Source=Microsoft.DataTransfer.Runtime.MongoDbAtlasConnector,''Type=MongoDB.Driver.MongoBulkWriteException`1[[MongoDB.Bson.BsonDocument, MongoDB.Bson, Version=2.10.4.0, Culture=neutral, PublicKeyToken=15b1115599983c50]],Message=A bulk write operation resulted in one or more errors.
  E11000 duplicate key error collection: JitResourcesV1.JitPolicySetV1. Failed _id or unique index constraint.
  E11000 duplicate key error collection: JitResourcesV1.JitPolicySetV1. Failed _id or unique index constraint.
  E11000 duplicate key error collection: JitResourcesV1.JitPolicySetV1. Failed _id or unique index constraint.
  E11000 duplicate key error collection:
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
970 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,448 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Rahul Randive 8,521 Reputation points Microsoft Employee
    2023-05-06T13:22:18.0233333+00:00

    Thanks for your question. 

    Kindly review the responses provided in the two threads below to see if they are applicable to your situation. Additionally, please take note of the comments as they may offer further insights.

     https://learn.microsoft.com/en-us/answers/questions/845141/azure-cosmos-api-for-mongodb-to-other-using-data-f

     https://learn.microsoft.com/en-us/answers/questions/1163177/adf-copy-data-to-cosmosdb-producing-errors

     Thank you!

    0 comments
  2. BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee
    2023-05-11T19:48:16.1+00:00

    Thanks, GeethaThatipatri-MSFT, Rahul Randive

    Hello @Bhagyashri Nivdunge

    Can you please check if you are inserting or creating a document with null values for the Unique key Properties. The insert would fail with "duplicate key error collection" if multiple documents are passed with null.

    Example: Unique Key on FirstName and LastName Property

    {

    'firstName' : 'My first name',

    'lastName': 'My lastName',

    }

    The null is passed if the document does not have any property defined and would lead to Duplicate Key Error

    Work arounds: The workaround is to pass a unique non meaningful values for Unique Properties. You can use GUID to avoid the duplicate error.

    https://learn.microsoft.com/en-us/azure/cosmos-db/unique-keys#define-a-unique-key

    0 comments