How to ignore copy activity failure rows in sinking to Cosmos DB in ADF

Peter Ma 206 Reputation points Microsoft Employee
2021-07-15T07:35:09.787+00:00

My cosmos db container has a unique constrain for columns.
There maybe some duplicate rows in source file in blob storage.
How can I set ignore the failure rows in copy activity when sinking to Cosmos DB.

By the way "Fault tolerance" doesn't work in this scenario!

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,509 questions
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2021-07-22T10:54:27.337+00:00

    Hi anonymous user-0048 ,

    Thank you for posting in Microsoft Q&A Platform.

    There might be two possible causes for this failure. Please check details below.

    Root Cause:

    • Cause 1: If you use Insert as the write behavior, this error means that your source data has rows or objects with same ID.
    • Cause 2: If you use Upsert as the write behavior and you set another unique key to the container, this error means that your source data has rows or objects with different IDs but the same value for the defined unique key.

    Resolution:

    • For cause 1, set Upsert as the write behavior.
    • For cause 2, make sure that each document has a different value for the defined unique key.

    Please check below link for more details.
    https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-guide#error-message-unique-index-constraint-violation

    Hope this will help. Thank you.

    ----------------------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.

0 additional answers

Sort by: Most helpful