primary key in adf pipeline

Vineet S 325

ADF package is failing due to Primarky and not null column available in target table due to which pipiline is failing how to handle such situation in adf..

Please prrimary key is only creating unique constraint

Vineet S 325 Reputation points

2024-07-06T06:26:15.11+00:00

https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-sql-data-warehouse?tabs=data-factory#error-row-handling

on error continue option is not available in copy activity
Harishga 5,750 Reputation points Microsoft Vendor

2024-07-08T07:59:00.1166667+00:00
Hi @Vineet S
Welcome to Microsoft Q&A platform and thanks for posting your question here.

It looks like your Azure Data Factory pipeline is having problems because your target table has primary key and NOT NULL constraints. Since you can’t use the “on error continue” option in the Copy activity, here are some other ways to deal with this:

Use a Data Flow activity to process your data first. You can split the data to separate rows that break the rules (like duplicate keys or missing values) from the valid rows. This way, you can handle the bad rows separately and only load the good ones into your target table.

Check your source data before copying it to make sure it meets the target table’s rules. You can do this with a stored procedure or a Data Flow activity to clean and validate the data.

Turn on logging to capture details about the rows that cause errors. This will help you find and fix the problematic data. You can log these errors to a separate table or file for further analysis.

Even though you can’t use “on error continue” in the Copy activity, you can enable fault tolerance to skip rows that cause problems and continue with the rest of the data. This can be set up in the Copy activity settings.

Reference
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-fault-tolerance

I hope this information helps you. Let me know if you have any further questions or concerns.
Harishga 5,750 Reputation points Microsoft Vendor

2024-07-09T10:22:38.8966667+00:00

Hi @Vineet S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Harishga 5,750 Reputation points Microsoft Vendor

2024-07-11T03:59:48.87+00:00

@Vineet S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Vineet S 325 Reputation points

2024-07-11T17:53:44.4533333+00:00

Even though you can’t use “on error continue” in the Copy activity, you can enable fault tolerance to skip rows that cause problems and continue with the rest of the data. This can be set up in the Copy activity setting... This option is not available and still pipeline is failing due less or more column in source

Share via

primary key in adf pipeline