Error in pipeline execution :- Job failed due to reason: Not started

Question

Error in pipeline execution :- Job failed due to reason: Not started

Anshulee Asthana 0

While executing a pipeline i am getting the following final error

User's image

All stages seem successful but there are no records in the sink. What is the issue?

User's image

KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee

2023-08-09T15:22:13.3033333+00:00
@Anshulee Asthana Welcome to Microsoft Q&A forum and thanks for reaching out here.

This seems very strange where everything looks green but end up with an error message.

I agree that the error message wasn't helpful to troubleshoot and identify the root cause of the problem, hence I am logging feedback to respective product owners to help improve the error message specific to the problem rathe being generic.

As per my analysis, it could be configuration issue or compute size related or partition related, it may also be product side issue too.

As the error wasn't helpful and it says configuration issue, I recommend playing around with below settings and see if any of them help resolve the issue.

Try to increase the Compute size of your data flow activity and see if that helps.

If above doesn't help, then try to play around with source and sink partition settings. If you have selected as single partition, then try to change settings to Use current partitioning or Set partitioning options and see if that helps.

Sometimes the issue could temporary, hence would also recommend implementing retry mechanism for your data flow activity.

If any of the above testing's didn't work, then I recommend filing a support ticket for deeper analysis.

Please do share the Support ticket number once it is opened so that we can track it internally.

Incase if you don't have a support plan, please let me know.

Let me know how it goes.

Thank you.
Anshulee Asthana 0 Reputation points

2023-08-10T10:29:34.0266667+00:00

@KranthiPakala-MSFT thank you for your recommendations. I realized this error is coming when the write to DB fails due to any error like Primary key violation etc. I would say the error could be better worded.

However when there is a a Violation like a Datatype mismatch of a PK violation then no rows are being written to the DB...i would have expected only the erroneous rows to be skipped?
Anshulee Asthana 0 Reputation points

2023-08-15T04:15:18.41+00:00

I am using postgres as the sink ans don't see these settings. Can I add them via script or will they simply not be supported?
KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee

2023-08-23T15:58:08.5033333+00:00

@Anshulee Asthana Thanks for the additional details. Since your sink is Azure DB for Postgres SQL I don't think the error row handling feature is available for it.

But what you can do is to use Assert data transformation to validate the assertion result and then do a conditional split and process only the good records to your sink.

OR

When an assertion fails, you can optionally direct those error rows to a file in Azure by using the "Errors" tab on the sink transformation. You will also have an option on the sink transformation to not output rows with assertion failures at all by ignoring error rows.

Hope this info helps.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

1 answer

Your answer

KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee

2023-08-09T15:22:13.3033333+00:00

@Anshulee Asthana Welcome to Microsoft Q&A forum and thanks for reaching out here.

This seems very strange where everything looks green but end up with an error message.

I agree that the error message wasn't helpful to troubleshoot and identify the root cause of the problem, hence I am logging feedback to respective product owners to help improve the error message specific to the problem rathe being generic.

As per my analysis, it could be configuration issue or compute size related or partition related, it may also be product side issue too.

As the error wasn't helpful and it says configuration issue, I recommend playing around with below settings and see if any of them help resolve the issue.

Try to increase the Compute size of your data flow activity and see if that helps.

If above doesn't help, then try to play around with source and sink partition settings. If you have selected as single partition, then try to change settings to Use current partitioning or Set partitioning options and see if that helps.

Sometimes the issue could temporary, hence would also recommend implementing retry mechanism for your data flow activity.

If any of the above testing's didn't work, then I recommend filing a support ticket for deeper analysis.

Please do share the Support ticket number once it is opened so that we can track it internally.

Incase if you don't have a support plan, please let me know.

Let me know how it goes.

Thank you.
Anshulee Asthana 0 Reputation points

2023-08-10T10:29:34.0266667+00:00

@KranthiPakala-MSFT thank you for your recommendations. I realized this error is coming when the write to DB fails due to any error like Primary key violation etc. I would say the error could be better worded.

However when there is a a Violation like a Datatype mismatch of a PK violation then no rows are being written to the DB...i would have expected only the erroneous rows to be skipped?
Anshulee Asthana 0 Reputation points

2023-08-15T04:15:18.41+00:00

I am using postgres as the sink ans don't see these settings. Can I add them via script or will they simply not be supported?
KranthiPakala-MSFT 46,632 Reputation points Microsoft Employee

2023-08-23T15:58:08.5033333+00:00

@Anshulee Asthana Thanks for the additional details. Since your sink is Azure DB for Postgres SQL I don't think the error row handling feature is available for it.

But what you can do is to use Assert data transformation to validate the assertion result and then do a conditional split and process only the good records to your sink.

OR

When an assertion fails, you can optionally direct those error rows to a file in Azure by using the "Errors" tab on the sink transformation. You will also have an option on the sink transformation to not output rows with assertion failures at all by ignoring error rows.

Hope this info helps.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Answer 1

@Anshulee Asthana Thanks for getting back and glad to know you were able to identify the root cause of the issue.

I agree that the error message could have been better described in the error message. I would highly encourage you to please share feedback directly from ADF Portal as below:

User's image

Since the issue is only with few records and you would like to continue the process on error by skipping errored rows, then you can utilize the Error row handling settings in Mapping data flow sink transformation. Using these settings your data will copied to database and all the bad records can be skipped and logged to storage file for fixing.

You can refer to this documentaion which has a demonstration video and in detailed information: Mapping data flow sink - Error row handling

User's image

In addition you can also refer to this document for additional information: Handle SQL truncation error rows in Data Factory mapping data flows

Hope this information helps.

Thank you

Share via

Error in pipeline execution :- Job failed due to reason: Not started

1 answer

Your answer