ADF - Error handling when using REST API as sink

Question

Hello everybody.

I am working on creating a data flow that basically post a data array into a rest api. This is working as expected. I want to increase the error handling properties when pushing data but apparently the sink is not as widely handled as those using SQL instead of REST. I wanted to induce an error for just one record of the 15 on the batch, just to be able to pinpoint what should be fixed, but all I can find is to regulate the input via "asserts" and then throw errors. What I want to do is to identify rows that the service may return as an http errors due to unforeseen conditions, let say one of the rows is faulty and the operation is terminated on the other end and we got an http 500 in return.

Is there anyway to trace to that error to an specific row to be reviewed and discard any error, or even let my trading partner know so they can track it on their end ? I understand the premise on data quality/data validation provided by assert, but even with precautions taken on what to be send, errors can still happen and I'd like to know about the culprit.

Thank you !

Answer 1

Hi @Rene Rugerio ,

Thankyou for using Microsoft Q&A platform and thankyou for posting your question here.

As I understand your query, you want to know if there is any way to error out a specific row out of the whole dataset using mapping dataflow. Please let me know if my understanding is incorrect.

You can make use of conditional split transformation to split the dataset into good rows or bad rows based on the condition you provide.

For example: If a column should hold date values and there are few records having invalid date then you can discard them using expression : isnull(toDate(ColumnName,'dd-MMM-yyyy')) . You can formulate your query based on the expected value for the column

For more details, kindly check out this video: Handle Error Rows in Data Factory Mapping Data Flows

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators