Azure synapse mapping dataflow Sink [error]

Question

Azure synapse mapping dataflow Sink [error]

Shreyash Choudhary 126

Hi, on sink side i am using azure synapse analytics sql dataset but , on sink errors tab i have used the setting but not able to see log file when pipeline got failed?

i refereed below sink error setting, please help with the soluton

https://learn.microsoft.com/en-us/answers/questions/284259/not-able-to-see-the-error-logging-in-sink-transfor

3 answers

Your answer

Answer 1

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hi Shreyash Choudhary,

Thank you for posting query in Microsoft Q&A Platform.

If error is specific to only data getting inserting part due to some bad data like that, then in that case below settings will work and you see that bad data in a file. But error is not specific to data it may be some other reason, in that case below settings will not come in pitch.

Below are settings which we can use for Azure Synapse Analytics connector type in Sink transformation. User's image

Hope this helps. If not, kindly share more details on exact error message and implementation, so that I can understand more and try to repro to help with resolution.

Please consider hitting Accept Answer button. Accepted answers help community as well.

Shreyash Choudhary 126 Reputation points

2023-03-01T05:53:21.6866667+00:00

Hi ShaikMaheer-MSFT ,

how to debug it? , i am passing a string value in date datatype column and its inserting null in it? not getting any output log file genrated..
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-03-01T16:46:37.4733333+00:00

Hi Shreyash Choudhary, sorry not able to understand clearly. Is data is getting inserting correctly in SQL table? Or any error you are seeing? What is the format of date in string which you are passing? Are you converting datatype of that string to date type using derived column from sink? Kindly share your implementation screenshots along with sample data and error details to understand more on this. Thank you.
Shreyash Choudhary 126 Reputation points

2023-03-03T06:13:49.98+00:00

Hi ShaikMaheer-MSFT ,

i am using azure synapse analytics database , for debug purpose i have made changes in database table schema like varchar 1 and incoming data has bigger length , when tried with above mentioned settings it didnt worked out.

please tell any othe rwa to debug it to make sure its logging error rows in folder, also it will be good if you can make video on it , it will be helpful.

Answer 2

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hi Shreyash Choudhary,

I reproduced your case, yes it seems when we use Azure synapse SQL table as sink then while writing error rows to storage, we are ending up with below error. I am checking more on this with internal team and I will share updates with you soon.

As a work around, I would suggest you use Assert transformation for your testing and debug for purpose. check this video to understand about assert transformation.

User's image

Hope this helps.

Please consider hitting Accept Answer button. Accepted answers help community as well.

Shreyash Choudhary 126 Reputation points

2023-03-07T14:00:38.4966667+00:00

Hi @ShaikMaheer-MSFT ,

thanks for the work around, but i have 200 columns coming from rest Api can't use assert I think, I need to log errors rows into a db table with error message, if you have some solution with assert row please share, i have not used assert, can't think of assert much.

please provide any other solution or any video or assert solution which can solve my problem which i can use.

Thanks
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-03-14T14:43:18.2866667+00:00

Hi Shreyash Choudhary,

Below video has sample use case of Assert transformation. You can check it and try to use according to your use case.

Assert Transformation in Mapping Data Flows in Azure Data Factory

You can consider using Conditional split transformation as well. Check below video.

Handle Error Rows in Data Factory Mapping Data Flows

Hope this helps.

Please consider hitting Accept Answer. Accepted answers help community as well.

Answer 3

Prashant Shukla 0

Any update on above error, i am getting same

Share via

Azure synapse mapping dataflow Sink [error]

3 answers

Your answer