ADF Pipeline Failing copying data from Azure SQL DB to Azure Databricks Delta Lake

Question

I'm trying to copy data from Azure SQL DB to Azure Databricks Delta Lake using a staging blob. The data is getting copied from DB to staging location, but when its trying to copy the data from staging location to Azure Databricks Delta Lake its throwing the below mentioned error. The same Pipeline is working on a different ADF.

Has anyone come across this particular error? if yes,It would be very helpful if you could share how you resolved this error.

Answer 1

Hello @MDFrazKhan-7900 ,

Thanks for the question and using MS Q&A platform.

When you say, "The same Pipeline is working on a different ADF." I am assuming that they are querying different set of data, correct? The reason I ask that is because from the error it looks to be one of the columns lengths on the delta table is not big enough for the data coming in. I think you can try two things

1. Enable fault tolerance and see if that helps . If the pipeline succeeds, revisit the logs and you will find which data row are the culprits ( it will log that)
2. Since in your case its SQL as the source , you can try to take a query the table and use the substring function eg .
   Select substring(columnName , 0,100) from table

This will pull only 100 characters of the column columnName

Hope this helps.

Thanks

Himanshu

Please accept as "Yes" if the answer provided is useful , so that you can help others in the community looking for remediation for similar issues.