data load issue from sfdc to azure blob storage

Question

Hi,
I have developed ADF pipeline to load data from sfdc to azure blob stage container.

I have total 24 objects from SFDC and loading into azure storage daily. But one table facing issue with alternatively. see below error. I have connected with sfdc team and there is no changes happened from their end. Pipeline runs daily successfully but sometimes facing this table error. Because of this table issue entire work load effecting. Please have a look into below error and advise.

Failure happened on 'Source' side. ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: 'stage/sfdc/DatedConversionRate'.,Source=Microsoft.DataTransfer.Common,''Type=System.FormatException,Message=String was not recognized as a valid DateTime.,Source=mscorlib,'

This is just loading data from SFDC to azure stage container.

Thanks

Answer 1

@Ram Babu It sounds like you’re encountering an intermittent issue with one specific table in your Azure Data Factory (ADF) pipeline. The error message indicates a problem with the date format during the file upload process. Ensure that the date formats in the DatedConversionRate table are consistent and valid. Sometimes, an unexpected date format can cause this error.

Use ADF’s error handling capabilities to manage failures gracefully. You can configure your pipeline to handle errors and continue processing other tables even if one fails.

Also, refer to suggestions on this post:

https://learn.microsoft.com/en-us/answers/questions/990316/copy-data-to-blob-storage-fails-with-usererrorfail

Please try the suggestions above and let me know via comment, if you are still seeing the issue. Thanks.