New MySql / MariaDb Azure Data Factory connector errors

Question

Previous version of the mysql/mariadb connector is scheduled for deprecation. So i changed the link service to use the latest verion of the connector. This resulted in multiple issues. The error messages are:
1 Boolean Conversion
"ErrorCode=TypeConversionBooleanNotTreatedAsNumber,Exception occurred in type conversion： ."
2 Java Array
"An error occurred when invoking java, message: java.lang.ArrayIndexOutOfBoundsException:64"
3 DateTime conversion
"The Ticks value '-621357696000000000' for the datetime column must be between valid datetime ticks range -621355968000000000 and 2534022144000000000.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,' "
4 Timeout
ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: 'xxx.parquet'.,Source=Microsoft.DataTransfer.Common,''Type=System.Net.Sockets.SocketException,Message=An existing connection was forcibly closed by the remote host,Source=MySqlConnector,'

error 1 could be fixed by forcing sink side mapping to set boolean to int
error 4 is discused also here:
https://learn.microsoft.com/en-us/answers/questions/1861737/azure-data-factory-copy-activity-with-mysql-unable.

I have adusted the link services to use the Legacy Azure Mariabd connector as sugested there.

But the azure connector is also gonna be depricated on 31-12-2024 .

The new connector obviously does not behave like the previous version, and i don't see the solution for errors 2 , 3 and 4.

So what now to do? Are this indeed bugs in de new connector version? And is MS gonna fix this then before deprication of the Legacy connector?

kind regards,
Sake van der Werf

Answer 1

Hi Werf, Sake van der,

Thanks for reaching out to Microsoft Q&A.

It looks like the issues youare encountering are likely due to the new MySQL/MariaDB adf connector not being fully backward compatible or lacking proper handling of certain data types and operations compared to the legacy version. These types of bugs, particularly around type conversion, datetime handling, and socket connection issues, could indeed be indicative of incomplete or improper functionality in the new connector version.

Here’s what you can do next:

Report the Issues: If you haven't already, I recommend reporting these issues through Azure support channels or the Microsoft Q&A forums. Providing detailed logs and error messages will help expedite any investigation into potential bugs.

Microsoft Roadmap and Fixes: Microsoft will likely address issues with the new connector before the legacy version is deprecated, especially if the issues are widespread. Monitor official Azure release notes, connector updates, and any announcements about bug fixes for the connector.

1. Workarounds for Errors 2 and 3:

• Java Array Out of Bounds (Error 2): This might occur due to improper data handling when fetching or writing large datasets. Try reducing the batch size or applying partitioning to limit the data sent in each batch.
  • DateTime Conversion (Error 3): This error could be related to invalid or null datetime values in your data. You could try filtering or cleaning the data before the copy activity to avoid invalid datetime ranges or transforming datetime values explicitly within a data flow before loading them.
  Handling Error 4 (Timeout): Since this is also mentioned in discussions and relates to socket exceptions, apart from using the legacy connector, ensure you’ve optimized network settings, applied retry policies, or considered increasing timeout configurations in your pipeline.

Although switching back to the legacy connector works for now, I would recommend staying updated on any fixes or patches to the new connector before the deprecation date.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.