Automatic type conversion for Azure Table Storage

Question

Currently we are spending quite alot of time (way too much time) to figure out how to automatically convert Decimal types (any column) to Double when populating Azure Table Storage from a parquet file sources (alternating schema).

ErrorCode=DataTypeNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Type 'System.Decimal' is not supported by Azure Table. Check the supported data type in Azure Table document.,Source=,'

Why can't we a setting where we can make these choices - as in something like this:

"source": { "type": "Decimal" }, "sink":{ "type": "Double" } 

That would be nice.

Answer 1

Hi @Ossi Kasurinen ,

Welcome to Microsoft Q&A forum and thanks for posting your query.

Could you please replace decimal data type with double data type in your source columns and see if that helps. The reason for this conversion is Single & Double interim data types in ADF are included in Float-point data type.

For more info please refer to this doc: ADF Data type mapping

And when you move data to and from Azure Table, the following mappings defined by Azure Table are used from Azure Table OData types to .NET type and vice versa. Here is the reference doc: Data type mapping for Azure Table

Hope this info helps. Do let us know if you have further query.

----------

• Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

This is not currently possible in the copy-data-activity.

As data-flow in the ADF is slightly more expensive than the just copy-activity, the current (cheaper) work-around is to do an Azure Function that peeks into the parquet file schema and creates a tabular translator mapping (as a json output) that you can use in the mapping part as an expression.

I truly wish this would be a standard feature in the ADF copy-data activity. Perhaps in the future.