Automatic type conversion for Azure Table Storage

Ossi Kasurinen 96 Reputation points
2021-10-13T09:48:40.017+00:00

Currently we are spending quite alot of time (way too much time) to figure out how to automatically convert Decimal types (any column) to Double when populating Azure Table Storage from a parquet file sources (alternating schema).

ErrorCode=DataTypeNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Type 'System.Decimal' is not supported by Azure Table. Check the supported data type in Azure Table document.,Source=,'

Why can't we a setting where we can make these choices - as in something like this: 

"source": { "type": "Decimal" }, "sink":{ "type": "Double" }

That would be nice.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,603 questions
0 comments
Accepted answer
  1. Ossi Kasurinen 96 Reputation points
    2021-11-12T21:10:03.033+00:00

    This is not currently possible in the copy-data-activity.

    As data-flow in the ADF is slightly more expensive than the just copy-activity, the current (cheaper) work-around is to do an Azure Function that peeks into the parquet file schema and creates a tabular translator mapping (as a json output) that you can use in the mapping part as an expression.

    I truly wish this would be a standard feature in the ADF copy-data activity. Perhaps in the future.

    0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
    2021-10-13T23:52:07.4+00:00

    Hi @Ossi Kasurinen ,

    Welcome to Microsoft Q&A forum and thanks for posting your query.

    Could you please replace decimal data type with double data type in your source columns and see if that helps. The reason for this conversion is Single & Double interim data types in ADF are included in Float-point data type.

    For more info please refer to this doc: ADF Data type mapping

    140318-image.png

    And when you move data to and from Azure Table, the following mappings defined by Azure Table are used from Azure Table OData types to .NET type and vice versa. Here is the reference doc: Data type mapping for Azure Table

    140381-image.png

    Hope this info helps. Do let us know if you have further query.

    ----------

    • Please don't forget to click on 130616-image.png and upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators