"Incompatible data type mapping" only in Flowlet

Axel B Andersen 106 Reputation points
2022-01-29T06:56:14.883+00:00

I'm building a flowlet to transform data from a DL2 pool of Json files to a Parquet sink.

First I built a traditional dataflow with the exact same transformation on DL2 datasets for 4 different countries. That works like a charm, but I would like to get rid of the redundant steps, so I selected all 3 steps (flatten, stringify and derived) from my dataflow and created a Flowlet.

The flowlet validates without any issues, but when trying to use the flowlet in a new dataflow, I get a "Incompatible data type mapping" error on the items property.

The items property is an array of subscriptions in the json files that are imported.

I think the error is caused by the "Input settings". In my traditional Data Sourceinput, the projection looks like this:
169562-image.png

But in my flowlet that translates to this:
169514-image.png

What really puzzles me is, that neither complex or array are available types in my Input settings mapping, but only the array is flagged as incompatible:
169574-image.png

What can I do to make my flowlet accept the complex array that I use in my traditional dataflow, so I do not have to repeat the same code for each new json sink?

The flowlet validates just fine, but when I add it to one of the data sources that run with the traditional dataflow, it complains:
169543-image.png

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,346 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,489 questions
{count} votes

4 answers

Sort by: Most helpful
  1. KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
    2022-02-01T06:37:50.197+00:00

    Hello @Axel B Andersen ,

    Just had a confirmation from the product team that it is identified a bug that complex data types are not appearing in Flowlets and product team is actively working on rolling out a fix for it.

    At this time, I do not have an ETA. In addition, I'm also checking to see if there is any possible workaround to unblock you untill the issue is fixed permanently. Will keep you posted as soon as we hear back from the team.

    Sorry for the inconvenience caused because of this issue.

    1 person found this answer helpful.
    0 comments No comments

  2. Gonçalo Sousa 6 Reputation points
    2023-01-03T11:16:21.4+00:00

    Hi @KranthiPakala-MSFT ,

    Similar problem here. Despite the flowlet accepts the complex datatype and recognizes it as an array, then the mapping is not valid with an error messsage:
    "Incompatible data type mapping from column <source_col_name> to column <destination_col_name>"
    275644-image.png

    This is the input column (an array)
    275664-image.png
    and this is the flowlet input expected (an array as well)
    275497-image.png

    Do you know the reason for this behaviour?

    1 person found this answer helpful.

  3. KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
    2022-02-17T06:38:30.76+00:00

    Hello @Axel B Andersen ,

    Wanted to update you that the bug has been fixed and now both binary/complex options are available as column type for selection in flowlets.

    175229-image.png

    Hope this helps. Appreciate for bringing this to our attention and helping us improve the product.

    ----------

    Please do consider clicking on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.

    0 comments No comments

  4. Thomas, Jinty 1 Reputation point
    2022-06-07T16:11:59.197+00:00

    Hi ,

    We could see the complex datatype but it looks like an object rather than array and when passing it to flatten it doesn't recognise it as an array.

    Error from flatten:

    The inner value type of the 'Unroll by' structure 'value' must be array type (e.g. structue of arrays, array of arrays)

    0 comments No comments