"Incompatible data type mapping" only in Flowlet

Question

I'm building a flowlet to transform data from a DL2 pool of Json files to a Parquet sink.

First I built a traditional dataflow with the exact same transformation on DL2 datasets for 4 different countries. That works like a charm, but I would like to get rid of the redundant steps, so I selected all 3 steps (flatten, stringify and derived) from my dataflow and created a Flowlet.

The flowlet validates without any issues, but when trying to use the flowlet in a new dataflow, I get a "Incompatible data type mapping" error on the items property.

The items property is an array of subscriptions in the json files that are imported.

I think the error is caused by the "Input settings". In my traditional Data Sourceinput, the projection looks like this:

But in my flowlet that translates to this:

What really puzzles me is, that neither complex or array are available types in my Input settings mapping, but only the array is flagged as incompatible:

What can I do to make my flowlet accept the complex array that I use in my traditional dataflow, so I do not have to repeat the same code for each new json sink?

The flowlet validates just fine, but when I add it to one of the data sources that run with the traditional dataflow, it complains:

Answer 1

Hello @Axel B Andersen ,

Just had a confirmation from the product team that it is identified a bug that complex data types are not appearing in Flowlets and product team is actively working on rolling out a fix for it.

At this time, I do not have an ETA. In addition, I'm also checking to see if there is any possible workaround to unblock you untill the issue is fixed permanently. Will keep you posted as soon as we hear back from the team.

Sorry for the inconvenience caused because of this issue.

Answer 2

Hi @KranthiPakala-MSFT ,

Similar problem here. Despite the flowlet accepts the complex datatype and recognizes it as an array, then the mapping is not valid with an error messsage:
"Incompatible data type mapping from column <source_col_name> to column <destination_col_name>"

This is the input column (an array)

and this is the flowlet input expected (an array as well)

Do you know the reason for this behaviour?

Answer 3

Hello @Axel B Andersen ,

Wanted to update you that the bug has been fixed and now both binary/complex options are available as column type for selection in flowlets.

Hope this helps. Appreciate for bringing this to our attention and helping us improve the product.

----------

Please do consider clicking on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.

Answer 4

Hi ,

We could see the complex datatype but it looks like an object rather than array and when passing it to flatten it doesn't recognise it as an array.

Error from flatten:

The inner value type of the 'Unroll by' structure 'value' must be array type (e.g. structue of arrays, array of arrays)