Azure Data Factory Mapping Data Flow type inference appears incorrect?

Kolas, Dave S 1 Reputation point
2021-11-30T21:01:16.947+00:00

The type inference for the map() function in the expression language in mapping data flows appears to be incorrect.

Below is a function which grabs the distanceInMiles sub-element of the location list, sorts this list, and then returns the first.

153794-image.png

Input schema of query_result.location for reference:

153740-image.png

The correct output of this function is a double. map(query_result.location, #item.distanceInMiles) returns a double[]. The enclosing sort also returns a double []. Adding the list index at the end should return double.

However, if I look in the schema output, it believes the output of this is a Complex with multiple sub-fields. This type appears to be derived from the #item in the map function instead of #item.distanceInMiles.

153832-image.png

If I run the step in debug mode, the column correctly outputs the double type and double values:

153769-image.png

This incorrect type inference makes every subsequent step in the mapping flow impossible. It is not possible for me to write this double into a database, because the validation fails because it thinks it is a complex type.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,196 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Kolas, Dave S 1 Reputation point
    2021-11-30T21:13:25.48+00:00

    I've found a workaround that avoids the problem for this case. The function can be rewritten as:

    sort(query_result.location, compare(#item1.distanceInMiles, #item2.distanceInMiles))[1].distanceInMiles

    which eliminates the need for calling the map() function, and thus avoids the incorrect type inference. This obviously does not eliminate the need to solve the type inference for map().