ADF Data Flow source transformation is not reading data types correctly while working with complex data types.

Varun Mourya Karanam 51 Reputation points
2022-11-18T07:51:00.033+00:00

I have a json file which looks like below. Though all attributes in both objects are in

string( '' ''). Data flow is reading them as boolean in case of "1" and as short in case of "2","3" etc.

Under OccurrenceSpans, object 1 elements are read as (date,short,bool,date) but in object 2 it is being read as (str,str,str,str) - though data is of same type in both objects.

261739-image.png
JSON File format Screenshot

261783-image.png
Data Flow Screenshot

When tried to Overwrite schema, it is not allowing me to define data types for sub elements. I am getting option only to define main object data type only.

Can you help me in solving this ?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,623 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Nasreen Akter 10,811 Reputation points Volunteer Moderator
    2022-11-18T17:09:28.38+00:00

    Hi @Varun Mourya Karanam ,

    Thank you for the ask.

    I think you are trying to change the schema from the DataFlow >> Source. You can change the Dataset-Schema by going into the Dataset >> code, and then modify the TYPE. Now, if you go in the DataFlow >> Source >> Projection and do reset Schema, it will show you the updated one. Thanks!

    261969-image.png


  2. Varun Mourya Karanam 51 Reputation points
    2022-11-21T06:19:02.887+00:00

    @Nasreen Akter Thank you for the response.

    Found a way to define datatypes for Inline Datasets. We have an option in Source>Projection > Enable overwrite schema >Define Complex type where we can change sub elements data type.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.