Combine two columns into one Azure Data Factory - Data Flow

Andres Esteban 156 Reputation points
2022-11-03T11:05:19.683+00:00

Hello

I have the following problem. I'm doing a data flow in ADF and the first thing I do is to decouple a JSON file via its body. When I decouple it I get two columns called "humidity" and "Hum" which work as follows. In some rows the humidity column contains the humidity data and in that row Hum is NULL and in others it is the other way around. How can I combine both columns into one so that I can delete both and the resulting column has the humidity data? The value is of type float.

Thank you

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,879 questions
{count} votes

Accepted answer
  1. MartinJaffer-MSFT 26,046 Reputation points
    2022-11-04T18:43:48.817+00:00

    Hello @Andres Esteban ,
    Thanks for the question and using MS Q&A platform.

    As I understand, you want to combine two columns into one. One of the columns is sometimes null.

    If your data is like:

    Humidity | Hum | desired result  
    23.4 | 23.4 | 23.4  
    55.1 | Null | 55.1  
    Null | 12.2 | 12.2  
    

    Then you can use coalesce( Humidity , Hum) in derived column. Coalesce takes the first non-null value. This works logically because you never have 2 numbers in disagreement. However if you have data like:

    Humidity | Hum  
    12.3 | Null  
    45.2 | 1.9  
    63.1 | 8.1  
    

    Things become much more complicated. What is the desired output when 2 non-null values disagree? Is it an average value, or a concatenation or something else? In such a case, coalesce would always take one column and ignore the other.

    Please do let me if you have any queries.

    Thanks
    Martin


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

1 additional answer

Sort by: Most helpful
  1. Sacher, Wolf-Rüdiger 21 Reputation points
    2023-10-05T07:39:02.77+00:00

    Hi @MartinJaffer-MSFT ,
    what would be your suggestion, if all column values were needed, as you described, that coalesce would always drop the second value?

    0 comments No comments