How to Handle Case Sensitivity in Dynamic JSON Input Keys ("A" vs "a") for Derived Columns in ADF Dataflow Pipeline?
I am working on a dataflow pipeline in ADF with an input dataset schema that includes a column 'A'. The dataflow processes various JSON files with dynamic schemas, where some JSON files use 'A' and others use 'a' as keys. Since the schema is case insensitive, I cannot include both 'A' and 'a' in the input dataset schema. Currently, I have a derived column dependent on the value of 'A'/'a' (it combines <A> + '/' + <date> and the sink stores the folders based on the derived column path data), and it functions correctly for JSON files containing 'A'. How can I modify the dataflow to handle JSON files with 'a' as the key? Is there any other workaround to achieve this in ADF?
Here are examples of two JSON files with dynamic schemas that I'm working with: