Column functions not supported in configurations - MDF Expression Error
I am building a pipeline with a Mapping Data Flow activity in synapse analytics. I am trying to create a derived column with an expression similar to below. However, it fails with the error "Column functions not supported in configurations".
I need to dynamically get the list of column names and get the values on the fly. Looks like it is not working when I use byNames() function along with columnNames() function. Is there any workaround for this?
PS: I cannot use columns() function because I need to reorder or filter some of the columns.
Just columnNames() alone will give you the column names
I'm having the same issue.
Trying to generate a hash of all incoming columns using- columns_hash = md5(byNames(columnNames())) which throws the error above.
If I pass a parameter in to generate the hash as in - columns_hash = md5(byNames(split($columns,','))) it works fine.
Why, when both the function split() and columnNames() return an array of string, which is then passed to the byNames() function, does the second expression work but the first fail?
Wouldn't you use just want to hash the column values instead? md5(columns())
Yep - that worked :) Stupidity error on my part doh!!!
Thanks for your response. However, just the columnNames() function does not work in my case because I actually want the values of the columns. I want to get all the columns dynamically first and then reorder the columns before calculating the hash value. This is to make sure the hash value is correct even if the column orders are different in the source file.
Reorder your columns with a Select transformation prior to the Derived Column. Then you can use just md5(columns()) to get the row hash of the column values using the column order you set.
Is there a way to reorder the columns with Select transformation if I do not know my columns at the design time.
Sign in to comment