Get max value from a parquet file by using ADF

Question

Hi, I am able to use ADF data flow to read a parquet file and get the row numbers of this parquet file. Following is the data flow.

But when I followed this https://stackoverflow.com/questions/74813503/how-to-take-max-value-and-replace-it-in-drived-colomn-in-data-factoryand tried to get max value of one column, it fail. As long as I added the window, I could not even do Data Preview from source. Here is the error message: Spark job failed: { "text/plain": "{"runId":"f128afcc-c3af-4bb7-a0a9-0544303c577b","sessionId":"0a1d08df-99ce-496f-b437-f7e098422c65","status":"Failed","payload":{"statusCode":400,"shortMessage":"DF-EXPR-010 at Window 'window1'(Line 16/Col 35): Column 'Hello' used in expression is unavailable or invalid.","detailedMessage":"Failure 2024-01-24 18:34:29.554 failed DebugManager.processJob, run=f128afcc-c3af-4bb7-a0a9-0544303c577b, errorMessage=DF-EXPR-010 at Window 'window1'(Line 16/Col 35): Column 'Hello' used in expression is unavailable or invalid."}}\n" } - RunId: f128afcc-c3af-4bb7-a0a9-0544303c577b Do you have any suggestions on how to get max value from a parquet file ? Thanks

Answer 1

Hello,

Looks like you havent imported the schema in source parquet file. Once you import schema, you will see number of columns in source transformation. Currently it doesnt show any column info, which means that the schema is not imported.

Go to the source data set and import schema from file. And then add a window transformation to find max key.

Please try this and let us know for any further query on it. Thank you.