Azure Data Factory Data Flow removing the first and last rows of a blob-stored file

Bakhshpour, Daniel 41 Reputation points
2021-06-03T21:06:47.52+00:00

Hello,

I have a file stored in blob storage that has a header and a footer. I want to remove the first and last rows and do some further manipulation within data flow. What I have done is created a surrogate key and tried to use the filter modifier to remove the rows. I have success removing the first row but am having trouble removing the last row because I can't call max(key) or last(key) in the filter expression. What is the best solution for removing these rows?

Thanks,

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

Answer accepted by question author
Mark Kromer MSFT 1,146 Reputation points
2021-06-04T00:22:55.893+00:00

Add an aggregate transform after the surrogate key to get the min() and max() key values for your filter.

Was this answer helpful?

2 people found this answer helpful.
0 comments

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.