Share via

Azure Data Factory Data Flow removing the first and last rows of a blob-stored file

Bakhshpour, Daniel 41 Reputation points
2021-06-03T21:06:47.52+00:00

Hello,

I have a file stored in blob storage that has a header and a footer. I want to remove the first and last rows and do some further manipulation within data flow. What I have done is created a surrogate key and tried to use the filter modifier to remove the rows. I have success removing the first row but am having trouble removing the last row because I can't call max(key) or last(key) in the filter expression. What is the best solution for removing these rows?

Thanks,

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,452 questions
Accepted answer
  1. Mark Kromer MSFT 1,146 Reputation points
    2021-06-04T00:22:55.893+00:00

    Add an aggregate transform after the surrogate key to get the min() and max() key values for your filter.

    2 people found this answer helpful.
    0 comments

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.