Azure data factory .CSV/TXT file row/column filtering WITHOUT using dataflow

Question

Azure data factory .CSV/TXT file row/column filtering WITHOUT using dataflow

Pankaj Joshi 411

I am copying .Csv file from storage blob through azure data factory. Then I need to filter few records based on specific column value AND also need to remove few columns from. Csv file.

Is above basic row level and column level filtering is possible for .csv/txt files in Adf WITHOUT using DATAFLOW?

Please advise.
(Also I am not using any database)

Nandan Hegde 36,716 Reputation points MVP Volunteer Moderator

2022-12-13T17:07:08.08+00:00

Hey, what is the amount of rows in the file?
If less, you can try to use look up activity
Pankaj Joshi 411 Reputation points

2022-12-13T17:21:12.89+00:00

There could be thousands of rows

1 answer

Your answer

Nandan Hegde 36,716 Reputation points MVP Volunteer Moderator

2022-12-13T17:07:08.08+00:00

Hey, what is the amount of rows in the file?
If less, you can try to use look up activity
Pankaj Joshi 411 Reputation points

2022-12-13T17:21:12.89+00:00

There could be thousands of rows

Answer 1

Nandan Hegde 36,716 MVP Volunteer Moderator

Hey
Based on my understanding, with no database involved and no dataflow usage and your file size exceeding the limitations of lookup activity , there is no out of the box feature within ADF to achieve this.
You would have to write your own custom logic in Azure automation/databricks or azure batch etc .

Note: There is something called power query but it too uses a spark cluster hence my thought is you would not be leveraging that as well

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-power-query-activity

MartinJaffer-MSFT 26,161 Reputation points

2022-12-22T02:25:15.04+00:00

@Anonymous , @Nandan Hegde is correct. You might want to take a look at Table storage in the Azure Storage accounts. I think ADF can query against that.

There is also a blob Storage feature called query acceleration. I haven't used it before, and there is no feature in ADF to use it. However I think it can be used as part of HTTP dataset, when pointed at this operation.

If either of us answered your question, please mark as accepted answer. Thank you!

Share via

Azure data factory .CSV/TXT file row/column filtering WITHOUT using dataflow

1 answer

Your answer