ADF Dataflow getting OutOfMemoryException on reading data from Blob, Blob size is 3.81 GB. I am using max number of cores with MemoryOptimised runtime integration

Nikunj Mittal 1

KranthiPakala-MSFT 46,827 Reputation points Moderator

2021-12-03T20:25:55.993+00:00

Hi @Nikunj Mittal ,

We still have not heard back from you. Just wanted to check if the below suggestions from Mark was helpful? If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

1 answer

MarkKromer-MSFT 5,231 Reputation points Microsoft Employee Moderator

2021-11-29T18:00:45.707+00:00

Data preview is for debugging sampled data sets. Use a smaller dataset when debugging with data preview.
Was this answer helpful?
Nikunj Mittal 1 Reputation point

2021-11-30T05:50:22.707+00:00

@MarkKromer-MSFT How do I check if it's working with the complete data? I triggered the pipeline as well to test if it runs with real data.

Is there is a way to read Blob in parts in DataFlow?

Mark Kromer MSFT 1,146 Reputation points

2021-11-30T16:37:45.547+00:00

Execute it from a pipeline. Split large files into smaller files for unit testing and debugging.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Your answer