ADF - ODP Connector - OutOfMemory issue

Joey Sun 1 Reputation point
2022-09-02T06:33:04.363+00:00

Hi,

Using the new ODP connector in ADF, we sometimes encounter a memory limitation issue when extracting large tables (even if we partition them into smaller pieces). According to an earlier Q&A memory issues should not be possible anymore with this connector, so this is strange.

What the cause of this error and how can we solve it?

237098-image.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,663 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. BhargavaGunnam-MSFT 26,501 Reputation points Microsoft Employee
    2022-09-02T23:59:01.76+00:00

    Hello @Joey Sun ,

    Welcome to the MS Q&A platform.

    Could you please check the resource usage and concurrent-activity execution on the self-hosted IR node? If it has high memory usage and too many concurrent activity executions, please adjust the trigger time and internal time of activity runs.

    If you can restrict too many activities execution on the same IR node simultaneously, see if it helps.

    Please let me know if you have any further questions.