TPL Performance slows down above 2 Gb data migration

Question

We have developed an Integration Tool using Task Parallel Library(TPL) PLINQ and are migrating data but its performing slow.

We observed that for 2 GB of data the tool performs well but above 2 GB the performance of tool slows down.

Below is the PC configuration on which the tool is running and tool is compiled as 'any cpu'

My PC's configuration is :- Processor : Intel(R) Core(TM) i3-7100 CPU @ 3.90GHz 3.91 GHz RAM : 8 GB RAM Windows 10 Pro.

IDE :- Visual Studio Express 2017 for Desktop.

Can someone tell us what can be the root cause of this performance issue. It will be very helpful to us.
Thankyou.

Answer 1

Hi @bharat choudhary ,
TPL is the preferred way to write multithreaded and parallel code. However, not all code is suitable for parallelization. For example, if a loop performs only a small amount of work on each iteration, or it doesn't run for many iterations, then the overhead of parallelization can cause the code to run more slowly. Furthermore, parallelization like any multithreaded code adds complexity to your program execution. Although the TPL simplifies multithreaded scenarios, we recommend that you have a basic understanding of threading concepts, for example, locks, deadlocks, and race conditions, so that you can use the TPL effectively.

You can refer to following document to get more details
https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/potential-pitfalls-in-data-and-task-parallelism

---

If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

---

Answer 2

most likely your threads are starved for a resource which longer processing exposes.

Maybe the number of connections to the Sharepoint server or from the CRM. maybe the CRM or Sharepoint servers are overloaded. Maybe the network is overloaded. Maybe you have poorly written code that doesn't scale (say overuse of memory and increased paging)