How to improve the performance using multithreading or parallel processing?

saurabh khandelwal 1 Reputation point
2021-09-28T07:03:47.42+00:00

Hi All,

We have to get data from thousands of API calls and insert it into the MSSQL Database table.
All the API call returns the same type of object so we have a single table where we have to store them.
In Data Flow we have used the Script Transformation component which takes the endpoints column as inputs, makes the call for each row, and then returns the output column JSONdata in unicode_text_stream format, and then further we have to parse the JSONdata and map to the output table columns --> then Union All and stored it into the database. Please see the below image of the Data flow.

135814-image.png

This complete process takes 2 hours for the 1500 api calls. so how can implement this using Multithreading/Parallel processing to improve the performance?

SQL Server Integration Services
SQL Server Integration Services
A Microsoft platform for building enterprise-level data integration and data transformations solutions.
2,611 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Tom Phillips 17,741 Reputation points
    2021-09-28T15:57:57.657+00:00

    Get a new API which returns batches instead of calling it thousands of times.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.