How to improve the performance using multithreading or parallel processing?

saurabh khandelwal 1 Reputation point
2021-09-28T07:03:47.42+00:00

Hi All,

We have to get data from thousands of API calls and insert it into the MSSQL Database table.
All the API call returns the same type of object so we have a single table where we have to store them.
In Data Flow we have used the Script Transformation component which takes the endpoints column as inputs, makes the call for each row, and then returns the output column JSONdata in unicode_text_stream format, and then further we have to parse the JSONdata and map to the output table columns --> then Union All and stored it into the database. Please see the below image of the Data flow.

135814-image.png

This complete process takes 2 hours for the 1500 api calls. so how can implement this using Multithreading/Parallel processing to improve the performance?

SQL Server Integration Services
SQL Server Integration Services
A Microsoft platform for building enterprise-level data integration and data transformations solutions.
2,460 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Tom Phillips 17,716 Reputation points
    2021-09-28T15:57:57.657+00:00

    Get a new API which returns batches instead of calling it thousands of times.

    0 comments