How to improve the performance using multithreading or parallel processing?

saurabh khandelwal 1 Reputation point

Hi All,

We have to get data from thousands of API calls and insert it into the MSSQL Database table.
All the API call returns the same type of object so we have a single table where we have to store them.
In Data Flow we have used the Script Transformation component which takes the endpoints column as inputs, makes the call for each row, and then returns the output column JSONdata in unicode_text_stream format, and then further we have to parse the JSONdata and map to the output table columns --> then Union All and stored it into the database. Please see the below image of the Data flow.


This complete process takes 2 hours for the 1500 api calls. so how can implement this using Multithreading/Parallel processing to improve the performance?

SQL Server Integration Services
SQL Server Integration Services
A Microsoft platform for building enterprise-level data integration and data transformations solutions.
2,084 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Tom Phillips 17,636 Reputation points

    Get a new API which returns batches instead of calling it thousands of times.

    0 comments No comments