How to improve the performance using multithreading or parallel processing?

Question

Hi All,

We have to get data from thousands of API calls and insert it into the MSSQL Database table.
All the API call returns the same type of object so we have a single table where we have to store them.
In Data Flow we have used the Script Transformation component which takes the endpoints column as inputs, makes the call for each row, and then returns the output column JSONdata in unicode_text_stream format, and then further we have to parse the JSONdata and map to the output table columns --> then Union All and stored it into the database. Please see the below image of the Data flow.

This complete process takes 2 hours for the 1500 api calls. so how can implement this using Multithreading/Parallel processing to improve the performance?

Answer 1

Get a new API which returns batches instead of calling it thousands of times.