How to connect to the foreach activity output to flatten the nested json data

Question

Hi,

I have just started to work with Azure Data factory.

Here is the breakdown of my pipeline

Lookup Activity: This activity retrieves the list of tickers, from adls gen2 directory.

Web Activity (Outside ForEach): This activity makes a web request to the Alpha Vantage API to fetch data based on the entire list of tickers obtained from the lookup activity. This might be used for some initialization or metadata retrieval.

ForEach Activity: This activity iterates over each item in the list of tickers obtained from the lookup activity.

Web Activity (Inside ForEach): This activity is placed inside the ForEach activity and makes a web request to the Alpha Vantage API for each individual ticker in the list. This fetches the specific data related to each ticker.

Now i want to flatten the nested json data and dump it to data factory before the next lookup data is picked up. Is there any way to do this? add a copy activity would require source and not sure what could be the source here since am getting output from the web activity.

Above is the pipeline image. I have also attached the sample json output file for reference.

Thanks in advance!

Answer 1

Ashish Could you try the Copy Activity inside the For-each? which contains the HTTP Dataset as Source Please try this and do let us know if that works?

Also please attach sample nested JSON from the data preview, which you want to flatten, so that we better understand the scenario!!!