DataFactory sink from variable array?

Question

Hi

I am looking for the best strategy or method to sink from a defined pipeline array. I have with success created a foreach loop, that iterates over a list of objects. Then each object have been stored within a Pipeline variable of type Array. Now I would like to sink the array to csv. How to do that?

Thanks
Klaus

Answer 1

Hello @Klaus Nørregaard , @Ryan Abbey . Welcome to Microsoft Q&A.

I don't want to interrupt the great collaboration you have going on, but I thought of suggesting another way.

If you just need to get the data from a web-based source, use the HTTP dataset or the REST dataset. Web activity is more for getting meta-data.

Web activity can be used to write via leveraging a storage api.

It is also possible to write from an array by using the "additional column" feature in a delimited text dataset. The trick in this, is to use an almost-empty file as source, put the array into additional column, and map to sink.

Answer 2

This was left unanswered but the way to do it is to store the output from the HTTP Request into a variable using the set variable, if there are subsequent HTTP calls then you'll require a ForEach loop that appends to your variable. Once that is done you'll need to pass your data to a JSON output you can use "add additional columns" and use the expression builder to access your variable. variables('variable1') and it will add this to your output.