DataFactory sink from variable array?

Question

DataFactory sink from variable array?

Hi

I am looking for the best strategy or method to sink from a defined pipeline array. I have with success created a foreach loop, that iterates over a list of objects. Then each object have been stored within a Pipeline variable of type Array. Now I would like to sink the array to csv. How to do that?

Thanks
Klaus

Ryan Abbey 1,106 Reputation points

2021-07-12T00:34:16.867+00:00

You'll need to provide some sampling... too many arrays to understand what you have and what you want.
Klaus Nørregaard 1 Reputation point

2021-07-12T06:50:56.663+00:00

It is just one array. In the Foreach loop how do I sink the data to a data store? The variable is located on the Pipeline and is called RecipientList. It is ready to be stored, but I dont know how to
Ryan Abbey 1,106 Reputation points

2021-07-12T21:33:20.18+00:00

If the array is the actual data you want saved, use the web request as your source dataset within a copy activity

If it's detail of the source that you need to subsequently fetch, use the output of the web request in a foreach loop
Klaus Nørregaard 1 Reputation point

2021-07-13T07:07:06.413+00:00

I cant see that it is possible to use a Web Request as a Source ? Is it possible? Please explain further..
Ryan Abbey 1,106 Reputation points

2021-07-13T20:08:48.927+00:00

How do you currently make the request? REST?
MartinJaffer-MSFT 25,651 Reputation points Microsoft Employee

2021-07-21T06:57:59.793+00:00

If you found your own solution, please share here with the community.

Ryan Abbey 1,106 Reputation points

2021-07-12T00:34:16.867+00:00

You'll need to provide some sampling... too many arrays to understand what you have and what you want.
Klaus Nørregaard 1 Reputation point

2021-07-12T06:50:56.663+00:00

It is just one array. In the Foreach loop how do I sink the data to a data store? The variable is located on the Pipeline and is called RecipientList. It is ready to be stored, but I dont know how to
Ryan Abbey 1,106 Reputation points

2021-07-12T21:33:20.18+00:00

If the array is the actual data you want saved, use the web request as your source dataset within a copy activity

If it's detail of the source that you need to subsequently fetch, use the output of the web request in a foreach loop
Klaus Nørregaard 1 Reputation point

2021-07-13T07:07:06.413+00:00

I cant see that it is possible to use a Web Request as a Source ? Is it possible? Please explain further..
Ryan Abbey 1,106 Reputation points

2021-07-13T20:08:48.927+00:00

How do you currently make the request? REST?
MartinJaffer-MSFT 25,651 Reputation points Microsoft Employee

2021-07-21T06:57:59.793+00:00

If you found your own solution, please share here with the community.

Answer 1

Hello @Klaus Nørregaard , @Ryan Abbey . Welcome to Microsoft Q&A.

I don't want to interrupt the great collaboration you have going on, but I thought of suggesting another way.

If you just need to get the data from a web-based source, use the HTTP dataset or the REST dataset. Web activity is more for getting meta-data.

Web activity can be used to write via leveraging a storage api.

It is also possible to write from an array by using the "additional column" feature in a delimited text dataset. The trick in this, is to use an almost-empty file as source, put the array into additional column, and map to sink.

DataFactory sink from variable array?

1 answer

Activity