DataFactory sink from variable array?

Klaus Nørregaard 1 Reputation point
2021-07-11T09:46:12.097+00:00

Hi

I am looking for the best strategy or method to sink from a defined pipeline array. I have with success created a foreach loop, that iterates over a list of objects. Then each object have been stored within a Pipeline variable of type Array. Now I would like to sink the array to csv. How to do that?

Thanks
Klaus

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,458 questions
{count} votes

2 answers

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,011 Reputation points
    2021-07-14T06:00:21.807+00:00

    Hello @Klaus Nørregaard , @Ryan Abbey . Welcome to Microsoft Q&A.

    I don't want to interrupt the great collaboration you have going on, but I thought of suggesting another way.

    If you just need to get the data from a web-based source, use the HTTP dataset or the REST dataset. Web activity is more for getting meta-data.

    Web activity can be used to write via leveraging a storage api.

    It is also possible to write from an array by using the "additional column" feature in a delimited text dataset. The trick in this, is to use an almost-empty file as source, put the array into additional column, and map to sink.

    0 comments No comments

  2. Jonathan @ SwitchCase Studios 0 Reputation points
    2023-06-27T18:01:35.6133333+00:00

    This was left unanswered but the way to do it is to store the output from the HTTP Request into a variable using the set variable, if there are subsequent HTTP calls then you'll require a ForEach loop that appends to your variable. Once that is done you'll need to pass your data to a JSON output you can use "add additional columns" and use the expression builder to access your variable. variables('variable1') and it will add this to your output.

    0 comments No comments