REST sink with non-array payload (body)

Matthew Campbell 1 Reputation point
2023-01-25T07:58:31.5+00:00

Is it possible to have a body (payload) other than an array sent by the REST sink?

Goal is to use a copy activity with a REST API source paging Azure Search documents to a REST API sink on another Azure Search index with the same schema.

String arrays (collections) har not supported by the Azure Search sink when mapping thereby eliminating that particular sink in a paging copy activity (on the collection reference $.value). A REST API source against an Azure Search index yields responses in the following schema (i.e. hierarchical data):

{
...
"value: [ <search documents> ]
}

The REST API as sink documentation seems to stipulate that the body of the JSON is always an array: "data will be sent in JSON with the following pattern" (see JSON example under REST as sink section which is an array). The schema mapping documentation gives another impression that hierarchical data from a source can be ported to a sink: "When copying data from hierarchical source to hierarchical sink, you can additionally preserve entire layer's hierarchy, by selecting the object/array and map to sink without touching the inner fields".

Posted additionally on Stackoverflow with a gist of the linked service, data set, pipeline and input + output.

The Azure Cognitive Search add || update || delete REST API (i.e. the potential sink) can accept responses of the searching documents API (paging source in copy activity) as a body (payload) and perform an upsert by default.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,525 questions
{count} votes