Data Factory Data Flow - External Transformation issues

Dmitriy Ryabin 6 Reputation points
2023-05-09T18:13:48.18+00:00

I .. do not understand what I should do with this. I have a REST call that gets the list of Projects. I set up that as a source to the data flow - having a Rest Linked service feed the data.

ADF Data Fkow

The JSON from the REST call is list of Projects objects Something like ...

{
	"_id": "5a04b1a0",
	"name": "Some name - as string",
	"number": "179",
	"value": null,
	"state": "PUBLISHED",
	"Sealed": 0,
	"ndaRequired": 0,
	"public": 1
    }

and there can be hundreds of those. After the list is obtained, I need to go through each one and make additiona API call to https://<baseUrl>/projects/<{_id}> where _id is a project ID (from above) This call will give me individual project with ADDITIONAL fields - nested "Packages" array

{
	"_id": "5a04b1a0",
	"name": "Some name - as string",
	"number": "179",
	"value": null,
	"state": "PUBLISHED",
	"Sealed": 0,
	"ndaRequired": 0,
	"public": 1,
    "Packages": [
        {
            "_id": "59a0471db3",
            "projectId": "5a04b1a0",
            "name": "some",
            "number": "9250",
            "keywords": [
                "keyWord"
            ],
            "state": "PUBLISHED",
            "dateStart": null
        },
        {
            "_id": "934234kkd93",
            "projectId": "5a04b1a0",
            "name": "some other",
            "number": "24349374",
            "keywords": [
                "keyWord, keyword, keyword"
            ],
            "state": "UNKNOWN",
            "dateStart": null
        }
    }

I do not know how (The syntax??) to specify the response schema for each individual "GetPackages" external call. please help. In the "Data Preview" of the FlattenProjects stage, I get the correct list for Projects, and after "Derived Column" I get the additional "ProjectID" as a url to be added. If I test the complete individual URL in postman - it works, question is NOT about correctness of the URL call. what do I put in the "body" and/or "Type", so to get the packages added. ? I can't find any documentation on this, or maybe I do not know / understand what I am looking for to post the rightly phrased question. I can and know how to use Flaten transformation if I need to flatten it afterwards

What do I need to put into the "Type" in the picture, so I can get the

External transformation

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
{count} votes

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.