Azure Date Factory - REST as source in Flows

Chris 146 Reputation points
2022-07-14T06:35:10.963+00:00

I have been using copy activities for a little while.

I would like to see if I can change to using a REST Source inside of a Data Flow

I was successful in connecting, however I return body and header. How do I retrieve only the columns of the dataset?

My guess is I need to add an activity after this, but I am a little lazy right now and am writing SOP's, so could use some help.

220629-image.png

Thank you!

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,135 questions
0 comments No comments
{count} votes

Accepted answer
  1. MartinJaffer-MSFT 25,996 Reputation points
    2022-07-16T00:57:34.117+00:00

    Hello @Chris ,
    Thanks for the question and using MS Q&A platform.

    As we understand the ask here is how to clear out the clutter in Data Flow REST source. Get just the data, not the headers or pagination info or other bits.

    Since you are feeling lazy I'll just toss some example code for you. Below is Data Flow script. The Dataset is Inline, using a REST Linked service pointing at a basure URL of https://reqres.in/api/

    source(output(  
            body as (data as (color as string, id as short, name as string, pantone_value as string, year as short)[], page as boolean, per_page as short, support as (text as string, url as string), total as short, total_pages as short),  
            headers as [string,string]  
        ),  
        allowSchemaDrift: true,  
        validateSchema: false,  
        format: 'rest',  
        timeout: 30,  
        entity: 'employees',  
        httpMethod: 'GET',  
        paginationRules: ['MaxRequestNumber' -> '1'],  
        responseFormat: ['type' -> 'json', 'documentForm' -> 'singleDocument']) \~> source1  
    source1 select(mapColumn(  
            each(body.data,match(true()))  
        ),  
        skipDuplicateMapInputs: true,  
        skipDuplicateMapOutputs: true) \~> select1  
    

    Please do let me if you have any queries.

    Thanks
    Martin


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

0 additional answers

Sort by: Most helpful