Azure Date Factory - REST as source in Flows

Question

I have been using copy activities for a little while.

I would like to see if I can change to using a REST Source inside of a Data Flow

I was successful in connecting, however I return body and header. How do I retrieve only the columns of the dataset?

My guess is I need to add an activity after this, but I am a little lazy right now and am writing SOP's, so could use some help.

Thank you!

Answer 1

Hello @Chris ,
Thanks for the question and using MS Q&A platform.

As we understand the ask here is how to clear out the clutter in Data Flow REST source. Get just the data, not the headers or pagination info or other bits.

Since you are feeling lazy I'll just toss some example code for you. Below is Data Flow script. The Dataset is Inline, using a REST Linked service pointing at a basure URL of https://reqres.in/api/

source(output(  
        body as (data as (color as string, id as short, name as string, pantone_value as string, year as short)[], page as boolean, per_page as short, support as (text as string, url as string), total as short, total_pages as short),  
        headers as [string,string]  
    ),  
    allowSchemaDrift: true,  
    validateSchema: false,  
    format: 'rest',  
    timeout: 30,  
    entity: 'employees',  
    httpMethod: 'GET',  
    paginationRules: ['MaxRequestNumber' -> '1'],  
    responseFormat: ['type' -> 'json', 'documentForm' -> 'singleDocument']) \~> source1  
source1 select(mapColumn(  
        each(body.data,match(true()))  
    ),  
    skipDuplicateMapInputs: true,  
    skipDuplicateMapOutputs: true) \~> select1  

Please do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators