Azure functions, web jobs or Azure Data Factory ADF - example scenarios ?

bitshift 71 Reputation points
2021-02-27T23:24:12.837+00:00

Trying to choose between Azure functions, web jobs or looking to something like Azure Data Factory
Anyone

Which Azure services fit this need - moving 1-10MB data files with transformations

  • Data files must be imported, read and the data "shaped" into a hieararchical JSON structure to feed another import process
  • During the process of reading/extracting the data in each file, lookups must be performed to fill in missing/required values
  • The transformations will also involve shifting different collections around within each record/object
  • This activity needs to be scueduled
  • As suspected, one or more steps might take more than the 5 min limit imposed on Azure functions

Since I am a .net coder, I tend to look for solutions that are code-centric. Hence, I am drawn to Azure functions but im not sure this is the right choice.
I have done a fair amount of work with SSIS and have started looking at ADF (Azure Data Factory) but I would tend to lean toward the code-centric approach.
The thing with functions that cause me to pause a bit is when I consider just how much processing must be done with each file, which will contain 100+ orders and their details.

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,517 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,939 questions
{count} vote

Accepted answer
  1. HimanshuSinha-msft 19,386 Reputation points Microsoft Employee
    2021-03-02T18:53:51.417+00:00

    Hello @bitshift ,

    Thanks for the ask and using the Microsoft Q&A platform .
    From all in the inputs which you have provided I see that you are more inclined towards Azure Functions . I am pretty confident that you will be able to achieve the ask with that also . Since you are more code centric , this may be more helpful.

    I could suggest you to evaluate the Mapping data flow (MDF ) as it does provide the the transformation option and its also lets you play with the JSON object . For MDF you will not need any coding ( just some expression ) as most of the thing are drag and drop .

    Please do let me know how it goes .
    Thanks
    Himanshu
    Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

    1 person found this answer helpful.
    0 comments No comments

0 additional answers

Sort by: Most helpful