ADF Architecture (or) how does pipeline executed in SHIR?

Question

Hi,

Please let us know how does the ADF pipeline execute in the VM or AZURE IR?

Does the pipeline converted into scala or java or .net program before submitting to IR?
What is the ADF Architecture?
What is the backbone of ADF?

Regards,

Answer 1

Hi @Srinivas Tallam ,

Thank you for posting query in Microsoft Q&A Platform.

Azure Data Factory is a managed cloud service that's built for complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects.

The Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory to provide the following data integration capabilities across different network environments:

• Data Flow
• Data Movement
• Activity Dispatch
• SSIS Package Execution

Integration runtime acts as bridge between Activity(Task which user wants to perform) and linked service(Target resource).

For example, if your task is to run SQL script using lookup activity, then IR will help ADF to connect with SQL server(linked service) and execute sql query on it. So, here your SQL server which is configured on linked service acts as compute environment. Now Query execution mechanism is purely depending your compute resource type.

So, ADF don't have its own compute environment, depending on resource your are connecting(linked service) to execute activity, it utilizes that resource compute to complete task.

Hope this will help. Please let us know if any further queries. Thank you.

--------------------------------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.