Share via

Azure Data Factory Intergration vs Inline Dataset

Santhi Dhanuskodi 66 Reputation points
2022-08-17T08:57:53.467+00:00

Hi,

I want to know the difference between integration data set and inline data set in ADF.

I know when multiple people in the team and pipelines look for same data set, we can go for integration data set.
That is sharable across branches.

What are pros and cons of these? Which DS will cost more?

Individual branches are created based on main branch and people already working on respective branches.
Now if i need an integration dataset, how do i proceed? Will creating a DS in main branch resolve it?
should i publish it? Will it be visible/accessible to other branches? Will it sync from main to individual branches?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,559 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,624 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator
    2022-08-18T11:46:52.057+00:00

    Hi @Santhi Dhanuskodi ,

    Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

    As you have understood correctly, physical dataset are shareable or usable in any of the pipelines within the ADF.

    However, if you are doing something just for the purpose of one dataflow, and you don't need that to be reused in other dataflows , then you can add that datastore as source or sink , without needing to define it as a shared resource .

    Inline datasets are recommended when you use flexible schemas, one-off source instances, or parameterized sources. If your source is heavily parameterized, inline datasets allow you to not create a "dummy" object. Inline datasets are based in Spark, and their properties are native to data flow.

    For more details, kindly visit: Inline datasets

    https://www.youtube.com/watch?v=fjNPL_Aqxl0
    https://www.youtube.com/watch?v=3799TA62rao

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    2 people found this answer helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.