Azure Databricks vs Adf

Question

HI, I have experience using ADF and SSIS. I am now trying to understand and implement Databricks at work. From what I can see ADF has a GUI which is a simple drag and drop which runs on preconfigured spark clusters called Integrated run times. Databricks you have the ability to configure the cluster yourself and program the ETL yourself and do data analytics too at the same time. Databricks is good at processing streaming data. Not sure if ADF can process streaming data.

I have few questions that I need your help to understand
1.) Is writeup above accurate? or anything wrong with my understanding
2.) Especially when you are doing data engineering i.e. ELT/ELT when would you use DataBricks over ADF activities, in real time? A few example scenarios might help.
3.) Can I say Azure databricks is like a stored procedure that can do complex data manipulations in just one script

We are about to start building a Azure synapse warehouse with internal/external sources from data lakes, so wondering in what scenarios I can use Databricks vs ADF activities while creating my pipelines.

Please help. Thanks in advance.

Answer 1

Hi @CzarR The real and notable difference is Azure Data Factory is batch process and natively could not handle heavy real time data intensive

ingestion that is coming across from devices or sensors for that ,other Azure services such as Event Hub and IoT hub with streaming analytics etc. are

used. Whereas the databricks is ideal platform for high volume real time streaming data. Typically , both ADF and Databricks are used in combination as

required.

https://www.mssqltips.com/sqlservertip/6438/azure-data-factory-vs-ssis-vs-azure-databricks/ Thanks

Answer 2

Hi @CzarR ,please go through below threads , it deals with your use case exhaustively.

https://learn.microsoft.com/en-us/answers/questions/108770/datafactory-vs-databricks.html

https://learn.microsoft.com/en-us/answers/questions/176624/data-factory-databricks.html

https://learn.microsoft.com/en-us/answers/questions/461614/what-are-the-major-pros-and-cons-of-using-synapse.html

https://learn.microsoft.com/en-us/answers/questions/367158/azure-data-bricks-and-azure-synapse-spark-pool.html

https://www.element61.be/en/resource/when-use-azure-synapse-analytics-andor-azure-databricks

Hope it helps. Thank you.

Answer 3

Hi @Samy Abdul , these artcles help. Thank you so much.

Still a real scenarios would help understand. From what I can see ADF can do everything that the databricks can do. Why put extra effort writing code into Databricks? Examples would help. Thanks again.