databricks vs ADF/synapse

Question

Hi,
I have started learning databricks...
It looks like databricks is more coding rather than ADF where it is mostly visual except where you add expressions, etc.
Am I right to assume that alot of the work carried out with databricks can be done with ADF or synapse workspace instead?

Thank you

Answer 1

Hey @arkiboys ,

ADF is basically similar to SSIS wherein you have components inbuilt and you drag and drop and create flows.

Whereas ADB(databricks) is completely code oriented wherein even to connect to source or sink you would need to write a code or to do a loop or iteration whereas in ADF it is a matter of creating a linked service directly and using For each loop.

ADF is still an evolving tool so there are many scenarios which cannot be handled via ADF for which you need to write a custom code (there you can leverage Azure fucntions, Azure batch,Azure automation or databricks)

So based on use cases, we take a call whether to leverage ADF or ADB

Answer 2

Hi @arkiboys , as Nandan has mentioned in prior post , Azure Data Factory(ADF) is a cloud orchestration tool for managing batch

data integration workloads that runs your data pipelines (like ETL). The Azure Databricks provides batch processing solutions. ADF does not support

streaming data natively and for that you have to rely on Event Hub etc. Below threads might be useful to go through. Thanks

https://www.mssqltips.com/sqlservertip/6438/azure-data-factory-vs-ssis-vs-azure-databricks/

https://learn.microsoft.com/en-us/answers/questions/108770/datafactory-vs-databricks.html