Delta lake vs Azure data lake

Question

Hi All, I understand we ingest the data from varied sources and through ADF and build the ADLS Gen2 Data lake. But Data lake is doesn't allow ACID transactions, where as Delta lake which mostly build through data bricks does provide ACID transactions feature, I understand by using Synapse we could overcome this challenge. But ,I am bit confused why can't we straightaway go for Delta lake and avail this feature. What scenarios does call for data lake instead of delta lake. Please elaborate on this. Thanks.

Answer 1

Hello @Samy Abdul and welcome to Microsoft Q&A.

My understanding is that a data lake is basically just a place to store all your less-structured data ( compared to relational database ).
The Delta Lake, if I understand correctly, is a practice where in your Data Lake you focus on writing the changes made, rather than updating the data itself. Like Change Capture. So here the only reason why Delta is atomic, is that you are just appending a line stating what is changed, rather than having to update a flat file.

To any experts out there, feel free to correct me if I got something wrong.

Answer 2

In my opinion, Azure Data Lake Storage is not competing against Delta Lake, actually Delta Lake is built on top of ADLS. They serve different purposes, for example, you should always have your raw data, to reprocess, and this can reside in ADLS, but if you are already using Databricks, you can read the raw data from ADLS, transforming, and ingest into Delta Lake. You can put add as a step into your ADF our Synpase, but if you are not using Databricks, maybe you want to save money, you can have a different strategy for rollback operations, maybe you can reprocess only one day, or one hour. First, as I always says, we need to understand the business need before define the architecture.

I found the following link explaining how ADLS and Delta Lake can work together - https://techcommunity.microsoft.com/t5/analytics-on-azure/simplify-your-lakehouse-architecture-with-azure-databricks-delta/ba-p/2027272