Azure datalake and data consistency

Question

Hi Experts, Azure Data Lake Storage (ADLS) does not natively provide full ACID (Atomicity, Consistency, Isolation, Durability) transaction support unlike traditional relational databases designed to support ACID transactions. This raises the following questions:

1. How does ADLS store data consistency and avoid duplication of data?
2. Since ADLS is a file-based system and lacks data atomicity when the data load/transaction fails in the process, the partial data load takes place and there is no fail-over process due to a lack of ACID property, this might cause data duplication, then ideally ADLS shall be data swap?   
3. ADLS has eventual consistency, but does it ensure data accuracy and uniqueness?
4. Considering the above, How would you ensure data integrity, isolation, and data consistency at all times in ADLS?

Please help me understand. Thank you.

Answer 1

Hello azure_learner,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like to have more clarity about Azure datalake and data consistency.

Regarding your questions:

How does ADLS store data consistency and avoid duplication of data?

ADLS uses a combination of file system semantics, file-level security, and scale to ensure data consistency and avoid duplication but does not inherently enforce data consistency across files or prevent duplication. It depends on your configurations and tools to implement data consistency strategies in your data ingestion processes. https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-best-practices and https://delta.io

Since ADLS is a file-based system and lacks data atomicity when the data load/transaction fails in the process, the partial data load takes place and there is no fail-over process due to a lack of ACID property, this might cause data duplication, then ideally ADLS shall be data swap?   

Yes, partial data loads can lead to incomplete or duplicated data, especially when failures occur during the load process but there are many ways to mitigate this, read more in the links above and continue with: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction and https://techcommunity.microsoft.com/t5/analytics-on-azure-blog/delta-lake-on-azure/ba-p/1869746

ADLS has eventual consistency, but does it ensure data accuracy and uniqueness?

ADLS does not guarantee immediate data accuracy and uniqueness, but there is eventual consistency that data will become consistent over time by you implementing additional measures, such as data validation and deduplication processes. https://learn.microsoft.com/en-us/azure/architecture/microservices/design/data-considerations

Considering the above, How would you ensure data integrity, isolation, and data consistency at all times in ADLS?

To ensure data integrity, isolation, and consistency, you can use Delta Lake on top of ADLS, which provides ACID transaction capabilities, schema enforcement, and time travel features, and also implement data validation and consistency checks in your data processing workflows can help maintain data quality: https://learn.microsoft.com/en-us/azure/databricks/lakehouse/acid and

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.