What's new and planned for OneLake in Microsoft Fabric
Important
The release plans describe functionality that may or may not have been released yet. The delivery timelines and projected functionality may change or may not ship. Refer to Microsoft policy for more information.
OneLake is the OneDrive for data. It provides you with data lake as a service without having to build it yourself. Every Microsoft Fabric tenant will have one OneLake. There is no infrastructure to set up or manage. All the data, which lands in OneLake will automatically take part in out of the box data governance such as data lineage, data protection, certification, and catalog integration. It breaks down data silos by enabling different parts of the organization to work independently while still contributing to the same data lake.
Microsoft Fabric items store your data in OneLake in an open file format. For structured tabular data, this format is delta parquet. This format allows every analytics engine in Microsoft Fabric to access the data from other analytics engines. This way it allows flexibility for data practitioners to use the tools of your choice.
OneLake is compatible with existing ADLS gen2 applications by supporting the ADLS gen2 DFS APIs. A key capability of OneLake is "Shortcuts" (also known as Symbolic Links). OneLake allows easy sharing of data between users and applications without having to move and duplicate information. Shortcuts allow OneLake to virtualize all the data across OneLake and other lake storages in ADLS G2 and Amazon S3. It enables developers to compose and analyze data across clouds with transparency.
To learn more, see the documentation and read our announcement blog.
Investment areas
| Feature | Estimated release timeline |
|---|---|
| Smart caching for Amazon S3 shortcuts (public preview) | Shipped |
| OneLake security model for all workloads (public preview) | Q4 2024 |
| Shortcuts API (public preview) | Shipped |
| Shortcuts Google cloud storage (public preview) | Shipped |
| Data Mesh enhancements | Q4 2023 |
Smart caching for Amazon S3 shortcuts (public preview)
Shipped
Smart caching for Amazon S3 shortcuts reduces egress costs and enhances performance by bringing data closer to the compute engine. Smart caching will egress data from S3 once, cache it locally in OneLake for a certain period. This way it eliminates the need for repeated data retrieval from S3. The cached data can be reused across multiple users, analytical engines, and scenarios that optimize the value a single egress.
OneLake security model for all workloads (public preview)
Estimated release timeline: Q4 2024
Managing data security across multiple analytical engines and copies of data is challenging. OneLake and Fabric simplify this by enabling the use of a single data copy across multiple analytical engines without any data movement or duplication. Taking the "one copy" concept further, OneLake is also enhancing security with a finer-grain model, allowing for table and folder access in addition to row and column level security. These security definitions live with the data and travel across shortcuts to wherever the data is used. Security defined at OneLake is universally enforced no matter which analytical engine is used to access the data.
Shortcuts API (public preview)
Shipped
A public REST API to automate creation and management of shortcut scenarios.
Shortcuts Google cloud storage (public preview)
Shipped
OneLake will expand shortcut support to Google Cloud Storage, allowing virtualization of data without moving or duplicating it. This enables the integration of Google Cloud Storage data with Microsoft services like ADLS gen2, OneLake, Dataverse, and Amazon S3. The data will appear and work as if it was in OneLake and gives you a simple data lake that can span clouds.
Data Mesh enhancements
Estimated release timeline: Q4 2023
Domains allow organizations to structure your data per business needs and to implement a data mesh paradigm on your OneLake. We're continuing to enhance the capabilities of domains in Fabric by introducing features like:
- Delegating selected tenant settings to control at the domain-level.
- Defining sub domains.
- Announcing the General Availability of Domains in Fabric.