Lakehouse deployment pipelines and git integration (Preview)
The Lakehouse integrates with the lifecycle management capabilities in Microsoft Fabric, providing a standardized collaboration between all development team members throughout the product's life. Lifecycle management facilitates an effective product versioning and release process by continuously delivering features and bug fixes into multiple environments. To learn more, see What is lifecycle management in Microsoft Fabric?.
Important
This feature is in preview.
Lakehouse git integration
The Lakehouse is an item that contains both metadata and data that is referenced in multiple objects in the workspace. Lakehouse contains tables, folders, and shortcuts as primary manageable data container items. From a development workflow perspective, the following dependent objects might reference a Lakehouse:
- Dataflows and Data Pipelines
- Spark Job Definitions
- Notebooks
- Semantic models and Power BI
The default semantic model and SQL analytics endpoint metadata are related to a Lakehouse and managed by the git update process by default. As a principle data is not tracked in git, only metadata is tracked.
Git representation
The following lakehouse information is serialized and tracked in a git connected workspace:
- Display name
- Description
- Logical guid
Note
The tracked logical guid is an automatically generated cross-workspace identifier representing an item and its source control representation.
Important
Only the Lakehouse container artifact is tracked in git in the current experience. Tables, folders, shortcuts and other metadata changes aren't tracked and versioned in git.
Lakehouse git integration capabilities
The following capabilities are available:
- Serialization of the Lakehouse object metadata to a git JSON representation.
- Apply changes directly or use pull request to control changes to upstream or downstream workspaces and branches.
- Renaming lakehouses are tracked in git. Updating a renamed lakehouse also renames the default semantic data model and SQL Analytics endpoint.
- No action is applied to tables, folders and shortcuts, metadata, and data of those items is always preserved.
Lakehouse in deployment pipelines
The Lakehouse is supported in Microsoft Fabric lifecycle management deployment pipelines. It enables environment segmentation best-practices.
Lakehouse deployment pipelines integration capabilities:
Deployment across dev, test, and production workspaces.
Lakehouse can be removed as a dependent object upon deployment. Mapping different Lakehouses within the deployment pipeline context is also supported.
If nothing is specified during deployment pipeline configuration, a new empty Lakehouse object with same name is created in the target workspace. Notebook and Spark Job Definitions are remapped to reference the new Lakehouse object in the new workspace.
If the Lakehouse dependency is configured to reference a different Lakehouse during deployment pipeline configuration time, such as the upstream Lakehouse, a new empty Lakehouse object with same name still is created in the target workspace, but Notebooks and Spark Job Definitions references are preserved to a different Lakehouse as requested.
SQL Analytics endpoints and semantic models are provisioned as part of the Lakehouse deployment.
No object inside the Lakehouse is overwritten.
Updates to Lakehouse name can be synchronized across workspaces in a deployment pipeline context.