What's new in Microsoft Fabric? archive
This archive page is periodically updated with an archival of content from What's new in Microsoft Fabric?
To follow the latest in Fabric news and features, see the Microsoft Fabric Blog. Also follow the latest in Power BI at What's new in Power BI?
New to Microsoft Fabric?
This section includes past articles and announcements that are useful to users new to Microsoft Fabric.
- Learning Paths for Fabric
- Get started with Microsoft Fabric
- End-to-end tutorials in Microsoft Fabric
- Definitions of terms used in Microsoft Fabric
| Month | Feature | Learn more |
|---|---|---|
| March 2024 | Microsoft Fabric is now HIPAA compliant | We're excited to announce that Microsoft Fabric, our all-in-one analytics solution for enterprises, has achieved new certifications for HIPAA and ISO 27017, ISO 27018, ISO 27001, ISO 27701. |
| March 2024 | Exam DP-600 is now available | Exam DP-600 is now available, leading to the Microsoft Certified: Fabric Analytics Engineer Associate certification. The Fabric Career Hub can help you learn quickly and get certified. |
| March 2024 | Fabric Copilot Pricing: An End-to-End example | Copilot in Fabric begins billing on March 1, 2024 as part of your existing Power BI Premium or Fabric Capacity. Learn how Fabric Copilot usage is calculated. |
| January 2024 | Microsoft Fabric Copilot for Data Science and Data Engineering | Copilot for Data Science and Data Engineering is now available worldwide. What can Copilot for Data Science and Data Engineering do for you? |
| December 2023 | Fabric platform Security Fundamentals | Learn more about the big-picture perspective of the Microsoft Fabric security architecture by describing how the main security flows in the system work. |
| November 2023 | Microsoft Fabric, explained for existing Synapse users | A focus on what customers using the current Platform-as-a-Service (PaaS) version of Synapse can expect. We explain what the general availability of Fabric means for your current investments (spoiler: we fully support them), but also how to think about the future. |
| November 2023 | Microsoft Fabric is now generally available | Microsoft Fabric is now generally available for purchase. Microsoft Fabric can reshape how your teams work with data by bringing everyone together on a single, AI-powered platform built for the era of AI. This includes: Power BI, Data Factory, Data Engineering, Data Science, Real-Time Analytics, Data Warehouse, and the overall Fabric platform. |
| November 2023 | Fabric workloads are now generally available! | Microsoft Fabric is now generally available! Microsoft Fabric Synapse Data Warehouse, Data Engineering & Data Science, Real-Time Analytics, Data Factory, OneLake, and the overall Fabric platform are now generally available. |
| November 2023 | Implement medallion lakehouse architecture in Microsoft Fabric | An introduction to medallion lake architecture and how you can implement a lakehouse in Microsoft Fabric. |
| October 2023 | Announcing the Fabric roadmap | Announcing the Fabric Roadmap. One place you can see what we are working on and when you can expect it to be available. |
| October 2023 | Get started with semantic link | Explore how semantic link seamlessly connects Power BI semantic models with Synapse Data Science within Microsoft Fabric. Learn more at Semantic link in Microsoft Fabric: Bridging BI and Data Science. You can also check out the semantic link sample notebooks that are now available in the fabric-samples GitHub repository. These notebooks showcase the use of semantic link's Python library, SemPy, in Microsoft Fabric. |
| September 2023 | Fabric Capacities – Everything you need to know about what's new and what's coming | Read more about the improvements we're making to the Fabric capacity management platform for Fabric and Power BI users. |
| August 2023 | Accessing Microsoft Fabric for developers, startups and enterprises! | Learn how to enable Microsoft Fabric as a developer, as a startup or as an enterprise has different steps. Learn more at Enabling Microsoft Fabric for developers, startups, and enterprises. |
| August 2023 | Strong, useful, beautiful: Designing a new way of getting data | From the Data Integration Design Team, learn about the strong, creative, and function design of Microsoft Fabric, as Microsoft designs for the future of data integration. |
| August 2023 | Learn Live: Get started with Microsoft Fabric | Calling all professionals, enthusiasts, and learners! On August 29, we'll be kicking off the "Learn Live: Get started with Microsoft Fabric" series in partnership with Microsoft's Data Advocacy teams and Microsoft WorldWide Learning teams to deliver 9x live-streamed lessons covering topics related to Microsoft Fabric! |
| July 2023 | Step-by-Step Tutorial: Building ETLs with Microsoft Fabric | In this comprehensive guide, we walk you through the process of creating Extract, Transform, Load (ETL) pipelines using Microsoft Fabric. |
| June 2023 | Get skilled on Microsoft Fabric - the AI-powered analytics platform | Who is Fabric for? How can I get skilled? This blog post answers these questions about Microsoft Fabric, a comprehensive data analytics solution by unifying many experiences on a single platform. |
| June 2023 | Introducing the end-to-end scenarios in Microsoft Fabric | In this blog, we explore four end-to-end scenarios that are typical paths our customers take to extract value and insights from their data using Microsoft Fabric. |
| May 2023 | Get Started with Microsoft Fabric - All in-one place for all your Analytical needs | A technical overview and introduction to everything from data movement to data science, real-time analytics, and business intelligence in Microsoft Fabric. |
| May 2023 | Microsoft OneLake in Fabric, the OneDrive for data | Microsoft OneLake brings the first multicloud SaaS data lake for the entire organization. |
Generally available features
The following table lists the features of Microsoft Fabric that have transitioned from preview to general availability (GA).
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | OneLake availability of Eventhouse in Delta Lake format | As part of the One logical copy promise, we're excited to announce that OneLake availability of Eventhouse in Delta Lake format is Generally Available. |
| May 2024 | Microsoft Fabric Private Links | Azure Private Link for Microsoft Fabric secures access to your sensitive data in Microsoft Fabric by providing network isolation and applying required controls on your inbound network traffic. For more information, see Announcing General Availability of Fabric Private Links. |
| May 2024 | Trusted workspace access | Trusted workspace access in OneLake shortcuts is now generally available. You can now create data pipelines to access your firewall-enabled Azure Data Lake Storage Gen2 (ADLS Gen2) accounts using Trusted workspace access (preview) in your Fabric Data Pipelines. Use the workspace identity to establish a secure and seamless connection between Fabric and your storage accounts. Trusted workspace access also enables secure and seamless access to ADLS Gen2 storage accounts from OneLake shortcuts in Fabric. |
| May 2024 | Managed private endpoints | Managed private endpoints for Microsoft Fabric allow secure connections over managed virtual networks to data sources that are behind a firewall or not accessible from the public internet. For more information, see Announcing General Availability of Fabric Private Links, Trusted Workspace Access, and Managed Private Endpoints. |
| May 2024 | Eventhouse | Eventhouse is a new, dynamic workspace hosting multiple KQL databases, generally available as part of Fabric Real-Time Intelligence. An Eventhouse offers a robust solution for managing and analyzing substantial volumes of real-time data. Get started with a guide to Create and manage an Eventhouse. |
| May 2024 | Data Engineering: Environment | The Environment in Fabric is now generally available. The Environment is a centralized item that allows you to configure all the required settings for running a Spark job in one place. At GA, we added support for Git, deployment pipelines, REST APIs, resource folders, and sharing. |
| May 2024 | Microsoft Fabric Core REST APIs | Microsoft Fabric Core APIs are now generally available. The Fabric user APIs are a major enabler for both enterprises and partners to use Microsoft Fabric as they enable end-to-end fully automated interaction with the service, enable integration of Microsoft Fabric into external web applications, and generally enable customers and partners to scale their solutions more easily. |
| May 2024 | Power Query Dataflow Gen2 SDK for VS Code | The Power Query SDK is now generally available in Visual Studio Code! To get started with the Power Query SDK in Visual Studio Code, install it from the Visual Studio Code Marketplace. |
| April 2024 | Semantic Link | Semantic links are now generally available! The package comes with our default VHD, and you can now use Semantic link in Fabric right away without any pip installation. |
| March 2024 | VNet Gateways in Dataflow Gen2 | VNet Data Gateway support for Dataflows Gen2 in Fabric is now generally available. The VNet data gateway helps to connect from Fabric Dataflows Gen2 to Azure data services within a VNet, without the need of an on-premises data gateway. |
| November 2023 | Microsoft Fabric is now generally available | Microsoft Fabric is now generally available for purchase. Microsoft Fabric can reshape how your teams work with data by bringing everyone together on a single, AI-powered platform built for the era of AI. This includes: Power BI, Data Factory, Data Engineering, Data Science, Real-Time Analytics, Data Warehouse, and the overall Fabric platform. |
Community
This section summarizes previous Microsoft Fabric community opportunities for prospective and current influencers and MVPs. To learn about the Microsoft MVP Award and to find MVPs, see mvp.microsoft.com.
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Solved Fabric Community posts are now available in the Fabric Help Pane | You can now find solved posts from Fabric Community discussions in the Fabric Help Pane. |
| May 2024 | Announcing Microsoft Fabric Community Conference Europe | Announcing the Microsoft Fabric Community Conference Europe on September 24, 2024. Register today! |
| May 2024 | Register for the Microsoft Build: Microsoft Fabric Cloud Skills Challenge | Starting May 21, 2024, sign up for the Microsoft Build: Microsoft Fabric Cloud Skills Challenge and prepare for Exam DP-600 and upskill to the Fabric Analytics Engineer Associate certification. |
| March 2024 | Exam DP-600 is now available | Exam DP-600 is now available, leading to the Microsoft Certified: Fabric Analytics Engineer Associate certification. The Fabric Career Hub can help you learn quickly and get certified. |
| March 2024 | Microsoft Fabric Community Conference | Join us in Las Vegas March 26-28, 2024 for the first annual Microsoft Fabric Community Conference. See firsthand how Microsoft Fabric and the rest of the data and AI products at Microsoft can help your organization prepare for the era of AI. Register today using code MSCUST for an exclusive discount! |
| March 2024 | Announcing the winners of "HackTogether: The Microsoft Fabric Global AI Hack" | We received 50 Hackathon project submissions from over 100 registrants, participating from every corner of the world. Our judges were blown away by the breadth, depth, and overall quality of submissions. Meet the winners of the Fabric Global AI Hack! |
| January 2024 | Announcing Fabric Career Hub | The new Fabric Career Hub is your one-stop-shop for professional growth! We've created a comprehensive learning journey with the best free on-demand and live training, plus exam discounts. |
| January 2024 | Hack Together: The Microsoft Fabric Global AI Hack | Hack Together is a global online hackathon that runs from February 15 to March 4, 2024. Join us for Hack Together: The Microsoft Fabric Global AI Hack, a virtual event where you can learn, experiment, and hack together with the new Copilot and AI features in Microsoft Fabric! For more information, see Microsoft Fabric Global AI Hack. |
| December 2023 | Microsoft Fabric Community Conference | Join us in Las Vegas March 26-28, 2024 for the first annual Microsoft Fabric Community Conference. See firsthand how Microsoft Fabric and the rest of the data and AI products at Microsoft can help your organization prepare for the era of AI. Register today to immerse yourself in the future of data and AI and connect with thousands of data innovators like yourself eager to share their insights. |
| November 2023 | Microsoft Fabric MVP Corner – Special Edition (Ignite) | A special edition of the "Microsoft Fabric MVP Corner" blog series highlights selected content related to Fabric and created by MVPs around the Microsoft Ignite 2023 conference, when we announced Microsoft Fabric generally available. |
| October 2023 | Microsoft Fabric MVP Corner – October 2023 | Highlights of selected content related to Fabric and created by MVPs from October 2023. |
| September 2023 | Microsoft Fabric MVP Corner – September 2023 | Highlights of selected content related to Fabric and created by MVPs from September 2023. |
| August 2023 | Microsoft Fabric MVP Corner – August 2023 | Highlights of selected content related to Fabric and created by MVPs from August 2023. |
| July 2023 | Microsoft Fabric MVP Corner – July 2023 | Highlights of selected content related to Fabric and created by MVPs in July 2023. |
| June 2023 | Microsoft Fabric MVP Corner – June 2023 | The Fabric MVP Corner blog series to highlight selected content related to Fabric and created by MVPs in June 2023. |
| May 2023 | Fabric User Groups | Power BI User Groups are now Fabric User Groups! |
| May 2023 | Learn about Microsoft Fabric from MVPs | Prior to our official announcement of Microsoft Fabric at Build 2023, MVPs had the opportunity to familiarize themselves with the product. For several months, they have been actively testing Fabric and gaining valuable insights. Now, their enthusiasm for the product is evident as they eagerly share their knowledge and thoughts about Microsoft Fabric with the community. |
Fabric samples and guidance
This section summarizes archived guidance and sample project resources for Microsoft Fabric.
| Month | Feature | Learn more |
|---|---|---|
| March 2024 | Protect PII information in your Microsoft Fabric Lakehouse with Responsible AI | One possible way to use Azure AI to identify and extract personally identifiable information (PII) in Microsoft Fabric is to use Azure AI Language to detect and categorize PII entities in text data, such as names, addresses, emails, phone numbers, social security numbers, etc. |
| February 2024 | Building Common Data Architectures with OneLake in Microsoft Fabric | Read more about common data architecture patterns and how they can be secured with Microsoft Fabric, and the basic building blocks of security for OneLake. |
| January 2024 | New Fabric certification and Fabric Career Hub | Beta availability of Microsoft Certification Exam DP-600: Implementing Analytics Solutions with Microsoft Fabric is available for a limited time. Passing this exam earns the Microsoft Certified: Fabric Analytics Engineer Associate certification. |
| December 2023 | Working with OneLake using Azure Storage Explorer | If you want to use an application that directly integrates with Windows File Explorer, check out OneLake file explorer. However, if you're accustomed to using Azure Storage Explorer for your data management tasks, you can continue to harness its functionalities with OneLake and some of its key benefits. |
| November 2023 | Semantic Link: OneLake integrated Semantic Models | Semantic Link adds support for the recently released OneLake integrated semantic models. You can now directly access data using your semantic model's name via OneLake using the read_table function and the new mode parameter set to onelake. |
| November 2023 | Integrate your SAP data into Microsoft Fabric | Using the built-in connectivity of Microsoft Fabric is the easiest and least-effort way of adding SAP data to your Fabric data estate. |
| November 2023 | Fabric Changing the game: Validate dependencies with Semantic Link – Data Quality | Follow this step-by-step example of how to explore the functional dependencies between columns in a table using the semantic link. The semantic link is a feature that allows you to establish a connection between Power BI datasets and Synapse Data Science in Microsoft Fabric. |
| November 2023 | Implement medallion lakehouse architecture in Microsoft Fabric | An introduction to medallion lake architecture and how you can implement a lakehouse in Microsoft Fabric. |
| October 2023 | Fabric Change the Game: Exploring the data | Follow this realistic example of reading data from Azure Data Lake Storage using shortcuts, organizing raw data into structured tables, and basic data exploration. Our data exploration uses as a source the diverse and captivating city of London with information extracted from data.london.gov.uk/. |
| September 2023 | Announcing an end-to-end workshop: Analyzing Wildlife Data with Microsoft Fabric | A new workshop guides you in building a hands-on, end-to-end data analytics solution for the Snapshot Serengeti dataset using Microsoft Fabric. The dataset consists of approximately 1.68M wildlife images and image annotations provided in .json files. |
| September 2023 | New learning path: Implement a Lakehouse with Microsoft Fabric | The new Implement a Lakehouse with Microsoft Fabric learning path introduces the foundational components of implementing a data lakehouse with Microsoft Fabric with seven in-depth modules. |
| September 2023 | Fabric Readiness repository | The Fabric Readiness repository is a treasure trove of resources for anyone interested in exploring the exciting world of Microsoft Fabric. |
| July 2023 | Connecting to OneLake | How do I connect to OneLake? This blog covers how to connect and interact with OneLake, including how OneLake achieves its compatibility with any tool used over ADLS Gen2! |
| June 2023 | Using Azure Databricks with Microsoft Fabric and OneLake | How does Azure Databricks work with Microsoft Fabric? This blog post answers that question and more details on how the two systems can work together. |
| July 2023 | Free preview usage of Microsoft Fabric experiences extended to October 1, 2023 | We're extending the free preview usage of Fabric experiences (other than Power BI). These experiences won't count against purchased capacity until October 1, 2023. |
Microsoft Copilot in Microsoft Fabric
This section summarizes archived announcements about Copilot in Fabric.
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Copilot privacy and security | For more information on the privacy and security of Copilot in Microsoft Fabric, and for detail information on each workload, see Privacy, security, and responsible use for Copilot in Microsoft Fabric (preview). |
| May 2024 | The AI and Copilot setting automatically delegated to capacity admins | In the tenant admin portal, you can delegate the enablement of AI and Copilot features to Capacity administrators. This AI and Copilot setting is automatically delegated to capacity administrators and tenant administrators won't be able to turn off the delegation. |
| February 2024 | Fabric Change the Game: How easy is it to use Copilot in Microsoft Fabric | This blog post shows how simple is to enable Copilot, a generative AI that brings new ways to transform and analyze data, generate insights, and create visualizations and reports in Microsoft Fabric. |
| February 2024 | Copilot for Data Factory in Microsoft Fabric | Copilot for Data Factory in Microsoft Fabric is now available in preview and included in the Dataflow Gen2 experience. For more information, see Copilot for Data Factory. |
| January 2024 | Microsoft Fabric Copilot for Data Science and Data Engineering | Copilot for Data Science and Data Engineering is now available worldwide. What can Copilot for Data Science and Data Engineering do for you? |
| January 2024 | How to enable Copilot in Fabric for Everyone | Follow this guide to get Copilot in Fabric enabled for everyone in your organization. For more information, see Overview of Copilot for Microsoft Fabric (preview). |
| January 2024 | Copilot in Fabric is available worldwide | Copilot in Fabric is now available to all customers, including Copilot for Power BI, Data Factory, and Data Science & Data Engineering. Read more in our Overview on Copilot in Fabric. |
| November 2023 | Empower Power BI users with Microsoft Fabric and Copilot | We're thrilled to announce the general availability of Microsoft Fabric and the preview of Copilot in Microsoft Fabric, including the experience for Power BI.. |
| November 2023 | Copilot for Power BI in Microsoft Fabric preview | We're thrilled to announce the preview of Copilot in Microsoft Fabric, including the experience for Power BI, which helps users quickly get started by helping them create reports in the Power BI web experience. For more information, see Copilot for Power BI. |
| October 2023 | Chat your data in Microsoft Fabric with Semantic Kernel | Learn how to construct Copilot tools based on business data in Microsoft Fabric. |
Data Factory in Microsoft Fabric
This section summarizes archived new features and capabilities of Data Factory in Microsoft Fabric. Follow issues and feedback through the Data Factory Community Forum.
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Dataflow Gen2 certified connector updates | New and updated Dataflow Gen2 connectors have been released. For more information, see the June 2024 Certified connector updates. |
| June 2024 | New data pipeline connector updates | More connectors are now available for data pipeline. For more information, see the June 2024 Fabric update. |
| June 2024 | Lakehouse schemas feature | The Lakehouse schemas feature (preview) introduces data pipeline support for reading the schema info from Lakehouse tables and supports writing data into tables under specified schemas. Lakehouse schemas allow you to group your tables together for better data discovery, access control, and more. |
| June 2024 | Move Data Across Workspace via Data pipeline Modern Get Data Experience | You can now move data among Lakehouses, warehouses, etc. across different workspaces. In Pipeline Modern Get Data, select a Fabric item from another workspace under Explorer on the left side of the OneLake data hub. |
| June 2024 | Create a new Warehouse as destination in Data pipeline | You can now create a new Warehouse as a destination in Data Pipeline, instead of only selecting an existing one. |
| May 2024 | Data Factory Announcements at Microsoft Build Recap | Don't miss any of the Data Factory in Fabric announcements, here's a recap of all new features in Data Factory in Fabric from Build 2024. |
| May 2024 | New certified connectors | The Power Query SDK and Power Query Connector Certification process has introduced several new Power Query connectors, including connectors for Oracle database, MySQL, Oracle Cloud Storage, Azure AI, Azure Files, Dynamics AX, Google Bigquery, Snowflake ADBC, and more coming soon. |
| May 2024 | API for GraphQL in Microsoft Fabric (preview) | The new API for GraphQL is a data access layer that allows us to query multiple data sources quickly and efficiently in Fabric. For more information, see What is Microsoft Fabric API for GraphQL? |
| May 2024 | Power Query Dataflow Gen2 SDK for VS Code GA | The Power Query SDK is now generally available in Visual Studio Code! To get started with the Power Query SDK in Visual Studio Code, install it from the Visual Studio Code Marketplace. |
| May 2024 | Refresh the Refresh History Dialog | The Refresh History details popup window now has a Refresh button. |
| May 2024 | New and updated certified connectors | The Power Query SDK and Power Query Connector Certification process has introduced four new and updated Power Query connectors. |
| May 2024 | Data workflows in Data Factory preview | Data workflows (preview) in Data Factory, powered by Apache Airflow, offer seamless authoring, scheduling, and monitoring experience for Python-based data processes defined as Directed Acyclic Graphs (DAGs). For more information, see Quickstart: Create a Data workflow. |
| May 2024 | Trusted Workspace Access in Fabric Data Pipelines preview | Use the workspace identity to establish a secure and seamless connection between Fabric and your storage accounts. You can now create data pipelines to access your firewall-enabled Azure Data Lake Storage Gen2 (ADLS Gen2) accounts using Trusted workspace access (preview) in your Fabric Data Pipelines. |
| May 2024 | Blob storage Event Triggers for Data Pipelines preview | Azure Blob storage event triggers (preview) in Fabric Data Factory Data Pipelines use Fabric Reflex alerts and event streams to create event subscriptions to your Azure storage accounts. |
| May 2024 | Azure HDInsight activity for data pipelines | The Azure HDInsight activity allows you to execute Hive queries, invoke a MapReduce program, execute Pig queries, execute a Spark program, or a Hadoop Stream program. |
| May 2024 | Copy data assistant | Start using the Modern Get Data experience by selecting Copy data assistant in the Pipeline landing page or Use copy assistant in the Copy data drop down. You can easily connect to recently used Fabric items and provides an intuitive way to read sources from sample data and new connections. |
| May 2024 | Edit the Destination Table Column Type when Copying Data | You can edit destination table column types when copying data for a new or autocreated destination table for many data stores. For more information, see Configure Lakehouse in a copy activity. |
| April 2024 | Spark job definition activity | With the new Spark job definition activity, you'll be able to run a Spark job definition in your pipeline. |
| April 2024 | Fabric Warehouse in ADF copy activity | You can now connect to your Fabric Warehouse from an Azure Data Factory/Synapse pipeline. You can find this new connector when creating a new source or sink destination in your copy activity, in the Lookup activity, Stored Procedure activity, Script activity, and Get Metadata activity. |
| April 2024 | Edit column type to destination table support added to Fabric Warehouse and other SQL data stores | When moving data from any supported data sources into Fabric Warehouse or other SQL data stores (SQL Server, Azure SQL Database, Azure SQL Managed Instance, or Azure Synapse Analytics) via data pipelines, users can now specify the data type for each column. |
| April 2024 | Performance improvements when writing data to SFTP | The SFTP connector has been improved to offer better performance when writing to SFTP as destination. |
| April 2024 | Service Principal Name authentication kind support for On-Premises and virtual network data gateways | Azure Service Principals (SPN) are now supported for on-premises data gateways and virtual network data gateways. Learn how to use the service principal authentication kind in Azure Data Lake Storage, Dataverse, Azure SQL Database, Web connector, and more. |
| April 2024 | New and updated Certified connectors | The Power Query SDK and Power Query Connector Certification process has introduced 11 new and updated custom Power Query connectors. |
| April 2024 | New Expression Builder Experience | A new experience in the Script activity in Fabric Data Factory pipelines to make it even easier to build expressions using the pipeline expression language. |
| April 2024 | Data Factory Increases Maximum Activities Per Pipeline to 80 | We have doubled the limit on number of activities you can define in a pipeline from 40 to 80. |
| April 2024 | REST APIs for Fabric Data Factory pipelines preview | The REST APIs for Fabric Data Factory Pipelines are now in preview. REST APIs for Data Factory pipelines enable you to extend the built-in capability in Fabric to create, read, update, delete, and list pipelines. |
| March 2024 | Fast copy in Dataflows Gen2 | With Fast copy, you can ingest terabytes of data with the easy experience of dataflows, but with the scalable backend of Pipeline's Copy activity. |
| March 2024 | Integrating On-Premises Data into Microsoft Fabric Using Data Pipelines in Data Factory preview | With the on-premises Data Gateway (preview), customers can connect to on-premises data sources using dataflows and data pipelines with Data Factory. For more information, see How to access on-premises data sources in Data Factory for Microsoft Fabric. |
| March 2024 | CI/CD for Fabric Data Pipelines preview | Git Integration and integration with built-in Deployment Pipelines to Data Factory data pipelines is now in preview. For more information, see Data Factory Adds CI/CD to Fabric Data Pipelines. |
| March 2024 | Browse Azure resources with Get Data | Learn how to browse and connect to all your Azure resources with the 'browse Azure' functionality in Get Data. You can browse Azure resources then connect to Synapse, blob storage, or ADLS Gen2 resources easily. |
| March 2024 | Dataflow Gen2 Support for VNet Gateways now generally available | VNet Data Gateway support for Dataflows Gen2 in Fabric is now generally available. The VNet data gateway helps to connect from Fabric Dataflows Gen2 to Azure data services within a VNet, without the need of an on-premises data gateway. |
| March 2024 | Privacy levels support in Dataflows | You can now set privacy levels for your connections in your Dataflow Gen2. Privacy levels are critical to configure correctly so that sensitive data is only viewed by authorized users. |
| March 2024 | Copy data to S3 Compatible via Fabric Data Factory Data Pipeline | Copying data to S3 Compatible is now available in Data pipeline of Fabric Data Factory! You can use Copy assistant and Copy activity in your Data pipeline to finish this data movement. |
| February 2024 | Dataflows Gen2 data destinations and managed settings | New features for Dataflows Gen2 include destinations, managed settings, and advanced topics. |
| February 2024 | Copilot for Data Factory in Microsoft Fabric | Copilot for Data Factory in Microsoft Fabric is now available in preview and included in the Dataflow Gen2 experience. For more information, see Copilot for Data Factory. |
| February 2024 | Certified Connector updates | The Power Query SDK enables you to create new connectors for both Power BI and Dataflow. New certified Power Query connectors are available to the list of Certified Connectors in Power Query. |
| February 2024 | Data pipeline connector updates | New connectors are available in your Data Factory data pipelines, including S3 compatible and Google Cloud Storage data sources. For more information, see Data pipeline connectors in Microsoft Fabric. |
| January 2024 | Automate Fabric Data Warehouse Queries and Commands with Data Factory | In Fabric Data Factory, there are many ways to query data, retrieve data, and execute commands from your warehouse using pipeline activities that can then be easily automated. |
| January 2024 | Use Fabric Data Factory Data Pipelines to Orchestrate Notebook-based Workflows | Guidance and good practices when building Fabric Spark Notebook workflows using Data Factory in Fabric with data pipelines. |
| December 2023 | Read and Write to the Fabric Lakehouse using Azure Data Factory (ADF) | You can now read and write data in the Microsoft Fabric Lakehouse from ADF (Azure Data Factory). Using either Copy Activity or Mapping Data Flows, you can read, write, transform, and process data using ADF or Synapse Analytics, currently in preview. |
| December 2023 | Set activity state for easy pipeline debugging | In Fabric Data Factory data pipelines, you can now set an activity's state to inactive so that you can save your pipeline even with incomplete, invalid configurations. Think of it as "commenting out" part of your pipeline code. |
| December 2023 | Connection editing in pipeline editor | You can now edit your existing data connections while you're designing your pipeline without leaving the pipeline editor! When setting your connection, select Edit and a pop-up appears. |
| December 2023 | Azure Databricks Notebook executions in Fabric Data Factory | You can now create powerful data pipeline workflows that include Notebook executions from your Azure Databricks clusters using Fabric Data Factory. Add a Databricks activity to your pipeline, point to your existing cluster, or request a new cluster, and Data Factory will execute your Notebook code for you. |
| November 2023 | Implement medallion lakehouse architecture in Microsoft Fabric | An introduction to medallion lake architecture and how you can implement a lakehouse in Microsoft Fabric. |
| November 2023 | Dataflow Gen2 General availability of Fabric connectors | The connectors for Lakehouse, Warehouse, and KQL Database are now generally available. We encourage you to use these connectors when trying to connect to data from any of these Fabric workloads. |
| November 2023 | Dataflow Gen2 Automatic refresh cancellation | To prevent unnecessary resources from being consumed, there's a new mechanism that stops the refresh of a Dataflow as soon as the results of the refresh are known to have no impact. This is to reduce consumption more proactively. |
| November 2023 | Dataflow Gen2 Error message propagation through gateway | We made diagnostics improvement to provide meaningful error messages when Dataflow refresh fails for those Dataflows running through the Enterprise Data Gateway. |
| November 2023 | Dataflow Gen2 Support for column binding for SAP HANA connector | Column binding support is enabled for SAP HANA. This optional parameter results in significantly improved performance. For more information, see Support for column binding for SAP HANA connector. |
| November 2023 | Dataflow Gen2 staging artifacts hidden | When using a Dataflow Gen2 in Fabric, the system will automatically create a set of staging artifacts. Now, these staging artifacts will be abstracted from the Dataflow Gen2 experience and will be hidden from the workspace list. No action is required by the user and this change has no impact on existing Dataflows. |
| November 2023 | Dataflow Gen2 Support for VNet Gateways preview | VNet Data Gateway support for Dataflows Gen2 in Fabric is now in preview. The VNet data gateway helps to connect from Fabric Dataflows Gen2 to Azure data services within a VNet, without the need of an on-premises data gateway. |
| November 2023 | Cross workspace "Save as" | You can now clone your data pipelines across workspaces by using the "Save as" button. |
| November 2023 | Dynamic content flyout integration with Email and Teams activity | In the Email and Teams activities, you can now add dynamic content with ease. With this new pipeline expression integration, you'll now see a flyout menu to help you select and build your message content quickly without needing to learn the pipeline expression language. |
| November 2023 | Copy activity now supports fault tolerance for Fabric Data Warehouse connector | The Copy activity in data pipelines now supports fault tolerance for Fabric Warehouse. Fault tolerance allows you to handle certain errors without interrupting data movement. By enabling fault tolerance, you can continue to copy data while skipping incompatible data like duplicated rows. |
| November 2023 | MongoDB and MongoDB Atlas connectors | MongoDB and MongoDB Atlas connectors are now available to use in your Data Factory data pipelines as sources and destinations. |
| November 2023 | Microsoft 365 connector now supports ingesting data into Lakehouse (preview) | The Microsoft 365 connector now supports ingesting data into Lakehouse tables. |
| November 2023 | Multi-task support for editing pipelines in the designer | You can now open and edit data pipelines from different workspaces and navigate between them using the multi-tasking capabilities in Fabric. |
| November 2023 | String interpolation added to pipeline return value | You can now edit your data connections within your data pipelines. Previously, a new tab would open when connections needed editing. Now, you can remain within your pipeline and seamlessly update your connections. |
| October 2023 | Category redesign of activities | We've redesigned the way activities are categorized to make it easier for you to find the activities you're looking for with new categories like Control flow, Notifications, and more. |
| October 2023 | Copy runtime performance improvement | We've made improvements to the Copy runtime performance. According to our tests results, with the improvements users can expect to see the duration of copying from parquet/csv files into Lakehouse table to improve by ~25%-35%. |
| October 2023 | Integer data type available for variables | We now support variables as integers! When creating a new variable, you can now choose to set the variable type to Integer, making it easier to use arithmetic functions with your variables. |
| October 2023 | Pipeline name now supported in System variables. | We've added a new system variable called Pipeline Name so that you can inspect and pass the name of your pipeline inside of the pipeline expression editor, enabling a more powerful workflow in Fabric Data Factory. |
| October 2023 | Support for Type editing in Copy activity Mappings | You can now edit column types when you land data into your Lakehouse tables. This makes it easier to customize the schema of your data in your destination. Simply navigate to the Mapping tab, import your schemas, if you don't see any mappings, and use the dropdown list to make changes. |
| October 2023 | New certified connector: Emplifi Metrics | Announcing the release of the new Emplifi Metrics connector. The Power BI Connector is a layer between Emplifi Public API and Power BI itself. For more information, see Emplifi Public API documentation. |
| October 2023 | SAP HANA (Connector Update) | The update enhances the SAP HANA connector with the capability to consume HANA Calculation Views deployed in SAP Datasphere by taking into account SAP Datasphere's additional security concepts. |
| October 2023 | Set Activity State to "Comment Out" Part of Pipeline | Activity State is now available in Fabric Data Factory data pipelines, giving you the ability to comment out part of your pipeline without deleting the definition. |
| August 2023 | Staging labels | The concept of staging data was introduced in Dataflows Gen2 for Microsoft Fabric and now you have the ability to define what queries within your Dataflow should use the staging mechanisms or not. |
| August 2023 | Secure input/output for logs | We've added advanced settings for the Set Variable activity called Secure input and Secure output. When you enable secure input or output, you can hide sensitive information from being captured in logs. |
| August 2023 | Pipeline run status added to Output panel | We've recently added Pipeline status so that developers can easily see the status of the pipeline run. You can now view your Pipeline run status from the Output panel. |
| August 2023 | Data pipelines FTP connector | The FTP connector is now available to use in your Data Factory data pipelines in Microsoft Fabric. Look for it in the New connection menu. |
| August 2023 | Maximum number of entities in a Dataflow | The new maximum number of entities that can be part of a Dataflow has been raised to 50. |
| August 2023 | Manage connections feature | The Manage Connections option now allows you to view the linked connections to your dataflow, unlink a connection, or edit connection credentials and gateway. |
| August 2023 | Power BI Lakehouses connector | An update to the Lakehouses connector in the August version of the Power BI Desktop and Gateway includes significant performance improvements. |
| July 2023 | New modern data connectivity and discovery experience in Dataflows | An improved experience aims to expedite the process of discovering data in Dataflow, Dataflow Gen2, and Datamart. |
| May 2023 | Introducing Data Factory in Microsoft Fabric | Data Factory enables you to develop enterprise-scale data integration solutions with next-generation dataflows and data pipelines. |
Data Factory in Microsoft Fabric samples and guidance
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | A Data Factory Pipeline Navigator mind map | The ultimate Data Factory Pipeline Mind Map helps you navigate Data Factory pipelines on your Data Factory journey to build a successful Data Integration project. |
| May 2024 | Semantic model refresh activity | Learn how to use the much-requested Semantic model refresh activity in Data pipelines and how you can now create a complete end-to-end solution that spans the entire pipeline lifecycle. |
| February 2024 | Fabric Data Pipelines – Advanced Scheduling Techniques | This blog series covers Advanced Scheduling techniques in Microsoft Fabric Data Pipelines. |
| December 2023 | Read data from Delta Lake tables with the DeltaLake.Table M function | The DeltaLake.Table is a new function in Power Query's M language for reading data from Delta Lake tables. This function is now available in Power Query in Power BI Desktop and in Dataflows Gen1 and Gen2 and replaces the need to use community-developed solution. |
| October 2023 | Microsoft Fabric Data Factory Webinar Series – October 2023 | You're invited to join our October webinar series, where we'll show you how to use Data Factory to transform and orchestrate your data in various scenarios. |
| September 2023 | Notify Outlook and Teams channel/group from a Microsoft Fabric pipeline | Learn how to send notifications to both Teams channels/groups and Outlook emails. |
| September 2023 | Microsoft Fabric Data Factory Webinar Series – September 2023 | Join our Data Factory webinar series where we'll show you how to use Data Factory to transform and orchestrate your data in various scenarios. |
| August 2023 | Metadata Driven Pipelines for Microsoft Fabric – Part 2, Data Warehouse Style | An overview of a metadata-driven pipeline in Microsoft Fabric that follows the medallion architecture with Data Warehouse serving as the Gold layer. |
| August 2023 | Metadata Driven Pipelines for Microsoft Fabric | An overview of a Metadata driven pipeline in Microsoft Fabric that follows the medallion architecture (Bronze, Silver, Gold). |
| August 2023 | Using Data pipelines for copying data to/from KQL Databases and crafting workflows with the Lookup activity | Real-Time Intelligence' KQL DB is supported as both a destination and a source with data pipelines, allowing you to build and manage various extract, transform, and load (ETL) activities, leveraging the power and capabilities of KQL DBs. |
| August 2023 | Incrementally amass data | With Dataflows Gen2 that comes with support for data destinations, you can setup your own pattern to load new data incrementally, replace some old data, and keep your reports up to date with your source data. |
| August 2023 | Data Pipeline Performance Improvement Part 3: Gaining more than 50% improvement for Historical Loads | Learn how to account for pagination given the current state of Fabric Data Pipelines in preview. This pipeline is performant when the number of paginated pages isn't too large. Read more at Gaining more than 50% improvement for Historical Loads. |
| August 2023 | Data Pipeline Performance Improvements Part 2: Creating an Array of JSONs | Examples from this blog series include how to merge two arrays into an array of JSON objects, and how to take a date range and create multiple subranges then store these as an array of JSONs. Read more at Creating an Array of JSONs. |
| July 2023 | Data Pipeline Performance Improvements Part 1: How to convert a time interval (dd.hh:mm:ss) into seconds | Part one of a series of blogs on moving data with multiple Copy Activities moving smaller volumes in parallel: How to convert a time interval (dd.hh:mm:ss) into seconds. |
| July 2023 | Construct a data analytics workflow with a Fabric Data Factory data pipeline | A blog covering data pipelines in Data Factory and the advantages you find by using pipelines to orchestrate your Fabric data analytics projects and activities. |
| July 2023 | Data Pipelines Tutorial: Ingest files into a Lakehouse from a REST API with pagination ft. AVEVA Data Hub | In this blog, we will act in the persona of an AVEVA customer who needs to retrieve operations data from AVEVA Data Hub into a Microsoft Fabric Lakehouse. |
| July 2023 | Data Factory Spotlight: Dataflow Gen2 | This blog spotlight covers the two primary high-level features Data Factory implements: dataflows and pipelines. |
Synapse Data Engineering in Microsoft Fabric
This section summarizes archived new features and capabilities of data engineering, including Data Factory in Microsoft Fabric.
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Fabric Spark connector for Fabric Synapse Data Warehouse in Spark runtime (preview) | The Fabric Spark connector for Synapse Data Warehouse (preview) enables a Spark developer or a data scientist to access and work on data from a warehouse or SQL analytics endpoint of the lakehouse (either from within the same workspace or from across workspaces) with a simplified Spark API. |
| June 2024 | External data sharing public API preview | REST APIs for OneLake external data sharing are now available in preview. Users can now scale their data sharing use cases by automating the creation of shares with the public API. |
| June 2024 | Capacity pools preview | Capacity administrators can now create custom pools (preview) based on their workload requirements, providing granular control over compute resources. Custom pools for Data Engineering and Data Science can be set as Spark Pool options within Workspace Spark Settings and environment items. |
| June 2024 | Native Execution Engine for Apache Spark | The Native Execution Engine for Apache Spark on Fabric Data Engineering and Data Science for Fabric Runtime 1.2 is now in preview. For more information, see Native execution engine for Fabric Spark. |
| June 2024 | OneLake data access roles API | Following the release of OneLake data access roles in preview, new APIs are available for managing data access roles. These APIs can be used to programmatically manage granular data access for your lakehouses. |
| May 2024 | Runtime 1.3 (Apache Spark 3.5, Delta Lake 3.1, R 4.3.3, Python 3.11) (preview) | The enhancements in Fabric Runtime 1.3 include the incorporation of Delta Lake 3.1, compatibility with Python 3.11, support for Starter Pools, integration with Environment, and library management capabilities. Additionally, Fabric Runtime now enriches the data science experience by supporting the R language and integrating Copilot. |
| May 2024 | Spark Run Series Analysis and Autotune feature preview | The Spark Monitoring Run Series Analysis features allow you to analyze the run duration trend and performance comparison for Pipeline Spark activity recurring run instances and repetitive Spark run activities, from the same Notebook or Spark Job Definition. |
| May 2024 | OneLake shortcuts to on-premises and network-restricted data sources (preview) | Connect to on-premises data sources with a Fabric on-premises data gateway on a machine in your environment, with networking visibility of your S3 compatible, Amazon S3, or Google Cloud Storage data source. Then, you create your shortcut and select that gateway. For more information, see Create shortcuts to on-premises data. |
| May 2024 | Comment @tagging in Notebook | Notebook now supports the ability to tag others in comments, just like the familiar functionality of using Office products. |
| May 2024 | Notebook ribbon upgrades | New features in the Fabric notebook ribbon including the Session connect control and Data Wrangler button on the Home tab, High concurrency sessions, new View session information control including the session timeout. |
| May 2024 | Data Engineering: Environment GA | The Environment in Fabric is now generally available. The Environment is a centralized item that allows you to configure all the required settings for running a Spark job in one place. At GA, we added support for Git, deployment pipelines, REST APIs, resource folders, and sharing. |
| May 2024 | Public API for Workspace Data Engineering/Science | REST API support for Fabric Data Engineering/Science workspace settings allows users to create/manage their Spark compute, select the default runtime/default environment, enable or disable high concurrency mode, or ML autologging. |
| April 2024 | Fabric Spark Optimistic Job Admission | Fabric Spark Optimistic Job Admission reduces the frequency of throttling errors (HTTP 430: Spark Capacity Limit Exceeded Response) and improves the job admission experience for our customers, especially during peak usage hours. |
| April 2024 | Single Node support for starter pools | The Single Node support for starter pools feature lets you set your starter pool to max one node and get super-fast session start times for your Spark sessions. |
| April 2024 | Container Image for Synapse VS Code | To simplify the development process, we have released a container image for Synapse VS Code that contains all the necessary dependencies for the extension. |
| April 2024 | Git integration with Spark Job definition | Git integration with Spark Job definitions allows you to check in the changes of your Spark Job Definitions into a Git repository, which will include the source code of the Spark jobs and other item properties. |
| April 2024 | New Revamped Object Explorer experience in the notebook | The new Object Explorer experience improves flexibility and discoverability of data sources in the explorer and improve the discoverability of Resource folders. |
| April 2024 | %Run your scripts in Notebook | Now you can use %run magic command to run your Python scripts and SQL scripts in Notebook resources folder, just like Jupyter notebook %run command. |
| April 2024 | OneLake shortcuts to S3-compatible data sources preview | OneLake shortcuts to S3-compatible data sources are now in preview. Create an Amazon S3 compatible shortcut to connect to your existing data through a single unified name space without having to copy or move data. |
| April 2024 | OneLake shortcuts to Google Cloud Storage preview | OneLake shortcuts to Google Cloud Storage are now in preview. Create a Google Cloud Storage shortcut to connect to your existing data through a single unified name space without having to copy or move data. |
| April 2024 | OneLake data access roles | OneLake data access roles for lakehouse are in preview. Role permissions and user/group assignments can be easily updated through a new folder security user interface. |
| March 2024 | New validation enhancement for "Load to table" | The new validation enhancement to the "Load to table" feature help mitigate any validation issues and make your data loading experience smoother and faster. |
| March 2024 | Queuing for Notebook Jobs | Now with Job Queueing for Notebook Jobs, jobs that are triggered by pipelines or job scheduler will be added to a queue and will be retried automatically when the capacity frees up. For more information, see Job queueing in Microsoft Fabric Spark. |
| March 2024 | Autotune Query Tuning feature for Apache Spark | The Autotune Query Tuning feature for Apache Spark is now available. Autotune leverages historical data from your Spark SQL queries and machine learning algorithms to automatically fine-tune your configurations, ensuring faster execution times and enhanced efficiency. |
| March 2024 | OneLake File Explorer: Editing via Excel | With our latest release v1.0.11.0 of file explorer, we're excited to announce that you can now update your files directly using Excel, mirroring the user-friendly experience available in OneDrive. |
| February 2024 | Trusted workspace access (preview) for OneLake Shortcuts | Trusted workspace access (preview) enables secure and seamless access to ADLS Gen2 storage accounts from OneLake shortcuts in Fabric. For more information, see Trusted workspace access (preview). |
| February 2024 | Reduce egress costs with S3 shortcuts in OneLake | Learn how OneLake shortcuts to S3 now support caching, which can greatly reduce egress costs. Use the new Enable Cache for S3 Shortcuts setting with an S3 shortcut. |
| February 2024 | OneLake Shortcuts API | New REST APIs for OneLake Shortcuts allow programmatic creation and management of shortcuts, currently in preview. You can now programmatically create, read, and delete OneLake shortcuts. For example, see Use OneLake shortcuts REST APIs. |
| February 2024 | Browse code snippet | The new Browse code snippet notebook feature allows you to easily access and insert code snippets for commonly used code snippets with multiple supported languages. |
| February 2024 | Configure session timeout | Notebooks now support configuring session timeout for the current live session. It can help you avoid wasting resources or losing context due to timeout. You can specify the maximum duration of your spark sessions, from minutes to hours, and also get alerts before the session expires and extend it. |
| February 2024 | Fabric notebook status bar upgrade | The new Fabric Notebook status bar has three persisted info buttons: session status, save status, and cell selection status. Plus, context features include info on the git connection state, a shortcut to extend session timeout, and a failed cell navigator. |
| January 2024 | Microsoft Fabric Copilot for Data Science and Data Engineering | Copilot for Data Science and Data Engineering is now available worldwide. What can Copilot for Data Science and Data Engineering do for you? |
| January 2024 | Newest version of OneLake File Explorer includes Excel Integration | With the newest version of OneLake file explorer (v1.0.11.0) we bring a few updates to enhance your experience with OneLake, including Excel Integration. |
| December 2023 | %%configure – personalize your Spark session in Notebook | Now you can personalize your Spark session with the magic command %%configure, in both interactive notebook and pipeline notebook activities. |
| December 2023 | Rich dataframe preview in Notebook | The display() function has been updated on Fabric Notebook, now named the Rich dataframe preview. Now when you use display() to preview your dataframe, you can easily specify the range, view the dataframe summary and column statistics, check invalid values or missing values, and preview the long cell. |
| December 2023 | Working with OneLake using Azure Storage Explorer | If you want to use an application that directly integrates with Windows File Explorer, check out OneLake file explorer. However, if you're accustomed to using Azure Storage Explorer for your data management tasks, you can continue to harness its functionalities with OneLake and some of its key benefits. |
| November 2023 | Accessibility support for Lakehouse | To provide a more inclusive and user-friendly interaction, we have implemented improvements so far to support accessibility in the Lakehouse, including screen reader compatibility, responsive design text reflow, keyboard navigation, alternative text for images, and form fields and labels. |
| November 2023 | Enhanced multitasking experience in Lakehouse | We've introduced new capabilities to enhance the multi-tasking experience in Lakehouse, including multitasking during running operations, nonblocking reloading, and clearer notifications. |
| November 2023 | Upgraded DataGrid capabilities in Lakehouse | An upgraded DataGrid for the Lakehouse table preview now features sorting, filtering, and resizing of columns. |
| November 2023 | SQL analytics endpoint re-provisioning | You can now retry the SQL analytics endpoint provisioning directly within the Lakehouse. This means that if your initial provisioning attempt fails, you have the option to try again without the need to create an entirely new Lakehouse. |
| November 2023 | Microsoft Fabric Runtime 1.2 | The Microsoft Fabric Runtime 1.2 is a significant advancement in our data processing capabilities. Microsoft Fabric Runtime 1.2 includes Apache Spark 3.4.1, Mariner 2.0 as the operating system, Java 11, Scala 2.12.17, Python 3.10, Delta Lake 2.4, and R 4.2.2, ensuring you have the most cutting-edge tools at your disposal. In addition, this release comes bundled with default packages, encompassing a complete Anaconda installation and essential libraries for Java/Scala, Python, and R, simplifying your workflow. |
| November 2023 | Multiple Runtimes Support | With the introduction of Runtime 1.2, Fabric supports multiple runtimes, offering users the flexibility to seamlessly switch between them, minimizing the risk of incompatibilities or disruptions. When changing runtimes, all system-created items within the workspace, including Lakehouses, SJDs, and Notebooks, will operate using the newly selected workspace-level runtime version starting from the next Spark Session. |
| November 2023 | Delta as the default table format in the new Runtime 1.2 | The default Spark session parameter spark.sql.sources.default is now delta. All tables created using Spark SQL, PySpark, Scala Spark, and Spark R, whenever the table type is omitted, will create the table as Delta by default. |
| November 2023 | Intelligent Cache | By default, the newly revamped and optimized Intelligent Cache feature is enabled in Fabric Spark. The intelligent cache works seamlessly behind the scenes and caches data to help speed-up the execution of Spark jobs in Microsoft Fabric as it reads from your OneLake or ADLS Gen2 storage via shortcuts. |
| November 2023 | Monitoring Hub for Spark enhancements | The latest enhancements in the monitoring hub are designed to provide a comprehensive and detailed view of Spark and Lakehouse activities, including executor allocations, runtime version for a Spark application, a related items link in the detail page. |
| November 2023 | Monitoring for Lakehouse operations | Users can now view the progress and status of Lakehouse maintenance jobs and table load activities. |
| November 2023 | Spark application resource Usage Analysis | Responding to customers' requests for monitoring Spark resource usage metrics for performance tuning and optimization, we're excited to introduce the Spark resource usage analysis feature, now available in preview. This newly released feature enables users to monitor allocated executors, running executors, and idle executors, alongside Spark executions. |
| November 2023 | REST API support for Spark Job Definition preview | REST Public APIs for Spark Job Definition are now available, making it easy for users to manage and manipulate SJD items. |
| November 2023 | REST API support for Lakehouse, Load to tables and table maintenance | As a key requirement for workload integration, REST Public APIs for Lakehouse are now available. The Lakehouse REST Public APIs makes it easy for users to manage and manipulate Lakehouse items programmatically. |
| November 2023 | Lakehouse support for git integration and deployment pipelines (preview) | The Lakehouse now integrates with the lifecycle management capabilities in Microsoft Fabric, providing a standardized collaboration between all development team members throughout the product's life. Lifecycle management facilitates an effective product versioning and release process by continuously delivering features and bug fixes into multiple environments. |
| November 2023 | Embed a Power BI report in Notebook | We're thrilled to announce that the powerbiclient Python package is now natively supported in Fabric notebooks. This means you can easily embed and interact with Power BI reports in your notebooks with just a few lines of code. To learn more about how to use the powerbiclient package to embed a Power BI component. |
| November 2023 | Mssparkutils new API – reference run multiple notebooks in parallel | A new runMultiple API in mssparkutils called mssparkutils.notebook.runMultiple() allows you to run multiple notebooks in parallel, or with a predefined topological structure. For more information, see Notebook utilities. |
| November 2023 | Notebook resources .JAR file support | We now support uploading the .jar files in the Notebook Resources explorer. You can add your own compiled libs, use drag & drop to generate a code snippet to install them in the session, and load the libraries in code conveniently. |
| November 2023 | Notebook Git integration preview | Fabric notebooks now offer Git integration for source control using Azure DevOps. It allows users to easily control the notebook code versions and manage the git branches by leveraging the Fabric Git functions and Azure DevOps. |
| November 2023 | Notebook in Deployment Pipeline Preview | Now you can also use notebooks to deploy your code across different environments, such as development, test, and production. You can also use deployment rules to customize the behavior of your notebooks when they're deployed, such as changing the default Lakehouse of a Notebook. Get started with deployment pipelines, and Notebook shows up in the deployment content automatically. |
| November 2023 | Notebook REST APIs Preview | With REST Public APIs for the Notebook items, data engineers/data scientists can automate their pipelines and establish CI/CD conveniently and efficiently. The notebook Restful Public API can make it easy for users to manage and manipulate Fabric notebook items and integrate notebook with other tools and systems. |
| November 2023 | Environment preview | We're thrilled to announce preview of the Environment in Fabric. The Environment is a centralized item that allows you to configure all the required settings for running a Spark job in one place. |
| November 2023 | Synapse VS Code extension in vscode.dev preview | With support for the Synapse VS Code extension on vsocde.dev, users can now seamlessly edit and execute Fabric Notebooks without ever leaving their browser window. Additionally, all the native pro-developer features of VS Code are now accessible to end-users in this environment. |
| October 2023 | Create multiple OneLake shortcuts at once | Creating multiple OneLake shortcuts just got easier. Rather than creating shortcuts one at a time, you can now browse to your desired location and select multiple targets at once. All your selected targets then get created as new shortcuts in a single operation. |
| October 2023 | Delta-RS introduces native support for OneLake | The OneLake team worked with the Delta-RS community to help introduce support for recognizing OneLake URLs in both Delta-RS and the Rust Object Store. |
| September 2023 | Import notebook to your Workspace | The new "Import Notebook" entry on the Workspace -> New menu lets you easily import new Fabric Notebook items in the target workspace. You can upload one or more files, including .ipynb, .py, .sql, .scala, and .r file formats. |
| September 2023 | Notebook file system support in Synapse VS Code extension | The Synapse VS Code extension now supports notebook File System for Data Engineering and Data Science in Microsoft Fabric. The Synapse VS Code extension empowers users to develop their notebook items directly within the Visual Studio Code environment. |
| September 2023 | Notebook sharing execute-only mode | We now support checking the "Run" operation separately when sharing a notebook, if you just selected the "Run" operation, the recipient would see a "Execution-only" notebook. |
| September 2023 | Notebook save conflict resolution | We now support viewing and comparing the differences between two versions of the same notebook when there are saving conflicts. |
| September 2023 | Mssparkutils new API for fast data copy | We now support a new method in mssparkutils that can enable large volume of data move/copy much faster, Mssparkutils.fs.fastcp(). You can use mssparkutils.fs.help("fastcp") to check the detailed usage. |
| September 2023 | Notebook resources .whl file support | We now support uploading .whl files in the Notebook Resources explorer. |
| August 2023 | Introducing High Concurrency Mode in Notebooks for Data Engineering and Data Science workloads in Microsoft Fabric | High concurrency mode allows you to run notebooks simultaneously on the same cluster without compromising performance or security when paying for a single session. High concurrency mode offers several benefits for Fabric Spark users. |
| August 2023 | Service principal support to connect to data in Dataflow, Datamart, Dataset and Dataflow Gen 2 | Azure service principal has been added as an authentication type for a set of data sources that can be used in Dataset, Dataflow, Dataflow Gen2 and Datamart. |
| August 2023 | Announcing XMLA Write support for Direct Lake datasets | Direct Lake datasets now support XMLA-Write operations. Now you can use your favorite BI Pro tools and scripts to create and manage Direct Lake datasets using XMLA endpoints. |
| July 2023 | Lakehouse Sharing and Access Permission Management | Share a lakehouse and manage permissions so that users can access lakehouse data through the Data Hub, the SQL analytics endpoint, and the default semantic model. |
| June 2023 | Virtualize your existing data into OneLake with shortcuts | Connect data silos without moving or copying data with OneLake, which allows you to create special folders called shortcuts that point to other storage locations. |
| May 2023 | Introducing Data Engineering in Microsoft Fabric | With Synapse Data Engineering, one of the core experiences of Microsoft Fabric, data engineers feel right at home, able to leverage the power of Apache Spark to transform their data at scale and build out a robust lakehouse architecture. |
Synapse Data Engineering samples and guidance
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Demystifying Data Ingestion in Fabric: Fundamental Components for Ingesting Data into a Fabric Lakehouse using Fabric Data Pipelines | Learn about a batch data Ingestion framework based on experience working with different customers while building a lakehouse in Fabric. |
| June 2024 | Boost performance and save costs with Fast Copy in Dataflows Gen2 | Learn how the Fast Copy feature helps to enhance the performance and cost-efficiency of your Dataflows Gen2. |
| May 2024 | Copy Data from Lakehouse in another Workspace using Data pipeline | Learn how to copy data between Lakehouse that cross different workspaces via Data pipeline. |
| May 2024 | Profiling Microsoft Fabric Spark Notebooks with Sparklens | In this blog, you will learn how to leverage Sparklens, an open-source Spark profiling tool, to profile Microsoft Fabric Spark Notebooks and improve the performance of your spark code. |
| March 2024 | Bridging Fabric Lakehouses: Delta Change Data Feed for Seamless ETL | Learn how to use the Delta Change Data Feed to facilitate seamless data synchronization across different lakehouses in your medallion architecture. |
| January 2024 | Use Fabric Data Factory Data Pipelines to Orchestrate Notebook-based Workflows | Guidance and good practices when building Fabric Spark Notebook workflows using Data Factory in Fabric with data pipelines. |
| November 2023 | Fabric Changing the game: Using your own library with Microsoft Fabric | A step-by-step guide to use your own Python library in the Lakehouse. It's quite simple to create your own library with Python and even simpler to reuse it on Fabric. |
| August 2023 | Fabric changing the game: Logging your workload using Notebooks | Learn more about logging your workload into OneLake using notebooks, using the OneLake API Path inside the notebook. |
Synapse Data Science in Microsoft Fabric
This section summarizes archived improvements and features for the Data Science experience in Microsoft Fabric.
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Capacity pools preview | Capacity administrators can now create custom pools (preview) based on their workload requirements, providing granular control over compute resources. Custom pools for Data Engineering and Data Science can be set as Spark Pool options within Workspace Spark Settings and environment items. |
| June 2024 | Native Execution Engine for Apache Spark | The Native Execution Engine for Apache Spark on Fabric Data Engineering and Data Science for Fabric Runtime 1.2 is now in preview. For more information, see Native execution engine for Fabric Spark. |
| June 2024 | Demystifying Data Ingestion in Fabric: Fundamental Components for Ingesting Data into a Fabric Lakehouse using Fabric Data Pipelines | Learn about a batch data Ingestion framework based on experience working with different customers while building a lakehouse in Fabric. |
| June 2024 | Boost performance and save costs with Fast Copy in Dataflows Gen2 | Learn how the Fast Copy feature helps to enhance the performance and cost-efficiency of your Dataflows Gen2. |
| May 2024 | Public API for Workspace Data Engineering/Science | REST API support for Fabric Data Engineering/Science workspace settings allows users to create/manage their Spark compute, select the default runtime/default environment, enable or disable high concurrency mode, or ML autologging. |
| April 2024 | Semantic Link GA | Semantic links are now generally available! The package comes with our default VHD. You can now use Semantic link in Fabric right away without any pip installation. |
| April 2024 | Capacity level delegation for AI and Copilot | Tenant admins can now enable AI and Copilot in Fabric for the entire organization, certain security groups, or for a specific Capacity. |
| March 2024 | EU customers can use AI and Copilot without cross-geo setting | Since mid-March EU customers can use AI and Copilot without turning on the cross-geo setting, and their AI and Copilot requests will be processed within EUDB. |
| March 2024 | Code-First Hyperparameter Tuning preview | FLAML is now integrated for hyperparameter tuning, currently a preview feature. Fabric's flaml.tune feature streamlines this process, offering a cost-effective and efficient approach to hyperparameter tuning. |
| March 2024 | Code-First AutoML preview | With the new AutoML feature, you can automate your machine learning workflow and get the best results with less effort. AutoML, or Automated Machine Learning, is a set of techniques and tools that can automatically train and optimize machine learning models for any given data and task type. |
| March 2024 | Compare Nested Runs | Parent and child runs in the Run List View for ML Experiments introduces a hierarchical structure, allowing users to effortlessly view various parent and child runs within a single view and seamlessly interact with them to visually compare results. |
| March 2024 | Support for Mandatory MIP Label Enforcement | ML Model and Experiment items in Fabric now offer enhanced support for Microsoft Information Protection (MIP) labels. |
| January 2024 | Microsoft Fabric Copilot for Data Science and Data Engineering | Copilot for Data Science and Data Engineering is now available worldwide. What can Copilot for Data Science and Data Engineering do for you? |
| December 2023 | Semantic Link update | We're excited to announce the latest update of Semantic Link! Apart from many improvements, we also added many new features for our Power BI engineering community that you can use from Fabric notebooks to satisfy all your automation needs. |
| December 2023 | Prebuilt Azure AI services in Fabric preview | The preview of prebuilt AI services in Fabric is an integration with Azure AI services, formerly known as Azure Cognitive Services. Prebuilt Azure AI services allow for easy enhancement of data with prebuilt AI models without any prerequisites. Currently, prebuilt AI services are in preview and include support for the Microsoft Azure OpenAI Service, Azure AI Language, and Azure AI Translator. |
| November 2023 | Copilot in notebooks preview | The Copilot in Fabric Data Science and Data Engineering notebooks is designed to accelerate productivity, provide helpful answers and guidance, and generate code for common tasks like data exploration, data preparation, and machine learning. You can interact and engage with the AI from either the chat panel or even from within notebooks cells using magic commands to get insights from data faster. For more information, see Copilot in notebooks. |
| November 2023 | Custom Python Operations in Data Wrangler | Data Wrangler, a notebook-based tool for exploratory data analysis, has always allowed users to browse and apply common data-cleaning operations, generating the corresponding code in real time. Now, in addition to generating code from the UI, users can also write their own code with custom operations in Data Wrangler. |
| November 2023 | Data Wrangler for Spark DataFrames preview | Data Wrangler now supports Spark DataFrames in preview. Until now, users have been able to explore and transform pandas DataFrames using common operations that can be converted to Python code in real time. The new release allows users to edit Spark DataFrames in addition to pandas DataFrames with Data Wrangler. |
| November 2023 | MLFlow Notebook Widget | The MLflow inline authoring widget enables users to effortlessly track their experiment runs along with metrics and parameters, all directly from within their notebook. |
| November 2023 | New Model & Experiment Item Usability Improvements | New enhancements to our model and experiment tracking features are based on valuable user feedback. The new tree-control in the run details view makes tracking easier by showing which run is selected. We've enhanced the comparison feature, allowing you to easily adjust the comparison pane for a more user-friendly experience. Now you can select the run name to see the Run Details view. |
| November 2023 | Recent Experiment Runs | It's now simpler for users to check out recent runs for an experiment directly from the workspace list view. This update makes it easier to keep track of recent activity, quickly jump to the related Spark application, and apply filters based on the run status. |
| November 2023 | Models renamed to ML Models | Microsoft has renamed "Models" to "ML Models" to ensure clarity and avoid any confusion with other Fabric elements. For more information, see Machine learning experiments in Microsoft Fabric. |
| November 2023 | SynapseML v1.0 | SynapseML v1.0 is now released. SynapseML v1.0 makes it easy to build production ready machine learning systems on Fabric and has been in use at Microsoft for over six years. |
| November 2023 | Train Interpretable Explainable Boosting Machines with SynapseML | We've introduced a scalable implementation of Explainable Boosting Machines (EBM) powered by Apache Spark in SynapseML. EBMs are a powerful machine learning technique that combines the accuracy of gradient boosting with a strong focus on model interpretability. |
| November 2023 | Prebuilt AI models in Microsoft Fabric preview | We're excited to announce the preview for prebuilt AI models in Fabric. Azure OpenAI Service, Text Analytics, and Azure AI Translator are prebuilt models available in Fabric, with support for both RESTful API and SynapseML. You can also use the OpenAI Python Library to access Azure OpenAI service in Fabric. |
| November 2023 | Reusing existing Spark Session in sparklyr | We have added support for a new connection method called "synapse" in sparklyr, which enables users to connect to an existing Spark session. Additionally, we have contributed this connection method to the OSS sparklyr project. Users can now use both sparklyr and SparkR in the same session and easily share data between them. |
| November 2023 | REST API Support for ML Experiments and ML Models | REST APIs for ML Experiment and ML Model are now available. These REST APIs for ML Experiments and ML Models begin to empower users to create and manage machine learning items programmatically, a key requirement for pipeline automation and workload integration. |
| October 2023 | Semantic link (preview) | Semantic Link is an innovative feature that seamlessly connects Power BI semantic models with Synapse Data Science within Microsoft Fabric. As the gold layer in a medallion architecture, Power BI semantic models contain the most refined and valuable data in your organization. |
| October 2023 | Semantic link in Microsoft Fabric: Bridging BI and Data Science | We're pleased to introduce the preview of semantic link, an innovative feature that seamlessly connects Power BI semantic models with Synapse Data Science within Microsoft Fabric. |
| October 2023 | Get started with semantic link (preview) | Explore how semantic link seamlessly connects Power BI semantic models with Synapse Data Science within Microsoft Fabric. Learn more at Semantic link in Microsoft Fabric: Bridging BI and Data Science. You can also check out the semantic link sample notebooks that are now available in the fabric-samples GitHub repository. These notebooks showcase the use of semantic link's Python library, SemPy, in Microsoft Fabric. |
| August 2023 | Harness the Power of LangChain in Microsoft Fabric for Advanced Document Summarization | Harness the potential of Microsoft Fabric and SynapseML LLM capabilities to effectively summarize and organize your own documents. |
| July 2023 | Unleashing the Power of SynapseML and Microsoft Fabric: A Guide to Q&A on PDF Documents | In this blog post, we delve into the exciting functionalities and features of Microsoft Fabric and SynapseML to demonstrate how to leverage Generative AI models or Large Language Models (LLMs) to perform question and answer (Q&A) tasks on any PDF document. |
| May 2023 | Introducing Synapse Data Science in Microsoft Fabric | With data science in Microsoft Fabric, you can utilize the power of machine learning features to seamlessly enrich data as part of your data and analytics workflows. |
Synapse Data Science samples and guidance
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Building Custom AI Applications with Microsoft Fabric: Implementing Retrieval Augmented Generation for Enhanced Language Models | This guide walks you through implementing a RAG (Retrieval Augmented Generation) system in Microsoft Fabric using Azure OpenAI and Azure AI Search. |
| March 2024 | New AI Samples | New AutoML sample, Model Tuning, and Semantic Link samples appear in the Quick Tutorial category of the Data Science samples on Microsoft Fabric. |
| December 2023 | Using Microsoft Fabric's Lakehouse Data and prompt flow in Azure Machine Learning Service to create RAG applications | A step-by-step RAG application through prompt flow in Azure Machine Learning Service combined with Microsoft Fabric's Lakehouse data. |
| November 2023 | New data science happy path tutorial in Microsoft Fabric | We've updated the Data Science Happy Path tutorial for Microsoft Fabric. This new comprehensive tutorial demonstrates the entire data science workflow, using a bank customer churn problem as the context. |
| November 2023 | New data science samples | We've expanded our collection of data science samples to include new end-to-end R samples and new quick tutorial samples for "Explaining Model Outputs" and "Visualizing Model Behavior.". |
| November 2023 | New data science forecasting sample | The new Data Science sample on sales forecasting was developed in collaboration with Sonata Software. This new sample encompasses the entire data science workflow, spanning from data cleaning to Power BI visualization. The notebook covers the steps to develop, evaluate, and score a forecasting model for superstore sales, harnessing the power of the SARIMAX algorithm. |
| August 2023 | New Machine failure and Customer churn samples | More samples have been added to the Microsoft Fabric Synapse Data Science Use a sample menu. To check these Data Science samples, select Synapse Data Science, then Use a sample. |
| August 2023 | Use Semantic Kernel with Lakehouse in Microsoft Fabric | Learn how Fabric allows data scientists to use Semantic Kernel with Lakehouse in Microsoft Fabric. |
Synapse Data Warehouse
This section summarizes archived improvements and features for Synapse Data Warehouse in Microsoft Fabric.
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Restore in place portal experience | You can now create user-created restore points in your warehouse via the Fabric portal. For more information, see Restore in-place of a warehouse in Microsoft Fabric. |
| June 2024 | Fabric Spark connector for Fabric Data Warehouse in Spark runtime (preview) | The Fabric Spark connector for Fabric Data Warehouse (preview) enables a Spark developer or a data scientist to access and work on data from Fabric DW and SQL analytics endpoint of the lakehouse (either from within the same workspace or from across workspaces) with a simplified Spark API. |
| May 2024 | Monitor Warehouse tools | You can Monitor Synapse Data Warehouse activity with a variety of tools, including: Billing and utilization reporting in Synapse Data Warehouse, monitor connections, sessions, and requests using DMVs, Query insights, and now Query activity. For more information, read Query activity: A one-stop view to monitor your running and completed T-SQL queries. |
| May 2024 | Copilot for Data Warehouse | Copilot for Data Warehouse (preview) is now available in limited preview, offering the Copilot chat pane, quick actions, and code completions. |
| May 2024 | Warehouse queries with time travel (preview) | Warehouse in Microsoft Fabric offers the capability to query the historical data as it existed in the past at the statement level, currently in preview. The ability to query data from a specific timestamp is known in the data warehousing industry as time travel. |
| May 2024 | COPY INTO enhancements | COPY INTO now supports Microsoft Entra ID authentication and access to firewall protected storage via the trusted workspace functionality. For more information, see COPY INTO enhancements and COPY INTO (Transact-SQL). |
| April 2024 | Fabric Warehouse in ADF copy activity | You can now connect to your Fabric Warehouse from an Azure Data Factory/Synapse pipeline. You can find this new connector when creating a new source or sink destination in your copy activity, in the Lookup activity, Stored Procedure activity, Script activity, and Get Metadata activity. |
| April 2024 | Git integration | Git integration for the Warehouse allows you to check in the changes of your Warehouse to an Azure DevOps Git repository as a SQL database project. |
| April 2024 | Partition elimination | Partition elimination is a performance improvement for tables with a large number of files. The SQL analytics endpoint of a Lakehouse uses partition elimination to read data from only those partitions that are relevant to the query. Recent improvements boosted performance even more when queries are aimed at a few partitions in a table that has many files. |
| March 2024 | Mirroring in Microsoft Fabric preview | With Mirroring in Fabric, you can easily bring your databases into OneLake in Microsoft Fabric, enabling seamless zero-ETL, near real-time insights on your data – and unlocking warehousing, BI, AI, and more. For more information, see What is Mirroring in Fabric?. |
| March 2024 | Cold cache performance improvements | Fabric stores data in Delta tables and when the data is not cached, it needs to transcode data from parquet file format structures to in-memory structures for query processing. Recent cold cache performance improvements further optimize transcoding and we observed up to 9% faster queries in our tests when data is not previously cached. |
| March 2024 | Extract and publish a SQL database project directly through the DW editor | The SQL Database Projects extension creates a SQL project (.sqlproj) file, a local representation of SQL objects that comprise the schema for a single database, such as tables, stored procedures, or functions. You can now extract and publish a SQL database project directly through the DW editor. |
| March 2024 | Change owner of Warehouse item | The new Takeover API allows you to change the warehouse owner from the current owner to a new owner, which can be an SPN or an Organizational Account. |
| March 2024 | Clone table RLS and CLS | A cloned table now inherits the row-level security (RLS) and dynamic data masking from the source of the clone table. |
| February 2024 | Experience performance improvements | Recent connectivity and performance enhancements include an improved experience for creating warehouses, T-SQL execution, automatic metadata discovery, and error messaging. |
| December 2023 | Automatic Log Checkpointing for Fabric Warehouse | Automatic Log Checkpointing is one of the ways that we help your Data Warehouse to provide you with great performance and best of all, it involves no additional work from you! |
| December 2023 | Restore points and restore in place | You can now create restore points and perform an in-place restore of a warehouse to a past point in time. The restore points and restore in place features are currently in preview. Restore in-place is an essential part of data warehouse recovery, which allows to restore the data warehouse to a prior known reliable state by replacing or over-writing the existing data warehouse from which the restore point was created. |
| November 2023 | TRIM T-SQL support | You can now use the TRIM command to remove spaces or specific characters from strings by using the keywords LEADING, TRAILING or BOTH in TRIM (Transact-SQL). |
| November 2023 | GENERATE_SERIES T-SQL support | Generates a series of numbers within a given interval with GENERATE_SERIES (Transact-SQL). The interval and the step between series values are defined by the user. |
| November 2023 | SSD metadata caching | File and rowgroup metadata are now also cached with in-memory and SSD cache, further improving performance. |
| November 2023 | PARSER 2.0 improvements for CSV ingestion | CSV file parser version 2.0 for COPY INTO builds an innovation from Microsoft Research's Data Platform and Analytics group to make CSV file ingestion blazing fast on Fabric Warehouse. For more information, see COPY INTO (Transact-SQL). |
| November 2023 | Fast compute resource assignment enabled | All query executions in Fabric Warehouse are now powered by the new technology recently deployed as part of the Global Resource Governance component that assigns compute resources in milliseconds. |
| November 2023 | REST API support for Warehouse | With the Warehouse public APIs, SQL developers can now automate their pipelines and establish CI/CD conveniently and efficiently. The Warehouse REST Public APIs makes it easy for users to manage and manipulate Fabric Warehouse items. |
| November 2023 | SQLPackage support for Fabric Warehouse | SQLPackage now supports Fabric Warehouse. SqlPackage is a command-line utility that automates the following database development tasks by exposing some of the public Data-Tier Application Framework (DacFx) APIs. The SqlPackage command line tool allows you to specify these actions along with action-specific parameters and properties. |
| November 2023 | Power BI semantic models | Microsoft has renamed the Power BI dataset content type to semantic model. This applies to Microsoft Fabric semantic models as well. For more information, see New name for Power BI datasets. |
| November 2023 | SQL analytics endpoint | Microsoft has renamed the SQL endpoint of a Lakehouse to the SQL analytics endpoint of a Lakehouse. |
| November 2023 | Dynamic data masking | Dynamic Data Masking (DDM) for Fabric Warehouse and the SQL analytics endpoint in the Lakehouse. For more information and samples, see Dynamic data masking in Fabric data warehousing and How to implement dynamic data masking in Synapse Data Warehouse. |
| November 2023 | Clone tables with time travel | You can now use table clones to create a clone of a table based on data up to seven calendar days in the past. |
| November 2023 | User experience updates | Several user experiences in Warehouse have landed. For more information, see Fabric Warehouse user experience updates. |
| November 2023 | Automatic data compaction | Automatic data compaction rewrites many smaller parquet files into a few larger parquet files, which will improve the performance of reading the table. Data Compaction is one of the ways that we help your Warehouse to provide you with great performance and no effort on your part. |
| October 2023 | Support for sp_rename | Support for the T-SQL sp_rename syntax is now available for both Warehouse and SQL analytics endpoint. For more information, see Fabric Warehouse support for sp_rename. |
| October 2023 | Query insights | The query insights feature is a scalable, sustainable, and extendable solution to enhance the SQL analytics experience. With historic query data, aggregated insights, and access to actual query text, you can analyze and tune your query performance. |
| October 2023 | Full DML to Delta Lake Logs | Fabric Warehouse now publishes all Inserts, Updates, and Deletes for each table to their Delta Lake Log in OneLake. |
| October 2023 | V-Order write optimization | V-Order optimizes parquet files to enable lightning-fast reads under the Microsoft Fabric compute engines such as Power BI, SQL, Spark, and others. Warehouse queries in general benefit from faster read times with this optimization, still ensuring the parquet files are 100% compliant to its open-source specification. Starting this month, all data ingested into Fabric Warehouses use V-Order optimization. |
| October 2023 | Burstable capacity | Burstable capacity allows workloads to use more resources to achieve better performance. Burstable capacity is finite, with a limit applied to the backend compute resources to greatly reduce the risk of throttling. For more information, see Warehouse SKU Guardrails for Burstable Capacity. |
| October 2023 | Throttling and smoothing in Synapse Data Warehouse | A new article details the throttling and smoothing behavior in Synapse Data Warehouse, where almost all activity is classified as background to take advantage of the 24-hr smoothing window before throttling takes effect. Learn more about how to observe utilization in Synapse Data Warehouse. |
| September 2023 | Default semantic model improvements | The default semantic model no longer automatically adds new objects. This can be enabled in the Warehouse item settings. |
| September 2023 | Deployment pipelines now support warehouses | Deployment pipelines enable creators to develop and test content in the service before it reaches the users. Supported content types include reports, paginated reports, dashboards, semantic models, dataflows, and now warehouses. Learn how to deploy content programmatically using REST APIs and DevOps. |
| September 2023 | SQL Projects support for Warehouse in Microsoft Fabric | Microsoft Fabric Data Warehouse is now supported in the SQL Database Projects extension available inside of Azure Data Studio and Visual Studio Code. |
| September 2023 | Announcing: Column-level & Row-level security for Fabric Warehouse & SQL analytics endpoint | Column-level and row-level security in Fabric Warehouse and SQL analytics endpoint are now in preview, behaving similarly to the same features in SQL Server. |
| September 2023 | Usage reporting | Utilization and billing reporting is available for Fabric data warehousing in the Microsoft Fabric Capacity Metrics app. For more information, read about Utilization and billing reporting Fabric data warehousing. |
| August 2023 | SSD Caching enabled | Local SSD caching stores frequently accessed data on local disks in highly optimized format, significantly reducing I/O latency. This benefits you immediately, with no action required or configuration necessary. |
| July 2023 | Sharing | Any Admin or Member within a workspace can share a Warehouse with another recipient within your organization. You can also grant these permissions using the "Manage permissions" experience. |
| July 2023 | Table clone | A zero-copy clone creates a replica of the table by copying the metadata, while referencing the same data files in OneLake. This avoids the need to store multiple copies of data, thereby saving on storage costs when you clone a table in Microsoft Fabric. For more information, see tutorials to Clone a table with T-SQL or Clone tables in the Fabric portal. |
| May 2023 | Introducing Synapse Data Warehouse in Microsoft Fabric | Synapse Data Warehouse is the next generation of data warehousing in Microsoft Fabric that is the first transactional data warehouse to natively support an open data format, Delta-Parquet. |
Synapse Data Warehouse samples and guidance
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Mastering Enterprise T-SQL ETL/ELT: A Guide with Data Warehouse and Fabric Pipelines | Learn about foundational elements of an enterprise-scale ETL/ELT framework using Fabric Pipelines and a Data Warehouse for performing our transformations in T-SQL. Additionally, we will examine a dynamic SQL script designed to incrementally process tables throughout your enterprise. |
| April 2024 | Fabric Change the Game: Azure SQL Database mirror into Microsoft Fabric | A step-by-step guide to mirror your Azure SQL Database into Microsoft Fabric. |
| February 2024 | Mapping Azure Synapse dedicated SQL pools to Fabric data warehouse compute | Read for guidance on mapping Data Warehouse Units (DWU) from Azure Synapse Analytics dedicated SQL pool to an approximate equivalent number of Fabric Capacity Units (CU). |
| January 2024 | Automate Fabric Data Warehouse Queries and Commands with Data Factory | In Fabric Data Factory, there are many ways to query data, retrieve data, and execute commands from your warehouse using pipeline activities that can then be easily automated. |
| November 2023 | Migrate from Azure Synapse dedicated SQL pools | A detailed guide with a migration runbook is available for migrations from Azure Synapse Data Warehouse dedicated SQL pools into Microsoft Fabric. |
| August 2023 | Efficient Data Partitioning with Microsoft Fabric: Best Practices and Implementation Guide | A proposed method for data partitioning using Fabric notebooks. Data partitioning is a data management technique used to divide a large dataset into smaller, more manageable subsets called partitions or shards. |
| May 2023 | Microsoft Fabric - How can a SQL user or DBA connect | This blog reviews how to connect to a SQL analytics endpoint of the Lakehouse or the Warehouse through the Tabular Data Stream, or TDS endpoint, familiar to all modern web applications that interact with a SQL Server endpoint. |
Real-Time Intelligence in Microsoft Fabric
This section summarizes archived improvements and features for Real-Time Intelligence in Microsoft Fabric.
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Graph Semantics in Eventhouse | Graph Semantics in Eventhouse allows users to model their data as graphs and perform advanced graph queries and analytics using the Kusto Query Language (KQL). |
| June 2024 | Set alerts on Real-time Dashboards with Data Activator triggers | Real-Time Dashboard visuals now support alerts, to extend monitoring support with Data Activator. With integration with Data Activator, you'll receive timely alerts as your key metrics change in real-time. |
| June 2024 | OneLake availability of Eventhouse in Delta Lake format GA | As part of the One logical copy promise, we're excited to announce that OneLake availability of Eventhouse in Delta Lake format is Generally Available. |
| June 2024 | Real-Time Dashboards | Real-Time Dashboards interact with data dynamically and in real time. Real-Time Dashboards natively visualize data stored in Eventhouses. Real-time Dashboards support ultra-low refresh rates of just 1 or 10 seconds. For more information, see Visualize and Explore Data with Real-Time Dashboards. |
| May 2024 | Eventhouse GA | Eventhouse is a new, dynamic workspace hosting multiple KQL databases, generally available as part of Fabric Real-Time Intelligence. An Eventhouse offers a robust solution for managing and analyzing substantial volumes of real-time data. Get started with a guide to Create and manage an Eventhouse. |
| May 2024 | Copilot for Real-Time Intelligence | Copilot for Real-Time Intelligence is now in preview! For those who are already fans of KQL or newcomers exploring its potential, Copilot can help you get started, and navigate data with ease. |
| May 2024 | Automating Fabric items with Real-Time Intelligence | Learn how to interact with data pipelines, notebooks, spark jobs in a more event-driven way. |
| May 2024 | Real-Time Intelligence | This month includes the announcement of Real-Time Intelligence, the next evolution of Real-Time Analytics and Data Activator. |
| May 2024 | Real-Time Intelligence new preview features | At Build 2024, a dozen new features and capabilities were announced for Real-Time Intelligence, organized into categories of Ingest & Process, Analyze & Transform, and Visualize & Act. |
| May 2024 | Real-Time hub preview | Real-Time hub is single, tenant-wide, unified, logical place for streaming data-in-motion. It enables you to easily discover, ingest, manage, and consume data-in-motion from a wide variety of sources. It lists all the streams and Kusto Query Language (KQL) tables that you can directly act on. It also gives you an easy way to ingest streaming data from Microsoft products and Fabric events. For more information, see Real-Time hub overview. |
| May 2024 | Get Events preview | The Get Events experience allows users to connect to a wide range of sources directly from Real-Time hub, Eventstreams, Eventhouse, and Data Activator. Using Get Events, bring streaming data from Microsoft sources directly into Fabric with a first-class experience. |
| May 2024 | Enhanced Eventstream capabilities preview | With enhanced Eventstream capabilities, you can now stream data not only from Microsoft sources but also from other platforms like Google Cloud, Amazon Kinesis, Database change data capture streams, and more, using our new messaging connectors. |
| May 2024 | Event streams - enhanced capabilities preview | The preview of enhanced capabilities supports many new sources - Google Cloud Pub/Sub, Amazon Kinesis Data Streams, Confluent Cloud Kafka, Azure SQL Database Change Data Capture (CDC), PostgreSQL Database CDC, MySQL Database CDC, Azure Cosmos DB CDC, Azure Blob Storage events, and Fabric workspace item events, and a new Stream destination. It supports two distinct modes, Edit mode and Live view, in the visual designer. It also supports routing based on content in data streams. For more information, see What is Fabric event streams. |
| April 2024 | Kusto Cache consumption preview | The preview of Kusto Cache consumption means that you will start seeing billable consumption of the OneLake Cache Data Stored meter from the KQL Database and Eventhouse items. For more information, see KQL Database consumption. |
| April 2024 | Pause and Resume in Eventstream preview | The Pause and Resume feature enables you to pause data streaming from various sources and destinations within Eventstream. You can then resume data streaming seamlessly from the paused time or a customized time, ensuring no data loss. |
| March 2024 | New Expressions "Changes by", "Increases by", and "Decreases by" | In Data Activator, when setting conditions on a trigger, we've added syntax to allows you to detect when there's been a change in your data by absolute number or percentage. See New Expressions "Changes by", "Increases by", and "Decreases by". |
| March 2024 | Fabric Real-Time Intelligence Integrates with Newly Announced Database Watcher for Azure SQL | Users of Azure SQL can use the Database Watcher monitoring solution with Microsoft Fabric. Database Watcher for Azure SQL (preview) provides advanced monitoring capabilities, and can integrate with Eventhouse KQL database. |
| March 2024 | Update records in a KQL Database preview | The .update command is now available, as a preview feature. Learn more about how to Update records in a Kusto database. |
| March 2024 | Query Azure Data Explorer data from Queryset | Connecting to and using data in Azure Data explorer cluster from Fabric's KQL Queryset is now available. |
| February 2024 | Eventhouse Overview: Handling Real-Time Data with Microsoft Fabric | Eventhouse (preview) is a dynamic workspace hosting multiple KQL databases, part of Fabric Real-Time Intelligence. An Eventhouse offers a robust solution for managing and analyzing substantial volumes of real-time data. Get started with a guide to Create and manage an Eventhouse. |
| February 2024 | KQL DB shortcut to Delta Lake tables support name-based column mapping | KQL DB now supports reading Delta tables with column name mappings. The column mapping feature allows Delta table columns and the underlying Parquet file columns to use different names. This enables Delta schema evolution operations such on a Delta table without the need to rewrite the underlying Parquet files and allows users to name Delta table columns by using characters that aren't allowed by Parquet. |
| February 2024 | KQL DB shortcut to Delta Lake tables support deletion vectors | KQL DB can now read delta tables with deletion vectors, resolving the current table state by applying the deletions noted by deletion vectors to the most recent table version. |
| February 2024 | Get Data in KQL DB now supports processing events before ingestion via Eventstream | The Process event before ingestion in Eventstream option enables you to process the data before it's ingested into the destination table. By selecting this option, the get data process seamlessly continues in Eventstream, with the destination table and data source details automatically populated. |
| February 2024 | KQL DB now supports data ingestion using Apache Flink | Using the open-source Flink connector, you can send data from Flink to your table. Using Azure Data Explorer and Apache Flink, you can build fast and scalable applications targeting data driven scenarios. |
| February 2024 | Route data from Splunk Universal Forwarder to KQL DB using Kusto Splunk Universal Connector | You can now use the Kusto Splunk Universal Connector to send data from Splunk Universal Forwarder to a table in your KQL DB. |
| December 2023 | Calculating distinct counts in Power BI running reports on KQL Databases | New Fabric KQL database dcount and dcountif functions use a special algorithm to return an estimate of distinct counts, even in extremely large datasets. The new functions count_distinct and count_distinctif calculate exact distinct counts. |
| December 2023 | Create a Notebook with pre-configured connection to your KQL DB | You can now just create a new Notebook from KQL DB editor with a preconfigured connection to your KQL DB and explore the data using PySpark. This option creates a PySpark Notebook with a ready-to execute code cell to read data from the selected KQL DB. |
| December 2023 | KQL Database schema validation | The new Kusto command .show database schema violations was designed to validate the current state of your database schema and find inconsistencies. You can use .show database schema violations for a spot check on your database or in CI/CD automation. |
| December 2023 | Enabling Data Availability of KQL Database in OneLake | Data availability of KQL Database in OneLake means you can enjoy the best of both worlds. You can query the data with high performance and low latency in their KQL database, and you can query the same data in Delta Parquet via Power BI Direct Lake mode, Warehouse, Lakehouse, Notebooks, and more. |
| December 2023 | Fabric Change the Game: Real–time Intelligence | Real-Time Intelligence is a formidable tool, diminishing complexity and streamlining data integration processes. Microsoft Fabric allows you to build Real-Time streaming analytics with eventstream or Spark Stream. |
| November 2023 | Announcing Delta Lake support in Real-Time Analytics KQL Database | You can now enable availability of KQL Database in Delta Lake format. Delta Lake is the unified data lake table format chosen to achieve seamless data access across all compute engines in Microsoft Fabric. |
| November 2023 | Real-Time Analytics in Microsoft Fabric general availability (GA) | Announcing the general availability of Real-Time Analytics in Microsoft Fabric! Real-Time Analytics offers countless features all aimed at making your data analysis more efficient and effective. |
| November 2023 | Delta Parquet support in KQL Database | As part of the one logical copy promise, we're excited to announce that data in KQL Database can now be made available in OneLake in delta parquet format. You can now access this Delta table by creating a OneLake shortcut from Lakehouse, Warehouse, or directly via Power BI Direct Lake mode. |
| November 2023 | Open Source Connectors for KQL Database | Several open-source connectors for real-time analytics are now supported to enable users to ingest data from various sources and process it using KQL DB. |
| November 2023 | REST API Support for KQL Database | We're excited to announce the launch of REST Public APIs for KQL DB. The Public REST APIs of KQL DB enables users to manage and automate their flows programmatically. |
| November 2023 | Eventstream now Generally Available | Eventstream is now generally available, adding enhancements aimed at taking your data processing experience to the next level. |
| November 2023 | Eventstream Data Transformation for KQL Database | Now, you can transform your data streams into real time within Eventstream before they're sent to your KQL Database. When you create a KQL Database destination in the eventstream, you can set the ingestion mode to "Event processing before ingestion" and add event processing logics such as filtering and aggregation to transform your data streams. |
| November 2023 | Splunk add-on preview | Microsoft Fabric add-on for Splunk allows users to ingest logs from Splunk platform into a Fabric KQL DB using the Kusto python SDK. |
| November 2023 | Get Data from Eventstream anywhere in Fabric | If you're working on other Fabric items and are looking to ingest data from Eventstream, our new "Get Data from Eventstream" feature simplifies the process, you can Get data from Eventstream while you're working with a KQL database and Lakehouse. |
| November 2023 | Two ingestion modes for Lakehouse Destination | We've introduced two distinct ingestion modes for your Lakehouse Destination: Rows per file and Duration. |
| November 2023 | Optimize Tables Before Ingesting Data to Lakehouse | The table optimization shortcut is now available inside Eventstream Lakehouse destination to compact numerous small streaming files generated on a Lakehouse table. Table optimization shortcut works by opening a Notebook with Spark job, which would compact small streaming files in the destination Lakehouse table. |
| November 2023 | Create a Cloud Connection within Eventstream | We've simplified the process of establishing a cloud connection to your Azure services within Eventstream. When adding an Azure resource, such as Azure IoT Hub and Azure Event Hubs, to your Eventstream, you can now create the cloud connection and enter your Azure resource credentials right within Eventstream. This enhancement significantly improves the process of adding new data sources to your Eventstream, saving time and effort. |
| November 2023 | Get Data in Real-Time Analytics: A New and Improved Experience | A new Get Data experience simplifies the data ingestion process in your KQL database. |
| October 2023 | Expanded Custom App Connections | New new custom app connections provide more flexibility when it comes to bringing your data streams into Eventstream. |
| October 2023 | Enhanced UX on Event Processor | New UX improvements on the no-code Event Processor provide an intuitive experience, allowing you to effortlessly add or delete operations on the canvas. |
| October 2023 | Eventstream Kafka Endpoints and Sample Code | The Custom App feature has new endpoints in sources and destinations, including sample Java code for your convenience. Simply add it to your application, and you're all set to stream your real-time event to Eventstream. |
| October 2023 | Event processing editor UX improvements | Recent UX improvements introduce a full-screen mode, providing a more spacious workspace for designing your data processing workflows. The insertion and deletion of data stream operations have been made more intuitive, making it easier to drag and drop and connect your data transformations. |
| October 2023 | KQL Database Auto scale algorithm improvements | Users do not need to worry about how many resources are needed to support their workloads in a KQL database. KQL Database has a sophisticated in-built, multi-dimensional, auto scaling algorithm. We recently implemented some optimizations that make some time series analysis more efficient. |
| October 2023 | Understanding Fabric KQL DB Capacity | Read more about how a KQL database is billed in the SaaS world of Microsoft Fabric. |
| September 2023 | OneLake shortcut to delta tables from KQL DB | Now you can create a shortcut from KQL DB to delta tables in OneLake, allowing in-place data queries. Now you query delta tables in your Lakehouse or Warehouse directly from KQL DB. |
| September 2023 | Model and Query data as graphs using KQL | Kusto Query Language (KQL) now allows you to model and query data as graphs. This feature is currently in preview. Learn more at Introduction to graph semantics in KQL and Graph operators and functions. |
| September 2023 | Easily connect to KQL Database from Power BI desktop | Power BI desktop released two new ways to easily connect to a KQL database, in the Get Data dialogue and in the OneLake data hub menus. |
| September 2023 | Eventstream now supports AMQP format connection string for data ingestion | AMQP stands for Advanced Message Queuing Protocol, a protocol that supports a wide range of messaging patterns. In Eventstream, you can now create a Custom App source or destination and select AMQP format connection string for ingesting data into Fabric or consuming data from Fabric. |
| September 2023 | Eventstream supports data ingestion from Azure IoT Hub | Azure IoT Hub is a cloud-hosted solution that provides secure communication channels for sending and receiving data from IoT devices. In Eventstream, you can now stream your Azure IoT Hub data into Fabric and perform real-time processing before storing it in a Kusto Database or Lakehouse. |
| September 2023 | Real-Time Data Sharing in Microsoft Fabric | A database shortcut in Real-Time Intelligence is an embedded reference within a KQL database to a source database in Azure Data Explorer (ADX) allowing in-place data sharing. The behavior exhibited by the database shortcut is similar to that of an Azure Data Explorer follower database. |
| August 2023 | Provisioning optimization | The KQL Database provisioning process has been optimized. Now you can create a KQL Database within a few seconds. |
| August 2023 | KQL Database support for inline Python | Fabric KQL Database supports running Python code embedded in Kusto Query Language (KQL) using the python() plugin. The plugin is disabled by default. Before you start, enable the Python plugin in your KQL database. |
| July 2023 | Microsoft Fabric eventstreams: Generating Real-time Insights with Python, KQL, and Power BI | Microsoft Fabric eventstreams are a high-throughput, low-latency data ingestion and transformation service. |
| July 2023 | Stream Real-time Events to Microsoft Fabric with eventstreams from a custom application | Eventstreams under Real-Time Intelligence are a centralized platform within Fabric, allowing you to capture, transform, and route real-time events to multiple destinations effortlessly, all through a user-friendly, no-code experience. |
| June 2023 | Unveiling the Epic Opportunity: A Fun Game to Explore the Real-Time Intelligence | As part of the Kusto Detective Agency Season 2, we're excited to introduce an epic opportunity for all investigators and data enthusiasts to learn about the new portfolio in a fun and engaging way. Recruiting now at https://detective.kusto.io/! |
| May 2023 | What's New in Kusto – Build 2023! | Announcing the Synapse Real Time Analytics in Microsoft Fabric! |
Real-Time Intelligence samples and guidance
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Power BI Admin portal Usage metrics dashboard retirement | Effective July 2024, the Power BI Admin portal Usage metrics dashboard is removed. Comparable insights are now supported out-of-the-box through the Admin monitoring workspace (preview). The Admin monitoring workspace provides several Power BI reports and semantic models, including the Feature Usage and Adoption report which focuses on Fabric tenant inventory and audit activity monitoring. |
| May 2024 | Alerting and acting on data from the Real-Time hub | Microsoft Fabric's new Real-Time hub and Data Activator provide a no-code experience for automatically taking actions when patterns or conditions are detected in changing data and is embedded around the Real-Time hub to make creating alerts always accessible. |
| May 2024 | Using APIs with Fabric Real-Time Intelligence: Eventhouse and KQL DB | Learn how to create/update/delete items in Fabric with the KQL APIs, accessing the data plane of a resource. |
| May 2024 | Connect and stream events with the Get events experience | The Get events experience streamlines the process of browsing and searching for sources and streams. |
| May 2024 | Acquiring Real-Time Data from New Sources with Enhanced Eventstream | Learn how to connect to new sources in Eventstream. Start by creating an eventstream and choosing "Enhanced Capabilities (preview)". |
| March 2024 | Browse Azure resources with Get Data | Learn how to browse and connect to all your Azure resources with the 'browse Azure' functionality in Get Data. You can browse Azure resources then connect to Synapse, blob storage, or ADLS Gen2 resources easily. |
| November 2023 | Semantic Link: Data validation using Great Expectations | Great Expectations Open Source (GX OSS) is a popular Python library that provides a framework for describing and validating the acceptable state of data. With the recent integration of Microsoft Fabric semantic link, GX can now access semantic models, further enabling seamless collaboration between data scientists and business analysts. |
| November 2023 | Explore Data Transformation in Eventstream for KQL Database Integration | Dive into a practical scenario using real-world bike-sharing data and learn to compute the number of bikes rented every minute on each street, using Eventstream's powerful event processor, mastering real-time data transformations, and effortlessly directing the processed data to your KQL Database.. |
| October 2023 | From RabbitMQ to PowerBI reports with Microsoft Fabric Real-Time Analytics | A walkthrough of an end-to-end scenario sending data from RabbitMQ to a KQL Database in Microsoft Fabric. |
| October 2023 | Stream Azure IoT Hub Data into Fabric Eventstream for Email Alerting | A demo of using Fabric Eventstream to seamlessly ingest and transform real-time data streams before they reach various Fabric destinations such as Lakehouse, KQL Database, and Reflex. Then, configure email alerts in Reflex with Data Activator triggers. |
| September 2023 | Real-Time Intelligence sample gallery | Real-Time Intelligence now offers a comprehensive sample gallery with multiple datasets allowing you to explore, learn, and get started quickly. Access the samples by selecting Use a sample from the Real-Time Intelligence experience home. |
| September 2023 | Quick start: Sending data to Real-Time Intelligence in Fabric from Apache Kafka Ecosystems using Java | Learn how to send data from Kafka to Real-Time Intelligence in Fabric. |
| June 2023 | From raw data to insights: How to ingest data from Azure Event Hubs into a KQL database | Learn about the integration between Azure Event Hubs and your KQL database. |
| June 2023 | From raw data to insights: How to ingest data from eventstreams into a KQL database | Learn about the integration between eventstreams and a KQL database, both of which are a part of the Real-Time Intelligence experience. |
| June 2023 | Discovering the best ways to get data into a KQL database | This blog covers different options for bringing data into a KQL database. |
| June 2023 | Get started with exploring your data with KQL – a purpose-built tool for petabyte scale data analytics | In this blog, we focus on the different ways of querying data in Real-Time Intelligence. |
| May 2023 | Ingest, transform, and route real-time events with Microsoft Fabric event streams | You can now ingest, capture, transform and route real-time events to various destinations in Microsoft Fabric with a no-code experience using Microsoft Fabric eventstreams. |
Microsoft Fabric core features
Archived news and feature announcements core to the Microsoft Fabric experience.
| Month | Feature | Learn more |
|---|---|---|
| May 2024 | Microsoft Fabric Private Links GA | Azure Private Link for Microsoft Fabric secures access to your sensitive data in Microsoft Fabric by providing network isolation and applying required controls on your inbound network traffic. For more information, see Announcing General Availability of Fabric Private Links. |
| May 2024 | Trusted workspace access GA | Trusted workspace access in OneLake shortcuts is now generally available. You can now create data pipelines to access your firewall-enabled Azure Data Lake Storage Gen2 (ADLS Gen2) accounts using Trusted workspace access (preview) in your Fabric Data Pipelines. Use the workspace identity to establish a secure and seamless connection between Fabric and your storage accounts. Trusted workspace access also enables secure and seamless access to ADLS Gen2 storage accounts from OneLake shortcuts in Fabric. |
| May 2024 | Fabric APIs walkthrough | Learn about using REST APIs in Fabric, including creating workspaces, adding permission, dropping, creating, executing data pipelines, and how to pause/resume Fabric activities using the management API. |
| May 2024 | Managed private endpoints GA | Managed private endpoints for Microsoft Fabric allow secure connections over managed virtual networks to data sources that are behind a firewall or not accessible from the public internet. For more information, see Announcing General Availability of Fabric Private Links, Trusted Workspace Access, and Managed Private Endpoints. |
| May 2024 | Fabric UX System | The Fabric UX System represents a leap forward in design consistency and extensibility for Microsoft Fabric. |
| May 2024 | Microsoft Fabric Core REST APIs | Microsoft Fabric Core APIs are now generally available. The Fabric user APIs are a major enabler for both enterprises and partners to use Microsoft Fabric as they enable end-to-end fully automated interaction with the service, enable integration of Microsoft Fabric into external web applications, and generally enable customers and partners to scale their solutions more easily. |
| May 2024 | Microsoft Fabric Admin APIs preview | Fabric Admin APIs are designed to streamline administrative tasks. Now, you can manage both Power BI and the new Fabric items (previously referred to as artifacts) using the same set of APIs. Before this enhancement, you had to navigate using two different APIs—one for Power BI items and another for new Fabric items. |
| May 2024 | Fabric workload dev kit (preview) | The Microsoft Fabric workload development kit extends to additional workloads and offers a robust developer toolkit for designing, developing, and interoperating with Microsoft Fabric using frontend SDKs and backend REST APIs. |
| May 2024 | Introducing external data sharing (preview) | External Data Sharing (preview) is a new feature that makes it possible for Fabric users to share data from within their Fabric tenant with users in another Fabric tenant. |
| May 2024 | Task flows in Microsoft Fabric (preview) | The preview of task flows in Microsoft Fabric is enabled for all Microsoft Fabric users. With Fabric task flows, when designing a data project, you no longer need to use a whiteboard to sketch out the different parts of the project and their interrelationships. Instead, you can use a task flow to build and bring this key information into the project itself. |
| May 2024 | Power BI: Subscriptions, licenses, and trials | Information on Power BI implementation planning and key considerations for planning subscriptions, licenses, and trials for Power BI and Fabric. |
| May 2024 | Register for the Microsoft Build: Microsoft Fabric Cloud Skills Challenge | Starting May 21, 2024, sign up for the Microsoft Build: Microsoft Fabric Cloud Skills Challenge and prepare for Exam DP-600 and upskill to the Fabric Analytics Engineer Associate certification. |
| March 2024 | Microsoft Fabric is now HIPAA compliant | We are excited to announce that Microsoft Fabric, our all-in-one analytics solution for enterprises, has achieved new certifications for HIPAA and ISO 27017, ISO 27018, ISO 27001, ISO 27701. |
| March 2024 | Folder in Workspace preview | As an organizational unit in the workspace, folder addresses this pain point by providing a hierarchical structure for organizing and managing your items. For more information, see Create folders in workspaces (preview). |
| March 2024 | Fabric Copilot Pricing: An End-to-End example | Copilot in Fabric begins billing on March 1, 2024 as part of your existing Power BI Premium or Fabric Capacity. Learn how Fabric Copilot usage is calculated. |
| March 2024 | Capacity Platform Updates for Pause/Resume, Capacity Metrics, virtualized items and workspaces for Copilot, and VNET Gateways | The Fabric Capacity Platform now supports usage reporting for Pause/Resume, virtualized items and workspaces supporting Copilot, Capacity Metrics, and VNET Gateway. For more information, read Capacity Platform Updates for Pause Resume and Capacity Metrics for Copilot and VNET Gateways. |
| February 2024 | Managed private endpoints for Microsoft Fabric (Preview) | Managed private endpoints for Microsoft Fabric (preview) allow secure connections to data sources that are behind a firewall or not accessible from the public internet. Workspaces with managed private endpoints have network isolation through a managed virtual network created by Microsoft Fabric. For more information, see Introducing Managed private endpoints for Microsoft Fabric preview. |
| February 2024 | Azure Private Link Support for Microsoft Fabric (Preview) | Azure Private Link for Microsoft Fabric secures access to your sensitive data in Microsoft Fabric by providing network isolation and applying required controls on your inbound network traffic. For more information, see Announcing Azure Private Link Support for Microsoft Fabric in Preview. |
| February 2024 | Domains in OneLake (preview) | Domains in OneLake help you organize your data into a logical data mesh, allowing federated governance and optimizing for business needs. You can now create sub domains, default domains for users, and move workspaces between domains. For more information, see Fabric domains. |
| February 2024 | Customizable Fabric navigation bar | You can now customize your preferred entry points in the navigation bar, including pinning common entry points and unpinning rarely used options. |
| February 2024 | Persistent filters in workspace | You can now save selected filters in workspace list view, and they'll be automatically applied the next time you open the workspace. |
| December 2023 | Microsoft Fabric Admin APIs preview | Fabric Admin APIs are designed to streamline administrative tasks. The initial set of Fabric Admin APIs is tailored to simplify the discovery of workspaces, Fabric items, and user access details. |
| December 2023 | Workspace retention changes in Fabric and Power BI | The retention period for collaborative workspaces is configurable from 7 to 90 days. The workspace retention setting is enabled by default and the default retention period is seven days. |
| November 2023 | Fabric workloads are now generally available! | Microsoft Fabric is now generally available! Microsoft Fabric Synapse Data Warehouse, Data Engineering & Data Science, Real-Time Analytics, Data Factory, OneLake, and the overall Fabric platform are now generally available. |
| November 2023 | Microsoft Fabric User APIs preview | We're happy to announce the preview of Microsoft Fabric User APIs. The Fabric user APIs are a major enabler for both enterprises and partners to use Microsoft Fabric as they enable end-to-end fully automated interaction with the service, enable integration of Microsoft Fabric into external web applications, and generally enable customers and partners to scale their solutions more easily. |
| October 2023 | Item type icons | Our design team has completed a rework of the item type icons across the platform to improve visual parsing. |
| October 2023 | Keyword-Based Filtering of Tenant Settings | Microsoft Fabric has recently introduced keyword-based filtering for the tenant settings page in the admin portal. |
| September 2023 | Monitoring hub – column options | Column options inside the monitoring hub give users a better customization experience and more room to operate. |
| September 2023 | OneLake File Explorer v1.0.10 | The OneLake file explorer automatically syncs all Microsoft OneLake items that you have access to in Windows File Explorer. With the latest version, you can seamlessly transition between using the OneLake file explorer app and the Fabric web portal. You can also right-click on the OneLake icon in the Windows notification area, and select Diagnostic Operations to view client-site logs. Learn more about easy access to open workspaces and items online. |
| August 2023 | Multitasking navigation improvement | Now, all Fabric items are opened in a single browser tab on the navigation pane, even in the event of a page refresh. This ensures you can refresh the page without the concern of losing context. |
| August 2023 | Monitoring Hub support for personalized column options | We have updated Monitoring Hub to allow users to personalize activity-specific columns. You now have the flexibility to display columns that are relevant to the activities you're focused on. |
| July 2023 | New OneLake file explorer update with support for switching organizational accounts | With OneLake file explorer v1.0.9.0, it's simple to choose and switch between different Microsoft Entra ID (formerly Azure Active Directory) accounts. |
| July 2023 | Help pane | The Help pane is feature-aware and displays articles about the actions and features available on the current Fabric screen. For more information, see Help pane in the monthly Fabric update. |
Continuous Integration/Continuous Delivery (CI/CD) in Microsoft Fabric
This section includes guidance and documentation updates on development process, tools, and versioning in the Microsoft Fabric workspace.
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Introducing New Branching Capabilities in Fabric Git Integration | New branching capabilities in Fabric Git integration include a redesigned Source Control pane, the ability to quickly create a new connected workspace and branch, and contextual related branches to find content related to the current workspace. |
| May 2024 | Deployment pipelines APIs for CI/CD | Fabric deployment pipelines APIs have been introduced, starting with the 'Deploy' API, which will allow you to deploy the entire workspace, or only selected items. |
| May 2024 | New items in Fabric CI/CD | Data pipelines, Warehouse, Spark, and Spark jobs are now available for CI/CD in git integration and deployment pipelines. |
| April 2024 | Introducing Trusted Workspace Access in Fabric Data Pipelines | Create data pipelines in Fabric to access your firewall-enabled ADLS Gen2 storage accounts with ease and security. This feature leverages the workspace identity to establish a secure and seamless connection between Fabric and your storage accounts. |
| March 2024 | CI/CD for Fabric Data Pipelines preview | Git Integration and integration with built-in Deployment Pipelines to Data Factory data pipelines is now in preview. For more information, see Data Factory Adds CI/CD to Fabric Data Pipelines. |
| March 2024 | System file updates for Git integration | The automatically generated system files item.metadata.json and item.config.json have been consolidated into a single system file .platform. For more information, see Automatically generated system files. |
| February 2024 | REST APIs for Fabric Git integration | REST APIs for Fabric Git integration enable seamless incorporation of Fabric Git integration into your team's end-to-end CI/CD pipeline, eliminating the need for manual triggering of actions from Fabric. |
| February 2024 | Delegation for Git integration settings | To enable more control over Git related settings, a tenant admin can now delegate these settings to both capacity admins and workspace admins via the What is the admin portal? |
| November 2023 | Microsoft Fabric User APIs | Microsoft Fabric User APIs are now available. The Fabric user APIs are a major enabler for both enterprises and partners to use Microsoft Fabric as they enable end-to-end fully automated interaction with the service, enable integration of Microsoft Fabric into external web applications, and generally enable customers and partners to scale their solutions more easily. |
| November 2023 | Notebook in Deployment Pipeline Preview | Now you can also use notebooks to deploy your code across different environments, such as development, test, and production. You can also use deployment rules to customize the behavior of your notebooks when they're deployed, such as changing the default Lakehouse of a Notebook. Get started with deployment pipelines, and Notebook shows up in the deployment content automatically. |
| November 2023 | Notebook Git integration preview | Fabric notebooks now offer Git integration for source control using Azure DevOps. It allows users to easily control the notebook code versions and manage the Git branches by leveraging the Fabric Git functions and Azure DevOps. |
| November 2023 | Notebook REST APIs Preview | With REST Public APIs for the Notebook items, data engineers/data scientists can automate their pipelines and establish CI/CD conveniently and efficiently. The notebook Restful Public API can make it easy for users to manage and manipulate Fabric notebook items and integrate notebook with other tools and systems. |
| November 2023 | Lakehouse support for git integration and deployment pipelines (preview) | The Lakehouse item now integrates with the lifecycle management capabilities in Microsoft Fabric, providing a standardized collaboration between all development team members throughout the product's life. Lifecycle management facilitates an effective product versioning and release process by continuously delivering features and bug fixes into multiple environments. |
| November 2023 | SQLPackage support for Fabric Warehouse | SQLPackage now supports Fabric Warehouse. SqlPackage is a command-line utility that automates the following database development tasks by exposing some of the public Data-Tier Application Framework (DacFx) APIs. The SqlPackage command line tool allows you to specify these actions along with action-specific parameters and properties. |
| September 2023 | SQL Projects support for Warehouse in Microsoft Fabric | Microsoft Fabric Data Warehouse is now supported in the SQL Database Projects extension available inside of Azure Data Studio and Visual Studio Code. |
| September 2023 | Notebook file system support in Synapse VS Code extension | The Synapse VS Code extension now supports notebook File System for Data Engineering and Data Science in Microsoft Fabric. The Synapse VS Code extension empowers users to develop their notebook items directly within the Visual Studio Code environment. |
| September 2023 | Deployment pipelines now support warehouses | Deployment pipelines enable creators to develop and test content in the service before it reaches the users. Supported content types include reports, paginated reports, dashboards, semantic models, dataflows, and now warehouses. Learn how to deploy content programmatically using REST APIs and DevOps. |
| September 2023 | Git integration with paginated reports in Power BI | You can now publish a Power BI paginated report and keep it in sync with your git workspace. Developers can apply their development processes, tools, and best practices. |
| August 2023 | Introducing the dbt adapter for Synapse Data Warehouse in Microsoft Fabric | The dbt adapter allows you to connect and transform data into Synapse Data Warehouse. The Data Build Tool (dbt) is an open-source framework that simplifies data transformation and analytics engineering. |
| May 2023 | Introducing git integration in Microsoft Fabric for seamless source control management | While developing in Fabric, developers can back up and version their work, roll back as needed, collaborate, or work in isolation using git branches. Read more about connecting the workspace to an Azure repo. |
Continuous Integration/Continuous Delivery (CI/CD) samples
| Month | Feature | Learn more |
|---|---|---|
| June 2024 | Getting started with development in isolation using a Private Workspace | In this walkthrough, we'll talk about how to set up git for a private workspace from a main branch, which is connected to a shared dev team workspace and then how to commit changes from the private workspace into the main branch of the shared workspace. |
| March 2024 | Microsoft Fabric Lifecycle Management – Getting started with Git Integration and Deployment Pipelines | Learn the essentials of Lifecycle Management through a demo scenario, and explore what Lifecycle Management is, and what it means in Fabric. |
Fabric Data Activator
This section summarizes archived new features and capabilities of Data Activator in Microsoft Fabric.
| Month | Feature | Learn more |
|---|---|---|
| October 2023 | Announcing the Data Activator preview | We're thrilled to announce that Data Activator is now in preview and is enabled for all existing Microsoft Fabric users. |
| August 2023 | Updated preview experience for trigger design | We have been working on a new experience for designing triggers and it's now available in our preview! You now see three cards in every trigger: Select, Detect, and Act. |
| May 2023 | Driving actions from your data with Data Activator | Data Activator is a new no-code Microsoft Fabric experience that empowers the business analyst to drive actions automatically from your data. To learn more, sign up for the Data Activator limited preview. |
Fabric and Microsoft 365
This section includes articles and announcements about Microsoft Fabric integration with Microsoft Graph and Microsoft 365.
| Month | Feature | Learn more |
|---|---|---|
| March 2024 | Analyze Dataverse tables from Microsoft Fabric | When creating a shortcut within Fabric, you will now see an option for Dataverse. When you choose this shortcut type and specify your Dataverse environment details, you can quickly see and work with the tables from that environment. |
| November 2023 | Fabric + Microsoft 365 Data: Better Together | Microsoft Graph is the gateway to data and intelligence in Microsoft 365. Microsoft 365 Data Integration for Microsoft Fabric enables you to manage your Microsoft 365 alongside your other data sources in one place with a suite of analytical experiences. |
| November 2023 | Microsoft 365 connector now supports ingesting data into Lakehouse (preview) | The Microsoft 365 connector now supports ingesting data into Lakehouse tables. |
| October 2023 | Microsoft OneLake adds shortcut support to Power Platform and Dynamics 365 | You can now create shortcuts directly to your Dynamics 365 and Power Platform data in Dataverse and analyze it with Microsoft Fabric alongside the rest of your OneLake data. There's no need to export data, build ETL pipelines, or use partner integration tools. |
| May 2023 | Step-by-Step Guide to Enable Microsoft Fabric for Microsoft 365 Developer Account | This blog reviews how to enable Microsoft Fabric with a Microsoft 365 Developer Account and the Fabric Free Trial. |
| May 2023 | Microsoft 365 Data + Microsoft Fabric better together | Microsoft 365 Data Integration for Microsoft Fabric enables you to manage your Microsoft 365 alongside your other data sources in one place with a suite of analytical experiences. |
Migration
This section includes guidance and documentation updates on migration to Microsoft Fabric.
| Month | Feature | Learn more |
|---|---|---|
| February 2024 | Mapping Azure Synapse dedicated SQL pools to Fabric data warehouse compute | Read for guidance on mapping Data Warehouse Units (DWU) from Azure Synapse Analytics dedicated SQL pool to an approximate equivalent number of Fabric Capacity Units (CU). |
| November 2023 | Migrate from Azure Synapse dedicated SQL pools | A detailed guide with a migration runbook is available for migrations from Azure Synapse Data Warehouse dedicated SQL pools into Microsoft Fabric. |
| November 2023 | Migrating from Azure Synapse Spark to Fabric | A detailed set of articles on migration of Azure Synapse Spark to Microsoft Fabric, including a migration process that can involve multiple scenarios and phases. |
| July 2023 | Fabric Changing the game – OneLake integration | This blog post covers OneLake integrations and multiple scenarios to ingest the data inside of Fabric OneLake, including ADLS, ADF, OneLake Explorer, Databricks. |
| June 2023 | Microsoft Fabric changing the game: Exporting data and building the Lakehouse | This blog post covers the scenario to export data from Azure SQL Database into OneLake. |
| June 2023 | Copy data to Azure SQL at scale with Microsoft Fabric | Did you know that you can use Microsoft Fabric to copy data at scale from supported data sources to Azure SQL Database or Azure SQL Managed Instance within minutes? |
| June 2023 | Bring your Mainframe DB2 z/OS data to Microsoft Fabric | In this blog, we review the convenience and ease of opening DB2 for z/OS data in Microsoft Fabric. |
Monitor
This section includes guidance and documentation updates on monitoring your Microsoft Fabric capacity and utilization, including the Monitoring hub.
| Month | Feature | Learn more |
|---|---|---|
| March 2024 | Capacity Metrics support for Pause and Resume | Fabric Capacity Metrics has been updated with new system events and reconciliation logic to simplify analysis of paused capacities. Fabric Pause and Resume is a capacity management feature that lets you pause F SKU capacities to manage costs. When your capacity isn't operational, you can pause it to enable cost savings and then later, when you want to resume work on your capacity you can reactivate it. |
| October 2023 | Throttling and smoothing in Synapse Data Warehouse | A new article helps you understand Fabric capacity throttling. Throttling occurs when a tenant's capacity consumes more capacity resources than it has purchased over a period of time. |
| September 2023 | Monitoring hub - column options | Users can select and reorder the columns according to their customized needs in the Monitoring hub. |
| September 2023 | Fabric Capacities – Everything you need to know about what's new and what's coming | Read more about the improvements we're making to the Fabric capacity management platform for Fabric and Power BI users. |
| September 2023 | Microsoft Fabric Capacity Metrics | The Microsoft Fabric Capacity Metrics app is available in App Source for a variety of billing and utilization reporting. |
| August 2023 | Monitoring Hub support for personalized column options | The Monitoring Hub to allow users to personalize activity-specific columns. You now have the flexibility to display columns that are relevant to the activities you're focused on. |
| May 2023 | Capacity metrics in Microsoft Fabric | Learn more about the universal compute capacities and Fabric's capacity metrics governance features that admins can use to monitor usage and make data-driven scale-up decisions. |
Microsoft Purview
This section summarizes archived announcements about governance and compliance capabilities with Microsoft Purview in Microsoft Fabric. Learn more about Information protection in Microsoft Fabric.
| Month | Feature | Learn more |
|---|---|---|
| May 2023 | Administration, Security and Governance in Microsoft Fabric | Microsoft Fabric provides built-in enterprise grade governance and compliance capabilities, powered by Microsoft Purview. |
Related content
- Modernization Best Practices and Reusable Assets Blog
- Azure Data Explorer Blog
- Get started with Microsoft Fabric
- Microsoft Training Learning Paths for Fabric
- End-to-end tutorials in Microsoft Fabric
- Fabric Known Issues
- Microsoft Fabric Blog
- Microsoft Fabric terminology
- What's new in Power BI?
- What's new in Microsoft Fabric?