What's new and planned for Real-Time Intelligence in Microsoft Fabric
Important
The release plans describe functionality that may or may not have been released yet. The delivery timelines and projected functionality may change or may not ship. Refer to Microsoft policy for more information.
Real-Time Intelligence enables data professionals, analysts, and business users to ingest, process, analyze, transform, visualize and act on large, time-sensitive and highly granular data to help organizations make faster and more informed business decisions. It is an end-to-end experience that enables seamless handling of real-time data without the need to land it first. Real-Time Intelligence builds on the existing Real-Time Analytics and Data Activator capabilities and offers new features making it even easier for users of all skill levels to get the most from their real-time data. This includes ingesting streaming data with high granularity, dynamically transform streaming data, query data in real-time for instant insights, and trigger actions like alerting a production manager when equipment is overheating or rerunning jobs when data pipelines fail.
Even though it’s called “real-time”, your data doesn’t have to be flowing at high rates and volumes to get true business value. Real-Time Intelligence gives you event-driven, rather than schedule-driven solutions, ensuring you can make the best decisions on time.
Behind this powerful workload is the Real-Time hub, a single place to discover, manage and use event steaming data from Fabric, other Microsoft data sources, and external data sources. Just like the OneLake data hub makes it easy to discover, manage, and use the data at rest, the Real-Time hub can help you do the same for data in motion. All events that flow through the Real-Time hub can be easily transformed and routed to any Fabric data stores and can create new streams that can be discovered and consumed.
For more information, see the Real-Time Intelligence documentation and visit the announcement blog.
The items available in Real-Time Intelligence are:
Real-Time hub, a single place to discover, manage and use event steaming data from Fabric, other Microsoft data sources, and external data sources
Eventstreams for capturing, transforming, and routing real-time events to various destinations with a no-code experience.
Eventhouse and KQL database are the ideal analytics engine to process data in motion. They're tailored to time-based, streaming events with structured, semi structured, and unstructured data. This data is automatically indexed and partitioned based on ingestion time, giving you incredibly fast and complex analytic querying capabilities on high-granularity data. Data stored in event houses can be made available in OneLake for consumption by other Fabric experiences.
KQL queryset to run queries, view, and customize query results on data. The KQL queryset allows you to save queries for future use, export, and share queries with others. It includes the option to generate a Power BI report.
Real-time dashboards, which contains a collection of tiles. Each tile has an underlying query and a visual representation perfect for data exploration, monitoring, and forensics.
Data Activator that integrates with Real-Time hub, eventstreams, real-time dashboards and KQL querysets, making it seamless to trigger on any patterns or changing in real-time data.
Investment areas
Azure Synapse Link from Cosmos DB to Fabric Eventhouse
Estimated release timeline: Q3 2024
Customers can stream all updates from Cosmos DB to a KQL database, enabling large scale, powerful analytics.
Multivariate Anomaly Detection
Estimated release timeline: Q3 2024
The Azure Anomaly Detector multi variate technology will be natively available in Fabric. Train the detector using spark and notebooks or using Eventhouses, and use it to detect anomalies in real time as new data arrives.
Eventhouse Platform Monitoring
Estimated release timeline: Q3 2024
Release Type: Public preview
Access Eventhouse logs and metrics to fine tune performance, troubleshoot issues and identify anomalies. The Eventhouse will report on Query logs, Ingestion metrics, and Ingestion failure events. The Eventhouse event and metric logs will be stored in a read only KQL database. Users will be able to analyze the events using KQL Querysets and Real time Dashboards.
CI/CD support for Eventstream items
Estimated release timeline: Q3 2024
Release Type: Public preview
Customers can easily version and deploy their eventstreams across development, testing, and production workspaces using the integrated Git integration and deployment experience in the Fabric platform.
More trigger options
Estimated release timeline: Q3 2024
Release Type: Public preview
To address further requests for monitoring different business conditions, Data Activator will support triggers that check date/time values, basic math functions etc.
Low latency triggers
Estimated release timeline: Q3 2024
We hear in customer feedback that triggers need to react faster when the specified condition is met. Data Activator will support lower-latency triggers in scenarios with simple conditions and lower volumes of data.This feature will be Private Preview in this time frame.
Set Alerts on KQL Querysets
Estimated release timeline: Q3 2024
Release Type: Public preview
Data Activator will be able to monitor the results of queries managed in a KQL Queryset, running the query on a regular schedule and acting on the results.
New streaming connectors for Kafka public clusters and more instances of SQL Server DB CDC
Estimated release timeline: Q3 2024
Release Type: Public preview
Additional connectors will be available through the Get Events experience, including Public Kafka clusters and SQL Server DB CDC.
Eventstream integration with Managed Private Endpoint
Estimated release timeline: Q3 2024
Release Type: Public preview
This integration with Managed Private Endpoints enables secure and private access to data sources behind a firewall or blocked from public internet access, allowing data to be streamed to Fabric Eventstream over outbound connections.
Eventhouse destination support Eventstreams
Estimated release timeline: Q3 2024
Release Type: Public preview
Eventstreams will add support for Eventhouse as a destination.
KQL Database Entities Diagram
Estimated release timeline: Q4 2024
Release Type: Public preview
a visual representation of the KQL DB entities and their relations (such as tables, functions, materialized views, and more).
Webhook support
Estimated release timeline: Q1 2025
Release Type: Public preview
Webhooks represent a large base of application integrations that can be used in different scenarios. Data Activator will support actions that call webhooks when triggers fire.
Support for parameters in triggering Fabric actions and webhooks
Estimated release timeline: Q1 2025
Release Type: Public preview
Triggers that take action by starting Fabric jobs or calling webhooks will be able to pass the value of properties in the reflex model as parameters to the job or webhook.
Shipped feature(s)
Set Alert on Real-Time Dashboards
Shipped (Q2 2024)
Release Type: Public preview
In addition to monitoring Power BI and Eventstream data, Data Activator will support periodic querying against KQL queries saved in Real-Time Dashboards. These queries can be used to return exceptions, or track results over time to detect data changes or threshold conditions.
Real-Time Hub for all data in motion
Shipped (Q2 2024)
Release Type: Public preview
Real-Time hub is the single estate for all data-in-motion across your entire organization.
Get events experience in Real-Time Hub (change to explain value or remove for 18/24)
Shipped (Q2 2024)
Release Type: Public preview
Whether data is coming from new or existing sources, streams, or available events, the Get Events experience allows users to connect to a wide range of sources directly from Real-Time hub, Eventstreams, Eventhouse and Data Activator.
New streaming connectors including Confluent Kafka, AWS Kinesis, PostgresSQL CDC and more
Shipped (Q2 2024)
Release Type: Public preview
Beyond Microsoft sources, users can now stream data from other platforms like Google Cloud, Amazon Kinesis, Database change data capture streams, etc. using our new messaging connectors. Our messaging connectors are powered by Kafka connect with Camel Kafka connectors that offer a pluggable, declarative data integration framework, and flexibility for diverse source connections on popular data platforms, as well as the Debezium that could connect databases for fetching the Change Data Capture (CDC) streams.
New sources of discrete events from Fabric system events and Azure storage events
Shipped (Q2 2024)
Release Type: Public preview
Discrete events (a.k.a. notification events) are significant (important) facts that have happened, and that trigger state changes or actions in the downstream consumer applications (subscribers). Examples include events like OrderPlaced in order system, blob storage changes in Azure sources, or item changes in Fabric system.
Source from Real-Time Hub in Enhanced Eventstream
Shipped (Q2 2024)
Release Type: Public preview
The Eventstream homepage makes it even easier to bring in data. By clicking on the “Add external source”, you will find these sources in the Get events wizard that helps you to set up the source in a few steps. After you add the source to your eventstream, you can publish it to stream the data into your eventstream. Using Eventstream with discrete sources to turn events into streams for more analysis. You can send the streams to different Fabric data destinations, like Lakehouse and KQL Database.
Eventstream Edit and Live modes
Shipped (Q2 2024)
Release Type: Public preview
Eventstream offers two distinct modes, Edit and Live, to provide flexibility and control over the development process of your eventstream. If you create a new Eventstream with Enhanced Capabilities enabled, you can modify it in an Edit mode. Here, you can design stream processing operations for your data streams using a no-code editor. Once you complete the editing, you can publish your Eventstream and visualize how it starts streaming and processing data in Live mode.
Default and Derived Streams
Shipped (Q2 2024)
Release Type: Public preview
Multiple streams are listed within the Real-Time Hub including Default and Derived streams. Default stream are automatically generated when a streaming source is added to Eventstream. Default stream captures raw event data directly from the source, ready for transformation or analysis. erived stream: A specialized stream that users can create as a destination within Eventstream. Derived stream can be created after a series of operations such as filtering and aggregating, and then it’s ready for further consumption or analysis by other users in the organization through the Real-Time Hub.
Co-pilot for Real-Time Intelligence
Shipped (Q2 2024)
Release Type: Public preview
Co-pilot allows you to write queries in natural language and have them translated into Kusto Query Language (KQL). You can use Co-pilot to ask your how-to queries, explore your data in a KQL database, and create Kusto entities such as tables, functions, and materialized views.
Real-Time Dashboard
Shipped (Q2 2024)
Release Type: Public preview
A dashboard is a collection of tiles, optionally organized in pages, where each tile has an underlying query and a visual representation. Natively export Kusto Query Language (KQL) queries to a dashboard as visuals and later modify their underlying queries and visual formatting as needed. In addition to ease of data exploration, this fully integrated dashboard experience provides improved query and visualization performance with light, flexible modeling on high granularity low latency data.
Content-based event routing to destinations in eventstreams
Shipped (Q2 2024)
Release Type: Public preview
Bringing the no-code event processor to the eventstream main canvas allows customers to route event data to their destinations based on event content, using event processing logic defined with the event processor.
Create actions and alerts with Data Activator
Shipped (Q2 2024)
This feature provides a low-code/no-code experience to drive actions and alerts from your KQL database data presented in a Real-Time Dashboard. Data Activator gives you a single place to define actionable patterns in your data. These patterns can range from simple thresholds (such as a value being exceeded) to more complex patterns over time (such a value trending down). When Data Activator detects an actionable pattern, it triggers an action. That action can be an email or a Teams alert to the relevant person in your organization. It can also trigger an automatic process, via a Power Automate flow or an action in one of your organization’s line-of-business apps.
Pause, stop, or restart eventstream flows
Shipped (Q2 2024)
This feature offers customers the capability to pause, stop, or restart real-time event flows, so they can iterate on transformation or event flow logic during the development phase.
Eventhouse
Shipped (Q2 2024)
Release Type: General availability
Eventhouse, a cutting-edge database workspace meticulously crafted to manage and store event-based data, is now officially available for general use. With Eventhouse, users can perform high-performance analysis of big data and real-time data querying, processing billions of events within seconds. The platform allows users to organize data into compartments (databases) within one logical item, facilitating efficient data management. Additionally, Eventhouse enables the sharing of compute and cache resources across databases, maximizing resource utilization.
Eventhouse OneLake Availability
Shipped (Q2 2024)
Release Type: General availability
Enabling data availability of Eventhouse in OneLake means that customers can enjoy the best of both worlds: they can query the data with high performance and low latency in their Eventhouse and query the same data in Delta Lake format via any other Fabric engines such as Power BI Direct Lake mode, Warehouse, Lakehouse, Notebooks, and more.
Create a database shortcut to another KQL Database
Shipped (Q2 2024)
Release Type: Public preview
A database shortcut in Eventhouse is an embedded reference to a source database. The database shortcut is attached in read-only mode, making it possible to view and run queries on the data that was ingested into the source KQL Database without ingesting it. This helps with data sharing scenarios where you can share data in-place either within teams, or even with external customers.
Support for AI Anomaly Detector
Shipped (Q2 2024)
Release Type: Public preview
AI Anomaly Detector algorithms are supported in Microsoft Fabric, allowing for real time scoring by KQL with inline Python in Real-Time Intelligence.
Eventhouse tenant level private endpoint support
Shipped (Q2 2024)
Release Type: Public preview
Customers can increase their network security by limiting access to Eventhouse at a tenant-level, from one or more virtual networks (VNets) via private links. This will prevent unauthorized access from public networks and only permit data plane operations from specific VNets.
New experience for data exploration from Real-Time Dashboards
Shipped (Q2 2024)
Release Type: Public preview
Directly from a real-time dashboard, users can refine their exploration using a user-friendly, form-like interface. This intuitive and dynamic experience is tailored for insights explorers craving insights based on real-time data. Add filters, create aggregations, and switch visualization types without writing queries to easily uncover insights.
Use Real-Time hub to Get Data in KQL Database in Eventhouse
Shipped (Q2 2024)
Release Type: Public preview
With Real-Time hub embedded in KQL Database experience, each user in the tenant can view and add streams which they have access and directly ingest it to a KQL Database table in Eventhouse.
Get data from Real-Time Hub within Data Activator
Shipped (Q2 2024)
Release Type: Public preview
Bring new events into reflex items by using the embedded Real-Time Hub browser.
Create triggers from Real-Time Hub
Shipped (Q2 2024)
Release Type: Public preview
Across the Real-Time Hub you'll see options to create new triggers and alerts on both business and system events.
Taking action through Fabric Items
Shipped (Q2 2024)
Release Type: Public preview
Reflex items will support starting Fabric item jobs as an action. You'll be able to kick off data pipelines and notebooks.
Enhanced trigger options
Shipped (Q2 2024)
To represent the business conditions for monitoring, Data Activator will support more trigger conditions such as percentage changes over time, date/time functions, basic mathematic functions, signal absence detection and more.
Notebook integration
Shipped (Q1 2024)
Customers can natively query KQL databases from Microsoft Fabric notebooks. Use notebooks to run queries on KQL databases, save the resulting data frames, ingest data into KQL databases, and more.
Scale up to 100 MB/s for eventstream processing
Shipped (Q1 2024)
An eventstream can support events ingestion throughput up to 100 MB/s while ingesting data to KQL databases and lakehouses.
Customizable event data retention in eventstreams
Shipped (Q1 2024)
Customers have the flexibility to set the event data retention time, ensuring that event data is retained in their eventstreams according to their specific business requirements.
Enhancements to Event processor in Eventstream
Shipped (Q1 2024)
This includes three enhancements. First, personalize operation nodes and easily filter out 'null' values from your data. Second, manage and rename your column fields easily in the Aggregate operation. Third, Change your values to different data types using the Manage Fields operation .
Table optimization shortcut in eventstream lakehouse destinations
Shipped (Q1 2024)
The table optimization shortcut helps users by opening a notebook with a Spark job that compacts small streaming files in the lakehouse table.
Enhanced Custom App Details tab offers comprehensive endpoint information
Shipped (Q1 2024)
The Details tab of the Custom App source and destination offers comprehensive information about the eventstream endpoint. Available information includes connection strings in Kafka, Eventhub, and AMQP formats, as well as sample Java code for event ingestion and consumption using these three protocols.
Improved no-code stream processing designer
Shipped (Q1 2024)
Users have an improved eventstream editing experience, with more intuitive gestures in the no-code designer.
"Get data from Eventstream" in multiple Fabric items
Shipped (Q1 2024)
Customers are able to get event data from eventstreams in multiple Fabric items, including Lakehouse, KQL Database, and Reflex.
Delta support to the KQL database OneLake representation
Shipped (Q1 2024)
Release Type: Public preview
KQL database data is already available in OneLake in parquet format. Adding Delta support enables other Fabric capabilities to use the OneLake presence as a table rather than as a list of files.
Administrative monitoring
Shipped (Q1 2024)
You can use administrative monitoring to search audit logs, and to track usage and adoption, capacity consumption, and availability.
New eventstream destination to Data Activator
Shipped (Q1 2024)
The Data Activator destination option helps customers route events to Data Activator, where they can build triggers based on the event data from eventstreams natively.
Autoscale eventstreams
Shipped (Q1 2024)
This feature provides the flexibility to automatically scale capacity in response to event data traffic volume, enabling seamless improvement of eventstream throughput without disrupting your business operations.
Eventstream data preprocessing to KQL database destination
Shipped (Q1 2024)
The addition of the KQL database destination with data transformation enables customers to transform, enrich, and reduce the data volume before it's routed to KQL databases.
Improved Get Data experience
Shipped (Q1 2024)
Users experience a simplified and improved Get Data experience to get their data into a KQL database.
Cloud connection creation embedded in eventstreams
Shipped (Q1 2024)
Customers can create a cloud connection directly within their eventstreams, without navigating to another webpage.
Two ingestion modes for eventstream lakehouse destinations
Shipped (Q1 2024)
Eventstream lakehouse destinations support two ingestion modes: low latency and high throughput.