Share via


What's new and planned for Real-Time Intelligence in Microsoft Fabric

Important

The release plans describe functionality that may or may not have been released yet. The delivery timelines and projected functionality may change or may not ship. Refer to Microsoft policy for more information.

Real-Time Intelligence enables data professionals, analysts, and business users to ingest, process, analyze, transform, visualize and act on large, time-sensitive and highly granular data to help organizations make faster and more informed business decisions. It is an end-to-end experience that enables seamless handling of real-time data without the need to land it first. Real-Time Intelligence builds on the existing Real-Time Analytics and Data Activator capabilities and offers new features making it even easier for users of all skill levels to get the most from their real-time data. This includes ingesting streaming data with high granularity, dynamically transform streaming data, query data in real-time for instant insights, and trigger actions like alerting a production manager when equipment is overheating or rerunning jobs when data pipelines fail.

Even though it’s called “real-time”, your data doesn’t have to be flowing at high rates and volumes to get true business value. Real-Time Intelligence gives you event-driven, rather than schedule-driven solutions, ensuring you can make the best decisions on time.

Behind this powerful workload is the Real-Time hub, a single place to discover, manage and use event steaming data from Fabric, other Microsoft data sources, and external data sources. Just like the OneLake data hub makes it easy to discover, manage, and use the data at rest, the Real-Time hub can help you do the same for data in motion. All events that flow through the Real-Time hub can be easily transformed and routed to any Fabric data stores and can create new streams that can be discovered and consumed.

For more information, see the Real-Time Intelligence documentation and visit the blog.

The items available in Real-Time Intelligence are:

  • Real-Time hub, a single place to discover, manage and use event steaming data from Fabric, other Microsoft data sources, and external data sources

  • Eventstreams for capturing, transforming, and routing real-time events to various destinations with a no-code experience.

  • Eventhouse and KQL database are the ideal analytics engine to process data in motion. They're tailored to time-based, streaming events with structured, semi structured, and unstructured data. This data is automatically indexed and partitioned based on ingestion time, giving you incredibly fast and complex analytic querying capabilities on high-granularity data. Data stored in event houses can be made available in OneLake for consumption by other Fabric experiences.

  • KQL queryset to run queries, view, and customize query results on data. The KQL queryset allows you to save queries for future use, export, and share queries with others. It includes the option to generate a Power BI report.

  • Real-time dashboards, which contains a collection of tiles. Each tile has an underlying query and a visual representation perfect for data exploration, monitoring, and forensics.

  • Activator that integrates with Real-Time hub, eventstreams, real-time dashboards and KQL querysets, making it seamless to take action on any patterns or changes in real-time data.

Investment areas

Feature Estimated release timeline
CI/CD support for Eventstream items Q4 2024
Real-Time Dashboards General Availability Q4 2024
Data Exploration General Availability Q4 2024
Copilot for Real-Time Intelligence Q4 2024
New streaming connector for Azure Service Bus Q4 2024
Activator performance & scale improvements Q4 2024
Activator alerts on Power BI, KQL Queryset and Real-Time Dashboard Q4 2024
"Get Events" from Activator Q4 2024
Fabric item job actions Q4 2024
Maximum number of rule actions Q4 2024
Fabric Monitoring integration Q4 2024
Power BI alert management Q4 2024
Migration from Synapse Data Explorer pools to Fabric Eventhouse Q4 2024
Query Acceleration over OneLake Delta Tables Q4 2024
Additional Fabric system events Q4 2024
Create Real time dashboards with Copilot Q4 2024
Eventstream connector sources GA Q4 2024
Data insights (metrics) on connector sources in Eventstream Q4 2024
Runtime logs on connector sources in Eventstream Q4 2024
Eventstream connector source pausing and resuming Q4 2024
Processing and routing events to Data Activator destination Q4 2024
Entra ID authentication support for custom endpoint in Eventstream Q4 2024
New Fabric events sources: job events and onelake item events Q4 2024
Data preview on new sources: CDC source in Eventstream Q4 2024
Public (REST) APIs support for Eventstream items Q4 2024
Eventhouse Platform Monitoring Q4 2024
Copilot assisted Real-Time dashboard creation for KQL table in Real-Time Hub Q4 2024
Rich sample experience in Real-Time hub landing page Q4 2024
Simplified Eventhub connection to Eventstream Q4 2024
KQL Database Entities Diagram Q4 2024
Evenhouse table no code creation and editing experience Q4 2024
Enhanced Eventstream GA: Content-based routing, derived streams and Edit/Live modes Q4 2024
Explore data for KQL table in Real-Time hub Q4 2024
Eventstream to Activator events Q1 2025
Support for parameters in triggering Fabric actions Q1 2025
Fabric job action parameters Q1 2025
Activator delay tolerance controls Q1 2025
Event catalog integration Q1 2025
New Fabric events sources: capacity utilization events Q1 2025
SQL native support in KQL querysets Q2 2025
Real-Time Hub for all data in motion GA Shipped (Q4 2024)
Get events experience in Real-Time Hub (change to explain value or remove for 18/24) Shipped (Q4 2024)
Eventhouse ALM Support Shipped (Q3 2024)
Eventstream integration with Managed Private Endpoint Shipped (Q3 2024)
Eventhouse destination support Eventstreams Shipped (Q3 2024)
New streaming connectors for Kafka public clusters and more instances of SQL Server DB CDC Shipped (Q3 2024)
Set Alerts on KQL Querysets Shipped (Q3 2024)
More trigger options Shipped (Q3 2024)
Low latency triggers Shipped (Q3 2024)
Azure Synapse Link from Cosmos DB to Fabric Eventhouse Shipped (Q3 2024)
Multivariate Anomaly Detection Shipped (Q3 2024)
Real time dashboard in Fabric OrgApps Shipped (Q3 2024)
Real-Time Dashboard Shipped (Q2 2024)
Content-based event routing to destinations in eventstreams Shipped (Q2 2024)
Create actions and alerts with Data Activator Shipped (Q2 2024)
Pause, stop, or restart eventstream flows Shipped (Q2 2024)
Co-pilot for Real-Time Intelligence Shipped (Q2 2024)
Eventhouse Shipped (Q2 2024)
Eventhouse OneLake Availability Shipped (Q2 2024)
Create a database shortcut to another KQL Database Shipped (Q2 2024)
Support for AI Anomaly Detector Shipped (Q2 2024)
Eventhouse tenant level private endpoint support Shipped (Q2 2024)
New experience for data exploration from Real-Time Dashboards Shipped (Q2 2024)
Use Real-Time hub to Get Data in KQL Database in Eventhouse Shipped (Q2 2024)
Get data from Real-Time Hub within Data Activator Shipped (Q2 2024)
Create triggers from Real-Time Hub Shipped (Q2 2024)
Taking action through Fabric Items Shipped (Q2 2024)
Set Alert on Real-Time Dashboards Shipped (Q2 2024)
New streaming connectors including Confluent Kafka, AWS Kinesis, PostgresSQL CDC and more Shipped (Q2 2024)
New sources of discrete events from Fabric system events and Azure storage events Shipped (Q2 2024)
Source from Real-Time Hub in Enhanced Eventstream Shipped (Q2 2024)
Eventstream Edit and Live modes Shipped (Q2 2024)
Default and Derived Streams Shipped (Q2 2024)
Enhanced trigger options Shipped (Q2 2024)
Notebook integration Shipped (Q1 2024)
Scale up to 100 MB/s for eventstream processing Shipped (Q1 2024)
Customizable event data retention in eventstreams Shipped (Q1 2024)
Enhancements to Event processor in Eventstream Shipped (Q1 2024)
Table optimization shortcut in eventstream lakehouse destinations Shipped (Q1 2024)
Enhanced Custom App Details tab offers comprehensive endpoint information Shipped (Q1 2024)
Improved no-code stream processing designer Shipped (Q1 2024)
"Get data from Eventstream" in multiple Fabric items Shipped (Q1 2024)
New eventstream destination to Data Activator Shipped (Q1 2024)
Autoscale eventstreams Shipped (Q1 2024)
Delta support to the KQL database OneLake representation Shipped (Q1 2024)
Administrative monitoring Shipped (Q1 2024)
Eventstream data preprocessing to KQL database destination Shipped (Q1 2024)
Improved Get Data experience Shipped (Q1 2024)
Cloud connection creation embedded in eventstreams Shipped (Q1 2024)
Two ingestion modes for eventstream lakehouse destinations Shipped (Q1 2024)

CI/CD support for Eventstream items

Estimated release timeline: Q4 2024

Release Type: Public preview

Customers can easily version and deploy their eventstreams across development, testing, and production workspaces using the integrated Git integration and deployment experience in the Fabric platform.

Real-Time Dashboards General Availability

Estimated release timeline: Q4 2024

Release Type: General availability

General availability of Fabric Real-Time Dashboards

Data Exploration General Availability

Estimated release timeline: Q4 2024

Release Type: General availability

General availability of no-code data exploration experiences for Real-Time Intelligence, including the ability to explore real-time dashboards tile data and query results.

Copilot for Real-Time Intelligence

Estimated release timeline: Q4 2024

Release Type: General availability

Copilot for Real-Time Intelligence is an advanced AI tool designed to help you explore your data and extract valuable insights. You can input questions about your data, which are then automatically translated into Kusto Query Language (KQL) queries. Copilot streamlines the process of analyzing data for both experienced KQL users and citizen data scientists.

New streaming connector for Azure Service Bus

Estimated release timeline: Q4 2024

Release Type: Public preview

Customers can leverage new connector to bring the streaming events from Azure Service Bus to Eventstream.

Activator performance & scale improvements

Estimated release timeline: Q4 2024

Release Type: General availability

Activator items will support up to 1000 events per second per stream, and activations typically happen within 30 seconds of the event being received.

Activator alerts on Power BI, KQL Queryset and Real-Time Dashboard

Estimated release timeline: Q4 2024

Release Type: General availability

Activator will support monitoring data from Power BI reports, Querysets, and Real-Time Dashboards. You set up an alert and configure it in a side pane.

"Get Events" from Activator

Estimated release timeline: Q4 2024

Release Type: General availability

From within an activator item, you choose ‘Get data’ to pull events from Real-Time Hub. You get search and catalog capabilities over all streams in your organization, and can connect to external sources with connectors to streams, change data capture feeds and more.

Fabric item job actions

Estimated release timeline: Q4 2024

Release Type: General availability

Activator lets you start Fabric items as an action. For example, you can run a Fabric pipeline when files are created in Azure data lake storage. Fabric pipelines and Spark notebooks are supported.

Maximum number of rule actions

Estimated release timeline: Q4 2024

Release Type: General availability

To avoid overloading messaging systems and generating too much noise, Activator has limits on how many times a rule can send an email/Teams message or trigger a Power Automate flow. This stops users being flooded with messages.

Fabric Monitoring integration

Estimated release timeline: Q4 2024

Release Type: Public preview

Fabric Workspace Monitoring is designed to provide a seamless and consistent monitoring experience with end-to-end visibility across all Fabric items. Workspace Monitoring is based on the Real-time Intelligence Eventhouse KQL database. Once Fabric Monitoring is enabled, a KQL database is created to store all the workspace items event logs. KQL databases are ideal for time series logs and metrics monitoring solutions.

Power BI alert management

Estimated release timeline: Q4 2024

Release Type: General availability

Users will be able to see and manage all alerts configured with Activator alerts on their current Power BI report. Choosing ‘Set alert’ for a report will present a list of all the alerts that you set up on that report, giving you the option to add new ones, start or stop rules, or delete them if they’re no longer necessary.

Migration from Synapse Data Explorer pools to Fabric Eventhouse

Estimated release timeline: Q4 2024

Release Type: Public preview

Seamless migration from Synapse Data Explorer pools to Fabric Eventhouse.

Query Acceleration over OneLake Delta Tables

Estimated release timeline: Q4 2024

Release Type: Public preview

Running queries on shortcuts is typically slow due to various factors such as network calls to fetch data from storage, the absence of indexes, and more.
Customers who are used to running performant queries on top of ingested data in Eventhouse, will be able to gain similar performance by caching Delta tables with query acceleration .

Additional Fabric system events

Estimated release timeline: Q4 2024

Release Type: Public preview

More Fabric Event categories will be available for use in Real-Time hub, namely Fabric Job events, OneLake events

Create Real time dashboards with Copilot

Estimated release timeline: Q4 2024

Release Type: Public preview

Auto create Real-Time Dashboard for KQL tables, including metadata tiles and insights tiles generated by Copilot

Eventstream connector sources GA

Estimated release timeline: Q4 2024

Release Type: General availability

Connector sources to be GA. Connector source list:

  • Azure SQL Database Change Data Capture (CDC)
  • SQL Server on VM DB (CDC)
  • Azure SQL Managed Instance CDC
  • PostgreSQL Database CDC
  • MySQL Database CDC
  • Azure Cosmos DB CDC
  • Google Cloud Pub/Sub
  • Amazon Kinesis Data Streams
  • Confluent Cloud Kafka
  • Amazon MSK Kafka

Data insights (metrics) on connector sources in Eventstream

Estimated release timeline: Q4 2024

Release Type: General availability

Support the data insights (metrics) on connector sources

Runtime logs on connector sources in Eventstream

Estimated release timeline: Q4 2024

Release Type: General availability

Support the runtime logs on connector sources

Eventstream connector source pausing and resuming

Estimated release timeline: Q4 2024

Release Type: General availability

Support to pause and resume the connector sources.

Processing and routing events to Data Activator destination

Estimated release timeline: Q4 2024

Release Type: Public preview

Support to process the events before routing the events to data activator destination.

Entra ID authentication support for custom endpoint in Eventstream

Estimated release timeline: Q4 2024

Release Type: General availability

Entra ID authentication support in custom endpoint

New Fabric events sources: job events and onelake item events

Estimated release timeline: Q4 2024

Release Type: Public preview

New Fabric events sources: job events and onelake item events

Data preview on new sources: CDC source in Eventstream

Estimated release timeline: Q4 2024

Release Type: General availability

Support to preview the data on the database CDC connector sources inside Eventstream.

Public (REST) APIs support for Eventstream items

Estimated release timeline: Q4 2024

Release Type: Public preview

Eventstream Public APIs enable the e2e automated interaction with Eventstream item, and the integration of Eventstream item with external applications.

Eventhouse Platform Monitoring

Estimated release timeline: Q4 2024

Release Type: Public preview

Access Eventhouse logs and metrics to fine tune performance, troubleshoot issues and identify anomalies. The Eventhouse will report on Query logs, Ingestion metrics, and Ingestion failure events. The Eventhouse event and metric logs will be stored in a read only KQL database. Users will be able to analyze the events using KQL Querysets and Real time Dashboards.

Copilot assisted Real-Time dashboard creation for KQL table in Real-Time Hub

Estimated release timeline: Q4 2024

Release Type: Public preview

Customers can select a KQL table and generate a real-time dashboard with copilot.

Rich sample experience in Real-Time hub landing page

Estimated release timeline: Q4 2024

Release Type: General availability

When users first time come to Real-TIme hub, provide rich samples for them to try out.

Simplified Eventhub connection to Eventstream

Estimated release timeline: Q4 2024

Release Type: General availability

We will autopopulate Eventhub information so that customers can connect to Eventhubs within several clicks.

KQL Database Entities Diagram

Estimated release timeline: Q4 2024

Release Type: Public preview

a visual representation of the KQL DB entities and their relations (such as tables, functions, materialized views, and more).

Evenhouse table no code creation and editing experience

Estimated release timeline: Q4 2024

Release Type: Public preview

Wizards to create/edit entities like functions, update policies, and materialized views.

Enhanced Eventstream GA: Content-based routing, derived streams and Edit/Live modes

Estimated release timeline: Q4 2024

Release Type: General availability

Enhanced Eventstream GA with content-based routing, derived streams and Edit/Live modes

Explore data for KQL table in Real-Time hub

Estimated release timeline: Q4 2024

Release Type: General availability

Allow customers to use 'explore data' capability for KQL tables in Real-Time hub.

Eventstream to Activator events

Estimated release timeline: Q1 2025

Release Type: General availability

Activator can monitor and act on events ingested to Fabric through an Eventstream. This will reach General Availability in early 2025.

Support for parameters in triggering Fabric actions

Estimated release timeline: Q1 2025

Release Type: Public preview

Rules that take action by starting Fabric jobs (notebooks or pipelines) will be able to pass the value of properties in the activator model as parameters to the job.

Fabric job action parameters

Estimated release timeline: Q1 2025

Release Type: General availability

When starting a Fabric job as an Activator action, users can pass values as parameters to the job. For example, the values from an event could be passed to the notebook or pipeline’s parameters for further use or transformation.

Activator delay tolerance controls

Estimated release timeline: Q1 2025

Release Type: General availability

Streams of events often arrive out-of-order. When comparing the change between a new event and the previous one, customers ask for control over how long to wait before ignoring late arriving events. This affects the overall latency of the rule, but allows users the manage the tradeoff between speed and accuracy.

Event catalog integration

Estimated release timeline: Q1 2025

Release Type: Public preview

Real-Time hub expands capabilities to support managing event types, providing a catalog for schema management and discovery.

New Fabric events sources: capacity utilization events

Estimated release timeline: Q1 2025

Release Type: Public preview

New Fabric events sources: capacity utilization events

SQL native support in KQL querysets

Estimated release timeline: Q2 2025

This feature enables customers to use a native SQL editor to run SQL over KQL databases in a queryset, alongside using KQL. With this capability, customers are able to use the SQL editor’s native capabilities, such as syntax highlighting, suggestions, and more.

Shipped feature(s)

Real-Time Hub for all data in motion GA

Shipped (Q4 2024)

Real-Time hub is the single estate for all data-in-motion across your entire organization.

Get events experience in Real-Time Hub (change to explain value or remove for 18/24)

Shipped (Q4 2024)

Release Type: General availability

Whether data is coming from new or existing sources, streams, or available events, the Get Events experience allows users to connect to a wide range of sources directly from Real-Time hub, Eventstreams, Eventhouse and Data Activator.

Eventhouse ALM Support

Shipped (Q3 2024)

Eventhouse ALM Support to allow for automatic deployments via Git Integration, Deployment Pipelines, and API with Definition.

Eventstream integration with Managed Private Endpoint

Shipped (Q3 2024)

Release Type: Public preview

This integration with Managed Private Endpoints enables secure and private access to data sources behind a firewall or blocked from public internet access, allowing data to be streamed to Fabric Eventstream over outbound connections.

Eventhouse destination support Eventstreams

Shipped (Q3 2024)

Release Type: Public preview

Eventstreams will add support for Eventhouse as a destination.

New streaming connectors for Kafka public clusters and more instances of SQL Server DB CDC

Shipped (Q3 2024)

Release Type: Public preview

Additional connectors will be available through the Get Events experience, including Public Kafka clusters and SQL Server DB CDC.

Set Alerts on KQL Querysets

Shipped (Q3 2024)

Release Type: Public preview

Data Activator will be able to monitor the results of queries managed in a KQL Queryset, running the query on a regular schedule and acting on the results.

More trigger options

Shipped (Q3 2024)

Release Type: Public preview

To address further requests for monitoring different business conditions, Data Activator will support triggers that check date/time values, basic math functions etc.

Low latency triggers

Shipped (Q3 2024)

We hear in customer feedback that triggers need to react faster when the specified condition is met. Data Activator will support lower-latency triggers in scenarios with simple conditions and lower volumes of data.This feature will be Private Preview in this time frame.

Shipped (Q3 2024)

Users can stream all changes from Cosmos DB to a Fabric Eventhouse KQL database, enabling large scale, powerful analytics via Eventstream Cosmos DB CDC.

  • /fabric/real-time-intelligence/event-streams/add-source-azure-cosmos-db-change-data-capture
  • /fabric/real-time-intelligence/get-data-eventstream

Multivariate Anomaly Detection

Shipped (Q3 2024)

The Azure Anomaly Detector multi variate technology will be natively available in Fabric. Train the detector using spark and notebooks or using Eventhouses, and use it to detect anomalies in real time as new data arrives.

Real time dashboard in Fabric OrgApps

Shipped (Q3 2024)

Org apps support for Real time dashboards

Real-Time Dashboard

Shipped (Q2 2024)

Release Type: Public preview

A dashboard is a collection of tiles, optionally organized in pages, where each tile has an underlying query and a visual representation. Natively export Kusto Query Language (KQL) queries to a dashboard as visuals and later modify their underlying queries and visual formatting as needed. In addition to ease of data exploration, this fully integrated dashboard experience provides improved query and visualization performance with light, flexible modeling on high granularity low latency data.

Content-based event routing to destinations in eventstreams

Shipped (Q2 2024)

Release Type: Public preview

Bringing the no-code event processor to the eventstream main canvas allows customers to route event data to their destinations based on event content, using event processing logic defined with the event processor.

Create actions and alerts with Data Activator

Shipped (Q2 2024)

This feature provides a low-code/no-code experience to drive actions and alerts from your KQL database data presented in a Real-Time Dashboard. Data Activator gives you a single place to define actionable patterns in your data. These patterns can range from simple thresholds (such as a value being exceeded) to more complex patterns over time (such a value trending down). When Data Activator detects an actionable pattern, it triggers an action. That action can be an email or a Teams alert to the relevant person in your organization. It can also trigger an automatic process, via a Power Automate flow or an action in one of your organization’s line-of-business apps.

Pause, stop, or restart eventstream flows

Shipped (Q2 2024)

This feature offers customers the capability to pause, stop, or restart real-time event flows, so they can iterate on transformation or event flow logic during the development phase.

Co-pilot for Real-Time Intelligence

Shipped (Q2 2024)

Release Type: Public preview

Co-pilot allows you to write queries in natural language and have them translated into Kusto Query Language (KQL). You can use Co-pilot to ask your how-to queries, explore your data in a KQL database, and create Kusto entities such as tables, functions, and materialized views.

Eventhouse

Shipped (Q2 2024)

Release Type: General availability

Eventhouse, a cutting-edge database workspace meticulously crafted to manage and store event-based data, is now officially available for general use. With Eventhouse, users can perform high-performance analysis of big data and real-time data querying, processing billions of events within seconds. The platform allows users to organize data into compartments (databases) within one logical item, facilitating efficient data management. Additionally, Eventhouse enables the sharing of compute and cache resources across databases, maximizing resource utilization.

Eventhouse OneLake Availability

Shipped (Q2 2024)

Release Type: General availability

Enabling data availability of Eventhouse in OneLake means that customers can enjoy the best of both worlds: they can query the data with high performance and low latency in their Eventhouse and query the same data in Delta Lake format via any other Fabric engines such as Power BI Direct Lake mode, Warehouse, Lakehouse, Notebooks, and more.

Create a database shortcut to another KQL Database

Shipped (Q2 2024)

Release Type: Public preview

A database shortcut in Eventhouse is an embedded reference to a source database. The database shortcut is attached in read-only mode, making it possible to view and run queries on the data that was ingested into the source KQL Database without ingesting it. This helps with data sharing scenarios where you can share data in-place either within teams, or even with external customers.

Support for AI Anomaly Detector

Shipped (Q2 2024)

Release Type: Public preview

AI Anomaly Detector algorithms are supported in Microsoft Fabric, allowing for real time scoring by KQL with inline Python in Real-Time Intelligence.

Eventhouse tenant level private endpoint support

Shipped (Q2 2024)

Release Type: Public preview

Customers can increase their network security by limiting access to Eventhouse at a tenant-level, from one or more virtual networks (VNets) via private links. This will prevent unauthorized access from public networks and only permit data plane operations from specific VNets.

New experience for data exploration from Real-Time Dashboards

Shipped (Q2 2024)

Release Type: Public preview

Directly from a real-time dashboard, users can refine their exploration using a user-friendly, form-like interface. This intuitive and dynamic experience is tailored for insights explorers craving insights based on real-time data. Add filters, create aggregations, and switch visualization types without writing queries to easily uncover insights.

Use Real-Time hub to Get Data in KQL Database in Eventhouse

Shipped (Q2 2024)

Release Type: Public preview

With Real-Time hub embedded in KQL Database experience, each user in the tenant can view and add streams which they have access and directly ingest it to a KQL Database table in Eventhouse.

Get data from Real-Time Hub within Data Activator

Shipped (Q2 2024)

Release Type: Public preview

Bring new events into reflex items by using the embedded Real-Time Hub browser.

Create triggers from Real-Time Hub

Shipped (Q2 2024)

Release Type: Public preview

Across the Real-Time Hub you'll see options to create new triggers and alerts on both business and system events.

Taking action through Fabric Items

Shipped (Q2 2024)

Release Type: Public preview

Reflex items will support starting Fabric item jobs as an action. You'll be able to kick off data pipelines and notebooks.

Set Alert on Real-Time Dashboards

Shipped (Q2 2024)

Release Type: Public preview

In addition to monitoring Power BI and Eventstream data, Data Activator will support periodic querying against KQL queries saved in Real-Time Dashboards. These queries can be used to return exceptions, or track results over time to detect data changes or threshold conditions.

New streaming connectors including Confluent Kafka, AWS Kinesis, PostgresSQL CDC and more

Shipped (Q2 2024)

Release Type: Public preview

Beyond Microsoft sources, users can now stream data from other platforms like Google Cloud, Amazon Kinesis, Database change data capture streams, etc. using our new messaging connectors. Our messaging connectors are powered by Kafka connect with Camel Kafka connectors that offer a pluggable, declarative data integration framework, and flexibility for diverse source connections on popular data platforms, as well as the Debezium that could connect databases for fetching the Change Data Capture (CDC) streams.

New sources of discrete events from Fabric system events and Azure storage events

Shipped (Q2 2024)

Release Type: Public preview

Discrete events (a.k.a. notification events) are significant (important) facts that have happened, and that trigger state changes or actions in the downstream consumer applications (subscribers). Examples include events like OrderPlaced in order system, blob storage changes in Azure sources, or item changes in Fabric system.

Source from Real-Time Hub in Enhanced Eventstream

Shipped (Q2 2024)

Release Type: Public preview

The Eventstream homepage makes it even easier to bring in data. By clicking on the “Add external source”, you will find these sources in the Get events wizard that helps you to set up the source in a few steps. After you add the source to your eventstream, you can publish it to stream the data into your eventstream. Using Eventstream with discrete sources to turn events into streams for more analysis. You can send the streams to different Fabric data destinations, like Lakehouse and KQL Database.

Eventstream Edit and Live modes

Shipped (Q2 2024)

Release Type: Public preview

Eventstream offers two distinct modes, Edit and Live, to provide flexibility and control over the development process of your eventstream. If you create a new Eventstream with Enhanced Capabilities enabled, you can modify it in an Edit mode. Here, you can design stream processing operations for your data streams using a no-code editor. Once you complete the editing, you can publish your Eventstream and visualize how it starts streaming and processing data in Live mode.

Default and Derived Streams

Shipped (Q2 2024)

Release Type: Public preview

Multiple streams are listed within the Real-Time Hub including Default and Derived streams. Default stream are automatically generated when a streaming source is added to Eventstream. Default stream captures raw event data directly from the source, ready for transformation or analysis. erived stream: A specialized stream that users can create as a destination within Eventstream. Derived stream can be created after a series of operations such as filtering and aggregating, and then it’s ready for further consumption or analysis by other users in the organization through the Real-Time Hub.

Enhanced trigger options

Shipped (Q2 2024)

To represent the business conditions for monitoring, Data Activator will support more trigger conditions such as percentage changes over time, date/time functions, basic mathematic functions, signal absence detection and more.

Notebook integration

Shipped (Q1 2024)

Customers can natively query KQL databases from Microsoft Fabric notebooks. Use notebooks to run queries on KQL databases, save the resulting data frames, ingest data into KQL databases, and more.

Scale up to 100 MB/s for eventstream processing

Shipped (Q1 2024)

An eventstream can support events ingestion throughput up to 100 MB/s while ingesting data to KQL databases and lakehouses.

Customizable event data retention in eventstreams

Shipped (Q1 2024)

Customers have the flexibility to set the event data retention time, ensuring that event data is retained in their eventstreams according to their specific business requirements.

Enhancements to Event processor in Eventstream

Shipped (Q1 2024)

This includes three enhancements. First, personalize operation nodes and easily filter out 'null' values from your data. Second, manage and rename your column fields easily in the Aggregate operation. Third, Change your values to different data types using the Manage Fields operation .

Table optimization shortcut in eventstream lakehouse destinations

Shipped (Q1 2024)

The table optimization shortcut helps users by opening a notebook with a Spark job that compacts small streaming files in the lakehouse table.

Enhanced Custom App Details tab offers comprehensive endpoint information

Shipped (Q1 2024)

The Details tab of the Custom App source and destination offers comprehensive information about the eventstream endpoint. Available information includes connection strings in Kafka, Eventhub, and AMQP formats, as well as sample Java code for event ingestion and consumption using these three protocols.

Improved no-code stream processing designer

Shipped (Q1 2024)

Users have an improved eventstream editing experience, with more intuitive gestures in the no-code designer.

"Get data from Eventstream" in multiple Fabric items

Shipped (Q1 2024)

Customers are able to get event data from eventstreams in multiple Fabric items, including Lakehouse, KQL Database, and Reflex.

New eventstream destination to Data Activator

Shipped (Q1 2024)

The Data Activator destination option helps customers route events to Data Activator, where they can build triggers based on the event data from eventstreams natively.

Autoscale eventstreams

Shipped (Q1 2024)

This feature provides the flexibility to automatically scale capacity in response to event data traffic volume, enabling seamless improvement of eventstream throughput without disrupting your business operations.

Delta support to the KQL database OneLake representation

Shipped (Q1 2024)

Release Type: Public preview

KQL database data is already available in OneLake in parquet format. Adding Delta support enables other Fabric capabilities to use the OneLake presence as a table rather than as a list of files.

Administrative monitoring

Shipped (Q1 2024)

You can use administrative monitoring to search audit logs, and to track usage and adoption, capacity consumption, and availability.

Eventstream data preprocessing to KQL database destination

Shipped (Q1 2024)

The addition of the KQL database destination with data transformation enables customers to transform, enrich, and reduce the data volume before it's routed to KQL databases.

Improved Get Data experience

Shipped (Q1 2024)

Users experience a simplified and improved Get Data experience to get their data into a KQL database.

Cloud connection creation embedded in eventstreams

Shipped (Q1 2024)

Customers can create a cloud connection directly within their eventstreams, without navigating to another webpage.

Two ingestion modes for eventstream lakehouse destinations

Shipped (Q1 2024)

Eventstream lakehouse destinations support two ingestion modes: low latency and high throughput.