Microsoft Fabric event streams - overview
The event streams feature in the Microsoft Fabric Real-Time Intelligence experience lets you bring real-time events into Fabric, transform them, and then route them to various destinations without writing any code (no-code). You create an eventstream, which is an instance of the Eventstream item in Fabric, add event data sources to the stream, optionally add transformations to transform the event data, and then route the data to supported destinations.
Bring events into Fabric
The event streams feature provides you with various source connectors to fetch event data from the various sources. There are more sources available when you enable Enhanced capabilities at the time of creating an eventstream.
Important
Enhanced capabilities of Fabric event streams are currently in preview.
| Sources | Description |
|---|---|
| Azure Event Hubs | If you have an Azure event hub, you can ingest event hub data into Microsoft Fabric using Eventstream. |
| Azure IoT Hub | If you have an Azure IoT hub, you can ingest IoT data into Microsoft Fabric using Eventstream. |
| Azure SQL Database Change Data Capture (CDC) | The Azure SQL Database CDC source connector allows you to capture a snapshot of the current data in an Azure SQL database. The connector then monitors and records any future row-level changes to this data. |
| PostgreSQL Database CDC | The PostgreSQL Database Change Data Capture (CDC) source connector allows you to capture a snapshot of the current data in a PostgreSQL database. The connector then monitors and records any future row-level changes to this data. |
| MySQL Database CDC | The Azure MySQL Database Change Data Capture (CDC) Source connector allows you to capture a snapshot of the current data in an Azure Database for MySQL database. You can specify the tables to monitor, and the eventstream records any future row-level changes to the tables. |
| Azure Cosmos DB CDC | The Azure Cosmos DB Change Data Capture (CDC) source connector for Microsoft Fabric event streams lets you capture a snapshot of the current data in an Azure Cosmos DB database. The connector then monitors and records any future row-level changes to this data. |
| Google Cloud Pub/Sub | Google Pub/Sub is a messaging service that enables you to publish and subscribe to streams of events. You can add Google Pub/Sub as a source to your eventstream to capture, transform, and route real-time events to various destinations in Fabric. |
| Amazon Kinesis Data Streams | Amazon Kinesis Data Streams is a massively scalable, highly durable data ingestion, and processing service optimized for streaming data. By integrating Amazon Kinesis Data Streams as a source within your eventstream, you can seamlessly process real-time data streams before routing them to multiple destinations within Fabric. |
| Confluent Cloud Kafka | Confluent Cloud Kafka is a streaming platform offering powerful data streaming and processing functionalities using Apache Kafka. By integrating Confluent Cloud Kafka as a source within your eventstream, you can seamlessly process real-time data streams before routing them to multiple destinations within Fabric. |
| Azure Blob Storage events | Azure Blob Storage events are triggered when a client creates, replaces, or deletes a blob. The connector allows you to link Blob Storage events to Fabric events in Real-Time hub. You can convert these events into continuous data streams and transform them before routing them to various destinations in Fabric. |
| Fabric workspace item events | Fabric workspace item events are discrete Fabric events that occur when changes are made to your Fabric Workspace. These changes include creating, updating, or deleting a Fabric item. With Fabric event streams, you can capture these Fabric workspace events, transform them, and route them to various destinations in Fabric for further analysis. |
| Sample data | You can choose Bicycles, Yellow Taxi, or Stock Market events as a sample data source to test the data ingestion while setting up an eventstream. |
| Custom endpoint | The custom app feature allows your applications or Kafka clients to connect to Eventstream using a connection string, enabling the smooth ingestion of streaming data into Eventstream. |
Process events using no-code experience
The drag and drop experience gives you an intuitive and easy way to create your event data processing, transforming, and routing logic without writing any code. An end-to-end data flow diagram in an eventstream can provide you with a comprehensive understanding of the data flow and organization. The event processor editor is a no-code experience that allows you to drag and drop to design the event data processing logic.
| Transformation | Description |
|---|---|
| Filter | Use the Filter transformation to filter events based on the value of a field in the input. Depending on the data type (number or text), the transformation keeps the values that match the selected condition, such as is null or is not null. |
| Manage fields | The Manage fields transformation allows you to add, remove, change data type, or rename fields coming in from an input or another transformation. |
| Aggregate | Use the Aggregate transformation to calculate an aggregation (Sum, Minimum, Maximum, or Average) every time a new event occurs over a period of time. This operation also allows for the renaming of these calculated columns, and filtering or slicing the aggregation based on other dimensions in your data. You can have one or more aggregations in the same transformation. |
| Group by | Use the Group by transformation to calculate aggregations across all events within a certain time window. You can group by the values in one or more fields. It's like the Aggregate transformation allows for the renaming of columns, but provides more options for aggregation and includes more complex options for time windows. Like Aggregate, you can add more than one aggregation per transformation. |
| Union | Use the Union transformation to connect two or more nodes and add events with shared fields (with the same name and data type) into one table. Fields that don't match are dropped and not included in the output. |
| Expand | Use the Expand array transformation to create a new row for each value within an array. |
| Join | Use the Join transformation to combine data from two streams based on a matching condition between them. |
If you enabled Enhanced capabilities while creating an eventstream, the transformation operations are supported for all destinations. If you didn't, the transformation operations are available only for the Lakehouse and KQL Database (event processing before ingestion) destinations.
Route events to destinations
The Fabric event streams feature supports sending data to the following supported destinations.
| Destination | Description |
|---|---|
| Custom app | With this destination, you can easily route your real-time events to a custom endpoint. You can connect your own applications to the eventstream and consume the event data in real time. This destination is useful when you want to egress real-time data to an external system outside Microsoft Fabric. |
| KQL Database | This destination lets you ingest your real-time event data into a KQL database, where you can use the powerful Kusto Query Language (KQL) to query and analyze the data. With the data in the Kusto database, you can gain deeper insights into your event data and create rich reports and dashboards. You can choose between two ingestion modes: Direct ingestion and Event processing before ingestion. |
| Lakehouse | This destination gives you the ability to transform your real-time events before ingesting them into your lakehouse. Real-time events convert into Delta Lake format and then store in the designated lakehouse tables. This destination supports data warehousing scenarios. |
| Reflex | This destination lets you directly connect your real-time event data to a Reflex. Reflex is a type of intelligent agent that contains all the information necessary to connect to data, monitor for conditions, and act. When the data reaches certain thresholds or matches other patterns, Reflex automatically takes appropriate action such as alerting users or kicking off Power Automate workflows. |
| Derived stream | Derived stream is a specialized type of destination that you can create after adding stream operations, such as Filter or Manage Fields, to an eventstream. The derived stream represents the transformed default stream following stream processing. You can route the derived stream to multiple destinations in Fabric, and view the derived stream in the Real-Time hub. |
You can attach multiple destinations in an eventstream to simultaneously receive data from your eventstreams without interfering with each other.
Note
We recommend that you use the Microsoft Fabric event streams feature with at least 4 capacity units (SKU: F4)
Enhanced capabilities (preview)
There are more features, sources, and destinations available when you enable the Enhanced capabilities (preview) option while creating an eventstream. Use the Enhanced capabilities (preview) and Standard capabilities tabs to learn about additional sources and destinations supported with the enhanced capabilities.
Here are a few other noteworthy features of Enhanced capabilities (preview):
- Edit Mode and Live View. Explore two distinct modes for visualizing and designing stream processing.
- Default and derived streams. Create a continuous flow of streams with the format you design, with an event processor that can be consumed later in Real-Time hub.
- Data stream routing based on content. Transform and route your data streams anywhere within Fabric based on the data stream content you designed with the event processor.
Related content
Commentaires
Bientôt disponible : Tout au long de 2024, nous allons supprimer progressivement GitHub Issues comme mécanisme de commentaires pour le contenu et le remplacer par un nouveau système de commentaires. Pour plus d’informations, consultez https://aka.ms/ContentUserFeedback.
Envoyer et afficher des commentaires pour