What's new and planned for Synapse Real-Time Analytics in Microsoft Fabric
Important
The release plans describe functionality that may or may not have been released yet. The delivery timelines and projected functionality may change or may not ship. Refer to Microsoft policy for more information.
Synapse Real-Time Analytics is a fully managed big data analytics platform optimized for streaming and time-series data. The services in Real-Time Analytics provide the capability to capture, transform, and route real-time events to and from different environments. You can even enable distinct endpoints where your custom application can directly produce or consume events from the Fabric Eventstream item. The dedicated Kusto Query Language (KQL) database can load data from any source in any data format, and run analytical queries directly on raw data without the need to build complex data models or create scripting to transform the data. Use either SQL or the native KQL to process massive amounts of data in real time with high performance and low latency.
Synapse Real-Time Analytics is useful for organizations that need high freshness from data ingestion for querying, require low query latency, or need to search or access data in different formats, including structured, semi-structured (such as JSON), or unstructured data (such as free text). You can scale your data needs with the service, to an unlimited amount of data (from gigabytes to petabytes) with unlimited scalability on concurrent queries and concurrent users. Real-Time Analytics is fully integrated with the entire suite of Fabric products to streamline your data analytics processes for data loading, data transformation, and advanced visualization scenarios.
For more information, see the Real-Time Analytics documentation and visit the announcement blog.
The items available in Real-Time Analytics are:
Eventstreams for capturing, transforming, and routing real-time events to various destinations with a no-code experience.
A KQL database for data storage, analytics, and management. Data loaded into a KQL database can be accessed in OneLake and is exposed to other Fabric experiences.
A KQL queryset to run queries, view, and customize query results on data. The KQL queryset allows you to save queries for future use, export, and share queries with others. It includes the option to generate a Power BI report.
A real-time dashboard, which contains a collection of tiles. Each tile has an underlying query and a visual representation perfect for data exploration, monitoring, and forensics.
Recent innovations include:
Mounting of Azure Data Explorer Database (follower) to Fabric; creating a database shortcut is now available (Announcement blog)
Sample gallery for Real-Time Analytics and KQL (Announcement blog)
Model and query data as graphs using Kusto Query Language (KQL) (Announcement blog)
Eventstreams support AMQP format connection string for data ingestion (Announcement blog)
Eventstreams support data ingestion from Azure IoT Hub (Announcement blog)
Investment areas
Real-Time Dashboard
Estimated release timeline: Q4 2023
A dashboard is a collection of tiles, optionally organized in pages, where each tile has an underlying query and a visual representation. Natively export Kusto Query Language (KQL) queries to a dashboard as visuals and later modify their underlying queries and visual formatting as needed. In addition to ease of data exploration, this fully integrated dashboard experience provides improved query and visualization performance with light, flexible modeling on high granularity low latency data.
SQL native support in KQL querysets
Estimated release timeline: Q2 2024
This feature enables customers to use a native SQL editor to run SQL over KQL databases in a queryset, alongside using KQL. With this capability, customers are able to use the SQL editor’s native capabilities, such as syntax highlighting, suggestions, and more.
Eventstream data preprocessing to KQL database destination
Estimated release timeline: Q4 2023
The addition of the KQL database destination with data transformation enables customers to transform, enrich, and reduce the data volume before it's routed to KQL databases.
Improved Get Data experience
Estimated release timeline: Q4 2023
Users experience a simplified and improved Get Data experience to get their data into a KQL database.
Co-pilot (Preview)
Estimated release timeline: Q4 2023
KQL Co-pilot allows you to write queries in natural language and have them translated into Kusto Query Language (KQL). You can use Co-pilot to ask your how-to queries, explore your data in a KQL database, and create Kusto entities such as tables, functions, and materialized views.
Monitoring Hub integration
Estimated release timeline: Q2 2024
Users can monitor Real-Time Analytics items in the monitoring hub with all other Fabric items.
New eventstream destination to Data Activator
Estimated release timeline: Q4 2023
The Data Activator destination option helps customers route events to Data Activator, where they can build triggers based on the event data from eventstreams natively.
Create actions and alerts with Data Activator
Estimated release timeline: Q4 2023
This feature provides a low-code/no-code experience to drive actions and alerts from your KQL database data. Data Activator gives you a single place to define actionable patterns in your data. These patterns can range from simple thresholds (such as a value being exceeded) to more complex patterns over time (such a value trending down). When Data Activator detects an actionable pattern, it triggers an action. That action can be an email or a Teams alert to the relevant person in your organization. It can also trigger an automatic process, via a Power Automate flow or an action in one of your organization’s line-of-business apps.
Pause, stop, or restart eventstream flows
Estimated release timeline: Q4 2023
This feature offers customers the capability to pause, stop, or restart real-time event flows, so they can iterate on transformation or event flow logic during the development phase.
Autoscale eventstreams
Estimated release timeline: Q3 2023
This feature provides the flexibility to automatically scale capacity in response to event data traffic volume, enabling seamless improvement of eventstream throughput without disrupting your business operations.
Notebook integration
Estimated release timeline: Q4 2023
Customers can natively query KQL databases from Microsoft Fabric notebooks. Use notebooks to run queries on KQL databases, save the resulting data frames, ingest data into KQL databases, and more.
Delta support to the KQL database OneLake representation
Estimated release timeline: Q4 2023
KQL database data is already available in OneLake in parquet format. Adding Delta support enables other Fabric capabilities to use the OneLake presence as a table rather than as a list of files.
Administrative monitoring
Estimated release timeline: Q4 2023
You can use administrative monitoring to search audit logs, and to track usage and adoption, capacity consumption, and availability.
Azure Synapse Link to KQL databases
Estimated release timeline: Q4 2023
Customers can stream all updates from Cosmos DB to a KQL database, enabling large scale, powerful analytics.
Data streaming from Kafka to eventstreams
Estimated release timeline: Q2 2024
Customers can bring event data from Kafka clusters into eventstreams with a simple no-code ingestion and stream processing experience, and route the data to other Fabric entities.
Data streaming from Azure Event Grid to eventstreams
Estimated release timeline: Q1 2024
Customers can subscribe to Azure resource events (like Blob storage events), partner events (like SAP), custom events, or MQTT topics, and then have them streamed to eventstreams.
Improved no-code stream processing designer
Estimated release timeline: Q3 2023
Users have an improved eventstream editing experience, with more intuitive gestures in the no-code designer.
"Get data from Eventstream" in multiple Fabric items
Estimated release timeline: Q4 2023
Customers are able to get event data from eventstreams in multiple Fabric items, including Lakehouse, KQL Database, and Reflex.
Cloud connection creation embedded in eventstreams
Estimated release timeline: Q4 2023
Customers can create a cloud connection directly within their eventstreams, without navigating to another webpage.
Enhanced Custom App Details tab offers comprehensive endpoint information
Estimated release timeline: Q4 2023
The Details tab of the Custom App source and destination offers comprehensive information about the eventstream endpoint. Available information includes connection strings in Kafka, Eventhub, and AMQP formats, as well as sample Java code for event ingestion and consumption using these three protocols.
Scale up to 100 MB/s for eventstream processing
Estimated release timeline: Q4 2023
An eventstream can support events ingestion throughput up to 100 MB/s while ingesting data to KQL databases and lakehouses.
Two ingestion modes for eventstream lakehouse destinations
Estimated release timeline: Q4 2023
Eventstream lakehouse destinations support two ingestion modes: low latency and high throughput.
Table optimization shortcut in eventstream lakehouse destinations
Estimated release timeline: Q4 2023
The table optimization shortcut helps users by opening a notebook with a Spark job that compacts small streaming files in the lakehouse table.
Enrich event data by joining reference data in eventstreams
Estimated release timeline: Q1 2024
During event data ingestion into destinations, an eventstream can enrich the event data by joining it with SQL reference data, enabling customers to extract more valuable insights from their event data.
Content-based event routing to destinations in eventstreams
Estimated release timeline: Q1 2024
Bringing the no-code event processor to the eventstream main canvas allows customers to route event data to their destinations based on event content, using event processing logic defined with the event processor.
Customizable event data retention in eventstreams
Estimated release timeline: Q1 2024
Customers have the flexibility to set the event data retention time, ensuring that event data is retained in their eventstreams according to their specific business requirements.
CI/CD support for Eventstream items
Estimated release timeline: Q1 2024
Customers can easily version and deploy their eventstreams across development, testing, and production workspaces using the integrated Git integration and deployment experience in the Fabric platform.
Monitor eventstream activities in Fabric monitoring hub
Estimated release timeline: Q1 2024
Integrating Microsoft Fabric eventstreams with Fabric monitoring hub allows customers to easily monitor their eventstream activities with other Fabric items in a centralized place.