Real-time streaming to Web App

Cloud Texavie 71 Reputation points

Is there any recommended way to stream real-time processed data (for e.g in Azure Databricks) to a custom web app deployed with App Service (not PowerBI dashboards) with minimum latency (data streaming at 50Hz)

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,391 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
4,936 questions
0 comments No comments
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 59,331 Reputation points Microsoft Employee

    Hello @Cloud Texavie ,

    Welcome to the Microsoft Q&A platform.

    After intensive research effort, unfortunately I could not a find recommending way to stream real-time data from Azure Databricks to a custom web app deployed with App Service.

    Here are the couple of alternatives:

    If Spark structured streaming (micro-batching), it's perhaps best to write the processed data to a low latency key-value/doc store like Cosmos DB. The webapp can then read from there but I've not seen any other architectures in production.

    If you are using Kafka/Confluent for the ingestion, and don't really require Spark streaming component behind it, the confluent ecosystem does suggest the use of http sink connector:

    One can also use Kafka + Spark streaming where Spark can write back the processed data back to another topic in Kafka and then use the http sink.

    That all said, all these options really depend on many different factors - SLAs, use-cases, cost considerations, performance/scalability, ease of operations etc.

    Structured streaming with Azure Databricks into Power BI & Cosmos DB

    Stream processing with Azure Databricks

    Hope this helps. Do let us know if you any further queries.


    Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

0 additional answers

Sort by: Most helpful