Is there any recommended way to stream real-time processed data (for e.g in Azure Databricks) to a custom web app deployed with App Service (not PowerBI dashboards) with minimum latency (data streaming at 50Hz)

Hello @Cloud Texavie , Welcome to the Microsoft Q&A platform. After intensive research effort, unfortunately I could not a find recommending way to stream real-time data from Azure Databricks to a custom web app deployed with App Service. Here are the couple of alternatives: If Spark structured streaming (micro-batching), it's perhaps best to write the processed data to a low latency key-value/doc store like Cosmos DB. The webapp can then read from there but I've not seen any other architectures in production. If you are using Kafka/Confluent for the ingestion, and don't really require Spark streaming component behind it, the confluent ecosystem does suggest the use of http sink connector: https://www.confluent.io/blog/webify-event-streams-using-kafka-connect-http-sink/ One can also use Kafka + Spark streaming where Spark can write back the processed data back to another topic in Kafka and then use the http sink. That all said, all these options really depend on many different factors - SLAs, use-cases, cost considerations, performance/scalability, ease of operations etc. Structured streaming with Azure Databricks into Power BI & Cosmos DB Stream processing with Azure Databricks Hope this helps. Do let us know if you any further queries. ------------ Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

Real-time streaming to Web App

Accepted answer

PRADEEPCHEEKATLA-MSFT 76,921 Reputation points Microsoft Employee

2021-03-10T10:19:10.743+00:00

Hello @Cloud Texavie ,

Welcome to the Microsoft Q&A platform.

After intensive research effort, unfortunately I could not a find recommending way to stream real-time data from Azure Databricks to a custom web app deployed with App Service.

Here are the couple of alternatives:

If Spark structured streaming (micro-batching), it's perhaps best to write the processed data to a low latency key-value/doc store like Cosmos DB. The webapp can then read from there but I've not seen any other architectures in production.

If you are using Kafka/Confluent for the ingestion, and don't really require Spark streaming component behind it, the confluent ecosystem does suggest the use of http sink connector: https://www.confluent.io/blog/webify-event-streams-using-kafka-connect-http-sink/

One can also use Kafka + Spark streaming where Spark can write back the processed data back to another topic in Kafka and then use the http sink.

That all said, all these options really depend on many different factors - SLAs, use-cases, cost considerations, performance/scalability, ease of operations etc.

Structured streaming with Azure Databricks into Power BI & Cosmos DB

Stream processing with Azure Databricks

Hope this helps. Do let us know if you any further queries.

------------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
Cloud Texavie 71 Reputation points

2021-03-10T22:42:39.06+00:00

Thank you very much for this answer! There are a few points I would like to clarify:

The webapp can then read from there but I've not seen any other architectures in production.

Do you mean that it is not very common to use low-latency key-value stores for real-time streaming in production ?

If you are using Kafka/Confluent for the ingestion, and don't really require Spark streaming component behind it, the confluent ecosystem does suggest the use of http sink connector

Would there be any major downside to processing within Kafka Streams instead of Spark

One can also use Kafka + Spark streaming where Spark can write back the processed data back to another topic in Kafka and then use the http sink.

Our current architecture uses IOT Hub instead of Kafka for message ingestion, would IOT Hub--> Spark --> Kafka be a viable architecture (since IOT Hub can not be used as a sink for Databricks) ?

PRADEEPCHEEKATLA-MSFT 76,921 Reputation points Microsoft Employee

2021-03-11T03:50:32.96+00:00

Hello @Cloud Texavie ,

Thanks for the follow-up questions.

We are reaching out to the internal team to get more help on this, I will update you once we hear back from them.

Cloud Texavie 71 Reputation points

2021-03-11T04:09:16.64+00:00

Great, thank you !

PRADEEPCHEEKATLA-MSFT 76,921 Reputation points Microsoft Employee

2021-03-11T12:28:36.653+00:00

Hello @Cloud Texavie ,

Spark streaming is actually mini-batched processing so if seconds level latency is required then you would go with Azure Stream Analytics or Kafka Stream (in Confluent) for streaming processing (e.g window operation). Streaming then writes result to a Cosmos DB or SQL DB from which your webapp can query continuously. The HTTP sink is a nice option but if you need to show a rich dashboard with not only current info then a serving DB is needed.

Cloud Texavie 71 Reputation points

2021-03-11T20:24:08.147+00:00

Thank you so much for the precisions. This helps a lot!
Would continously querying still be appropriate for milliseconds latency ? Also, if my understanding is correct, Stream Analytics only supports the SQL querying language so we will not be able to use Cosmos DB for MongoDB, is that right ? In that case, would Storm on HDInsight be a good option to consider ?

PRADEEPCHEEKATLA-MSFT 76,921 Reputation points Microsoft Employee

2021-03-12T07:40:53.533+00:00

Hello @Cloud Texavie ,

At this time, Stream Analytics supports connection to Azure Cosmos DB only through the SQL API. Other Azure Cosmos DB APIs are not yet supported.
You can use Storm/Kafka on HDInsight would be a good option to consider.

Cloud Texavie 71 Reputation points

2021-03-12T08:04:37.2+00:00

Thank you for all the insights you provided !

Cloud Texavie 71 Reputation points

2021-03-12T23:34:21.36+00:00

Hi again,

Just one quick follow up question I would like to ask is whether one instance of a CosmosDB database can serve both for real-time streaming and persistent storage, or would we need two databases, one for streaming and one for serving historical data to the web app.

PRADEEPCHEEKATLA-MSFT 76,921 Reputation points Microsoft Employee

2021-03-15T09:21:26.55+00:00

Hello @Cloud Texavie ,

Yes, you can use one instance of a Cosmos DB database can serve both for real-time streaming and persistent storage.
Sign in to comment

Real-time streaming to Web App

0 additional answers