Design Real Time Streaming with Azure

Question

Design Real Time Streaming with Azure

Relay 100

I am in process of designing a real time (15 sec latency) data streaming system.

Please share your thoughts on Pros and Cons for below scenerio.

I have to replace SAP Connector with Azure Tech stack with lowest latency for process data, as shown in pic below .

which Component in Azure Stack best to replace SAP Connector.

SAP-Connector

2nd scenerio:

can we directly connect Stream Analytics with Azure SQL DB, or will it be fine connecting with service bus queue and function app. as shown in diagram.

There will be multiple data source to be connect to system.

Azure- SQL

My Doubt is Is this solution is scalable and cost effective or shall I go with different approach.

Which all area I should focus before do the system design.

Please share your expertise suggestion.

Thanks

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2025-05-14T05:31:52.0666667+00:00
@Relay You're welcome! Let's address your concerns about connecting Azure Stream Analytics directly to Azure SQL Database.

Handling Throttle or Timeouts Issues

To manage throttling and timeouts when connecting Azure Stream Analytics to Azure SQL Database, consider the following strategies:

Retry Logic: Implement retry policies to handle transient errors. Azure Stream Analytics supports retry logic for output failures, which can help mitigate issues caused by temporary throttling or timeouts 1.

Parallel Writes: Enable parallel writes to increase throughput performance. This can help reduce the likelihood of timeouts by distributing the load across multiple connections 2.

Resource Monitoring: Regularly monitor the resource utilization of your Azure SQL Database. Ensure you are not hitting DTU (Database Transaction Unit) limits, which can cause throttling 3.

Handling Partial Failures and Logging

To effectively manage partial failures and logging at each phase, you can use the following approaches:

Diagnostic Logs: Enable diagnostic logs for your Azure Stream Analytics job. These logs provide detailed information about data errors, including serialization and deserialization issues 4. You can use these logs to identify and troubleshoot partial failures.

Error Handling: Implement robust error handling in your Stream Analytics queries. Use the TRY_CAST and TRY_CONVERT functions to handle data conversion errors gracefully.

Monitoring Tools: Utilize Azure Monitor and Application Insights to track the performance and health of your Stream Analytics job. Set up alerts for critical issues to ensure timely interventio
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2025-05-15T04:35:08.09+00:00

@Relay We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

2 answers

Your answer

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2025-05-14T05:31:52.0666667+00:00

@Relay You're welcome! Let's address your concerns about connecting Azure Stream Analytics directly to Azure SQL Database.

Handling Throttle or Timeouts Issues

To manage throttling and timeouts when connecting Azure Stream Analytics to Azure SQL Database, consider the following strategies:

Retry Logic: Implement retry policies to handle transient errors. Azure Stream Analytics supports retry logic for output failures, which can help mitigate issues caused by temporary throttling or timeouts 1.

Parallel Writes: Enable parallel writes to increase throughput performance. This can help reduce the likelihood of timeouts by distributing the load across multiple connections 2.

Resource Monitoring: Regularly monitor the resource utilization of your Azure SQL Database. Ensure you are not hitting DTU (Database Transaction Unit) limits, which can cause throttling 3.

Handling Partial Failures and Logging

To effectively manage partial failures and logging at each phase, you can use the following approaches:

Diagnostic Logs: Enable diagnostic logs for your Azure Stream Analytics job. These logs provide detailed information about data errors, including serialization and deserialization issues 4. You can use these logs to identify and troubleshoot partial failures.

Error Handling: Implement robust error handling in your Stream Analytics queries. Use the TRY_CAST and TRY_CONVERT functions to handle data conversion errors gracefully.

Monitoring Tools: Utilize Azure Monitor and Application Insights to track the performance and health of your Stream Analytics job. Set up alerts for critical issues to ensure timely interventio
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2025-05-15T04:35:08.09+00:00

@Relay We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

@Relay

It sounds like you're diving into an exciting project with real-time data streaming in Azure. Let's break down your queries:

Replacing SAP Connector: For low-latency data processing in Azure, you might want to consider using Azure Stream Analytics. It's designed for real-time analytics and processing, providing low-latency capabilities. It can seamlessly integrate with various Azure services, making it a solid replacement for your SAP Connector.

Pros of Azure Stream Analytics:

Low latency (under 100 ms for high throughput).
Simplified management (no need to manage clusters).
Built-in support for temporal processing and a SQL-like query language.

Cons:

Limited scalability compared to a full Spark cluster if you need to process a massive amount of data.
Some learning curve if your team is unfamiliar with the service.

Connecting Stream Analytics with Azure SQL DB: You can definitely connect Azure Stream Analytics directly to Azure SQL Database for real-time analytics without adding an intermediate step like Service Bus queues or Function Apps. However, if your architecture demands more control or you need to decouple components, using a Service Bus can also be a good option.

Scalability and Cost-Effectiveness: Given you're considering multiple data sources:

Using Azure Stream Analytics is cost-effective due to per-job billing and no clusters to manage. Ensure you assess your expected data volumes to choose an appropriate number of Streaming Units.
If you anticipate significant growth or variable loads, consider Azure Databricks for flexibility.

Areas to Focus On for System Design:

Data Sources: Identify all data sources and their formats. Ensure you consider how they'll integrate into your streaming solution.
Latency Requirements: Make sure your chosen components will consistently meet your latency goals, especially under peak loads.
Error Handling: Plan for message processing failures and retries.
Monitoring and Analytics: Implementing monitoring tools to assess performance and reliability will be crucial for maintenance.
Cost Management: Regularly monitor costs associated with Azure services to adjust as necessary.

Answer 2

Relay 100

@phemanth : Thanks a lot.

But few doubts when directly connect :Azure Stream Analytics to SQL

How to handle throttle or timeouts issue.
How to handle partial failures, logging at each phase.

Appreciate your suggestion very much.

Thanks a lot

Share via

Design Real Time Streaming with Azure

2 answers

Your answer