What is the Hardware config of a Streaming Unit in Stream Analytics

SRIRAMANENI CHAITANYA 1

We are doing a study on Spark Streaming (using DataBricks Infra) and Stream Analytics.

WRT pricing,

1 Databricks worker node price >> 1 Stream Analytics SU

But we do not know if these two units are directly comparable or not with respect to their throughput and hardware.

The official documentation states to check SU% and take a decision on increasing or decreasing the #SUs in a job

If anyone can provide details on Streaming Unit Hardware Specifications it'll be helpful

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2022-09-29T21:01:53.87+00:00

Hello @SRIRAMANENI CHAITANYA ,

Thanks for the question and using MS Q&A platform.

We are reaching out to internal team to get more information related to the comparison of hardware and their pricing. Will keep you posted as soon as we hear back from the team.

Thank you
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2022-10-19T06:36:33.117+00:00

Hello @SRIRAMANENI CHAITANYA ,

After further discussion with internal teams. Below are inputs with regards to your query.

Databricks has several costs, DBU which as far as I know is not really something you can clearly measure and define and it differs based on the workload type as well as features, premium or standard etc. To this adds up some costs for VMs and storage as well.

The advice from experts would be to focus not on costs, but on needs and features. (Eg: Whether you already have an architecture in place for the other components, i.e. do you already have a setup for other components like an Event Hub or IOT or SQL Data Warehouse etc? What type of processing is you're expecting to do on the streaming data and where will you write it afterwards or what will you do with it afterwards?

The lower cost solution might or might not give you the capabilities you need and best way to estimate would be to actually run some tests/POC with a test workload to assess both costs and performance for the type of workload.

I’d rather start the with this https://learn.microsoft.com/en-us/azure/architecture/data-guide/technology-choices/stream-processing#capability-matrix than with costs.

Hope this info helps.
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2022-10-19T06:38:26.49+00:00

I have also checked with other internal team about your ask related to hardware but no luck, seems like streaming units correspond to hardware isn't publicly shared.