what is the best IoT architecture for AI in Azure ?

Question

I'm here to ask for advice on an architecture I've set up for real-time data processing. The goal is to have a Dashboard able to display all the metrics coming from the connected objects and to be able to make predictions in real time (cost prediction, maintenance prediction etc..) using AI.

I set up the architecture below with a script simulating a machine. But I have a lot of doubts about this one.

Especially for Stream Analytics. I would like to be able to clean the raw data, before storing them (aggregation in particular) to calculate an OEE and other things.

But SA is limited in terms of data aggregation, so I would like to know what would be the best alternative for real-time data processing before storing it.

Should I use Azure Function combined with SA?
Can Azure Machine Learning studio do this processing work with or without SA?
Azure Databricks seemed to be a good alternative but I'm having a lot of trouble setting up a real-time data processing stream and CSV/Json storage in my data lake but I don't know if I'm doing it wrong or if it's just impossible.

Azure Data Factory didn't seem to be adapted to real time data processing so I abandoned this option

There are so many options and I have no idea in which direction to go, I've been working on it for several months.

Thanks to all the people who will take the time to answer.

Answer 1

Take a look at examples in Azure Architecture Center and Real-time processing and Stream processing with Azure Stream Analytics
, Stream Analytics does seem to be solution.