Any suggestion for Azure Data Architecture?

Question

Any suggestion for Azure Data Architecture?

Pankaj Rathod 1

Below is the customer requirements:
• Platform should be Azure.
• Volume of the data coming in would be 5 GB per day.
• Data should be available near to real time. Currently it is made available in one hour.
• Azure Blob Storage Gen2(Delta) is already available.
• Already leveraging Power BI. Expected 1000 plus Power BI users.
• Data should have be single version of truth.
• We should also cater to Machine Learning capabilities.
• The data should be exposed to down stream through API as well.

Sander van de Velde | MVP 36,761 Reputation points MVP Volunteer Moderator

2022-11-09T12:20:22.723+00:00

Hello @Pankaj Rathod ,

can you tell us more about the incoming data (Which source, batch/stream, format, interval, size of individual messages)?
Pankaj Rathod 1 Reputation point

2022-11-09T12:59:18.91+00:00

source is ADLS, Azure SQL Database.
Sander van de Velde | MVP 36,761 Reputation points MVP Volunteer Moderator

2022-11-10T14:38:05.783+00:00

Hello @Pankaj Rathod ,

just to be sure, each day, this 5GB of data is a combination of files being added to the Azure Data Lake (Azure Storage Gen2) and records being added to a Sql Database?
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2022-11-14T09:29:24.06+00:00
Hello @Pankaj Rathod ,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

---------

Please don't forget to click on or upvote button whenever the information provided helps you.

1 answer

Your answer

Sander van de Velde | MVP 36,761 Reputation points MVP Volunteer Moderator

2022-11-09T12:20:22.723+00:00

Hello @Pankaj Rathod ,

can you tell us more about the incoming data (Which source, batch/stream, format, interval, size of individual messages)?
Pankaj Rathod 1 Reputation point

2022-11-09T12:59:18.91+00:00

source is ADLS, Azure SQL Database.
Sander van de Velde | MVP 36,761 Reputation points MVP Volunteer Moderator

2022-11-10T14:38:05.783+00:00

Hello @Pankaj Rathod ,

just to be sure, each day, this 5GB of data is a combination of files being added to the Azure Data Lake (Azure Storage Gen2) and records being added to a Sql Database?
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2022-11-14T09:29:24.06+00:00

Hello @Pankaj Rathod ,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

---------

Please don't forget to click on or upvote button whenever the information provided helps you.

Answer 1

Hello @Pankaj Rathod ,

Thanks for the question and using MS Q&A platform.

Analytics end-to-end with Azure Synapse - combines a range of Azure services that will ingest, store, process, enrich, and serve data and insights from different sources (structured, semi-structured, unstructured, and streaming).

With respective to Microsoft Azure Data warehousing and analytics architecture:

This example scenario demonstrates a data pipeline that integrates large amounts of data from multiple sources into a unified analytics platform in Azure. This specific scenario is based on a sales and marketing solution, but the design patterns are relevant for many industries requiring advanced analytics of large datasets such as e-commerce, retail, and healthcare.

For more details, refer to Data warehousing in Microsoft Azure, Data warehousing and analytics and Analytics end-to-end with Azure Synapse.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.

Share via

Any suggestion for Azure Data Architecture?

1 answer

Your answer