Analytics architecture design

With the exponential growth in data, organizations rely on the limitless compute, storage, and analytical power of Azure to scale, stream, predict, and see their data. Analytics solutions turn volumes of data into useful business intelligence (BI), such as reports and visualizations, and inventive artificial intelligence (AI), such as forecasts based on machine learning.

Whether your organization is just starting to evaluate cloud-based analytics tools or is looking to expand your current implementation, Azure provides many options. The workflow starts with learning about common approaches and aligning processes and roles around a cloud mindset.

Data can be processed in batches or in real-time, on-premises or in the cloud, but the goal of any analytics solution is to make use of data at scale. Increasingly, organizations want to create a single source of truth for all the relational and nonrelational data being generated by people, machines, and the Internet of Things (IoT). It's common to use a big data architecture or an IoT architecture to transform raw data into a structured form, then move it to an analytical data store. This store becomes the single source of truth that can power a multitude of insightful analytics solutions.

Architecture

Diagram showing the solution journey for analytics on Azure. The journey starts with learning and organizational readiness, then moves to choosing storage solutions and Azure BI or AI technologies based on workload requirements.

Download a Visio file of this architecture.

The diagram above demonstrates a typical basic/baseline analytics implementation. Refer to the architectures provided in this section to find real-world solutions that you can build in Azure.

Explore analytics architectures and guides

The articles in this section include fully developed architectures that you can deploy in Azure and expand to production-grade solutions and guides. These can help you make important decisions about how you use analytics technologies in Azure. You can also review solution ideas, which give you a taste of what is possible as you plan your analytics POC development.

Architectures

• Analytics end-to-end with Microsoft Fabric - Build a modern analytics platform using Microsoft Fabric's integrated capabilities.
• Data warehousing and analytics - Integrate large amounts of data from multiple sources into a unified analytics platform.
• Use Microsoft Fabric to design an enterprise BI solution - Design an enterprise business intelligence solution using Microsoft Fabric.
• Near real-time lakehouse data processing - Use Azure Synapse Analytics and Azure Data Lake Storage for near real-time data lakehouse processing.
• Real-time sync of MongoDB Atlas to Azure Synapse Analytics - Synchronize MongoDB Atlas data to Azure Synapse Analytics in real time.
• Stream processing with Azure Databricks - Create an end-to-end stream processing pipeline using Azure Databricks.
• Stream processing with Azure Stream Analytics - Build a stream processing pipeline that ingests data, correlates records, and calculates rolling averages.
• Modern data warehouse for small and medium business - Build a modern data warehouse solution designed for small and medium businesses.

Solution ideas

• Ingestion, ETL, and stream processing pipelines with Azure Databricks - Create ETL pipelines for batch and streaming data to simplify data lake ingestion.
• Modern analytics architecture with Azure Databricks - Collect, process, analyze, and visualize data using a modern data architecture.
• Modern data platform for small and medium businesses - Build a modern data platform architecture for small and medium businesses using Microsoft Fabric and Azure Databricks.
• Real-time analytics with Azure Data Explorer - Analyze data in real time using Azure Data Explorer and Azure Service Bus.

Guides

Technology choices

• Analytics and reporting - Compare options for data analysis and visualization in Azure.
• Batch processing - Evaluate batch processing technologies for big data workloads.
• Stream processing - Compare stream processing technologies for real-time analytics.
• Choose an analytical data store - Guidance on selecting the right analytical data store.
• Choose an analytical data store in Microsoft Fabric - Guidance on selecting data stores in Microsoft Fabric.

Disaster recovery for Azure data platform

• Overview - Overview of disaster recovery strategies for Azure data platforms.
• Architecture - Architecture patterns for disaster recovery in Azure data platforms.
• Scenario details - Detailed scenarios for implementing disaster recovery.
• Recommendations - Best practice recommendations for disaster recovery.

Learn about analytics on Azure

If you're new to analytics on Azure, the best place to learn more is with Microsoft Learn, a free, online training platform. You'll find videos, tutorials, and hands-on learning for specific products and services, plus learning paths based on your job role, such as developer or data analyst.

Here are some resources to get you started:

• Browse Azure data topics
• Introduction to Microsoft Azure Data core data concepts
• Get started with Microsoft Fabric

Learning paths by role

• Data analyst: Get started with Microsoft data analytics
• Data engineer: Implement a Data Analytics Solution with Azure Databricks
• Data scientist: Build machine learning solutions using Azure Databricks

Organizational readiness

If your organization is new to the cloud, the Cloud Adoption Framework can help you get started. This collection of documentation and best practices offers proven guidance from Microsoft designed to accelerate your cloud adoption journey. For more information on clouds-scale analytics, see Cloud-scale analytics.

To help assure the quality of your analytics solution on Azure, we recommend following the Azure Well-Architected Framework. It provides prescriptive guidance for organizations seeking architectural excellence and discusses how to design, provision, and monitor cost-optimized Azure solutions.

For data workload guidance aligned to the Well-Architected Framework pillars, see Azure Well-Architected Framework for data workloads.

Path to production

Knowing how to store your data is one of the first decisions you need to make in your journey to analytics on Azure. Then you can choose the best data analytics technology for your scenario.

Key decision points include:

• Data storage: Choose between data lakes, data warehouses, or lakehouses based on your data structure and query patterns. For guidance on selecting and designing database solutions that power analytics workloads, see Databases architecture design.

• Processing model: Determine whether batch processing, stream processing, or a combination best fits your workload requirements.

• Analytics tools: Select BI and AI technologies that align with your team's skills and business needs.

To view different architecture styles for analytics solutions, see architectures.

Best practices

High-quality analytics start with robust, trustworthy data. At the highest level, information security practices help ensure that your data is protected in transit and at rest. Access to that data must also be trusted. Trustworthy data implies a design that implements:

• Governance policies - Define clear data ownership, classification, and access policies.

• Identity and access management - Implement role-based access control and least-privilege principles.

• Network security controls - Protect data flows between services and prevent unauthorized access.

• Data protection - Encrypt data at rest and in transit.

At the platform level, the following big data best practices contribute to trustworthy analytics on Azure:

• Orchestrate data ingestion - Use a data workflow or pipeline solution such as those supported by Azure Data Factory or Microsoft Fabric pipelines.

• Process data in place - Use a distributed data store, a big data approach that supports larger volumes of data and a wider range of formats.

• Scrub sensitive data early - Remove or mask sensitive data as part of the ingestion workflow to avoid storing it in your data lake.

• Consider total cost - Balance the per-unit cost of the compute nodes needed against the per-minute cost of using those nodes to complete a job.

• Create a unified data lake - Combine storage for files in multiple formats, whether structured, semi-structured, or unstructured. Use Azure Data Lake Storage Gen2 as your single source of truth. For example, see BI solution architecture in the Center of Excellence.

Stay current with analytics

Azure analytics services are evolving to address modern data challenges. Stay informed about the latest updates and planned features:

Get the latest updates on Azure products and features.

Stay current with these key analytics services:

• What's new in Microsoft Fabric
• Azure Databricks release notes
• What's new in Azure Data Explorer
• What's new in Power BI

Additional resources

Analytics is a broad category and covers a range of solutions. The following resources can help you discover more about Azure.

Hybrid

The vast majority of organizations need a hybrid approach to analytics because their data is hosted both on-premises and in the cloud. Organizations often extend on-premises data solutions to the cloud. To connect environments, organizations must choose a hybrid network architecture.

Key hybrid analytics scenarios:

• Mainframe modernization: Modernize mainframe and midrange data - Integrate legacy data sources with modern analytics platforms.
• On-premises integration: Unified hybrid and multicloud operations - Connect on-premises databases to cloud analytics.
• Edge analytics: Process data at the edge and aggregate insights in the cloud.

Real-time analytics

Real-time analytics enables organizations to act on data as it arrives. Here are some resources to help you get started with real-time analytics on Azure:

• Real-time analytics on big data architecture - Process and analyze streaming data at scale.
• IoT analytics with Azure Data Explorer - Analyze IoT telemetry data in real time.
• Stream processing with Azure Stream Analytics - Build serverless streaming solutions.
• Create a modern analytics architecture by using Azure Databricks - Enterprise-grade analytics using Apache Spark.

Browse more analytics examples in the Azure Architecture Center

AWS or Google Cloud professionals

These articles can help you ramp up quickly by comparing Azure analytics options to other cloud services:

• Analytics and big data in Database technologies on Azure and AWS

• Big data and analytics in Google Cloud to Azure services comparison