Get started with analytics architecture design

Organizations rely on the compute, storage, and analytical power of Azure to scale, stream, predict, and view their data. Analytics solutions transform volumes of data into useful business intelligence (BI), such as reports and visualizations, and inventive AI, such as forecasts based on machine learning. Azure offers a range of cloud-based analytics tools for organizations that are new to analytics and organizations that need to expand their implementation. Analytics solutions help organizations use data at scale. You can use a big data architecture or an Internet of Things (IoT) architecture to process raw data and then move it to an analytical data store. This data store becomes a single source of truth that can power insightful analytics solutions.

Architecture

Diagram that shows four columns, labeled Learn, Assign roles, Choose storage, and Choose technology. Two tiles appear in the Learn column, labeled Azure data services and Data modeling. Three user groups appear in the Assign roles column: Data analysts, Data engineers, and Self-service users. One square tile labeled Analytical data store appears in the Choose storage column. Four functions appear in the Choose technology column: Report and visualize, Stream data, Make predictions, and Scale analytics.

Download a Visio file of this architecture.

The previous diagram demonstrates a typical basic or baseline analytics implementation. For real-world solutions that you can build in Azure, see Analytics architectures.

Explore analytics guides, architectures, and solution ideas

The articles in this section include guides and fully developed architectures that you can deploy in Azure and expand to production-grade solutions. Solution ideas demonstrate implementation patterns and possibilities to consider as you plan your analytics proof-of-concept (POC) development. These articles can help you decide how to use analytics technologies in Azure.

Analytics guides

The following articles help you evaluate and select the best analytics technologies for your workload requirements:

• Choose a data analytics and reporting service: Compare options for data analysis and visualization in Azure.

• Choose a batch processing service: Evaluate batch processing technologies for big data workloads.

• Choose a stream processing service: Compare stream processing technologies for real-time analytics.

• Choose an analytical data store: Get guidance on analytical data store selection.

• Choose an analytical data store in Microsoft Fabric: Get guidance on data stores in Fabric.

The following articles provide guidance about disaster recovery (DR) strategies for Azure data platforms:

• Overview: Overview of DR strategies for Azure data platforms.

• Architecture: Architecture patterns for DR in Azure data platforms.

• Scenario details: Detailed scenarios for DR implementation.

• Recommendations: Best practices for DR.

Analytics architectures

The following production-ready architectures demonstrate end-to-end analytics solutions that you can deploy and customize:

• Analytics end-to-end with Fabric: Build a modern analytics platform by using Fabric.

• Data warehousing and analytics: Integrate data from multiple sources into a unified analytics platform.

• Use Fabric to design an enterprise BI solution: Design an enterprise BI solution by using Fabric.

• Near real-time lakehouse data processing: Use Azure Synapse Analytics and Azure Data Lake Storage for near real-time data lakehouse processing.

• Real-time sync of MongoDB Atlas to Azure Synapse Analytics: Synchronize MongoDB Atlas data to Azure Synapse Analytics in real time.

• Stream processing with Azure Databricks: Create an end-to-end stream processing pipeline by using Azure Databricks.

• Stream processing with Azure Stream Analytics: Build a stream processing pipeline that ingests data, correlates records, and calculates rolling averages.

• Modern data warehouse for small and medium businesses: Build a modern data warehouse solution designed for small and medium businesses.

Analytics solution ideas

The following analytics solution ideas demonstrate implementation patterns and possibilities to explore:

• Ingestion, extract, transform, and load (ETL), and stream processing pipelines with Azure Databricks: Create ETL pipelines for batch and streaming data to simplify data lake ingestion.

• Modern analytics architecture with Azure Databricks: Collect, process, analyze, and visualize data by using a modern data architecture.

• Modern data platform for small and medium businesses: Build a modern data platform architecture for small and medium businesses by using Fabric and Azure Databricks.

• Real-time analytics with Azure Data Explorer: Analyze data in real time by using Azure Data Explorer and Azure Service Bus.

Learn about analytics on Azure

Microsoft Learn provides free online training resources for Azure analytics technologies. The platform offers videos, tutorials, and hands-on labs for specific products and services, along with learning paths organized by job role.

The following resources provide foundational knowledge for analytics implementations on Azure:

• Browse Azure data articles
• Introduction to Microsoft Azure data core data concepts
• Get started with Fabric
• Get started with Microsoft data analytics
• Implement a data analytics solution with Azure Databricks
• Build machine learning solutions by using Azure Databricks

Organizational readiness

Organizations at the beginning of the cloud adoption process can use the Cloud Adoption Framework for Azure to access proven guidance that accelerates cloud adoption.

To help ensure the quality of your analytics solution on Azure, follow the guidance in the Azure Well-Architected Framework. The Well-Architected Framework provides prescriptive guidance for organizations that seek architectural excellence and describes how to design, provision, and monitor cost-optimized Azure solutions.

Best practices

Best practices in analytics ensure that solutions are scalable, reliable, cost efficient, and secure.

Data analytics

To use analytics on Azure, you need to decide how to store your data. Then you can choose the best data analytics technology for your scenario. Consider the following factors:

• Data storage: Choose between data lakes, data warehouses, and lakehouses based on your data structure and query patterns. For more information about the database solutions that power analytics workloads, see Database architecture design.

• Processing model: Determine whether batch processing, stream processing, or a combination best fits your workload requirements.

• Analytics tools: Select BI and AI technologies that meet your team's skills and business needs.

Trustworthy data

For high-quality analytics, you need robust, trustworthy data. Information security practices help ensure that your data is protected in transit and at rest. Access to your data must also be secure. To help produce trustworthy data, consider the following practices and controls:

• Governance policies: Define clear data ownership, classification, and access policies.

• Identity and access management: Implement role-based access control and least-privilege principles.

• Network security controls: Protect data flows between services and prevent unauthorized access.

• Data protection: Encrypt data at rest and in transit.

At the platform level, the following big data best practices contribute to trustworthy analytics on Azure:

• Orchestrate data ingestion: Use an Azure Data Factory or Fabric pipelines-supported data workflow or pipeline solution.

• Process data in place: Use a distributed data store, which is a big data approach that supports larger volumes of data and a wider range of formats.

• Scrub sensitive data early: To avoid accidental storage of sensitive data in your data lake, remove or mask this data as part of the ingestion workflow.

• Consider total cost: Balance the per-unit cost of the required compute nodes against the per-minute cost to run a job on those nodes.

• Create a unified data lake: Combine storage for files in multiple formats, whether structured, semi-structured, or unstructured. Use Data Lake Storage as your single centralized source. For more information, see BI solution architecture in the Center of Excellence.

Stay current with analytics

Azure analytics services evolve to address modern data challenges. Stay informed about the latest updates and features.

To stay current with key analytics services, see the following articles:

• What's new in Fabric?
• Azure Databricks release notes
• What's new in Azure Data Explorer
• What's new in Power BI?

Other resources

The following resources can help you discover more about analytics.

Real-time analytics

Organizations can use real-time analytics to act on data as it arrives. The following resources can help you get started with real-time analytics on Azure:

• Real-time analytics on big data architecture: Process and analyze streaming data at scale.

• IoT analytics with Azure Data Explorer: Analyze IoT personal data in real time.

• Stream processing with Stream Analytics: Build serverless streaming solutions.

• Create a modern analytics architecture by using Azure Databricks: Discover enterprise-grade analytics by using Apache Spark.

• For more analytics examples, see the Azure Architecture Center

Amazon Web Services (AWS) or Google Cloud professionals

To help you get started quickly, the following articles compare Azure analytics options to other cloud services and provide migration guidance:

• Relational database technologies on Azure and AWS
• Google Cloud to Azure services comparison