Analytics data estate
Microsoft Cloud for Financial Services comprises starter applications developed by Microsoft that include industry-specific data models and tailored processes. While the solution's primary focus is to facilitate business scenarios through modular components, its ready-to-use analytics capabilities are limited.
Overview
This article explains how to create an analytics data estate for Microsoft Cloud for Financial Services by following the configure-first and custom report approaches.
Key components of Analytics data estate
The key components of an Analytics data estate typically include:
Data sources: The origin points of your data, which can include Dataverse and any other structured and unstructured data from databases, files, and more.
Data ingestion: The process of collecting and importing data from various sources into a centralized storage or data lake for further processing and analysis.
Data storage: The location where your data is stored, often in data lakes or lake databases, allowing for efficient data retrieval and management.
Data transformation: Involves data cleaning, normalization, enrichment, and structuring to prepare raw data for analysis. Data transformation can include ETL (Extract, Transform, Load) processes.
Data visualization: The presentation of data in graphical or visual formats, making it easier for users to interpret and understand the insights generated.
Data governance and security: Ensures data is handled in a compliant and secure manner, including data access controls, encryption, and audit trails.
User access and collaboration: Tools and interfaces that allow users to access and collaborate on data analysis, reports, and dashboards.
Monitoring and performance tuning: The ongoing monitoring of data processes and system performance to ensure efficiency and reliability.
Scalability and elasticity: The ability to scale your analytics data estate horizontally or vertically to accommodate growing data volumes and user demands.
Backup and disaster recovery: Implementing strategies and solutions for data backup and recovery if there's data loss or a system failure.
These components collectively form an analytics data estate, which serves as the foundation for data-driven decision-making and advanced analytics within an organization. Each component's specific technologies and tools might vary depending on the organization's needs and infrastructure.
Implementation approaches
We recommend following a configure-first approach before creating a data estate for custom reporting. You might need to follow the custom reporting approach based on factors such as:
- Unique business needs that the configure-first approach can't satisfy
- Level of transformation required
- Integration and consolidation of data with other data sources
- Formatting and visualization that align with your organization's branding and presentation standards
- Any limitations on the built-in methods such as transformation capability and number of records
Configure-first approach
Power Apps includes many extensible built-in reporting capabilities, such as:
- Dynamics 365 Charts / Dashboards
- SSRS reports running in Dynamics 365
- Power BI dashboards and embedded reporting
The figure illustrates how these reporting capabilities interact with Dataverse, especially different connectors you can use for Power BI:
Download a printable PDF of this diagram.
You can use the following table to compare and select the option you want for your implementation.
| Attribute | Dynamics 365 charts/Dashboards | SSRS reporting | Power BI |
|---|---|---|---|
| Number of records | Queries can span up to 50,000 rows | No Limit | No Limit |
| Data Source | Dataverse Only | Dataverse Only | Dataverse with the capability to integrate with other data sources |
| Data Latency | Real-time | Real-time | Near Real-Time |
| Observes Dynamics 365 Security Model | Yes | Yes | Extra development required to apply Row-Level Security in Power BI |
| Considerations | Considerations | Exporting to Excel is limited to 65,536 rows or 256 columns | Additional licensing might be required. Some considerations for each data connector.1. Dynamics 365 Connector (legacy) - To be superseded by Dataverse Connectors 2. Dataverse - Can't access virtual or elastic tables.3. Direct Query |
| Use When | Recommend using where applicable as long as business requirements and technical considerations are met. | For situations; - necessitating report export - require filtering by end-use - other cases where the utilization of Dynamics 365 and Power BI options isn't viable, SSRS reports can be created and employed. | Use for complex reporting requirements and where Dynamics 365 dashboards aren't an option. |
Custom reporting approach
Custom reporting approach can combine a range of Azure services to ingest, store, transform, and serve data and insights from different sources (structured, semi-structured, unstructured, and Azure data services including Dataverse). The services covered by this architecture are only a subset of a larger family of Azure services. Similar outcomes can be achieved by using Synapse Analytics only or by Fabric only. This section aims to provide both these different technical options together however your implementation can scope one technical option over the other.
Custom reporting architecture
Download a printable PDF of this diagram.
Data flow
1. Ingestion
You can use Azure Synapse pipelines, Fabric (preview) shortcuts, pipelines & data flows, notebook or any other applicable ETL tools to pull data from a wide variety of databases, both on-premises and in the cloud. Pipelines can be triggered based on a predefined schedule in response to an event, or can be explicitly called via REST APIs.
2. Store - Raw Data
The way you structure your data transformation layers can vary depending on how intricate the transformation process needs to be when moving from raw data to an analytics data model. You can take the medallion pattern to have flexible access and extendable data processing. Medallion pattern is composed of Bronze, Silver, and Gold storage layers:
Bronze tables provide the entry point for raw data when it lands in Data Lake Storage. The data is taken in its raw source format and converted to the open, transactional Delta Lake format for processing. Within the raw data lake layer, organize your data lake follow the best practices around which layers to create, what folder structures to use in each layer, and what files format to use for each analytics scenario.
Silver tables store data while it's being optimized for BI and data science use cases. The Bronze layer ingests raw data, and then transformation tasks are done to filter, clean, transform, join, and aggregate the data into Silver curated datasets.
Gold tables contain enriched data, ready for analytics and reporting.
You can use the technical options in the transform layer to stage the data copied from the nonstructured data sources into the raw layer of your Azure Data Lake Store Gen 2 data lake. Save data by preserving the original format acquired from the data sources.
3. Synapse Link
Azure Synapse Link for Dataverse enable you to run near real-time analytics over operational and business application data from Dataverse. You can access the selected Dataverse tables (only tables enabled for change tracking) at your Synapse Analytics workspace. You can then combine this data with the data transformed in your data lake (Step-2) and you can also opt to enhance it with more data elements using the Banking Synapse database template.
Microsoft Fabric announced direct integration with Dataverse that enables organizations to extend their existing analytics in Dataverse into Fabric. You can view the Azure Synapse Analytics lakehouse, SQL endpoint, and the default dataset generated by Dataverse in the Fabric workspace. You can enable Fabric links for your existing links in Azure Synapse Link. Fabric links simplify downstream data pipelines, and you can use new features like Power BI DirectLake mode reports.
4. Transform
The transformation layer plays a pivotal role in the data loading process. This layer enables the data to be efficiently loaded into streamlined, compact Delta Lake tables or directories located within the bronze layer of the data lake storage. Additionally, it serves as a critical component for the conversion of data sourced from Dataverse into the Banking Synapse database template or any other structured format. At this stage, the data undergoes a sequence of operations, including filtering, cleansing, transformation, joining, and aggregation.
Ultimately, the transformation culminates with data conversion into the Gold tables format, rendering it immediately accessible and suitable for direct utilization in analytics and reporting endeavors. You can use either data flows, SQL serverless queries, or Spark notebooks and some other methods in Synapse Analytics or Fabric.
5. Serve
Power BI, alongside other data sources, can access the populated analytics data model can be accessed to create comprehensive and insightful dashboards. To provide end-users with access to these enriched analytics, you can seamlessly embed Power BI dashboards into Power Apps to make it accessible right within the operation system.
Dataverse generates an enterprise-ready lakehouse and SQL endpoint for your Dynamics 365 data. As a result, it's easier for data engineers and database admins to combine data from multiple sources and build custom analytics with Spark, Python, or SQL.
6. Platform
Following are some of those technologies mentioned as platform technologies in the architecture:
- Microsoft Entra ID: Identity services, single sign-on and multifactor authentication across Azure workloads.
- Microsoft Cost Management: Financial governance over your Azure workloads.
- Azure Key Vault: Secure credential and certificate management. For example, Azure Synapse Pipelines can retrieve credentials and certificates from Azure Key Vault used to securely access data stores.
- Azure Monitor: Collect, analyze, and act on telemetry information of your Azure resources to proactively identify problems and maximize performance and reliability.
- Microsoft Defender for Cloud: Strengthen and monitor the security posture of your Azure workloads.
- Azure DevOps and GitHub: Implement DevOps practices to enforce automation and compliance with your workload development and deployment pipelines for Azure Synapse.
- Azure Policy: Implement organizational standards and governance for resource consistency, regulatory compliance, security, cost, and management.
See also
- Analytics end-to-end with Azure Synapse
- Microsoft Fabric Lakehouse end-to-end scenario
- View Dataverse data in Microsoft Fabric (preview)