Overview of data estate strategy

  • Article

Integrating financial data from various systems and applications is a costly and time-intensive process. To address the issues, implementing a data estate strategy becomes crucial, as it establishes uniform standards for organizations to efficiently manage all their data, regardless of its storage location or format.

Data estate strategy is a comprehensive and structured approach that organizations adopt to manage their entire data ecosystem effectively. The strategy is to develop a well-defined plan and set of guidelines to acquire, store, process, secure, and using data across various sources, systems, and applications within the organization. Financial institutions handle a diverse range of data, including customer transactions, investment portfolios, market trends, and regulatory compliance. Hence, effective data management becomes crucial to maintain stability and gain a competitive edge.

Data integration and interoperability challenges in financial services

Addressing data integration and interoperability challenges is crucial for financial institutions to create a cohesive and data-driven ecosystem, enabling them to make informed decisions, provide better customer experiences, and stay ahead in a competitive financial landscape. Following are some common data integration and interoperability challenges:

  • Diverse data sources: Financial institutions often deal with data from various internal systems, legacy applications, third-party providers, and external data feeds, leading to challenges in consolidating and integrating data from multiple sources.

  • Data silos: Data might be isolated and stored in separate systems within different departments, hindering cross-functional collaboration. Hence, it is difficult to gain a comprehensive view of the organization's operations and customer interactions.

  • Inconsistent data formats: Different systems might use varying data formats, structures, and naming conventions, which makes it difficult to cohesively merge and analyze data. The varying data formats can lead to data quality and reporting issues.

  • Real-time data updates: Many financial processes require real-time data updates, and ensuring timely data synchronization across systems becomes crucial to avoid discrepancies and potential errors.

  • Legacy systems compatibility: Financial institutions might have legacy systems that don't easily integrate with modern data platforms or APIs, resulting in a technology gap that impedes seamless data flow.

  • Compliance and security concerns: Integrating data across systems must comply with industry regulations and data security standards, adding complexity to data integration projects. Such as handling sensitive customer data and consent to share data with other parties.

  • Scalability and performance: As financial data grows exponentially, data integration solutions must be scalable to handle increasing data volumes without compromising performance.

  • Data mapping and transformation: Mapping data from one system to another and transforming data to match different formats require careful planning and validation to prevent data loss or corruption.

  • API and data exchange challenges: Interoperability issues arise when financial institutions need to exchange data with external partners, clients, or regulatory bodies, necessitating standardized APIs and data exchange protocols.

  • Data governance and ownership: Defining clear data ownership and governance policies becomes essential to ensure data accuracy, accountability, and regulatory compliance during data integration processes.

Operational data categories

Operational Data Categories refer to the different types or classifications of data used in operational processes within an organization. Effective management of operational data categories is essential for smooth business operations, decision-making, and compliance with relevant regulations. Within the context of Microsoft Cloud for Financial Services, this type of data typically falls into one of four categories: master data, transactional data, configuration data, and inferred data.

Microsoft Cloud for Financial Services offers a set of comprehensive and tailored data models to address the unique data requirements of the industry. These data models represent commonly used concepts and activities, such as customer profiles, financial holdings, relationships, and applications. These activities simplify the analysis of financial data by relationship managers and manage onboarding processes by agents.

Some high-level data categories that construct the Microsoft Cloud for Financial Services data models are:

Data category Description Example
Master Data Typically represents real-world entities or common business data, which is maintained over time and therefore receives a comparably low to medium number of updates. Sometimes also referred to as Static data, which might be misleading. Company (Account)
Contact
Person
Transactional Data This type of data is high in volume due to the nature of its use. Transactional data typically refers to events or activities related to the master data tables. The information is either created automatically or recorded by a user. Financial Holdings
Application
Tasks
Documents
Configuration Data Configuration data is data about different setups needed to prepare your environment for use. Life Events Configuration
Document Definition
Inferred Data Represents activity that is related to the master data entities but, which has a level of uncertainty Age
Birthday Alert

Types of operational data

When diving into Microsoft Cloud for Financial Services data models, it's important to know that you can encounter different types of data. These data types vary based on how often they change, how much data there is, and how the data is structured. The following table shows the different data types within these models. You can use these definitions to organize data logically and help with tasks such as creating data catalogs, data classifications, and data maps, which are crucial for data management and compliance.

Data category Type of Data Description Example
Master Data Profile Represents profile of a customer or company Account
Contact
Person
Master Data Bank Represents financial institution organization Bank, Branch
Transactional Data Financial Holdings Represent the financial holdings of both individual customers and groups. Financial holding
Financial holding instrument
Financial market product
Transactional Data Life Events and Financial Goals Represent life events and their associated financial goals. Financial Goal
Life Event
Transactional Data Business Process Data related to management of business processes like onboarding application Application
Transactional Data Document Customer and process documents Annotation
Transactional Data Relationship Group memberships like household Customer financial holding
Group financial holding
Configuration Data Reference Data used to be referenced from other data types. Currency
Choice(optionset) definitions
Configuration Data Definition Defining parameters for related capability Task Definition
Document Definition
Life Events Configuration
Inferred Data Calculated Data elements calculated based on other information Tenure field in Contact table calculated based on Join Date field
Inferred Data Alerts Any alerts related to customer financial holdings and customers calculated and displayed in user interface Birthday alert, Expired Soon status for Cards

Data persistency approaches

Before you go deeper into implementation guidance, start by defining the data persistency approaches you can take. The approach you take shapes your design decision for the data estate strategy.

Microsoft Cloud for Financial Services data models are deployed into a Dataverse environment. Data persisted in Dataverse is a requirement for the Cloud for FSI first-party applications like Unified Customer Profile to operate natively. However, there can be some scenarios where you might consider choosing a hybrid-path with approaches such as:

  • Real-time data needs: If the business requirement can only be satisfied by accessing data from master systems in real-time, such as to determine the current available limit in credit cards, or current account balance.
  • High-volume scenarios: You might have a business requirement where you want to access data elements that are high-volume in nature, like interaction histories or account transactions. Persisting this high-volume data in Dataverse might not be the recommended approach for cost-optimization and overall system performance as described in the performance efficiency pillar. You might want to take a different approach to prevent data proliferation or cope with the side effects of substantial data volumes.
  • Availability of APIs and access from public cloud: Financial organizations usually have a strict access policy while systems access customer and financial data. The availability of APIs and access restrictions from public cloud might necessitate a client-based scripting approach to access API Data.

Based on the specified scenarios, you might need to consider a different approach from Dataverse for data persistence. The following table compares different data persistence approaches with the Dataverse option and shares considerations for the use of other approaches.

Attribute Data persistence in Dataverse standard tables Data persistence in Dataverse elastic tables (Preview) Cache in Dataverse Presentational integration
Approach Master systems for financial data are hydrating and feeding data elements to the Microsoft Cloud for Financial Services data model. Activity data such as interaction history is ingested into elastic tables in Dataverse. No initial data load except for master and configuration data. Data is pulled in real-time via plug-ins / web-hooks. Data is fetched on-demand to display via custom user interfaces developed or embedded. The approach doesn’t utilize any existing user interface, controls, and data model.
Data Persistence At least bare minimum data is persisted in Dataverse. Data stored in Dataverse elastic tables powered by Azure Cosmos DB. Persisted temporarily for only customers viewed by agents. Records are removed from tables based on retention rules. No data persisted in Dataverse.
Data Recency Mostly near-real time or daily Mostly near-real time or daily Real-time with some latency to persist data before rendering. No regular updates to cached data Not applicable
User Interface Using native forms and controls before customization Can introduce elastic tables to forms with configuration Using native forms and controls before customization Low code or customization required to develop PCF control / Custom Pages in Power Apps.
Consideration Supports async and batch integration patterns to hydrate and feed data to data model. The data volume to be synced and data latency scenarios should be considered in design.

Real-time data fetch is supported with customization.		 Currently in preview and not recommended for production use.
Elastic tables don't support multi-record transactions where multi-records aren't transactional with each other.
Known Issues		 Higher latency as data persistence is fulfilled before rendering UI. The complexity of transformation requirements from source to data model can make it even worse for performance and potential timeout issues in plug-ins.

Real-time integrations are more prone to time out and aren't scalable as other methods.

Tightly coupled with APIs. Changes in data model require changes in the integration code.		 Heavy high customization on user interface.

Not real use of built-in data model and controls.
Use When Recommended approach to take a configure-first approach with cost-optimization and performance. Use standard tables in following situations:
- Your application requires strong data consistency.
- Your application requires relational modeling and needs transactional capability across tables or during plug-in execution.
- Your application requires complex joins.		 High-volume scenarios where you need to scale horizontally to handle large amounts of data and high levels of throughput with low latency. Use elastic tables in these situations:
- Your data might be unstructured or semi-structured, or your data model might constantly be changing.
- You need horizontal scaling to handle workload growth over time or bursty workload at a given point.
- You need to handle a high volume of read and write requests.		 Short-term lived elements like Alerts & Notifications where temporary persistence is aligned with the lifecycle of the data elements For scenarios where real-time integration is the only approach to fulfill the business requirement and APIs to access the information is restricted from public cloud

Next steps