Create data standardization

Important

Some of the functionality described in this release plan has not been released. Delivery timelines may change and projected functionality may not be released (see Microsoft policy). Learn more: What's new and planned

Enabled for Public preview General availability
Users by admins, makers, or analysts Mar 2024 Mar 2024

Business value

The AgriTech sector is characterized by a myriad of data providers, each following their own methodologies and standards. This fragmentation often presents challenges when attempting to make cross-provider comparisons or aggregations. Consider the instance where multiple brands and models are used in a single farm. As each provider has a unique method for calculating wet yield, along with a unique file format and structure in which the data is delivered, direct comparisons and aggregate calculations become questionable due to inconsistency in how the input values were generated, presented, and in processing multiple file types. Our new feature aims to overcome these challenges with a consistent, user-friendly query experience across diverse data sources. It offers a transformation and harmonization path within the Extensible Data Connector, alongside provisions in the Azure Data Manager for Agriculture (ADMA) model for a standardized format and shape as a silver store collection that does not in any way interfere with or modify the bronze store.

Feature details

Our new feature aims to overcome these challenges with a consistent, user-friendly query experience across diverse data sources. It offers a transformation and harmonization path within the Extensible Data Connector, alongside provisions in the ADMA model for a standardized format and shape as a silver store collection that does not in any way interfere with or modify the bronze store.

  • Data Transformation Pathway via the Data Connector model: This feature provides a pathway to standardize diverse data formats as a subsequent step to a primary data ingress connector. Using an orchestrated ETL pipeline, it bridges the gap between different methodologies and standards, aligning them to a common measure, format, and shape. This enables reliable aggregation and comparison across data from different providers.
  • Harmonized Data Format: To tackle the issue of incompatible file formats, our solution incorporates translation to standardize diverse formats into a universally accepted format. This allows for smoother data merging and more consistent data management.
  • User-friendly Query Experience: By adopting standardized data formats and measures, we simplify data querying for organizations. This ensures a more streamlined and efficient user experience, even when working with data from various sources.