Edit

Mining equipment monitoring

Azure Analysis Services
Azure Logic Apps
Azure Data Lake Storage
Azure Databricks
Azure Monitor

Solution ideas

This article is a solution idea. If you'd like us to expand the content with more information, such as potential use cases, alternative services, implementation considerations, or pricing guidance, let us know by providing GitHub feedback.

Mining companies can have Azure continually monitor the performance data from their equipment or from other assets. Analysis of the data identifies anomalies and results in recommendations for maintenance and repair. Such monitoring can prevent failures and reduce operating costs.

Architecture

Diagram showing the architecture for monitoring mining equipment.

Download a Visio file of this architecture.

Dataflow

The data flows through the solution as follows:

  1. Equipment and other assets have integrated sensor systems that deliver sensor data (in CSV files) to a folder in an FTP server or to Azure Storage.
  2. Azure Logic App monitors the folder for new or modified files.
  3. Logic App triggers the Data Factory pipeline when a file is added to the folder, or when a previously added file is modified.
  4. Azure Data Factory obtains the data from the FTP server or from Azure Storage, and stores it to a data lake that Azure Data Lake provides. The Delta Lake open-source software augments Data Lake capabilities.
  5. The cloudFiles feature of Azure Databricks Auto Loader automatically processes new files as they arrive at the data lake, and can also process existing files.
    1. cloudFiles uses structured streaming APIs to check if sensor values exceed thresholds. If so, it copies the values to a separate storage folder (Alerts).
    2. After appropriate cleansing and transforming of the data, it moves the data to Delta Lake Bronze/Silver/Gold folders. The folders contain various transformations of the data; for example, ingested (Bronze), to refined (Silver), to aggregated (Gold).
  6. An Azure Synapse connector in Azure Databricks moves the data from the data lake to an Azure Synapse Analytics dedicated SQL pool.
  7. Whenever a new alert arrives in the Alerts folder, Azure Function Apps sends notifications to Azure Notification Hub.
  8. Notification Hub then sends notifications to various mobile platforms to alert operators and administrators of events that require attention.
  9. Monitoring advisors can create visual reports to explore the data. They can publish and share them, and collaborate with others. Power BI integrates with other tools, including Power Apps. Advisors can integrate Power BI reports into a Canvas App in Power Apps for a good user experience.

Components

Data is loaded from these different data sources using several Azure components:

  • Azure Data Lake Storage makes Azure Storage the foundation for building enterprise data lakes on Azure. It can quickly process massive amounts of data (petabytes).
  • Azure Data Factory is a managed service that orchestrates and automates data movement and data transformation. In this architecture, it copies the data from the source to Azure Storage.
  • Azure Logic Apps are automated workflows for common enterprise orchestration tasks. Logic Apps includes connectors for many popular cloud services, on-premises products, and other applications.
  • Azure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. Databricks is integrated with Azure to provide one-click setup, streamlined workflows, and an interactive workspace that was designed in collaboration with the founders of Apache Spark.
  • Azure Databricks – Auto Loader provides a structured streaming source called cloudFiles. The cloudFiles source automatically processes new files as they arrive at a directory, and can also process other files in the directory.
  • Azure Synapse Analytics is a distributed system for storing and analyzing large datasets. Its use of massive parallel processing (MPP) makes it suitable for running high-performance analytics.
  • Azure Functions allows you to run small pieces of code (called "functions") without worrying about application infrastructure. Azure Functions is a great solution for processing bulk data, integrating systems, working with the internet-of-things (IoT), and building simple APIs and micro-services.
  • Power BI is a suite of business analytics tools to analyze data and provide insights. Power BI can query a semantic model stored in Analysis Services, or it can query Azure Synapse directly.
  • Power Apps is a suite of apps, services, and connectors for building custom business apps. It includes an underlying data platform (Microsoft Dataverse) and a rapid development environment.

Scenario details

Potential use cases

  • Monitor mining equipment and other equipment that can provide the needed data. This solution is ideal for the energy industry.

Contributors

This article is maintained by Microsoft. It was originally written by the following contributors.

Principal author:

Next steps

Information about the Delta Lake open-source project for building a Lakehouse architecture:

See the following related database architectural guidance:

See the following related IoT architectural guidance: