Azure Logic Apps framework for first-party connectors

We're building a framework that will enable partners and solution providers to build connectors and integrate them with the Microsoft Sustainability Manager solution. This article provides details about the first group of connectors we built using Azure Logic Apps in collaboration with the partner team. This article provides guidance on how to build a Logic Apps connector for Microsoft Sustainability Manager and share the data required to calculate the carbon and sustainability footprint for customers.

Key terms

You'll need to know the following definitions used in this article:

Provider: A provider is a partner offering one or more services that enables an enriched, pre-configured connection with a data solution. Organizations select the providers that apply to their business by connecting to them. After they connect, an instance of that provider is created, and it can include more than one provider solution. Customers can establish a connection with each provider solution to bring their data into Microsoft Sustainability Manager.

Connector: A connector is software that is either built within the product or delivered as a separate module that allows a user to ingest (or import) data into Microsoft Sustainability Manager. Each provider can have one or more connectors for each of their solutions that a user connects to.

Logic Apps: Azure Logic Apps is a cloud platform where you can create and run automated workflows with little to no code. By using the visual designer and selecting from prebuilt operations, you can quickly build a workflow that integrates and manages your apps, data, services, and systems.

The following table briefly defines core terminology and concepts in Azure Logic Apps.

Term Description
Logic App The Azure resource you create when you want to build a workflow. There are multiple logic app resource types that run in different environments.
Workflow A series of steps that define a task, business process, or workload. Each workflow starts with a single trigger, after which you must add one or more actions.
Trigger Always the first step in any workflow and specifies the condition for running any further steps in that workflow. For example, in this article, we'll use an HTTP trigger to trigger the logic apps and pass the trigger parameters as body content.
Action Each subsequent step in a workflow that follows the trigger. Every action runs some operation in a workflow.
Azure Resource Manager (ARM) template An ARM template is a JavaScript Object Notation (JSON) file that defines the infrastructure and configuration for your project. The template uses declarative syntax. In declarative syntax, you describe your intended deployment without writing the sequence of programming commands to create the deployment.

Provider solution requirements

Microsoft Cloud for Sustainability and its partners will build connectors for the Microsoft Sustainability Manager solution to get complete data sets for one or more carbon emissions and sustainability categories. The connectors must meet the following requirements:

  • Be available across geographies and serve different customer industry types and operational sizes.

  • Allow separation of users for connection credentials (target system administrator) and connection configuration (Microsoft Sustainability Manager user).

  • Provide developer tools, sandbox, and supporting documentation to build POC if necessary.

In addition, each connector must provide the following items:

  • Activity data, reference data, or pre-calculated emissions critical to define carbon and sustainability entitlement

  • Ability to retrieve data from partner solution by authenticating via user credentials

  • Ability to automate data mapping of entities and attributes

High-level design of Logic Apps Connectors

The following high-level design of the Logic Apps connectors includes a brief outline of the ingestion flow from the user's perspective. These steps aren't the ones you would follow to create a connector, but an overview of how the connector is used.

  1. Connector selection:

    1. User is prompted with the type of data they’d like to ingest along with the scope and category.
    2. In the next step, a list displays Power Query and custom connectors that support the data type, scope, and category that the user selected.
  2. Credential selection:

    1. To create a new credential for a Logic Apps connector, users need to specify the credentials.
    2. Microsoft Sustainability Manager validates the credentials using the protocol specified by the partner and submits the connection request to the backend.
  3. Connection creation and trigger:

    1. To fulfill the connection request, Microsoft deploys a Logic Apps instance using the ARM template specified by the partner. The ARM template should follow the connector paradigm described later in this article.
    2. Once the Logic Apps instance is deployed, it's triggered once, and the user can schedule it for daily runs. The connection creation is now complete.
    3. The user can optionally trigger the Logic Apps instance themselves from the user interface.
  4. Ingestion into Dataverse:

    1. Each Logic Apps workflow execution (whether triggered manually or scheduled) sends the transformed data in chunks over HTTPS call to the Sustainability microservice. The data would then be validated and ingested to the customer's environment.
  5. Connection deletion:

    1. Connection deletion is handled by the Logic Apps framework. When a user deletes a Logic Apps connection, we deprovision the Logic Apps instance for the given connection, then delete the connection itself.
    2. The deletion of connection entity would also cascade delete the connection refresh IDs.

Build a Logic Apps connector

To build a first-party connector in Microsoft Sustainability Manager, follow these steps to compile connector definition, provider definition, Logic Apps ARM template and credentials configuration. Add these files to a private GitHub repository and add our team as a contributor: mcfsconnectors@microsoft.com.

Step 1: Data review

The first step in building a connector involves reviewing the data type available from the partner solution and its application within Microsoft Sustainability Manager. These data types can include activity data, reference data or pre-calculated emissions, all of which are critical to define carbon and sustainability entitlement.

Step 2: Define connector and provider or partner

The next step for building the connector includes gathering additional information on the solution that is being integrated with Microsoft Sustainability Manager.

  • Provider definition:

    Information Description
    Name Name of your company.
    Description A brief description of your company.
    Logo Company logo in SVG format.
    URL Company URL.
  • Connector definition:

    Information Description
    Name Connector display name.
    Description A brief, two-sentence description about the connector.
    Logo Icon for the connector in SVG format.
    Data type The data type this connector ingests for the user. For example: Activity data, scope 1, category 2. For more information about data types, go to Record.

This image provides an example of how a connector would display. This example shows the sample logo. Distribution data connector is the display name, Johnnie McConnel is the provider's name, and Our internal data is the description.

Sample display of a Logic Apps connector.

Step 3: Define data source

Before creating a logic app, you need to define the data source and how it can be accessed using user credentials. Here are a few examples for reference:

  • HTTP REST: You can expose an HTTP REST API to expose data to the logic app. You can use HTTP authentication mechanisms to access this data from our logic app.

  • Azure Data Lake Storage: <https://<adlsname>.blob.core.windows.net/<folder name>/<file name>>, in which case the logic app would need relevant access to this URL with the appropriate SAS token.

  • MySQL table: <jdbc:mysql:replication//myUser:myPassword@[address=(host=myHost1)(port=1111)(key1=value1)]> The data could be stored in a database. In this case, the logic app may require the endpoint of the database along with a valid user ID and password. It may also require the name of the table where the data to be ingested is stored. For more information about connecting to an SQL database, go to Connect to an SQL database from workflows in Azure Logic Apps.

  • AWS S3: <s3.Region.amazonaws.com> Data might be stored in an S3 bucket or a similar file storage. Relevant access control would be required by the Logic Apps instance to access this storage bucket.

Step 4: Obtain credential information

After you define the data source and access details, you must provide the credentials information that must be collected from the user to authenticate to the source data.

For first-party connectors, Microsoft Sustainability Manager currently doesn't support SSO. The credentials will be collected and stored in the customer's Dataverse environment.

When users create a Logic Apps connection from Microsoft Sustainability Manager, they're prompted with a page similar to the following image to input relevant connection details between the Microsoft Sustainability Manager hosted logic app and the partner.

Screenshot of an Add credentials page.

Note

The Add credentials page is a dynamic page that can vary based on the partner connector. These fields are defined by a JSON file and can be different for each connector, as defined by the partner. A validation check is performed to ensure that these credentials are valid before they're saved.

Microsoft will construct the actual JSON file for the credentials page. You just need to answer the following questions for us to generate the file.

Question Example
What credential information would you need from the user? Username, password, secrets, API key, etc.
What are the various field types? We support all HTML input types like textbox, checkbox, dropdown, radio button, etc. Username: textbox, Password: textbox, Region: dropdown.
Enter the name of the fields, which need to be masked. Password, secret.
Enter the URL with headers and body where we can make an HTTP call with the credentials to validate it. Also add a response with response content and status code. POST: https:api.partner.com/token
Body:
{
“username”:”xyz”, “password”:”test”
}
Response:
“status”:
“message”:
“type”:
}

Step 5: Create the logic app

The next step involves creating a logic app. Microsoft will provide a sample ARM template that you can upload to your Azure subscription and start using it to develop a logic app that performs the following actions:

  • Use user credentials to authenticate their system and get data from the data source in a paginated way.

  • Apply transformations and map the data to the sustainability data model or water data model.

  • Add the transformed data in a JSON array.

To automatically deploy a logic app template to Azure, you can select the following Deploy to Azure button, which signs you into the Azure portal and prompts for the Subscription ID and resource group name where this logic app will be deployed.

Deploy to Azure

This sample logic app is a starting point for the first party connector. The inputs for this connector are sent via HTTPS trigger from the customer's Microsoft Sustainability Manager instance. This HTTP trigger contains details provided during the setup of the connector (including custom endpoint and credentials).

After deploying the logic app, you can view and edit it from the app designer:

Screenshot of the app designer.

Brief description of the actions:

  1. Trigger: Microsoft Sustainability Manager will trigger the Partner Logic App to run via an HTTP trigger passing in the Customer supplied connection details from Step 4: Obtain credential information. Partner logic apps will use a subset of the following trigger parameters:

    Screenshot of the trigger parameters.

    • Client ID and client secret: Credentials that the user enters in the credentials page from within Microsoft Sustainability Manager. These parameters can differ based on how you authenticate your data source. For example, instead of client ID and secret, you can use username, password, or API key. Replace them with the credentials that you identified in the previous step.

    • Last refresh time: This parameter is a timestamp in UTC timezone and is used for incremental refreshes. The logic app should only pull and process data where dataCreateTime >= lastRefreshTime.

  2. InitializeContinuationToken: This action initializes the ContinuationToken variable to null. This variable will then be used to exit the until loop, which is pulls data in a paginated way.

  3. Try: Any logic to pull and transform data should be inside the try block. If any step inside the try block fails, the execution flow will be skipped to the catch block, where the Microsoft team will add logic to handle error scenarios. The following image shows the actions inside the try block.

    Screenshot of the try block.

The get token action uses the credentials from the user to get the access token and use it to call the partner endpoint and connect to the data source to get user data. Depending on data size, data from your data source should be accessed in a paginated way to avoid any timeouts. The next step will map your data to the Microsoft Sustainability Manager data model.

Screenshot of mapping data to the Microsoft Sustainability Manager data model.

This template uses the Purchased Electricity entity as an example to demonstrate the mapping logic, but you'll need to map the data to the entity that your connector supports. After the data is mapped, you'll store the data array in a variable and send the data to Microsoft Sustainability Manager in a POST request. PostDataToMSM is an example of how the HTTP request should be constructed.

You can either send data to Microsoft Sustainability Manager in multiple batches by making multiple POST requests or at once depending on data size. The Complete Ingestion action shows how the last request to Microsoft Sustainability Manager should look. Microsoft calls the same endpoint and sets the isIngestionComplete variable to true. After this request, Microsoft Sustainability Manager will start processing the data to ingest it into the customer's environment.

The following image provides an example of how the data records should be structured:

Screenshot of how data records should be structured.

Step 6: Submit the connector files

You must add all the files created in the previous steps in a private GitHub repository and add your Microsoft team as a contributor. The root folder of the repository should be a version number (for example, v1.0) and the following files should be in the folder:

  • logic_apps_armtemplate.json
  • credentials_info.csv
  • connector_definition.csv
  • provider_definition.csv
  • company_logo.svg
  • connector_logo.svg

Your Microsoft team will then begin the process of validating the logic app and adding more actions to complete the ingestion process. After the validation step, Microsoft will create a Microsoft Sustainability Manager environment with the operational connector for the partner to test the functionality. Microsoft will also conduct its own functional testing before launching the capability for the users in preview or general availability.

Provider or partner support requirements

The partner owns the upkeep and operation of the data source. We require a contact person with a Service Level Agreement (SLA) that the Microsoft team can reach out to during outages and solution-related errors. The following scenarios are examples that will require partner support:

  • Data model change: One of the most critical steps in building the integration with Microsoft Sustainability Manager is mapping the partner solution data model with the sustainability data model. If the partner solution data model changes, the partner will need to work with Microsoft Sustainability Manager team to remap the data model or provide an updated ARM template and ensure that the required data attributes can be obtained from their solution.

  • Credential management: The primary mechanism to access the partner solution within Microsoft Sustainability Manager is through the credentials from the partner solution. If the users aren't able to authenticate their credentials within Microsoft Sustainability Manager, the partner will provide support to authenticate and update user credentials.

  • Solution not operational: The partner will also need to assist Microsoft Sustainability Manager and its users in situations when their solution isn't operational or functional. Microsoft Sustainability Manager will work with the partner to determine the cause and resolution timelines so that it can communicate to its end users.

  • Solution updates: The partner must provide prior notification before any major updates along with potential impact to user access within Microsoft Sustainability Manager. Prior to integrating a solution within Microsoft Sustainability Manager, the partner will provide tentative timelines for their solution update cadence along with supporting documentation on how it will extend support to Microsoft Sustainability Manager and its users in the event they aren't able to access the provider solution and import data.

  • Test environment: The partner must provide a test environment for Microsoft Sustainability Manager to test capabilities before rolling them out to its customers.

Update scenarios

For any updates to the logic app or connector configuration, the partner must create a new folder in the root of the repository with the new version number (for example, v2.0) and add all the files in that folder. The partner must also include a readme.md file in the folder with a brief explanation of what was updated. After Microsoft receives the updated files, we'll follow the same process as we do for publishing the first version of the connector. After the updated logic app is released to the end user, the logic apps for existing connections will be deprovisioned and the latest version will be deployed. This process is handled by our system by comparing the version numbers in the ARM template.

Compliance requirements

Microsoft product features and services must go through privacy, security, and compliance reviews before they're moved into production. Microsoft team will own and manage this process, but will require coordination with the partner team to gather required information on their solution being integrated with Microsoft Sustainability Manager.

See also

Import data
Azure Logic Apps documentation
Microsoft Sustainability Manager
Microsoft Cloud for Sustainability data model