Edit

Tutorial: Sample steps to perform a manifest-based file ingestion

  • Article

Manifest ingestion provides the capability to ingest manifests into Azure Data Manager for Energy instance

In this tutorial, you will learn how to:

  • Ingest sample manifests into the Azure Data Manager for Energy instance using Postman
  • Search for storage metadata records created during the manifest ingestion using Postman

Prerequisites

Before beginning this tutorial, the following prerequisites must be completed:

Get Azure Data Manager for Energy instance details

  • Azure Data Manager for Energy instance is created already. If not, follow the steps outlined in Quickstart: Create an Azure Data Manager for Energy instance

  • For this tutorial, you will need the following parameters:

    Parameter Value to use Example Where to find these values?
    CLIENT_ID Application (client) ID 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx App ID or Client_ID used when registering the application with the Microsoft Identity Platform. See Register an application
    CLIENT_SECRET Client secrets _fl****************** Sometimes called an application password, a client secret is a string value your app can use in place of a certificate to identity itself. See Add a client secret
    TENANT_ID Directory (tenant) ID 72f988bf-86f1-41af-91ab-xxxxxxxxxxxx Hover over your account name in the Azure portal to get the directory or tenant ID. Alternately, search and select Azure Active Directory > Properties > Tenant ID in the Azure portal.
    SCOPE Application (client) ID 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx Same as App ID or Client_ID mentioned above
    refresh_token Refresh Token value 0.ATcA01-XWHdJ0ES-qDevC6r........... Follow the How to Generate a Refresh Token to create a refresh token and save it. This refresh token is required later to generate a user token.
    DNS URI <instance>.energy.Azure.com Overview page of Azure Data Manager for Energy instance
    data-partition-id Data Partition(s) <instance>-<data-partition-name> Overview page of Azure Data Manager for Energy instance

  • Follow the Manage users guide to add appropriate entitlements for the user running this tutorial

Set up Postman and execute requests

  • Download and install Postman desktop app

  • Import the following files into Postman:

  • Update the CURRENT_VALUE of the postman environment with the information obtained in Azure Data Manager for Energy instance details

  • The Postman collection for manifest ingestion contains multiple requests, which will have to be executed in a sequential manner.

  • Make sure to choose the Ingestion Workflow Environment before triggering the Postman collection. Screenshot of the Postman environment.

  • Each request can be triggered by clicking the Send Button.

  • On every request, Postman will validate the actual API response code against the expected response code; if there is any mismatch the test section will indicate failures.

Successful Postman request

Screenshot of a successful Postman call.

Failed Postman request

Screenshot of a failure Postman call.

Ingest sample manifests into the Azure Data Manager for Energy instance using Postman

  1. Get a user token - Generate the User token, which will be used to authenticate further API calls.
  2. Create a legal tag - Create a legal tag that will be added to the Manifest data for data compliance purpose
  3. Get a signed url for uploading a file - Get the signed URL path to which the Manifest file will be uploaded
  4. Upload a file - Download the sample Wellbore.csv to your local machine (it could be any filetype - CSV, LAS, JSON, etc.), and select this file in Postman by clicking the Select File option as shown in the Screenshot below. Screenshot of a select file option.
  5. Upload file metadata - Upload the file metadata information such as file location & other relevant fields
  6. Get the file metadata - Call to validate if the metadata got created successfully
  7. Ingest Master, Reference and Work Product Component (WPC) data - Ingest the Master, Reference and Work Product Component manifest metadata.
  8. Get manifest ingestion workflow status - The workflow will start and will be in the running state. Keep querying until it changes state to finished (typically 20-30 seconds)

Search for storage metadata records created during the manifest ingestion using Postman

  • Search Work Products - Call Search service to retrieve the Work Product metadata records
  • Search Work Product Components - Call Search service to retrieve the Work Product Component metadata records
  • Search for Dataset - Call Search service to retrieve the Dataset metadata records
  • Search for Master data - Call Search service to retrieve the Master metadata records
  • Search for Reference data - Call Search service to retrieve the Reference metadata records

Next steps