Edit

Tutorial: Sample steps to perform a manifest-based file ingestion

  • Article
  • 4 minutes to read

Manifest ingestion provides the capability to ingest manifests into Azure Data Manager for Energy Preview instance

In this tutorial, you will learn how to:

  • Ingest sample manifests into the Azure Data Manager for Energy Preview instance using Postman
  • Search for storage metadata records created during the manifest ingestion using Postman

Important

Microsoft Azure Data Manager for Energy Preview is currently in preview. For legal terms that apply to features that are in beta, in preview, or otherwise not yet released into general availability, see the Supplemental Terms of Use for Microsoft Azure Previews.

Azure Data Manager for Energy Preview is now available to all customers and partners during the preview period. Get started with by creating an instance of the product here.

Prerequisites

Before beginning this tutorial, the following prerequisites must be completed:

Get Azure Data Manager for Energy Preview instance details

  • Azure Data Manager for Energy Preview instance is created already. If not, follow the steps outlined in Quickstart: Create an Azure Data Manager for Energy Preview instance

  • For this tutorial, you will need the following parameters:

    Parameter Value to use Example Where to find these values?
    CLIENT_ID Application (client) ID 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx App ID or Client_ID used when registering the application with the Microsoft Identity Platform. See Register an application
    CLIENT_SECRET Client secrets _fl****************** Sometimes called an application password, a client secret is a string value your app can use in place of a certificate to identity itself. See Add a client secret
    TENANT_ID Directory (tenant) ID 72f988bf-86f1-41af-91ab-xxxxxxxxxxxx Hover over your account name in the Azure portal to get the directory or tenant ID. Alternately, search and select Azure Active Directory > Properties > Tenant ID in the Azure portal.
    SCOPE Application (client) ID 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx Same as App ID or Client_ID mentioned above
    refresh_token Refresh Token value 0.ATcA01-XWHdJ0ES-qDevC6r........... Follow the How to Generate a Refresh Token to create a refresh token and save it. This refresh token is required later to generate a user token.
    DNS URI <instance>.energy.Azure.com Overview page of Azure Data Manager for Energy Preview instance
    data-partition-id Data Partition(s) <instance>-<data-partition-name> Overview page of Azure Data Manager for Energy Preview instance

  • Follow the Manage users guide to add appropriate entitlements for the user running this tutorial

Set up Postman and execute requests

  • Download and install Postman desktop app
  • Import the following files into Postman:
  • Update the CURRENT_VALUE of the postman environment with the information obtained in Get Azure Data Manager for Energy Preview instance details
  • The Postman collection for manifest ingestion contains multiple requests, which will have to be executed in a sequential manner.
  • Make sure to choose the Ingestion Workflow Environment before triggering the Postman collection. Screenshot of the Postman environment.
  • Each request can be triggered by clicking the Send Button.
  • On every request, Postman will validate the actual API response code against the expected response code; if there is any mismatch the test section will indicate failures.

Successful Postman request

Screenshot of a successful Postman call.

Failed Postman request

Screenshot of a failure Postman call.

Ingest sample manifests into the Azure Data Manager for Energy Preview instance using Postman

  1. Get a user token - Generate the User token, which will be used to authenticate further API calls.
  2. Create a legal tag - Create a legal tag that will be added to the Manifest data for data compliance purpose
  3. Get a signed url for uploading a file - Get the signed URL path to which the Manifest file will be uploaded
  4. Upload a file - Download the sample Wellbore.csv to your local machine (it could be any filetype - CSV, LAS, JSON, etc.), and select this file in Postman by clicking the Select File option as shown in the Screenshot below. Screenshot of a select file option.
  5. Upload file metadata - Upload the file metadata information such as file location & other relevant fields
  6. Get the file metadata - Call to validate if the metadata got created successfully
  7. Ingest Master, Reference and Work Product Component (WPC) data - Ingest the Master, Reference and Work Product Component manifest metadata.
  8. Get manifest ingestion workflow status - The workflow will start and will be in the running state. Keep querying until it changes state to finished (typically 20-30 seconds)

Search for storage metadata records created during the manifest ingestion using Postman

  • Search Work Products - Call Search service to retrieve the Work Product metadata records
  • Search Work Product Components - Call Search service to retrieve the Work Product Component metadata records
  • Search for Dataset - Call Search service to retrieve the Dataset metadata records
  • Search for Master data - Call Search service to retrieve the Master metadata records
  • Search for Reference data - Call Search service to retrieve the Reference metadata records

Next steps

Advance to the next tutorial to learn about sdutil

Tutorial: Seismic store sdutil