Edit

Tutorial: Perform CSV parser ingestion

  • Article

Comma-separated values (CSV) parser ingestion provides the capability to ingest CSV files into an Azure Data Manager for Energy instance.

In this tutorial, you learn how to:

  • Ingest a sample wellbore data CSV file into an Azure Data Manager for Energy instance by using Postman.
  • Search for storage metadata records created during CSV ingestion by using Postman.

Prerequisites

Before you start this tutorial, complete the following prerequisites.

Get details for the Azure Data Manager for Energy instance

  • You need an Azure Data Manager for Energy instance. If you don't already have one, create one by following the steps in Quickstart: Create an Azure Data Manager for Energy instance.

  • For this tutorial, you need the following parameters:

    Parameter Value to use Example Where to find this value
    CLIENT_ID Application (client) ID 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx You use this app or client ID when registering the application with the Microsoft identity platform. See Register an application.
    CLIENT_SECRET Client secrets _fl****************** Sometimes called an application password, a client secret is a string value that your app can use in place of a certificate to identity itself. See Add a client secret.
    TENANT_ID Directory (tenant) ID 72f988bf-86f1-41af-91ab-xxxxxxxxxxxx Hover over your account name in the Azure portal to get the directory or tenant ID. Alternately, search for and select Microsoft Entra ID > Properties > Tenant ID in the Azure portal.
    SCOPE Application (client) ID 3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx This value is the same as the app or client ID mentioned earlier.
    refresh_token Refresh token value 0.ATcA01-XWHdJ0ES-qDevC6r........... Follow How to generate auth token to create a refresh token and save it. You need this refresh token later to generate a user token.
    DNS URI <instance>.energy.Azure.com Find this value on the overview page of the Azure Data Manager for Energy instance.
    data-partition-id Data partitions <instance>-<data-partition-name> Find this value on the overview page of the Azure Data Manager for Energy instance.

Follow the Manage users guide to add appropriate entitlements for the user who's running this tutorial.

Set up Postman and execute requests

  1. Download and install the Postman desktop app.

  2. Import the following files into Postman:

    To import the Postman collection and environment variables, follow the steps in Importing data into Postman.

  3. Update CURRENT VALUE for the Postman environment with the information that you obtained in the details of the Azure Data Manager for Energy instance.

  4. The Postman collection for CSV parser ingestion contains 10 requests that you must execute sequentially.

    Be sure to choose Ingestion Workflow Environment before you trigger the Postman collection.

    Screenshot of the Postman environment.

  5. Trigger each request by selecting the Send button.

    On every request, Postman validates the actual API response code against the expected response code. If there's any mismatch, the test section indicates failures.

Here's an example of a successful Postman request:

Screenshot of a successful Postman call.

Here's an example of a failed Postman request:

Screenshot of a failed Postman call.

Ingest wellbore data by using Postman

To ingest a sample wellbore data CSV file into the Azure Data Manager for Energy instance by using the Postman collection, complete the following steps:

  1. Get a User Access Token: Generate the user token, which will be used to authenticate further API calls.

  2. Create a Schema: Generate a schema that adheres to the columns present in the CSV file.

  3. Get Schema details: Get the schema created in the previous step and validate it.

  4. Create a Legal Tag: Create a legal tag that will be added to the CSV data for data compliance purposes.

  5. Get a signed URL for uploading a CSV file: Get the signed URL path to which the CSV file will be uploaded.

  6. Upload a CSV file: Download the Wellbore.csv sample to your local machine, and then select this file in Postman by clicking the Select File button.

    Screenshot of uploading a CSV file.

  7. Upload CSV file metadata: Upload the file metadata information, such as file location and other relevant fields.

  8. Create a CSV Parser Ingestion Workflow: Create the directed acyclic graph (DAG) for the CSV parser ingestion workflow.

  9. Trigger a CSV Parser Ingestion Workflow: Trigger the DAG for the CSV parser ingestion workflow.

  10. Search for ingested CSV Parser Ingestion Workflow status: Get the status of the CSV parser's DAG run.

Search for ingested wellbore data by using Postman

To search for the storage metadata records created during the CSV ingestion by using the Postman collection, complete the following step:

  • Search for ingested CSV records: Search for the CSV records created earlier.

    Screenshot of searching ingested CSV records.

Next step

Advance to the next tutorial:

Tutorial: Perform manifest-based file ingestion