Tutorial: Perform CSV parser ingestion

Comma-separated values (CSV) parser ingestion provides the capability to ingest CSV files into an Azure Data Manager for Energy instance.

In this tutorial, you learn how to:

• Ingest a sample wellbore data CSV file into an Azure Data Manager for Energy instance by using Postman.
• Search for storage metadata records created during CSV ingestion by using Postman.

Prerequisites

Before you start this tutorial, complete the following prerequisites.

Get details for the Azure Data Manager for Energy instance

• You need an Azure Data Manager for Energy instance. If you don't already have one, create one by following the steps in Quickstart: Create an Azure Data Manager for Energy instance.

• For this tutorial, you need the following parameters:

  Parameter	Value to use	Example	Where to find this value
  CLIENT_ID	Application (client) ID	3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx	You use this app or client ID when registering the application with the Microsoft identity platform. See Register an application.
  CLIENT_SECRET	Client secrets	_fl******************	Sometimes called an application password, a client secret is a string value that your app can use in place of a certificate to identity itself. See Add a client secret.
  TENANT_ID	Directory (tenant) ID	72f988bf-86f1-41af-91ab-xxxxxxxxxxxx	Hover over your account name in the Azure portal to get the directory or tenant ID. Alternately, search for and select Microsoft Entra ID > Properties > Tenant ID in the Azure portal.
  SCOPE	Application (client) ID	3dbbbcc2-f28f-44b6-a5ab-xxxxxxxxxxxx	This value is the same as the app or client ID mentioned earlier.
  refresh_token	Refresh token value	0.ATcA01-XWHdJ0ES-qDevC6r...........	Follow How to generate auth token to create a refresh token and save it. You need this refresh token later to generate a user token.
  DNS	URI	<instance>.energy.Azure.com	Find this value on the overview page of the Azure Data Manager for Energy instance.
  data-partition-id	Data partitions	<instance>-<data-partition-name>	Find this value on the overview page of the Azure Data Manager for Energy instance.

Follow the Manage users guide to add appropriate entitlements for the user who's running this tutorial.

Set up Postman and execute requests

1. Download and install the Postman desktop app.

2. Import the following files into Postman:

   • CSV workflow Postman collection
   • CSV workflow Postman environment

   To import the Postman collection and environment variables, follow the steps in Importing data into Postman.

3. Update CURRENT VALUE for the Postman environment with the information that you obtained in the details of the Azure Data Manager for Energy instance.

4. The Postman collection for CSV parser ingestion contains 10 requests that you must execute sequentially.

   Be sure to choose Ingestion Workflow Environment before you trigger the Postman collection.

   

5. Trigger each request by selecting the Send button.

   On every request, Postman validates the actual API response code against the expected response code. If there's any mismatch, the test section indicates failures.

Here's an example of a successful Postman request:

Here's an example of a failed Postman request:

Ingest wellbore data by using Postman

To ingest a sample wellbore data CSV file into the Azure Data Manager for Energy instance by using the Postman collection, complete the following steps:

1. Get a User Access Token: Generate the user token, which will be used to authenticate further API calls.

2. Create a Schema: Generate a schema that adheres to the columns present in the CSV file.

3. Get Schema details: Get the schema created in the previous step and validate it.

4. Create a Legal Tag: Create a legal tag that will be added to the CSV data for data compliance purposes.

5. Get a signed URL for uploading a CSV file: Get the signed URL path to which the CSV file will be uploaded.

6. Upload a CSV file: Download the Wellbore.csv sample to your local machine, and then select this file in Postman by clicking the Select File button.

   

7. Upload CSV file metadata: Upload the file metadata information, such as file location and other relevant fields.

8. Create a CSV Parser Ingestion Workflow: Create the directed acyclic graph (DAG) for the CSV parser ingestion workflow.

9. Trigger a CSV Parser Ingestion Workflow: Trigger the DAG for the CSV parser ingestion workflow.

10. Search for ingested CSV Parser Ingestion Workflow status: Get the status of the CSV parser's DAG run.

Search for ingested wellbore data by using Postman

To search for the storage metadata records created during the CSV ingestion by using the Postman collection, complete the following step:

• Search for ingested CSV records: Search for the CSV records created earlier.

  

Next step

Advance to the next tutorial:

Tutorial: Perform manifest-based file ingestion