Επεξεργασία

Κοινή χρήση μέσω

Deploy Geospatial Consumption Zone

  • Άρθρο

This guide shows you how to deploy the Geospatial Consumption Zone (GCZ) service integrated with Azure Data Manager for Energy (ADME).

Important

While the Geospatial Consumption Zone (GCZ) service is a graduated service in the OSDU Forum, it has limitations in terms of security and usage. We will deploy some additional services and policies to secure the environment, but encourage you to follow the service's development on the OSDU Gitlab.

Description

The OSDU Geospatial Consumption Zone (GCZ) is a service that enables enhanced management and utilization of geospatial data. The GCZ streamlines the handling of location-based information. It abstracts away technical complexities, allowing software applications to access geospatial data without needing to deal with intricate details. By providing ready-to-use map services, the GCZ facilitates seamless integration with OSDU-enabled applications.

Create an App Registration in Microsoft Entra ID

To deploy the GCZ, you need to create an App Registration in Microsoft Entra ID. The App Registration is to authenticate the GCZ APIs with Azure Data Manager for Energy to be able to generate the cache of the geospatial data.

  1. See Create an App Registration in Microsoft Entra ID for instructions on how to create an App Registration.
  2. Grant the App Registration permission to read the relevant data in Azure Data Manager for Energy. See How to add members to an OSDU group for further instructions.

Setup

There are two main deployment options for the GCZ service:

  • Azure Kubernetes Service (AKS): Deploy the GCZ service on an AKS cluster. This deployment option is recommended for production environments. It requires more setup, configuration, and maintenance. It also has some limitations in the provided container images.
  • Windows: Deploy the GCZ service on a Windows. This deployment option recommended for development and testing environments, as it's easier to set up and configure, and requires less maintenance.

Deploy Geospatial Consumption Zone (GCZ) on Azure Kubernetes Service (AKS)

Learn how to deploy Geospatial Consumption Zone (GCZ) on Azure Kubernetes Service (AKS).

Important

The current deployment of GCZ using AKS is limited to a default configuration of included schemas, please see OSDU GitLab for information regarding the supported schemas. To add or change schemas (i.e. newer versions) a custom container image will need to be created.

Prerequisites

Deploy Geospatial Consumption Zone (GCZ) HELM Chart

  1. Clone the GCZ repository to your local environment:

    git clone https://community.opengroup.org/osdu/platform/consumption/geospatial.git

  2. Change directory to the geospatial folder:

    cd geospatial/devops/azure/charts/geospatial

  3. Define variables for the deployment:

    # Define the variables for Azure Data Manager for Energy
AZURE_DNS_NAME="<instanceName>.energy.azure.com"  # Example: demo.energy.azure.com
DATA_PARTITION_ID="<dataPartitionId>" # Data partition ID. Example: opendes
AZURE_TENANT_ID="<tenantId>" # Entra ID tenant ID. Example: 557963fb-ede7-4a88-9e3e-19ace7f1e36b 
AZURE_CLIENT_ID="<clientId>" # App Registration client ID. Example: b149dc73-ed8c-4ad3-bbaf-882a208f87eb
AZURE_CLIENT_SECRET="<clientSecret>" # App Registration client secret.
CALLBACK_URL="http://localhost:5050" #ie: http://localhost:8080

# Define the variables for AKS
AKS_NAME="<aksName>" # Name of the AKS cluster. Example: gcz-aks-cluster.
RESOURCE_GROUP="<resourceGroupName>" # Name of the resource group. Example: gcz-rg.
NAMESPACE="ignite" # Name of the AKS namespace you want to deploy to. We recommend to leave it default.
GCZ_IGNITE_SERVICE="ignite-service" # Name of the ignite service. We recommend to leave it default.
GCZ_IGNITE_NAMESPACE=$NAMESPACE
CHART=osdu-gcz-service
VERSION=0.1.0

  4. Create the HELM chart:

    cat > osdu_gcz_custom_values.yaml << EOF
# This file contains the essential configs for the gcz on azure helm chart

################################################################################
# Specify the values for each service.
#
global:
ignite:
    namespace: $NAMESPACE
    name: ignite
    image:
    name: community.opengroup.org:5555/osdu/platform/consumption/geospatial/gridgain-community
    tag: 8.8.34
    configuration:
    gcz_ignite_namespace: "$GCZ_IGNITE_NAMESPACE"
    gcz_ignite_service: "$GCZ_IGNITE_SERVICE"
provider:
    namespace: $NAMESPACE
    image:
    repository: community.opengroup.org:5555
    name: osdu/platform/consumption/geospatial/geospatial-provider-master
    tag: latest
    service:
    type: LoadBalancer
    annotations:
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
transformer:
    namespace: $NAMESPACE
    image:
    repository: community.opengroup.org:5555
    name: osdu/platform/consumption/geospatial/geospatial-transformer-master
    tag: latest
    service:
    type: LoadBalancer
    annotations:
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    configuration:
    datapartitionid: $DATA_PARTITION_ID
    clientId: $AZURE_CLIENT_ID
    tenantId: $AZURE_TENANT_ID
    callbackURL: $CALLBACK_URL
    searchQueryURL: "https://$AZURE_DNS_NAME/api/search/v2/query"
    searchCursorURL: "https://$AZURE_DNS_NAME/api/search/v2/query_with_cursor"
    schemaURL: "https://$AZURE_DNS_NAME/api/schema-service/v1/schema"
    entitlementsURL: "https://$AZURE_DNS_NAME/api/entitlements/v2"
    fileRetrievalURL: "https://$AZURE_DNS_NAME/api/dataset/v1/retrievalInstructions"
    crsconvertorURL: "https://$AZURE_DNS_NAME/api/crs/converter/v3/convertTrajectory"
    storageURL: "https://$AZURE_DNS_NAME/api/storage/v2/records"
    clientSecret: $(echo "$AZURE_CLIENT_SECRET" | base64)
    gcz_ignite_namespace: "$GCZ_IGNITE_NAMESPACE"
    gcz_ignite_service: "$GCZ_IGNITE_SERVICE"   
EOF

  5. Change service type to LoadBalancer for the provider and transformer services configuration files.

    cat > ../provider/templates/service.yaml << EOF
apiVersion: v1
kind: Service
metadata:
    name: gcz-provider
    namespace: {{ $.Values.global.provider.namespace }}
    annotations:
        {{- range $key, $value := $.Values.global.provider.service.annotations }}
        {{ $key }}: {{ $value | quote }}
        {{- end }}
spec:
    selector:
        app: provider
    ports:
    - port: 80
        protocol: TCP
        targetPort: 8083
    type: {{ $.Values.global.provider.service.type }}
EOF

cat > ../transformer/templates/service.yaml << EOF
apiVersion: v1
kind: Service
metadata:
    name: gcz-transformer
    namespace: {{ $.Values.global.transformer.namespace }}
    annotations:
        {{- range $key, $value := $.Values.global.transformer.service.annotations }}
        {{ $key }}: {{ $value | quote }}
        {{- end }}
spec:
    selector:
        app: transformer
    ports:
    - port: 80
        protocol: TCP
        targetPort: 8080
    type: {{ $.Values.global.transformer.service.type }}
EOF

  6. Authenticate to the Azure Kubernetes Service (AKS) cluster:

    az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_NAME --admin

  7. Deploy HELM dependencies:

    helm dependency build

  8. Deploy the GCZ HELM chart:

    helm install $CHART ../$CHART --values osdu_gcz_custom_values.yaml

  9. Verify the deployment:

    kubectl get pods -n $NAMESPACE

    Now you should see the pods for the ignite, provider, and transformer services.

  10. Next get note the External IPs for the provider and transformer services.

    kubectl get service -n $NAMESPACE

    These IPs are used to connect to the GCZ API endpoints.

Deploy Geospatial Consumption Zone (GCZ) on a Windows Virtual Machine

Learn how to deploy Geospatial Consumption Zone (GCZ) on Windows. This deployment option is recommended for development and testing environments, as it's easier to set up and configure, and requires less maintenance.

Prerequisites

Deploy GCZ on Windows

  1. Connect to your Windows Virtual Machine.

  2. Download the following files from the OSDU GitLab repository:

    1. GCZ Provider
    2. GCZ Transformer
    3. Python dependencies

  3. Open PowerShell as an administrator and navigate to the folder where you downloaded the files.

  4. Run the following commands to extract the files:

    Expand-Archive -Path .\GCZ_PROVIDER.zip -DestinationPath C:\gcz\
Expand-Archive -Path .\GCZ_TRANSFORMER.zip -DestinationPath C:\gcz\
Expand-Archive -Path .\GCZ_PYTHON_DEPENDENCIES.zip -DestinationPath C:\gcz\

  5. Configure the environment variables:

    $ADME_HOSTNAME = "<adme-hostname>" # ADME Hostname, e.g. "https://contoso.energy.azure.com"
$GCZ_DATA_PARTITION_ID = "<data-partition-id>" # ADME Data Partition ID, e.g. "opendes"
$GCZ_QUERY_URL = "$ADME_HOSTNAME/api/search/v2/query" # ADME Query Endpoint
$GCZ_QUERY_CURSOR_URL = "$ADME_HOSTNAME/api/search/v2/query_with_cursor" # ADME Query with Cursor Endpoint
$GCZ_SCHEMA_URL = "$ADME_HOSTNAME/api/schema-service/v1/schema" # ADME Schema Endpoint
$GCZ_ENTITLEMENT_SERVICE_URL = "$ADME_HOSTNAME/api/entitlements/v2" # ADME Entitlement Service Endpoint
$GCZ_FILE_RETRIEVAL_URL = "$ADME_HOSTNAME/api/dataset/v1/retrievalInstructions" # ADME File Retrieval Endpoint
$GCZ_CONVERT_TRAJECTORY_URL = "$ADME_HOSTNAME/api/crs/converter/v3/convertTrajectory" # ADME Convert Trajectory Endpoint
$GCZ_STORAGE_URL = "$ADME_HOSTNAME/api/storage/v2/records/" # ADME Storage Endpoint

    For more environment variables, see the OSDU GitLab documentation.

  6. Validate the configuration files for the GCZ Provider and Transformer by opening the configuration files in a text editor and updating the values if needed.

    • Provider: C:\gcz\gcz-provider\gcz-provider-core\config\koop-config.json
    • Transformer: C:\gcz\gcz-transformer-core\config\application.yml

    Important

    If you make changes to the schemas in the configuration files, you have to make sure that those schemas are represented in both of the configuration files.

  7. (optional) Install Python Dependencies (only required for Well Log Interpolation).

    pip install -r C:\gcz\gcz-transformer-core\src\main\resources\script\requirements.txt --no-index --find-links python-dependencies

  8. Start the GCZ Transformer.

    C:\gcz\transformer\transformer.bat local

  9. Build the GCZ Provider.

    cd C:\gcz\gcz-provider\gcz-provider-core
npm install
npm start

By default the Provider is listening on http://localhost:8083 and the Transformer is listening on http://localhost:8080.

Publish GCZ APIs publicly (optional)

If you want to expose the GCZ APIs publicly, you can use Azure API Management (APIM). Azure API Management allows us to securely expose the GCZ service to the internet, as the GCZ service doesn't yet have authentication and authorization built in. Through APIM we can add policies to secure, monitor, and manage the APIs.

Prerequisites

Important

The Azure API Management instance will need to be injected into a virtual network that is routable to the AKS cluster to be able to communicate with the GCZ API's.

Add the GCZ APIs to Azure API Management

Download the GCZ OpenAPI specifications

  1. Download the two OpenAPI specification to your local computer.

  2. Open each OpenAPI specification file in a text editor and replace the servers section with the corresponding IPs of the AKS GCZ Services' Load Balancer (External IP).

    servers:
- url: "http://<GCZ-Service-External-IP>/ignite-provider"

Add the GCZ APIs to Azure API Management

  1. Navigate to your Azure API Management service in the Azure portal.

  2. In the left-hand navigation pane, select APIs.

  3. Select + Add API.

  4. Select OpenAPI.

  5. Select Select a file and upload the gcz-openapi-provider.yaml file.

  6. In the API URL suffix field, enter ignite-provider.

  7. Select Create.

  8. Repeat the steps for the gcz-openapi-transformer.yaml file, but use gcz/transformer/admin as the API URL suffix.

    Add GCZ API to APIM

Configure policies

Next we need to configure the policies to validate the JSON Web Tokens (JWT).

You need the following information:

  • Your Microsoft Entra ID tenant ID.
  • The Azure Data Manager for Energy client ID (or token-issuing client ID if separate).

Note

If you have multiple App Registrations issuing tokens, you can add multiple <application-id> elements to the <client-application-ids> element.

  1. In the newly created Geospatial Consumption Zone - Provider API, make sure All operations is selected.

  2. Under Inbound processing, select ... and then Code editor.

  3. Paste the following policy definition in the editor:

    <policies>
    <!-- Throttle, authorize, validate, cache, or transform the requests -->
    <inbound>
        <base />
        <validate-azure-ad-token tenant-id="%tenant-id%" failed-validation-httpcode="401">
        <client-application-ids>
            <application-id>%client-id%</application-id>
        </client-application-ids>
    </inbound>
    <!-- Control if and how the requests are forwarded to services  -->
    <backend>
        <base />
    </backend>
    <!-- Customize the responses -->
    <outbound>
        <base />
    </outbound>
    <!-- Handle exceptions and customize error responses  -->
    <on-error>
        <base />
    </on-error>
</policies>

  4. Replace %tenant-id% with your Microsoft Entra ID tenant ID, and %client-id% with the Azure Data Manager for Energy client ID.

  5. Select Save.

  6. Repeat the steps for the Geospatial Consumption Zone - Transformer API.

Testing the GCZ service

  1. Download the API client collection from the OSDU GitLab and import it into your API client of choice (for example, Postman).

  2. Add the following environment variables to your API client:

    • PROVIDER_URL - The URL to the GCZ Provider API.
    • AMBASSADOR_URL - The URL to the GCZ Transformer API.
    • access_token - A valid ADME access token.

  3. To verify that the GCZ is working as expected, run the API calls in the collection.

Next steps

After you have a successful deployment of GCZ, you can:

  • Visualize your GCZ data using the GCZ WebApps from the OSDU GitLab.

Important

The GCZ WebApps are currently in development and does not support authentication. We recommend deploying the WebApps in a private network and exposing them using Azure Application Gateway or Azure Front Door to enable authentication and authorization.

You can also ingest data into your Azure Data Manager for Energy instance:

References

  • For information about Geospatial Consumption Zone, see OSDU GitLab.