Cluster Jobs - Run Job

Service:

HDInsight On Aks

API Version:

2023-06-01-preview

Operations on jobs of HDInsight on AKS cluster.

POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.HDInsight/clusterpools/{clusterPoolName}/clusters/{clusterName}/runJob?api-version=2023-06-01-preview

URI Parameters

Name	In	Required	Type	Description
clusterName	path	True	string	The name of the HDInsight cluster.
clusterPoolName	path	True	string	The name of the cluster pool.
resourceGroupName	path	True	string	The name of the resource group. The name is case insensitive.
subscriptionId	path	True	string uuid	The ID of the target subscription. The value must be an UUID.
api-version	query	True	string	The API version to use for this operation.

Request Body

Name	Required	Type	Description
properties	True	ClusterJobProperties: FlinkJobProperties	Properties of cluster job.

Responses

Name	Type	Description
200 OK	ClusterJob	Successful.
202 Accepted		Run cluster job action request accepted. Headers location: string
Other Status Codes	ErrorResponse	Error Response describing why the operation failed.

Security

azure_auth

Azure Active Directory OAuth2 Flow.

Type: oauth2
Flow: implicit
Authorization URL: https://login.microsoftonline.com/common/oauth2/authorize

Scopes

Name	Description
user_impersonation	impersonate your user account

Examples

RunClusterJob

Sample Request

HTTP

POST https://management.azure.com/subscriptions/10e32bab-26da-4cc4-a441-52b318f824e6/resourceGroups/hiloResourcegroup/providers/Microsoft.HDInsight/clusterpools/clusterpool1/clusters/cluster1/runJob?api-version=2023-06-01-preview

{
  "properties": {
    "jobType": "FlinkJob",
    "jobName": "flink-job-name",
    "action": "START",
    "jobJarDirectory": "abfs://flinkjob@hilosa.dfs.core.windows.net/jars",
    "jarName": "flink-sleep-job-0.0.1-SNAPSHOT.jar",
    "entryClass": "com.microsoft.hilo.flink.job.streaming.SleepJob",
    "flinkConfiguration": {
      "parallelism": "1",
      "savepoint.directory": "abfs://flinkjob@hilosa.dfs.core.windows.net/savepoint"
    }
  }
}

Python

from azure.identity import DefaultAzureCredential
from azure.mgmt.hdinsightcontainers import HDInsightContainersMgmtClient

"""
# PREREQUISITES
    pip install azure-identity
    pip install azure-mgmt-hdinsightcontainers
# USAGE
    python run_cluster_job.py

    Before run the sample, please set the values of the client ID, tenant ID and client secret
    of the AAD application as environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID,
    AZURE_CLIENT_SECRET. For more info about how to get the value, please see:
    https://docs.microsoft.com/azure/active-directory/develop/howto-create-service-principal-portal
"""


def main():
    client = HDInsightContainersMgmtClient(
        credential=DefaultAzureCredential(),
        subscription_id="10e32bab-26da-4cc4-a441-52b318f824e6",
    )

    response = client.cluster_jobs.begin_run_job(
        resource_group_name="hiloResourcegroup",
        cluster_pool_name="clusterpool1",
        cluster_name="cluster1",
        cluster_job={
            "properties": {
                "action": "START",
                "entryClass": "com.microsoft.hilo.flink.job.streaming.SleepJob",
                "flinkConfiguration": {
                    "parallelism": "1",
                    "savepoint.directory": "abfs://flinkjob@hilosa.dfs.core.windows.net/savepoint",
                },
                "jarName": "flink-sleep-job-0.0.1-SNAPSHOT.jar",
                "jobJarDirectory": "abfs://flinkjob@hilosa.dfs.core.windows.net/jars",
                "jobName": "flink-job-name",
                "jobType": "FlinkJob",
            }
        },
    ).result()
    print(response)


# x-ms-original-file: specification/hdinsight/resource-manager/Microsoft.HDInsight/HDInsightOnAks/preview/2023-06-01-preview/examples/RunClusterJob.json
if __name__ == "__main__":
    main()

To use the Azure SDK library in your project, see this documentation. To provide feedback on this code sample, open a GitHub issue

Go

package armhdinsightcontainers_test

import (
	"context"
	"log"

	"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
	"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/hdinsightcontainers/armhdinsightcontainers"
)

// Generated from example definition: https://github.com/Azure/azure-rest-api-specs/blob/7f70e351393addbc31d790a908c994c7c8644d9c/specification/hdinsight/resource-manager/Microsoft.HDInsight/HDInsightOnAks/preview/2023-06-01-preview/examples/RunClusterJob.json
func ExampleClusterJobsClient_BeginRunJob() {
	cred, err := azidentity.NewDefaultAzureCredential(nil)
	if err != nil {
		log.Fatalf("failed to obtain a credential: %v", err)
	}
	ctx := context.Background()
	clientFactory, err := armhdinsightcontainers.NewClientFactory("<subscription-id>", cred, nil)
	if err != nil {
		log.Fatalf("failed to create client: %v", err)
	}
	poller, err := clientFactory.NewClusterJobsClient().BeginRunJob(ctx, "hiloResourcegroup", "clusterpool1", "cluster1", armhdinsightcontainers.ClusterJob{
		Properties: &armhdinsightcontainers.FlinkJobProperties{
			JobType:    to.Ptr(armhdinsightcontainers.JobTypeFlinkJob),
			Action:     to.Ptr(armhdinsightcontainers.ActionSTART),
			EntryClass: to.Ptr("com.microsoft.hilo.flink.job.streaming.SleepJob"),
			FlinkConfiguration: map[string]*string{
				"parallelism":         to.Ptr("1"),
				"savepoint.directory": to.Ptr("abfs://flinkjob@hilosa.dfs.core.windows.net/savepoint"),
			},
			JarName:         to.Ptr("flink-sleep-job-0.0.1-SNAPSHOT.jar"),
			JobJarDirectory: to.Ptr("abfs://flinkjob@hilosa.dfs.core.windows.net/jars"),
			JobName:         to.Ptr("flink-job-name"),
		},
	}, nil)
	if err != nil {
		log.Fatalf("failed to finish the request: %v", err)
	}
	res, err := poller.PollUntilDone(ctx, nil)
	if err != nil {
		log.Fatalf("failed to pull the result: %v", err)
	}
	// You could use response here. We use blank identifier for just demo purposes.
	_ = res
	// If the HTTP response code is 200 as defined in example definition, your response structure would look as follows. Please pay attention that all the values in the output are fake values for just demo purposes.
	// res.ClusterJob = armhdinsightcontainers.ClusterJob{
	// 	Properties: &armhdinsightcontainers.FlinkJobProperties{
	// 		JobType: to.Ptr(armhdinsightcontainers.JobTypeFlinkJob),
	// 		Action: to.Ptr(armhdinsightcontainers.ActionSTART),
	// 		EntryClass: to.Ptr("com.microsoft.hilo.flink.job.streaming.SleepJob"),
	// 		FlinkConfiguration: map[string]*string{
	// 			"parallelism": to.Ptr("1"),
	// 			"savepoint.directory": to.Ptr("abfs://flinkjob@hilosa.dfs.core.windows.net/savepoint"),
	// 		},
	// 		JarName: to.Ptr("flink-sleep-job-0.0.1-SNAPSHOT.jar"),
	// 		JobJarDirectory: to.Ptr("abfs://flinkjob@hilosa.dfs.core.windows.net/jars"),
	// 		JobName: to.Ptr("flink-job-name"),
	// 	},
	// }
}

To use the Azure SDK library in your project, see this documentation. To provide feedback on this code sample, open a GitHub issue

JavaScript

const { HDInsightContainersManagementClient } = require("@azure/arm-hdinsightcontainers");
const { DefaultAzureCredential } = require("@azure/identity");

/**
 * This sample demonstrates how to Operations on jobs of HDInsight on AKS cluster.
 *
 * @summary Operations on jobs of HDInsight on AKS cluster.
 * x-ms-original-file: specification/hdinsight/resource-manager/Microsoft.HDInsight/HDInsightOnAks/preview/2023-06-01-preview/examples/RunClusterJob.json
 */
async function runClusterJob() {
  const subscriptionId =
    process.env["HDINSIGHT_SUBSCRIPTION_ID"] || "10e32bab-26da-4cc4-a441-52b318f824e6";
  const resourceGroupName = process.env["HDINSIGHT_RESOURCE_GROUP"] || "hiloResourcegroup";
  const clusterPoolName = "clusterpool1";
  const clusterName = "cluster1";
  const clusterJob = {
    properties: {
      action: "START",
      entryClass: "com.microsoft.hilo.flink.job.streaming.SleepJob",
      flinkConfiguration: {
        parallelism: "1",
        savepointDirectory: "abfs://flinkjob@hilosa.dfs.core.windows.net/savepoint",
      },
      jarName: "flink-sleep-job-0.0.1-SNAPSHOT.jar",
      jobJarDirectory: "abfs://flinkjob@hilosa.dfs.core.windows.net/jars",
      jobName: "flink-job-name",
      jobType: "FlinkJob",
    },
  };
  const credential = new DefaultAzureCredential();
  const client = new HDInsightContainersManagementClient(credential, subscriptionId);
  const result = await client.clusterJobs.beginRunJobAndWait(
    resourceGroupName,
    clusterPoolName,
    clusterName,
    clusterJob
  );
  console.log(result);
}

To use the Azure SDK library in your project, see this documentation. To provide feedback on this code sample, open a GitHub issue

dotnet

using System;
using System.Threading.Tasks;
using Azure;
using Azure.Core;
using Azure.Identity;
using Azure.ResourceManager;
using Azure.ResourceManager.HDInsight.Containers;
using Azure.ResourceManager.HDInsight.Containers.Models;

// Generated from example definition: specification/hdinsight/resource-manager/Microsoft.HDInsight/HDInsightOnAks/preview/2023-06-01-preview/examples/RunClusterJob.json
// this example is just showing the usage of "ClusterJobs_RunJob" operation, for the dependent resources, they will have to be created separately.

// get your azure access token, for more details of how Azure SDK get your access token, please refer to https://learn.microsoft.com/en-us/dotnet/azure/sdk/authentication?tabs=command-line
TokenCredential cred = new DefaultAzureCredential();
// authenticate your client
ArmClient client = new ArmClient(cred);

// this example assumes you already have this HDInsightClusterResource created on azure
// for more information of creating HDInsightClusterResource, please refer to the document of HDInsightClusterResource
string subscriptionId = "10e32bab-26da-4cc4-a441-52b318f824e6";
string resourceGroupName = "hiloResourcegroup";
string clusterPoolName = "clusterpool1";
string clusterName = "cluster1";
ResourceIdentifier hdInsightClusterResourceId = HDInsightClusterResource.CreateResourceIdentifier(subscriptionId, resourceGroupName, clusterPoolName, clusterName);
HDInsightClusterResource hdInsightCluster = client.GetHDInsightClusterResource(hdInsightClusterResourceId);

// invoke the operation
ClusterJob clusterJob = new ClusterJob(new FlinkJobProperties("flink-job-name")
{
    JobJarDirectory = "abfs://flinkjob@hilosa.dfs.core.windows.net/jars",
    JarName = "flink-sleep-job-0.0.1-SNAPSHOT.jar",
    EntryClass = "com.microsoft.hilo.flink.job.streaming.SleepJob",
    Action = FlinkJobAction.Start,
    FlinkConfiguration =
    {
    ["parallelism"] = "1",
    ["savepoint.directory"] = "abfs://flinkjob@hilosa.dfs.core.windows.net/savepoint",
    },
});
ArmOperation<ClusterJob> lro = await hdInsightCluster.RunJobClusterJobAsync(WaitUntil.Completed, clusterJob);
ClusterJob result = lro.Value;

Console.WriteLine($"Succeeded: {result}");

To use the Azure SDK library in your project, see this documentation. To provide feedback on this code sample, open a GitHub issue

Sample Response

Status code:

202

location: http://azure.async.operation/status

Status code:

200

{
  "properties": {
    "jobType": "FlinkJob",
    "jobName": "flink-job-name",
    "action": "START",
    "jobJarDirectory": "abfs://flinkjob@hilosa.dfs.core.windows.net/jars",
    "jarName": "flink-sleep-job-0.0.1-SNAPSHOT.jar",
    "entryClass": "com.microsoft.hilo.flink.job.streaming.SleepJob",
    "flinkConfiguration": {
      "parallelism": "1",
      "savepoint.directory": "abfs://flinkjob@hilosa.dfs.core.windows.net/savepoint"
    }
  }
}

Definitions

Name	Description
action	A string property that indicates the action to be performed on the Flink job. It can have one of the following enum values => NEW, UPDATE, STATELESS_UPDATE, STOP, START, CANCEL, SAVEPOINT, LIST_SAVEPOINT, or DELETE.
ClusterJob	Cluster job.
createdByType	The type of identity that created the resource.
ErrorAdditionalInfo	The resource management error additional info.
ErrorDetail	The error detail.
ErrorResponse	Error response
FlinkJobProperties	Properties of flink job.
systemData	Metadata pertaining to creation and last modification of the resource.

action

A string property that indicates the action to be performed on the Flink job. It can have one of the following enum values => NEW, UPDATE, STATELESS_UPDATE, STOP, START, CANCEL, SAVEPOINT, LIST_SAVEPOINT, or DELETE.

Name	Type	Description
CANCEL	string
DELETE	string
LIST_SAVEPOINT	string
NEW	string
SAVEPOINT	string
START	string
STATELESS_UPDATE	string
STOP	string
UPDATE	string

ClusterJob

Cluster job.

Name	Type	Description
id	string	Fully qualified resource ID for the resource. E.g. "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}"
name	string	The name of the resource
properties	ClusterJobProperties: FlinkJobProperties	Properties of cluster job.
systemData	systemData	Azure Resource Manager metadata containing createdBy and modifiedBy information.
type	string	The type of the resource. E.g. "Microsoft.Compute/virtualMachines" or "Microsoft.Storage/storageAccounts"

createdByType

The type of identity that created the resource.

Name	Type	Description
Application	string
Key	string
ManagedIdentity	string
User	string

ErrorAdditionalInfo

The resource management error additional info.

Name	Type	Description
info	object	The additional info.
type	string	The additional info type.

ErrorDetail

The error detail.

Name	Type	Description
additionalInfo	ErrorAdditionalInfo[]	The error additional info.
code	string	The error code.
details	ErrorDetail[]	The error details.
message	string	The error message.
target	string	The error target.

ErrorResponse

Error response

Name	Type	Description
error	ErrorDetail	The error object.

FlinkJobProperties

Properties of flink job.

Name	Type	Description
action	action	A string property that indicates the action to be performed on the Flink job. It can have one of the following enum values => NEW, UPDATE, STATELESS_UPDATE, STOP, START, CANCEL, SAVEPOINT, LIST_SAVEPOINT, or DELETE.
actionResult	string	Action result of job.
args	string	A string property representing additional JVM arguments for the Flink job. It should be space separated value.
entryClass	string	A string property that specifies the entry class for the Flink job.
flinkConfiguration	object	Additional properties used to configure Flink jobs. It allows users to set properties such as parallelism and jobSavePointDirectory. It accepts additional key-value pairs as properties, where the keys are strings and the values are strings as well.
jarName	string	A string property that represents the name of the job JAR.
jobId	string	Unique id for identifying a job
jobJarDirectory	string	A string property that specifies the directory where the job JAR is located.
jobName	string	Name of job
jobOutput	string	Output of job.
jobType	string: FlinkJob	Type of cluster job.
lastSavePoint	string	The last savepoint.
savePointName	string	A string property that represents the name of the savepoint for the Flink job
status	string	Status of job.

systemData

Metadata pertaining to creation and last modification of the resource.

Name	Type	Description
createdAt	string	The timestamp of resource creation (UTC).
createdBy	string	The identity that created the resource.
createdByType	createdByType	The type of identity that created the resource.
lastModifiedAt	string	The timestamp of resource last modification (UTC)
lastModifiedBy	string	The identity that last modified the resource.
lastModifiedByType	createdByType	The type of identity that last modified the resource.