Best practices for submitting spark batch jobs in Azure HDinsights.

Question

Hi, I'm looking to submit my pyspark scripts in HDInsight.
Currently, HDInsight provided Livy for job submission, using curl.
However, If I want to productionize it, then what authentication mechanism to use.
Also, How can I check the progress of my spark application and create an alert on various instances.

Thanks

Answer 1

Hello @Bipin Singh and welcome to Microsoft Q&A.

Under 'Best Practicies' documentation, there are a number of options for submitting, and monitoring jobs.

These options include making an app in VScode, an application calling REST API for livy, as part of a Data Factory pipeline, and others.
Curl is a client application. You can write a script to automate using curl calls, or, make a full application to do the same thing using some other url library. In the end, everything is at some point an http request.

It all really depends upon what tools you want to use, and what sort of application you are building.

Or you can go direct to the livy docs.

Does this help?