Spark Batch - Create Spark Batch Job

Service:

Synapse

API Version:

2020-12-01

Buat pekerjaan batch spark baru.

POST {endpoint}/livyApi/versions/{livyApiVersion}/sparkPools/{sparkPoolName}/batches

With optional parameters:

POST {endpoint}/livyApi/versions/{livyApiVersion}/sparkPools/{sparkPoolName}/batches?detailed={detailed}

Parameter URI

Nama	Dalam	Diperlukan	Jenis	Deskripsi
endpoint	path	True	string	Titik akhir pengembangan ruang kerja, misalnya https://myworkspace.dev.azuresynapse.net.
livyApiVersion	path	True	string	Versi api yang valid untuk permintaan.
sparkPoolName	path	True	string	Nama kumpulan spark.
detailed	query		boolean	Param kueri opsional yang menentukan apakah respons terperinci dikembalikan di luar livy biasa.

Isi Permintaan

Nama	Diperlukan	Jenis	Deskripsi
file	True	string
name	True	string
archives		string[]
args		string[]
artifactId		string
className		string
conf		object
driverCores		integer
driverMemory		string
executorCores		integer
executorMemory		string
files		string[]
jars		string[]
numExecutors		integer
pyFiles		string[]
tags		object

Respons

Nama	Jenis	Deskripsi
200 OK	SparkBatchJob	Berhasil

Contoh

Create new spark batch job.

Sample Request

HTTP

POST myWorkspace.dev.azuresynapse.net/livyApi/versions/2019-11-01-preview/sparkPools/mySparkPool/batches?detailed=True

{
  "tags": null,
  "artifactId": "fill in here",
  "name": "jobname",
  "file": "abfss://",
  "className": "classname",
  "args": [],
  "jars": [],
  "files": [],
  "archives": [],
  "conf": null,
  "driverMemory": "4g",
  "driverCores": 4,
  "executorMemory": "2g",
  "executorCores": 4,
  "numExecutors": 2
}

Sample Response

Status code:

200

{
  "livyInfo": null,
  "name": "jobname",
  "workspaceName": "myWorkspace",
  "sparkPoolName": "mySparkPool",
  "submitterName": "thetime",
  "submitterId": "thesubmitterid",
  "artifactId": "fill in here",
  "jobType": "SparkBatch",
  "result": "Failed",
  "schedulerInfo": null,
  "pluginInfo": null,
  "errorInfo": [],
  "tags": null,
  "id": 1,
  "appId": "fill in here",
  "appInfo": null,
  "state": "the state",
  "log": []
}

Definisi

Nama	Deskripsi
LivyStates	Status batch
PluginCurrentState
SchedulerCurrentState
SparkBatchJob
SparkBatchJobOptions
SparkBatchJobResultType	Hasil pekerjaan batch Spark.
SparkBatchJobState
SparkErrorSource
SparkJobType	Jenis pekerjaan.
SparkRequest
SparkScheduler
SparkServiceError
SparkServicePlugin

LivyStates

Status batch

Nama	Jenis	Deskripsi
busy	string
dead	string
error	string
idle	string
killed	string
not_started	string
recovering	string
running	string
shutting_down	string
starting	string
success	string

PluginCurrentState

Nama	Jenis	Deskripsi
Cleanup	string
Ended	string
Monitoring	string
Preparation	string
Queued	string
ResourceAcquisition	string
Submission	string

SchedulerCurrentState

Nama	Jenis	Deskripsi
Ended	string
Queued	string
Scheduled	string

SparkBatchJob

Nama	Jenis	Deskripsi
appId	string	Id aplikasi sesi ini
appInfo	object	Info aplikasi terperinci.
artifactId	string	Pengidentifikasi artefak.
errorInfo	SparkServiceError[]	Informasi kesalahan.
id	integer	Id sesi.
jobType	SparkJobType	Jenis pekerjaan.
livyInfo	SparkBatchJobState
log	string[]	Baris log.
name	string	Nama batch.
pluginInfo	SparkServicePlugin	Informasi plugin.
result	SparkBatchJobResultType	Hasil pekerjaan batch Spark.
schedulerInfo	SparkScheduler	Informasi penjadwal.
sparkPoolName	string	Nama kumpulan Spark.
state	LivyStates	Status batch
submitterId	string	Pengidentifikasi pengirim.
submitterName	string	Nama pengirim.
tags	object	Tag.
workspaceName	string	Nama ruang kerja.

SparkBatchJobOptions

Nama	Jenis	Deskripsi
archives	string[]
args	string[]
artifactId	string
className	string
conf	object
driverCores	integer
driverMemory	string
executorCores	integer
executorMemory	string
file	string
files	string[]
jars	string[]
name	string
numExecutors	integer
pyFiles	string[]
tags	object

SparkBatchJobResultType

Hasil pekerjaan batch Spark.

Nama	Jenis	Deskripsi
Cancelled	string
Failed	string
Succeeded	string
Uncertain	string

SparkBatchJobState

Nama	Jenis	Deskripsi
currentState	string	status pekerjaan Spark.
deadAt	string	waktu di mana "mati" negara livy pertama kali terlihat.
jobCreationRequest	SparkRequest
killedAt	string	waktu ketika "membunuh" negara livy pertama kali terlihat.
notStartedAt	string	waktu ketika status livy "not_started" pertama kali terlihat.
recoveringAt	string	waktu ketika "memulihkan" status livy pertama kali terlihat.
runningAt	string	waktu ketika status livy "berjalan" pertama kali terlihat.
startingAt	string	waktu saat "memulai" status livy pertama kali terlihat.
successAt	string	waktu ketika status livy "sukses" pertama kali terlihat.

SparkErrorSource

Nama	Jenis	Deskripsi
Dependency	string
System	string
Unknown	string
User	string

SparkJobType

Jenis pekerjaan.

Nama	Jenis	Deskripsi
SparkBatch	string
SparkSession	string

SparkRequest

Nama	Jenis	Deskripsi
archives	string[]
args	string[]
className	string
conf	object
driverCores	integer
driverMemory	string
executorCores	integer
executorMemory	string
file	string
files	string[]
jars	string[]
name	string
numExecutors	integer
pyFiles	string[]

SparkScheduler

Nama	Jenis	Deskripsi
cancellationRequestedAt	string
currentState	SchedulerCurrentState
endedAt	string
scheduledAt	string
submittedAt	string

SparkServiceError

Nama	Jenis	Deskripsi
errorCode	string
message	string
source	SparkErrorSource

SparkServicePlugin

Nama	Jenis	Deskripsi
cleanupStartedAt	string
currentState	PluginCurrentState
monitoringStartedAt	string
preparationStartedAt	string
resourceAcquisitionStartedAt	string
submissionStartedAt	string