Spark jobs scheduling

Bakhruz Dzhafarov 60

Hi,

When I use Spark jobs (Serverless Spark Compute) like this:

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-submit-spark-jobs?view=azureml-api-2&tabs=sdk#spark-component-in-a-pipeline-job

can I schedule them in a similar way as described here:

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-schedule-pipeline-job?view=azureml-api-2&tabs=ui

Because I don't see such an option after starting.

Amira Bedhiafi 26,656 Reputation points

2024-06-17T07:39:51.63+00:00

Define your Spark jon as a component within a pipeline. You can follow the guidelines from the first link to set up your Spark job as part of an Azure Machine Learning pipeline.

Once your Spark job is set up as a component, you need to create a pipeline that includes this Spark job component. This can be done using the Azure Machine Learning SDK or through the UI in the Azure Machine Learning Studio.

After creating the pipeline, you can schedule it similarly to other pipeline jobs. Follow the instructions in the second link to set up a recurring schedule for your pipeline job.

Your answer