I am trying to schedule my azure ml command job. It runs fine the first time but from second occurrence, it directly goes into completed state!

Question

I am trying to schedule a command job using bicep. It gets executed as expected the first time but then from 2nd time the job directly goes into completed state.

resource jobResource 'Microsoft.MachineLearningServices/workspaces/schedules@2023-10-01' = { name: '${workspaceName}/${jobName}' properties: { action: { actionType: 'CreateJob' jobDefinition: { jobType: 'Command' command: 'echo hello world' environmentId: environmentVersion.id experimentName: experimentName computeId: compute.id description: 'Schedule for running model training' displayName: 'Model Train Job' } } trigger: { triggerType: 'Cron' expression: '42,52 13 20 * *' } } }

Answer 1

Hi @Makarand Batchu

I encountered the same situation too. In my case the reason was that the parameters of pipeline component I set hit the automatic reuse strategy where orchestrator doesn't re-run step if it same parameters and same input path. So I changed the output settings of component and set the "Regenerate output" to be choosen. Then the scheduled job goes perfectly.

Here's my reference: https://github.com/Azure/MachineLearningNotebooks/issues/270

Any my component is Execute Python Script. You can see that the second job dose not run any script.

First job:

Second job:

Hope it will be helpful.

After setting this, if still failed in schedule scenario you can check whether the job definition was successful attached to the schedule. If not you can set job definition via azure cli or sdk of your language.

I think it's maybe a bug that we can see job definition of pipeline in portal/UI, but when list the schedules by sdk we can see that no job definition in the schedule.

Answer 2

Hi @Makarand Batchu

Welcome to the Microsoft Q&A and thank you for posting your questions here. You would like to know the reasons you job schedule in your Azure ML command job runs fine the first time but from second occurrence directly goes into completed state.

The issue you're encountering might be related to the job definition or the trigger configuration.

NOTE: Your Bicep code defines a scheduled job with a command action that executes echo hello world. The job is triggered using a cron expression to run at 13:42 and 13:52 on the 20th of every month.

The job going directly into the completed state after the first execution could be due to various reasons however, the command echo hello world is very basic and will likely complete very quickly. After the first execution, it's possible that the job completes almost immediately, hence why it appears to go directly into the completed state.

My Advice:

1. Add error handling mechanisms within your command to capture any potential errors that might occur during execution. This will help you understand if there are any issues preventing the job from running successfully.
2. Make sure that the job type is appropriate for your use case. If your intention is to run a recurring task, a command job might not be the most suitable option.
3. Verify that the cron expression is correctly set up for your desired schedule. It's possible that the expression is triggering the job more frequently than expected.

Finally, make sure to tailor your solution based on your specific requirements and environment setup by making the environmentVersion.id and compute.id are pointing to valid resources that can execute the command job.

I hope this is helpful! Do not hesitate to let me know if you have any other questions. Please remember to "Accept Answer" if answer helped, so that others in the community facing similar issues can easily find the solution. Best Regards, Sina

Answer 3

@Makarand Batchu

Thanks for your response, please check on the private message part and share your Azure Subscription ID to so that I can process it.

Screenshot to find the private message part is here -

You can respond on either your thread I will process both of them. I hope this helps!

Regards, Yutong

-Please kindly accept the answer if you feel helpful to support the community and vote 'Yes' to support the community, thanks a lot.