Job scheduling in Azure Databricks

Question

I'm trying to create a Job in Databricks using python code,

Also the cluster I'm trying to attach is an all-purpose cluster which has all the libraries installed for the job.

   response = requests.post(  
       'https://%s/api/2.0/jobs/create' % (DOMAIN),  
       headers={'Authorization': 'Bearer {}'.format(TOKEN)},  
       json={  
           "settings": {  
               "existing_cluster_id": "cluster-id",  # all-purpose cluster  
               "notebook_task": {  
                   "notebook_path": "/Notebooks/prediction/f2.py"  
               },  
               "email_notifications": {  
                   "on_failure": [  
                       "abc@org.com"  
                   ]  
               },  
               "name": "Job 01",  
               "schedule": {  
                   "quartz_cron_expression": "24 59 7 * * ?",  
                   "timezone_id": "UTC",  
                   "pause_status": "UNPAUSED"  
               },  
               "max_concurrent_runs": 1  
           }  
       }  
       )  

but I get the following error

Answer 1

Hi @Krishnamoorthy, Rajamannar

I assume you manually created the job in databricks and copied the JSON from the job. You do not need the settings object when using the JSON API as it is the settings actually. Please try the below code and that should work.

response = requests.post(  
    'https://%s/api/2.0/jobs/create' % (DOMAIN),  
    headers={'Authorization': 'Bearer {}'.format(TOKEN)},  
    json={  
            "existing_cluster_id": "cluster-id",  # all-purpose cluster  
            "notebook_task": {  
                "notebook_path": "/Notebooks/prediction/f2.py"  
            },  
            "email_notifications": {  
                "on_failure": [  
                    "abc@org.com"  
                ]  
            },  
            "name": "Job 01",  
            "schedule": {  
                "quartz_cron_expression": "24 59 7 * * ?",  
                "timezone_id": "UTC",  
                "pause_status": "UNPAUSED"  
            },  
            "max_concurrent_runs": 1  
        }  
    )  

Thanks,
Shalvin