Using the Jobs API to get job failures in the last 12 hours

Question

Using the Jobs API to get job failures in the last 12 hours

Kenneth Huddleston 145

For my use case, I am looking to use the Databricks Job API to query for a list of jobs that have 'failed' in the last 12 hours. As best I can tell, the Job API only lets you filter based on 'start time' not 'end time'. To address this limitation, my script is searching the entire job history, parsing the 'end time', and manually discarding results that are more than 12 hours old. While this 'works', it is incredibly inefficient.

Has anyone else been in this situation? Is there a more efficient way to get job failures for a given amount of time?

KranthiPakala-MSFT 46,642 Microsoft Employee Moderator

Hi Kenneth Huddleston,

Thanks for using Microsoft Q&A forum and posting your query.
To better assist on your query, could you please share the script (please remove/mask any sensitive information from the script) you are using to pull the data?
Are you looking for something like below?

import requests
import json
import datetime
# Define the endpoint URL for the Databricks Jobs API
endpoint = "https://.cloud.databricks.com/api/2.0/jobs/list"
# Define the API token for authentication
headers = {
    "Authorization": "Bearer <api-token>"
}
# Get the current time and subtract 12 hours to find the start time for the search
current_time = datetime.datetime.utcnow()
start_time = current_time - datetime.timedelta(hours=12)
# Define the parameters for the API call to list all jobs in the past 12 hours
params = {
    "status": "FAILED",
    "from_time": int(start_time.timestamp() * 1000)
}
# Call the Databricks Jobs API
response = requests.get(endpoint, headers=headers, params=params)
# Parse the response from the API
jobs = json.loads(response.text)["jobs"]
# Print all job failures in the past 12 hours
for job in jobs:
    print("Job ID:", job["job_id"], "Start Time:", datetime.datetime.fromtimestamp(job["start_time"]/1000), "Status:", job["state"]["life_cycle_state"])

Kenneth Huddleston 145 Reputation points

2023-02-07T03:55:11.4666667+00:00

What you have there is exactly what I am looking for. Is that a valid script? My script is very similar to yours, except I am querying the "https://.cloud.databricks.com/api/2.1/jobs/runs/list" endpoint. According to the documentation, the only params are 'completed only', 'active only', 'start time', 'limit', and 'offset'...
Kenneth Huddleston 145 Reputation points

2023-02-07T03:58:07.9166667+00:00

I didn't see any 'status' filters in the documentation. Is that just an example you gave, or is that a valid program?

Your answer

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-02-06T01:15:45.51+00:00

Hi Kenneth Huddleston,

Thanks for using Microsoft Q&A forum and posting your query.
To better assist on your query, could you please share the script (please remove/mask any sensitive information from the script) you are using to pull the data?
Are you looking for something like below?

import requests import json import datetime # Define the endpoint URL for the Databricks Jobs API endpoint = "https://.cloud.databricks.com/api/2.0/jobs/list" # Define the API token for authentication headers = { "Authorization": "Bearer <api-token>" } # Get the current time and subtract 12 hours to find the start time for the search current_time = datetime.datetime.utcnow() start_time = current_time - datetime.timedelta(hours=12) # Define the parameters for the API call to list all jobs in the past 12 hours params = { "status": "FAILED", "from_time": int(start_time.timestamp() * 1000) } # Call the Databricks Jobs API response = requests.get(endpoint, headers=headers, params=params) # Parse the response from the API jobs = json.loads(response.text)["jobs"] # Print all job failures in the past 12 hours for job in jobs: print("Job ID:", job["job_id"], "Start Time:", datetime.datetime.fromtimestamp(job["start_time"]/1000), "Status:", job["state"]["life_cycle_state"])
Kenneth Huddleston 145 Reputation points

2023-02-07T03:55:11.4666667+00:00

What you have there is exactly what I am looking for. Is that a valid script? My script is very similar to yours, except I am querying the "https://.cloud.databricks.com/api/2.1/jobs/runs/list" endpoint. According to the documentation, the only params are 'completed only', 'active only', 'start time', 'limit', and 'offset'...
Kenneth Huddleston 145 Reputation points

2023-02-07T03:58:07.9166667+00:00

I didn't see any 'status' filters in the documentation. Is that just an example you gave, or is that a valid program?

Share via

Using the Jobs API to get job failures in the last 12 hours

Your answer