Using the Jobs API to get job failures in the last 12 hours

Kenneth Huddleston 145 Reputation points
2023-02-02T19:53:40.4766667+00:00

For my use case, I am looking to use the Databricks Job API to query for a list of jobs that have 'failed' in the last 12 hours. As best I can tell, the Job API only lets you filter based on 'start time' not 'end time'. To address this limitation, my script is searching the entire job history, parsing the 'end time', and manually discarding results that are more than 12 hours old. While this 'works', it is incredibly inefficient.

Has anyone else been in this situation? Is there a more efficient way to get job failures for a given amount of time?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.