Using the Jobs API to get job failures in the last 12 hours
Kenneth Huddleston
145
Reputation points
For my use case, I am looking to use the Databricks Job API to query for a list of jobs that have 'failed' in the last 12 hours. As best I can tell, the Job API only lets you filter based on 'start time' not 'end time'. To address this limitation, my script is searching the entire job history, parsing the 'end time', and manually discarding results that are more than 12 hours old. While this 'works', it is incredibly inefficient.
Has anyone else been in this situation? Is there a more efficient way to get job failures for a given amount of time?
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,534 questions
Sign in to answer