How to efficiently get the latest Spark job status by name using Synapse Monitoring API

Question

How to efficiently get the latest Spark job status by name using Synapse Monitoring API

Satya 0 Microsoft Employee

I'm using the following Synapse Monitoring API to fetch spark application runs:

Currently it returns all Spark runs, but it does not support filtering by job name in url.

My use case: Before triggering a new job via web activity I want to check latest status of a specific job name to avoid duplicate submissions.

https://<workspaceName>.dev.azuresynapse.net/monitoring/workloadTypes/spark/Applications?api-version=2019-11-01-preview

1 answer

Your answer

Answer 1

Deepanshukatara-6769 15,275

Hello Satya,

While the API may not directly support filtering by job name in the URL, you can filter the list of applications in Synapse Studio by specifying fields such as job name. This allows you to focus on specific applications that match your criteria.

For more detailed guidance, you can refer to the following resources:

Please let us know if any ques further

Thanks

Deepanshu

Satya 0 Reputation points Microsoft Employee

2025-03-28T10:06:25.9966667+00:00

But will this API - https://<workspaceName>.dev.azuresynapse.net/monitoring/workloadTypes/spark/Applications?api-version=2019-11-01-preview
scale well in the future, when the number of jobs increases significantly?
J N S S Kasyap 1,525 Reputation points Microsoft External Staff

2025-03-31T02:33:30.94+00:00
@Satya
The API you're currently using (api-version=2019-11-01-preview) may face scalability challenges as the number of jobs increases significantly, especially if you're fetching and processing all applications without specific filtering.

As the number of Spark jobs grows, retrieving all applications could lead to performance issues, such as long response times or high memory usage. If the API doesn’t provide specific filtering by job name, you would need to fetch all jobs and then filter them locally, which might not scale well in the future.

If scalability is a concern for your use case, it’s important to monitor the performance of the API as the workload increases. If performance degrades, consider implementing alternative solutions such as:

Using caching to store job statuses and avoid frequent API calls.

Exploring pagination to handle large responses more efficiently.

Integrating Azure Monitor with your Synapse workspace allows for more efficient querying and monitoring of Spark job statuses.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
J N S S Kasyap 1,525 Reputation points Microsoft External Staff

2025-04-01T01:45:05.6133333+00:00

@Satya
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
J N S S Kasyap 1,525 Reputation points Microsoft External Staff

2025-04-02T00:58:13.5433333+00:00

@Satya

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

How to efficiently get the latest Spark job status by name using Synapse Monitoring API

1 answer

Your answer