Azure synapse batch job results to be returned from livy via API (without intermediate storage)

David Beavon 991 Reputation points
2022-09-26T23:54:34.27+00:00

Is there a recommended way to return a small amount of output from a batch job that is executed via livy?

In the best case I would return all of stdout from the driver.

Otherwise it might be enough to return a short string that says something useful/informative like "SUCCESSFULLY PROCESSED 1234 RECORDS".

Below is an image showing Livy batch response from the API. I'm wondering if I can use "log" or "appInfo" or "tags" to send a custom response, so that it can be consumed by the client that submitted the batch job. Any help would be appreciated.

244950-image.png

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.