Azure synapse batch job results to be returned from livy via API (without intermediate storage)
David Beavon
991
Reputation points
Is there a recommended way to return a small amount of output from a batch job that is executed via livy?
In the best case I would return all of stdout from the driver.
Otherwise it might be enough to return a short string that says something useful/informative like "SUCCESSFULLY PROCESSED 1234 RECORDS".
Below is an image showing Livy batch response from the API. I'm wondering if I can use "log" or "appInfo" or "tags" to send a custom response, so that it can be consumed by the client that submitted the batch job. Any help would be appreciated.
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
Sign in to answer