Can Livy be used when I run a remote spark job in databricks.

Liupeng Zhuang 1 Reputation point
2021-04-28T02:32:27.723+00:00

Hi, I am surveying the feature of databricks runing remote spark jobs. Now I have some questions:
(1) Can Livy api be used to run remote spark jobs as HDInsight do ?
(2) We can build different versions of spark cluster (3.1.1, 3.0.1, 2.4.5, 2.4.3), if Livy api is useful to databricks, what are their version respective ?
(3) If Livy is not useful, are there some api similiar to Livy which can be installed by NuGet using c-sharp?

Thanks.

.NET
.NET
Microsoft Technologies based on the .NET software framework.
3,367 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,916 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 76,921 Reputation points Microsoft Employee
    2021-04-28T11:48:19.597+00:00

    Hello @Liupeng Zhuang ,

    Thanks for the question and using Microsoft Q&A platform.

    Apache Livy’s is a dead project. Why would you want to install Apache Livy on Azure Databricks clusters?

    Similar to what Apache Livy has, Azure Databricks also provides a REST API.

    Azure Databricks REST API which does what Apache Livy was designed for.

    Hope this helps. Do let us know if you any further queries.

    ------------

    Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.