Stable release for Azure.Analytics.Synapse.Spark

Jiten Patil 0

I can connect spark endpoint(livyapi version -2019-11-01-preview) using Azure.Analytics.Synapse.Spark 1.0.0-preview.8 but can't see any stable release and last updated date for package is 10/5/2021.

What is stable release url that we can use? Is there any other options to connect spark endpoint in dot net core with same way need to pass additional parameters as well?

POST myWorkspace.dev.azuresynapse.net/livyApi/versions/2019-11-01-preview/sparkPools/mySparkPool/batches?detailed=True

{

"tags": null, "artifactId": "fill in here", "name": "jobname", "file": "abfss://", "className": "classname", "args": [], "jars": [], "files": [], "archives": [], "conf": null, "driverMemory": "4g", "driverCores": 4, "executorMemory": "2g", "executorCores": 4, "numExecutors": 2

}

References - https://www.nuget.org/packages/Azure.Analytics.Synapse.Spark/1.0.0-preview.8 https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/spark-batch/create-spark-batch-job?view=rest-synapse-data-plane-2020-12-01&tabs=HTTP

phemanth 6,555 Microsoft Vendor

@Jiten Patil

Thanks for using MS Q&A platform and posting your query.

While there isn't a formally designated "stable" release for the Azure.Analytics.Synapse.Spark package at this time, it's generally considered safe to use the latest preview version

this is not sure going to be work but I'm just providing a workaround solution on your requirement.

Navigate to the NuGet Package Manager in your Visual Studio project.
Search for Azure.Analytics.Synapse.Spark.
In the search results, ensure "Include prerelease" is checked.
Select the latest version (likely higher than 1.0.0-preview.8).
Install the package.

Connecting to Spark Endpoint

Here's a code example demonstrating how to connect and submit a Spark job using the latest Azure.Analytics.Synapse.Spark package:

using System;
using System.Collections.Generic;
using System.Net.Http;
using Azure.Analytics.Synapse.Spark;
public class SparkJobSubmitter
{
    private readonly string _workspaceUrl;
    private readonly string _sparkPoolName;
    public SparkJobSubmitter(string workspaceUrl, string sparkPoolName)
    {
        _workspaceUrl = workspaceUrl;
        _sparkPoolName = sparkPoolName;
    }
    public async Task SubmitSparkJobAsync(string jobName, string className, string driverMemory,
                                          string driverCores, string executorMemory, string executorCores,
                                          int numExecutors, string[] args = null, string[] jars = null,
                                          string[] files = null, string[] archives = null)
    {
        var sparkBatchClient = new SparkBatchClient(_workspaceUrl, new HttpClient());
        var livyApiVersion = "2019-11-01-preview"; // Assuming your endpoint uses this version
        var job = new SparkBatchJob
        {
            Name = jobName,
            File = "abfss://", // Replace with your file path
            ClassName = className,
            Args = args ?? new string[0],
            Jars = jars ?? new string[0],
            Files = files ?? new string[0],
            Archives = archives ?? new string[0],
            Conf = null, // Optional configuration settings
            DriverMemory = driverMemory,
            DriverCores = Convert.ToInt33(driverCores),
            ExecutorMemory = executorMemory,
            ExecutorCores = Convert.ToInt33(executorCores),
            NumExecutors = numExecutors
        };
        await sparkBatchClient.CreateSparkBatchJobAsync(livyApiVersion, _sparkPoolName, job);
    }
}

Usage:

var workspaceUrl = "https://myWorkspace.dev.azuresynapse.net";
var sparkPoolName = "mySparkPool";
var jobSubmitter = new SparkJobSubmitter(workspaceUrl, sparkPoolName);
await jobSubmitter.SubmitSparkJobAsync(
    "MySparkJob",
    "com.example.MySparkApp",
    "4g", "4", "2g", "4", 2,
    new string[] { "arg1", "arg2" } // Optional arguments
);

Remember to replace placeholders with your actual values (workspace URL, pool name, job details, etc.).

Refer to the Azure Synapse Analytics documentation for the latest information on Spark pool configuration and job submission: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-job-definitions

Hope this helps. Do let us know if you any further queries.

1 answer

phemanth 6,555 Reputation points Microsoft Vendor

2024-05-01T14:35:51.1933333+00:00

@Jiten Patil We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Stable release for Azure.Analytics.Synapse.Spark

1 answer