Remotly connect to AzureHDInsight 5.0 spark cluster

Question

Hi,

I have a local jupyter notebook and Im trying to connect to an AzureHDInsight 5.0 spark cluster.

I found tutorials on how to create a jupyter notebook directly inside the Azure Cloud Interface of the cluster, but what if I want to connect remotly?

Something like:

# Replace with your HDInsight cluster information
cluster_name = "NAME"
client_id = "CLIENT-ID"
client_secret = "CLIENT-SECRET"
tenant_id = "TENANT-ID"
container = "CONTAINER-NAME"

spark = SparkSession.builder \
    .appName("Write to HDFS") \
    .config("spark.hadoop.fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem") \
    .config("spark.hadoop.fs.azure.account.auth.type." + cluster_name + ".dfs.core.windows.net", "OAuth") \
    .config("spark.hadoop.fs.azure.account.oauth.provider.type." + cluster_name + ".dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider") \
    .config("spark.hadoop.fs.azure.account.oauth2.client.id." + cluster_name + ".dfs.core.windows.net", client_id) \
    .config("spark.hadoop.fs.azure.account.oauth2.client.secret." + cluster_name + ".dfs.core.windows.net", client_secret) \
    .config("spark.hadoop.fs.azure.account.oauth2.client.endpoint." + cluster_name + ".dfs.core.windows.net", "https://login.microsoftonline.com/" + tenant_id+ "/oauth2/token") \
    .config("spark.hadoop.fs.defaultFS", "wasb://"+ container + "@" + cluster_name + ".blob.core.windows.net/")\
    .getOrCreate()

Is it possible?

Answer 1

Hi @Dario Bertolino Welcome to Microsoft Q&A forum and thanks for reaching out here.

Here is the MS documentation which explains below steps to achieve your requirement - Install Jupyter Notebook on your computer and connect to Apache Spark on HDInsight

There are four key steps involved in installing Jupyter Notebook locally and connecting to Apache Spark on HDInsight.

• Configure Spark cluster.
• Install Jupyter Notebook.
• Install the PySpark and Spark kernels with the Spark magic.
• Configure Spark magic to access Spark cluster on HDInsight.

Another additional helpful resource but it a bit outdated - Remotely execute a Spark job on an HDInsight cluster

Hope this helps. Let us know how it goes.

Thank you

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.