Synapse Notebook: Passing variables from %%pyspark cell to %%sparkr cell

Question

I would like to pass variables values from one cell executing Python (%%pyspark) in one cell to another cell executing R (%%sparkr). In the same notebook.

Is it possible?

My code looks like this :

#cell 1: executes pyspark (python code)

%%pyspark

var = 'A'

#cel2: executes sparkr (R code)

%%sparkr

print(paste0("value of var=",var))

Thanks in advance for the help.

Answer 1

One of the easiest ways to pass data between %%pyspark and %%sparkr cells is by creating a Spark DataFrame in Python and then accessing it in R.

   %%pyspark

   var = 'A'

   # Create a Spark DataFrame

   df = spark.createDataFrame([(var,)], ["var"])

   df.createOrReplaceTempView("var_df")

In your R (%%sparkr) cell:

   %%sparkr

   # Access the DataFrame

   df <- sql("SELECT * FROM var_df")

   var_value <- collect(df)$var[1]

   print(paste0("value of var=", var_value))