Using parameters defined in pyspark parameter cell in scala-spark?

Question

Eshwar Sundar 46 Microsoft Employee

I have a parameter cell defined in pyspark
with values like

The above cell is a parameter cell and the language is PySpark

Now I want to use this in ScalaSpark. I assume it is doable since the above is a parameter cell, like:

Can someone help here?

Dillon Silzer 60,726 Reputation points Volunteer Moderator

2022-07-16T04:15:03.36+00:00

Are you talking about creating a Scala notebook in Databricks and creating a spark.sql statement?
Eshwar Sundar 46 Reputation points Microsoft Employee

2022-07-16T04:18:36.527+00:00

No this is a notebook in Azure Synapse.
I want to share variables between cells.
Eshwar Sundar 46 Reputation points Microsoft Employee

2022-07-16T04:19:25.86+00:00

The catch is -> the cells are in different languages. but the cell that defines the variables is a parameter cell.

Answer accepted by question author

0 additional answers

Dillon Silzer 60,726 Reputation points Volunteer Moderator

2022-07-16T04:15:03.36+00:00

Are you talking about creating a Scala notebook in Databricks and creating a spark.sql statement?
Eshwar Sundar 46 Reputation points Microsoft Employee

2022-07-16T04:18:36.527+00:00

No this is a notebook in Azure Synapse.
I want to share variables between cells.
Eshwar Sundar 46 Reputation points Microsoft Employee

2022-07-16T04:19:25.86+00:00

The catch is -> the cells are in different languages. but the cell that defines the variables is a parameter cell.

Answer 1

Dillon Silzer 60,726 Volunteer Moderator

See section Use temp tables to reference data across languages

----------------------

If this helps please mark as correct answer.

Eshwar Sundar 46 Reputation points Microsoft Employee

2022-07-16T04:57:19.483+00:00

This is actually a really good option. Thanks.
But can we not use parameter cell for this purpose?
I thought Azure Synapse injects parameters in runtime and abstracts away the language.
Check this out :
https://diangermishuizen.com/parameterize-synapse-analytics-spark-notebooks-efficiently/#:~:text=In%20Synapse%20Analytics%2C%20when%20calling%20a%20Notebook%20activity,cell.%20There%20can%20only%20be%20one%20per%20notebook.

I am thinking this might be more efficient than sharing tables across nodes. But I want to do this in across runtime.
Dillon Silzer 60,726 Reputation points Volunteer Moderator

2022-07-16T05:42:15.397+00:00

I think you can use both ways, but I'd say the Microsoft Learn example looks less complex if you are trying to pass values from language to language.
Eshwar Sundar 46 Reputation points Microsoft Employee

2022-07-16T18:23:18.07+00:00

Thanks Dillon.