Saving Spark SQL result in a new dataframe

dan_banan 6 Reputation points
2021-11-17T13:58:30.617+00:00

Hey,

A simple question. Wondering how to save query results into a dataframe from magic command in Spark notebook in Synapse.

This allows me to save query result directly into a new data frame.
dataframe = spark.sql("<query>")
However long queries must be added with "" + \n "" + etc.

Can I achieve the same thing with a magic command? (no need to turn query into a string) Namely:
%%sql
<query>
So that the result would be assigned into a new dataframe. So far it seems that I can only 'INSERT INTO' previously defined dataframe.

In order words, could I create a spark dataframe from %%sql?

Thanks in advance

.NET
.NET
Microsoft Technologies based on the .NET software framework.
646 questions
.NET Machine learning
.NET Machine learning
.NET: Microsoft Technologies based on the .NET software framework.Machine learning: A type of artificial intelligence focused on enabling computers to use observed data to evolve new behaviors that have not been explicitly programmed.
112 questions
No comments
{count} votes