Saving Spark SQL result in a new dataframe

asked 2021-11-17T13:58:30.617+00:00
dan_banan 6 Reputation points


A simple question. Wondering how to save query results into a dataframe from magic command in Spark notebook in Synapse.

This allows me to save query result directly into a new data frame.
dataframe = spark.sql("<query>")
However long queries must be added with "" + \n "" + etc.

Can I achieve the same thing with a magic command? (no need to turn query into a string) Namely:
So that the result would be assigned into a new dataframe. So far it seems that I can only 'INSERT INTO' previously defined dataframe.

In order words, could I create a spark dataframe from %%sql?

Thanks in advance

Microsoft Technologies based on the .NET software framework.
596 questions
.NET Machine learning
.NET Machine learning
.NET: Microsoft Technologies based on the .NET software framework.Machine learning:
111 questions
No comments
{count} votes