Saving Spark SQL result in a new dataframe

dan_banan 6 Reputation points


A simple question. Wondering how to save query results into a dataframe from magic command in Spark notebook in Synapse.

This allows me to save query result directly into a new data frame.
dataframe = spark.sql("<query>")
However long queries must be added with "" + \n "" + etc.

Can I achieve the same thing with a magic command? (no need to turn query into a string) Namely:
So that the result would be assigned into a new dataframe. So far it seems that I can only 'INSERT INTO' previously defined dataframe.

In order words, could I create a spark dataframe from %%sql?

Thanks in advance

Microsoft Technologies based on the .NET software framework.
630 questions
.NET Machine learning
.NET Machine learning
.NET: Microsoft Technologies based on the .NET software framework.Machine learning: A type of artificial intelligence focused on enabling computers to use observed data to evolve new behaviors that have not been explicitly programmed.
110 questions
No comments
{count} votes