Saving Spark SQL result in a new dataframe
Hey,
A simple question. Wondering how to save query results into a dataframe from magic command in Spark notebook in Synapse.
This allows me to save query result directly into a new data frame.
dataframe = spark.sql("<query>")
However long queries must be added with "" + \n "" + etc.
Can I achieve the same thing with a magic command? (no need to turn query into a string) Namely:
%%sql
<query>
So that the result would be assigned into a new dataframe. So far it seems that I can only 'INSERT INTO' previously defined dataframe.
In order words, could I create a spark dataframe from %%sql?
Thanks in advance