question

danbanan-9611 avatar image
0 Votes"
danbanan-9611 asked danbanan-9611 edited

Saving Spark SQL result in a new dataframe

Hey,

A simple question. Wondering how to save query results into a dataframe from magic command in Spark notebook in Synapse.

This allows me to save query result directly into a new data frame.
dataframe = spark.sql("<query>")
However long queries must be added with "" + \n "" + etc.

Can I achieve the same thing with a magic command? (no need to turn query into a string) Namely:
%%sql
<query>
So that the result would be assigned into a new dataframe. So far it seems that I can only 'INSERT INTO' previously defined dataframe.

In order words, could I create a spark dataframe from %%sql?

Thanks in advance

dotnet-ml-big-data
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

0 Answers