sql against dataframe

Question

Hi,
I have a dataframe which I am querying using pyspark. in azure databricks
Question:
How is it possible to query this dataframe using sql?
Thank you

Answer 1

Hello @arkiboys ,

Thanks for the question and using MS Q&A platform.

In order to run SQL queries on Data frame, first we need to convert them to temporary tables in our spark session as shown in below example. (*Please note that these tables are defined for current session only and will be deleted once Spark session is expired. *)

Once you have your temporary table created for your data frame, you can use SQL queries as shown below:

Here is the referenced article where a detailed explanation is provided by a community volunteer: Running SQL queries on Spark DataFrames
Here is another helpful document: Run SQL queries on DataFrames

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators