sql against dataframe

arkiboys 9,686 Reputation points
2022-05-05T06:43:54.353+00:00

Hi,
I have a dataframe which I am querying using pyspark. in azure databricks
Question:
How is it possible to query this dataframe using sql?
Thank you

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,072 questions
0 comments No comments
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 46,442 Reputation points Microsoft Employee
    2022-05-06T22:33:36.973+00:00

    Hello @arkiboys ,

    Thanks for the question and using MS Q&A platform.

    In order to run SQL queries on Data frame, first we need to convert them to temporary tables in our spark session as shown in below example. (*Please note that these tables are defined for current session only and will be deleted once Spark session is expired. *)

    199750-image.png

    Once you have your temporary table created for your data frame, you can use SQL queries as shown below:

    199809-image.png

    199810-image.png

    Here is the referenced article where a detailed explanation is provided by a community volunteer: Running SQL queries on Spark DataFrames
    Here is another helpful document: Run SQL queries on DataFrames

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

0 additional answers

Sort by: Most helpful