Using pyspark dataframe input insert data into a table temp

Question

Using pyspark dataframe input insert data into a table temp

Sadio Jean-Marie 0

I want to insert the results of my query into a temporary table, but I get this error:

User's image

QuantumCache 20,366 Moderator

Hello @Sadio Jean-Marie

Below is an example code snippet that demonstrates how to create a PySpark DataFrame from the query results and then insert it into a temporary table.

from pyspark.sql import SparkSession 
# create a SparkSession 
spark = SparkSession.builder.appName("Insert into Temp Table").getOrCreate() 
# create a DataFrame from the query results 
df = spark.sql("SELECT * FROM my_table") 
# insert the DataFrame into a temporary table 
df.createOrReplaceTempView("temp_table")

1 answer

Your answer

QuantumCache 20,366 Reputation points Moderator

2023-10-31T05:44:49.04+00:00

Hello @Sadio Jean-Marie

Below is an example code snippet that demonstrates how to create a PySpark DataFrame from the query results and then insert it into a temporary table.

from pyspark.sql import SparkSession # create a SparkSession spark = SparkSession.builder.appName("Insert into Temp Table").getOrCreate() # create a DataFrame from the query results df = spark.sql("SELECT * FROM my_table") # insert the DataFrame into a temporary table df.createOrReplaceTempView("temp_table")

Answer 1

Amira Bedhiafi 33,071 Volunteer Moderator

The key part of the error is InvalidAbfsRestOperationException,it says that there is an issue with the interaction between your Spark environment and Azure Blob Storage.

You might have misconfigured the Azure Blob Storage account details in your Spark configuration. Make sure the access key, account name, and other necessary configurations for azurebfs are correctly set.
Can you share what you have so far ?

Share via

Using pyspark dataframe input insert data into a table temp

1 answer

Your answer