spark.readStream.text() text file from adls gen2 in databricks

Sasidhar R. Kolli 20

file_path = "adl://<account-name>.dfs.core.windows.net/<container>/<path>/*.txt"

streaming_df = spark.readStream \
    .schema(schema) \
    .text(file_path) 

query = streaming_df_transformed.writeStream \
    .outputMode("append") \
    .format("console") \
    .start()

query.awaitTermination()

this is the code I'm using but when i read simple spark.read.text it is able to read my df but when I do spark.readStream.text() I'm able to read but when I do query it is throwing me error could not find adls gen2 token. Please provide me solution

Bhargava-MSFT 31,191 Reputation points Microsoft Employee

2023-10-05T20:07:17.6666667+00:00

Hello Sasidhar R. Kolli,

Welcome to the Microsoft Q&A forum.

Are you using Credential passthrough here?

If yes, credential passthrough doesn't work for jobs. You need to set the spark config properties in the job cluster directly.

https://learn.microsoft.com/en-us/azure/databricks/storage/azure-storage#--access-azure-data-lake-storage-gen2-or-blob-storage-using-oauth-20-with-an-azure-service-principal

Please see this StackOverflow thread.

I hope this helps.
Bhargava-MSFT 31,191 Reputation points Microsoft Employee

2023-10-09T19:29:48.76+00:00

Hello Sasidhar R. Kolli,

I am checking to see if you had a chance to look into my answer above. Please let me know if you have any further questions.
Bhargava-MSFT 31,191 Reputation points Microsoft Employee

2023-10-11T16:35:12.9466667+00:00

Hello Sasidhar R. Kolli,

I am checking to see if you need any further help here.
Sasidhar R. Kolli 20 Reputation points

2023-10-11T16:51:42.8066667+00:00

I have used sas token and able to stream the directory of adls but im unable to stream the text file present in adls. Please give me solution to stream the text file in adls gen2
Bhargava-MSFT 31,191 Reputation points Microsoft Employee

2023-10-11T19:49:58.1766667+00:00
Can you please check the SAS token is valid and has the required permissions to access the data in the storage account?

Also, please check your file path exists in the specified location.

And add error handling to your spark code to capture exceptions.

try: # Your Spark Structured Streaming code here query.awaitTermination() except Exception as e: print("An error occurred:", str(e))
Bhargava-MSFT 31,191 Reputation points Microsoft Employee

2023-10-17T18:56:28.1633333+00:00

Hello Sasidhar R. Kolli,

I am checking to see if you need any further assistance here.

Share via

spark.readStream.text() text file from adls gen2 in databricks

Your answer