Databricks 'PRIMARY' filegroup is full

Oussama KIASSI 31 Reputation points
2022-11-27T07:37:03.787+00:00

Hi all,

I have a Databricks workflow that fails after too much time of execution. It actually writes a delta table to a location. It raises the following error with the job being aborted: 

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 670.0 failed 4 times, most recent failure: Lost task 0.3 in stage 670.0 (TID 16234) (172.16.9.135 executor 9): com.microsoft.sqlserver.jdbc.SQLServerException: Could not allocate space for object 'dbo.SORT temporary run storage:  142825730473984' in database 'tempdb' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.

I appreciate any help.
Best regards,

264521-image.png

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,942 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 77,901 Reputation points Microsoft Employee
    2022-11-29T11:07:56.54+00:00

    Hello @Oussama KIASSI ,

    Thanks for the question and using MS Q&A platform.

    Could you please clarify on the which location are you trying to write the delta tables?

    From the error message it looks like your are trying to write to temp storage and the error message is pretty self explanatory - your tempdb is full, and can't be grown to perform the operation it needs tempdb for.

    To resolve this issue: Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.
  2. Mark Taylor 311 Reputation points
    2022-11-29T11:20:54.17+00:00

    As PRADEEP says. Also look to see if your transaction is much bigger than you are expecting, for some reason.