How to fix broken read/write Pythonic operations in Azure Databricks?

Question

How to fix broken read/write Pythonic operations in Azure Databricks?

Robert Tinn 1

My team has a production pipeline running in Azure Databricks using PySpark. This pipeline is automated and has been running for months with zero issues. Recently it has broken because I can no longer use certain read/write to files in a mounted blob storage account. No changes have been to the code or environment during this time so it is unclear why it has suddenly broken.

Spark read/write operations such as "sc.textFile()" work perfectly well as before.

However standard Pythonic operations such as "open()" now always throw the error "FileNotFoundError: [Errno 2] No such file or directory" when the files definitely exist and could always be read fine before.

Please advise how to fix this.

PRADEEPCHEEKATLA 91,866 Reputation points

2021-12-22T07:45:12.697+00:00

Hello @Robert Tinn ,

Thanks for the question and using MS Q&A platform.

This is strange. Could you please share the complete code which you are running along with the stack trace of the error message?

And also, please do confirm the file or directory exists by running the same path using dbutils.fs.ls?
PRADEEPCHEEKATLA 91,866 Reputation points

2021-12-24T10:26:04.217+00:00

Hello @Robert Tinn ,

Did you get a chance to accept it as answer(). This can be beneficial to other community members. Thank you.

1 answer

Your answer

PRADEEPCHEEKATLA 91,866 Reputation points

2021-12-22T07:45:12.697+00:00

Hello @Robert Tinn ,

Thanks for the question and using MS Q&A platform.

This is strange. Could you please share the complete code which you are running along with the stack trace of the error message?

And also, please do confirm the file or directory exists by running the same path using dbutils.fs.ls?
PRADEEPCHEEKATLA 91,866 Reputation points

2021-12-24T10:26:04.217+00:00

Hello @Robert Tinn ,

Did you get a chance to accept it as answer(). This can be beneficial to other community members. Thank you.

Answer 1

Robert Tinn 1

Thanks for the quick response. I just figured out the issue. It turns out someone rotated the keys for the storage account and somehow that was responsible for this unusual error. By simply unmounting and then remounting the storage account in Databricks resolved this issue :)

PRADEEPCHEEKATLA 91,866 Reputation points

2021-12-23T02:50:55.207+00:00

Hello @Robert Tinn ,

Glad to know that your issue has resolved. You can accept it as answer(). This can be beneficial to other community members. Thank you.

Share via

How to fix broken read/write Pythonic operations in Azure Databricks?

1 answer

Your answer