Notebook workflow with R dataframe

Question

I'm relatively new to Databricks and I'm looking to create a workbook flow where data preparation and modelling are performed in separate notebooks using R. The final cell of the data preparation results in a dataframe called "rdf_all" with around 4.7 million rows and 13 columns. I'm then looking to call this dataframe in the modelling notebook, please can you advise how best to do this? I've found various answers using db.utils but that this only works for Python and Scala and I'm also unsure how to go about saving the dataframe in DBFS as an alternative.

Many thanks for your help!

Answer 1

Hello @Ross Hamilton ,
Thanks for the question and using MS Q&A platform.
As we understand the ask here is how to read one dataframe which is in a different notebook , please do let us know if its not accurate.
Well since the dataframe in question is not small , I think one way is to write it to a intermediate storage and then read it from the other notebook .

Writing such an huge data in DBFS is not recommended .

Please read here

https://stackoverflow.com/questions/62028296/what-is-the-data-size-limit-of-dbfs-in-azure-databricks#:~:text=For%20Azure%20Databricks%20Filesystem%20(DBFS,less%20than%202GB%20in%20size.

Please do let me if you have any queries.
Thanks
Himanshu

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators