Hello @Birajdar, Sujata ,
Welcome to the Microsoft Q&A platform.
Yes, Azure Databricks support 50 million records using pyspark.
For more details, refer to the below articles:
- How to process a DataFrame with millions of rows in seconds
- Experimenting with PySpark to Match Large Data Sources
You may checkout the below articles which describes more on optimize good performance:
Hope this will help. Please let us know if any further queries.
------------------------------
- Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
- Want a reminder to come back and check responses? Here is how to subscribe to a notification
- If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators