Hello @Rohit Kulkarni ,
Thanks for the question and using MS Q&A platform.
Below are couple of stack overflow threads where a similar requirement has been discussed by community volunteers using pyspark in Databricks. You can follow similar approach to achieve your requirement.
- Merge CSV files in ADLS2 that are prepared through DataBricks
- Azure Data-bricks : How to read part files and save it as one file to blob?
Hope this helps. Let us know how it goes.