unable to load the dataframe from cosmos

Question

Hello,

I am trying to load the dataframe from cosmos and unable to do that. I am trying the following command:

   def load_dataframe(self, query):      
       return self.spark_session.read.format('com.microsoft.azure.cosmosdb.spark').options(**self._make_connection(query)).load()  

And I am seeing the below error:

I have tried the following steps as well:

   You need to build the repository into the jar file first using SBT. Then include it to your spark cluster.  
     
   I know there will be a lot of people having trouble with buiding this jar file (include myself of several hours ago), so I will guide you how to build the jar file, step by step:  
     
   Go to https://www.scala-sbt.org/download.html to download SBT, then install it.  
     
   Go to https://github.com/Azure/azure-cosmosdb-spark and download the zip file.  
     
   Open the folder of the repository you have just downloaded, right click in the blank space and click "Open PowerShell windows here" . https://i.stack.imgur.com/Fq7NX.png  
     
   In the Shell windows, type "sbt" then press enter. It may require you to download the Java Development Kit. If so, go to https://www.oracle.com/java/technologies/javase-downloads.html to download and install it. You may need to close and reopen the shell windows after installing.  
     
   If things go right, you may see this screen: https://i.stack.imgur.com/fMxVr.png  
     
   After the above step has done its job, type "package". The shell may show you something like this, and it may take you a long time to finish the job. https://i.stack.imgur.com/hr2hw.png  
     
   After the build is done, go to the "target" folder, then "scala-2.12" folder to get the jar file. https://i.stack.imgur.com/Aziqy.png  
     
   After you got the jar file, include it to the Spark cluster.  

Can anybody help me with this problem? Or what can I do to fix this.

Answer 1

Hello @Shubham Mehta ,

Thanks for the question and using MS Q&A platform.

I would suggest you to use the below connector to read the Cosmos data into Dataframe:

https://search.maven.org/artifact/com.azure.cosmos.spark/azure-cosmos-spark_3-1_2-12/4.1.0/jar

For more details, refer to Azure Cosmos DB Apache Spark 3 OLTP Connector for Core (SQL) API: Release notes and resources.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators