Unable to set spark.cosmos.changeFeed.batchCheckpointLocation to adls storage location while reading change feed from cosmos

Question

Hello Team,

I am using below property to set the check point location to read the change feed from cosmos db using batch API.

"spark.cosmos.changeFeed.batchCheckpointLocation" -> checkPointLocation

It throws me below error if I set check point location to abfss path. However, it is working fine if I set check point location to dbfs path.

IllegalArgumentException: abfss:/adlspath/startOffset has invalid authority.
Caused by: InvalidUriAuthorityException: abfss:/adlspath/startOffset has invalid authority.

Could you please let me know if there is any limitation in using abfss path for storing checkpoint information and resolution for this error.

Answer 1

Hello @Ramakrishna Gunimanikala ,
The issue is with the slash. It seems like you are using only one slash. ABFS URI should start with abfss://
I hope this helps.

------------------------------

• Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators