Failed to unmount Blob storage in synapse Notebook

Question

Hello everyone,

Need support from Microsoft Community!

I'm getting error while unmounting blob storage mount. I actually have two mounts one for datalake and other for blob storage. Unmounting datalake is working fine but when trying to unmount blob storage getting exception.

Below is the code screenshot for mounting datalake and data storage

Note: '/dest_dir' is actually blob storage mount.

Unmounting using below code

Please find stack trace below

---------------------------------------------------------------------------

Py4JJavaError Traceback (most recent call last)
/tmp/ipykernel_5906/3279281236.py in <module>
1 mssparkutils.fs.unmount('/source_dir')
----> 2 mssparkutils.fs.unmount('/dest_dir')

~/cluster-env/env/lib/python3.8/site-packages/notebookutils/mssparkutils/fs.py in unmount(mountPoint, isLH)
40
41 def unmount(mountPoint, isLH=False):
---> 42 return fs.unmount(mountPoint, isLH)
43
44 def mountLH(source, mountPoint):

~/cluster-env/env/lib/python3.8/site-packages/notebookutils/mssparkutils/handlers/fsHandler.py in unmount(self, mountPoint, isLH)
119 def unmount(self, mountPoint, isLH=False):
120 self.check_types([(mountPoint, string_types), (isLH, bool)])
--> 121 return self.fsutils.unmount(mountPoint, isLH)
122
123 def mounts(self):

~/cluster-env/env/lib/python3.8/site-packages/py4j/java_gateway.py in call(self, *args)
1302
1303 answer = self.gateway_client.send_command(command)
-> 1304 return_value = get_return_value(
1305 answer, self.gateway_client, self.target_id, self.name)
1306

/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py in deco(*a, **kw)
109 def deco(*a, **kw):
110 try:
--> 111 return f(*a, **kw)
112 except py4j.protocol.Py4JJavaError as e:
113 converted = convert_exception(e.java_exception)

~/cluster-env/env/lib/python3.8/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTERtype
325 if answer1 == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)

Py4JJavaError: An error occurred while calling z:mssparkutils.fs.unmount.
: com.microsoft.spark.notebook.msutils.UnMountOperationFailedException: operation with id - e95535ec-ebc8-4fd4-8242-e46c801af675 failed with error: Checking operation rollbackAction execution status failed - service return Failed status for unmount operation. detail info:
node b814926c-b742-4dac-86a6-1bef6c1c82c1 in {notStarted} with unknown error info None
node 69fdccbb-325d-4ccc-a703-a291e81efdd2 in failed status and the error is FailedWithNonZeroExitCode
The rollback action for operation e95535ec-ebc8-4fd4-8242-e46c801af675 failed with non-zero exit code 1

stdout:

stderr:
[fusermount] unmount /synfs/114/dest_dir failed with error fusermount: entry for /synfs/114/dest_dir not found in /etc/mtab
with correlationId: e95535ec-ebc8-4fd4-8242-e46c801af675

node a2b68b89-51e7-4db9-afd0-6daea6cfecb8 in {notStarted} with unknown error info None

at com.microsoft.spark.notebook.msutils.impl.mount.MSFsMountUtilsImpl.tryDoUnMount(MSFsMountUtilsImpl.scala:160)  
at com.microsoft.spark.notebook.msutils.impl.mount.MSFsMountUtilsImpl.$anonfun$unmount$1(MSFsMountUtilsImpl.scala:83)  
at com.microsoft.spark.notebook.msutils.impl.mount.MSFsMountUtilsImpl$.doOpWithLock(MSFsMountUtilsImpl.scala:220)  
at com.microsoft.spark.notebook.msutils.impl.mount.MSFsMountUtilsImpl.unmount(MSFsMountUtilsImpl.scala:83)  
at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.unmount(MSFsUtilsImpl.scala:579)  
at mssparkutils.fs$.unmount(fs.scala:46)  
at mssparkutils.fs.unmount(fs.scala)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
at py4j.Gateway.invoke(Gateway.java:282)  
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)  
at py4j.commands.CallCommand.execute(CallCommand.java:79)  
at py4j.GatewayConnection.run(GatewayConnection.java:238)  
at java.lang.Thread.run(Thread.java:748)  

Caused by: com.microsoft.spark.notebook.msutils.OperationExecutionFailedException: operation with id - e95535ec-ebc8-4fd4-8242-e46c801af675 failed with error: Checking operation rollbackAction execution status failed - service return Failed status for unmount operation. detail info:
node b814926c-b742-4dac-86a6-1bef6c1c82c1 in {notStarted} with unknown error info None
node 69fdccbb-325d-4ccc-a703-a291e81efdd2 in failed status and the error is FailedWithNonZeroExitCode
The rollback action for operation e95535ec-ebc8-4fd4-8242-e46c801af675 failed with non-zero exit code 1

stdout:

stderr:
[fusermount] unmount /synfs/114/dest_dir failed with error fusermount: entry for /synfs/114/dest_dir not found in /etc/mtab
with correlationId: e95535ec-ebc8-4fd4-8242-e46c801af675

node a2b68b89-51e7-4db9-afd0-6daea6cfecb8 in {notStarted} with unknown error info None

at com.microsoft.spark.notebook.operations.BaseOperation.process(BaseOperation.scala:53)  
at com.microsoft.spark.notebook.msutils.impl.mount.MSFsMountUtilsImpl.tryDoUnMount(MSFsMountUtilsImpl.scala:148)  
... 17 more

Answer 1

Hello @Qureshi, Umer ,

Thanks for the question and using MS Q&A platform.

This could be an intermittent issue, I would request you to try again and do let us know.

As per the repro from our end it's working as excepted.

In case, if you are facing the same issue, I would suggest you to try on different cluster and do let us know how does it goes.

In case, if you are still experiencing the same issue, I would request you to share the below details:

• Synapse Workspace Name:
• Spark Cluster Name:
• Workspace Region:

Looking forward for our response.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators