Fabric Native Execution

Sriram Ganarajan 0 Reputation points
2024-09-17T01:48:42.8833333+00:00

Hi ,

Is the native execution avaialble for free trails? I signed up for a free trial , but when i had attempted the Fabric Native Execuiton, The launch of Spark App failed as below

	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:757)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.shuffle.sort.ColumnarShuffleManager

My Spark launc prop as below

%%configure 
{ 
   "conf": {
       "spark.native.enabled": "true", 
       "spark.shuffle.manager": "org.apache.spark.shuffle.sort.ColumnarShuffleManager" 
   } 
}

The way intepreted this, is the additional jars needed for this shuffle may not be in the classpath load. Which made me suspect if the Fabic Spark Pool i attemped is having the needed feature to support .

To test that , i had separately executed the property

"spark.native.enabled": "true", 

I could not notice any Changes to Spark Plan and it was as before. Anything am missing here?

Azure Service Fabric
Azure Service Fabric
An Azure service that is used to develop microservices and orchestrate containers on Windows and Linux.
267 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.