Single-Node Cluster Configuration for Azure Synapse Spark Pool

Question

Is it possible to configure an Azure Synapse Spark pool as a single-node cluster, where only the master node is used, and no additional nodes are involved in the setup? My requirement is to create a Spark pool that operates with just the master node, avoiding the use of executor or worker nodes. This configuration would be ideal for scenarios where I don't need distributed processing but still require the capabilities of Spark for small-scale tasks or testing. Is this kind of setup achievable in Azure Synapse?

Answer 1

Hello @Anwar

Thanks for the question and using MS Q&A platform.

In a Synapse workspace, you can create a Spark pool with a minimum of three nodes, which includes one driver node and two worker nodes. It is not possible to create a single-node cluster in Synapse.

For more information, please refer - https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-autoscale

Similar thread for reference: https://learn.microsoft.com/en-us/answers/questions/1692387/setting-up-a-single-node-cluster-in-synapse

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.