Purpose of "spark.yarn.executor.decommission.enabled" in Synapse Spark

Question

Purpose of "spark.yarn.executor.decommission.enabled" in Synapse Spark

David Beavon 996

The Microsoft implementation of Spark that is found in Synapse has an unusual property named "spark.yarn.executor.decommission.enabled". This seems to be totally undocumented.

User's image

Whenever it is present, and set to "true", it will print a message in the driver logs like so: INFO YarnAllocator [main]: Yarn Executor Decommissioning Enabled User's image

I have not found documentation, but I believe this is related to the YARN resource manager implementation in Synapse, and I believe it is related to "autoscale" in particular, which allows nodes of the cluster to be removed if needed.

It would be nice to have a better understanding of this property. I'm finding that this may be a Microsoft customization to Yarn, and it might be incompatible with certain Spark features. For example this property seems to conflict (or supercede) the behavior of the idle timeout for dynamically allocated executors: spark.dynamicAllocation.executorIdleTimeout

When I set the spark.dynamicAllocation.executorIdleTimeout to ten minutes or so, it is not respected. The underlying node will be powered off at an unexpected time, even though the Spark executor that was running on it was not idle for the full ten minutes. This is causing a number of problems, especially for features like "localCheckpoint". I would prefer it if the Synapse Yarn cluster would not power off the nodes that are being used for Spark jobs.

Can anyone share the intended behavior of spark.yarn.executor.decommission.enabled? Is it compatible with dynamic executor allocation? Is this a Microsoft enhancement to Spark? Is it found in the other Spark offerings (Fabric and HDInsight)? Should it be causing problems with the expected behavior of spark.dynamicAllocation.executorIdleTimeout? How do I get support when these properties are conflicting with each other? I have tried to open a couple cases with support (CSS) and they don't appear to have any information to share, despite the fact that this Yarn-related property seems to be something that was home-grown at Microsoft.

Any help would be very much appreciated.

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-17T22:16:37.2466667+00:00

Hello David Beavon,

I am sorry to hear that you are encountering difficulties with the spark undocumented property.

Can you please provide me the source URL from where you downloaded the spark config so that I can try to reproduce this issue on my end.

Additionally, if you could share the CSS case number, I can reach out to the case owner for more details.
David Beavon 996 Reputation points

2024-01-17T23:40:35.1+00:00

You simply need to create an autoscale cluster in synapse, and submit a job that supports dynamic executor allocation. In this scenario, you will find the undocumented spark property on the "environment" tab of the Spark UI.

The main reason this issue is impacting us is because our executors are being terminated prematurely (Synapse Spark isn't respecting the idle timeout as configured via spark.dynamicAllocation.executorIdleTimeout). If it wasn't for the premature termination of our executors, I would not be bothered by a Microsoft-specific configuration.

Feel free to open your own CSS case once you observe the undocumented property in Synapse. I would not like to share the customer-specific case number; I haven't been given an ICM number yet or I would share that with you.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-18T19:26:54.25+00:00

Hello David Beavon,

Thank you for the clear instructions. I can see this undocumented property on the envionment tab of the spark UI.

I am reaching out to my internal team for further clarification regarding this. I will get back to you as soon as I hear from them.
David Beavon 996 Reputation points

2024-01-19T01:18:53.35+00:00

@Bhargava-MSFT
I am currently a Synapse customer but the level of support on that platform can be hit-and-miss. Would you please find out if a similar property exists in the HDInsight implementation of Spark? If so then I will create a temporary resource over there, and a support case after I am able to recreate a similar problem. Hopefully the support from the HDInsight team is a bit better than it is on the Synapse side.
David Beavon 996 Reputation points

2024-01-19T01:20:50.6466667+00:00

@Bhargava-MSFT

Can you tell me if the same property exists on HDInsight?
If so then I may try to recreate over there as well.
My hope is that the support will be a bit better on a more mature Spark offering. On the Synapse side, the CSS support is sometimes hit-and-miss.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-19T20:51:19.0833333+00:00

Hello David Beavon, Based on my research HDInsights of spark uses Yarn as its resource manager. So, I believe the same property exists on the HDInsights as well. But I am reaching out to my internal team to confirm the same.
David Beavon 996 Reputation points

2024-01-24T23:39:28.14+00:00

@Bhargava-MSFT
Did you hear back from your internal team?

I tested HDInsight and I don't see the property over there. It may be specific to "Synapse" workspaces. I'm pretty unsatisfied with Synapse. These SaaS products that have bundled Spark are pretty poor when it comes to support (ie. in Fabric, and ADF, and Synapse there are repackaged implementations of Spark that are presented to customers in a way that makes Spark more accessible to "citizen developers"). While it may seem like a good idea to make Spark available to everyone, there are obviously some unfortunate trade-offs since these implementations can misbehave and they lack the support that would be needed to find out why.
David Beavon 996 Reputation points

2024-01-24T23:39:59.29+00:00

Any help would be appreciated.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-25T21:21:44.0866667+00:00

Hello David Beavon,

I apologize for the support experience you had. I contacted the Product group on this issue, but unfortunately, I haven't received any response from them. Since you already have an open case with CSS, they have the ability to create ICMs and collaborate directly with the product group. Our community support team has limited visibility on CSS cases and ICMs. I would recommend waiting for the CSS team to get back to you. If you are willing to share the CSS case number, I can reach out to the CSS case owner and expedite the request. I apologize again for not being helpful in this situation
David Beavon 996 Reputation points

2024-01-26T00:03:13.7433333+00:00

@Bhargava-MSFT Case is TrackingID#2401110040014664.

The primary question I have is why the YARN cluster in Synapse does not seem to respect the Spark idle timeouts (spark.dynamicAllocation.executorIdleTimeout).
It seems that the YARN implementation in Synapse is completely ignoring the Spark properties that govern executor timeouts. This seems pretty frustrating, considering that YARN on synapse is hidden from me, and also considering the fact that its whole purpose in life is to serve the needs of Spark but it doesn't do that very well.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-30T19:20:40.5233333+00:00

Hello David Beavon,, I apologize for the inconvenience. Currently, an ICM has been created and assigned to the Product Group to address your concerns. I will get back to you as soon as I hear from them.
David Beavon 996 Reputation points

2024-01-31T00:15:47.5266667+00:00

Thanks, please let me know.

As I mentioned, I tested HDInsight and I don't see the property over there. It may be specific to "Synapse" workspaces.

With HDInsight, I tested both the dynamically allocated executors AND the autoscale. In neither case did I see this undocumented property that appears to be home-grown at Microsoft. Ideally Microsoft would create documentation for any modifications they make to OSS Spark. The documentation for Synapse is a bit sparse; perhaps it is because the audience of that platform does not have much use for documentation!
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-02-06T19:54:05.38+00:00

Hello, Here is an update from the PG.

spark.yarn.executor.decommission.enabled is not Microsoft owned property. Microsoft works with open-source Spark and contributes to bug fixes/improvements. So, public documentation of Spark mostly applies to Microsoft Spark. This is because bug fixes identified by Microsoft shall have priority as we have flexibility to deploy to Spark in our code base before open-source spark does that.

This property does influence YARN. If the property is sent through spark configuration, "spark." need to be prefixed. In case of HDInsight, you have a way to update configs on YARN directly, in that case, you don't prepend "spark." for the property and update directly. Since Synapse do not have a way to update configs directly on YARN, we pass the properties thru Spark, hence we prepend "spark." for the YARN properties.
David Beavon 996 Reputation points

2024-02-07T16:43:16.1633333+00:00

Hi @Bhargava-MSFT

Thanks for the update.
The property, "spark.yarn.executor.decommission.enabled" is totally undocumented on the Internet and that is not something that would happen with OSS spark. It seems like something your Microsoft teams would do. FYI There are some other examples poorly documented properties in the home-grown spark from Microsoft (like the ones for "vegas" options for configuring the "intelligent cache" or whatever).

You are implying that "yarn.executor.decommission.enabled" is documented (without prepending spark). Of course that is not the case either. I already searched for every permutation of this property name and it seems clear that Microsoft doesn't want to explain it to anyone. Finally, the information about passing a customized version of this property to yarn via spark is directly conflicting with what I was told by a CSS engineer (Mousam). Can you give a name or ICM number to reference so that I can re-open discussion with him? Can you see how troubling it is when Microsoft customizes Spark with their own "flexible" changes, yet chooses not to explain them? This is not just about the documentation itself. My executors in synapse will suddenly die within moments of creation, and will ignore my idle timeouts. Yet customers are unable to investigate because (1) no visibility to review yarn, (2) lots of customizations as compared to OSS spark including custom properties, (3) missing documentation for custom properties, and (4) no way for customers to independently verify whether the sudden death of executors is "by design" as CSS and PG would lead you to believe.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-02-07T22:45:17.82+00:00

Hello David, I'm sorry to hear about all the trouble you're experiencing due to this undocumented property. Once again, I apologize for not being more helpful, as my scope is limited by the undocumented nature of this property. If you require further assistance from my end, please don't hesitate to ask or open a new thread for future support.

Here is the ICM number for your reference: 463973337 (opened and assigned to PG)
David Beavon 996 Reputation points

2024-03-23T21:26:41.5633333+00:00

Thanks @Bhargava-MSFT . We have been able to encourage the team to provide us with some public docs . I think you may have played a role in this but I'm not sure. I think this information applies to the "Fabric" side as well, and that might be the reason they finally prioritized the topic, and provided the needful docs.

1 answer

Your answer

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-17T22:16:37.2466667+00:00

Hello David Beavon,

I am sorry to hear that you are encountering difficulties with the spark undocumented property.

Can you please provide me the source URL from where you downloaded the spark config so that I can try to reproduce this issue on my end.

Additionally, if you could share the CSS case number, I can reach out to the case owner for more details.
David Beavon 996 Reputation points

2024-01-17T23:40:35.1+00:00

You simply need to create an autoscale cluster in synapse, and submit a job that supports dynamic executor allocation. In this scenario, you will find the undocumented spark property on the "environment" tab of the Spark UI.

The main reason this issue is impacting us is because our executors are being terminated prematurely (Synapse Spark isn't respecting the idle timeout as configured via spark.dynamicAllocation.executorIdleTimeout). If it wasn't for the premature termination of our executors, I would not be bothered by a Microsoft-specific configuration.

Feel free to open your own CSS case once you observe the undocumented property in Synapse. I would not like to share the customer-specific case number; I haven't been given an ICM number yet or I would share that with you.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-18T19:26:54.25+00:00

Hello David Beavon,

Thank you for the clear instructions. I can see this undocumented property on the envionment tab of the spark UI.

I am reaching out to my internal team for further clarification regarding this. I will get back to you as soon as I hear from them.
David Beavon 996 Reputation points

2024-01-19T01:18:53.35+00:00

@Bhargava-MSFT
I am currently a Synapse customer but the level of support on that platform can be hit-and-miss. Would you please find out if a similar property exists in the HDInsight implementation of Spark? If so then I will create a temporary resource over there, and a support case after I am able to recreate a similar problem. Hopefully the support from the HDInsight team is a bit better than it is on the Synapse side.
David Beavon 996 Reputation points

2024-01-19T01:20:50.6466667+00:00

@Bhargava-MSFT

Can you tell me if the same property exists on HDInsight?
If so then I may try to recreate over there as well.
My hope is that the support will be a bit better on a more mature Spark offering. On the Synapse side, the CSS support is sometimes hit-and-miss.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-19T20:51:19.0833333+00:00

Hello David Beavon, Based on my research HDInsights of spark uses Yarn as its resource manager. So, I believe the same property exists on the HDInsights as well. But I am reaching out to my internal team to confirm the same.
David Beavon 996 Reputation points

2024-01-24T23:39:28.14+00:00

@Bhargava-MSFT
Did you hear back from your internal team?

I tested HDInsight and I don't see the property over there. It may be specific to "Synapse" workspaces. I'm pretty unsatisfied with Synapse. These SaaS products that have bundled Spark are pretty poor when it comes to support (ie. in Fabric, and ADF, and Synapse there are repackaged implementations of Spark that are presented to customers in a way that makes Spark more accessible to "citizen developers"). While it may seem like a good idea to make Spark available to everyone, there are obviously some unfortunate trade-offs since these implementations can misbehave and they lack the support that would be needed to find out why.
David Beavon 996 Reputation points

2024-01-24T23:39:59.29+00:00

Any help would be appreciated.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-25T21:21:44.0866667+00:00

Hello David Beavon,

I apologize for the support experience you had. I contacted the Product group on this issue, but unfortunately, I haven't received any response from them. Since you already have an open case with CSS, they have the ability to create ICMs and collaborate directly with the product group. Our community support team has limited visibility on CSS cases and ICMs. I would recommend waiting for the CSS team to get back to you. If you are willing to share the CSS case number, I can reach out to the CSS case owner and expedite the request. I apologize again for not being helpful in this situation
David Beavon 996 Reputation points

2024-01-26T00:03:13.7433333+00:00

@Bhargava-MSFT Case is TrackingID#2401110040014664.

The primary question I have is why the YARN cluster in Synapse does not seem to respect the Spark idle timeouts (spark.dynamicAllocation.executorIdleTimeout).
It seems that the YARN implementation in Synapse is completely ignoring the Spark properties that govern executor timeouts. This seems pretty frustrating, considering that YARN on synapse is hidden from me, and also considering the fact that its whole purpose in life is to serve the needs of Spark but it doesn't do that very well.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-01-30T19:20:40.5233333+00:00

Hello David Beavon,, I apologize for the inconvenience. Currently, an ICM has been created and assigned to the Product Group to address your concerns. I will get back to you as soon as I hear from them.
David Beavon 996 Reputation points

2024-01-31T00:15:47.5266667+00:00

Thanks, please let me know.

As I mentioned, I tested HDInsight and I don't see the property over there. It may be specific to "Synapse" workspaces.

With HDInsight, I tested both the dynamically allocated executors AND the autoscale. In neither case did I see this undocumented property that appears to be home-grown at Microsoft. Ideally Microsoft would create documentation for any modifications they make to OSS Spark. The documentation for Synapse is a bit sparse; perhaps it is because the audience of that platform does not have much use for documentation!
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-02-06T19:54:05.38+00:00

Hello, Here is an update from the PG.

spark.yarn.executor.decommission.enabled is not Microsoft owned property. Microsoft works with open-source Spark and contributes to bug fixes/improvements. So, public documentation of Spark mostly applies to Microsoft Spark. This is because bug fixes identified by Microsoft shall have priority as we have flexibility to deploy to Spark in our code base before open-source spark does that.

This property does influence YARN. If the property is sent through spark configuration, "spark." need to be prefixed. In case of HDInsight, you have a way to update configs on YARN directly, in that case, you don't prepend "spark." for the property and update directly. Since Synapse do not have a way to update configs directly on YARN, we pass the properties thru Spark, hence we prepend "spark." for the YARN properties.
David Beavon 996 Reputation points

2024-02-07T16:43:16.1633333+00:00

Hi @Bhargava-MSFT

Thanks for the update.
The property, "spark.yarn.executor.decommission.enabled" is totally undocumented on the Internet and that is not something that would happen with OSS spark. It seems like something your Microsoft teams would do. FYI There are some other examples poorly documented properties in the home-grown spark from Microsoft (like the ones for "vegas" options for configuring the "intelligent cache" or whatever).

You are implying that "yarn.executor.decommission.enabled" is documented (without prepending spark). Of course that is not the case either. I already searched for every permutation of this property name and it seems clear that Microsoft doesn't want to explain it to anyone. Finally, the information about passing a customized version of this property to yarn via spark is directly conflicting with what I was told by a CSS engineer (Mousam). Can you give a name or ICM number to reference so that I can re-open discussion with him? Can you see how troubling it is when Microsoft customizes Spark with their own "flexible" changes, yet chooses not to explain them? This is not just about the documentation itself. My executors in synapse will suddenly die within moments of creation, and will ignore my idle timeouts. Yet customers are unable to investigate because (1) no visibility to review yarn, (2) lots of customizations as compared to OSS spark including custom properties, (3) missing documentation for custom properties, and (4) no way for customers to independently verify whether the sudden death of executors is "by design" as CSS and PG would lead you to believe.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-02-07T22:45:17.82+00:00

Hello David, I'm sorry to hear about all the trouble you're experiencing due to this undocumented property. Once again, I apologize for not being more helpful, as my scope is limited by the undocumented nature of this property. If you require further assistance from my end, please don't hesitate to ask or open a new thread for future support.

Here is the ICM number for your reference: 463973337 (opened and assigned to PG)
David Beavon 996 Reputation points

2024-03-23T21:26:41.5633333+00:00

Thanks @Bhargava-MSFT . We have been able to encourage the team to provide us with some public docs . I think you may have played a role in this but I'm not sure. I think this information applies to the "Fabric" side as well, and that might be the reason they finally prioritized the topic, and provided the needful docs.

Answer 1

David Beavon 996

There is now a limited amount of documentation available from the Microsoft Spark team. My support ticket with that team is being closed.

See:

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-autoscale#prepare-for-scaling-down

By default, spark.yarn.executor.decommission.enabled is set to true, enabling the automatic shutdown of underutilized nodes to optimize compute efficiency. If less aggressive scaling down is preferred, this configuration can be set to false.

User's image

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2024-03-25T16:55:00.98+00:00

David Beavon,

Thank you for sharing the details and the document reference URL

Share via

Purpose of "spark.yarn.executor.decommission.enabled" in Synapse Spark

1 answer

Your answer