Why is Microsoft not Approving Azure Spot vCPU quota increases, and will Spot Virtual Machines ever work again?

Timo Lindqvist 16

I have not been able to increase Spot vCPU quotas in some regions for months now. I opened a support request and got the response below. What I do not understand is why Microsoft is controlling spot consumption by limiting Spot vCPU quota increases? Isn't the whole point of Spot pricing that if Azure encounters "high consumption", the spot prices should increase, evicting workloads that are not prepared to pay a higher price? The mechanism seems to a) not work as intended b) not work as documented in https://learn.microsoft.com/en-us/azure/virtual-machines/spot-vms. Is there any plan or timelines on when Azure Spot Virtual Machines start working again (meaning Spot vCPU quotas can be increased)?

Due to very high rates of Spot consumption, Microsoft is unable to approve additional quota at this time. The” high consumption” is overall with Azure, not your personal consumption on this subscription or your account. Microsoft has decided to limit SPOT access for new requests; however, you’ll still be able to use what you have.

As SPOT is now restricted for new access and increases, we can only recommend that you go with dedicated ARM cores for those SKUs that are needed in a different region.

We have no path for exceptions at this time. Please let me know if you have any questions, and if nothing else is needed in this case, let us know and we will archive it.

Timo Lindqvist 16 Reputation points

2022-11-10T08:09:19.703+00:00

It looks like Azure "spot" pricing is adjusted only once per month. When there is less spot capacity available, does Azure take the maximum price set for the VM into account at all? During the 12h period shown, this VM was evicted even if it had the maximum price way above the spot price. Immediately after eviction, the VM could be restarted, indicating that all the capacity had not disappeared. How does Azure choose the spot workloads to be evicted? If the maximum price must be set to -1 for not to become evicted, how does Azure choose between the workloads that have maximum price set to -1. Randomly?
Timo Lindqvist 16 Reputation points

2022-11-10T08:15:52.437+00:00

Another hypothesis is that Azure considers that maximum price only when the workload is started. This would confirm that Azure evicts workloads randomly. And, as spot prices are adjusted only once per month, imply that everybody should set maximum price to -1 and automatically restart their workloads after eviction. But this means the spot prices do nothing of the sort you would expect from "spot pricing". It would be nice if somebody who knows how this actually works would help us learn how the system works, thx!
Timo Lindqvist 16 Reputation points

2022-11-10T08:31:14.853+00:00

From this question I'm starting to converge into a mental model where Azure handles capacity and price totally separate from each other. Eviction based on price can only happen once a month when they adjust the spot price: workloads that have maximum price set to less than the new price will be evicted. And the process of evicting workloads when all workloads cannot be supported does not consider pricing at all - it's just random (may take into account some hidden heuristics like uptime ofc).

But to be frank, this has nothing to do with "spot pricing" and calling this system "Spot VM" is wildly misleading. It's just VMs without an SLA, with pricing adjusted monthly.

https://learn.microsoft.com/en-us/answers/questions/1018525/azure-vmss-with-spot.html
Timo Lindqvist 16 Reputation points

2022-11-10T08:38:17.643+00:00

I would be really interested to see a comment from the person who is responsible for the business plan for the "Spot VM" service. With the current way things work, Microsoft loses significant revenue every month as they let the available spot capacity be used for the semi-fixed monthly price, without adjusting the price upwards every 5 minutes based on actual capacity. It just does not make business sense to evict workloads randomly when offered workloads that want to pay more for the capacity.
Timo Lindqvist 16 Reputation points

2022-11-10T08:59:20.61+00:00
I suggest a rewrite of the "Eviction type" UI text in VM Configuration > Azure Spot to more clearly reflect the impact of these controls.

Capacity only: Your virtual machine may be randomly selected for eviction when Azure excess capacity is reduced

Price or capacity: Your virtual machine may be randomly selected for eviction when Azure's excess capacity is reduced, or when the resource price changes and exceeds your specified max price