NIC models used in Linux VM change randomly

Question

NIC models used in Linux VM change randomly

flexiWAN Networks 21

Hello,

We notice that, from time to time, when rebooting our Azure Linux VM with multiple NICs attached, the NIC model detected on the guest OS for all of them change. This is our scenario:

Azure VM type Standard DS3 v2
Running Linux (Ubuntu 18.04 LTS)
4 NICs attached to the VM with accelerated network enabled on all of them

According to the information provided by lspci tool, we have identified the following two NIC models:

Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80) --> The one that is used most of the times, and the one we need for our use case.
Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] --> The one that, from time to time, Azure decides to use within the VM.

We have not modified any setting on the above scenario, however, this change happens without any apparent reaseon. Our qestions are the following:

Is this behavior expected? If so, can you please explain the reason behind doing this change in order to better understand what's happening under the hood?
Is there any method provided by Azure that we can use on our VM to stick to the same NIC model all the time? If not, would using a dedicated host solve the problem?

We are asking these questions because our use case relies on using the same NIC model inside the VM, even if it needs to be rebooted from time to time. So, changing dynamically the NIC model prevents us to progress our work on Azure, unfortunately.

Thank you so much in advance.

KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator

2022-07-12T11:42:10.51+00:00
Hi @flexiWAN Networks ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand you are leveraging accelerated networking and you notice that the NIC models are switching between,

Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80) ----------> preferred.

Ethernet controller: Mellanox Technologies MT27500/MT27520 Family

This is by design.

To explain why this is happening,

When you stop the virtual machine it will be deallocated from the current physical server and when you start the machine again it will be started/allocated on the new physical server in the data center and that particular machine will have a new hardware MAC address.

If you shut down the VM through the OS, the VM is stopped and not deallocated. In that case, there would be no ghosted NIC because the lease is never lost for the existing hardware. The VM would start on the same hardware/physical server in the data center, have the same resources, and no ghosted NIC would appear.
-Refer: https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-overview, for the difference in the design for accelerated and normal networking.

Refer : https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-how-it-works
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator

2022-07-12T11:42:10.51+00:00
Hi @flexiWAN Networks ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand you are leveraging accelerated networking and you notice that the NIC models are switching between,

Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80) ----------> preferred.

Ethernet controller: Mellanox Technologies MT27500/MT27520 Family

This is by design.

To explain why this is happening,

When you stop the virtual machine it will be deallocated from the current physical server and when you start the machine again it will be started/allocated on the new physical server in the data center and that particular machine will have a new hardware MAC address.

If you shut down the VM through the OS, the VM is stopped and not deallocated. In that case, there would be no ghosted NIC because the lease is never lost for the existing hardware. The VM would start on the same hardware/physical server in the data center, have the same resources, and no ghosted NIC would appear.
-Refer: https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-overview, for the difference in the design for accelerated and normal networking.

Refer : https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-how-it-works
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator

2022-07-12T11:56:44.043+00:00
(Continued in comments)

As stated above, we will not have any control over the physical host where the VM is going to get deployed.

With respect to using Dedicated Host,

With Dedicated Host, again, you might get any physical server allocated and there is no guarantee that this server is going to use Mellanox Technologies MT27710 Family .

You may delete and recreate the Host until you land in MT27710 Family (not a good practice)

However, usage of Dedicated Host introduces another challenge - platform updates and maintenance

During maintenance events, there is a possibility that you are given a different physical server (still isolated from other subscriptions).
I am currently checking internally if the host update is going to change the physical server or not.
Kindly give me some time, I shall update the thread once I have an update.

Thanks,
Kapil.
flexiWAN Networks 21 Reputation points

2022-07-12T13:15:15.033+00:00

Hi anonymous user-MSFT,

Thank you so much for your detailed answer and additional explanations. You understood 100% our situation, so your answer is perfect in that sense. I had already come up with the documentation that you point out in your reply, and I understood that once you stopped a VM, Azure is deallocates it from the physical server where it run previously, so the next time the user start it again, Azure can re-allocate it into another server where other physical NICs may be used. I wanted to have confirmation from other experts that my previous understandings were right.

Howerver, it's interesting the point that you mention about start/stop the VM from Azure management vs. from withtin the guest OS. Does it mean that, if we stop the VM through its guest OS from now on, we can prevent the NIC change model behavior? Can we stick to this workaround for now?

Regarding the usage of a Dedicated Host to overcome this situation, I understand your points, make lot of sense. Of course, take your time to get the missing information about platform updates and maintenance for Dedicated Hosts.

Once again, many thanks for your help; we appreciate your explanations!
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator

2022-07-13T14:54:23.06+00:00
Hi @flexiWAN Networks ,

You are most welcome.

To answer your question,
" if we stop the VM through its guest OS from now on, we can prevent the NIC change model behavior?"

This is not a good approach.

Yes, you may use it as a temporary workaround for now

However, the challenge is platform updates and maintenance.

Since we have little control over when these updates are going to happen, there is a chance that the VMs are spun up in a new Physical host altogether(which does not have the required NIC) while the current physical host undergoes maintenance even when the VM is running.

Refer to https://learn.microsoft.com/en-us/azure/virtual-machines/maintenance-and-updates

In fact, currently, I am trying to get some details if this is the same behavior for Azure Dedicated Host or not.

P.S :

Please be aware that if you are not deallocating the VM, i.e., not turning it off via Portal, you will be charged for the VM as if it were in a started state.

This is because, the underlying hardware are reserved for your VM and the reserve was never released.

Thanks,
Kapil
flexiWAN Networks 21 Reputation points

2022-07-14T09:24:52.317+00:00

Hello again anonymous user-MSFT,

Many thanks for keeping us informed. For now we are going to use the workaournd of just stop the VM (not deallocate it) to have the same environment where work. Of course we will need in the future support for both NIC models, but first we want to focus our efforts on Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80) compatibility. Then, we will switch to the Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] NIC model. Although this will take more time, for sure it's the best solution in the future. And many thanks also to highlight the charging method based on VM state, we were aware of that.

Thank you so much!
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator

2022-07-14T17:59:58.453+00:00

Hi @flexiWAN Networks ,

Greetings.
I got some updates internally with respect to the dedicated Host and have posted a summary of our entire discussion as Answer.

Please don’t forget to click "Accept the answer" and Up-Vote for the same, which might be beneficial to other community members reading this thread. And, if you have any further queries, do let us know and I shall try my best to address them.

Thanks,
Kapil

Accepted answer

2 additional answers

Your answer

KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator

2022-07-12T11:42:10.51+00:00

Hi @flexiWAN Networks ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand you are leveraging accelerated networking and you notice that the NIC models are switching between,

Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80) ----------> preferred.

Ethernet controller: Mellanox Technologies MT27500/MT27520 Family

This is by design.

To explain why this is happening,

When you stop the virtual machine it will be deallocated from the current physical server and when you start the machine again it will be started/allocated on the new physical server in the data center and that particular machine will have a new hardware MAC address.

If you shut down the VM through the OS, the VM is stopped and not deallocated. In that case, there would be no ghosted NIC because the lease is never lost for the existing hardware. The VM would start on the same hardware/physical server in the data center, have the same resources, and no ghosted NIC would appear.
-Refer: https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-overview, for the difference in the design for accelerated and normal networking.

Refer : https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-how-it-works
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator

2022-07-12T11:56:44.043+00:00

(Continued in comments)

As stated above, we will not have any control over the physical host where the VM is going to get deployed.

With respect to using Dedicated Host,

With Dedicated Host, again, you might get any physical server allocated and there is no guarantee that this server is going to use Mellanox Technologies MT27710 Family .

You may delete and recreate the Host until you land in MT27710 Family (not a good practice)

However, usage of Dedicated Host introduces another challenge - platform updates and maintenance

During maintenance events, there is a possibility that you are given a different physical server (still isolated from other subscriptions).
I am currently checking internally if the host update is going to change the physical server or not.
Kindly give me some time, I shall update the thread once I have an update.

Thanks,
Kapil.
flexiWAN Networks 21 Reputation points

2022-07-12T13:15:15.033+00:00

Hi anonymous user-MSFT,

Thank you so much for your detailed answer and additional explanations. You understood 100% our situation, so your answer is perfect in that sense. I had already come up with the documentation that you point out in your reply, and I understood that once you stopped a VM, Azure is deallocates it from the physical server where it run previously, so the next time the user start it again, Azure can re-allocate it into another server where other physical NICs may be used. I wanted to have confirmation from other experts that my previous understandings were right.

Howerver, it's interesting the point that you mention about start/stop the VM from Azure management vs. from withtin the guest OS. Does it mean that, if we stop the VM through its guest OS from now on, we can prevent the NIC change model behavior? Can we stick to this workaround for now?

Regarding the usage of a Dedicated Host to overcome this situation, I understand your points, make lot of sense. Of course, take your time to get the missing information about platform updates and maintenance for Dedicated Hosts.

Once again, many thanks for your help; we appreciate your explanations!
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator

2022-07-13T14:54:23.06+00:00

Hi @flexiWAN Networks ,

You are most welcome.

To answer your question,
" if we stop the VM through its guest OS from now on, we can prevent the NIC change model behavior?"

This is not a good approach.

Yes, you may use it as a temporary workaround for now

However, the challenge is platform updates and maintenance.

Since we have little control over when these updates are going to happen, there is a chance that the VMs are spun up in a new Physical host altogether(which does not have the required NIC) while the current physical host undergoes maintenance even when the VM is running.

Refer to https://learn.microsoft.com/en-us/azure/virtual-machines/maintenance-and-updates

In fact, currently, I am trying to get some details if this is the same behavior for Azure Dedicated Host or not.

P.S :

Please be aware that if you are not deallocating the VM, i.e., not turning it off via Portal, you will be charged for the VM as if it were in a started state.

This is because, the underlying hardware are reserved for your VM and the reserve was never released.

Thanks,
Kapil
flexiWAN Networks 21 Reputation points

2022-07-14T09:24:52.317+00:00

Hello again anonymous user-MSFT,

Many thanks for keeping us informed. For now we are going to use the workaournd of just stop the VM (not deallocate it) to have the same environment where work. Of course we will need in the future support for both NIC models, but first we want to focus our efforts on Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80) compatibility. Then, we will switch to the Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] NIC model. Although this will take more time, for sure it's the best solution in the future. And many thanks also to highlight the charging method based on VM state, we were aware of that.

Thank you so much!
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator

2022-07-14T17:59:58.453+00:00

Hi @flexiWAN Networks ,

Greetings.
I got some updates internally with respect to the dedicated Host and have posted a summary of our entire discussion as Answer.

Please don’t forget to click "Accept the answer" and Up-Vote for the same, which might be beneficial to other community members reading this thread. And, if you have any further queries, do let us know and I shall try my best to address them.

Thanks,
Kapil

Answer 1

Hi @flexiWAN Networks ,

I am summarizing our discussion and posting it as a single answer as it might be beneficial to other community members reading this thread.

You had a specific requirement that you wanted your Accelerated Networking VM to get hosted in a specific Physical server with Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] NIC.

**To Explain why this is happening, **

When you stop the virtual machine, it will be deallocated from the current physical server and when you start the machine again it will be started/allocated on the new physical server in the data center and that particular machine will have a new hardware MAC address.
So, for every reboot, the VM may land in any Host and there is no guarantee that it would land on X-4 family
Currently, there is no mechanism to specify the host where a VM would land in Azure
Refer: https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-how-it-works
You wanted to know if the same can be accomplished via Azure Dedicated Host

Now, with respect to Azure Dedicated Host,

With Azure Dedicated Host, yes, this can be achieved.
Once you land up in a Host with X-4 family, the host will not change, even for Host maintenance and updates
There is also a mechanism to control the maintenance events
Refer : https://learn.microsoft.com/en-us/azure/virtual-machines/maintenance-configurations
https://learn.microsoft.com/en-us/azure/virtual-machines/dedicated-hosts#maintenance-control
The only limitation is that if there is a node failure, and if you have disabled autoReplaceOnFailure, the VMs will not self-heal
https://learn.microsoft.com/en-us/azure/virtual-machines/dedicated-hosts
-In terms of host failure, if you want to recover the host, it’s possible that the host could resolve itself and move back to a healthy state. Otherwise, you will need to provision a new host (which would be deployed on new underlying hardware) and migrate the VMs from their old host onto the new host. This would require downtime.
However, the chance of this happening is comparatively very low, and you can work around this by having any High Availability set up.

I hope this provides you with more clarity on your question and how you may achieve this.

Thanks,
Kapil

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.

flexiWAN Networks 21 Reputation points

2022-07-19T19:18:34.623+00:00

Hi anonymous user-MSFT,

Thank you so much for summarizing the answer and updating the informatin regarding dedicated hosts. Marked your asnwer as accepted and give you 5 stars, too.

Regards.
KapilAnanth-MSFT 49,611 Reputation points Microsoft Employee Moderator

2022-07-20T04:41:22.377+00:00

Hi @flexiWAN Networks ,

Thank you.
Your feedback is highly appreciated and welcomed.

Best Regards,
Kapil.

Answer 2

Hi @flexiWAN Networks ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand you are leveraging accelerated networking and you notice that the NIC models are switching between,

Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80) ----------> preferred.
Ethernet controller: Mellanox Technologies MT27500/MT27520 Family

This is by design.

To explain why this is happening,

When you stop the virtual machine it will be deallocated from the current physical server and when you start the machine again it will be started/allocated on the new physical server in the data center and that particular machine will have a new hardware MAC address.
If you shut down the VM through the OS, the VM is stopped and not deallocated. In that case, there would be no ghosted NIC because the lease is never lost for the existing hardware. The VM would start on the same hardware/physical server in the data center, have the same resources, and no ghosted NIC would appear.
-Refer: https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-overview, for the difference in the design for accelerated and normal networking.

Refer : https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-how-it-works

Answer 3

Andreas Baumgarten 123.6K MVP Volunteer Moderator

Hi @flexiWAN Networks ,

my guess, I am not 100% sure, the reason for seeing different types of NICs is related to different NIC types of the Azure hosts the VM is running on.

This might be the same "issue" if you move the VMs on a dedicated host. The main benefit of running VMs on a dedicated host in Azure is that the host is "reserved" for your organization. As far as I know, a dedicated host does not guarantee any specific hardware setup or device type.

As far as I know there is no method/option available by default to stick with the same NIC model of a VM on an Azure host.

----------

(If the reply was helpful please don't forget to upvote and/or accept as answer, thank you)

Regards
Andreas Baumgarten

flexiWAN Networks 21 Reputation points

2022-07-12T12:58:33.983+00:00

Hi AndreasBaumgarten,

First fo all many thanks for invest time on reading and answer our questions. My guess was exactly the same as you mentioned in your answer, but I wanted to also have confirmation from other experts in Microsoft Azure.

I will leave the question open in case other members want to provide additional information on that.

Once again , many thanks!

Share via

NIC models used in Linux VM change randomly

2 additional answers

Your answer