Edit

Prepare Linux for imaging in Azure

Applies to: ✔️ Linux VMs ✔️ Flexible scale sets

The Azure platform service-level agreement (SLA) applies to virtual machines (VMs) running the Linux operating system only when you're using one of the endorsed distributions. For endorsed distributions, Azure Marketplace provides preconfigured Linux images. For more information, see:

All other distributions running on Azure, including community-supported and non-endorsed distributions, have some prerequisites.

This article focuses on general guidance for running your Linux distribution on Azure. This article can't be comprehensive, because every distribution is different. Even if you meet all the criteria that this article describes, you might need to significantly tweak your Linux system for it to run properly.

General Linux installation notes

Important

The remote NVMe preparation steps in this article are optional. Complete them only if you're creating an image that you intend to deploy with a remote NVMe disk controller. If the target VM uses SCSI, skip every step labeled for remote NVMe and retain the standard settings. Remote NVMe requires a Gen2 image and a supported operating system version. Verify the exact target VM size's DiskControllerTypes capability and review Supported OS images for remote NVMe. Don't infer the controller type only from the VM series generation.

  • Azure doesn't support the Hyper-V virtual hard disk (VHDX) format. Azure supports only fixed VHD. You can convert the disk to VHD format by using Hyper-V Manager or the Convert-VHD cmdlet. If you're using VirtualBox, select Fixed size rather than the default (Dynamically allocated) when you're creating the disk.

  • Azure supports Gen1 (BIOS boot) and Gen2 (UEFI boot) virtual machines.

  • The virtual file allocation table (VFAT) kernel module must be enabled in the kernel.

  • The maximum size allowed for the VHD is 1,023 GB.

  • When you're installing the Linux system, we recommend that you use standard partitions rather than Logical Volume Manager (LVM). LVM is the default for many installations.

    Using standard partitions will avoid LVM name conflicts with cloned VMs, particularly if an OS disk is ever attached to another identical VM for troubleshooting. You can use LVM or RAID on data disks.

  • Kernel support for mounting user-defined function (UDF) file systems is necessary. At first boot on Azure, the provisioning configuration is passed to the Linux VM via UDF-formatted media that are attached to the guest. The Azure Linux agent must mount the UDF file system to read its configuration and provision the VM.

  • Linux kernel versions earlier than 2.6.37 don't support Non-Uniform Memory Access (NUMA) on Hyper-V with larger VM sizes. This issue primarily affects older distributions that use the upstream Red Hat 2.6.32 kernel. It was fixed in Red Hat Enterprise Linux (RHEL) 6.6 (kernel-2.6.32-504).

    Systems running custom kernels older than 2.6.37, or RHEL-based kernels older than 2.6.32-504, must set the boot parameter numa=off on the kernel command line in grub.conf. For more information, see Red Hat KB 436883.

  • Don't configure a swap partition on the OS disk. You can configure the Linux agent to create a swap file on the temporary resource disk, as described later in this article.

  • All VHDs on Azure must have a virtual size aligned to 1 MB (1024 x 1024 bytes). When you're converting from a raw disk to VHD, ensure that the raw disk size is a multiple of 1 MB before conversion, as described later in this article.

  • Use the most up-to-date distribution version, packages, and software.

  • Remove users and system accounts, public keys, sensitive data, unnecessary software, and applications.

Note

Cloud-init version 21.2 or later removes the UDF requirement. But without the udf module enabled, the CD-ROM won't mount during provisioning, which prevents the custom data from being applied. A workaround is to apply user data. However, unlike custom data, user data isn't encrypted. For more information, see User data formats in the cloud-init documentation.

Install kernel modules without Hyper-V

Azure runs on the Hyper-V hypervisor, so Linux requires certain kernel modules to run in Azure. If you have a VM that was created outside Hyper-V, the Linux installers might not include the drivers for Hyper-V in the initial RAM disk (initrd or initramfs), unless the VM detects that it's running in a Hyper-V environment.

When you're using a different virtualization system (such as VirtualBox or KVM) to prepare your Linux image, you might need to rebuild initrd so that at least the hv_vmbus and hv_storvsc kernel modules are available on the initial RAM disk. This known issue is for systems based on the upstream Red Hat distribution, and possibly others.

The mechanism for rebuilding the initrd or initramfs image can vary, depending on the distribution. Consult your distribution's documentation or support for the proper procedure. Here's one example for rebuilding initrd by using the mkinitrd utility:

  1. Back up the existing initrd image:

    cd /boot
    sudo cp initrd-`uname -r`.img  initrd-`uname -r`.img.bak
    
  2. Rebuild initrd by using the hv_vmbus and hv_storvsc kernel modules:

    sudo mkinitrd --preload=hv_storvsc --preload=hv_vmbus -v -f initrd-`uname -r`.img `uname -r`
    

Resize VHDs

VHD images on Azure must have a virtual size aligned to 1 MB. Typically, VHDs created through Hyper-V are aligned correctly. If the VHD isn't aligned correctly, you might get an error message similar to the following example when you try to create an image from your VHD:

The VHD http://<mystorageaccount>.blob.core.windows.net/vhds/MyLinuxVM.vhd has an unsupported virtual size of 21475270656 bytes. The size must be a whole number (in MBs).

In this case, resize the VM by using either the Hyper-V Manager console or the Resize-VHD PowerShell cmdlet. If you aren't running in a Windows environment, we recommend using qemu-img to convert (if needed) and resize the VHD.

Note

There's a known bug in qemu-img for QEMU version 2.2.1 and some later versions that results in an improperly formatted VHD. The issue was fixed in QEMU 2.6. We recommend using version 2.2.0 or earlier, or using version 2.6 or later.

  1. Resizing the VHD directly by using tools such as qemu-img or vbox-manage might result in an unbootable VHD. We recommend first converting the VHD to a raw disk image by using the following code.

    If the VM image was created as a raw disk image, you can skip this step. Creating the VM image as a raw disk image is the default in some hypervisors, such as KVM.

    sudo qemu-img convert -f vpc -O raw MyLinuxVM.vhd MyLinuxVM.raw
    
  2. Calculate the required size of the disk image so that the virtual size is aligned to 1 MB. The following Bash shell script uses qemu-img info to determine the virtual size of the disk image, and then calculates the size to the next 1 MB:

    rawdisk="MyLinuxVM.raw"
    vhddisk="MyLinuxVM.vhd"
    
    MB=$((1024*1024))
    size=$(qemu-img info -f raw --output json "$rawdisk" | \
    gawk 'match($0, /"virtual-size": ([0-9]+),/, val) {print val[1]}')
    
    rounded_size=$(((($size+$MB-1)/$MB)*$MB))
    
    echo "Rounded Size = $rounded_size"
    
  3. Resize the raw disk by using $rounded_size:

    sudo qemu-img resize MyLinuxVM.raw $rounded_size
    
  4. Convert the raw disk back to a fixed-size VHD:

    sudo qemu-img convert -f raw -o subformat=fixed,force_size -O vpc MyLinuxVM.raw MyLinuxVM.vhd
    

    Or, with QEMU versions before 2.6, remove the force_size option:

    sudo qemu-img convert -f raw -o subformat=fixed -O vpc MyLinuxVM.raw MyLinuxVM.vhd
    

Linux kernel requirements

The Linux Integration Services (LIS) drivers for Hyper-V and Azure are contributed directly to the upstream Linux kernel. Many distributions that include a recent Linux kernel version (such as 3.x) have these drivers available already, or otherwise provide backported versions of these drivers with their kernels.

LIS drivers are constantly being updated in the upstream kernel with new fixes and features. When possible, we recommend running an endorsed distribution that includes these fixes and updates.

If you're running a variant of RHEL versions 6.0 to 6.3, you need to install the latest LIS drivers for Hyper-V. Beginning with RHEL 6.4+ (and derivatives), the LIS drivers are already included with the kernel, so you don't need additional installation packages.

If a custom kernel is required, we recommend a recent kernel version (such as 3.8+). For distributions or vendors that maintain their own kernel, you need to regularly backport the LIS drivers from the upstream kernel to your custom kernel.

Even if you're already running a relatively recent kernel version, we highly recommend keeping track of any upstream fixes in the LIS drivers and backporting them as needed. The locations of the LIS driver source files are specified in the MAINTAINERS file in the Linux kernel source tree:

    F:    arch/x86/include/asm/mshyperv.h
    F:    arch/x86/include/uapi/asm/hyperv.h
    F:    arch/x86/kernel/cpu/mshyperv.c
    F:    drivers/hid/hid-hyperv.c
    F:    drivers/hv/
    F:    drivers/input/serio/hyperv-keyboard.c
    F:    drivers/net/hyperv/
    F:    drivers/scsi/storvsc_drv.c
    F:    drivers/video/fbdev/hyperv_fb.c
    F:    include/linux/hyperv.h
    F:    tools/hv/

The VM's active kernel must include the following patches. This list can't be complete for all distributions.

Azure Linux Agent

The Azure Linux Agent (waagent) provisions a Linux virtual machine in Azure. You can get the latest version, report problems, or submit pull requests at the Linux Agent GitHub repo.

Here are some considerations for using the Azure Linux Agent:

  • The Linux agent is released under the Apache 2.0 license. Many distributions already provide .rpm or .deb packages for the agent. You can easily install and update these packages.
  • The Azure Linux Agent requires Python v2.6+.
  • The agent also requires the python-pyasn1 module. Most distributions provide this module as a separate package to be installed.
  • In some cases, the Azure Linux Agent might not be compatible with NetworkManager. Many of the packages (.rpm or .deb) provided by distributions configure NetworkManager as a conflict to the waagent package. In these cases, the agent will uninstall NetworkManager when you install the Linux agent package.
  • The Azure Linux Agent must be at or above the minimum supported version.

Note

Make sure the udf and vfat modules are enabled. Disabling the udf module will cause a provisioning failure. Disabling the vfat module will cause both provisioning and boot failures. Cloud-init version 21.2 or later can provision VMs without requiring UDF if both of these conditions exist:

  • You created the VM by using SSH public keys and not passwords.
  • You didn't provide any custom data.

General Linux system requirements

  1. Modify the kernel boot line in GRUB or GRUB2 to include the following parameters, so that all console messages are sent to the first serial port. These messages can assist Azure support with debugging any issues.

    GRUB_CMDLINE_LINUX="rootdelay=300 console=ttyS0 earlyprintk=ttyS0 net.ifnames=0"
    

    We also recommend removing the following parameters if they exist:

    rhgb quiet crashkernel=auto
    

    Graphical and quiet boot aren't useful in a cloud environment, where you want all logs sent to the serial port. You can leave the crashkernel option configured if needed, but this parameter reduces the amount of available memory in the VM by at least 128 MB. Reducing available memory might be problematic for smaller VM sizes.

  2. After you finish editing /etc/default/grub, rebuild the GRUB configuration. Run only the command appropriate for your distribution.

    For distros based on Debian or Ubuntu, run:

    sudo update-grub
    

    For distributions that provide grub2-mkconfig, such as RHEL, Oracle Linux, and SUSE, run:

    sudo grub2-mkconfig -o /boot/grub2/grub.cfg
    

    On distributions that provide grub-mkconfig instead, run:

    sudo grub-mkconfig -o /boot/grub/grub.cfg
    

    The GRUB configuration path can differ for UEFI installations. If your distribution uses another path, use the path documented for that distribution.

  3. Add the Hyper-V modules to the initial RAM disk. Run only the procedure that matches the tools used by your distribution.

    1. For a dracut-based distribution, run:
    cd /boot
    sudo cp initramfs-<kernel-version>.img <kernel-version>.img.bak
    sudo dracut -f -v initramfs-<kernel-version>.img <kernel-version> --add-drivers "hv_vmbus hv_netvsc hv_storvsc"
    
    1. For a distribution that uses mkinitramfs, run this procedure instead of the dracut procedure:
    cd /boot
    sudo cp initrd.img-<kernel-version> initrd.img-<kernel-version>.bak
    sudo mkinitramfs -o initrd.img-<kernel-version> <kernel-version> --with=hv_vmbus,hv_netvsc,hv_storvsc
    
  4. Only if you're creating an image for a remote NVMe disk controller, configure and verify NVMe boot support. Otherwise, skip this step. The exact commands can vary by distribution.

    1. Rebuild initramfs with the NVMe drivers by using the method for your distribution.

      For a dracut-based distribution, make the NVMe drivers persistent in future initramfs images and rebuild the current initramfs:

      sudo mkdir -p /etc/dracut.conf.d
      printf '%s\n' 'add_drivers+=" nvme nvme_core "' | sudo tee /etc/dracut.conf.d/azure-nvme.conf
      sudo dracut --force
      

      For an initramfs-tools-based distribution, make the NVMe drivers persistent and rebuild initramfs:

      printf '%s\n' nvme nvme_core | sudo tee -a /etc/initramfs-tools/modules
      sudo update-initramfs -u -k all
      
    2. Add nvme_core.io_timeout=240 to the existing kernel command line in /etc/default/grub, and then rebuild the GRUB configuration by using the command appropriate for your distribution.

    3. Verify that both NVMe driver modules are available on the system. Run both commands:

      modinfo nvme
      modinfo nvme_core
      

      Each command must return information about its module. If either module isn't found, install or enable it by following your distribution's documentation before you continue.

    4. Verify that both NVMe drivers are included in initramfs. Run only the command for your distribution.

      For a dracut-based distribution, run:

      sudo lsinitrd /boot/initramfs-$(uname -r).img | grep -E 'nvme(_core)?\.ko'
      

      For an initramfs-tools-based distribution, run:

      sudo lsinitramfs /boot/initrd.img-$(uname -r) | grep -E 'nvme(_core)?\.ko'
      

      The output must show both the nvme and nvme_core drivers. If it doesn't, consult your distribution's documentation and rebuild the initial RAM disk before you continue.

  5. Ensure that the SSH server is installed and configured to start at boot time. This configuration is usually the default.

  6. Install the Azure Linux Agent.

    The Azure Linux Agent is required for provisioning a Linux image on Azure. Many distributions provide the agent as an .rpm or .deb package. The package is typically called WALinuxAgent or walinuxagent. You can also install the agent manually by following the steps in the Azure Linux Agent guide.

    Note

    Make sure the udf and vfat modules are enabled. Removing or disabling them will cause a provisioning or boot failure. Cloud-init version 21.2 or later removes the UDF requirement.

    Install the Azure Linux Agent, cloud-init, and other necessary utilities. Run only the command for the package manager available on your system.

    For dnf, run:

     sudo dnf install -y WALinuxAgent cloud-init cloud-utils-growpart gdisk hyperv-daemons
    

    If your system provides yum instead of dnf, replace dnf with yum in the preceding command.

    For apt, run:

    sudo apt install walinuxagent cloud-init cloud-utils-growpart gdisk hyperv-daemons
    

    For zypper, run:

    sudo zypper install python-azure-agent cloud-init cloud-utils-growpart gdisk hyperv-daemons
    

    Then enable the agent and cloud-init on all distributions:

    sudo systemctl enable waagent.service
    sudo systemctl enable cloud-init.service
    
  7. Don't create swap space on the OS disk.

    You can use the Azure Linux Agent or cloud-init to configure swap space via the local resource disk. Azure attaches this resource disk to the VM after provisioning. The local resource disk is temporary and might be emptied when the VM is deprovisioned.

    Important

    These instructions apply whether the remote disk controller is SCSI or NVMe. They require a VM size that includes a local temporary resource disk. The remote disk controller type doesn't determine whether a resource disk is available.

    1. Choose and configure one method to manage swap space.

      To use the Azure Linux Agent, modify the following parameters in /etc/waagent.conf:

      ResourceDisk.Format=y
      ResourceDisk.Filesystem=ext4
      ResourceDisk.MountPoint=/mnt/resource
      ResourceDisk.EnableSwap=y
      ResourceDisk.SwapSizeMB=2048    ## NOTE: Set this to your desired size.
      

      To use cloud-init instead, configure cloud-init to handle provisioning and prevent the Azure Linux Agent from formatting the resource disk or creating swap space:

      sudo sed -i 's/Provisioning.Agent=auto/Provisioning.Agent=cloud-init/g' /etc/waagent.conf
      sudo sed -i 's/ResourceDisk.Format=y/ResourceDisk.Format=n/g' /etc/waagent.conf
      sudo sed -i 's/ResourceDisk.EnableSwap=y/ResourceDisk.EnableSwap=n/g' /etc/waagent.conf
      
    2. If you chose cloud-init, choose how to provide the swap configuration.

      You can pass a cloud-init configuration through customdata every time you create a VM. Use this method.

      Alternatively, add a cloud-init directive to the image so that swap space is configured every time a VM is created. To use this method, create a .cfg file:

      echo 'DefaultEnvironment="CLOUD_CFG=/etc/cloud/cloud.cfg.d/00-azure-swap.cfg"' | sudo tee -a /etc/systemd/system.conf
      cat << EOF | sudo tee /etc/cloud/cloud.cfg.d/00-azure-swap.cfg
      #cloud-config
      # Generated by Azure cloud image build
      disk_setup:
        ephemeral0:
          table_type: mbr
          layout: [66, [33, 82]]
          overwrite: True
      fs_setup:
        - device: ephemeral0.1
          filesystem: ext4
        - device: ephemeral0.2
          filesystem: swap
      mounts:
        - ["ephemeral0.1", "/mnt/resource"]
        - ["ephemeral0.2", "none", "swap", "sw,nofail,x-systemd.requires=cloud-init.service,x-systemd.device-timeout=2", "0", "0"]
      EOF
      
  8. Configure cloud-init to handle the provisioning:

    1. Configure waagent for cloud-init:

      sudo sed -i 's/Provisioning.Agent=auto/Provisioning.Agent=cloud-init/g' /etc/waagent.conf
      sudo sed -i 's/ResourceDisk.Format=y/ResourceDisk.Format=n/g' /etc/waagent.conf
      sudo sed -i 's/ResourceDisk.EnableSwap=y/ResourceDisk.EnableSwap=n/g' /etc/waagent.conf
      

      If you're migrating a specific virtual machine and don't want to create a generalized image, set Provisioning.Agent=disabled in the /etc/waagent.conf configuration.

    2. Configure mounts:

      echo "Adding mounts and disk_setup to init stage"
      sudo sed -i '/ - mounts/d' /etc/cloud/cloud.cfg
      sudo sed -i '/ - disk_setup/d' /etc/cloud/cloud.cfg
      sudo sed -i '/cloud_init_modules/a\\ - mounts' /etc/cloud/cloud.cfg
      sudo sed -i '/cloud_init_modules/a\\ - disk_setup' /etc/cloud/cloud.cfg
      
      
    3. Configure the Azure data source:

      echo "Allow only Azure datasource, disable fetching network setting via IMDS"
      cat << EOF | sudo tee /etc/cloud/cloud.cfg.d/91-azure_datasource.cfg
      datasource_list: [ Azure ]
      datasource:
         Azure:
           apply_network_config: False
      EOF
      
    4. Remove the existing swap file if you configured one:

      if [[ -f /mnt/resource/swapfile ]]; then
      echo "Removing swapfile" #RHEL uses a swap file by default
      swapoff /mnt/resource/swapfile
      rm /mnt/resource/swapfile -f
      fi
      
    5. Configure cloud-init logging:

      echo "Add console log file"
      cat << EOF | sudo tee -a /etc/cloud/cloud.cfg.d/05_logging.cfg
      
      # This tells cloud-init to redirect its stdout and stderr to
      # 'tee -a /var/log/cloud-init-output.log' so the user can see output
      # there without needing to look on the console.
      output: {all: '| tee -a /var/log/cloud-init-output.log'}
      EOF
      
  9. Validate the disk mount configuration before you deprovision the virtual machine. This validation applies to both SCSI and remote NVMe images.

    1. Review /etc/fstab. Don't use device names such as /dev/sd* or /dev/nvme*, because device names can change across reboots or when the disk controller changes. Use file-system UUIDs or other persistent identifiers instead.

      List the block devices and their persistent identifiers so that you can compare them with the entries in /etc/fstab:

      sudo blkid
      
    2. Check /etc/fstab for syntax errors, invalid mount options, and references that can't be resolved:

      sudo findmnt --verify --verbose
      
    3. Only if you're creating an image for a remote NVMe disk controller, verify the configured I/O timeout. If the source VM currently uses NVMe, run:

      cat /sys/module/nvme_core/parameters/io_timeout
      

      The value must be 240. A SCSI source VM might not expose this runtime parameter; in that case, verify that nvme_core.io_timeout=240 is present in the GRUB configuration and that the rebuilt configuration contains it.

  10. Run the following commands to deprovision the virtual machine.

    Caution

    If you're migrating a specific virtual machine and don't want to create a generalized image, skip the deprovisioning step. Running the command waagent -force -deprovision+user will render the source machine unusable. This step is intended only to create a generalized image.

    sudo rm -f /var/log/waagent.log
    sudo cloud-init clean
    sudo waagent -force -deprovision+user
    sudo rm -f ~/.bash_history
    sudo export HISTSIZE=0
    

    On VirtualBox, you might see an error message after you run waagent -force -deprovision that says [Errno 5] Input/output error. This error message is not critical, and you can ignore it.

  11. Shut down the virtual machine and upload the VHD to Azure.

Next steps

Create a Linux VM from a custom disk by using the Azure CLI

For an existing Azure VM that you need to move from SCSI to NVMe, use Convert Linux and Windows VMs from SCSI to NVMe. For architecture and support information, see NVMe overview.