HD insight Cluster, Worker node, E32_V3 (256 GB), memory issue

Question

HD insight Cluster - Worker node - E32_V3 (256 GB). It is showing 911 GB memory on ambari portal. MS document for E32_v3 of HD insight worker node showing 1600 GB space. Why there is a discrepancy.

Answer 1

Hello @Thakur, Prabhat ,

Thanks for the question and using MS Q&A platform.

The one which you are viewing the Azure HDInsight Ambari portal is the OS disk size and not the Temporary storage.

Note: To find the disk usage details. SSH to the node first, then run df to list disk usage for all mounts.

In Azure HDInsight - Each virtual machine runs linux with the applicable components from the selected cluster type installed. The amount of RAM and quantity of CPU's are determined by the selected virtual machine size.

A Linux Virtual Machine in Azure, including those used in an HDInsight cluster, has two disks associated with it. The disk at /dev/sda is your OS disk, /dev/sdb is your temporary disk. The OS disk (/dev/sda) contains the operating system and is optimized for fast VM boot time and does not provide good performance for your workloads. While the OS disk is backed by durable storage, as node VM's are managed by HDInsight, the VHD file backing the OS disk is not accessible to you (nor is the Storage Account which contains it).

The temporary disk (/dev/sdb) is not durable, and can be lost if specific events like VM resizing, redeployment, or maintenance forces a restart of your VM. The temporary drive itself is backed by local HDD disk-based storage or SSD based storage depending on the VM type selected. However, as this temporary storage drive is present on the physical machine which is hosting your VM, it can have higher IOPs and lower latency when compared to the persistent storage like a standard data disk. The size of your temporary disk is related to the VM size for the node type you chose at cluster deployment time.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators