Shweta Nayak’s Blog

Creating a local Red hat repository for use within the specific environment.

shweta nayak — Thu, 23 Nov 2017 15:08:43 GMT

We do have an update infrastructure in place by default for Red Hat VMs created from the Platform image on Azure : Red Hat Update Infrastructure for on-demand Red Hat Enterprise Linux VMs in Azure https://docs.microsoft.com/en-us/azure/virtual-machines/linux/update-infrastructure-redhat However, we have seen that some customers having large environments choose to manage the updates centrally and in a more granular manner within their organization. For this, they need to configure a local repository on one of their servers, by downloading the packages from the RHUI. Here is how we can create this local repository on Azure. Red Hat shares an article below for creating a local mirror : How to create a local mirror of the latest update for Red Hat Enterprise Linux 5, 6, 7 without using Satellite server? https://access.redhat.com/solutions/23016 Run the command below on the server where the repo needs to be configured : yum install yum-utils createrepo Create the directory to be used for the repo : mkdir /testrepo Run the commands below to check the available repos from RHUI : yum repolist Run the command below to sync them locally in the specified directory : reposync --gpgcheck -l --repoid=rhui-rhel-7-server-rhui-rpms --download_path=/testrepo Note there are multiple repo ids available. We need to pick the ones needed, and run the command multiple times if required. We can download all the packages locally from RHUI using this method. Note : The packages get updated in RHUI every day and every few hours, and these would need to be synced on the local repo too. Hence, we may determine an interval as practical for the environment and sync them locally at a appropriate frequency. Once the local repository is ready, this may need to be accessible over a shared url. We can use FTP or https share for this. Steps for FTP below : Install vsftpd package : yum install -y vsftpd* Edit the configuration file to have the content below : vi /etc/vsftpd/vsftpd.conf #Disable anonymous user Access to secure FTP server anonymous_enable=NO Allow local users to login in vsftpd. local_enable=YES Enable write access to local users. write_enable=YES Uncomment the line chroot_local_user chroot_local_user=YES Enable writable chroot. allow_writeable_chroot=YES systemctl restart vsftpd.service systemctl enable vsftpd.service Apply SELinux context to ftp directory : semanage fcontext -a -t public_content_rw_t "/(/.*)?" restorecon -R / setsebool -P ftpd_anon_write 1 OR on chown ftp / The client would be able to access the share using url like below : ftp://ftp.example.com/path/

'Capture'd the virtual machine. Didn’t intend to generalize it. Now what?

shweta nayak — Sun, 19 Mar 2017 18:49:28 GMT

If you are using Azure Resource Manager virtual machines with Managed Disks, you would see a 'Capture' option when you click on the Virtual Machine overview on the Portal If you click on the button, it clearly mentions its use. You must proceed only if you have run a sysprep or generalized the OS inside the VM. This option helps us capture a VM Image that can be used to create multiple such VMs that use managed disks. It basically has the same outcome as the article below describes Capture a managed image of a generalized VM in Azure <https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-capture-image-resource> If 'Image' creation is not what you intend to do, this is not the right option to use. If you intend to just take a copy or backup of the VM, you must use either Snapshot creation for managed disks Create a copy of a VHD stored as an Azure Managed Disk by using Managed Snapshots <https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-snapshot-copy-managed-disk> Or Recovery Services Back up Azure virtual machines to Recovery Services vaults <https://docs.microsoft.com/en-us/azure/backup/backup-azure-vms-first-look-arm> Having established that, we have had a couple of scenarios where customers have proceeded with the 'Capture' option without generalizing the OS. If you have faced a similar situation, here's how you could recover the machine. Proceeding with the capture button marks the VM with a 'Generalized' flag since it basically assumes the administrator has run the steps at the OS level. The flag is also carried over to the existing disk, hence recreating the VM using the same disk does not fix the issue. Instead, we could capture a snapshot of the disk using the steps below Create a copy of a VHD stored as an Azure Managed Disk by using Managed Snapshots <https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-snapshot-copy-managed-disk> Once you have the snapshot, you may delete the VM from the Portal. Then recreate the machine using the existing NIC resources , but a new managed disk created from the snapshot, using commands like below Login-AzureRmAccount $rgName = ‘ResourceGroupName’ $vmName = ‘VMName’ $location = ‘South India’ $storageType = ‘PremiumLRS’ $snapshotName = 'snapshot1' $snapshotResourceGroupName = 'SnapshotRGName' $vmsize = 'Standard_DS2_v2' $nicname = 'nic' $vm1 = New-AzureRmVMConfig -VMName $vmname -VMSize $vmsize $snapshot = Get-AzureRmSnapshot -SnapshotName $snapshotName -ResourceGroupName $snapshotResourceGroupName $diskConfig = New-AzureRmDiskConfig -AccountType ‘PremiumLRS’ -Location $location -CreateOption copy -SourceResourceId $snapshot.Id $OsDisk = New-AzureRmDisk -DiskName mdnew -Disk $diskConfig -ResourceGroupName $rgName $vm1 = Set-AzureRmVMOSDisk -VM $vm1 -ManagedDiskId $osDisk.Id -CreateOption Attach -Windows $nic = Get-AzureRMNetworkInterface -Name $nicname -ResourceGroupName $rgName $vm1 = Add-AzureRmVMNetworkInterface -VM $vm1 -ID $nic.id New-AzureRMVM -ResourceGroupName $rgName -Location $location -VM $vm1 -Verbose Once the VM comes up, it will be running as good as it was originally.

Case Scenario : "no such partition" error while trying to start the VM after attempting to extend the OS drive

shweta nayak — Sun, 12 Mar 2017 17:30:22 GMT

This particular scenario is from my recent experience working with a customer facing the error "no such partition" when he tried to boot his Red Hat 7.2 machine on Azure. The issue began after customer attempted to extend his OS drive which was running out of space.

The VM was showing up as Running from the Portal, however there was a boot issue with the Operating system.

This error could be seen under Virtual Machine > Boot Diagnostics > Screenshot in the portal

In such cases, we have to look into the OS vhd by attaching it to a working VM.

We exported the VM properties using the command below, since it was Classic VM

Export-AzureVm -Name VMName -ServiceName CloudServiceName -Path "E:\Cs logs\redhat1.xml"

We attached the OS disk to working VM of the same linux flavor and version.

Note :- We strongly recommend taking a backup copy of the OS vhd prior to making any changes to it.

When we ran fdisk on the attached data drive, we found that there was no boot partition present.

Ideally we should see a device /dev/sdc1 with the Boot flag '*' marked in the output below

Now, it is important to understand the steps that the administrator performed on the boot partition, prior to running into this issue.

Since, in this scenario, customer was trying to extend the OS drive using the steps below, I suspected he had accidentally deleted the boot partition and missed recreating it.

https://blogs.msdn.microsoft.com/cloud_solution_architect/2016/05/24/step-by-step-how-to-resize-a-linux-vm-os-disk-in-azure-arm/

We recreated the boot partition by running the command fdisk /dev/sdc

Command 'n' for new partition.

Note that, since sdc2 begins from sector 1026048, sdc1 has to end at 1026047

Used command 'a' to toggle the boot flag.

Used command 'p' to list the partition table and verify that the settings are correct and 'w' to commit the writes.

Note :- In Linux, the partitions can be deleted and recreated, as long as the actual data at the File System level is not disturbed.

However, while doing so, make sure the sector numbers match exactly with the original partitions.

Here, the admin had not made any changes at all to the file system, hence simply the recreation of boot partition was needed.

Since the boot partition had now been recreated, we were able to recreate the VM using the xml file saved earlier and it booted up fine, and we were successfully able to SSH to the machine.

-Shweta Nayak

Learnings : Azure Availability Sets

shweta nayak — Fri, 30 Dec 2016 16:15:56 GMT

Availability Sets work great to ensure high availability of VMs. The article below describes the feature in detail, hope you find this useful. Manage the availability of virtual machines https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-manage-availability Here are some recommendations in order to make the best use of the Availability Sets : Note that these are 'recommendations' and do not affect the supportability.

Place VMs belonging to same Application Tier in a single Availability set.
Maintain the same VM size or family for machines in the same Availability set.

Here are the reasons behind these recommendations Load Balancing and VM failures The concept of Availability set focuses on the idea of having ‘replaceable’ VMs in each Application Tier. “If your virtual machines are all nearly identical and serve the same purpose for your application, we recommend that you configure an availability set for each tier of your application. If you place two different tiers in the same availability set, all virtual machines in the same application tier can be rebooted at once. By configuring at least two virtual machines in an availability set for each tier, you guarantee that at least one virtual machine in each tier will be available.” As described above, the idea is to have at least one VM running in case of multiple points of failure. This could be any one of the VMs in the Availability set. During normal operations, the intent is to load balance the requests among the VMs in the Availability set. In both the scenarios, it is practical and most beneficial to maintain the VMs of the same size ( or family ) to ensure equal distribution of load and least impact, and hence the recommendation. Supportability of the VM size on the Physical Hardware cluster All running VMs in the availability set must be using the same physical hardware cluster. This is by design. “ If all VMs are in the same VM Size family, then you can be assured they will all be supported on the physical hardware cluster which is used to host the VMs of the availability set. “ Resize virtual machines https://azure.microsoft.com/en-in/blog/resize-virtual-machines/ Due to this reason, we may run into Provisioning errors while Starting, Resizing or Provisioning one or more of the VMs when we try to use different size families in an Availability Set. VM size family chart : Refer to the table under section “Performance considerations” in the article below to get an idea of the VM size families. https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-sizes Managing an Availability Set with multiple VM sizes Since the above are ideal recommendations, some administrators may still want to use multiple VM sizes in the same Availability Set. There are chances we may run into the error “Provisioning failed. Allocation failed. Please try reducing the VM size or number of VMs, retry later, or try deploying to a different Availability Set or different Azure location" while managing such setups. Here is how we would need to perform some of the operations in such scenarios Start Operations - Stop all the VMs in the Availability Set. Start the one with the largest size first. This would ensure the hardware cluster allocated is compatible with the largest VM size and the ones started later ( smaller or same size ) can be hosted on the same hardware cluster. Resize Operations - Stop all the VMs. Resize the VMs one by one - Once again first resize the ones with the largest target size. Refer : Resize virtual machines https://azure.microsoft.com/en-in/blog/resize-virtual-machines/ Adding a new VM of a different size family to the Availability Set - Stop all the VMs in the Availability set. Introduce the new VM belong to the different size family ( assuming larger one ). Then start the other VMs. We would not need these steps if the new VMs being added belongs to a smaller VM size family. Of course, the above guidance assumes we are trying to use sizes that are supported in those regions and supported for resizing.

Exchange Service Pack / Cumulative Update Installation Checklist

shweta nayak — Wed, 02 Mar 2016 19:39:00 GMT

Here are the set of steps to go through, to ensure the Exchange 2010 Service Pack upgrades or the Exchange 2013 Cumulative Update installations happen smoothly

Make sure the Execution policies are set to unrestricted by running the command :

Get-ExecutionPolicy -list

You can run the command Set-ExecutionPolicy Undefined -Scope PolicyName

Refer

You receive error 1603 when you try to install the Exchange Server 2010 RU1

<https://support.microsoft.com/en-us/kb/981474>

This is also applicable to Exchange 2013

Since the server is already in production, the Windows OS updates and system requirements will already be met.

However, I would still suggest to go through the hotfixes section.

https://technet.microsoft.com/en-us/library/bb691354(v=exchg.141).aspx

https://technet.microsoft.com/en-us/library/aa996719(v=exchg.141).aspx

Make sure we do not have anything additional installed that is unsupported.

Exchange Server Supportability Matrix

<https://technet.microsoft.com/en-us/library/ff728623(v=exchg.150).aspx>

For example, the .Net framework and Windows Management Framework installed should fall into the supportability matrix. There are instances where admnistrators inevitably or unknowingly install one of the unsupported versions of .Net or WMF that prevents future upgrades.

Check the Powershell version by running the command $PSVersionTable

.Net framework version could be checked under the registry location :

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full

Interim Updates if any, need to uninstalled before Exchange 2010 Service Pack installation. For Exchange 2013, they need not be uninstalled, we can go ahead with the Cumulative Update installation.

If at all there are UM-LanguagePacks installed, they need to uninstalled before upgrading, for Exchange 2010 and Exchange 2013 both.
Make sure we have the latest back-up of the Exchange databases and also system state backup of the AD environment

Note down any customizations such as OWA, config files on servers, registry changes, Lync integration, or third party add-ons, prior to the installation.

We could perform the PrepareAD in advance separately. This only needs to be done once for the whole environment.

Exchange 2010

setup.com /preparead

Exchange 2013

setup.exe /preparead /IAcceptExchangeServerLicenseTerms

If this is not done, setup will automatically perform it at the time of the upgrade of the first server.

Set the server to be upgraded to maintenance mode if it is mailbox role in Exchange 2013 ( with or without DAG) and if is a DAG member in Exchange 2010.

For Exchange 2010 :

.\StartDagServerMaintenance.ps1

Remove it from maintenance mode using the command below

.\StopDagServerMaintenance.ps1

<https://technet.microsoft.com/en-IN/library/ee861125(v=exchg.141).aspx>

For Exchange 2013 :

Use the steps as suggested in the article below,

Exchange 2013 Maintenance mode

<http://blogs.technet.com/b/nawar/archive/2014/03/30/exchange-2013-maintenance-mode.aspx>

Or the script ,

Exchange Server 2013 Maintenance Mode Script (Start)

<https://gallery.technet.microsoft.com/office/Exchange-Server-2013-ff6c942f>

For CAS, take the node to be upgraded out of the Load Balancer.

Then proceed with the installation from GUI or from Administrator command prompt by pointing to the location where the files have been downloaded. Files should be downloaded from Microsoft Download Center

Exchange 2010

Setup.com /m:upgrade

Exchange 2013

Setup.exe /m:upgrade /IAcceptExchangeServerLicenseTerms

Make sure to use an elevated command prompt and to close all the other Exchange windows at the time of setup.

Also make sure there were no pending reboots prior to beginning the installation.

The order of installation for Exchange 2010 :

Client Access Server (if you have multiple sites, internet facing sites first)

Hub Transport

Mailbox

The order of installation for Exchange 2013 :

Mailbox (Internet-facing site first)

CAS

Edge

Post the installation we must reboot the Exchange server.

I would also recommend to test the setup in a lab before installing it in production.

- Shweta

Usage of Get-StoreQuery to query against the managed store

shweta nayak — Fri, 30 May 2014 08:08:00 GMT

Exchange15 focusses on providing more Diagnostic information to customers. The Introduction of Managed Store feature in E15 provides a read-only access to the Store table.

Store Query is a function that basically uses Get-ExchangeDiagnosticInfo command in the background and provides output that would make sense to humans.

Through this Blog I am sharing a few ways I found the command could be used for some of the Basic troubleshooting,

However first, let us see how to load this command.

We first have to run the ManageStoreDiagnosticFunctions.ps1 script in the Shell.

[PS] C:\Windows\system32>cd "C:\Program Files\Microsoft\Exchange Server\V15\Scripts"

[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>. .\ManagedStoreDiagnosticFunctions.ps1

The method above is called Dot sourcing as script. It loads the script into Powershell session instead of just running it.

Display all the Tables in the Catalog

Get-StoreQuery -Database dbname -query "select * from Catalog" |ft

This command helps you get the list of all the Tables and TableFunctions in the catalog. Tables are basically present on the Disk, whereas the TableFunctions are present in the Memory.

How to find out the mailbox Number

Since you may need to use the MailboxNumber instead of displayname in most of the queries, and it is also easier to use the mailboxNumber, here is how we could get the list of Mailboxes on a specific database.

Get-StoreQuery -Database dbname -query "select * from Mailbox" |ft MailboxNumber,DisplayName

Alternately, if we know the displayname, we can find out the MailboxNumber and other details as below :

Get-StoreQuery -Database dbname -query "select * from Mailbox where DisplayName = 'tester1'"

Display information about a specific message in the Message table from the MID and MailboxNumber

Get-StoreQuery -Database dbname -query "select * from Message where MessageID = '0x9EAD721616F9D74B9CBF644606B6F49B000004A0D79D00000100' and MailboxPartitionNumber = '106'"

We can obtain the properties of a message as below :

Find out the mailbox details based upon a UserSID

Scenario : We find out a specific Mailbox SID as the culprit, however we do not know which mailbox does the SID belong to.

Get-StoreQuery -Database dbname -query "select * from MailboxInfo where UserSID = 'S-1-5-21-3088144668-2606371049-731046914-2053'" |fl

Here, we have found out that the SID "S-1-5-21-3088144668-2606371049-731046914-2053" Belongs to the Health Mailbox, which wouldn’t have been listed from the Shell Window.

Find out the transactions that took place on a specific MessageID/FID or a Mailbox

Get-StoreQuery -Database dbname -query "select * from Events where MID = '0x812782A445F65B40A0A6608432BF2EBD0000E550867E0000'"

We may use this to track pervious transactions at database and mailbox level.

The results give out transactions which occurred in that order. It also mentions the Object Class, which may be the Message Class or the Folder Type on which the transaction took place.

Lastly, in order to make sure the Store Query function is working fine, we could run the command

Test-StoreQuery

We could use store querying in lot many other ways, do share if you come across other helpful usages.

Hope this was a helpful start to using Managed Store querying!

- Shweta