1. Install Node and npm on on your Mac, as that is one of the easier ways to get the Azure CLI (Command Line Interface) installed. Follow the steps here to do so: http://blog.teamtreehouse.com/install-node-js-npm-mac One thing that the above blog does not go into the details of is how to install the correct Xcode on your Mac. If you go to the Apple Store, you will only get the latest version of Xcode. But your Mac may be running an older version of OSX, which cannot install the latest version of Xcode. If that happens, you will have to find which Xcode version is good for your OSX (usually Xcode 6.2 works well on OSX 10.X, please verify the support matrix by running some searches). Once you know which Xcode version to install, download it from Apple developer download center and install it.
2. Install the Azure CLI npm install -g azure-cli
3. Check if the CLI is working by running some basic commands azure help azure --version
4. Login to your Azure account (the following command is for interactive login using a browser window, there are other ways too) azure login
5. If you have multiple subscriptions, run this command to see all of them so that you can verify whether the CLI is using the correct one. The "Current" column of the output will be "true" for the subscription being currently used. azure account list
6. If you have multiple subscriptions, and if the above command shows that the CLI session is using the incorrect one, use the following command to set the correct one as default: azure account set <<name of the subscription you want to set as default from the "Name" column displayed in the output of the "azure account list" command>>
7. Go to Azure ARM mode (if you do not know the differences between the two modes or deployment models supported by Azure - the older "classic" model and the newer "ARM" model, where ARM stands for "Azure Resource Manager", see this): azure config mode arm
8. An easy way to create a Docker host on Azure is to just execute a pre-existing JSON template that Microsoft template developers have contributed to github. You can find a lot of templates in this quickstart template repository. I picked one that says docker-simple-on-ubuntu. The beauty of these templates is that you just run a CLI command with the template file's location on github and your infrastructure, whatever the template is written to deploy, will get created and deployed within a few minutes! azure group create --name file-upload-container-demo --location "West US" --template-uri https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/docker-simple-on-ubuntu/azuredeploy.json In the above command, I passed, as value for the --name parameter, "file-upload-container-demo". That creates an Azure Resource Group by that name. The benefit of creating your entire infrastructure within a new Resource Group is that you can, later, simply delete the Resource Group, and all the VM-s, VNET-s, Load Balancers, IP Addresses, Network Security Groups, NIC-s, etc. you had deployed withing that Resource Group will all be removed in one go - so cleaning up is easy. You may be thinking... why am I going to deploy all these things to run a simple Docker host? The answer to that is, you need some of these basic infrastructure components to have a usable Virtual Machine. For example, you can spin up a VM, but unless you have a public IP address, you cannot access it easily. Unless you have a NSG (Network Security Group), you cannot selectively open ports (like 22 for SSH or 80 for web). When you create a VM from the Azure portal, these peripheral infrastructure components are automatically created for you. But when you are creating something from a command line or code, you will need to create all of these. Luckily, that is where the template helps. The template here has instruction to create all of these, and just one CLI command to execute the ARM Template is enough to stand up the whole gamut of components needed to start using your Docker host quickly! One possible down-side of using this pre-written template is that you may not want your Docker Host to be of size standard F1. But that is what you will get, as the template specifies that. You may want your storage disks to be either HDD or SSD, but you will get whatever the template specifies. In short, if you need to customize your deployment like what you want, you will have to write your own template. Some template authors go the extra mile to make these choices configurable. In that case, you will have to pass these as parameters for the template.
9. The above command is interactive, and it will ask you a few things. Once it succeeds, your Docker host is ready. It will interactively ask you some questions like the DNS name for your VM, use a unique name for the region you are using in the command. In my case, I specified the DNS Name "uploadnshare0123" and user name "user_name", so trying to access the VM through the DNS name associated with it via SSH from my Mac is done by a simple ssh command ssh user_name@uploadnshare0123.westus.cloudapp.azure.com
10. Once you are in, start running docker commands sudo docker ps
11. If you are wondering how exactly Docker is running on the machine without you installing it, it is all magic done inside the JSON ARM template. Read it here to understand it in greater details. Good luck!

After the Microsoft - Red Hat alliance, engineering teams from both sides are working on the integration aspects. There is a lot to integrate - OpenShift, CloudForms, Ansible, etc. Speaking of Ansible, I recently had to set it up on an Azure Red Hat VM for a demo. Like any developer-at-heart would, I was determined to make Ansible work from source (as opposed to a downloaded binary package). To my surprise, the packages needed for the Ansible master to run on an Azure Red Hat VM were mostly missing! I went through the usual package-hunting process and finally got it going. Thought I would put a blog out to help others in my shoes.

My take on Ansible

Get set, ready, go

Step 1. Spin your Linux Infrastructure up

Step 2. Install and run Ansible on the controller

1. sudo yum install git

2. git clone git://github.com/ansible/ansible.git --recursive

3. cd ./ansible

4. source ./hacking/env-setup

5. sudo easy_install pip

6. sudo yum install python-devel

7. sudo yum groupinstall "Development Tools"

8. sudo yum install libffi-devel

9. sudo yum install openssl-devel

10. .sudo pip install paramiko PyYAML Jinja2 httplib2 six

11. sudo pip install pycrypto

 pushd ansible
 source ./hacking/env-setup
 popd

Step 3. Create your own custom inventory file and see it work

[webservers]
 apacheweb ansible_host=10.6.0.5

[dbservers]
 mysqldb ansible_host=10.6.0.6

Step 4. Deploy Web Server and Database Server and see it all work

1. First of all, the playbook "lamp_simple_rhel7" came with its own hosts file in the lamp_simple_rhel7 directory. I renamed it to something else, as I was trying to use /etc/ansible/hosts
2. I commented out the 3 lines, each saying "remote_user=root" from site.yml, the playbook that I was trying to run. I used a leading '#' to comment them out
3. For each *.yml file in lamp_simple_rhel7 directory and all subdirectories (except site.yml), I added "sudo: yes" to every task and include. To do this, first cd into lamp_simple_rhel7 directory and run a find command: find . -name "*.yml" I found 7 files except site.yml. Ignore site.yml. In each of these 7 files, I added a line "sudo: yes" after proper indentation to each task/ include. As an example, I will show the final form of /roles/web/tasks/copy_code.yml file:

- name: Copy the code from repository
  git: repo={{ repository }} dest=/var/www/html/
  sudo: yes

- name: Creates the index.php file
  template: src=index.php.j2 dest=/var/www/html/index.php
  sudo: yes

Failure talking to yum: Could not contact any CDS load balancers

Step 5. Verify installation

Conclusion

1. Web Apps
2. MySQL and SQL resource providers to act as back-ends for the Web Apps
3. Updated X-Plat CLI and Powershell support for Native SDK
4. Productivity Boosters like Native Visual Studio support (enabling things like CI/CD right from inside Visual Studio)

What About "Multi-Cloud" Strategy?

Partnership with Red Hat

We are trying to work hard to make the RHEL Pay-As-You-Go images available on Azure as standard supported Gallery images, as of the writing of this blog post. The developer and partner community should not need to wait much longer for this to happen! However, if someone wants to bring their own custom RHEL image, upload it as a VHD to Azure Blob Storage and provision Virtual Machines from the same, they can do so even today. That's called the "BYOL" model - Bring your own license.

The terms of the much-anticipated partnership included, other than the RHEL OS, the following Red Hat solutions to be made available on the Azure Platform: JBoss Middleware Suite, GlusterFS, Openshift platform as well as CloudForms. It included support for Hybrid environments - partially on the cloud and partially on-premises or 3rd party hosted sites. It also assured us that RHEL will be the primary OS for .NET Core on Linux.

Containerization and Docker - Windows integration and Azure Container Service

Microsoft's efforts to integrate Docker Engine and Windows OS has made great progress! Though it is Work in Progress, you can now, as of the writing of this blog post, go to the Azure Portal (and remember that the "new" portal is now the "current" portal), spin up a VM from the image "Windows Server 2016 Core with Containers Tech Preview 4", and start creating Windows Containers on it! The step by step preview guide for creating Windows Containers and playing around with them can be found here: https://msdn.microsoft.com/en-us/virtualization/windowscontainers/containers_welcome.

This incredible speed is also evident in the work done on the Azure Container Service front. After we announced it at AzureCon in September 2015, we opened Preview by self-nomination in December. The Azure Container Service is a Compute Resource Provider - it creates a cluster of VM-s acting as container hosts together with master VM-s that orchestrate deployment and execution of containers, providing HA, Fault Tolerance and Utilization Density.

Availability of Debian as an Azure-endorsed OS

Debian (7 and 8) are now fully endorsed on Azure, thanks to our partnership with Credactiv! This piece of news was welcomed by the cloud community: http://www.zdnet.com/article/debian-gnulinux-now-supported-on-microsofts-azure/.

Though Debian is widely considered a 'developer-friendly' Operating System (as opposed to Ubuntu's perceived ease of use), recent changes in Debian are clearly attempting to erase that gap. In my opinion, the structure that Canonical has provided to Ubuntu's feature releases, software and hardware compatibility, usability etc. have resulted in a more streamlined user experience than the completely open source community developing Debian. However, Debian remains the Rock of Gibraltar when it comes to stability, and is also known to have outperformed Ubuntu, though not by much, in performance tests. They cater to slightly different audiences. Availability of both on the Azure Platform, therefore, is critical to opening up Azure to the widest possible spectrum of Linux users!

General Availability of HDInsight: Run Hadoop clusters on Linux on Azure

The ability to create Hadoop, Hbase, STORM and SPARK clusters on a LINUX based collection of VM-s (in addition to the Windows option) – which is fully managed, monitored and supported by us with 99.9% SLA – was greeted with enthusiasm by entire OSS and Big Data Community in September of last year! It is amazing that Microsoft runs a major PAAS service entirely on Linux!

Node.js support for Azure Mobile Apps Backend

We released Node SDK for Mobile Apps under Azure App Service early September 2015. Complete with push notifications, mobile auth, offline sync and other rich features, this addition to Azure App Service was an instant hit with OSS Mobile Developers, which is a significant chunk of the IT community. We followed up with a beta update recently a few days back, adding even more features!

Not directly related to Azure, but our recent open-sourcing of CNTK definitely deserves mention here

Microsoft’s brain is now available for anyone to use in their apps! We have open-sourced the artificial intelligence framework we use to power speech recognition in our Cortana Digital Assistant and Skype Translate applications. The framework, called, CNTK (Computational Network Toolkit), is based on a branch of Artificial intelligence called Deep Learning, which seeks to help machines do things like recognize photos and videos or understanding human speech by mimicking the structure and functions of the human brain.

CNTK's advantage over Google TensorFlow (quoted from http://www.wired.com/2016/01/microsoft-tries-to-one-up-google-in-the-open-source-ai-race/): CNTK has a big advantage over TensorFlow for people outside of academia: it can take advantage of the power of many servers at the same time. That’s important because it’s rare that a single computer is powerful enough to handle a real-world artificial intelligence application, such as speech recognition on an app used by millions of people. Internally, Google likely uses TensorFlow on thousands of servers at a time. But the version Google released to the public, Huang says, can’t be used in this way. In fact, few deep learning frameworks other than CNTK support running across multiple servers right out of the box, though it’s possible to do with other open source software such as Torch, which is used by Facebook and Google.

The code can be found on github: https://github.com/Microsoft/CNTK

As we continue to drive the momentum of innovation, openness and empowerment even harder, you can expect bigger and better things from Microsoft in the future! I would like to thank you for the support!

There are several ways to set up a SWARM cluster of Docker hosts on Azure. The documentation out there is out-of-date or incomplete or simply does not work. I wanted to share one way that seemed easy to me.

Here is a diagrammatic representation of the SWARM cluster we will create on Azure [ 1 Swarm Manager (Ubuntu), 2 Swarm Nodes (Ubuntu) in a VNET, being controlled from a Docker Client of any sort (I use my Windows laptop) ]:

(In addition, I used another Linux VM to issue Azure CLI commands to actually create the set-up. Instead, you could use the same machine for the Docker client and CLI client)

Steps involved:

1. Create the SWARM Master using Azure CLI, then configure it:

a. Pre-requisite: Install Azure CLI. Steps for an Azure Ubuntu VM are given below:

i. sudo apt-get update

ii. sudo apt-get install nodejs-legacy

iii. sudo apt-get install npm

iv. sudo npm install -g azure-cli

v. (Log yourself in) azure login

b. Change mode to ARM:

i. azure config mode arm

c. Create a separate resource group which will contain all the resources for this set-up

i. azure group create -n "cliSouthCentralSwarmRG" -l "South Central US"

d. Create the SWARM master VM with its own NIC, Public IP Address and a new storage account, also creating the VNET and subnet in the process

i. azure vm docker create --resource-group cliSouthCentralSwarmRG --name cli-swarm-master --location southcentralus --os-type linux --nic-name clitestnic --public-ip-name clitestpip --vnet-name cliswarmvnet --vnet-subnet-name cliswarmsubnet --storage-account-name clidockerswarmstorage --image-urn canonical: UbuntuServer :14.04.3- LTS:latest --admin-username Azure123 --admin-password 'Azure!23'

ii. (When prompted for VNET address prefix, use something that works for you like 10.10.0/16, for subnet address prefix, you can use 10.10.1.0/24 in that case. When prompted for Public IP domain name, use something unique, e.g., “clidockerswarm” – I have already taken that one for South Central US though)

iii. Check the portal for the FQDN of the Public IP you just created, use that and port 22 to SSH into the box (you can use Putty)

e. Disable TLS on the Swarm Master – for demo purposes, we will disable TLS, as using TLS for Swarm cluster is slightly more work than enabling it for just one server, and we can dedicate some time later on if we are working on a real production or dev/QA workload, adding TLS back

i. Once logged into the Swarm Master host on SSH, enter a sudo shell (“sudo su”) as the following operations will need root access

ii. Edit /etc/default/docker (use vi or some other editor) and remove the parameters tlsverify, tlscacert, tlscert and tlskey. The shortened line should look like this: DOCKER_OPTS="-H=unix:// -H=0.0.0.0:2376"

iii. service docker restart

iv. (To verify that your changes have taken effect) ps aux | grep docker

f. Connect to the SWARM master host from your docker client. If you are not sure how to set up a docker client on your windows laptop, follow the first 2 sub-steps in “Step 2: Set up your Docker client on your Windows desktop/ laptop and connect to the server” of my blog here . There is no need to follow the sub-steps beyond the first two as we will not be using TLS. On your WINDOWS laptop command prompt/ Cygwin terminal running on WINDOWS laptop:

i. Make sure that you have no user or user/environment variables DOCKER_HOST or DOCKER_TLS_VERIFY or DOCKER_CERT_PATH. If you have set them previously to connect to another docker server, you can do the following: (a) if you are running Cygwin, temporarily disable these for the current session by running: unset DOCKER_HOST; unset DOCKER_TLS_VERIFY; unset DOCKER_CERT_PATH (b) If you are running on command prompt, these will turn the variables off permanently: setx DOCKER_HOST “”; setx DOCKER_TLS_VERIFY “”; setx DOCKER_CERT_PATH “”

ii. docker -H tcp://{domain name of your docker VM swarm master's Public IP Address}:2376 {some docker command}
Example: docker -H tcp://clidockerswarm.southcentralus.cloudapp.azure.com:2376 ps -a

2. Create the two SWARM nodes using Azure CLI and configure them:

a. Azure CLI command to create the 2 VM-s – basically the same command as for creating the swarm master, but with new values for (a) --name (b) --nic-name (c) --public-ip-name, but with existing values of vnet-name, vnet-subnet-name and storage-account-name. Which means that we are, with each command, creating a new VM with a new NIC and a new Public IP Address but we are adding it to the same vnet/subnet and using the same storage account. When prompted, use some unique name for Public IP domain name for each node (like “clidockerswarmnodeone”, “clidockerswarmnodetwo”):

i. azure vm docker create --resource-group cliSouthCentralSwarmRG --name cli-node-1 --location southcentralus --os-type linux --nic-name clitestnicnodeone --public-ip-name clitestpipnodeone --vnet-name cliswarmvnet --vnet-subnet-name cliswarmsubnet --storage-account-name clidockerswarmstorage --image-urn canonical:UbuntuServer:14.04.3-LTS:latest --admin-username Azure123 --admin-password 'Azure!23'

ii. azure vm docker create --resource-group cliSouthCentralSwarmRG --name cli-node-2 --location southcentralus --os-type linux --nic-name clitestnicnodetwo --public-ip-name clitestpipnodetwo --vnet-name cliswarmvnet --vnet-subnet-name cliswarmsubnet --storage-account-name clidockerswarmstorage --image-urn canonical:UbuntuServer:14.04.3-LTS:latest --admin-username Azure123 --admin-password 'Azure!23'

b. Like you did with the SWARM Master host, disable TLS on each and connect to each node from your docker client repeating the steps 1(e) and 1(f) for each of the nodes, with ONE IMPORTANT DIFFERENCE. When you edit /etc/default/docker on the nodes, ALSO CHANGE THE PORT NUMBER TO 2375 (instead of 2376). The DOCKER_OPTS line should look like this for the nodes: DOCKER_OPTS="-H=unix:// -H=0.0.0.0:2375".

3. Create the swarm cluster and join the nodes to the cluster (all commands from now on should be executed on the docker client, using the -H parameter to point it to the correct docker server, be it the swarm master or one of the swarm nodes). NOTE: We are installing swarm using docker, which means swarm will run as a container:

a. Pointing client towards swarm master (commands show the FQDN as created in the above examples, replace with your own FQDN):

i. docker -H tcp://clidockerswarm.southcentralus.cloudapp.azure.com:2376 pull swarm

ii. docker -H tcp://clidockerswarm.southcentralus.cloudapp.azure.com:2376 run --rm swarm create

iii. Save the cluster id reported back by the last command, we will need it later on

b. Pointing client towards swarm node one (commands show the FQDN as created in the above examples, replace with your own FQDN):

i. docker -H tcp://clidockerswarmnodeone.southcentralus.cloudapp.azure.com:2375 pull swarm

ii. In the following command, use the public ip address of node one for the value of addr parameter (get it from the portal) and use the cluster id saved in 3(a)(iii) above after token://

iii. docker -H tcp://clidockerswarmnodeone.southcentralus.cloudapp.azure.com:2375 run -d swarm join --addr=40.84.184.174:2375 token://cb12d74b9b5c45deb054f92cf6578efe

iv. (to verify) docker -H tcp://clidockerswarmnodeone.southcentralus.cloudapp.azure.com:2375 ps -a

c. Pointing client towards swarm node two (commands show the FQDN as created in the above examples, replace with your own FQDN):

i. docker -H tcp://clidockerswarmnodetwo.southcentralus.cloudapp.azure.com:2375 pull swarm

ii. In the following command, use the public ip address of node two for the value of addr parameter (get it from the portal) and use the cluster id saved in 3(a)(iii) above after token://

iii. docker -H tcp://clidockerswarmnodetwo`.southcentralus.cloudapp.azure.com:2375 run -d swarm join --addr=40.84.190.103:2375 token://cb12d74b9b5c45deb054f92cf6578efe

iv. (to verify) docker -H tcp://clidockerswarmnodetwo.southcentralus.cloudapp.azure.com:2375 ps -a

d. Pointing client towards swarm master (commands show the FQDN as created in the above examples, replace with your own FQDN):

i. In the following command, use the cluster id saved in 3(a)(iii) above after token:// and be careful with the port numbers, note that we are connecting the docker client to port 2376 where the docker daemon is running, but we are asking to start the swarm manager daemon to start up and listen on port 2375, and pass docker commands to port 2375 on the nodes:

ii. docker -H tcp://clidockerswarm.southcentralus.cloudapp.azure.com:2376 run -d -p 2375:2375 swarm manage token://cb12d74b9b5c45deb054f92cf6578efe

iii. Once the manager is up and running, our cluster is ready, and we should be able to list the nodes. There are 2 ways to list the nodes

iv. (First way to list the nodes): docker -H tcp://clidockerswarm.southcentralus.cloudapp.azure.com:2375 info

v. (Second way to list the nodes): docker -H tcp://clidockerswarm.southcentralus.cloudapp.azure.com:2376 run --rm swarm list token://cb12d74b9b5c45deb054f92cf6578efe

e. It may happen that when you try to list the nodes, you do not see two of them, you only see one. That would be a bug whereby you joined one of the nodes too soon. Just join it back again (run 3b or 3c again depending on which node needs re-joining) and verify using the listing methods

4. Put SWARM to work – well what use of setting a SWARM cluster up unless you also use it for something. I will cover the basics of how you could use the cluster:

a. From this point on, issue ALL docker commands (from your docker client) against the SWARM master port 2375 (where the SWARM manager daemon is running):

i. First, verify that no containers are running on any node: docker -H tcp://clidockerswarm.southcentralus.cloudapp.azure.com:2375 ps

ii. Then, run mysql on your cluster and see SWARM master pick one of the nodes automatically: docker -H tcp://clidockerswarm.southcentralus.cloudapp.azure.com:2375 run -d -P --name db mysql

iii. Then, check which node it chose with the ps command again, it should show which node it chose in the ‘names’ column. Also, note the container id

iv. Then, stop the container: docker -H tcp://clidockerswarm.southcentralus.cloudapp.azure.com:2375 stop <container id of mysql container>

b. That was an example of SWARM selecting the node for you. However, SWARM provides several ways by which you can specify your preference of nodes. See the filters topic here

c. Once you combine the filters with strategies, you can start putting together the rules of how to design an end-to-end solution/application (e.g., an n-tier backend) with containers. I will write back, in another session, on the application design and architecture aspects of using docker, swarm, etc.

Thanks

Typically, the engineering group runs the initial tests on performance. They spin up an Azure Linux VM, install the database software on it, point a load testing client at it and run tests. As this is meant to be a quick go/no-go assessment, they do not bother to add additional data disks and move the data location from the OS disk to the data disk.

The results, therefore, are misleading. When you install any of these databases using the standard package managers (apt-get or rpm), the data location is still on the OS disk (/usr/local or /etc or something like that).

On an Azure VM, the OS Disk is not optimized for IO performance. Instead, it is optimized for faster boot times. That is why before running any initial performance tests, we should add at least one data disk and move the data location to the new additional disk. After all, that is the way production is going to look like - so the initial tests will mimic production in that sense.

That alone will improve the IO performance by several times. If we want to get even more performance gain, we should add multiple data disks and stripe them in a RAID 0 array. See this blog post.

Build better things on Azure!

MySQL Cluster, if used correctly, has proven to be one of the most fault tolerant, scalable and real-time distributed Active/Active-capable enterprise data storage solutions. Web providers with tons of daily visits like Paypal and Zillow, as well as leading telecommunications and network equipment providers, including Alcatel-Lucent, AT&T, BT, and Ericsson have successfully deployed MySQL Clusters. You can listen to Paypal Chief Architect talk about using MySQL cluster for Paypal fraud detection in this video starting at the 24 minutes 20 seconds. Check out the Zillow case study here. They use the MySQL Cluster Carrier Grade Edition ("CGE") which includes support and useful enterprise tools - but the MySQL Cluster Community edition is free to try out and play around with.

Why another best practice guide?

So why another best practices guideline for MySQL Cluster? Because when it comes to deploying and getting the best out of MySQL on Azure as an IaaS solution, we have to keep certain things in mind that is not necessarily covered very well out there today. Oracle has a whitepaper that is intended to be on the topic of MySQL for cloud - it is called "A Guide to MySQL for SaaS". However, I found that it is more of a marketing material than being a technical guide. Also, it covers all MySQL offerings broadly, and none very deeply, including MySQL Cluster.

Distinguishing MySQL Cluster from MySQL

Which brings me to the distinction - an important one for this blog post - MySQL Cluster is not the same as MySQL Database (which, in short, is just "MySQL"). They do not follow the same release cycle and version number patterns. As of this blog post, the GA version of MySQL Cluster is 7.4, but the GA version of (simply) MySQL is 5.7. MySQL Cluster does not use the same engine as MySQL - it uses a different engine called "NDB Cluster". MySQL Cluster does not store the data co-located with the mysqld daemon, it uses separate data nodes. It is a very different beast in that sense. Of course, it is still the same in one sense, but the difference is important to note for this blog post, because we are going to explore MySQL Cluster on Azure. On the contrary, this article and this blog post of mine deals with certain best practices of using MySQL on Azure.

PaaS versus IaaS MySQL Cluster for Azure

ClearDB now provides PAAS MySQL Clusters on Azure as well. In fact, their offering seems very robust and technically a lot more attractive than managing your own instances of MySQL Clusters because of the recently announced option of Premium Dedicated Clusters. I say that because personally, PaaS is always more attractive than IaaS, because you do not have to worry about maintaining/ monitoring/ baby-sitting/ upgrading your infrastructure, you do not have to worry about the up-time SLA or handle scaling your VM-s yourself. However, I am not sure of how ClearDB stacks up against Azure IaaS cost-wise. Though I have not done an accurate cost comparison analysis between these choices, it is clear to me that MySQL Cluster demands Azure VM-s of bigger sizes because of its memory needs. See Point 3 in the next section "What about Performance?". Therefore, none of the options promise to be cheap, but I am sure both of them are cheaper than having on-premises servers which tend to be severely under-utilized and almost always end up having a higher TCO because of man-hours spent in maintaining these servers, money spent behind refreshing hardware, etc.

This blog post is about IaaS - where you set up the MySQL Cluster yourself on Azure. I wrote another blog post about the actual process of creating a MySQL Cluster on Azure. I will not cover the same here again. This one will be about architectural best practices regarding high availability, fault tolerance and geo-replication.

What about Performance? 

Today, we will cover High Availability, Fault Tolerance, Scalability and some thoughts on Geo Replication related to MySQL Cluster on Azure. We will leave the performance for another day. Why? A few thoughts:

1. There are a lot of articles on the internet that focuses on performance optimization, fine tuning and cluster configuration for optimum MySQL cluster performance. Most of these techniques are specific to MySQL Cluster and will work regardless of where it is running – on-premises or cloud, AWS or Azure. However, in my opinion, HA and Fault Tolerance aspects are different on the cloud, and need special considerations.
    
2. You will hear a lot of great things about MySQL Cluster performance. Sometimes it gets difficult to sieve through results published by people trying to sell something versus people stating facts in a blunt way. Not every product is suitable for every use case. You get a feature at the cost of another, trade-offs are made - but these trade-offs are sometimes not discussed openly. What is the trade-off involved with MySQL Cluster? This is a quote from an official MySQL Cluster white paper: "MySQL Cluster is a distributed, shared-nothing data store that provides very high, scalable performance for applications that largely use primary key access and have high concurrency, but it incurs a cost in network access when accessing data between a MySQL Server and tables distributed across Data Nodes". Well of course it does! If you are writing your data spread across multiple data nodes which can be spread out in different regions, you will incur a latency cost. It will slow things down. Software is not magic. So why do we do it? Well, if the benefits outweigh the costs, then sure why not? But the important point to note here is this: Every piece of software tool out there wants to be "enterprise grade" - meaning they want to be fault-tolerant, BCP-enabled, DR-enabled, able to scale up and down as well as in and out. But for an ACID compliant database, this does not come free - they come at the cost of latency and performance. Therefore, as MySQL evolved into MySQL Cluster, some performance had to be sacrificed. But good engineers do not just accept that - they provide knobs and levers to adjust this and adjust that, so that users can find the sweet spot between performance and features. The same is true for MySQL cluster. When JOIN performance started suffering because of data being distributed across several nodes, the MySQL Cluster team came up with AQL in version 7.2. Various configurations are provided to fine tune performance, sometimes at the cost of durability, sometimes at the cost of RTO/RPO. However, most of that is extremely well documented. The focus of this blog post happens to be HA and Fault Tolerance because these are the reasons why we have so many performance levers in the first place!
   
   
3. In the cloud, performance of MySQL Cluster will depend on, among several other factors, the size, CPU and memory of the Virtual Machines you are running the data nodes on. MySQL uses a lot of memory as it keeps a large part of the database in memory for performance. Everything has knobs and levers of course, but the recommended RAM size on MySQL Cluster data nodes for a 50 GB database is 64 GB (see page 20 of this whitepaper). In terms of Azure VM-s, a D14 or G3 will be the right choice. It is this realization that makes me say that ClearDB is worth a shot - because when you spin up large VM-s, you want density and utilization. None of the two options (MySQL Cluster PAAS versus IAAS) will be very cheap, but in most cases either will be cheaper than having on-premises physical servers or 3rd party hosting.

Evolution of HA in the MySQL product line

Before we jump into the meat of this discussion, there is one other thing that should be kept in mind, and that is: MySQL Cluster represents the peak of MySQL's HA journey - 99.999% up-time guarantee if the architecture is done correctly. There are several other offerings that are less complex and provide less HA. Before studying MySQL Cluster, you should study these other offerings because they will be less complex to set up and definitely cheaper. If one them happens to suit your needs, there is no need to over-provision yourself in that case.

The best way to understand all these other offerings leading up to MySQL Cluster, in my opinion, is to follow the evolution path. Imagine the single server MySQL Database. It is a good ACID database, but enterprises do not just need a database to be ACID compliant - they are in constant fear of vandalism, earthquakes and dinosaur level extinction events. They want to replicate. They want Disaster Recovery with low RTO and RPO. So backing up the database once a day is no longer sufficient, it needs to be continuous.

Stage 1 - single server MySql with replication

Hence the single server MySQL Server evolved to provide simple replication mechanisms - an inbuilt feature that does not need any plugins/ add-ons. It is a simple master-slave topology. Replication can be synchronous - where the write on the master will be committed only when the write gets committed on the slave, thereby ensuring 100% consistency of the replica but at the cost of write performance - or it can be asynchronous, where the master propagates the changes out to the slave after the fact, thereby improving write latency but increasing the chance of data loss in case of disaster. The master-slave topology also provides the additional benefit of directing readers to the slave if they can work with slightly stale data, thereby reducing the load on the master, which can then funnel these resources to write operations, thereby isolating the two operations from each other.

While that sounds wonderful, there came a day when that was inadequate. As financial transactions moved to the web, even more rigor was required from the replication process. Compliance to stringent standards required multiple replicas. Keeping all of them in sync was a challenge. On top of that, who would oversee the replication itself? What if replication was failing silently for days? It would leave a bad taste in the mouth if we found that out only after the replica was actually needed. And the million dollar question of availability then started glaring people in the face: how quickly can we fail over? What use is a replica if we cannot quickly fail over to it? As every second of availability became critical to businesses, the need for sub-second fail-over arose, and with that came the need for near-real-time replication without sacrificing performance (what use is a quick fail-over if the replica is stale?)

Stage 2 - clustered and virtualized systems

At this stage, the simple master-slave replication evolved to LAN/WAN based wider "failover clustering" approaches. Balance was achieved between completely asynchronous replication (which improves write performance but increases possibility of stale reads from the slave) and completely synchronous replication (which degrades write performance but improves the chances of consistent replica on the slave). This balance was called "semi-synchronous replication" where the master commits a write when the change just reaches the slave - thereby re-defining the meaning of error and success. Multiple copies of the replica are handled in a "failover cluster", with a "cluster resource manager" constantly monitoring each copy using a heart beat mechanism, ensuring the health of the copies themselves, so that failover, when and if needed, is quick and seamless. All such mechanisms constantly monitor hardware, OS, network and database processes so that it can automatically fail over to standby servers in the event of a failure being detected, redirecting applications to new master as part of the process. Automatic failover was one of the biggest wins in this step of the evolution - lower downtime, less losses. Such solutions provided a new breed of HA-aware MySQL products: Oracle Clusterware (on Linux/ Solaris), Oracle VM Template (on Linux), Solaris Cluster (on Solaris), Windows Cluster (on Windows) and DRBD (on Linux). You can explore some aspects of these solutions in this white paper, which calls such systems "Clustered and Virtualized systems"

While that sounds even more wonderful, limitations still exist in the above systems. Availability guarantees are still lower than what mission critical and massive for-profit organizations needed. As business started operating globally, their user bases started interacting with the database from multiple regions. The concept of master-slave suddenly felt inadequate because the traditional master-slave arrangement only allows writes against the master. All of a sudden, the need for master-master arose, because we now needed the ability to write from different parts of the world. Multi-master topology brought its own challenges, like data consistency issues due to update conflicts.

Stage 3 - MySQL Cluster

MySQL Cluster, therefore, represents the peak of this evolution as of today. It attempts to resolve the above challenges, and provides 99.999% availability guarantee if the architecture is right - that comes to less than 5.5 minutes of downtime each year, including planned maintenance. I will not go into the details of how MySQL Cluster actually works. My earlier blog describes the topology of a MySQL Cluster. Consider reading this whitepaper to find out how MySQL Cluster attempts to solve these problems. MySQL Cluster is shared-nothing and geo-replicated. It has three layers - management nodes, application nodes (aka SQL Nodes), and data nodes. The most interesting part is the data nodes. We will soon see how data is spread across the data nodes for fault tolerance.

This picture shows the evolution I just described:



 

What does High Availability and Fault Tolerance mean w.r.t MySQL Cluster?

From a MySQL Cluster perspective, it means that no single failure can result in the loss of service. That guarantee, however, is dependent upon the architecture of the cluster. Not all architecture can provide that guarantee. That is one of the things that we are going to explore – how to architect the MySQL cluster on Azure so that its intrinsic guarantee of HA/ BCP is realized.

An important point to note is MySQL Cluster’s CAP Theorem stance. In case of catastrophic network partition, does MySQL choose availability or consistency? The answer is consistency (as is truly becoming of an ACID compliant database) It shuts the cluster down to prevent inconsistency in data. Therefore, it is even more important to architect the cluster properly – because a MySQL cluster with improper architecture will shut itself down in case of node or network failure, thereby failing to provide HA. Hence, the 99.999% guarantee will not be reached. That is why architecture plays a big role in determining your HA mileage out of MySQL Cluster, 99.999% being the best case scenario.

The Data Node Algorithm

MySQL Cluster, as I described earlier, has at least three layers, not counting the clients that connect to it. In the diagram below, borrowed from http://www.handybackup.net/, the "storage nodes" is the most important layer - we will call them "data nodes". The "query processing nodes" are the ones where MySQL daemon (mysqld) runs - which means clients connect to this node to run queries against - we will call them "SQL Nodes". The management nodes run the cluster orchestrating processes - they distribute workloads and arbitrate in case of node failures in case leader election is needed:




Two very important points to note about MySQL Cluster is that data is written to the data nodes (a) auto-sharded (partitioned) and (b) replicated in a synchronous manner.

In the config.ini file in the management node (typically /var/lib/mysql-cluster/config.ini), there is a key "NoOfReplicas". In the same file, the data nodes are defined - so the cluster knows the number of data nodes. From these two pieces of information (#replicas, #data nodes), MySQL Cluster calculates the number of node groups.

#Node Groups = #Data Nodes / #Replicas

It then arranges the data nodes into as many node groups. A node group is defined as the group of data nodes which contain the same partition of data, replicated {#Replicas} times. Inside a given node group, MySQL cluster establishes a primary node for a given partition, and one or more secondary nodes for the same partition - and it does this in such a way that each partition has a distinct primary and distinct secondary(ies).

Confusing? Let us try this diagram (click on it to make it bigger):




In this example, #Data Nodes = 12, #Replicas = 3 (the two inputs).

Therefore, here are the computed counts:
# Node Groups = 12/3 = 4 (marked as Node Group 1, Node Group 2, Node Group 3 and Node Group 4 in the diagram, with a-l being the names of the actual data nodes)
# Partitions per table = #Data Nodes = 12
# Partitions per node group = #Replicas = 3

What that means is that a given table will be partitioned into 12 parts (based on the hash of the primary key, unless a different algorithm is provided). In the above example, the table shown is, therefore, split into 12 partitions (12 sets of rows): Partition 1, Partition 2, ... Partition 12.

Out of these 12 partitions, 3 are selected for Node Group 1, 3 selected for Node Group 2 and so on. In the above example, say Partitions 1, 2 and 3 are selected for Node Group 1. These 3 partitions are stored in the Node Group 1 - but remember that Node Group 1 has 3 data nodes. Therefore, MySQL Cluster chooses one primary and two secondaries for each of Partitions 1, 2, and 3 in such a way that no single node in that group is the primary of more than one. See the table embedded in the diagram. One way to read the table is: Data Node a is the primary for Partition 1, hence all reads/writes for rows in this partition will first come to data node a, and then will be synchronously replicated to data nodes b and c.

This is a simplified algorithm - but one that is sufficient to explain HA in MySQL Cluster.


Fault Scenarios - the goal of MySQL Cluster Architecture

So what happens when a network partition happens and the data nodes can no longer talk to each other? Or one or more data nodes fail, thereby severing communication with other data nodes?

The data nodes are split into separate communities - each community containing nodes than can talk to each other. Each community follows this flow chart (again, clicking should give you a bigger diagram to look at):




The goal of MySQL Cluster architecture is to reach one of those two green circles for all possible single point failures. If you reach one of the red ones, you lose service, and the promise of 99.999% availability is vaporware.

At this point, I urge you to, if you have not yet done so, go through this excellent blog to understand how the architecture would impact availability if you were to do this on physical servers.

Azure specific thoughts

1. This is the recommended architecture on Azure (assuming that this cluster is contained in one single data center):
   
   
   
   
   
   Here is why: Because each node group is in its own Availability Set, at least one of them will always be up and running (unless the whole Azure data center goes down, we will talk about that later). Therefore, any residual community of data nodes after one or more failures will always be a viable community of data nodes (see definition of viable community in the flow chart above, "viable community" means a group of data nodes where there is at least one node from each Node Group). Therefore, as we traverse the flow chart, we come down to the second diamond. At this point, it either continues operating, or even if it needs arbitration, it will always be able to connect to the management node, as we have two of them sitting in an availability set - so one of them will always be available. Therefore, loss of service is prevented under all conditions except loss of entire cluster (i.e., entire Azure data center)
   

2. In order to protect your data from catastrophic disasters which can take an entire Azure data center down, you can choose to employ MySQL Cluster's geo-replication feature in Azure. Geo-replication provides geographic redundancy - while the internal (synchronous) replication provides High Availability between data nodes co-located within the cluster, the asynchronous inter-region replication to a remote site guards against site failures
   

3. A very interesting option for MySQL Cluster with Azure is the "Multi-site clustering", where two data nodes belonging to the same cluster can reside in two regions. This is normally not going to fly, as replication between data nodes is synchronous. Synchronous inter-data-center writes are normally prohibitive in terms of latency. However, Azure provides the ExpressRoute - which can provide up to 10 GBPS of throughput between Azure data centers (or even your premises and Azure cloud). A typical use case would be when we need writes happening at both ends - Active/Active configuration with no need for conflict resolution, as MySQL handles conflicts as an inbuilt feature (using time stamps and other techniques). When the user base is geographically distributed and writes must be supported from both ends, this is a viable architecture. Azure ExpressRoute can make this possible, though I do not know of anyone who has actually used MySQL Cluster this way. But MySQL Cluster provides the feature and you can give it a try on Azure!
   
   
   
   
   This option is also a viable one if the two sites are in the same geographical regions (like Azure East and East 2) - and for some reason we want to set replication up, say, for example, between corporate locations.

4.  The other option for writes happening at both ends is Active/Active Multi-Master geo-replication on two Azure data centers. This would typically increase your compute costs compared to the previous option ("Multi Site Clustering"), but the previous option will need Express Route, while this option may be feasible using IPSEC tunnels. Again, MySQL Cluster supports full multi-master Active/Active geo-replication where both clusters are active - and there is no need for application level conflict resolution. Replication can happen asynchronously or semi-synchronously in this case. An important aspect to remember about Azure is that traffic between Azure data centers traverses Microsoft owned network backbones. This is key to high geo-replication performance - one of the several ways that Azure makes it easier to use MySQL Cluster.
   
   
   
   

5.  Another use case where geo-replication make sense on Azure is reporting. Several reporting tools can act on a MySQL data store and run complex queries - including Power BI. However, the recommended MySQL Engine optimized for such a use case is InnoDB, not NDB Cluster. We can use Active/Passive asynchronous geo-replication to offload the data to an InnoDB engine store where we can point our report and dashboard generation applications to
   
   

6.  For applications that read/write/update a bunch of records per transaction, it is recommended that we build in "distribution awareness" in the application - where partitioning happens based on a sub-key common to all the rows being affected by the transaction (e.g., transaction or session id) - so that all records go to the same data node. This is especially required if we are using Multi Site Clustering
   
   

7.  Azure can provide the perfect platform for real time applications that need fast propagation of updates across nodes or clusters or regions. Basically, applications that need low latency real time updates (e.g., reflecting status updates on financial transactions) either across the nodes of a cluster or across multiple clusters, can use a feature in MySQL where most or all data can be stored just in memory (RAM). This provides millisecond level cross node/cross cluster updates, but the cost incurred is durability/RTO/RPO. On a typical on-premises installation, power outages taking down the whole cluster will result in loss of the data that was still held in RAM. MySQL provides a way to mitigate this problem – continuous/ frequent disk-check-pointing and redo-logging, but that can reduce performance as it uses CPU and disk. However, a commercial cloud like Azure can take these trade-offs away. A MySQL cluster can be easily deployed with individual nodes distributed across availability sets, thus protecting the cluster from power/ rack failures. Therefore, real time applications can use the in-memory feature without the fear of losing data – and also without the need for continuous disk check-pointing and redo-logging which compromise performance
   
   

8. MySQL Cluster's operational agility features (online scaling by adding more slaves or nodes) are very nicely complemented by Azure auto-scaling features

Conclusion

In my opinion, ClearDB is a good example of what can be done with MySQL on Azure. They are running a successful commercial PAAS MySQL Cluster service on Azure, which is simply fantastic. As we work with more and more partners to deploy complex clusters on Azure, I will come back and update this blog. Thanks! 

Introduction

In an earlier blog post, I wrote about setting up a MySQL single server instance on Azure Linux (Ubuntu) Virtual Machine. We also did some performance bench-marking around READ-s with lots of joins.

However, most enterprises would be interested in exploring a full-fledged MySQL cluster as opposed to just a single server. MySQL cluster uses the NDB engine to manage the different nodes, and data is saved in the data nodes in such a way that it is extremely fault-resilient, scalable and high performing.

In this blog post, I will explore how to set up a 4 node MySQL IAAS cluster on Azure.

Note: ClearDB now provides PAAS MySQL Clusters on Azure as well. In fact, their offering seems very robust and technically a lot more attractive than setting your own IAAS instance because of the recently announced option of Premium Dedicated Clusters. I said "technically" because I have not had the chance to compare the cost model of going with them versus hosting your own IAAS cluster on Azure. In this blog post, we will focus on the IAAS option.

Architecture

These steps are listed on mysql website in a generic fashion - http://dev.mysql.com/doc/mysql-cluster-excerpt/5.1/en/mysql-cluster-installation.html

However, these steps need some modifications on Azure Virtual Machines running Ubuntu LTS 14.04. We will try to capture the differences here.

Here is the conceptual architecture diagram of what we are trying to achieve. We will, as is obvious from this diagram, we will spin up 4 Virtual Machines - 2 for the data nodes, 1 for SQL Node, and 1 for the NDB Management node. You can set up a 5th node as a client if you want to run performance tests, and we will also briefly cover that. However, to hold against a large performance test run on the cluster, the cluster nodes need to be beefy, as MySQL has a tendency to consume large amounts of memory unless tuned well - especially queries with JOINS needs AQL. Discussing such optimizations in not within the scope of this blog post.

Introduction

In an earlier blog post, I wrote about how we can configure an Azure Linux (Ubuntu) VM with a RAID 0 array of data disks and mount XFS on the array. The goal of doing this was gaining disk I/O, hence performance.

In order to substantiate this claim, I am following up with some performance tests run with the popular Open Source RDBMS database - MySQL. This blog shows how to set up the test using Azure Linux (Ubuntu) VM-s from scratch, and also discusses the results.

Install Mysql

Before we dive into the details of the performance test and its results, let us quickly review what we need to do to install and run MySQL database on an Azure Linux (Ubuntu) VM. Here are the steps:

1. Provision an Azure Linux (Ubuntu 14.04 LTS) VM through the portal, Azure CLI, Powershell or REST API-s - or one of the several other ways now available to do so (like this open source go sdk).
   
   NOTE A: Please remember that we are going to benchmark SQL Server performance on this machine - hence the VM instance type should be carefully chosen. We recommend at least D-series for this purpose. If you plan to add data disks to this VM later on in order to create a RAID array, please pay close attention to the maximum number of data disks allowed as it varies per VM instance type. In order to complete this tutorial as-is, you should choose A2 or higher if you are going with A series, D2 or higher if you are going with D series. That is because we stripe together 4 data disks, and A1/D1 does not allow four data disks.
   
   NOTE B: Please use the "Resource Manager" deployment method (as opposed to the "Classic" method). "Resource Manager" is the newer method and provides expanded manageability of your VM. Also, please create a new Resource Group (name it something meaningful like "mysqltest") and also create a new Virtual Network for this VM (also name it something meaningful). This is important because we will later create a load testing client machine and add it to the same Resource Group and Virtual Network. Adding the client machine to the same network as the server machine results in the load traffic not needing to cross any switches, thus avoiding variance.
    
2. Log into the VM using Putty (or any other SSH client)
3. sudo apt-get update
4. sudo apt-get upgrade
5. sudo apt-get install mysql-server
6. sudo mysql_secure_installation (set root password, make other changes to make your installation as necessary - you can leave anonymous access on for convenience during testing)
7. mysql -u root -p (test whether you can use the root password that you set in the above step)
8. sudo netstat -tap | grep mysql (test if you can detect the mysqld daemon running)
9. sudo service mysql restart (test if mysql has registered as a service)
10. mysqlslap -uroot -p{Root password} (test whether the load testing client tool has been installed properly)
11. vi /etc/mysql/my.cnf (or use any other editor - review this configuration file carefully, we are going to make changes to this file down the line to change the way mysql behaves)

[ These steps are consolidated as a shell script here (check out "install-mysql-on-azure-ubuntu-vm-needs-sudo.sh"): https://github.com/kbxkb/mysql-on-azure ]

1. mkdir employees_sample_db; cd employees_sample_db/
2. wget "https://launchpad.net/test-db/employees-db-1/1.0.6/+download/employees_db-full-1.0.6.tar.bz2" (NOTE: Please make sure that you are using the latest and correct URL, launchpad may have a more current version)
3. tar -xjf employees_db-full-1.0.6.tar.bz2
4. cd employees_db/
5. mysql -uroot -p{Root password} -t < employees.sql
6. Follow the steps here to install, validate and get the database up and running (note that the last few steps above are also included in this guide, so adjust your starting point accordingly)
7. Log into the mysql> prompt (mysql -uroot -p{root password}) and run show databases; to verify that the employees database is now installed on your server

[ These steps are consolidated as a shell script here (check out "install-employees-db-pass-mysql-root-pwd-as-parameter.sh"): https://github.com/kbxkb/mysql-on-azure ]

CREATE USER 'perftest'@'localhost' IDENTIFIED BY 'Azure123';
GRANT ALL PRIVILEGES ON *.* TO 'perftest'@'localhost' WITH GRANT OPTION;
CREATE USER 'perftest'@'%' IDENTIFIED BY 'Azure123';
GRANT ALL PRIVILEGES ON *.* TO 'perftest'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;
SELECT * from information_schema.user_privileges where grantee like "'perftest'%";

select salaries.emp_no, salaries.to_date, dept_emp.dept_no FROM salaries INNER JOIN dept_emp ON salaries.emp_no = dept_emp.emp_no WHERE salaries.to_date BETWEEN '1990-01-01' AND "1991-12-31" ORDER BY salaries.emp_no, salaries.to_date;

SELECT e.first_name, e.last_name, d.dept_name, t.title, t.from_date, t.to_date FROM employees e INNER JOIN dept_emp de ON e.emp_no=de.emp_no INNER JOIN departments d ON de.dept_no=d.dept_no INNER JOIN titles t ON e.emp_no=t.emp_no ORDER BY e.first_name, e.last_name, d.dept_name, t.from_date;

Mysqlslap

(We should now inspect the load test tool that we will use: mysqlslap. It is a good idea to read up on the tool's features, parameters etc. so that you can make the best use of it.)

NOTE A: mysqlslap has several parameters and supports running load tests in many different ways. You should use it in several other ways as well to get a well-rounded picture. The "--debug-info" parameter results in the output containing interesting system information during the test like memory, CPU, etc. I found the "involuntary context switches" statistics to be one of the few interesting things that mysqlslap reports.

NOTE B: Use the correct value for --host (it should be the Private IP Address of your MySQL server VM that you can easily look up from the Azure portal). Use the correct user id and password as created above or otherwise.

NOTE C: Interpreting mysqlslap's output, quite interestingly, has been an non-intuitive journey for myself. For a given value of --concurrency parameter and a given value of --iterations parameter, its output may look like this:

Average number of seconds to run all queries: 167.015 seconds
Minimum number of seconds to run all queries: 166.215 seconds
Maximum number of seconds to run all queries: 167.995 seconds

Naturally, if my concurrency was say 20, and iterations was say 5, I would want to know what exactly took 167 seconds? The phrase "all queries" initially threw me off - I was thinking "wait, do I again need to divide it by the number of queries to get the real mean I need?"

Turns out that I do not need to. If concurrency was 20, mysqlslap created 20 client threads, each running my given query once (provided I have not used the --number-of-queries parameter). That was one single test. It then repeated that single test 5 times if I have said --iterations=5. So it ran my query 20 * 5 = 100 times. 167.015 seconds is the average across all these 100 runs. It says "all queries" because my query could have easily been a set of queries instead of just one (it also accepts a .sql file with several queries in it). That is why it resorts to a generic phrase "all queries". I suspected this all along, but had to take an incredibly long journey to actually confirm this.

Here is the mysqlslap command that fires off the above query. Please note that the value of --concurrency parameter, as quoted below, is 'XX' because that is what I varied during different test runs.

mysqlslap -uperftest -pAzure123 --host=10.1.0.4 --create-schema=employees --query="SELECT e.first_name, e.last_name, d.dept_name, t.title, t.from_date, t.to_date FROM employees e INNER JOIN dept_emp de ON e.emp_no=de.emp_no INNER JOIN departments d ON de.dept_no=d.dept_no INNER JOIN titles t ON e.emp_no=t.emp_no ORDER BY e.first_name, e.last_name, d.dept_name, t.from_date;" --concurrency=XX --iterations=10 --verbose --debug-info

Results on OS disk

Here are the results with the employees database still residing on the OS disk:

Switch to RAID 0 array as data directory

Now that we have executed our READ test against the ext4 OS Disk, we can now, in order to compare, run it against an alternate partition - the RAID 0 array of 4 standard storage data disks mounted with XFS. In order to do so, you need to follow my earlier blog and add 4 data disks to the mysql server VM, stripe them in a RAID 0 array and mount it as an XFS partition - if you have not already done so.

NOTE: You can add data disks and follow all the steps to stripe, partition and mount them without disturbing the software installed on the OS disk

Once you have followed those steps, you should have a mount point "/RAID0" on your server VM:

$ df -T
 Filesystem Type 1K-blocks Used Available Use% Mounted on
 /dev/sda1 ext4 30202916 4062200 24874684 15% /
 none tmpfs 4 0 4 0% /sys/fs/cgroup
 udev devtmpfs 1756712 12 1756700 1% /dev
 tmpfs tmpfs 352320 452 351868 1% /run
 none tmpfs 5120 0 5120 0% /run/lock
 none tmpfs 1761584 0 1761584 0% /run/shm
 none tmpfs 102400 0 102400 0% /run/user
 none tmpfs 64 0 64 0% /etc/network/interfaces.dynamic.d
 /dev/sdb1 ext4 139203560 60928 132048464 1% /mnt
 /dev/md127 xfs 1048047616 741848 1047305768 1% /RAID0
$

Now, we have to copy the entire MySQL database part to /RAID0 partition from / (i.e., to /dev/md127 from /dev/sda) and also point mySQL at the new location.

Follow these steps to do so:

1. sudo service mysql stop
2. sudo netstat -tap | grep mysql #check if stop worked
3. rm -rf /RAID0/mysql #because rsync will create the mysql folder
4. sudo rsync -avzh /var/lib/mysql /RAID0 #rsync parameters will make sure that all directory permissions and ownerships will be preserved
5. edit /etc/mysql/my.cnf and change (line following '-' is replaced by the line following '+' without the -/+ sign themselves being part of the lines):
   
   -datadir     = /var/lib/mysql
   +datadir     = /RAID0/mysql
   
   
6. sudo mv /var/lib/mysql /var/lib/mysql-old
7. edit /etc/apparmor.d/usr.sbin.mysqld and change (line following '-' is replaced by the line following '+' without the -/+ sign themselves being part of the lines):
   
   -  /var/lib/mysql/ r,
   -  /var/lib/mysql/** rwk,
   +  /RAID0/mysql/ r,
   +  /RAID0/mysql/** rwk,
   
   
8. sudo /etc/init.d/apparmor reload
9. sudo service mysql start
10. sudo netstat -tap | grep mysql #check if start worked

[ These steps are consolidated as a shell script here (check out "switch-mysql-data-from-var-lib-to-new-dir-needs-sudo-needs-newdir-as-param.sh"): https://github.com/kbxkb/mysql-on-azure ]

 
 

mysqlslap -uperftest -pAzure123 --host=10.1.0.4 --auto-generate-sql --auto-generate-sql-load-type=write --concurrency=100 --iterations=10 --verbose