1. Enable bash for windows : Follow the steps mentioned in Enable bash for Windows. Once you have bash has been configured we can proceed with the next steps.
   
   
2. Installing docker client for bash on windows: Follow the steps mention in Install docker client for bash on windows. The commands executed by me are shown below (please follow link for latest instructions) sudo apt-get update sudo apt-get install apt-transport-https ca-certificates curl software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo apt-key fingerprint 0EBFCD88 sudo apt-get update sudo apt-get install docker-ce
   
3. Installing docker-compose for bash on windows: Follow the steps mention in Install docker-compose in bash on windows. The commands executed by me are shown below (please follow link for latest instructions) sudo curl -L https://github.com/docker/compose/releases/download/1.16.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose
   
4. Configuring bash using .bashrc to connect to the docker daemon running on the windows machine: This step will differ depending on how the docker engine is running on your machine, i.e. whether you are using docker for windows, docker-machine, or minikube. If you dont have any of these then would recommend you install docker for windows following the steps mentioned in the link Install docker for windows. Depending on the engine you will need to make different enteries in your ~/.bashrc file, lets take a look:
   
   
   
   • If using minikube: We first fire the command minikube docker-env command, and then set the corresponding environment variables in our bash for windows ~/.bashrc file. You can execute the following minikube command from cmdshell , powershell or gitbash i.e. from where ever you normally execute minikube commands
     $ minikube docker-env export DOCKER_TLS_VERIFY="1" export DOCKER_HOST="tcp://192.168.99.100:2376" export DOCKER_CERT_PATH="C:\Users\username\.minikube\certs" export DOCKER_API_VERSION="1.23" # Run this command to configure your shell: # eval $(minikube docker-env) Now we set corresponding values in the ~/bashrc file. We will need to change the value of the DOCKER_CERT_PATH, as C:\ is mounted as /mnt/c for bash on windows . # value set at bottom of ~/.bashrc export DOCKER_TLS_VERIFY="1" export DOCKER_HOST="tcp://192.168.99.100:2376" export DOCKER_CERT_PATH="/mnt/c/Users/username/.minikube/certs" export DOCKER_API_VERSION="1.23"
     
   • If using docker for windows: In this case first we can open docker for windows settings and enable the " Expose daemon on tcp://localhost:2375 without TLS" option (Note when enabling this you need to aware about possibility of remote execution attacks). Once this option has been enabled you can add the following to the end of the ~/.bashrc file export DOCKER_HOST=tcp://0.0.0.0:2375
     
   • if using docker-machine: Similar to the minikube steps we will first execute docker-machine env machine-name command, and then set the environment variables in the ~/.bashrc file. $ docker-machine env dmach SET DOCKER_TLS_VERIFY=1 SET DOCKER_HOST=tcp://192.168.99.102:2376 SET DOCKER_CERT_PATH=C:\Users\username\.docker\machine\machines\dmach SET DOCKER_MACHINE_NAME=dmach SET COMPOSE_CONVERT_WINDOWS_PATHS=true REM Run this command to configure your shell: REM @FOR /f "tokens=*" %i IN ('docker-machine env dmach') DO @%i . We will modify the DOCKER_CERT_PATH and then set the environment variables as follows: # value set at bottom of ~/.bashrc export DOCKER_TLS_VERIFY=1 export DOCKER_HOST=tcp://192.168.99.102:2376 export DOCKER_CERT_PATH=/mnt/c/Users/username/.docker/machine/machines/dmach export DOCKER_MACHINE_NAME=dmach export COMPOSE_CONVERT_WINDOWS_PATHS=true
     
   We can verify if we have configured this step correctly by opening a new bash shell and typing docker ps, this should show us the currently running containers.
5. Installing the vscode docker extension: You can read more about the vscode docker extension at vscode docker extension. To install you can press F1 in vscode and type "ext install vscode-docker". Alternately you can use the vscode extensions tab to install the extension.
6. Configuring vscode to use bash as the default terminal shell: Press Ctrl+, or go File->Preferences->Settings. Then add the value below to the JSON file "terminal.integrated.shell.windows": "C:\\Windows\\sysnative\\bash.exe". After this step if you open up a new terminal in vscode a bash shell should open up. .

Release Overview

1. Build immutable image using packer: In this step we use the Build immutable image task. We pass the path to the phpapp-packer.json file to this task. We will need VSTS Variables ARM_SUBSCRIPTION_ID, ARM_CLIENT_ID, ARM_CLIENT_SECRET, and ARM_TENANT_ID to be set. You can revisit my post to find out how to get value of these variables. Additionally we provide name of VSTS variable (IMAGEURI) to this task, this variable needs to be created in VSTS. Once the VM image is baked the task will save the image uri for the baked VM image to this variable (IMAGEURI). Screenshot below shows configuration of this task:
   
2. Replace Tokens Task: This task replace tokens in the Azure Deployment Parameters File with values from VSTS variables. Snippet below is from parameters file: "adminUsername": { "value": "@@VMUSERNAME@@" }, "imageUri":{ "value": "@@IMAGEURI@@" }, "adminPassword": { "value": "@@VMPASSWORD@@" } The replace token task will replace token @@VMUSERNAME@@ with value of the VSTS variable VMUSERNAME, similarly token @@IMAGEURI@@ will be replaced by the VSTS variable IMAGEURI value, which was set by the previous packer task. screenshot below shows configuration of this task:
   
3. Azure Resource Group Deployment Task: This task will update the VM Image associated with the VM Scaleset. The ARM template file azuredeploy.json and parameter file (where tokens have been replaced by variables by previous task) are passed to this task. Screenshot below shows configuration of this task:
   

• The bot relied on pattern matching for intent detection. Which means if the exact pattern was not found the intent would be missed
• The bot was utilized the node-slack-client library and only connected to the slack channel. If I wanted to connect the bot to a different channel like skype I would need to use channel specific connectors to connect the bot, which would mean more work

• The file vsts-bot-LUIS-model.json is the LUIS Model to recognize intents and entities associated with this application. This model can be imported when creating your application at luis.ai . In the old version of the application patterns like "/task #(\d{1,9})(.*)/i" (view task/issue pattern) were used to identify user intents. In this version LUIS identify intents based on the training utterances shown in the LUIS model JSON file
• Since This version of the application is built using the Microsoft Bot Framework, after we have built our bot we can separately enable the bot on the several supported channels including slack, skype, facebook messenger etc
• Interaction with VSTS is similar to the earlier version of the bot except the fact that earlier version of code was in coffeescript, and this one is in javascript
• The main entry point of this solution is the app.js file. The key lines of code are shown below : var builder = require('botbuilder'); var restify = require('restify'); var request = require('request'); builder represents the Microsoft Botframework nodejs SDK which has abstractions for communicating with the Bot Connector (which connects bot to various channels) and also abstractions to communicate with LUIS to get the intents and associated entities. restify is used to expose a REST endpoint which the bot connector can use to communicate with our bot. request library will be used to make call to the VSTS REST endpoints. // Bind to REST endpoint where the Bot Connector will communicate with our bot var server = restify.createServer(); server.listen(process.env.port || process.env.PORT || 4141, function () { console.log('%s listening to %s', server.name, server.url); }); . . // Create chat bot which can be linked to various channels like slack and skype var connector = new builder.ChatConnector({ appId: process.env.MICROSOFT_APP_ID, appPassword: process.env.MICROSOFT_APP_PASSWORD }); var bot = new builder.UniversalBot(connector); server.post('/api/messages', connector.listen()); . // Initialize the LUIS intent recognizer // This Url can be obtained by uploading or creating your model from the LUIS portal: https://www.luis.ai/ const LuisModelUrl = process.env.LUIS_MODEL_URL; . // Main dialog with LUIS var recognizer = new builder.LuisRecognizer(LuisModelUrl); var intents = new builder.IntentDialog({ recognizers: [recognizer] }) . // If User greeted the bot, i.e. LUIS identified the intent as greeting the send message to user .matches('greeting', (session, args) => { session.send('Hi There'); }) . . // Similar code for other intents // Simlarly match next set of events and perform actions
• This version of the bot acts on an additional intent "userfacingissue". When user confirms this intent the bot creates an issue in VSTS

Build and Release Pipeline Overview

1. Configuring the custom linux agent for build and release: Have a look at the link VSTS Custom Linux Agent
to create you own linux agent
2. Configuring the custom agent to execute Packer: Information on installing latest Packer executable can be found at install packer. The script below shows installation of packer version 0.12.0 . (Note : Built in VSTS packer task is now available, You don't need to configure agent to use packer if you are planning to use that task. You can read my post to configure the built in VSTS Packer task) sudo apt-get install unzip wget https://releases.hashicorp.com/packer/0.12.0/packer_0.12.0_linux_amd64.zip unzip packer_0.12.0_linux_amd64.zip sudo mv packer /usr/bin/packer
3. Configuring the custom agent to execute phpunit and selenium scripts: Follow the link to install phpunit (only if you are using a php based application) and then you can move it to /usr/bin/phpunit. You can use the link to install firefox. You can use the link to install xvfb and selenium stand alone as a service. Once these tools are installed on the agent, we are good to configure the release pipeline.
4. Build configuration : For simplicity the build is configured to publish all files are folders in the repository.
   
5. Configuring the Release. The file VSTS-Release-Def.json under the build-files folder has the sample release definition which can be imported in VSTS :
   • A new release is configured to be created on each successful build.
     
   • Following are the environment variables which are configured for the release. The ARM_CLIENT_ID, ARM_RESOURCE_GROUP, ARM_STORAGE_ACCOUNT, ARM_SUBSCRIPTION_ID, and ARM_TENANT_ID are used by packer to save the baked VM Image VHD into the configured Azure storage account. ADMIN_USERNAME, and ADMIN_PASSWORD are the the logon credentials for the VM scale set VMs. DEPLOYMENT_RESOURCE_GROUP is the resource group where the VM scale set is deployed. These are passed to the different tasks in the release (as we will see when we go through each task in detail).
     
   • Bake VM image with latest application updates baked in, using packer: In this step we use the execute shell script task to call the build-vm-image.sh file, and pass it the parameters required by packer to create the vm image in the configured Azure storage account.Image below shows configuration of this shell script step: ( Note : Built in VSTS packer task is now available, to use that task instead you can read my post Using the VSTS Packer Task and Replace Tokens Task )
     
     Following code block shows key lines from the shell script. The environment variables are passed as positional parameters to the script in this example: export ARM_CLIENT_ID=$1 export ARM_CLIENT_SECRET=$2 export ARM_RESOURCE_GROUP=$3 export ARM_STORAGE_ACCOUNT=$4 export ARM_SUBSCRIPTION_ID=$5 export ARM_TENANT_ID=$6 export ADMIN_USERNAME=$7 export ADMIN_PASSWORD=$8 packer build ./packer-files/phpapp-packer.json 2>&1 | tee packer-build-output.log The packer build configuration file php-packer.json has all the configuration to enable packer to package the latest application code and bake the VM image (including base VM image and Azure region). Following configuration section from the packer configuration file demonstrates how the application dist folder is copied into the baked image using the file provisioner: "provisioners": [ . . { "type": "file", "source": "../dist", "destination": "/tmp/dist" }, . . . ] In the example other than file and shell script provisioners, we have used the ansible provisioner, but packer supports several other provisioners like chef, puppet, powershell etc. The output of the packer build command is written to the file packer-build-command-output.log. Packer writes the image uri of the newly created vhd in the format : OSDiskUri: https://*********/system/Microsoft.Compute/Images/images/packer-osDisk.5a892bb0-*********-410f-8198-92c8b140390e.vhd (see file packer-build-output.log for complete output of packer build command)
   • Replace image uri in ARM template deployment parameters file, with uri of vhd generated by packer in preceding step.
     
     The shell script overwrite-azure-deployment-parameters.sh is invoked, which parses the packer out log file to retrieve the uri of the newly created vhd. it then replaces the parameter value azuredeploy.parameters.json file. Code extract of file is shown below : export imageodisk=$(cat packer-build-output.log | grep OSDiskUri: | awk '{print $2}') sed -i 's/@@VMUSERNAME@@/'"$ADMIN_USERNAME"'/g' azuredeploy.parameters.json sed -i 's/@@VMPASSWORD@@/'"$ADMIN_PASSWORD"'/g' azuredeploy.parameters.json sed -i 's|@@IMAGEURI@@|'"$imageodisk"'|g' azuredeploy.parameters.json Note : The VSTS VMSS task is now available, if the VSTS Packer task is used along with the VSTS VMSS task then this step and the next step "Push updated VM image to VM scale set" may not be required. See the Post on how to use these tasks.
   • Push updated VM image to VM scale set using Azure Cli group deployment: This is done by using the Azure cli task which calls the deploy-vm-image.sh file and passes the resource group where the VM Scale set is deployed.
     
     This script then executes the azure group deployment command passing it the ARM template and parameters file. The script is shown below : azure group deployment create -f azuredeploy.json -e azuredeploy.parameters.json -g $1 -n rocdashdeploy$2
   • Execute Selenium based Smoke test on Application.
     
     Phpunit is used to then execute the site smoke tests: phpunit $1site-smoke-test.php $2
   • Execute Load tests on Application
     <img src="https://raw.githubusercontent.com/maniSbindra/vsts-packer-vmss-php-webapp-release/master/images/execute-jmeter-load-tests.PNG" alt="Execute JMeter Load tests on application" width="60%" height="auto" />
     The JMeter Load test execution task then fires JMeter Load Tests against the site site using the JMX file site-load-test.jmx (which has duration assertions) .

Why Docker? I wanted to make it seamless for anyone wanting to replicate this solution, and then tweak it as per their needs, and let’s face it once you start using docker, it is difficult to turn back, you want to use it for every solution. The hypriot image comes loaded with the docker engine, and is one of the easiest ways to get started with docker on the pi.

Overview : Noise Pollution Monitor (packaged as a docker container) using Lego mindstorms sensors, Brickpi and Raspberry Pi

Raspberry pi is loaded with the Hypriot version 0.8.0 - Barbossa image. Once the pi has booted up with this image, docker is available on the pi. All dependencies needed for the pi to control the Mindstorms sensors, and the Brick pi (including the Brickpi_Python library) are built into the docker container image, using the Dockerfile shown in the project's github repository.

To start the monitor we can run the container as follows  :

docker run -e INTENSITY_THRESHOLD=40 -d --device /dev/ttyAMA0:/dev/ttyAMA0 --device /dev/mem:/dev/mem --privileged -ti maninderjit/pi-brickpi

(Sound intensity threshold configured to 40% using -e switch of the docker run command above)

• drupalVmSku: "Standard_D1_v2" (Size of the Drupal VMs)
• drupalUbuntuOSVersion: "14.04.4-LTS" (OS Version for the Drupal VMs)
• drupalAdminUser: "drupalAdmin" (Drupal Site Administrator)
• drupalAdminPassword: "drupalAdmin-123" (Password for Drupal Administrator)
• drupalVersion: "8.1.1" (The Drupal Version)
• vmssName: "dru25vmss" (Name of the VM Scaleset)
• instanceCount: "2" (Default and minimum number of Drupal VMs in the VM scaleset)
• maximumInstanceCount: "10" (Maximum Number of Drupal VMs in the scale set)
• adminUsername: "drupvmuser" (The ssh username for the drupal and gluster vms)
• adminPassword: "DrupVm-pass1234" (The ssh password for drupal and gluster vms)
• drupalInstallationDatabaseName: "drupaldb" (the drupal database name using which the drupal site will be installed)
• createNewMySQLServer: "yes" (No is default, details of existing SQL Server need to be provided)
• existingMySqlFQDN: "anything" (FQDN of existing MySQL server). This can be set to any value. Will be ignored when creating new mysql server .
• existingMysqlUser: "anything". When using the Mysql-Replication template the user name created will be "admin"
• mysqlUserPassword: "Mysql-Pass1234" (The mysql admin users password will be set to this)
• glusterTemplateLocation: "https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/gluster-file-system/" (Base location of ARM template used to create the Gluster cluster)
• templateLocation: "https://raw.githubusercontent.com/maniSbindra/azure-quickstart-templates/master/301-drupal8-vmss-glusterfs-mysql" (Base location of the the Drupal ARM template and scripts)
• storageAccountNamePrefix: "dru25stacc" (prefix for the VM storage accounts, which need to be unique)
• glusterVmSize: "Standard_D1_v2" (Size of the gluster VMs)
• glusterInstanceCount: "2" (Number of nodes in the gluster cluster)
• newMySqlDnsName: "mysqlserverdnsname" (The FQDN of the MySQL server will be dnsName.location.cloudapp.azure.com)
• newMySqlVmUserName: "mysqlVmUser" (user name used for ssh)
• newMySqlVmPassword: "MySQLVM-Pass1234" (password used for ssh)
• newMySqlVmSize: "Standard_D1_v2"
• dbCentosOSVersion: "6.6"

• drupalVmSku: "Standard_D1_v2" (Size of the Drupal VMs)
• drupalUbuntuOSVersion: "14.04.4-LTS" (OS Version for the Drupal VMs)
• drupalAdminUser: "drupalAdmin" (Drupal Site Administrator)
• drupalAdminPassword: "drupalAdmin-123" (Password for Drupal Administrator)
• drupalVersion: "8.1.1" (The Drupal Version)
• vmssName: "dru25vmss" (Name of the VM Scaleset)
• instanceCount: "2" (Default and minimum number of Drupal VMs in the VM scaleset)
• maximumInstanceCount: "10" (Maximum Number of Drupal VMs in the scale set)
• adminUsername: "drupvmuser" (The ssh username for the drupal and gluster vms)
• adminPassword: "drupvmpass" (The ssh password for drupal and gluster vms)
• drupalInstallationDatabaseName: "drupaldb" (the drupal database name using which the drupal site will be installed)
• createNewMySQLServer: "no" (No is default, details of existing SQL Server need to be provided)
• existingMySqlFQDN: "host123.southcentralus.cloudapp.azure.com" (FQDN of existing MySQL server) . The MySQL Server needs to be accessible from the Drupal VMs.
• existingMysqlUser: "admin" (the user name used to connect to the mysql server instance)
• mysqlUserPassword: "Mysql-Pass1234" (The password used to connect to the MySQL server instance)
• glusterTemplateLocation: "https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/gluster-file-system/" (Base location of ARM template used to create the Gluster cluster)
• templateLocation: "https://raw.githubusercontent.com/maniSbindra/azure-quickstart-templates/master/301-drupal8-vmss-glusterfs-mysql" (Base location of the the Drupal ARM template and scripts)
• storageAccountNamePrefix: "dru25stacc" (prefix for the VM storage accounts, which need to be unique)
• glusterVmSize: "Standard_D1_v2" (Size of the gluster VMs)
• glusterInstanceCount: "2" (Number of nodes in the gluster cluster)

• newMySqlDnsName: "NA"
• newMySqlVmUserName: "NA"
• newMySqlVmPassword: "NA"
• newMySqlVmSize: "NA"
• dbCentosOSVersion: "6.6"

1. Setting up migration script execution machine : I have used the Centos 6.7 VM image from Azure, to create the script execution VM. This VM is needed to execute the backup and restore commands. Once the the VM has been created ssh on the vm and execute the following command to install the mysql client utilities:
   sudo yum install mysql
    
2. Backing up AWS RDS MySql database to file : you will need the FQDN, database name, username and password of your AWS RDS mysql instance to execute the following command, which will create backup file for your database. Please make sure that the RDS MySql instance is accessible from the script execution vm (setting in AWS):
   mysqldump -h rdsmysqlserverfqdn -P 3306 databasename > dbbackupfile.sql -u dbusername -p
   You will be prompted to enter your RDS MySql password after you execute this command. After execution is complete you will see the backup file dbbackupfile.sql in your directory
3. Restoring database toAzure : Azure has both PaaS (ClearDB) and IaaS option for MySql.
   • Restore to ClearDB (PaaS) : This link shows how to create a ClearDB MySql instance. We need the ClearDB server name, database name, username and password to restore the database from backup. Once you have these details execute the following command :
     mysql -h cleardb_server_name -P 3306 -u cleardb_username -p cleardb_databasename < dbbackupfile.sql
     You will be prompted to enter your ClearDB password after you execute this command. After execution is complete you will see that the database has been restored
   • Restore to MySql deployed to VMs: One of the most popular MySQL Master-Slave setups can be achieved by deploying the mysql-replication ARM template (by clicking on deploy to azure button). The Readme.md file of this template has instructions to do various things including backing up the database to azure blob storage. Once this template has been deployed we need to restore the backup using the following script:
     mysql -h serverfqdn -P 3306 -u admin -p -D test < dbbackupfile.sql
     You will be prompted to enter your admin password after you execute this command. After execution is complete you will see that the test database has been restored
   
   
   Thanks for reading my blog. I hope you liked it. Please feel free to write your comments and views about the same over here or at @manisbindra