Automate Everything (Building Real-World Cloud Apps with Azure)
Download Fix It Project or Download E-book
The Building Real World Cloud Apps with Azure e-book is based on a presentation developed by Scott Guthrie. It explains 13 patterns and practices that can help you be successful developing web apps for the cloud. For an introduction to the e-book, see the first chapter.
The first three patterns we'll look at actually apply to any software development project, but especially to cloud projects. This pattern is about automating development tasks. It's an important topic because manual processes are slow and error-prone; automating as many of them as possible helps set up a fast, reliable, and agile workflow. It's uniquely important for cloud development because you can easily automate many tasks that are difficult or impossible to automate in an on-premises environment. For example, you can set up whole test environments including new web server and back-end VMs, databases, blob storage (file storage), queues, etc.
DevOps Workflow
Increasingly you hear the term "DevOps." The term developed out of a recognition that you have to integrate development and operations tasks in order to develop software efficiently. The kind of workflow you want to enable is one in which you can develop an app, deploy it, learn from production usage of it, change it in response to what you've learned, and repeat the cycle quickly and reliably.
Some successful cloud development teams deploy multiple times a day to a live environment. The Azure team used to deploy a major update every 2-3 months, but now it releases minor updates every 2-3 days and major releases every 2-3 weeks. Getting into that cadence really helps you be responsive to customer feedback.
In order to do that, you have to enable a development and deployment cycle that is repeatable, reliable, predictable, and has low cycle time.
In other words, the period of time between when you have an idea for a feature and when the customers are using it and providing feedback must be as short as possible. The first three patterns – automate everything, source control, and continuous integration and delivery -- are all about best practices that we recommend in order to enable that kind of process.
Azure management scripts
In the introduction to this e-book, you saw the web-based console, the Azure Management Portal. The management portal enables you to monitor and manage all of the resources that you have deployed on Azure. It's an easy way to create and delete services such as web apps and VMs, configure those services, monitor service operation, and so forth. It's a great tool, but using it is a manual process. If you're going to develop a production application of any size, and especially in a team environment, we recommend that you go through the portal UI in order to learn and explore Azure, and then automate the processes that you'll be doing repetitively.
Nearly everything that you can do manually in the management portal or from Visual Studio can also be done by calling the REST management API. You can write scripts using Windows PowerShell, or you can use an open source framework such as Chef or Puppet. You can also use the Bash command-line tool in a Mac or Linux environment. Azure has scripting APIs for all those different environments, and it has a .NET management API in case you want to write code instead of script.
For the Fix It app we've created some Windows PowerShell scripts that automate the processes of creating a test environment and deploying the project to that environment, and we'll review some of the contents of those scripts.
Environment creation script
The first script we'll look at is named New-AzureWebsiteEnv.ps1. It creates an Azure environment that you can deploy the Fix It app to for testing. The main tasks that this script performs are the following:
- Create a web app.
- Create a storage account. (Required for blobs and queues, as you'll see in later chapters.)
- Create a SQL Database server and two databases: an application database, and a membership database.
- Store settings in Azure that the app will use to access the storage account and databases.
- Create settings files that will be used to automate deployment.
Run the script
Note
This part of the chapter shows examples of scripts and the commands that you enter in order to run them. This a demo and doesn't provide everything you need to know in order to run the scripts. For step-by-step how-to-do-it instructions, see Appendix: The Fix It Sample Application.
To run a PowerShell script that manages Azure services you have to install the Azure PowerShell console and configure it to work with your Azure subscription. Once you're set up, you can run the Fix It environment creation script with a command like this one:
.\New-AzureWebsiteEnv.ps1 -Name <websitename> -SqlDatabasePassword <password>
The Name
parameter specifies the name to be used when creating the database and storage accounts, and the SqlDatabasePassword
parameter specifies the password for the admin account that will be created for SQL Database. There are other parameters you can use that we'll look at later.
After the script finishes you can see in the management portal what was created. You'll find two databases:
A storage account:
And a web app:
On the Configure tab for the web app, you can see that it has the storage account settings and SQL database connection strings set up for the Fix It app.
The Automation folder now also contains a <websitename>.pubxml file. This file stores settings that MSBuild will use to deploy the application to the Azure environment that was just created. For example:
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<WebPublishMethod>MSDeploy</WebPublishMethod>
<SiteUrlToLaunchAfterPublish>http://fixitdemo.azurewebsites.net</SiteUrlToLaunchAfterPublish>
<ExcludeApp_Data>False</ExcludeApp_Data>
<MSDeployServiceURL>waws-prod-bay-003.publish.azurewebsites.windows.net:443</MSDeployServiceURL>
<DeployIisAppPath>fixitdemo</DeployIisAppPath>
<RemoteSitePhysicalPath />
<SkipExtraFilesOnServer>True</SkipExtraFilesOnServer>
<MSDeployPublishMethod>WMSVC</MSDeployPublishMethod>
<EnableMSDeployBackup>True</EnableMSDeployBackup>
<UserName>$fixitdemo</UserName>
<PublishDatabaseSettings></PublishDatabaseSettings>
</PropertyGroup>
</Project>
As you can see, the script has created a complete test environment, and the whole process is done in about 90 seconds.
If someone else on your team wants to create a test environment, they can just run the script. Not only is it fast, but also they can be confident that they are using an environment identical to the one you're using. You couldn't be quite as confident of that if everyone was setting things up manually by using the management portal UI.
A look at the scripts
There are actually three scripts that do this work. You call one from the command line and it automatically uses the other two to do some of the tasks:
New-AzureWebSiteEnv.ps1 is the main script.
- New-AzureStorage.ps1 creates the storage account.
- New-AzureSql.ps1 creates the databases.
Parameters in the main script
The main script, New-AzureWebSiteEnv.ps1, defines several parameters:
[CmdletBinding(PositionalBinding=$True)]
Param(
[Parameter(Mandatory = $true)]
[ValidatePattern("^[a-z0-9]*$")]
[String]$Name,
[String]$Location = "West US",
[String]$SqlDatabaseUserName = "dbuser",
[String]$SqlDatabasePassword,
[String]$StartIPAddress,
[String]$EndIPAddress
)
Two parameters are required:
- The name of the web app that the script creates. (This is also used for the URL:
<name>.azurewebsites.net
.) - The password for the new administrative user of the database server that the script creates.
Optional parameters enable you to specify the data center location (defaults to "West US"), database server administrator name (defaults to "dbuser"), and a firewall rule for the database server.
Create the web app
The first thing the script does is create the web app by calling the New-AzureWebsite
cmdlet, passing in to it the web app name and location parameter values:
# Create a new website
$website = New-AzureWebsite -Name $Name -Location $Location -Verbose
Create the storage account
Then the main script runs the New-AzureStorage.ps1 script, specifying "*<websitename>*storage" for the storage account name, and the same data center location as the web app.
$storageAccountName = $Name + "storage"
$storage = $scriptPath\New-AzureStorage.ps1" -Name $storageAccountName -Location $Location
New-AzureStorage.ps1 calls the New-AzureStorageAccount
cmdlet to create the storage account, and it returns the account name and access key values. The application will need these values in order to access the blobs and queues in the storage account.
# Create a new storage account
New-AzureStorageAccount -StorageAccountName $Name -Location $Location -Verbose
# Get the access key of the storage account
$key = Get-AzureStorageKey -StorageAccountName $Name
# Generate the connection string of the storage account
$connectionString = "BlobEndpoint=http://$Name.blob.core.windows.net/;QueueEndpoint=http://$Name.queue.core.windows.net/;TableEndpoint=http://$Name.table.core.windows.net/;AccountName=$Name;AccountKey=$primaryKey"
#Return a hashtable of storage account values
Return @{AccountName = $Name; AccessKey = $key.Primary; ConnectionString = $connectionString}
You might not always want to create a new storage account; you could enhance the script by adding a parameter that optionally directs it to use an existing storage account.
Create the databases
The main script then runs the database creation script, New-AzureSql.ps1, after setting up default database and firewall rule names:
$sqlAppDatabaseName = "appdb"
$sqlMemberDatabaseName = "memberdb"
$sqlDatabaseServerFirewallRuleName = $Name + "rule"
# Create a SQL Azure database server, app and member databases
$sql = $scriptPath\New-AzureSql.ps1 `
-AppDatabaseName $sqlAppDatabaseName `
-MemberDatabaseName $sqlMemberDatabaseName `
-UserName $SqlDatabaseUserName `
-Password $SqlDatabasePassword `
-FirewallRuleName $sqlDatabaseServerFirewallRuleName `
-StartIPAddress $StartIPAddress `
-EndIPAddress $EndIPAddress `
-Location $Location
The database creation script retrieves the dev machine's IP address and sets a firewall rule so the dev machine can connect to and manage the server. The database creation script then goes through several steps to set up the databases:
Creates the server by using the
New-AzureSqlDatabaseServer
cmdlet.$databaseServer = New-AzureSqlDatabaseServer -AdministratorLogin $UserName -AdministratorLoginPassword $Password -Location $Location
Creates firewall rules to enable the dev machine to manage the server and to enable the web app to connect to it.
# Create a SQL Azure database server firewall rule for the IP address of the machine in which this script will run # This will also whitelist all the Azure IP so that the website can access the database server New-AzureSqlDatabaseServerFirewallRule -ServerName $databaseServerName -RuleName $FirewallRuleName -StartIpAddress $StartIPAddress -EndIpAddress $EndIPAddress -Verbose New-AzureSqlDatabaseServerFirewallRule -ServerName $databaseServer.ServerName -AllowAllAzureServices -RuleName "AllowAllAzureIP" -Verbose
Creates a database context that includes the server name and credentials, by using the
New-AzureSqlDatabaseServerContext
cmdlet.# Create a database context which includes the server name and credential # These are all local operations. No API call to Azure $credential = New-PSCredentialFromPlainText -UserName $UserName -Password $Password $context = New-AzureSqlDatabaseServerContext -ServerName $databaseServer.ServerName -Credential $credential
New-PSCredentialFromPlainText
is a function in the script that calls theConvertTo-SecureString
cmdlet to encrypt the password and returns aPSCredential
object, the same type that theGet-Credential
cmdlet returns.Creates the application database and the membership database by using the
New-AzureSqlDatabase
cmdlet.# Use the database context to create app database New-AzureSqlDatabase -DatabaseName $AppDatabaseName -Context $context -Verbose # Use the database context to create member database New-AzureSqlDatabase -DatabaseName $MemberDatabaseName -Context $context -Verbose
Calls a locally defined function to create a connection string for each database. The application will use these connection strings to access the databases.
$appDatabaseConnectionString = Get-SQLAzureDatabaseConnectionString -DatabaseServerName $databaseServerName -DatabaseName $AppDatabaseName -UserName $UserName -Password $Password $memberDatabaseConnectionString = Get-SQLAzureDatabaseConnectionString -DatabaseServerName $databaseServerName -DatabaseName $MemberDatabaseName -UserName $UserName -Password $Password
Get-SQLAzureDatabaseConnectionString is a function defined in the script that creates the connection string from the parameter values supplied to it.
Function Get-SQLAzureDatabaseConnectionString { Param( [String]$DatabaseServerName, [String]$DatabaseName, [String]$UserName, [String]$Password ) Return "Server=tcp:$DatabaseServerName.database.windows.net,1433;Database=$DatabaseName;User ID=$UserName@$DatabaseServerName;Password=$Password;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;" }
Returns a hash table with the database server name and the connection strings.
Return @{ ` Server = $databaseServer.ServerName; UserName = $UserName; Password = $Password; ` AppDatabase = @{Name = $AppDatabaseName; ConnectionString = $appDatabaseConnectionString}; ` MemberDatabase = @{Name = $MemberDatabaseName; ConnectionString = $memberDatabaseConnectionString} ` }
The Fix It app uses separate membership and application databases. It's also possible to put both membership and application data in a single database.
Store app settings and connection strings
Azure has a feature that enables you to store settings and connection strings that automatically override what is returned to the application when it tries to read the appSettings
or connectionStrings
collections in the Web.config file. This is an alternative to applying Web.config transformations when you deploy. For more information, see Store sensitive data in Azure later in this e-book.
The environment creation script stores in Azure all of the appSettings
and connectionStrings
values that the application needs to access the storage account and databases when it runs in Azure.
# Configure app settings for storage account and New Relic
$appSettings = @{ `
"StorageAccountName" = $storageAccountName; `
"StorageAccountAccessKey" = $storage.AccessKey; `
"COR_ENABLE_PROFILING" = "1"; `
"COR_PROFILER" = "{71DA0A04-7777-4EC6-9643-7D28B46A8A41}"; `
"COR_PROFILER_PATH" = "C:\Home\site\wwwroot\newrelic\NewRelic.Profiler.dll"; `
"NEWRELIC_HOME" = "C:\Home\site\wwwroot\newrelic" `
}
# Configure connection strings for appdb and ASP.NET member db
$connectionStrings = ( `
@{Name = $sqlAppDatabaseName; Type = "SQLAzure"; ConnectionString = $sql.AppDatabase.ConnectionString}, `
@{Name = "DefaultConnection"; Type = "SQLAzure"; ConnectionString = $sql.MemberDatabase.ConnectionString}
)
# Add the connection string and storage account name/key to the website
Set-AzureWebsite -Name $Name -AppSettings $appSettings -ConnectionStrings $connectionStrings
New Relic is a telemetry framework that we demonstrate in the Monitoring and Telemetry chapter. The environment creation script also restarts the web app to make sure that it picks up the New Relic settings.
# Restart the website to let New Relic hook kick in
Restart-AzureWebsite -Name $websiteName
Preparing for deployment
At the end of the process, the environment creation script calls two functions to create files that will be used by the deployment script.
One of these functions creates a publish profile (<websitename>.pubxml file). The code calls the Azure REST API to get the publish settings, and it saves the information in a .publishsettings file. Then it uses the information from that file along with a template file (pubxml.template) to create the .pubxml file that contains the publish profile. This two-step process simulates what you do in Visual Studio: download a .publishsettings file and import that to create a publish profile.
The other function uses another template file (website-environment.template) to create a website-environment.xml file that contains settings the deployment script will use along with the .pubxml file.
Troubleshooting and error handling
Scripts are like programs: they can fail, and when they do you want to know as much as you can about the failure and what caused it. For this reason, the environment creation script changes the value of the VerbosePreference
variable from SilentlyContinue
to Continue
so that all verbose messages are displayed. It also changes the value of the ErrorActionPreference
variable from Continue
to Stop
, so that the script stops even when it encounters non-terminating errors:
# Set the output level to verbose and make the script stop on error
$VerbosePreference = "Continue"
$ErrorActionPreference = "Stop"
Before it does any work, the script stores the start time so that it can calculate the elapsed time when it's done:
# Mark the start time of the script execution
$startTime = Get-Date
After it completes its work, the script displays the elapsed time:
# Mark the finish time of the script execution
$finishTime = Get-Date
# Output the time consumed in seconds
Write-Output ("Total time used (seconds): {0}" -f ($finishTime - $startTime).TotalSeconds)
And for every key operation the script writes verbose messages, for example:
Write-Verbose "[Start] creating $websiteName website in $Location location"
$website = New-AzureWebsite -Name $websiteName -Location $Location -Verbose
Write-Verbose "[Finish] creating $websiteName website in $Location location"
Deployment script
What the New-AzureWebsiteEnv.ps1 script does for environment creation, the Publish-AzureWebsite.ps1 script does for application deployment.
The deployment script gets the name of the web app from the website-environment.xml file created by the environment creation script.
[Xml]$envXml = Get-Content "$scriptPath\website-environment.xml"
$websiteName = $envXml.environment.name
It gets the deployment user password from the .publishsettings file:
[Xml]$xml = Get-Content $scriptPath\$websiteName.publishsettings
$password = $xml.publishData.publishProfile.userPWD[0]
$publishXmlFile = Join-Path $scriptPath -ChildPath ($websiteName + ".pubxml")
It executes the MSBuild command that builds and deploys the project:
& "$env:windir\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" $ProjectFile `
/p:VisualStudioVersion=12.0 `
/p:DeployOnBuild=true `
/p:PublishProfile=$publishXmlFile `
/p:Password=$password
And if you've specified the Launch
parameter on the command line, it calls the Show-AzureWebsite
cmdlet to open your default browser to the website URL.
If ($Launch)
{
Show-AzureWebsite -Name $websiteName
}
You can run the deployment script with a command like this one:
.\Publish-AzureWebsite.ps1 ..\MyFixIt\MyFixIt.csproj -Launch
And when it's done, the browser opens with the site running in the cloud at the <websitename>.azurewebsites.net
URL.
Summary
With these scripts you can be confident that the same steps will always be executed in the same order using the same options. This helps ensure that each developer on the team doesn't miss something or mess something up or deploy something custom on his own machine that won't actually work the same way in another team member's environment or in production.
In a similar way, you can automate most Azure management functions that you can do in the management portal, by using the REST API, Windows PowerShell scripts, a .NET language API, or a Bash utility that you can run on Linux or Mac.
In the next chapter we'll look at source code and explain why it's important to include your scripts in your source code repository.
Resources
- Install and Configure Windows PowerShell for Azure. Explains how to install the Azure PowerShell cmdlets and how to install the certificate that you need on your computer in order to manage your Azure account. This is a great place to get started because it also has links to resources for learning PowerShell itself.
- Azure Script Center. WindowsAzure.com portal to resources for developing scripts that manage Azure services, with links to getting started tutorials, cmdlet reference documentation and source code, and sample scripts
- Weekend Scripter: Getting Started with Azure and PowerShell. In a blog dedicated to Windows PowerShell, this post provides a great introduction to using PowerShell for Azure management functions.
- Install and Configure the Azure Cross-Platform Command-Line Interface. Getting-started tutorial for an Azure scripting framework that works on Mac and Linux as well as Windows systems.
- Command-line tools section of the Download Azure SDKs and Tools topic. Portal page for documentation and downloads related to command-line tools for Azure.
- Automating everything with the Azure Management Libraries and .NET. Scott Hanselman introduces the .NET management API for Azure.
- Using Windows PowerShell Scripts to Publish to Dev and Test Environments. MSDN documentation that explains how to use publish scripts that Visual Studio automatically generates for web projects.
- PowerShell Tools for Visual Studio 2013. Visual Studio extension that adds language support for Windows PowerShell in Visual Studio.