Edit

Automate Everything (Building Real-World Cloud Apps with Azure)

  • Article

by Rick Anderson, Tom Dykstra

Download Fix It Project or Download E-book

The Building Real World Cloud Apps with Azure e-book is based on a presentation developed by Scott Guthrie. It explains 13 patterns and practices that can help you be successful developing web apps for the cloud. For an introduction to the e-book, see the first chapter.

The first three patterns we'll look at actually apply to any software development project, but especially to cloud projects. This pattern is about automating development tasks. It's an important topic because manual processes are slow and error-prone; automating as many of them as possible helps set up a fast, reliable, and agile workflow. It's uniquely important for cloud development because you can easily automate many tasks that are difficult or impossible to automate in an on-premises environment. For example, you can set up whole test environments including new web server and back-end VMs, databases, blob storage (file storage), queues, etc.

DevOps Workflow

Increasingly you hear the term "DevOps." The term developed out of a recognition that you have to integrate development and operations tasks in order to develop software efficiently. The kind of workflow you want to enable is one in which you can develop an app, deploy it, learn from production usage of it, change it in response to what you've learned, and repeat the cycle quickly and reliably.

Some successful cloud development teams deploy multiple times a day to a live environment. The Azure team used to deploy a major update every 2-3 months, but now it releases minor updates every 2-3 days and major releases every 2-3 weeks. Getting into that cadence really helps you be responsive to customer feedback.

In order to do that, you have to enable a development and deployment cycle that is repeatable, reliable, predictable, and has low cycle time.

DevOps workflow

In other words, the period of time between when you have an idea for a feature and when the customers are using it and providing feedback must be as short as possible. The first three patterns – automate everything, source control, and continuous integration and delivery -- are all about best practices that we recommend in order to enable that kind of process.

Azure management scripts

In the introduction to this e-book, you saw the web-based console, the Azure Management Portal. The management portal enables you to monitor and manage all of the resources that you have deployed on Azure. It's an easy way to create and delete services such as web apps and VMs, configure those services, monitor service operation, and so forth. It's a great tool, but using it is a manual process. If you're going to develop a production application of any size, and especially in a team environment, we recommend that you go through the portal UI in order to learn and explore Azure, and then automate the processes that you'll be doing repetitively.

Nearly everything that you can do manually in the management portal or from Visual Studio can also be done by calling the REST management API. You can write scripts using Windows PowerShell, or you can use an open source framework such as Chef or Puppet. You can also use the Bash command-line tool in a Mac or Linux environment. Azure has scripting APIs for all those different environments, and it has a .NET management API in case you want to write code instead of script.

For the Fix It app we've created some Windows PowerShell scripts that automate the processes of creating a test environment and deploying the project to that environment, and we'll review some of the contents of those scripts.

Environment creation script

The first script we'll look at is named New-AzureWebsiteEnv.ps1. It creates an Azure environment that you can deploy the Fix It app to for testing. The main tasks that this script performs are the following:

  • Create a web app.
  • Create a storage account. (Required for blobs and queues, as you'll see in later chapters.)
  • Create a SQL Database server and two databases: an application database, and a membership database.
  • Store settings in Azure that the app will use to access the storage account and databases.
  • Create settings files that will be used to automate deployment.

Run the script

Note

This part of the chapter shows examples of scripts and the commands that you enter in order to run them. This a demo and doesn't provide everything you need to know in order to run the scripts. For step-by-step how-to-do-it instructions, see Appendix: The Fix It Sample Application.

To run a PowerShell script that manages Azure services you have to install the Azure PowerShell console and configure it to work with your Azure subscription. Once you're set up, you can run the Fix It environment creation script with a command like this one:

.\New-AzureWebsiteEnv.ps1 -Name <websitename> -SqlDatabasePassword <password>

The Name parameter specifies the name to be used when creating the database and storage accounts, and the SqlDatabasePassword parameter specifies the password for the admin account that will be created for SQL Database. There are other parameters you can use that we'll look at later.

PowerShell window

After the script finishes you can see in the management portal what was created. You'll find two databases:

Databases

A storage account:

Storage account

And a web app:

Web site

On the Configure tab for the web app, you can see that it has the storage account settings and SQL database connection strings set up for the Fix It app.

appSettings and connectionStrings

The Automation folder now also contains a <websitename>.pubxml file. This file stores settings that MSBuild will use to deploy the application to the Azure environment that was just created. For example:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <WebPublishMethod>MSDeploy</WebPublishMethod>
    <SiteUrlToLaunchAfterPublish>http://fixitdemo.azurewebsites.net</SiteUrlToLaunchAfterPublish>
    <ExcludeApp_Data>False</ExcludeApp_Data>
    <MSDeployServiceURL>waws-prod-bay-003.publish.azurewebsites.windows.net:443</MSDeployServiceURL>
    <DeployIisAppPath>fixitdemo</DeployIisAppPath>
    <RemoteSitePhysicalPath />
    <SkipExtraFilesOnServer>True</SkipExtraFilesOnServer>
    <MSDeployPublishMethod>WMSVC</MSDeployPublishMethod>
    <EnableMSDeployBackup>True</EnableMSDeployBackup>
    <UserName>$fixitdemo</UserName>
    <PublishDatabaseSettings></PublishDatabaseSettings>
  </PropertyGroup>
</Project>

As you can see, the script has created a complete test environment, and the whole process is done in about 90 seconds.

If someone else on your team wants to create a test environment, they can just run the script. Not only is it fast, but also they can be confident that they are using an environment identical to the one you're using. You couldn't be quite as confident of that if everyone was setting things up manually by using the management portal UI.

A look at the scripts

There are actually three scripts that do this work. You call one from the command line and it automatically uses the other two to do some of the tasks:

  • New-AzureWebSiteEnv.ps1 is the main script.

    • New-AzureStorage.ps1 creates the storage account.
    • New-AzureSql.ps1 creates the databases.

Parameters in the main script

The main script, New-AzureWebSiteEnv.ps1, defines several parameters:

[CmdletBinding(PositionalBinding=$True)]
Param(
    [Parameter(Mandatory = $true)]
    [ValidatePattern("^[a-z0-9]*$")]
    [String]$Name,                             
    [String]$Location = "West US",             
    [String]$SqlDatabaseUserName = "dbuser",   
    [String]$SqlDatabasePassword,              
    [String]$StartIPAddress,                   
    [String]$EndIPAddress                      
    )

Two parameters are required:

  • The name of the web app that the script creates. (This is also used for the URL: <name>.azurewebsites.net.)
  • The password for the new administrative user of the database server that the script creates.

Optional parameters enable you to specify the data center location (defaults to "West US"), database server administrator name (defaults to "dbuser"), and a firewall rule for the database server.

Create the web app

The first thing the script does is create the web app by calling the New-AzureWebsite cmdlet, passing in to it the web app name and location parameter values:

# Create a new website
$website = New-AzureWebsite -Name $Name -Location $Location -Verbose

Create the storage account

Then the main script runs the New-AzureStorage.ps1 script, specifying "*<websitename>*storage" for the storage account name, and the same data center location as the web app.

$storageAccountName = $Name + "storage"
 
$storage = $scriptPath\New-AzureStorage.ps1" -Name $storageAccountName -Location $Location

New-AzureStorage.ps1 calls the New-AzureStorageAccount cmdlet to create the storage account, and it returns the account name and access key values. The application will need these values in order to access the blobs and queues in the storage account.

# Create a new storage account
New-AzureStorageAccount -StorageAccountName $Name -Location $Location -Verbose
 
# Get the access key of the storage account
$key = Get-AzureStorageKey -StorageAccountName $Name
 
# Generate the connection string of the storage account
$connectionString = "BlobEndpoint=http://$Name.blob.core.windows.net/;QueueEndpoint=http://$Name.queue.core.windows.net/;TableEndpoint=http://$Name.table.core.windows.net/;AccountName=$Name;AccountKey=$primaryKey"
 
#Return a hashtable of storage account values
Return @{AccountName = $Name; AccessKey = $key.Primary; ConnectionString = $connectionString}

You might not always want to create a new storage account; you could enhance the script by adding a parameter that optionally directs it to use an existing storage account.

Create the databases

The main script then runs the database creation script, New-AzureSql.ps1, after setting up default database and firewall rule names:

$sqlAppDatabaseName = "appdb"
$sqlMemberDatabaseName = "memberdb"
$sqlDatabaseServerFirewallRuleName = $Name + "rule"

# Create a SQL Azure database server, app and member databases
$sql = $scriptPath\New-AzureSql.ps1 `
    -AppDatabaseName $sqlAppDatabaseName `
    -MemberDatabaseName $sqlMemberDatabaseName `
    -UserName $SqlDatabaseUserName `
    -Password $SqlDatabasePassword `
    -FirewallRuleName $sqlDatabaseServerFirewallRuleName `
    -StartIPAddress $StartIPAddress `
    -EndIPAddress $EndIPAddress `
    -Location $Location

The database creation script retrieves the dev machine's IP address and sets a firewall rule so the dev machine can connect to and manage the server. The database creation script then goes through several steps to set up the databases:

  • Creates the server by using the New-AzureSqlDatabaseServer cmdlet.

    $databaseServer = New-AzureSqlDatabaseServer -AdministratorLogin $UserName -AdministratorLoginPassword $Password -Location $Location

  • Creates firewall rules to enable the dev machine to manage the server and to enable the web app to connect to it.

    # Create a SQL Azure database server firewall rule for the IP address of the machine in which this script will run
# This will also whitelist all the Azure IP so that the website can access the database server
New-AzureSqlDatabaseServerFirewallRule -ServerName $databaseServerName -RuleName $FirewallRuleName -StartIpAddress $StartIPAddress 
-EndIpAddress $EndIPAddress -Verbose
New-AzureSqlDatabaseServerFirewallRule -ServerName $databaseServer.ServerName -AllowAllAzureServices 
-RuleName "AllowAllAzureIP" -Verbose

  • Creates a database context that includes the server name and credentials, by using the New-AzureSqlDatabaseServerContext cmdlet.

    # Create a database context which includes the server name and credential
# These are all local operations. No API call to Azure
$credential = New-PSCredentialFromPlainText -UserName $UserName -Password $Password
$context = New-AzureSqlDatabaseServerContext -ServerName $databaseServer.ServerName -Credential $credential

    New-PSCredentialFromPlainText is a function in the script that calls the ConvertTo-SecureString cmdlet to encrypt the password and returns a PSCredential object, the same type that the Get-Credential cmdlet returns.

  • Creates the application database and the membership database by using the New-AzureSqlDatabase cmdlet.

    # Use the database context to create app database
New-AzureSqlDatabase -DatabaseName $AppDatabaseName -Context $context -Verbose
 
# Use the database context to create member database
New-AzureSqlDatabase -DatabaseName $MemberDatabaseName -Context $context -Verbose

  • Calls a locally defined function to create a connection string for each database. The application will use these connection strings to access the databases.

    $appDatabaseConnectionString = Get-SQLAzureDatabaseConnectionString -DatabaseServerName $databaseServerName -DatabaseName $AppDatabaseName -UserName $UserName -Password $Password
$memberDatabaseConnectionString = Get-SQLAzureDatabaseConnectionString -DatabaseServerName $databaseServerName -DatabaseName $MemberDatabaseName -UserName $UserName -Password $Password

    Get-SQLAzureDatabaseConnectionString is a function defined in the script that creates the connection string from the parameter values supplied to it.

    Function Get-SQLAzureDatabaseConnectionString
{
    Param(
        [String]$DatabaseServerName,
        [String]$DatabaseName,
        [String]$UserName,
        [String]$Password
    )

    Return "Server=tcp:$DatabaseServerName.database.windows.net,1433;Database=$DatabaseName;User ID=$UserName@$DatabaseServerName;Password=$Password;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;"
}

  • Returns a hash table with the database server name and the connection strings.

    Return @{ `
    Server = $databaseServer.ServerName; UserName = $UserName; Password = $Password; `
    AppDatabase = @{Name = $AppDatabaseName; ConnectionString = $appDatabaseConnectionString}; `
    MemberDatabase = @{Name = $MemberDatabaseName; ConnectionString = $memberDatabaseConnectionString} `
}

The Fix It app uses separate membership and application databases. It's also possible to put both membership and application data in a single database.

Store app settings and connection strings

Azure has a feature that enables you to store settings and connection strings that automatically override what is returned to the application when it tries to read the appSettings or connectionStrings collections in the Web.config file. This is an alternative to applying Web.config transformations when you deploy. For more information, see Store sensitive data in Azure later in this e-book.

The environment creation script stores in Azure all of the appSettings and connectionStrings values that the application needs to access the storage account and databases when it runs in Azure.

# Configure app settings for storage account and New Relic
$appSettings = @{ `
    "StorageAccountName" = $storageAccountName; `
    "StorageAccountAccessKey" = $storage.AccessKey; `
    "COR_ENABLE_PROFILING" = "1"; `
    "COR_PROFILER" = "{71DA0A04-7777-4EC6-9643-7D28B46A8A41}"; `
    "COR_PROFILER_PATH" = "C:\Home\site\wwwroot\newrelic\NewRelic.Profiler.dll"; `
    "NEWRELIC_HOME" = "C:\Home\site\wwwroot\newrelic" `
}

# Configure connection strings for appdb and ASP.NET member db
$connectionStrings = ( `
    @{Name = $sqlAppDatabaseName; Type = "SQLAzure"; ConnectionString = $sql.AppDatabase.ConnectionString}, `
    @{Name = "DefaultConnection"; Type = "SQLAzure"; ConnectionString = $sql.MemberDatabase.ConnectionString}
)

# Add the connection string and storage account name/key to the website
Set-AzureWebsite -Name $Name -AppSettings $appSettings -ConnectionStrings $connectionStrings

New Relic is a telemetry framework that we demonstrate in the Monitoring and Telemetry chapter. The environment creation script also restarts the web app to make sure that it picks up the New Relic settings.

# Restart the website to let New Relic hook kick in
Restart-AzureWebsite -Name $websiteName

Preparing for deployment

At the end of the process, the environment creation script calls two functions to create files that will be used by the deployment script.

One of these functions creates a publish profile (<websitename>.pubxml file). The code calls the Azure REST API to get the publish settings, and it saves the information in a .publishsettings file. Then it uses the information from that file along with a template file (pubxml.template) to create the .pubxml file that contains the publish profile. This two-step process simulates what you do in Visual Studio: download a .publishsettings file and import that to create a publish profile.

The other function uses another template file (website-environment.template) to create a website-environment.xml file that contains settings the deployment script will use along with the .pubxml file.

Troubleshooting and error handling

Scripts are like programs: they can fail, and when they do you want to know as much as you can about the failure and what caused it. For this reason, the environment creation script changes the value of the VerbosePreference variable from SilentlyContinue to Continue so that all verbose messages are displayed. It also changes the value of the ErrorActionPreference variable from Continue to Stop, so that the script stops even when it encounters non-terminating errors:

# Set the output level to verbose and make the script stop on error
$VerbosePreference = "Continue"
$ErrorActionPreference = "Stop"

Before it does any work, the script stores the start time so that it can calculate the elapsed time when it's done:

# Mark the start time of the script execution
$startTime = Get-Date

After it completes its work, the script displays the elapsed time:

# Mark the finish time of the script execution
$finishTime = Get-Date
# Output the time consumed in seconds
Write-Output ("Total time used (seconds): {0}" -f ($finishTime - $startTime).TotalSeconds)

And for every key operation the script writes verbose messages, for example:

Write-Verbose "[Start] creating $websiteName website in $Location location"
$website = New-AzureWebsite -Name $websiteName -Location $Location -Verbose
Write-Verbose "[Finish] creating $websiteName website in $Location location"

Deployment script

What the New-AzureWebsiteEnv.ps1 script does for environment creation, the Publish-AzureWebsite.ps1 script does for application deployment.

The deployment script gets the name of the web app from the website-environment.xml file created by the environment creation script.

[Xml]$envXml = Get-Content "$scriptPath\website-environment.xml"
$websiteName = $envXml.environment.name

It gets the deployment user password from the .publishsettings file:

[Xml]$xml = Get-Content $scriptPath\$websiteName.publishsettings 
$password = $xml.publishData.publishProfile.userPWD[0]
$publishXmlFile = Join-Path $scriptPath -ChildPath ($websiteName + ".pubxml")

It executes the MSBuild command that builds and deploys the project:

& "$env:windir\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" $ProjectFile `
    /p:VisualStudioVersion=12.0 `
    /p:DeployOnBuild=true `
    /p:PublishProfile=$publishXmlFile `
    /p:Password=$password

And if you've specified the Launch parameter on the command line, it calls the Show-AzureWebsite cmdlet to open your default browser to the website URL.

If ($Launch)
{
    Show-AzureWebsite -Name $websiteName
}

You can run the deployment script with a command like this one:

.\Publish-AzureWebsite.ps1 ..\MyFixIt\MyFixIt.csproj -Launch

And when it's done, the browser opens with the site running in the cloud at the <websitename>.azurewebsites.net URL.

Fix It app deployed to Windows Azure

Summary

With these scripts you can be confident that the same steps will always be executed in the same order using the same options. This helps ensure that each developer on the team doesn't miss something or mess something up or deploy something custom on his own machine that won't actually work the same way in another team member's environment or in production.

In a similar way, you can automate most Azure management functions that you can do in the management portal, by using the REST API, Windows PowerShell scripts, a .NET language API, or a Bash utility that you can run on Linux or Mac.

In the next chapter we'll look at source code and explain why it's important to include your scripts in your source code repository.

Resources

Previous Next