Here is a summary of the steps to follow for analyzing your storage logs:

1. Install the Windows Azure HDInsight Log Analysis Toolkit.
2. Turn on logging for your storage account.
3. Run a Hadoop job. (I’ll run the 10GB GraySort sample in my example below.)
4. Download your storage logs using the Log Analysis Tool (LAT).
5. Analyze your storage logs using the LAT.

Read on for details about each step…

Install the Log Analysis Tool (LAT)

Download and run the .msi installer here: Windows Azure HDInsight Log Analysis Toolkit. The .msi will install the toolkit in the C:\Program Files (x86)\Windows Azure HDInsight Log Analysis Toolkit\ directory. The lat.exe will not be added to your Path environment variable by default.

Turn on Storage Account Logging

You can get detailed information about successful and failed requests to your storage account by turning on storage account logging. To do this, follow the instructions here. (For general information about storage analytics, see About Storage Analytics Logging.)

Run a job

For example purposes, I’ll run the 10GB GraySort sample. The jars for this sample are already in place when you provision an HDInsight cluster. The instructions for running this sample are here: The 10GB GraySort Sample. The PowerShell code I used to submit the job is below. A few things to note:

• As one of the arguments for the “teragen” job, I have “1000000000” instead of “100000000” as is used in the documentation. This is to increase the amount of data generated, sorted, and validated. This is a more realistic example.
• I submitted this job to a 16-node HDInsight cluster. Clusters with fewer nodes are not likely to run into throttling by Windows Azure Storage as they simply cannot produce enough throughput to go beyond the storage throttling limits.
• I am manually setting the cluster’s self-throttling factors to “1.0”, which essentially turns self-throttling off. My intention is to produce some level of throttling by Azure Storage for the purposes of this blog post.

$clusterName = "bswandemo"

#-----------TERAGEN------------------

$teragenJob = New-AzureHDInsightMapReduceJobDefinition `

    -JobName "teragenJob" `

    -JarFile "wasb:///example/jars/hadoop-examples.jar" `

    -ClassName "teragen" `

    –Arguments "-Dmapred.map.tasks=50", "1000000000", "wasb:///example/data/10GB-sort-input" `

    -Defines @{"fs.azure.selfthrottling.write.factor" = "1.0"}

$teragenJob | Start-AzureHDInsightJob -Cluster $clusterName `

            | Wait-AzureHDInsightJob

#----------TERASORT-------------------

$terasortJob = New-AzureHDInsightMapReduceJobDefinition `

    -JobName "terasortjob" `

    -JarFile “/example/jars/hadoop-examples.jar” `

    -ClassName "terasort" `

    –Arguments "-Dmapred.map.tasks=50", “-Dmapred.reduce.tasks=25”, “wasb:///example/data/10GB-sort-input”, “wasb:///example/data/10GB-sort-output”

$terasortJob | Start-AzureHDInsightJob -Cluster $clusterName `

             | Wait-AzureHDInsightJob 

#----------TERAVALIDATE----------------

$teravalidateJob = New-AzureHDInsightMapReduceJobDefinition `

    -JarFile “/example/jars/hadoop-examples.jar” `

    -ClassName "teravalidate" `

    –Arguments "-Dmapred.map.tasks=50", “-Dmapred.reduce.tasks=25”, “wasb:///example/data/10GB-sort-output”, “wasb:///example/data/10GB-sort-validate” `

    -Defines @{"fs.azure.selfthrottling.read.factor" = "1.0"}

$teravalidateJob | Start-AzureHDInsightJob -Cluster $clusterName `

                 | Wait-AzureHDInsightJob 

Download Storage Logs

Use the Log Analysis Tool (LAT) that you downloaded as part of the .NET SDK for Hadoop to download the storage logs that generated when you ran the job above. (Note that you will have to wait up to 10 minutes for the logs to be generated before you can download them.) To use the download utility, you need 5 pieces of information:

• -start: The start time of the job.
• -end: The end time of the job.
• -logcache: The directory to which the logs will be downloaded.
• -account: Your storage account name (e.g. bswanstorage).
• -key: Your storage account key.

With that information, you can run a command similar to this one (the -force option is only necessary if the log cache directory doesn't exist):

lat.exe download -start "2014-01-02T22:49:00Z" -end "2014-01-02T23:23:00Z" -logcache "C:\logcache" -account "Your account name" -key "Your account key" -force

Here’s the output:

Navigating to my c:\logcache directory, I see the downloaded files:

Analyze Storage Logs

The next step is to analyze the storage logs. To do this, you again need 5 pieces of information (the 6th is optional):

• -logcache: The folder you specified when downloading your storage logs.
• -account: The name of your storage account.
• -start: The start time of the job.
• -end: The end time of the job.
• -name: The name of the job. This name will be used as part of the name of the file to which the storage analyses are written.
• -note: (Optional) This is a note about the job being analyzed.

With this information, you can execute a command similar to this:

lat.exe throttlinganalysis -logcache "C:\logcache" -account "Your storage account name" -start "2014-01-02T22:49:00Z" -end "2014-01-02T23:23:00Z" -name "teragen3" -note "throttling factor set to 1, 16 node cluster"

Here’s the output from the command line:

The analysis is written to two .csv files: jobname.summary.csv and jobname.details.csv (teragen3.summary.csv and teragen3.details.csv in my case). The summary file will contain information similar to this:

We can see from the summary that some throttling did occur in my case. To get a better idea of when the throttling occurred, we need to look at the details file, which contains information about what happened at each 1-second interval of the job:

What’s nice about the details file is that, because it’s a .csv file, you can use Excel features to create a picture of throughput for your job. To do this, select columns A through G, select INSERT from the main menu, and choose Scatter from the Charts options, and finally, select the Straight Lines option:

You should see a graph similar to this one:

As you can see, Azure Storage throttling occurred during the first job (Teragen) because of a high write load.  In this particular example, the peak write loads were around 5Gbps which happens to be the throughput quota for uploads to a geo-replicated storage account. So, this example was straying into the red-zone a little.  One solution to prevent the Teragen job from issuing writes to Azure Storage above the quota rate is to adjust fs.azure.selftthrottling.write to produce some self-throttling of storage requests by the Hadoop tasks.  Other solutions include increasing storage bandwidth quota, reducing cluster size or reducing the cluster usage for the Teragen job.

Please note that the Log Analysis Tool is still in the “experimental” phase. We’d love to hear your feedback and/or have you contribute to the project.

Thanks.

-Brian

1. Azure setup logs, Hadoop installation logs, and Hadoop Service logs are written to Azure Table Storage, and
2. Task logs are written to Azure Blob Storage for jobs submitted through Templeton.

Read on to find out more.

Logs Written to Windows Azure Table Storage

When you create an HDInsight cluster, three tables are automatically created in Azure Table Storage:

• setuplog: Log of events/exceptions encountered in provisioning/set up of HDInsight VMs.
• hadoopinstalllog: Log of events/exceptions encountered when installing Hadoop on the cluster. This table may be useful in debugging issues related to clusters created with custom parameters.
• hadoopservicelog: Log of events/exceptions recorded by all Hadoop services. This table may be useful in debugging issues related to job failures on HDInsight clusters.

Note: To make these table names unique, the names are in this format: uclusternameDDmonYYYYatHHMMSSssshadoopservicelog.

There are many tools available for accessing data in these tables (e.g. Visual Studio, Azure Storage Explorer, Power Query for Excel). Here, I’ll use Power Query first, then Visual Studio, to explore the hadoopservicelog table. In both cases, I’ll assume you have created an HDInsight cluster.

Using Power Query

I’ll assume you have Power Query installed:

1. Connect to Azure Table Storage from Excel.

2. Right click the hadoopservicelog table in the Navigator pane and select Edit Query.

3. Optional: Delete the Partition Key, Row Key, and Timestamp columns by selecting them, then clicking Remove Columns from the options in the ribbon.

4. Click on the Content column to choose the columns you want to include. For my example, I chose TraceLevel, Role, and ComponentName:

5. Select the TraceLevel, Role, and ComponentName columns, and then click Group By control in the ribbon.

6. Click OK in the Group By dialog box:

7. Finally, click Apply & Close:

You can now use Excel to filter and sort as necessary. Obviously, you may want to include other columns (e.g. Message) in order to drill down into issues when they occur, but selecting and grouping the columns described above provides a decent picture of what is happening with Hadoop services. The same idea can be applied to the setuplog and hadoopinstalllog tables.

Using Visual Studio

1. In Visual Studio, select Server Explorer from the View menu:

2. Expand Windows Azure, right-click Storage, and select Attach External Storage…

3. Provide your storage account name and key.

4. Expand your storage account, expand Tables, and double-click the hadoopservicelog table.

5. Add a filter to exclude rows in which TraceLevel = ‘INFO’ (more info about constructing filters here: Constructing Filter Strings for the Table Designer):

If you are a developer, using Visual Studio provides you the extra level of detail I omitted when using Excel, and you don’t need an “extra” tool.

Logs Written to Windows Azure Blob Storage

The logs written to Windows Azure Tables (as described above) provide one level of insight into what is happening with an HDInsight cluster. However, these tables do not provide task-level logs, which can be helpful in drilling further into issues when they occur. To provide this next level of detail, HDInsight clusters are configured to write task logs to your Blob Storage account for any job that is submitted through Templeton. (Practically, this means jobs submitted using the Windows Azure PowerShell cmdlets or the .NET Job Submission APIs, not jobs submitted through RDP/command-line access to the cluster.) I’ll illustrate how to access these logs by submitting a job and then using CloudXPlorer to view the logs.

Submit a job

I’ll run the WordCount example that is installed with an HDInsight cluster. You can submit this job using either the PowerShell cmdlets or the .NET Job Submission APIs. Note that in each example I set the StatusFolder to “wordcountstatus”. This will be the folder that contains the task logs. Also note that if you use the PowerShell cmdlets, task logs are collected by default. With the .NET job submission APIs, you must set EnableTaskLogs = true to collect task logs.

Powershell cmdlets

$clusterName = "bswandemo"

$wordCountJobDef = New-AzureHDInsightMapReduceJobDefinition `

    -JarFile "wasb:///example/jars/hadoop-examples.jar" `

    -ClassName "wordcount" `

    -Arguments "wasb:///example/data/gutenberg/davinci.txt", `

               "wasb:///wordcountoutput" 

$wordCountJob = Start-AzureHDInsightJob `

    -Cluster $clusterName `

    -JobDefinition $wordCountJobDef

Wait-AzureHDInsightJob `

    -Job $wordCountJob `

    -WaitTimeoutInSeconds 3600

Get-AzureHDInsightJobOutput `

    -Cluster $clusterName `

    -JobId $wordCountJob.JobId `

    -StandardError

.NET Job Submission APIs

string clusterName = "bswandemo2";

var store = new X509Store();

store.Open(OpenFlags.ReadOnly);

var cert = store.Certificates.Cast<X509Certificate2>().First(item => item.Thumbprint == "{Thumbprint of certificate}");

var jobSubmissionCreds = new JobSubmissionCertificateCredential(new Guid("{Your Subscription ID}"), cert, clusterName);

var jobClient = JobSubmissionClientFactory.Connect(jobSubmissionCreds);

var MRJob = new MapReduceJobCreateParameters()

{

    ClassName = "wordcount",

    JobName = "WordCount",

    JarFile = "wasb:///example/jars/hadoop-examples.jar",

    StatusFolder = "wordcountstatus",

    EnableTaskLogs = true

};

MRJob.Arguments.Add("wasb:///example/data/gutenberg/davinci.txt");

MRJob.Arguments.Add("wasb:///wordcountoutput");

var jobResults = jobClient.CreateMapReduceJob(MRJob);

Note: If you don’t specify a StatusFolder, one will be created for you using a GUID for the folder name.

View logs

After your job completes, you can view the logs using any of the many available tools for viewing Azure Blob Storage. Here, I’m using CloudXplorer. Note that a new folder has been created (templeton-hadoop) for containing job output information. The files in each of the job_ folders contain high-level job completion information (percentComplete, exitValue, userArgs, etc.). However, under the admin folder (under this folder since this was the user who submitted the job), you will find the wordcountstatus (specified at when I submitted the job) that contains task attempt information for each task attempt of the job.

Each of the attempt_ folders contains stderr, stdout, and syslog files. You can drill into these to get a better understanding of if/when something when wrong:

Also note that the job.xml.html file contains all the Hadoop settings that were used when running the job you submitted:

Disabling Logging

Hopefully, you will find the information logged in your Azure storage account to be useful in debugging Hadoop jobs. I do want to point out, however, that because these logs are written to your storage account, you are being billed for storing them. You should create a retention/clean-up policy that is consistent with your development needs to avoid unnecessary charges.

Currently, collection of task logs (the logs written to Blob Storage) is off by default when using the .NET job submission APIs (December update, version 1.0.4.0). To enable it, set EnableTaskLogs = true when defining your job (see the example above). In previous versions, collection of task logs in on by default.

The ability to turn off logging will be available via the PowerShell cmdlets soon.

As always, we’re interested in your feedback on the usefulness of these logs.

Thanks.

-Brian

Note: In HDInsight, HDFS is intended to be used as a cache or for intermediary storage. When a cluster is deleted, data in HDFS will be discarded. Data intended for long-term storage should be stored in Windows Azure Storage (BLOBS).

Overview

If you run a heavy I/O workload on an HDInsight cluster of sufficient size*, reads and/or writes may be throttled by Windows Azure Storage. Throttling can result in jobs running slowly, tasks failing, and (in rare cases) jobs failing. Throttling occurs when the aggregate load that a cluster puts on a storage account exceeds the allotted bandwidth for the storage account. To address this, HDInsight clusters have a tunable self-throttling mechanism that can slow read and/or write traffic to a storage account. The self-throttling mechanism exposes two parameters: fs.azure.selfthrottling.read.factor and fs.azure.selftthrottling.write.factor. These parameters govern the rate of read and write traffic from an HDInsight cluster to a storage account. Values for these parameters are set at job submission time. Values must be in the range (0, 1], where 1 corresponds to no self-throttling, 0.5 corresponds to roughly 1/2 the unrestricted throughput rate, and so on. Conservative default values for these parameters are set based on cluster size at cluster creation time ("conservative" here means that values are such that throttling is highly unlikely to occur at all, but bandwidth utilization may be below the allocated amount). To arrive at optimal values for the self-throttling parameters, you should turn on storage account logging prior to running a job, analyze the logs to understand if/when throttling occurred, and adjust the parameter values accordingly.

Note: We are currently working on ways for a cluster to self-tune its throughput rate to avoid throttling and maximize bandwidth utilization.

* The number of nodes required to trigger throttling by Windows Azure Storage depends on whether geo-replication is enabled for the storage account (because bandwidth allocation is different for each case). If geo-replication is enabled, clusters with more than 7 nodes may encounter throttling. If geo-replication is not enabled, clusters with more than 10 nodes may encounter throttling.

What is throttling?

Limits are placed on the bandwidth allocated to Windows Azure Storage accounts to guarantee high availability for all customers. Limiting bandwidth is done by rejecting requests to a storage account (HTTP response 500 or 503) when the request rate exceeds the allocated bandwidth. Windows Azure Storage imposes the following bandwidth limits on a single storage account:

• Bandwidth for a Geo Redundant storage account (geo-replication on)
• Ingress - up to 5 gigabits per second
• Egress - up to 10 gigabits per second
• Bandwidth for a Locally Redundant storage account (geo-replication off)
• Ingress - up to 10 gigabits per second
• Egress - up to 15 gigabits per second

Note that these limits are subject to change. For more information, see Windows Azure’s Flat Network Storage and 2012 Scalability Targets. For information about enabling or disabling geo-replication for a storage account, see How to manage storage accounts.

When will my cluster be throttled?

An HDInsight cluster will be throttled if/when its throughput rates to Windows Azure Storage exceed those stated above. Throughput, in turn, is dependent on the nature of the job being run. Perhaps the best way to understand in advance if a job will encounter throttling is by comparing it to a well-known workload, the Terasort benchmark. With the fs.azure.selfthrottling.read.factor and fs.azure.selftthrottling.write.factor  parameters each set to 1 (i.e. no self-throttling), HDInsight clusters generally encounter throttling during the Teragen and Teravalidate phases of the Terasort workload* under the following conditions:

• Geo-replication for the storage account is on and the cluster has more than 15 nodes, or
• Geo-replication for the storage account is off and the cluster has more than 31 nodes.

These numbers are for reference only. A cluster will only encounter throttling if the job that it is running produces throughput in excess of that allocated for the storage account.

* Run with 4 map slots and 2 reduce slots.

How do I know my cluster is being throttled?

Initial indications that a cluster workload is being throttled by Windows Azure Storage may include the following:

• Longer-than-expected job completion times
• A high number of task failures
• Job failures (in rare cases). If this occurs, task-attempt error messages will be of the form “java.io.IOException … caused by com.microsoft.windowsazure.services.core.storage.StorageException: The server encountered an unknown failure: The server is busy.”

While the above are indications that your cluster is being throttled, the best way to understand if your workload is being throttled is by inspecting responses returned by Windows Azure Storage. Responses with response code (http status code) of 500 or 503 indicate that a request has been throttled. One way to collect WA Storage responses is to turn on storage logging (https://www.windowsazure.com/en-us/manage/services/storage/how-to-monitor-a-storage-account/#configurelogging).

How can throttling be avoided?

If you have a workload that encounters throttling, there are three ways avoid it:

1. Reduce your cluster size
2. Adjust the settings that control the cluster’s self-throttling mechanism
3. Request an increase in bandwidth allocated for your storage account.

The sections below go into more detail.

Reduce your cluster size

The first question to answer in avoiding throttling by Windows Azure Storage is this: Do I need all the CPUs in my cluster? In many cases, the answer here might be yes (e.g. the Terasort benchmark), in which case you can skip this section. However, some workloads that are truly I/O dominant may not require the CPUs available in a large cluster. By reducing the number of nodes in your cluster, you can reduce the load on storage and (potentially) avoid throttling (in addition to saving money!).

Adjust settings that control self-throttling

The fs.azure.selfthrottling.read.factor and fs.azure.selftthrottling.write.factor settings control the rate at which an HDInsight cluster reads and writes to Windows Azure Storage. Values for these settings must be in the range (0, 1], where 1 corresponds to no self-throttling, 0.5 corresponds to roughly 1/2 the unrestricted throughput rate, and so on. Default values for these settings are determined at cluster creation time according to the following formulas (n = number of nodes in cluster):

fs.azure.selfthrottling.read/write.factor = 1, n <= 7

fs.azure.selfthrottling.read/write.factor = 32/(5n), n > 7

The formula for n > 7 is conservative, based on the “worst-case” scenario (for a storage account with geo-replication enabled) in which the throughput capacity for each node in the cluster is maximized. In practice, this is rare. You can override the default values for these settings at job submission time. Depending on your workload, you may find that increasing the value for either or both of these settings when you submit a job improves job performance. However, increasing the default value by too much may result in throttling by Windows Azure Storage.

How do high latencies affect the self-throttling mechanism?

One of the assumptions built into the self-throttling mechanism is that end-to-end request latencies are low (in the 500ms to 1000ms range). If this assumption does not apply, bandwidth utilization may be low and/or jobs may take longer-than-expected to complete. In this case, increasing the values for fs.azure.selfthrottling.read.factor and fs.azure.selfthrottling.write.factor (within the range of (0, 1]) may improve performance.

That’s it for today. I’d be interested in feedback on this feature, so please use the comments below. And, as I mentioned earlier, we are currently working on ways for a cluster to self-tune its throughput rate to avoid throttling and maximize bandwidth utilization without the need for any manual intervention.

Thanks.

-Brian

If you have PHP-related questions (the topic of the vast majority of the blog to date), feel free to ask them. I’ll do the best I can to answer them, but as I mentioned earlier, my recent focus has been HDInsight for the past several months.

Thanks.

-Brian 

Overview

Drupal is an open source content management system that runs on PHP. Windows Azure offers a flexible platform for hosting, managing, and scaling Drupal deployments. This paper focuses on an approach to host Drupal sites on Windows Azure, based on learning from a BPD Customer Programs Design Win engagement with the Screen Actors Guild Awards Drupal website. This paper covers guidelines and best practices for managing an existing Drupal web site in Windows Azure. For more information on how to migrate Drupal applications to Windows Azure, see Azure Real World: Migrating a Drupal Site from LAMP to Windows Azure.

The target audience for this paper is Drupal administrators who have some exposure to Windows Azure. More detailed pointers to Windows Azure content is provided throughout the paper as links.

Drupal Application Architecture on Windows Azure

Before reviewing the management and monitoring guidelines, it is important to understand the architecture of a typical Drupal deployment on Windows Azure. First, the following diagram displays the basic architecture of Drupal running on Windows and IIS7.

In the Windows Server scenario, you could have one or more machines hosting the web site in a farm. Those machines would either persist the site content to the file system or point to other network shares.

For Windows Azure, the basic architecture is the same, but there are some differences. In Windows Azure the site is hosted on a web role. A web role instance is hosted on a Windows Server 2008 virtual machine within the Windows Azure datacenter. Like the web farm, you can have multiple instances running the site. But there is no persistence guarantee for the data on the file system. Because of this, much of the shared site content should be stored in Windows Azure Blob storage. This allows them to be highly available and durable. Usually, a large portion of the site caters to static content which lends well to caching. And caching can be applied in a set of places – browser level caching, CDN to cache content in the edge closer to the browser clients, caching in Azure to reduce the load on backend, etc. Finally, the database can be located in SQL Azure. The following diagram shows these differences.

For monitoring and management, we will look at Drupal on Windows Azure from three perspectives:

• Availability: Ensure the web site does not go down and that all tiers are setup correctly. Apply best practices to ensure that the site is deployed across data centers and perform backup operations regularly.

• Scalability: Correctly handle changes in user load. Understand the performance characteristics of the site.

• Manageability: Correctly handle updates. Make code and site changes with no downtime when possible.

Although some management tasks span one or more of these categories, it is still helpful to discuss Drupal management on Windows Azure within these focus areas.

Availability

One main goal is that the Drupal site remains running and accessible to all end-users. This involves monitoring both the site and the SQL Azure database that the site depends on. In this section, we will briefly look at monitoring and backup tasks. Other crossover areas that affect availability will be discussed in the next section on scalability.

Monitoring

With any application, monitoring plays an important role with managing availability. Monitoring data can reveal whether users are successfully using the site or whether computing resources are meeting the demand. Other data reveals error counts and possibly points to issues in a specific tier of the deployment.

There are several monitoring tools that can be used.

• The Windows Azure Management Portal.

• Windows Azure diagnostic data.

• Custom monitoring scripts.

• System Center Operations Manager.

• Third party tools such as Azure Diagnostics Manager and Azure Storage Explorer.

The Windows Azure Management Portal can be used to ensure that your deployments are successful and running. You can also use the portal to manage features such as Remote Desktop so that you can directly connect to machines that are running the Drupal site.

Windows Azure diagnostics allows you to collect performance counters and logs off of the web role instances that are running the Drupal site. Although there are many options for configuring diagnostics in Azure, the best solution with Drupal is to use a diagnostics configuration file. The following configuration file demonstrates some basic performance counters that can monitor resources such as memory, processor utilization, and network bandwidth.

<DiagnosticMonitorConfiguration xmlns="https://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration" 

                                configurationChangePollInterval="PT1M" 

                                overallQuotaInMB="4096">

  <DiagnosticInfrastructureLogs bufferQuotaInMB="10"

                                scheduledTransferLogLevelFilter="Error"

                                scheduledTransferPeriod="PT1M" />

  <Logs bufferQuotaInMB="0"

        scheduledTransferLogLevelFilter="Verbose"

        scheduledTransferPeriod="PT1M" />

  <Directories bufferQuotaInMB="0" scheduledTransferPeriod="PT5M">

    <!-- These three elements specify the special directories that are set up for the log types -->

    <CrashDumps container="wad-crash-dumps" directoryQuotaInMB="256" />

    <FailedRequestLogs container="wad-frq" directoryQuotaInMB="256" />

    <IISLogs container="wad-iis" directoryQuotaInMB="256" />

  </Directories>

  <PerformanceCounters bufferQuotaInMB="0" scheduledTransferPeriod="PT1M">

    <!-- The counter specifier is in the same format as the imperative diagnostics configuration API -->

    <PerformanceCounterConfiguration counterSpecifier="\Memory\Available MBytes" sampleRate="PT10M" />

    <PerformanceCounterConfiguration counterSpecifier="\Network Interface(*)\Bytes Total/sec" sampleRate="PT10M" />

    <PerformanceCounterConfiguration counterSpecifier="\Processor(*)\% Processor Time" sampleRate="PT10M" />

    <PerformanceCounterConfiguration counterSpecifier="\System\Processor Queue Length" sampleRate="PT10M" />

    <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Current Connections" sampleRate="PT10M" />

    <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Bytes Total/sec" sampleRate="PT10M" />

    <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Connection Attempts/sec" sampleRate="PT10M" />

    <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Get Requests/sec" sampleRate="PT10M" />

    <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Files Sent/sec" sampleRate="PT10M" />

  </PerformanceCounters>

  <WindowsEventLog bufferQuotaInMB="0"

                   scheduledTransferLogLevelFilter="Verbose"

                   scheduledTransferPeriod="PT5M">

    <!-- The event log name is in the same format as the imperative diagnostics configuration API -->

    <DataSource name="System!*" />

  </WindowsEventLog>

</DiagnosticMonitorConfiguration>

For more information about setting up diagnostic configuration files, see How to Use the Windows Azure Diagnostics Configuration File. This information is stored locally on each role instance and then transferred to Windows Azure storage per a defined schedule or on-demand. See Getting Started with Storing and Viewing Diagnostic Data in Windows Azure Storage. Various monitoring tools, such as Azure Diagnostics Manager, help you to more easily analyze diagnostic data.

Monitoring the performance of the machines hosting the Drupal site is only part of the story. In order to plan properly for both availability and scalability, you should also monitor site traffic, including user load patterns and trends. Standard and custom diagnostic data could contribute to this, but there are also third-party tools that monitor web traffic. For example, if you know that spikes occur in your application during certain days of the week, you could make changes to the application to handle the additional load and increase the availability of the Drupal solution.

Backup Tasks

To remain highly available, it is important to backup your data as a defense-in-depth strategy for disaster recovery. This is true even though SQL Azure and Windows Azure Storage both implement redundancy to prevent data loss. One obvious reason is that these services cannot prevent administrator error if data is accidentally deleted or incorrectly changed.

SQL Azure does not currently have a formal backup technology, although there are many third-party tools and solutions that provide this capability. Usually the database size for a Drupal site is relatively small. In the case of SAG Awards, it was only ~100-150 MB. So performing an entire backup using any strategy was relatively fast. If your database is much larger, you might have to test various backup strategies to find the one that works best.

Apart from third-party SQL Azure backup solutions, there are several strategies for obtaining a backup of your data:

· Use the Drush tool and the portabledb-export command.

· Periodically copy the database using the CREATE DATABASE Transact-SQL command.

· Use Data-tier applications (DAC) to assist with backup and restore of the database.

SQL Azure backup and data security techniques are described in more detail in the topic, Business Continuity in SQL Azure.

Note that bandwidth costs accrue with any backup operation that transfers information outside of the Windows Azure datacenter. To reduce costs, you can copy the database to a database within the same datacenter. Or you can export the data-tier applications to blob storage in the same datacenter.

Another potential backup task involves the files in Blob storage. If you keep a master copy of all media files uploaded to Blob storage, then you already have an on-premises backup of those files. However, if multiple administrators are loading files into Blob storage for use on the Drupal site, it is a good idea to enumerate the storage account and to download any new files to a central location. The following PHP script demonstrates how this can be done by backing up all files in Blob storage after a specified modification date.

<?php

/**

* Backup the blob storage to a local folder 

* 

* PHP version 5.3

* 

* @category Blob Backup

* @package Windows Azure

* @author Bibin Kurian

* @copyright 2011 Microsoft Corp. (https://www.microsoft.com)

* @license New BSD license, (https://www.opensource.org/licenses/bsd-license.php)

* @version SVN: 1.0

* @link

* 

*/

/** Microsoft_WindowsAzure_Storage_Blob */

require_once 'Microsoft/WindowsAzure/Storage/Blob.php';

//Windows Azure BLOB storage account name

define("AZURE_STORAGE_ACCOUNT", "YOUR_ACCOUNTNAME");

//Windows Azure BLOB storage container name

define("AZURE_STORAGE_CONTAINER", "YOUR_STORAGECONTAINERNAME");

//Windows Azure BLOB storage secret key

define(

    "AZURE_STORAGE_KEY", 

    "YOUR_SECRETKEY"

);

//backup folder

define("STORAGE_BACKUP_DIR", "YOUR_LOCALDRIVE");

//backup from date

define("DEFAULT_BACKUP_FROM_DATE", strtotime("Mon, 19 Dec 2011 22:00:00 GMT"));

//backup to date

define("DEFAULT_BACKUP_TO_DATE", time());

//directory seperator

define("DS", "\\");

//start backup logic

//current datetime to create the backup folder

$now = date("F j, Y, g.i A");

$fullBackupDirPath = STORAGE_BACKUP_DIR . DS . $now . DS. AZURE_STORAGE_CONTAINER;

mkdir($fullBackupDirPath, 0755, true);

//For the directory creations, pointing the current directory to user specified directory

chdir($fullBackupDirPath);

//BLOB object

$blobObj = new Microsoft_WindowsAzure_Storage_Blob(

    'blob.core.windows.net', 

    AZURE_STORAGE_ACCOUNT,

    AZURE_STORAGE_KEY

);

//$blobObj->setProxy(true, 'YOUR_PROXY_IF_NEEDED', 80);

$blobs = (array)$blobObj->listBlobs(AZURE_STORAGE_CONTAINER, '', '', 35000);

backupBlobs($blobs, $blobObj);

function backupBlobs($blobs, $blobObj) {

    foreach ($blobs as $blob) {

        if (strtotime($blob->lastmodified) >= DEFAULT_BACKUP_FROM_DATE && strtotime($blob->lastmodified) <= DEFAULT_BACKUP_TO_DATE) {

            $path = pathinfo($blob->name);

            if ($path['basename'] != '$$$.$$$') {

                $dir = $path['dirname'];

                $oldDir = getcwd();

                if (handleDirectory($dir)) {

                    chdir($dir);

                    $blobObj->getBlob(

                        AZURE_STORAGE_CONTAINER,

                        $blob->name,

                        $path['basename']

                    );

                    chdir($oldDir);

                }

            }

        }

    }

}

function handleDirectory($dir) {

    if (!checkDirExists($dir)) {

        return mkdir($dir, 0755, true);

    }

    return true;

}

function checkDirExists($dir) {

    if(file_exists($dir) && is_dir($dir)) {

        return true;

    }

    return false;

}

?>

This script has a dependency on the Windows Azure SDK for PHP. Also note there are several parameters that you must modify such as the storage account, secret, and backup location. As with SQL Azure, bandwidth and transaction charges apply to a backup script like this.

Scalability

Drupal sites on Windows Azure can scale as load increased through typical strategies of scale-up, scale-out, and caching. The following sections describe the specifics of how these strategies are implemented in Windows Azure.

Typically you make scalability decisions based on monitoring and capacity planning. Monitoring can be done in staging during testing or in production with real-time load. Capacity planning factors in projections for changes in user demand.

Scale Up

When you configure your web role prior to deployment, you have the option of specifying the Virtual Machine (VM) size, such as Small or ExtraLarge. Each size tier adds additional memory, processing power, and network bandwidth to each instance of your web role. For cost efficiency and smaller units of scale, you can test your application under expected load to find the smallest virtual machine size that meets your requirements.

The workload usually in most popular Drupal websites can be separated out into a limited set of Drupal admins making content changes and a large user base who perform mostly read-only workload. End users can be allowed to make ‘writes’, such as uploading blogs or posting in forums, but those changes are not ‘content changes’. Drupal admins are setup to operate without caching so that the writes are made directly to SQL Azure or the corresponding backend database. This workload performs well with Large or ExtraLarge VM sizes. Also, note that the VM size is closely tied to all hardware resources, so if there are many content-rich pages that are streaming content, then the VM size requirements are higher.

To make changes to the Virtual Machine size setting, you must change the vmsize attribute of the WebRole element in the service definition file, ServiceDefinition.csdef. A virtual machine size change requires existing applications to be redeployed.

Scale Out

In addition to the size of each web role instance, you can increase or decrease the number of instances that are running the Drupal site. This spreads the web requests across more servers, enabling the site to handle more users. To change the number of running instances of your web role, see How to Scale Applications by Increasing or Decreasing the Number of Role Instances.

Note that some configuration changes can cause your existing web role instances to recycle. You can choose to handle this situation by applying the configuration change and continue running. This is done by handling the RoleEnvironment.Changing event. For more information see, How to Use the RoleEnvironment.Changing Event.

A common question for any Windows Azure solution is whether there is some type of built-in automatic scaling. Windows Azure does not provide a service that provides auto-scaling. However, it is possible to create a custom solution that scales Azure services using the Service Management API. For an example of this approach, see An Auto-Scaling Module for PHP Applications in Windows Azure.

Caching

Caching is an important strategy for scaling Drupal applications on Windows Azure. One reason for this is that SQL Azure implements throttling mechanisms to regulate the load on any one database in the cloud. Code that uses SQL Azure should have robust error handling and retry logic to account for this. For more information, see Error Messages (SQL Azure Database). Because of the potential for load-related throttling as well as for general performance improvement, it is strongly recommended to use caching.

Although Windows Azure provides a Caching service, this service does not currently have interoperability with PHP. Because of this, the best solution for caching in Drupal is to use a module that uses an open-source caching technology, such as Memcached.

Outside of a specific Drupal module, you can also configure Memcached to work in PHP for Windows Azure. For more information, see Running Memcached on Windows Azure for PHP. Here is also an example of how to get Memcached working in Windows Azure using a plugin: Windows Azure Memcached plugin.

In a future paper, we hope to cover this architecture in more detail. For now, here are several design and management considerations related to caching.

For ease of use and cost savings, collocation of the cache across the web role instances of the Drupal site works best. However, this assumes that there is available reserve memory on each instance to apply toward caching. It is possible to increase the virtual machine size setting to increase the amount of available memory on each machine. It is also possible to add additional web role instances to add to the overall memory of the cache while at the same time improving the ability of the web site to respond to load. It is possible to create a dedicated cache cluster in the cloud, but the steps for this are beyond the scope of this paper[RR1] .

For Windows Azure Blob storage, there is also a caching feature built into the service called the Content Delivery Network (CDN). CDN provides high-bandwidth access to files in Blob storage by caching copies of the files in edge nodes around the world. Even within a single geographic region, you could see performance improvements as there are many more edge nodes than Windows Azure datacenters. For more information, see Delivering High-Bandwidth Content with the Windows Azure CDN.

Manageability

It is important to note that each hosted service has a Staging environment and a Production environment. This can be used to manage deployments, because you can load and test and application in staging before performing a VIP swap with production.

From a manageability standpoint, Drupal has an advantage on Windows Azure in the way that site content is stored. Because the data necessary to serve pages is stored in the database and blob storage, there is no need to redeploy the application to change the content of the site.

Another best practice is to use a separate storage account for diagnostic data than the one that is used for the application itself. This can improve performance and also helps to separate the cost of diagnostic monitoring from the cost of the running application.

As mentioned previously, there are several tools that can assist with managing Windows Azure applications. The following table summarizes a few of these choices.

Additional Drupal Resources

The following list contains additional resources for managing your Drupal deployment on Windows Azure:

• Install Drupal for Windows
• Azure Real World: Migrating a Drupal Site from LAMP to Windows Azure
• Drupal on Windows Azure
• Getting Started with Drupal 7 Administration
• Drupal Statistics
• Windows Azure Traffic Manager (WATM)
• System Center Operations Manager MP for Azure applications
• System Center Operations Manager MP for SQL Azure

Share this on Twitter

I saw a Tweet on Monday that nicely summed up the start to my week:The only thing worse than a Monday is a Monday after #DrupalCon. I’m not sure I’ve quoted it exactly, but I’m sure I’ve captured the sentiment. My Monday wasn’t bad as in 3-day-hangover bad, it was just bad in that it was a let down after the fun and intensity of DrupalCon Denver. I met lots of great people, had many interesting conversations, and learned a ton. Mix in a few parties, and it was an excellent week.

The highlights for me were Alessandro Pilotti’s pre-conference training: Deploying Drupal at Scale on the Microsoft Platform. Alessandro did a live demo of using Application Request Routing and Web Farm Framwork to very quickly create a Drupal web farm that could be easily scaled out. That he was able to do this in less than an hour (the demo, not the entire training) was impressive (as was the 480 GB SSD, 16 GB RAM laptop he used to simultaneously run 4 Windows VMs!). You can read a summary of the demo and get the detailed training slides here.

Alessandro followed Monday’s training with a Lightning talk on Tuesday about using PhoneGap to develop mobile applications for any platform, including Windows Phone. After hearing Luke Wroblewski’s keynote address on Thursday (he talked about the growth of mobile, the importance of developing for mobile, and the opportunities mobile creates…nice summary here), I’m keen to dig a bit deeper into PhoneGap.

Training aside, the highlight of the conference for me was meeting people and learning from the conversations I had with them. I still got a lot of “So, what is Microsoft doing at DrupalCon?” questions that I got 2 years ago at DrupalCon SF, and it was nice to be able to say what we’ve been doing. We’ve been working with community members to improve PHP performance on Windows, improve IIS functionality and tooling around PHP, improve the Drupal installation experience on Windows, develop Drupal modules that leverage the Microsoft platform, improve Drush on Windows, and develop free training courses (like Alessandro’s). And, this time around, some people even had specific Microsoft-related questions…

The question I was asked most often was something along the lines of “When will you support Drupal/Sharepoint integration?” The short answer was, “We do now!” Sharepoint 2010 provides several APIs – the tricky part is in determining which one to use. Since most people I talked to wanted some way to surface Sharepoint data (from internal-facing Sharepoint deployment) on an public facing Drupal deployment, I’m guessing that the Sharepoint 2010 Web Services API is going to be the easiest path (I’m assuming familiarity with REST). And, another option is the Content Manageability Interoperability Services (CMIS) connector.

That’s it for me. Hopefully, you got a chance to stop by our booth and chat a bit. If you did (and even if you didn’t), I’d be interested in hearing your thoughts in the comments below.

Thanks.

-Brian

Share this on Twitter

As you can see from the training agenda, there is no way I can re-cap the day’s work in a single blog post, so I won’t try. I will, however, do my best to summarize his final demo of the day: creating a Drupal Web Farm using the Microsoft Web Farm Framework. What stood out at the end of the demo was the ease and speed with which he had a Drupal Web Farm up and running (almost completely from scratch!). Alessandro’s slides, which are attached to this post, go into much more detail than I will, so check them out if you want more than an interview.

Note: I will capture the high-level steps that Alessandro went through in creating a Drupal Web Farm. I’ve selected several slides from his presentation to to this – all of which show how to use IIS GUI tooling. However, Alessandro made a point of showing how everything he did could also be done from the command line using appcmd.exe and PowerShell commands.

Alessandro did a live demo of the following scenario: He had 4 Windows Server VM’s running on his laptop (480 GB SSD with 16 MB RAM !), which would serve as load balancer (Application Request Routing + Web Farm Framwork), two web servers (each hosting Drupal), and database server/file server.

Note: For the purposes of the demo, he used one VM for both the database server and file server, but he pointed out that having each on a separate machine (as shown in the diagram) would be the way to go for a production site.

Next, he used the Web Platform Installer to install Drupal  and configure IIS on the primary server.

Again using the Web Platform Installer, he then installed Application Request Routing and Web Farm Framework.

With ARR and WFF installed, he could create a Web Farm…

…and add a primary host to the farm. (He also added the 2nd web server as a secondary host using the same UI.)

Finally, he provisioned the second server (and could have just as easily done more) using WFF. (Note that WFF uses Web Deploy. Complete details are in the attached slide deck.)

And, Voila! A scalable Drupal Web Farm. Of course, it wasn’t quite as simple as Voila!, but he did manage to go from four “blank” Windows Server VMs to a running web farm in about 45 minutes…very impressive IMHO.

Be sure to check out the attached slides from Alessandro’s presentation.

Thanks.

-Brian

Share this on Twitter

DrupalCon2012_DrupalOnIIS.zip

Note: Rama Ramani, Senior Program Manager on the Customer Advisory Team at Microsoft,  was co-author of this post.

Background

The Screen Actors Guild (SAG) is the United States’ largest union representing working actors. In January of every year since 1995, SAG has hosted the Screen Actors Guild Awards (SAG Awards) to honor performers in motion pictures and TV series. In 2011, the SAG Awards Drupal website, deployed on a LAMP stack, was impacted by site outages and slow performance during peak-usage days, with SAG having to consistently upgrade their hardware to meet demand for those days. That upgraded hardware was then not optimally used during the rest of the year. In late 2011, SAG Awards engineers began working with Microsoft engineers to migrate its website to Windows Azure in anticipation of its 2012 show. In January of 2012, the SAG Website had over 350K unique visitors and 1.1M page views, with traffic spiking to over 160K visitors during the show.

Overview and Challenges

In many ways, the SAG Awards website was a perfect candidate for Windows Azure. The website has moderate traffic throughout most of the year, but has a sustained traffic spike shortly before, during, and after the awards show in January. The elastic scalability and fast storage services offered by the Azure platform were designed to handle this type of usage.

The main challenge that SAG Awards and Microsoft engineers faced in moving the SAG Awards website to Windows Azure was in architecting for a very high, sustained traffic spike while accommodating the need of SAG Awards administrators to frequently update media files during the awards show. Both intelligent use of Windows Azure Blob Storage and a custom module for invalidating cached pages when content was updated were key to delivering a positive user experience.

Note: In this post I will focus on how the Drupal website was moved to Windows Azure, as well as how content and data were moved to Windows Azure Blob Storage and SQL Azure. I won’t cover the details of the caching strategy.

The process for moving the SAG-Awards website from a LAMP environment to the Windows Azure platform can be broken down into five high-level steps:

1. Export data. A custom Drush command (portabledb-export) was used to create a database dump of MySQL data. A .zip archive of media files was created for later use.
2. Install Drupal on Windows. The Drupal files that comprised the installation in the LAMP environment were copied to Windows Server/IIS as an initial step in discovering compatibility issues.
3. Import data to SQL Azure. A custom Drush command (portabledb-import) was used together with the database dump created in step 1 to import data to SQL Azure.
4. Copy media files to Azure Blob Storage. After unpacking the .zip archive in step 1, CloudXplorer was used to copy these files to Windows Azure Blob Storage.
5. Package and deploy Drupal. The Azure packaging tool cspack was used to package Drupal for deployment. Deployment was done through the Windows Azure Portal.

Note: The portabledb commands mentioned above are authored and maintained by Damien Tournoud.

Details for each of these high-level steps are in the sections below.

Export Data

Microsoft and SAG engineers began investigating the best way to export MySQL data by looking at Damien Tournoud’s portabledb Drush commands. They found that this tool worked perfectly when moving Drupal to Windows and SQL Server, but they needed to make some modifications to the tool for exporting data to SQL Azure. (These modifications have since been incorporated into the portabledb commands, which are now available as part of the Windows Azure Integration Module.)

The names of media files stored in the file_managed table were of the form public://field/image/file_name.avi. In order for these files to be streamed from Windows Azure Blob Storage (as they would be by the Windows Azure Integration module when deployed in Azure), the file names needed to be modified to this form: azurepublic://field/image/file_name.avi. This was an easy change to make.

Because the SAG Awards website would be retrieving all data from the cloud, Windows Azure Storage connection information needed to be stored in the database. The portabledb tool was modified to create a new table, azure_storage, for containing this information.

Finally, to allow all media files to be retrieved from Blob Storage, the file_default_scheme table needed to be updated with the stream wrapper name: azurepublic.

Using the modified portabledb tool, the following command produced the database dump:

drush portabledb-export  --use-windows-azure-storage=true --windows-azure-stream-wrapper-name=azurepublic --windows-azure-storage-account-name=azure_storage_account_name --windows-azure-storage-account-key=azure_storage_account_key  --windows-azure-blob-container-name=azure_blob_container_name --windows-azure-module-path=sites/all/modules --ctools-module-path=sites/all/modules  > drupal.dump

Note that the portabledb-export command does not copy media files themselves. Instead, the local media files were compressed in a .zip archive for use in a later step.

Install Drupal on Windows

In order to use the portabledb-import command (the counter part to the portabledb-export command above), a Drupal installation needed to be set up on Windows (with Drush for Windows installed). This was necessary, in part, because connectivity to SQL Azure was to be managed by the Commerce Guys’ SQL Server/SQL Azure module for Drupal, which relies on the SQL Server Drivers for PHP, a Windows-only PHP extension. Having a Windows installation of Drupal would also make it possible to package the application for deployment to Windows Azure. For this reason, Microsoft and SAG Awards engineers copied the Drupal files from the LAMP environment to a Windows Server machine. The team incrementally moved the rest of the application to an IIS/SQL Server Express stack before moving the backend to SQL Azure.

Note: The Windows Server machine was actually a virtual machine running in a Windows Azure Web Role in the same data center as SQL Azure. The Web Role was configured to allow RDP connections, which the team used to install and configure the SAG website installation. This was done to avoid timeouts that occurred when attempting to upload data from an on-premises machine to SQL Azure.

There were, however, some customizations made to the Drupal installation before running the portabledb-import command. Specifically,

• The SQL Server/SQL Azure module for Drupal was installed and enabled.
• The memcache module for Drupal was installed and enabled.
• The Windows Azure Integration module for Drupal was installed and enabled. Note that this module has a dependency on the CTools module. It also requires Damien Tournoud’s branch of the Windows Azure SDK for PHP, which must be unpacked and put into a folder called phpazure in the module’s main directory. (As of this writing, Damien Tournoud’s changes have not been merged with the Windows Azure SDK for PHP. However, they may be merged in the future.)
• Database connection information in the settings.php file was modified to connect to SQL Azure.
• A custom caching module was installed and enabled.

Some customizations to PHP were also necessary since this PHP installation would be packaged with the application itself:

• The php_pdo_sqlsrv.dll extension was installed and enabled. This extension provided connectivity to SQL Azure.
• The php_memcache.dll extension was installed an enabled. This would be used for caching purposes.
• The php_azure.dll extension was installed and enabled. This extension allowed configuration information to be retrieved from the Windows Azure service configuration file after the application was deployed. This allowed changes to be made without having to re-package and re-deploy the entire application. For example, database connection information could be retrieved in the settings.php file like this:

$databases['default']['default']['driver'] = 'sqlsrv';

$databases['default']['default']['username'] = azure_getconfig('sql_azure_username');

$databases['default']['default']['password'] = azure_getconfig('sql_azure_password');

$databases['default']['default']['host'] = azure_getconfig('sql_azure_host');

$databases['default']['default']['database'] = azure_getconfig('sql_azure_database');

With Drupal running on Windows, and with the customizations to Drupal and PHP outlined above, the importing of data could begin.

Import Data to SQL Azure

There were two phases to importing the SAG Awards website data: importing database data to SQL Azure and copying media files to Windows Azure Blob Storage. As alluded to above, importing data to SQL Azure was done with the portabledb-import Drush command. With SQL Azure connection information specified in Drupal’s settings.php file, the following command copied data from the drupal.dump file (which was copied to Drupal’s root directory on the Windows installation) to SQL Azure:

drush portabledb-import --delete-local-files=false --copy-files-blob-storage=false --use-production-storage=true mysite.dump

Note: The copy-files-blob-storage flag was set to false in the command above. While the portabledb-import command can copy media files to Blob Storage, Microsoft and SAG engineers had some work to do in modifying media file names (discussed in the next section). For this reason, they chose not to use this tool for uploading files to Blob Storage.

The next step was to create stored procedures on SQL Azure that are designed to handle some SQL that is specific to MySQL. The SQL Server/SQL Azure module for Drupal normally creates these stored procedures when the module is enabled, but since Drupal would be deployed with the module already enabled, these stored procedures needed to be created manually. Engineers executed the stored procedure creation DDL that is defined in the module by accessing SQL Azure through the management portal.

After the import was complete, the Windows installation of the SAG Awards website was now retrieving all database data from SQL Azure. However, recall that the portabledb-export command modified the names of media files in the file_managed table so that the Drupal Azure module would retrieve media files from Blob Storage. The final phase in importing data was to copy media files to Blob Storage.

Note: After this phase was complete, engineers cleared the cache through the Drupal admin panel.

Copy Media Files to Blob Storage

The main challenge in copying media files to Windows Azure Blob Storage was in handling Linux file name conventions that are not supported on Windows. While Linux supports a colon (:) as part of a file name, Windows does not. Consequently, when the .zip archive of media files was unpacked on Windows, file names were automatically changed: all colons were converted to underscores (_). However, colons are supported in Blob Storage as part of blob names. This meant that files could be uploaded to Blob Storage from Windows with underscores in the blob names, but the blob names would have to be modified manually to match the names stored in SQL Azure.

Engineers used WinRAR to unpack the .zip archive of media files. WinRAR provided a record of all file names that were changed in the unpacking process. Engineers then used CloudXplorer to upload the media files to Blob Storage and to change the modified files names, replacing underscores with colons.

At this point in the migration process, the SAG Awards website was fully functional on Windows and was retrieving all data (database data and media files) from the cloud.

Package and Deploy Drupal

There were two main challenges in packaging the SAG Awards website for deployment to Drupal: packaging a custom installation of PHP and creating the necessary startup tasks.

Because customizations were made to the PHP installation powering Drupal, engineers needed to package their custom PHP installation for deployment to Windows Azure. The other option was to rely on Microsoft’s Web Platform Installer to install a “vanilla” installation of PHP and then write scripts to modify it on start up. Since it is relatively easy to package a custom PHP installation for deployment to Azure, engineers chose to go that route. (For more information, see Packaging a Custom PHP Installation for Windows Azure.)

The startup tasks that needed to be performed were the following:

• Configure IIS to use the custom PHP installation.
• Register the Service Runtime COM Wrapper (which played a role in the caching strategy).
• Put Drush (which was packaged with the deployment) in the PATH environment variable.

The final project structure, prior to packaging, was the following:

\SAGAwards
    \WebRole
    \bin
        \php
        install-php.cmd
        register-service-runtime-COM-Wrapper.cmd
        WinRMConfig.cmd
        startup-tasks-errorlog.txt
        startup-tasks-log.txt
    \resources
        \drush
        \ServiceRuntimeCOMWrapper
        \WebPICmdLine
        (Drupal files and folders)

Finally, the Windows Azure Memcached Plugin was added to the Windows Azure SDK prior to packaging so that memcache would run on startup and restart if killed.

The SAG Awards website was then packaged using cspack and deployed to Windows Azure through the developer portal.

Summary

The work in moving the SAG Awards Drupal website to the Windows Azure platform was an excellent example of Microsoft’s commitment to supporting popular OSS applications on the Windows Azure platform. The collaboration between engineers from SAG and from Microsoft’s Interoperability and Customer Advisory Teams resulted in a win for SAG (the SAG Awards website was able to handle sustained spikes in traffic that it could not handle previously) and in valuable lessons learned for the Windows Azure team about supporting migration and scalability of OSS applications on the Azure Platform.

-Brian

Share this on Twitter

I am, of course, very eager to hear what folks at the conference have to say, but going in, I feel good about our level of commitment. At DrupalCon 2010 in S.F., we announced the beta release of our PDO driver for SQL Server and our engagement with Commerce Guys to build integrated support for SQL Server into Drupal 7. Since then, we have continued to engage the Drupal community by continuing to sponsor and be engaged at subsequent DrupalCons and other Drupal events worldwide, sponsoring and providing guidance for various integrations, tools, training, and much more. I’m sure I’m not capturing everything in this list, but here are some of the results:   

• Drush 5 for Windows (still in beta).
• Open Data Government Initiative (OGDI) integration with Drupal.
• Screen Actors Guild Awards website deployment to Windows Azure. This was no small feat – look for more detail in a post in the coming weeks.
• DrupalWinCache. We’d like feedback on this module.
• Deploying Drupal at Scale on Microsoft Platform training. This was first run at BadCAMP and will be run again next week at DrupalCon Denver (details below).
• Windows Azure Integration module, by the Commerce Guys.
• A scaffold for deploying Drupal to Windows Azure.
• We’ve pretty much killed IE6. Yeah!

Note: I need to point out that the results called out above were, in part, due to the ongoing work of many people at Microsoft over the last couple of years.

What do you think? I’d love to hear your thoughts at the conference or in comments below.

OK, so now let’s talk about DrupalCon Denver. What are we doing there? At a high-level, I hope we are continuing to collaborate with people and support efforts to broaden opportunities for Drupal deployments. Specifically, here are the Microsoft-related events and people to watch for:

• Monday, March 19, 9:00-6:00 (pre-conference training): Deploying Drupal at Scale on the Microsoft Platform. This training is for anyone “who wants to explore the IIS ecosystem and gain knowledge on optimizing Drupal and PHP on Windows…” This training will be run by Alessandro Pilotti, CEO of Cloudbase Solutions Srl, and will be full of useful technical information. (I’ll be there!)
• Tuesday, March 20, 3:45-4:45: Lightning Talk: Mobile Essentials for Drupal. Alessandro Pilotti will present Windows Phone: a Platform for Drupal Mobile Apps. During this session, Alessandro will demonstrate how to get the best out of Drupal and the next generation of Windows Phone devices, including some practical Javascript debugging tips.
• Throughout the conference:
• Microsoft booth. Stop by, say hi, and ask questions. We’ll have some goodies as long as they last.
• Giveaways. We’ll be giving away three LG Quantum Windows Phones. One of these will be given away on Tuesday at Alessandro Pilotti’s lightning talk (details above), and the other two will be given away on Wednesday and Thursday via Twitter (look for the #windowsphone, #drupal, and #drupalcon hash tags).
• The folks from Microsoft who will be at the conference (and who will be happy to answer any questions) are Grace Francisco (@gracefr), Jerry Nixon (@jerrynixon), and myself (@brian_swan). You can read Jerry’s pre-conference thoughts here: Drupal and Microsoft? Yes!

Looking forward to it!

-Brian

Share this on Twitter

Easier Setup of Your Development Environment

If you have used SQL Server Express in the past, you may not have been overjoyed at the time it took to download, install, and configure your database server. With the introduction of SQL Server Express LocalDB, you can have this done in less than 5 minutes. Here’s how:

1. Download SQL Server Express LocalDB. Choose either the ENU\x64\SqlLocalDB.MSI or the ENU\x86\SqlLocalDB.MSI depending on your computer architecture.

2. Download the command line Tools. Find “Microsoft® SQL Server® 2012 Command Line Utilities” and select the download for your computer architecture.

With those two steps complete, you can connect to LocalDB and begin creating databases with the sqlcmd utility:

There is also a SqlLocalDB utility that lets you create and manage an instance of SQL Server Express LocalDB.

3. Download the Microsoft Drivers for SQL Server for PHP. Add the driver to your PHP extension directory and enable it in your php.ini file, and you can connect to it like this:

$serverName = '(localdb)\v11.0';

$connOptions = array('AttachDBFileName'=>'C:\Users\bswan\ExampleDB.mdf','Database'=>'ExampleDB');

$conn = sqlsrv_connect($serverName, $connOptions);

if($conn === false)      

    die(print_r(sqlsrv_errors(), true)); 

else

    echo "Connected via sqlsrv!<br />";

That’s it. From zero to writing SQL Server database queries in PHP much faster than before.

Of course, that’s not entirely it. There can be some gotchas, and there may be a couple more steps, depending on what you want to do. Here are a few things I ran into when getting set up:

• If you are installing LocalDB and the command line tools on a machine that has a previous version of SQL Server, you will need to alter your PATH environment variable. You will need to move both the C:\Program Files\Microsoft SQL Server\110\Tools\Binn\ and c:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\ paths before any other paths to SQL Server Tools (they will likely be at the end of your PATH variable by default – the order corresponds to installation order). This will make sure that when you use sqlcmd, the right version will be called.
• Note that in my PHP code, I’m providing an absolute path to the .mdf file. You must provide an absolute path. By default, the sqlcmd utility will create .mdf files in the User directory. However, you can use any .mdf file you might have…just provide an absolute path to it (and make sure that the sqlcmd utility has access to the directory). Beware that if the .mdf file was created with a previous version of SQL Server, it will be upgraded the first time you connect to it (which will produce a whole host of warnings). After it has been upgraded, everything should work smoothly.
• Also note that in my PHP code, I’m connecting using Windows Authentication (I’ve omitted the UID and PWD elements from the connection options array). When using LocalDB, you can only connect with Windows Authentication. And, if you are using IIS, it is the credentials of the application pool for PHP that are used to login. So, since we are talking about a dev environment here, I suggest setting the application pool credentials to an identity that is also a login for the targeted database. I did this by starting IIS Manager, clicking Application Pools, right clicking the DefaultAppPool (PHP is running under this pool in my case), then following these steps:

Friendlier Syntax: OFFSET…FETCH NEXT

I’ve written a few articles on strategies for paging data with the SQL Server Driver for PHP, all of which relied on the ROW_NUMBER…OVER syntax or server-side cursors. The ROW_NUMBER…OVER syntax is cumbersome compared to the MySQL LIMIT…OFFSET syntax, and server-side cursors are not appropriate for all scenarios. Enter the OFFSET…FETCH NEXT syntax in SQL Server 2012. Now it’s easy to get exactly the subset of data you want from a much larger set:

$sql = "SELECT Name, ProductNumber 

        FROM Production.Product

        ORDER BY ProductNumber

        OFFSET ? ROWS

        FETCH NEXT ? ROWS ONLY";

$params = array((int) $_GET['low'], (int) $_GET['high']);    

$stmt = sqlsrv_query($conn, $sql, $params);

More Flexibility: Client-Side Cursors

Prior to the release of v3.0 of the PHP driver, the only options available for moving through a result set were “forward only” (the default, which forces you to consume rows in order, and you can’t move back to previous rows) or to use server-side cursors (which, as mentioned earlier, are not ideal for all scenarios). However, in the 3.0 release, client-side cursors are available for getting random access to the rows of a result set. To enable this, set the Scrollable option to buffered on sqlsrv_query or sqlsrv_prepare:

$sql = "SELECT Name, ProductNumber 

        FROM Production.Product";

$stmt = sqlsrv_query($conn, $sql, null, array("Scrollable"=>"buffered"));

if($stmt === false)

    die(print_r(sqlsrv_errors(), true));

With client-side queries (a.k.a “buffered” queries) enabled, you can use the sqlsrv_num_rows function to get the number of rows, and you can use the following constants with sqlsrv_fetch, sqlsrv_fetch_array,  or sqlsrv_fetch_object to randomly access rows:

• SQLSRV_SCROLL_NEXT
• SQLSRV_SCROLL_PRIOR
• SQLSRV_SCROLL_FIRST
• SQLSRV_SCROLL_LAST
• SQLSRV_SCROLL_ABSOLUTE
• SQLSRV_SCROLL_RELATIVE

Just for fun, here’s one way you could use those constants to work backwards through a result set:

$num_rows = sqlsrv_num_rows($stmt);

for($i = $num_rows - 1; $i >= 0; $i--)

{

    $row = sqlsrv_fetch_array($stmt, SQLSRV_FETCH_ASSOC, SQLSRV_SCROLL_ABSOLUTE, $i);

    // process row here

}

Of course, you can also use client-side cursors with the PDO extension. There are plenty of examples in the documentation.

There’s More

The improvements mentioned above are those that I think have the biggest impact for PHP developers now. But there are other things you might be interested in, such as support for AlwaysOn Connectivity and for the PDO::ATTR_EMULATE_PREPARES constant in the PHP Driver. And, the PHP driver still provides access to SQL Azure, the SQL Server ODBC Driver for Linux is in a beta release (which some PHP developers are already taking advantage of), and Hadoop for Azure is on its way. The only down note in this release is the the PHP driver does not currently support PHP 5.4, but that’s a temporary thing. I know the team is working hard to quickly add that support.

As I mentioned at the beginning, I think the release of SQL Server 2012 takes some big steps toward improving the development experience for PHP developers. Thanks to all the folks who have provided feedback to help get us here…we’ve been listening. I’d be very interested to hear if you agree.

Thanks.

-Brian

Share this on Twitter