---

November 2018

Volume 33 Number 11

Azure Service Bus - Web Site Background Processing with Service Bus Queues

By Will Stott

I was working at the United Kingdom Collaborative Trial of Ovarian Cancer Screening (UKCTOCS), when I ran into a problem. Tasked with demonstrating a logistic regression classifier I had developed for a trial project, I began to build a simple, low-budget Web site to support it. But I needed the site to both support long-running background processing and start up the classifier server on-demand.

It was a two-pronged challenge, and one that I was able to tackle using key Microsoft technologies. In this article, I’ll explore how I leveraged Azure Functions and Service Bus queues to enable background processing, while in a future piece I’ll dive into the Azure Container Service and how it enabled on-demand provisioning of server resources. In that second article I’ll show you how to programmatically start and terminate an Azure Container using the Azure.Management.Fluent API.

As for the underlying Web site and its database, they’re hosted on Azure as a standard Web App based on ASP.Net Core 2.1 MVC, Entity Framework and SQL Server. Just in case you’re new to this technology, I’ve provided a companion article on the Web that gives step-by-step instructions for building the Web App and its database, as well as provisioning the associated Azure resources. You can find it at bit.ly/2Qbifod. However, most readers should be able to construct what’s required from the system overview shown in Figure 1 and the OvaryVis model shown in Figure 2. Really, the project is just a Web Form that captures three integers representing measurements made of an ovary, saves them in a database record, and displays a results page that shows whether or not the classifier judges the dimensions to be consistent with those of an ovary.

Figure 1 System Overview

Figure 2 The OvaryVis Class in the Models Folder

Recreating the project in this article doesn’t require much more than rudimentary Web development skills. I do assume that you’ve already created the ASP.NET Core 2.1 MVC Web App and its SQL Server database, as well as the associated Azure Resources (which are described in the companion article). In terms of tools, you’ll need Visual Studio 2017 v15.7 with .NET Core 2.1 SDK and the Web development workload. You’ll also need an Azure Subscription, but you can get what you need for free if you’re a new customer. All the source code and instructions for creating the Azure resources for the solution shown in Figure 1 are available at the GitHub repository (bit.ly/2NSiIuh).

Overview

A Web App, like the one developed for this article, supports multiple users by deciding what each of them wants and then returning an appropriate HTML page (View) to each individual browser. The time taken for the Web App to return a view determines the responsiveness of your Web site and the number of simultaneous users it can support. For this reason there’s a pressing need to avoid long-running process when obtaining the data needed by your Web site views. A Service Bus Queue is a good way to address this type of problem, as it allows your Web App to store messages containing the processing details, and then provides an immediate response to the user—even if it’s just a view that says come back later for the result. Another software component operates in the background to read the queue and perform the required processing for each message.

The advantage of Service Bus is that when lots of users are submitting data for classification, the queue simply gets longer. Yes, this means people may need to wait longer for their results, but it ensures that the Web site remains responsive. It also de-couples your system by separating foreground Web App processing from the component running your background processing. Azure services like Storage queues, EventHubs and EventGrid perform a similar function, but target other types of usage. You can find a useful comparison of Azure Storage queues and Service Bus queues at bit.ly/2QJVwNp, and a comparison of Azure messaging services at bit.ly/2DoY1T5.

In the case of my application, Service Bus queue is ideal. It’s also cheap and very easy to use, particularly now that there’s good integration with Azure Functions, which are perfect for doing my background processing.

Provisioning Azure Resources

Azure Cloud Shell is built into the Azure Portal Web site and allows you to provision Azure Resources using a series of simple commands from the PowerShell Console, as described in the online companion article. To create the Azure Service Bus queue and Function App, you must issue commands from the Cloud Shell, with appropriate values for your subscription, resource group and location. These values will be the same ones you used to provision your Web app and database. You also need to give yourself a unique Service Bus namespace rather than MSDNOvaryVisSBQ. Here are the commands to use:

az account set --subscription MsdnOvaryVis
az servicebus namespace create  --name MSDNOvaryVisSBQ --location "WestEurope"
  --resource-group resMSDNOvaryVis --sku Basic
az servicebus queue create --name dimsubmission --namespace-name MSDNOvaryVisSBQ
  --resource-group resMSDNOvaryVis --max-size 1024

You’ll notice that I opted for the lowest service tier (--sku Basic), which allows me to receive 1 million messages up to 256KB each in size for $0.05 with no monthly charge. In addition to creating a Service Bus queue you also need to create a Function app and a new storage account for it, as shown here:

az storage account create -name msdnovaryvisstorageac
  --resource-group resMSDNOvaryVis
  --location "WestEurope" --sku Standard_LRS
az functionapp create --name MSDNOvaryVisFnApp --resource-group resMSDNOvaryVis
  --storage-account msdnovaryvisstorageac --consumption-plan-location westeurope

The name of the Function app service is publicly accessible, so you’ll need to enter your own name in place of MSDNOvary­VisFnApp. You should also note that creating your first Azure Function results in the creation of a consumption plan, which is how you’ll be billed for usage, though for a project of this nature the cost should be almost nothing.

Microsoft’s intention is that you create Azure Functions without having to explicitly provision a server to run them—known as serverless execution. Microsoft uses consumption plans to bill customers based only on how often a function executes, how long it takes to execute and how much memory it uses. Yes, you can get a fixed-priced bill by electing to run your Azure Functions app service on the same virtual machine (VM) as your Web app, but this option isn’t available for Web apps that use shared infrastructure. As a result for this project, it was cheaper to use a consumption plan than to upgrade my Web app service plan.

Now let’s apply application settings to the new Function app to complete the provisioning step. Note that you must replace the connection value MMM with the one for your Service Bus, by copying the RootManageSharedAccessKey primary connection string from its Shared access policies blade in the Azure Portal. Here’s the code:

az functionapp config appsettings set --name MSDNOvaryVisFnApp
  --resource-group resMSDNOvaryVis --settings 'AzureServiceBusQueueName=dimsubmission'
az functionapp config appsettings set --name MSDNOvaryVisFnApp
  --resource-group resMSDNOvaryVis --settings 'AzureWebJobsServiceBus=MMM’
az functionapp config appsettings set --name MSDNOvaryVisFnApp
  --resource-group resMSDNOvaryVis
  --settings 'FUNCTIONS_EXTENSION_VERSION=2.0.12050.0'

Of course, in production code you would create an additional key in the Shared access policies blade with just send privileges, and then use its connection string instead of the one from the root key. You also should be aware that your Azure Function executes on a common runtime, which may change if there’s an update unless you define a particular version using the FUNCTIONS_EXTENSION_VERSION setting.

I discovered the need to specify my runtime the hard way, as Microsoft updated the pre-release runtime I was working with soon after I completed the development work for this article, causing my function to suddenly stop working. Fortunately, I was able to force Azure Functions to employ a specific runtime version to ensure that my function would only be executed on a server with the same runtime used during development. I don’t expect such breaking changes to happen now that the runtime has entered its beta releases, but it’s something worth knowing. You can see the runtime version for your own functions by visiting the Azure Function app settings blade in the Azure Portal.

Implementing an Azure Function App Service

The Azure Function needed to perform your Web site’s background processing will be developed from a Visual Studio Azure Function app project. The required template is installed on your PC as part of the .NET Core 2.1 SDK and the Web development workload.

Creating an Azure Function project takes less than a minute. First open the New Project dialog (File | New Project), select Azure Functions from the Cloud template folder in Visual C# and provide a suitable name for your project (OvaryVisFnApp), making sure Add to Solution is selected. In the next dialog provide the type of project (in this case, Empty) and select the storage account you just created. The new OvaryVisFnApp Project will then appear in your Solution Explorer together with an initial set of files.

Now is a good time to make sure you have the latest versions of the packages needed for the project. These are as displayed in the NuGet Solution window when you click Tools | NuGet Package Manager | Manage NuGet Packages for Solution. I used the following for this article, but you might want to try later versions for your own work:

• NETStandard.Library v2.0.3
• Microsoft.NET.Sdk.Functions v1.0.19

To implement the Azure Function itself, you need to add a new class to your OvaryVisFnApp project. Again Visual Studio provides good support. Select the project, right-click Add | New Azure Function and name the file OvaryVisSubmitProc.cs. Next, select Service Bus Queue Trigger as the type of Azure Function to create with AzureWebJobs­ServiceBus as the connection string setting and dimsubmission as the queue name (see Figure 3). This creates the required class, which you should update by providing myQueueItem as the log.Info parameter, as follows:

public static class OvaryVisSubmitProc
{
  [FunctionName("OvaryVisSubmitProc")]
  public static async Task Run([ServiceBusTrigger("dimsubmission", 
    Connection = "AzureWebJobsServiceBus")]string myQueueItem,
    TraceWriter log)
  {
    log.Info(myQueueItem));
  }
}

Figure 3 Creating an Azure Function for Service Bus Queue Trigger

Recall that AzureWebJobsServiceBus is the name of the Appli­cation setting you applied to your Azure Function app earlier, while dimsubmission is the actual name of your Service Bus queue.

Now you can publish your Azure Function project. This is done in much the same way as the Web app, which I describe in the companion article: select the project and right-click Publish. You need to select an existing (rather than new) Azure Function App Service so you can publish your project to the MSDNOvaryVisFn­App resource you created earlier. Helpfully, MSDNOvaryVisFnApp appears in the Resource Group folder in the next dialog box. The only thing to worry about is that the Azure App Service is halted before you publish from Visual Studio, as its DLLs cannot be overwritten when in use by a running process. You can achieve this by issuing the following command from the Cloud Shell:

az functionapp stop --name MSDNOvaryVisFnApp --resource-group resMSDNOvaryVis

After successfully publishing your project, you can start the service by either issuing a start command from the Cloud Shell, or by clicking the start button in the Overview blade, as shown in Figure 4.

Figure 4 Azure Function App Service Overview Blade

Testing your function in the Azure Portal is a good idea at this point, particularly if you’ve already opened the Overview blade of your Azure Function App Service. The OvaryVisSubmitProc function is listed on the left of the blade, and selecting it displays a page that allows you to test it, though you might have to click the Test item on the right vertical menu to show this page fully.

The Test page is not particularly pretty, as you can see in Figure 5, but it is useful. The first thing you need to do is type a suitable message in the Request body box—for example, “hello world.” Later we’ll replace this string by the OvaryVis record ID created by the Web app HomeController.Index (HttpPost) method. However, be aware this message can have much larger and complex content, as well as metadata to describe payload and handling instructions (for more on that, see the Azure Service Messaging document, “Messages, Payloads, and Serialization,” at bit.ly/2OCnIjX). Indeed, depending on your selected service tier, you can create Service Bus queues able to hold up to 80GB of messages with each message being up to 1MB in size.

Figure 5 Azure Function App Service Test Blade

Once you’ve created the Request body text in the Test page, you need to click Run to simulate an event being sent to your Run function. The processing of this event can be seen in the Log window, shown at the bottom of Figure 5. You should note that the value of the data field in the log is the same as that in the request body. When your function returns, it will cause a Status 202 Accepted message to appear in the Test page. This indicates that your function and the Azure Function App Service are working correctly.

Sending Messages to the Service Bus Queue from a Web App

It’s presumed that you’ve already built a Web app like the one described in the accompanying online article. It just needs to have two Controller methods that handle the creation of a view with a simple form and its subsequent posting back to the Web app. In our case that’s a HomeController with Index methods for HttpGet and HttpPost.

Installing Azure.ServiceBus package in your Web app project will allow it to use the component needed for sending an event message to your Service Bus queue. The Visual Studio NuGet Package Manager Console enables you to add such packages, as described in Microsoft Docs at bit.ly/2QIiOmZ. You should open the console from the Tools menu (Tools | NuGet Package Manager | Package Manager Console) and then issue the following command:

Install-Package Microsoft.Azure.ServiceBus -Version 3.1.0 -Project OvaryVisWebApp

This also installs dependencies like WebJobs.Extensions.ServiceBus (3.0.0-beta8), so you should now have everything you need to send messages to your Service Bus queue.

Now it’s time to set up the Service Bus queue client. This requires you to pass to it the name of your queue and its connection. I found it more convenient to keep these values as application configuration settings, so I initialized the Service Bus queue client in the Web app’s Startup class constructor, as shown in Figure 6. I then used the Cloud Shell to give the following commands, again replacing MMM with the same Service Bus connection string used previously when setting up the Azure Function app:

az webapp config appsettings set --name MSDNOvaryVisWebApp
  --resource-group resMSDNOvaryVis --settings 'OvaryVisServiceBus:QueueName=dimsubmission'
az webapp config appsettings set --name MSDNOvaryVisWebApp
  --resource-group resMSDNOvaryVis --settings 'OvaryVisServiceBus:Connection=MMM'

You’ll see in Figure 6 that the queue client is held in a static variable, which is initially set as null and then initialized by the SetupServiceBus method. The use of lock makes this method thread safe, while testing that _queueClient is null avoids more than one client being constructed. This is overkill for a Web app as the Startup class is initialized only once, but it does allow the SetupServiceBus method to be copied to the Azure Function you’ll create in the next article, where such protection will be needed.

Figure 6 Setting Up the Service Bus Queue Client in the Web App’s Startup Class

public class Startup
{
  private static IQueueClient _queueClient = null;
  private static readonly object _accesslock = new object();
  private static void SetupServiceBus(string connection, string queueName)
  {
    lock (_accesslock)
    {
      if (_queueClient == null)
        _queueClient = new QueueClient(connection, queueName);
    }
  }
  public static IQueueClient GetQueueClient() { return _queueClient; }
  public Startup(IConfiguration configuration)
  {
    Configuration = configuration;
    SetupServiceBus(Configuration["OvaryVisServiceBus:Connection"],
      Configuration["OvaryVisServiceBus:QueueName"]);
  }
}

Next, I need to update the HttpPost Index method of the HomeController class so that its code is like that shown in Figure 7. This allows you to send an event message to your Service Bus queue in the same way you did using the Azure Function App Service Test page. However, you also need to add the following using statements at the top of the file:

using Microsoft.Azure.ServiceBus;
using System.Net.Http;

You’ll see from Figure 7 that after initially saving the form data as a record in the OvaryVis table, you create a Message object with its body set as the record ID string. After you build and run your Web App locally (Ctrl+F5), the Result page will display Job Created as its StatusMsg upon submission of the Index page’s form. In addition, you’ll see the record ID value displayed in the Portal’s Azure Function log. This indicates that you have not only succeeded in saving a record to the database, but you’ve also transmitted its ID to the Azure Function. All you need to do now is update your Azure Function so it uses this ID to read the corresponding record from the database in the Azure Function.

Figure 7 HomeController Index Method

[HttpPost]
[ValidateAntiForgeryToken]
public async Task<IActionResult> Index([Bind("Id,D1mm,D2mm,D3mm")] OvaryVis form)
{
  OvaryVis record = new OvaryVis
  {
    D1mm = form.D1mm,  // First ovary dimension
    D2mm = form.D2mm,  // Second ovary dimension
    D3mm = form.D3mm,  // Third ovary dimension
    JobSubmitted = DateTime.UtcNow,
    ResultVis = -1,    // Result code: -1 = not yet processed 
    StatusMsg = String.Format("Job Created at {0}",
      DateTime.UtcNow.ToString("HH:mm:ss"))
  };
                       // Add record to database
  _context.Add(record);
  await _context.SaveChangesAsync();
                       // Send message with ID value to Service Bus Queue
  var message = new Message(Encoding.UTF8.GetBytes(record.Id));
  await Startup.GetQueueClient().SendAsync(message);
                       // Call Result method to return Result page
  return RedirectToAction("Result", new { id = record.Id });
}

Adding Entity Framework to the Azure Function

Entity Framework provides a simple way of retrieving the Ovary­Vis record from your database using the ID value passed in the Service Bus event message. Implementation involves adding the Entity Framework package to your Azure Function project, using the same Package Manager Console you used to install the Service Bus package, like so:

Install-Package Microsoft.EntityFrameworkCore.SqlServer
  -Project OvaryVisFnApp –version 2.1.2

You also need to provide a connection string so your Azure Function can access your Azure SQL Database. This is the same connection string used by your Web app and it’s best set from your Azure Function’s Applications Settings, which is accessible from its Overview blade in the Portal shown in Figure 4. You need to scroll down to the Connection Strings section and then click Add new connection string with the name DefaultConnection, type SQLAzure and the following value, though with the name of your own database server and its username and password:

Server=tcp:msdnovaryvisdbsvr.database.windows.net,1433;Database=Msdn.OvaryVisDb;
  User  ID=XXX;Password=YYY;Encrypt=true;Connection Timeout=30;

After adding the string, don’t forget to scroll back to the top and click Save.

The last thing you need to do to access your database from your Azure Function is copy the DataContext class and the OvaryVis model class from your Web app project to your Function app project. You’ll need to change the namespace of both files to Ovary­VisFnApp, but otherwise you should be able to rebuild your solution without any problems (press F7). Finally, you need to implement the following method and then call it from the Run method of the OvaryVisSubmitProc function. Here’s the code:

private static async Task<string> FormSubmittedProc(IConfigurationRoot config,
  ApplicationDbContext dbContext, string queueItem)
{
  string rc = "FormSubmittedProc: ";
  var record = await dbContext.OvaryVis.SingleOrDefaultAsync(a => a.Id == queueItem);
  if (record == null)
    rc += string.Format("record not found: Id={0}", msg.Id);
  else
    rc += string.Format("record Id={0} found, ", record.Id);
  return rc;
}

The config parameter passed to FormSubmittedProc by the Azure Function Run method is obtained using ConfigurationBuilder and provides access to its application settings, including the database connection string. This is then used to create the ApplicationDbContext object that forms the second parameter. You can find the exact implementation details in the sample code download. 

You’ve completed the development work for this article, so now it’s time to rebuild your Visual Studio Solution, publish its Azure Function app project and give it a quick smoke test. If you open the Azure Function App Service Test Blade that I described earlier (Figure 5), you should see the ovary record Id displayed in the log whenever you submit an input form from your Web site, as shown in Figure 8. This indicates that your Azure Function has successfully found the record corresponding to the Id value contained in the Service Bus message sent from the Web app, so all the parts of the system shown in Figure 1 are working as intended.

Figure 8 The Web Site Input Form and Result Page

Wrapping Up

It’s worth reflecting how easy Azure makes it to develop a robust background processing mechanism for any Web site. With a production site, you’d want to provide some way to automatically update the results page, rather than rely on the user pressing F4—but this isn’t too difficult to achieve using SignalR or a similar mechanism. However, the provisioning of an Azure Function App Service and Service Bus queue was achieved with just a few commands from the Azure Portal, and integration with your Web app’s Home Controller required only a few lines of code.

There’s very little associated cost for this sort of implementation beyond the small monthly charge for an App Service Plan and database server. If you’re running on the free tier of the App Service Plan, you should be able to host a test Web site for less than the cost of a couple cups of coffee a month. This makes it a truly compelling solution.

In the next article, I’ll explore how to extend an Azure Function by submitting the dimension values in the record to the classifier running on a server provisioned from a Docker image. You’ll also see how to start this server automatically when traffic arrives at your Web site, and then stop it after the traffic ceases, implementing an on-demand server that’s billed only for the actual time used.

---

Dr. Will Stott has more than 25 years of experience working as a contractor/consultant for a wide range of companies in the United Kingdom and Europe, including IBM, Cap-Gemini, Logica CMG and Accenture. However, for the last 10 years most of his time has been spent doing research at University College London (UCL) on Ovarian Cancer Screening. Dr. Stott has spoken at many conferences both in the United Kingdom and internationally. He’s also the author of papers published in various journals, as well as the book “Visual Studio Team System: Better Software Development for Agile Teams” (Addison-Wesley Professional, 2007).

Thanks to the following Microsoft technical expert for reviewing this article: David Barkol

---

Discuss this article in the MSDN Magazine forum