Microsoft Azure - How Azure, Web API and Redis Helped Deliver Data Faster

By Johnny Webb | December 2015

In today’s worldof real-time marketing, the spinning circle of death that too often accompanies waiting for a page to load or a search query to populate can quite literally mean the death of a sale. As more and more data records are created, stored and analyzed, the generation of rich, real-time data insights grows ever more challenging. The question becomes how to choose the right technology tools and architectures to best sift through millions of data records and deliver instantaneous results for a better customer experience.

This article will explore a recent use case in which we helped a client implement Redis and Web API. We’ll discuss our implementation approach, the challenges we encountered and how we ultimately achieved the performance requirements.

Business Challenge

Several months ago, a Fortune 100 diversified insurance and financial services organization approached Harte Hanks, a global marketing company, with a new initiative. The company wanted to provide a better, richer customer experience during the online insurance quote process. To do this, it needed to enable fast and proactive access to the customer’s policy data.

The objective was to allow customers to view and select one or more insurance policies through Internet self-service or agent-facing applications. Ultimately, this would lead to higher business value, greater customer satisfaction, higher quote-to-bind rates, and an improved agent commissioning process. In order to make this a reality, during the quote process the customers’ stored details and data needed to be processed, matched and provided proactively, in a near instantaneous manner.

The client’s database contained more than 50 million records, which meant with any traditional database a request of this nature typically would take multiple seconds to load. We were tasked with providing a Web service capable of executing queries on this large data source and delivering results to consumers within milliseconds.

For this task, we evaluated the implementation of Redis and Web API. The overall measure factors were:

Transaction Requests
Object Requests
Responses Per Second
Total Bandwidth of Data
Total Throughput of the Site

Implementation: How We Did It

Building a Web service to expose data requires multiple layers of components providing specific functionality. As the demand for the Web service grows, the infrastructure supporting these layers must adapt and scale quickly. And though scalability is essential, availability is equally important and must always be considered in order to keep the consumer happy. Furthermore, exposed endpoints must be monitored and wrapped with specific authentication protocols to ensure the data is protected and delivered securely with the maximum performance. To help accommodate these demands for our solution, we leveraged Microsoft Azure. Within Azure, we deployed a combination of resources including virtual machines, Web apps, virtual networking, resource groups and availability sets to build a fully functional, high-quality solution.

The Data Layer

Because the majority of our solutions are built using the Microsoft stack, it makes sense that we leverage Microsoft SQL Server most of the time. However, the SLA requirements for this solution specified a service response time of less than 1 second per request, at a rate of about 6,000 requests per hour, on a dataset with more than 50 million records. Because traditional databases like SQL Server store data to disk and are bottlenecked by IOPS, we couldn’t guarantee that every query would pass that requirement. To further complicate the matter, the subset of data we needed to expose already belonged to a traditional database containing terabytes of unrelated data. For that reason, we began evaluating solutions that could support large datasets quickly. That’s when we discovered Redis.

Redis is an in-memory data structure store that supports multiple datatypes. The essential server requirements include a Linux distribution, enough RAM to encapsulate your data and enough disk space for data persistence. Redis can also be configured as a cluster, but this is more suitable for solutions with terabytes of data. Because our dataset was less than 50GB, we decided to keep it simple and set up a single master/slave configuration. To host this configuration, we created two CentOS 7.1 virtual machines within a virtual network on Azure. Each VM contained 56GB of memory, a static IP address and an SSH endpoint with AES256 encryption. Because both VMs shared an available set, Azure provided a 99.95 percent SLA guaranteed uptime. As an added bonus, all the resources we created were appended to a resource group created specifically for this solution, allowing us to monitor and manage billing on a monthly basis. Within just a few minutes, our VMs were deployed and available to us.

Standing up our Redis servers was simple and accomplished well within an hour. After downloading and installing the latest stable release onto our newly created servers, first and foremost we modified the configuration file to allow what Redis calls append-only file (AOF) persistence. In short, this means that every command sent to our instance is stored to disk. If the server were to restart, all commands are re-executed, restoring the server to its original state. To eliminate a bloated AOF, a BGREWRITEAOF command is triggered occasionally, rewriting the file with the shortest sequence of commands needed to rebuild the current dataset in memory. Additionally, we configured a maximum memory use of 50GB, creating a buffer and preventing Redis from consuming all system memory if too much data was loaded. With that in mind, if our solution ever needed more memory, we could easily resize the VM using the Azure Portal and update the configuration later. Next, we configured the slave instance to acknowledge the master, creating data replication. If the master instance were to become unavailable, the slave instance would be ready to serve our clients. Finally, we restarted our Redis servers for the configurations to take effect. Here are the configurations we used:

appendonly yes
maxmemory 50000000000
slaveof 10.0.0.x 6379
# cluster-enabled no

Data Loading and Benchmarking

The data we needed to load into Redis from our traditional database contained approximately 4GB of customer metadata. This metadata was updated daily, so we needed to create a process to transform it into a Redis datatype and bulk load it in order to keep our solution current. To execute this, we created an automated process to extract the daily change sets into Redis protocol-formatted files and transferred them to the master server using the SSH endpoint available. Within the master instance, we created a bash script to bulk load the files using pipe mode. To emphasize, pipe mode was a key element of our solution, as we were able to load 28 million records within 5 minutes. Note, however, that pipe mode is not yet compatible with a cluster implementation. During our initial prototyping phase, we discovered that loading 28 million records into a cluster would take hours because the records were transferred individually. This was a significant factor in our decision to keep the design a simple master/slave implementation. The pipe mode command and response for a single instance looks like this:

$ cat data.txt | redis-cli –pipe
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 1000000

After pipe mode executed, the response indicated how many errors and replies were received. We parsed this response within the script, archived the file and sent an e-mail notification to the appropriate technical teams. For each error, the techs could evaluate the extract and quickly identify any data that would need to be reloaded. After completion of the daily processing script, we benchmarked our implementation using the Redis benchmark utility. Figure 1 shows the performance results of our solution, reassuring us that it would meet our SLA requirements.

Figure 1 The Redis Benchmark

Service Layer

It was critical that our client be the only consumer of our solution. Luckily, Azure makes this easy with Access Control Service, allowing us to create an OAuth provider specifically for our solution. Once implemented, our services required a specific token per request, with a new token regenerated after 5 minutes. As a side note, the token should always persist until expiration. Requesting a new token for every service call would essentially bottleneck your solution or, worse, make it inaccessible if you exceed the request limit specified by Azure. We strongly recommend that anyone leveraging this feature read the Azure documentation before implementing it in a production environment.

There are multiple C# Redis clients available, but we used StackExchange.Redis because we felt it was the most popular and mature. Because all of our data was stored in sets, we used the LRANGE command to query the data. To prevent a reconnection for each query, the connection used by the StackExchange.Redis was managed in a singleton pattern. The example in Figure 2 demonstrates how we retrieve data.

Figure 2 Connecting to Redis and Retrieving Data

private static Lazy<ConnectionMultiplexer> lazyConnection =
  new Lazy<ConnectionMultiplexer>(() => {
  return ConnectionMultiplexer.Connect(
    ConfigurationManager.AppSettings[Constants.AppSettings.RedisConnection]);
});
public static ConnectionMultiplexer Connection
{
  get
  {
    return lazyConnection.Value;
  }
}
public async static Task<MemberResponse> MemberLookup(MemberRequest request)
{
  MemberResponse response = new MemberResponse();
  try
  {
    if (request != null && request.customer != null && request.customer.Count() > 0)
    {
      foreach (var customer in request.customer)
      {
        Customer c = customer;
        RedisKey key = GetLookupKey(c);
        IDatabase db = Connection.GetDatabase();
        c.policies = await db.ListRangeAsync(key, 0, -1, CommandFlags.None);
        response.customer.Add(c);
      }
      response.success = true;
    }
    else
    {
      response.exceptions = new List<ServiceException>();
      response.exceptions.Add(Classes.ErrorCodes.Code_101_CustomerRequired);
      response.success = false;
        }
  }
  catch (Exception ex)
  {
    response.exceptions = new List<ServiceException>();
    response.exceptions.Add(Classes.ErrorCodes.Code_100_InternalError);
    response.success = false;
    Logging.LogException(ex);
  }
  response.executedOn =
    Utils.FormatEST(DateTime.UtcNow).ToString(Constants.DateTimeFormat);
  return response;
}

In order to host our solution in Azure, we created a Web app. For maximum performance, it was critical that we deployed the Web app to the same region as the Redis instances. Once the app was deployed, we created a VNET connection to our Redis virtual network, allowing the service to obtain direct access to the data. This connection is demonstrated in Figure 3.

Figure 3 VNET Integration Connection

This Web app was configured to scale to 10 instances based on CPU usage. Because the traffic for our solution varied, the Web app would scale as necessary. As an added layer of security, an SSL certificate was applied to the app, along with IP filtering specifically for our client. Although the app was configured to scale automatically, it didn’t do automatic failover. To maximize the availability of the solution to our client, we created a clone of the primary Web app, but placed it in a different region. Both apps were added to an Azure Traffic Manager, which we leveraged for automatic failover.

Finally, we purchased a custom domain and created a CNAME record pointing to our Traffic Manager URL, completing our implementation. In order to monitor the daily performance of our solution, we purchased an instance of New Relic directly from the Azure marketplace. By utilizing New Relic, we could quickly identify areas that needed improvement, as well as server availability.

Wrapping Up

By deploying this solution to Azure, we learned how to make different technology stacks work together quickly to provide a powerful, successful solution. Although deploying a solution to Azure doesn’t eliminate server maintenance, if you follow the patterns in place, you will have success. Depending on the average duration of your solutions, you could save on hardware costs by moving all your solutions to the cloud.

By the end of the implementation, the average response time was 25 milliseconds for 50 concurrent users. Based on these results, the response time is extended roughly 10 milliseconds per 15 added concurrent users.

Sean Iannuzzi is head of technology for Harte Hanks and has been in the technology industry for more than 20 years, playing a pivotal role in bridging the gap between technology and business visions for a plethora of social networking, Big Data, database solutions, cloud computing, e-commerce and financial applications of today. Iannuzzi has experience with more than 50 unique technology platforms, has achieved more than a dozen technical awards/certifications and specializes in driving technology direction and solutions to help achieve business objectives.

Johnny Webb is a software architect for Harte Hanks and has implemented high-quality solutions using cutting-edge technologies for well-known clients for the past 10 years. His expertise includes the full software development lifecycle, as well as the Microsoft .NET Framework, software architecture, cloud computing, Web development, mobile development, API development, SQL databases and NoSQL databases.

Thanks to the following technical expert for reviewing this article: Eric Riddle (Liquidhub)
Eric Riddle is a technical manager for Liquid Hub in Philadelphia. He has been developing and designing solutions with the .NET Framework since its inception. His expertise includes front-end web development, REST API, WCF, Windows Services, and SQL Server. He has in-depth experience in the financial services and direct marketing industry.