Tutorial: Build a highly available application with Blob storage
Article
14 minutes to read
This tutorial is part one of a series. In it, you learn how to make your application data highly available in Azure.
When you've completed this tutorial, you'll have a console application that uploads and retrieves a blob from a read-access geo-zone-redundant (RA-GZRS) storage account.
Geo-redundancy in Azure Storage replicates transactions asynchronously from a primary region to a secondary region that is hundreds of miles away. This replication process guarantees that the data in the secondary region is eventually consistent. The console application uses the circuit breaker pattern to determine which endpoint to connect to, automatically switching between endpoints as failures and recoveries are simulated.
Download the sample project, extract (unzip) the storage-dotnet-circuit-breaker-pattern-ha-apps-using-ra-grs.zip file, then navigate to the v12 folder to find the project files.
You can also use git to clone the repository to your local development environment. The sample project in the v12 folder contains a console application.
Download the sample project, extract (unzip) the storage-dotnet-circuit-breaker-pattern-ha-apps-using-ra-grs.zip file, then navigate to the v11 folder to find the project files.
You can also use git to download a copy of the application to your development environment. The sample project in the v11 folder contains a console application.
Download the sample project and extract (unzip) the storage-python-circuit-breaker-pattern-ha-apps-using-ra-grs.zip file. You can also use git to download a copy of the application to your development environment. The sample project contains a basic Python application.
Download the sample project and unzip the file. You can also use git to download a copy of the application to your development environment. The sample project contains a basic Node.js application.
Application requests to Azure Blob storage must be authorized. Using the DefaultAzureCredential class provided by the Azure.Identity client library is the recommended approach for connecting to Azure services in your code. The .NET v12 code sample uses this approach. To learn more, please see the DefaultAzureCredential overview.
You can also authorize requests to Azure Blob Storage by using the account access key. However, this approach should be used with caution to protect access keys from being exposed.
In the application, you must provide the connection string for your storage account. You can store this connection string within an environment variable on the local machine running the application. Follow one of the examples below depending on your Operating System to create the environment variable.
In the Azure portal, navigate to your storage account. Select Access keys under Settings in your storage account. Copy the connection string from the primary or secondary key. Run one of the following commands based on your operating system, replacing <yourconnectionstring> with your actual connection string. This command saves an environment variable to the local machine. In Windows, the environment variable isn't available until you reload the Command Prompt or shell you're using.
In the application, you must provide your storage account credentials. You can store this information in environment variables on the local machine running the application. Follow one of the examples below depending on your Operating System to create the environment variables.
In the Azure portal, navigate to your storage account. Select Access keys under Settings in your storage account. Paste the Storage account name and Key values into the following commands, replacing the <youraccountname> and <youraccountkey> placeholders. This command saves the environment variables to the local machine. In Windows, the environment variable isn't available until you reload the Command Prompt or shell you're using.
In Visual Studio, press F5 or select Start to begin debugging the application. Visual Studio automatically restores missing NuGet packages if package restore is configured. See Installing and reinstalling packages with package restore to learn more.
When the console window launches, the app will get the status of the secondary region and write that information to the console. Then the app will create a container in the storage account and upload a blob to the container. Once the blob is uploaded, the app will continuously check to see if the blob has replicated to the secondary region. This check continues until the blob is replicated, or we reach the maximum number of iterations as defined by the loop conditions.
Next, the application enters a loop with a prompt to download the blob, initially reading from primary storage. Press any key to download the blob. If there's a retryable error reading from the primary region, a retry of the read request is performed against the secondary region endpoint. The console output will show when the region switches to secondary.
To exit the loop and clean up resources, press the Esc key at the blob download prompt.
In Visual Studio, press F5 or select Start to begin debugging the application. Visual Studio automatically restores missing NuGet packages if package restore is configured, visit Installing and reinstalling packages with package restore to learn more.
A console window launches and the application begins running. The application uploads the HelloWorld.png image from the solution to the storage account. The application checks to ensure the image has replicated to the secondary RA-GZRS endpoint. It then begins downloading the image up to 999 times. Each read is represented by a P or an S. Where P represents the primary endpoint and S represents the secondary endpoint.
In the sample code, the RunCircuitBreakerAsync task in the Program.cs file is used to download an image from the storage account using the DownloadToFileAsync method. Prior to the download, an OperationContext is defined. The operation context defines event handlers that fire when a download completes successfully, or if a download fails and is retrying.
To run the application on a terminal or command prompt, go to the circuitbreaker.py directory, then enter python circuitbreaker.py. The application uploads the HelloWorld.png image from the solution to the storage account. The application checks to ensure the image has replicated to the secondary RA-GZRS endpoint. It then begins downloading the image up to 999 times. Each read is represented by a P or an S. Where P represents the primary endpoint and S represents the secondary endpoint.
In the sample code, the run_circuit_breaker method in the circuitbreaker.py file is used to download an image from the storage account using the get_blob_to_path method.
The Storage object retry function is set to a linear retry policy. The retry function determines whether to retry a request, and specifies the number of seconds to wait before retrying the request. Set the retry_to_secondary value to true, if request should be retried to secondary in case the initial request to primary fails. In the sample application, a custom retry policy is defined in the retry_callback function of the storage object.
Before the download, the Service object retry_callback and response_callback function is defined. These functions define event handlers that fire when a download completes successfully or if a download fails and is retrying.
To run the sample, open a command prompt, navigate to the sample folder, then enter node index.js.
The sample creates a container in your Blob storage account, uploads HelloWorld.png into the container, then repeatedly checks whether the container and image have replicated to the secondary region. After replication, it prompts you to enter D or Q (followed by ENTER) to download or quit. Your output should look similar to the following example:
Created container successfully: newcontainer1550799840726
Uploaded blob: HelloWorld.png
Checking to see if container and blob have replicated to secondary region.
[0] Container has not replicated to secondary region yet: newcontainer1550799840726 : ContainerNotFound
[1] Container has not replicated to secondary region yet: newcontainer1550799840726 : ContainerNotFound
...
[31] Container has not replicated to secondary region yet: newcontainer1550799840726 : ContainerNotFound
[32] Container found, but blob has not replicated to secondary region yet.
...
[67] Container found, but blob has not replicated to secondary region yet.
[68] Blob has replicated to secondary region.
Ready for blob download. Enter (D) to download or (Q) to quit, followed by ENTER.
> D
Attempting to download blob...
Blob downloaded from primary endpoint.
> Q
Exiting...
Deleted container newcontainer1550799840726
The sample creates a BlobServiceClient object configured with retry options and a secondary region endpoint. This configuration allows the application to automatically switch to the secondary region if the request fails on the primary region endpoint.
string accountName = "<YOURSTORAGEACCOUNTNAME>";
Uri primaryAccountUri = new Uri($"https://{accountName}.blob.core.windows.net/");
Uri secondaryAccountUri = new Uri($"https://{accountName}-secondary.blob.core.windows.net/");
// Provide the client configuration options for connecting to Azure Blob storage
BlobClientOptions blobClientOptions = new BlobClientOptions()
{
Retry = {
// The delay between retry attempts for a fixed approach or the delay
// on which to base calculations for a backoff-based approach
Delay = TimeSpan.FromSeconds(2),
// The maximum number of retry attempts before giving up
MaxRetries = 5,
// The approach to use for calculating retry delays
Mode = RetryMode.Exponential,
// The maximum permissible delay between retry attempts
MaxDelay = TimeSpan.FromSeconds(10)
},
// Secondary region endpoint
GeoRedundantSecondaryUri = secondaryAccountUri
};
// Create a BlobServiceClient object using the configuration options above
BlobServiceClient blobServiceClient = new BlobServiceClient(primaryAccountUri, new DefaultAzureCredential(), blobClientOptions);
When the GeoRedundantSecondaryUri property is set in BlobClientOptions, retries for GET or HEAD requests will switch to use the secondary endpoint. Subsequent retries will alternate between the primary and secondary endpoint. However, if the status of the response from the secondary Uri is 404, then subsequent retries for the request will no longer use the secondary Uri, as this error code indicates the resource hasn't replicated to the secondary region.
Retry event handler
The OperationContextRetrying event handler is called when the download of the image fails and is set to retry. If the maximum number of retries defined in the application are reached, the LocationMode of the request is changed to SecondaryOnly. This setting forces the application to attempt to download the image from the secondary endpoint. This configuration reduces the time taken to request the image as the primary endpoint isn't retried indefinitely.
private static void OperationContextRetrying(object sender, RequestEventArgs e)
{
retryCount++;
Console.WriteLine("Retrying event because of failure reading the primary. RetryCount = " + retryCount);
// Check if we have had more than n retries in which case switch to secondary.
if (retryCount >= retryThreshold)
{
// Check to see if we can fail over to secondary.
if (blobClient.DefaultRequestOptions.LocationMode != LocationMode.SecondaryOnly)
{
blobClient.DefaultRequestOptions.LocationMode = LocationMode.SecondaryOnly;
retryCount = 0;
}
else
{
throw new ApplicationException("Both primary and secondary are unreachable. Check your application's network connection. ");
}
}
}
Request completed event handler
The OperationContextRequestCompleted event handler is called when the download of the image is successful. If the application is using the secondary endpoint, the application continues to use this endpoint up to 20 times. After 20 times, the application sets the LocationMode back to PrimaryThenSecondary and retries the primary endpoint. If a request is successful, the application continues to read from the primary endpoint.
private static void OperationContextRequestCompleted(object sender, RequestEventArgs e)
{
if (blobClient.DefaultRequestOptions.LocationMode == LocationMode.SecondaryOnly)
{
// You're reading the secondary. Let it read the secondary [secondaryThreshold] times,
// then switch back to the primary and see if it's available now.
secondaryReadCount++;
if (secondaryReadCount >= secondaryThreshold)
{
blobClient.DefaultRequestOptions.LocationMode = LocationMode.PrimaryThenSecondary;
secondaryReadCount = 0;
}
}
}
The retry_callback event handler is called when the download of the image fails and is set to retry. If the maximum number of retries defined in the application are reached, the LocationMode of the request is changed to SECONDARY. This setting forces the application to attempt to download the image from the secondary endpoint. This configuration reduces the time taken to request the image as the primary endpoint isn't retried indefinitely.
def retry_callback(retry_context):
global retry_count
retry_count = retry_context.count
sys.stdout.write(
"\nRetrying event because of failure reading the primary. RetryCount= {0}".format(retry_count))
sys.stdout.flush()
# Check if we have more than n-retries in which case switch to secondary
if retry_count >= retry_threshold:
# Check to see if we can fail over to secondary.
if blob_client.location_mode != LocationMode.SECONDARY:
blob_client.location_mode = LocationMode.SECONDARY
retry_count = 0
else:
raise Exception("Both primary and secondary are unreachable. "
"Check your application's network connection.")
Request completed event handler
The response_callback event handler is called when the download of the image is successful. If the application is using the secondary endpoint, the application continues to use this endpoint up to 20 times. After 20 times, the application sets the LocationMode back to PRIMARY and retries the primary endpoint. If a request is successful, the application continues to read from the primary endpoint.
def response_callback(response):
global secondary_read_count
if blob_client.location_mode == LocationMode.SECONDARY:
# You're reading the secondary. Let it read the secondary [secondaryThreshold] times,
# then switch back to the primary and see if it is available now.
secondary_read_count += 1
if secondary_read_count >= secondary_threshold:
blob_client.location_mode = LocationMode.PRIMARY
secondary_read_count = 0
With the Node.js V10 SDK, callback handlers are unnecessary. Instead, the sample creates a pipeline configured with retry options and a secondary endpoint. This configuration allows the application to automatically switch to the secondary pipeline if it fails to reach your data through the primary pipeline.
