Edit

Troubleshoot client application errors in Azure storage accounts

  • Article

This article helps you investigate client application errors by using metrics, client side logs, and resource logs in Azure Monitor.

Diagnosing errors

Users of your application may notify you of errors reported by the client application. Azure Monitor also records counts of different response types (ResponseType dimensions) from your storage services, such as NetworkError, ClientTimeoutError, or AuthorizationError. While Azure Monitor only records counts of different error types, you can obtain more detail about individual requests by examining server-side, client-side, and network logs. Typically, the HTTP status code returned by the storage service will give an indication of why the request failed.

Note

Remember that you should expect to see some intermittent errors. For example, errors due to transient network conditions or application errors.

The following resources are useful for understanding storage-related status and error codes:

The client is receiving HTTP 403 (Forbidden) messages

If your client application is throwing HTTP 403 (Forbidden) errors, a likely cause is that the client is using an expired Shared Access Signature (SAS) when it sends a storage request (although other possible causes include clock skew, invalid keys, and empty headers).

The Storage Client Library for .NET enables you to collect client-side log data that relates to storage operations performed by your application. For more information, see Client-side Logging with the .NET Storage Client Library.

The following table shows a sample from the client-side log generated by the Storage Client Library that illustrates this issue occurring:

Source Verbosity Verbosity Client request ID Operation text
Microsoft.Azure.Storage Information 3 85d077ab-… Starting operation with location Primary per location mode PrimaryOnly.
Microsoft.Azure.Storage Information 3 85d077ab -… Starting synchronous request to <https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Synchronous_and_Asynchronous_Requests#Synchronous_request>
Microsoft.Azure.Storage Information 3 85d077ab -… Waiting for response.
Microsoft.Azure.Storage Warning 2 85d077ab -… Exception thrown while waiting for response: The remote server returned an error: (403) Forbidden.
Microsoft.Azure.Storage Information 3 85d077ab -… Response received. Status code = 403, Request ID = <Request ID>, Content-MD5 = , ETag = .
Microsoft.Azure.Storage Warning 2 85d077ab -… Exception thrown during the operation: The remote server returned an error: (403) Forbidden..
Microsoft.Azure.Storage Information 3 85d077ab -… Checking if the operation should be retried. Retry count = 0, HTTP status code = 403, Exception = The remote server returned an error: (403) Forbidden..
Microsoft.Azure.Storage Information 3 85d077ab -… The next location has been set to Primary, based on the location mode.
Microsoft.Azure.Storage Error 1 85d077ab -… Retry policy did not allow for a retry. Failing with The remote server returned an error: (403) Forbidden.

In this scenario, you should investigate why the SAS token is expiring before the client sends the token to the server:

  • Typically, you shouldn't set a start time when you create a SAS for a client to use immediately. If there are small clock differences between the host generating the SAS using the current time and the storage service, then it's possible for the storage service to receive a SAS that isn't yet valid.

  • Don't set a very short expiry time on a SAS. Again, small clock differences between the host generating the SAS and the storage service can lead to a SAS expiring earlier than anticipated.

  • Does the version parameter in the SAS key (for example, sv=2015-04-05) match the version of the Storage Client Library you're using? We recommend that you always use the latest version of the storage client library.

  • If you regenerate your storage access keys, any existing SAS tokens may be invalidated. This issue may arise if you generate SAS tokens with a long expiry time for client applications to cache.

If you're using the Storage Client Library to generate SAS tokens, then it's easy to build a valid token. However, if you're using the Storage REST API and constructing the SAS tokens by hand, see Delegating Access with a Shared Access Signature.

The client is receiving HTTP 404 (Not found) messages

If the client application receives an HTTP 404 (Not found) message from the server, this implies that the object the client was attempting to use (such as an entity, table, blob, container, or queue) doesn't exist in the storage service. There are a number of possible reasons for this, such as:

  • The client or another process previously deleted the object.

  • A Shared Access Signature (SAS) authorization issue.

  • Client-side JavaScript code doesn't have permission to access the object.

  • Network failure.

The client or another process previously deleted the object

In scenarios where the client is attempting to read, update, or delete data in a storage service, it's easy to identify in the storage resource logs a previous operation that deleted the object in question from the storage service. Often, the log data shows that another user or process deleted the object. The Azure Monitor logs (server-side) show when a client deleted an object.

In the scenario where a client is attempting to insert an object, it may not be immediately obvious why this results in an HTTP 404 (Not found) response, given that the client is creating a new object. However, if the client is creating a blob, it must be able to find the blob container. If the client is creating a message, it must be able to find a queue. And if the client is adding a row, it must be able to find the table.

You can use the client-side log from the Storage Client Library to better understand when the client sends specific requests to the storage service.

The following client-side log generated by the Storage Client library illustrates the problem when the client can't find the container for the blob it's creating. This log includes details of the following storage operations:

Request ID Operation
07b26a5d-... DeleteIfExists method to delete the blob container. This operation includes a HEAD request to check for the existence of the container.
e2d06d78… CreateIfNotExists method to create the blob container. This operation includes a HEAD request that checks for the existence of the container. The HEAD returns a 404 message but continues.
de8b1c3c-... UploadFromStream method to create the blob. The PUT request fails with a 404 message

Log entries:

Request ID Operation Text
07b26a5d-... Starting synchronous request to https://domemaildist.blob.core.windows.net/azuremmblobcontainer.
07b26a5d-... StringToSign = HEAD............x-ms-client-request-id:07b26a5d-....x-ms-date:Tue, 03 Jun 2014 10:33:11 GMT.x-ms-version:2014-02-14./domemaildist/azuremmblobcontainer.restype:container.
07b26a5d-... Waiting for response.
07b26a5d-... Response received. Status code = 200, Request ID = eeead849-...Content-MD5 = , ETag = &quot;0x8D14D2DC63D059B&quot;.
07b26a5d-... Response headers were processed successfully, proceeding with the rest of the operation.
07b26a5d-... Downloading response body.
07b26a5d-... Operation completed successfully.
07b26a5d-... Starting synchronous request to https://domemaildist.blob.core.windows.net/azuremmblobcontainer.
07b26a5d-... StringToSign = DELETE............x-ms-client-request-id:07b26a5d-....x-ms-date:Tue, 03 Jun 2014 10:33:12 GMT.x-ms-version:2014-02-14./domemaildist/azuremmblobcontainer.restype:container.
07b26a5d-... Waiting for response.
07b26a5d-... Response received. Status code = 202, Request ID = 6ab2a4cf-..., Content-MD5 = , ETag = .
07b26a5d-... Response headers were processed successfully, proceeding with the rest of the operation.
07b26a5d-... Downloading response body.
07b26a5d-... Operation completed successfully.
e2d06d78-... Starting asynchronous request to https://domemaildist.blob.core.windows.net/azuremmblobcontainer.
e2d06d78-... StringToSign = HEAD............x-ms-client-request-id:e2d06d78-....x-ms-date:Tue, 03 Jun 2014 10:33:12 GMT.x-ms-version:2014-02-14./domemaildist/azuremmblobcontainer.restype:container.
e2d06d78-... Waiting for response.
de8b1c3c-... Starting synchronous request to https://domemaildist.blob.core.windows.net/azuremmblobcontainer/blobCreated.txt.
de8b1c3c-... StringToSign = PUT...64.qCmF+TQLPhq/YYK50mP9ZQ==........x-ms-blob-type:BlockBlob.x-ms-client-request-id:de8b1c3c-....x-ms-date:Tue, 03 Jun 2014 10:33:12 GMT.x-ms-version:2014-02-14./domemaildist/azuremmblobcontainer/blobCreated.txt.
de8b1c3c-... Preparing to write request data.
e2d06d78-... Exception thrown while waiting for response: The remote server returned an error: (404) Not Found..
e2d06d78-... Response received. Status code = 404, Request ID = 353ae3bc-..., Content-MD5 = , ETag = .
e2d06d78-... Response headers were processed successfully, proceeding with the rest of the operation.
e2d06d78-... Downloading response body.
e2d06d78-... Operation completed successfully.
e2d06d78-... Starting asynchronous request to https://domemaildist.blob.core.windows.net/azuremmblobcontainer.
e2d06d78-... StringToSign = PUT...0.........x-ms-client-request-id:e2d06d78-....x-ms-date:Tue, 03 Jun 2014 10:33:12 GMT.x-ms-version:2014-02-14./domemaildist/azuremmblobcontainer.restype:container.
e2d06d78-... Waiting for response.
de8b1c3c-... Writing request data.
de8b1c3c-... Waiting for response.
e2d06d78-... Exception thrown while waiting for response: The remote server returned an error: (409) Conflict..
e2d06d78-... Response received. Status code = 409, Request ID = c27da20e-..., Content-MD5 = , ETag = .
e2d06d78-... Downloading error response body.
de8b1c3c-... Exception thrown while waiting for response: The remote server returned an error: (404) Not Found..
de8b1c3c-... Response received. Status code = 404, Request ID = 0eaeab3e-..., Content-MD5 = , ETag = .
de8b1c3c-... Exception thrown during the operation: The remote server returned an error: (404) Not Found..
de8b1c3c-... Retry policy did not allow for a retry. Failing with The remote server returned an error: (404) Not Found..
e2d06d78-... Retry policy did not allow for a retry. Failing with The remote server returned an error: (409) Conflict..

In this example, the log shows that the client is interleaving requests from the CreateIfNotExists method (request ID e2d06d78…) with the requests from the UploadFromStream method (de8b1c3c-...). This interleaving happens because the client application is invoking these methods asynchronously. Modify the asynchronous code in the client to ensure that it creates the container before attempting to upload any data to a blob in that container. Ideally, you should create all your containers in advance.

A Shared Access Signature (SAS) authorization issue

If the client application attempts to use a SAS key that doesn't include the necessary permissions for the operation, the storage service returns an HTTP 404 (Not found) message to the client. At the same time, in Azure Monitor metrics, you'll also see an AuthorizationError for the ResponseType dimension.

Investigate why your client application is attempting to perform an operation for which it hasn't been granted permissions.

Client-side JavaScript code doesn't have permission to access the object

If you're using a JavaScript client and the storage service is returning HTTP 404 messages, check for the following JavaScript errors in the browser:

SEC7120: Origin http://localhost:56309 not found in Access-Control-Allow-Origin header.
SCRIPT7002: XMLHttpRequest: Network Error 0x80070005, Access is denied.

Note

You can use the F12 Developer Tools in Internet Explorer to trace the messages exchanged between the browser and the storage service when you're troubleshooting client-side JavaScript issues.

These errors occur because the web browser implements the same origin policy security restriction that prevents a web page from calling an API in a different domain from the domain the page comes from.

To work around the JavaScript issue, you can configure Cross-Origin Resource Sharing (CORS) for the storage service the client is accessing. For more information, see Cross-Origin Resource Sharing (CORS) Support for Azure Storage Services.

The following code sample shows how to configure your blob service to allow JavaScript running in the Contoso domain to access a blob in your blob storage service:

var connectionString = Constants.connectionString;

 BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);

 BlobServiceProperties sp = blobServiceClient.GetProperties();

 // Set the service properties.
 sp.DefaultServiceVersion = "2013-08-15";
 BlobCorsRule bcr = new BlobCorsRule();
 bcr.AllowedHeaders = "*";

 bcr.AllowedMethods = "GET,POST";
 bcr.AllowedOrigins = "http://www.contoso.com";
 bcr.ExposedHeaders = "x-ms-*";
 bcr.MaxAgeInSeconds = 5;
 sp.Cors.Clear();
 sp.Cors.Add(bcr);
 blobServiceClient.SetProperties(sp);

Network failure

In some circumstances, lost network packets can lead to the storage service returning HTTP 404 messages to the client. For example, when your client application is deleting an entity from the table service, you see the client throw a storage exception reporting an "HTTP 404 (Not Found)" status message from the table service. When you investigate the table in the table storage service, you see that the service did delete the entity as requested.

The exception details in the client include the request ID (7e84f12d…) assigned by the table service for the request: you can use this information to locate the request details in the storage resource logs in Azure Monitor by searching in Fields that describe how the operation was authenticated of log entries. You could also use the metrics to identify when failures such as this occur and then search the log files based on the time the metrics recorded this error. This log entry shows that the delete failed with an "HTTP (404) Client Other Error" status message. The same log entry also includes the request ID generated by the client in the client-request-id column (813ea74f…).

The server-side log also includes another entry with the same client-request-id value (813ea74f…) for a successful delete operation for the same entity and from the same client. This successful delete operation took place shortly before the failed delete request.

The most likely cause of this scenario is that the client sent a delete request for the entity to the table service, which succeeded but didn't receive an acknowledgment from the server (perhaps due to a temporary network issue). The client then automatically retried the operation (using the same client-request-id), and this retry failed because the entity had already been deleted.

If this problem occurs frequently, you should investigate why the client is failing to receive acknowledgments from the table service. If the problem is intermittent, you should trap the "HTTP (404) Not Found" error and log it in the client, but allow the client to continue.

The client is receiving HTTP 409 (Conflict) messages

When a client deletes blob containers, tables, or queues, there's a brief period before the name becomes available again. If the code in your client application deletes and then immediately recreates a blob container using the same name, the CreateIfNotExists method eventually fails with the HTTP 409 (Conflict) error.

The client application should use unique container names whenever it creates new containers if the delete/recreate pattern is common.

Metrics show low PercentSuccess or analytics log entries have operations with transaction status of ClientOtherErrors

A ResponseType dimension equal to a value of Success captures the percent of operations that were successful based on their HTTP Status Code. Operations with status codes of 2XX count as successful, whereas operations with status codes in 3XX, 4XX, and 5XX ranges are counted as unsuccessful and lower the Success metric value. In storage resource logs, these operations are recorded with a transaction status of ClientOtherError.

These operations have completed successfully and therefore don't affect other metrics, such as availability. Some examples of operations that execute successfully but that can result in unsuccessful HTTP status codes include:

  • ResourceNotFound (Not Found 404), for example, from a GET request to a blob that doesn't exist.
  • ResourceAlreadyExists (Conflict 409), for example, from a CreateIfNotExist operation where the resource already exists.
  • ConditionNotMet (Not Modified 304), for example, from a conditional operation such as when a client sends an ETag value and an HTTP If-None-Match header to request an image only if it has been updated since the last operation.

You can find a list of common REST API error codes that the storage services return on the page Common REST API Error Codes.

See also

Contact us for help

If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.