Remove the background in images

Important

This feature is now deprecated. On January 10, 2025, the Azure AI Image Analysis 4.0 Segment API and background removal service will be retired. All requests to this service will fail after this date.

To maintain a smooth operation of your models, install the open-source Florence 2 model and use its Region to segmentation feature, which allows for a similar background removal operation.

This article demonstrates how to call the Image Analysis 4.0 API to segment an image (separate the foreground from background). It also shows you how to parse the returned information.

Important

Background removal is only available through direct REST API calls. It is not available through the SDKs.

Prerequisites

This guide assumes you have successfully followed the steps mentioned in the quickstart page. This means:

• You have created a Vision resource and obtained a key and endpoint URL.
• You have successfully made a curl.exe call to the service (or used an alternative tool). You modify the curl.exe call based on the examples here.

The quickstart shows you how to extract visual features from an image. However, the concepts are similar to background removal. Therefore you benefit from starting with the quickstart and making modifications.

Important

Background removal is only available in certain Azure regions. See Region availability

Authenticate against the service

To authenticate against the Image Analysis service, you need a Computer Vision key and endpoint URL.

Tip

Don't include the key directly in your code, and never post it publicly. See the Azure AI services security article for more authentication options like Azure Key Vault.

Authentication is done by adding the HTTP request header Ocp-Apim-Subscription-Key and setting it to your vision key. The call is made to the URL <endpoint>/computervision/imageanalysis:segment?api-version=2023-02-01-preview, where <endpoint> is your unique Computer Vision endpoint URL. See Select a mode section for another query string you add to this URL.

Select the image to analyze

The code in this guide uses remote images referenced by URL. You may want to try different images on your own to see the full capability of the Image Analysis features.

When analyzing a remote image, you specify the image's URL by formatting the request body like this: {"url":"https://learn.microsoft.com/azure/ai-services/computer-vision/images/windows-kitchen.jpg"}. The Content-Type should be application/json.

To analyze a local image, you'd put the binary image data in the HTTP request body. The Content-Type should be application/octet-stream or multipart/form-data.

Select a mode

Set the query string mode to one of these two values. This query string is mandatory if you want to do image segmentation.

URL parameter	Value	Description
mode	backgroundRemoval	Outputs an image of the detected foreground object with a transparent background.
mode	foregroundMatting	Outputs a gray-scale alpha matte image showing the opacity of the detected foreground object.

A populated URL for backgroundRemoval would look like this: <endpoint>/computervision/imageanalysis:segment?api-version=2023-02-01-preview&mode=backgroundRemoval

Get results from the service

This section shows you how to make the API call and parse the results.

The service returns a 200 HTTP response on success with Content-Type: image/png, and the body contains the returned PNG image in the form of a binary stream.

As an example, assume background removal is run on the following image:

On a successful background removal call, The following four-channel PNG image is the response for the backgroundRemoval mode:

The following one-channel PNG image is the response for the foregroundMatting mode:

The API returns an image the same size as the original for the foregroundMatting mode, but at most 16 megapixels (preserving image aspect ratio) for the backgroundRemoval mode.

Error codes

On error, the Image Analysis service response contains a JSON payload that includes an error code and error message. It may also include other details in the form of and inner error code and message. For example:

{
    "error":
    {
        "code": "InvalidRequest",
        "message": "Analyze query is invalid.",
        "innererror":
        {
            "code": "NotSupportedVisualFeature",
            "message": "Specified feature type is not valid"
        }
    }
}

Following is a list of common errors and their causes. List items are presented in the following format:

• HTTP response code
  • Error code and message in the JSON response
    • [Optional] Inner error code and message in the JSON response

List of common errors:

• 400 Bad Request
  • InvalidRequest - Image URL is badly formatted or not accessible. Make sure the image URL is valid and publicly accessible.
  • InvalidRequest - The image size is not allowed to be zero or larger than 20971520 bytes. Reduce the size of the image by compressing it and/or resizing, and resubmit your request.
  • InvalidRequest - The feature 'Caption' is not supported in this region. The feature is only supported in specific Azure regions. See Quickstart prerequisites for the list of supported Azure regions.
  • InvalidRequest - The provided image content type ... is not supported. The HTTP header Content-Type in the request isn't an allowed type:
    • For an image URL, Content-Type should be application/json
    • For a binary image data, Content-Type should be application/octet-stream or multipart/form-data
  • InvalidRequest - Either 'features' or 'model-name' needs to be specified in the query parameter.
  • InvalidRequest - Image format is not valid
    • InvalidImageFormat - Image format is not valid. See the Image requirements section for supported image formats.
  • InvalidRequest - Analyze query is invalid
    • NotSupportedVisualFeature - Specified feature type is not valid. Make sure the features query string has a valid value.
    • NotSupportedLanguage - The input language is not supported. Make sure the language query string has a valid value for the selected visual feature, based on the following table.
    • BadArgument - 'smartcrops-aspect-ratios' aspect ratio is not in allowed range [0.75 to 1.8]
• 401 PermissionDenied
  • 401 - Access denied due to invalid subscription key or wrong API endpoint. Make sure to provide a valid key for an active subscription and use a correct regional API endpoint for your resource.
• 404 Resource Not Found
  • 404 - Resource not found. The service couldn't find the custom model based on the name provided by the model-name query string.

Tip

While working with Azure AI Vision, you might encounter transient failures caused by rate limits enforced by the service, or other transient problems like network outages. For information about handling these types of failures, see Retry pattern in the Cloud Design Patterns guide, and the related Circuit Breaker pattern.

Next steps

Background removal concepts