Share via

Azure Cognitive Services Computer Vision SDK for Python

  • Article

The Computer Vision service provides developers with access to advanced algorithms for processing images and returning information. Computer Vision algorithms analyze the content of an image in different ways, depending on the visual features you're interested in.

You can use Computer Vision in your application to:

  • Analyze images for insight
  • Extract text from images
  • Generate thumbnails

Looking for more documentation?

Prerequisites

If you need a Computer Vision API account, you can create one with this Azure CLI command:

RES_REGION=westeurope
RES_GROUP=<resourcegroup-name>
ACCT_NAME=<computervision-account-name>

az cognitiveservices account create \
    --resource-group $RES_GROUP \
    --name $ACCT_NAME \
    --location $RES_REGION \
    --kind ComputerVision \
    --sku S1 \
    --yes

Installation

Install the Azure Cognitive Services Computer Vision SDK with pip, optionally within a virtual environment.

Configure a virtual environment (optional)

Although not required, you can keep your base system and Azure SDK environments isolated from one another if you use a virtual environment. Execute the following commands to configure and then enter a virtual environment with venv, such as cogsrv-vision-env:

python3 -m venv cogsrv-vision-env
source cogsrv-vision-env/bin/activate

Install the SDK

Install the Azure Cognitive Services Computer Vision SDK for Python package with pip:

pip install azure-cognitiveservices-vision-computervision

Authentication

Once you create your Computer Vision resource, you need its region, and one of its account keys to instantiate the client object.

Use these values when you create the instance of the ComputerVisionClient client object.

Get credentials

Use the Azure CLI snippet below to populate two environment variables with the Computer Vision account region and one of its keys (you can also find these values in the Azure portal). The snippet is formatted for the Bash shell.

RES_GROUP=<resourcegroup-name>
ACCT_NAME=<computervision-account-name>

export ACCOUNT_REGION=$(az cognitiveservices account show \
    --resource-group $RES_GROUP \
    --name $ACCT_NAME \
    --query location \
    --output tsv)

export ACCOUNT_KEY=$(az cognitiveservices account keys list \
    --resource-group $RES_GROUP \
    --name $ACCT_NAME \
    --query key1 \
    --output tsv)

Create client

Once you've populated the ACCOUNT_REGION and ACCOUNT_KEY environment variables, you can create the ComputerVisionClient client object.

from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from azure.cognitiveservices.vision.computervision.models import VisualFeatureTypes
from msrest.authentication import CognitiveServicesCredentials

import os
region = os.environ['ACCOUNT_REGION']
key = os.environ['ACCOUNT_KEY']

credentials = CognitiveServicesCredentials(key)
client = ComputerVisionClient(
    endpoint="https://" + region + ".api.cognitive.microsoft.com/",
    credentials=credentials
)

Usage

Once you've initialized a ComputerVisionClient client object, you can:

  • Analyze an image: You can analyze an image for certain features such as faces, colors, tags.
  • Generate thumbnails: Create a custom JPEG image to use as a thumbnail of the original image.
  • Get description of an image: Get a description of the image based on its subject domain.

For more information about this service, see What is Computer Vision?.

Examples

The following sections provide several code snippets covering some of the most common Computer Vision tasks, including:

Analyze an image

You can analyze an image for certain features with analyze_image. Use the visual_features property to set the types of analysis to perform on the image. Common values are VisualFeatureTypes.tags and VisualFeatureTypes.description.

url = "https://upload.wikimedia.org/wikipedia/commons/thumb/1/12/Broadway_and_Times_Square_by_night.jpg/450px-Broadway_and_Times_Square_by_night.jpg"

image_analysis = client.analyze_image(url,visual_features=[VisualFeatureTypes.tags])

for tag in image_analysis.tags:
    print(tag)

Get subject domain list

Review the subject domains used to analyze your image with list_models. These domain names are used when analyzing an image by domain. An example of a domain is landmarks.

models = client.list_models()

for x in models.models_property:
    print(x)

Analyze an image by domain

You can analyze an image by subject domain with analyze_image_by_domain. Get the list of supported subject domains in order to use the correct domain name.

domain = "landmarks"
url = "https://images.pexels.com/photos/338515/pexels-photo-338515.jpeg"
language = "en"

analysis = client.analyze_image_by_domain(domain, url, language)

for landmark in analysis.result["landmarks"]:
    print(landmark["name"])
    print(landmark["confidence"])

Get text description of an image

You can get a language-based text description of an image with describe_image. Request several descriptions with the max_description property if you are doing text analysis for keywords associated with the image. Examples of a text description for the following image include a train crossing a bridge over a body of water, a large bridge over a body of water, and a train crossing a bridge over a large body of water.

domain = "landmarks"
url = "http://www.public-domain-photos.com/free-stock-photos-4/travel/san-francisco/golden-gate-bridge-in-san-francisco.jpg"
language = "en"
max_descriptions = 3

analysis = client.describe_image(url, max_descriptions, language)

for caption in analysis.captions:
    print(caption.text)
    print(caption.confidence)

Get text from image

You can get any handwritten or printed text from an image. This requires two calls to the SDK: read and get_read_result. The call to read is asynchronous. In the results of the get_read_result call, you need to check if the first call completed with OperationStatusCodes before extracting the text data. The results include the text as well as the bounding box coordinates for the text.

# import models
from azure.cognitiveservices.vision.computervision.models import OperationStatusCodes

url = "https://github.com/Azure-Samples/cognitive-services-python-sdk-samples/raw/master/samples/vision/images/make_things_happen.jpg"
raw = True
numberOfCharsInOperationId = 36

# SDK call
rawHttpResponse = client.read(url, language="en", raw=True)

# Get ID from returned headers
operationLocation = rawHttpResponse.headers["Operation-Location"]
idLocation = len(operationLocation) - numberOfCharsInOperationId
operationId = operationLocation[idLocation:]

# SDK call
result = client.get_read_result(operationId)

# Get data
if result.status == OperationStatusCodes.succeeded:

    for line in result.analyze_result.read_results[0].lines:
        print(line.text)
        print(line.bounding_box)

Generate thumbnail

You can generate a thumbnail (JPG) of an image with generate_thumbnail. The thumbnail does not need to be in the same proportions as the original image.

This example uses the Pillow package to save the new thumbnail image locally.

from PIL import Image
import io

width = 50
height = 50
url = "http://www.public-domain-photos.com/free-stock-photos-4/travel/san-francisco/golden-gate-bridge-in-san-francisco.jpg"

thumbnail = client.generate_thumbnail(width, height, url)

for x in thumbnail:
    image = Image.open(io.BytesIO(x))

image.save('thumbnail.jpg')

Troubleshooting

General

When you interact with the ComputerVisionClient client object using the Python SDK, the ComputerVisionErrorException class is used to return errors. Errors returned by the service correspond to the same HTTP status codes returned for REST API requests.

For example, if you try to analyze an image with an invalid key, a 401 error is returned. In the following snippet, the error is handled gracefully by catching the exception and displaying additional information about the error.


domain = "landmarks"
url = "http://www.public-domain-photos.com/free-stock-photos-4/travel/san-francisco/golden-gate-bridge-in-san-francisco.jpg"
language = "en"
max_descriptions = 3

try:
    analysis = client.describe_image(url, max_descriptions, language)

    for caption in analysis.captions:
        print(caption.text)
        print(caption.confidence)
except HTTPFailure as e:
    if e.status_code == 401:
        print("Error unauthorized. Make sure your key and region are correct.")
    else:
        raise

Handle transient errors with retries

While working with the ComputerVisionClient client, you might encounter transient failures caused by rate limits enforced by the service, or other transient problems like network outages. For information about handling these types of failures, see Retry pattern in the Cloud Design Patterns guide, and the related Circuit Breaker pattern.

Next steps

More sample code

Several Computer Vision Python SDK samples are available to you in the SDK's GitHub repository. These samples provide example code for additional scenarios commonly encountered while working with Computer Vision:

Additional documentation

For more extensive documentation on the Computer Vision service, see the Azure Computer Vision documentation on docs.microsoft.com.