Edit

Call the Image Analysis 4.0 Analyze API (preview)

  • Article

This article demonstrates how to call the Image Analysis 4.0 API to return information about an image's visual features. It also shows you how to parse the returned information.

Prerequisites

This guide assumes you have successfully followed the steps mentioned in the quickstart page. This means:

  • You have created a Computer Vision resource and obtained a key and endpoint URL.
  • If you're using the client SDK, you have the appropriate SDK package installed and you have a running quickstart application. You can modify this quickstart application based on code examples here.
  • If you're using 4.0 REST API calls directly, you have successfully made a curl.exe call to the service (or used an alternative tool). You modify the curl.exe call based on the examples here.

Authenticate against the service

To authenticate against the Image Analysis service, you need a Computer Vision key and endpoint URL.

Tip

Don't include the key directly in your code, and never post it publicly. See the Cognitive Services security article for more authentication options like Azure Key Vault.

The SDK example assumes that you defined the environment variables VISION_KEY and VISION_ENDPOINT with your key and endpoint.

Start by creating a VisionServiceOptions object using one of the constructors. For example:

var serviceOptions = new VisionServiceOptions(
    Environment.GetEnvironmentVariable("VISION_ENDPOINT"),
    new AzureKeyCredential(Environment.GetEnvironmentVariable("VISION_KEY")));

Select the image to analyze

The code in this guide uses remote images referenced by URL. You may want to try different images on your own to see the full capability of the Image Analysis features.

Create a new VisionSource object from the URL of the image you want to analyze, using the static constructor VisionSource.FromUrl.

VisionSource implements IDisposable, therefore create the object with a using statement or explicitly call Dispose method after analysis completes.

using var imageSource = VisionSource.FromUrl(
    new Uri("https://learn.microsoft.com/azure/cognitive-services/computer-vision/media/quickstarts/presentation.png"));

Tip

You can also analyze a local image by passing in the full-path image file name. See VisionSource.FromFile.

Select analysis options

Select visual features when using the standard model

The Analysis 4.0 API gives you access to all of the service's image analysis features. Choose which operations to do based on your own use case. See the overview for a description of each feature. The example in this section adds all of the available visual features, but for practical usage you likely need fewer.

Visual features 'Captions' and 'DenseCaptions' are only supported in the following Azure regions: East US, France Central, Korea Central, North Europe, Southeast Asia, West Europe, West US.

Note

The REST API uses the terms Smart Crops and Smart Crops Aspect Ratios. The SDK uses the terms Crop Suggestions and Cropping Aspect Ratios. They both refer to the same service operation. Similarly, the REST API users the term Read for detecting text in the image, whereas the SDK uses the term Text for the same operation.

Create a new ImageAnalysisOptions object and specify the visual features you'd like to extract, by setting the Features property. ImageAnalysisFeature enum defines the supported values.

var analysisOptions = new ImageAnalysisOptions()
{
    Features =
          ImageAnalysisFeature.CropSuggestions
        | ImageAnalysisFeature.Caption
        | ImageAnalysisFeature.DenseCaptions
        | ImageAnalysisFeature.Objects
        | ImageAnalysisFeature.People
        | ImageAnalysisFeature.Text
        | ImageAnalysisFeature.Tags
};

Set model name when using a custom model

You can also do image analysis with a custom trained model. To create and train a model, see Create a custom Image Analysis model. Once your model is trained, all you need is the model's name. You do not need to specify visual features if you use a custom model.

To use a custom model, create the ImageAnalysisOptions object and set the ModelName property. You don't need to set any other properties on ImageAnalysisOptions. There's no need to set the Features property, as you do with the standard model, since your custom model already implies the visual features the service extracts.

var analysisOptions = new ImageAnalysisOptions()
{
    ModelName = "MyCustomModelName"
};

Specify languages

You can specify the language of the returned data. The language is optional, with the default being English. See Language support for a list of supported language codes and which visual features are supported for each language.

Language option only applies when you're using the standard model.

Use the Language property of your ImageAnalysisOptions object to specify a language.

analysisOptions.Language = "en";

Select gender neutral captions

If you're extracting captions or dense captions, you can ask for gender neutral captions. Gender neutral captions is optional, with the default being gendered captions. For example, in English, when you select gender neutral captions, terms like woman or man are replaced with person, and boy or girl are replaced with child.

Gender neutral caption option only applies when you're using the standard model.

Set the GenderNeutralCaption property of your ImageAnalysisOptions object to true to enable gender neutral captions.

analysisOptions.GenderNeutralCaption = true;

Select smart cropping aspect ratios

An aspect ratio is calculated by dividing the target crop width by the height. Supported values are from 0.75 to 1.8 (inclusive). Setting this property is only relevant when the smartCrop option (REST API) or CropSuggestions (SDK) was selected as part the visual feature list. If you select smartCrop/CropSuggestions but don't specify aspect ratios, the service returns one crop suggestion with an aspect ratio it sees fit. In this case, the aspect ratio is between 0.5 and 2.0 (inclusive).

Smart cropping aspect rations only applies when you're using the standard model.

Set the CroppingAspectRatios property of your ImageAnalysisOptions to a list of aspect ratios. For example, to set aspect ratios of 0.9 and 1.33:

analysisOptions.CroppingAspectRatios = new List<double>() { 0.9, 1.33 };

Get results from the service

Get results using the standard model

This section shows you how to make an analysis call to the service using the standard model, and get the results.

  1. Using the VisionServiceOptions, VisionSource and ImageAnalysisOptions objects, construct a new ImageAnalyzer object. ImageAnalyzer implements IDisposable, therefore create the object with a using statement, or explicitly call Dispose method after analysis completes.

  2. Call the Analyze method on the ImageAnalyzer object, as shown here. This is a blocking (synchronous) call until the service returns the results or an error occurred. Alternatively, you can call the nonblocking AnalyzeAsync method.

  3. Check the Reason property on the ImageAnalysisResult object, to determine if analysis succeeded or failed.

  4. If succeeded, proceed to access the relevant result properties based on your selected visual features, as shown here. Additional information (not commonly needed) can be obtained by constructing the ImageAnalysisResultDetails object.

  5. If failed, you can construct the ImageAnalysisErrorDetails object to get information on the failure.

using var analyzer = new ImageAnalyzer(serviceOptions, imageSource, analysisOptions);

var result = analyzer.Analyze();

if (result.Reason == ImageAnalysisResultReason.Analyzed)
{
    Console.WriteLine($" Image height = {result.ImageHeight}");
    Console.WriteLine($" Image width = {result.ImageWidth}");
    Console.WriteLine($" Model version = {result.ModelVersion}");

    if (result.Caption != null)
    {
        Console.WriteLine(" Caption:");
        Console.WriteLine($"   \"{result.Caption.Content}\", Confidence {result.Caption.Confidence:0.0000}");
    }

    if (result.DenseCaptions != null)
    {
        Console.WriteLine(" Dense Captions:");
        foreach (var caption in result.DenseCaptions)
        {
            Console.WriteLine($"   \"{caption.Content}\", Bounding box {caption.BoundingBox}, Confidence {caption.Confidence:0.0000}");
        }
    }

    if (result.Objects != null)
    {
        Console.WriteLine(" Objects:");
        foreach (var detectedObject in result.Objects)
        {
            Console.WriteLine($"   \"{detectedObject.Name}\", Bounding box {detectedObject.BoundingBox}, Confidence {detectedObject.Confidence:0.0000}");
        }
    }

    if (result.Tags != null)
    {
        Console.WriteLine($" Tags:");
        foreach (var tag in result.Tags)
        {
            Console.WriteLine($"   \"{tag.Name}\", Confidence {tag.Confidence:0.0000}");
        }
    }

    if (result.People != null)
    {
        Console.WriteLine($" People:");
        foreach (var person in result.People)
        {
            Console.WriteLine($"   Bounding box {person.BoundingBox}, Confidence {person.Confidence:0.0000}");
        }
    }

    if (result.CropSuggestions != null)
    {
        Console.WriteLine($" Crop Suggestions:");
        foreach (var cropSuggestion in result.CropSuggestions)
        {
            Console.WriteLine($"   Aspect ratio {cropSuggestion.AspectRatio}: "
                + $"Crop suggestion {cropSuggestion.BoundingBox}");
        };
    }

    if (result.Text != null)
    {
        Console.WriteLine($" Text:");
        foreach (var line in result.Text.Lines)
        {
            string pointsToString = "{" + string.Join(',', line.BoundingPolygon.Select(pointsToString => pointsToString.ToString())) + "}";
            Console.WriteLine($"   Line: '{line.Content}', Bounding polygon {pointsToString}");

            foreach (var word in line.Words)
            {
                pointsToString = "{" + string.Join(',', word.BoundingPolygon.Select(pointsToString => pointsToString.ToString())) + "}";
                Console.WriteLine($"     Word: '{word.Content}', Bounding polygon {pointsToString}, Confidence {word.Confidence:0.0000}");
            }
        }
    }

    var resultDetails = ImageAnalysisResultDetails.FromResult(result);
    Console.WriteLine($" Result details:");
    Console.WriteLine($"   Image ID = {resultDetails.ImageId}");
    Console.WriteLine($"   Result ID = {resultDetails.ResultId}");
    Console.WriteLine($"   Connection URL = {resultDetails.ConnectionUrl}");
    Console.WriteLine($"   JSON result = {resultDetails.JsonResult}");
}
else
{
    var errorDetails = ImageAnalysisErrorDetails.FromResult(result);
    Console.WriteLine(" Analysis failed.");
    Console.WriteLine($"   Error reason : {errorDetails.Reason}");
    Console.WriteLine($"   Error code : {errorDetails.ErrorCode}");
    Console.WriteLine($"   Error message: {errorDetails.Message}");
}

Get results using custom model

This section shows you how to make an analysis call to the service, when using a custom model.

The code is similar to the standard model case. The only difference is that results from the custom model are available on the CustomTags and/or CustomObjects properties of the ImageAnalysisResult object.

using var analyzer = new ImageAnalyzer(serviceOptions, imageSource, analysisOptions);

var result = analyzer.Analyze();

if (result.Reason == ImageAnalysisResultReason.Analyzed)
{
    if (result.CustomObjects != null)
    {
        Console.WriteLine(" Custom Objects:");
        foreach (var detectedObject in result.CustomObjects)
        {
            Console.WriteLine($"   \"{detectedObject.Name}\", Bounding box {detectedObject.BoundingBox}, Confidence {detectedObject.Confidence:0.0000}");
        }
    }

    if (result.CustomTags != null)
    {
        Console.WriteLine($" Custom Tags:");
        foreach (var tag in result.CustomTags)
        {
            Console.WriteLine($"   \"{tag.Name}\", Confidence {tag.Confidence:0.0000}");
        }
    }
}
else
{
    var errorDetails = ImageAnalysisErrorDetails.FromResult(result);
    Console.WriteLine(" Analysis failed.");
    Console.WriteLine($"   Error reason : {errorDetails.Reason}");
    Console.WriteLine($"   Error code : {errorDetails.ErrorCode}");
    Console.WriteLine($"   Error message: {errorDetails.Message}");
}

Error codes

The sample code for getting analysis results shows how to handle errors and get the ImageAnalysisErrorDetails object that contains the error information. The error information includes:

  • Error reason. See enum ImageAnalysisErrorReason.
  • Error code and error message. Click on the REST API tab to see a list of some common error codes and messages.

In addition to those errors, the SDK has a few other error messages, including:

  • Missing Image Analysis options: You must set at least one visual feature (or model name) for the 'analyze' operation. Or set segmentation mode for the 'segment' operation
  • Invalid combination of Image Analysis options: You cannot set both visual features (or model name), and segmentation mode

Make sure the ImageAnalysisOptions object is set correctly to fix these errors.

To help resolve issues, look at the Image Analysis Samples repository and run the closest sample to your scenario. Search the GitHub issues to see if your issue was already address. If not, create a new.

Next steps