Product Recognition (version 4.0 preview)
Important
This feature is now deprecated. On January 10, 2025, Azure AI Image Analysis 4.0 Custom Image Classification, Custom Object Detection, and Product Recognition preview API will be retired. After this date, API calls to these services will fail.
To maintain a smooth operation of your models, transition to Azure AI Custom Vision, which is now generally available. Custom Vision offers similar functionality to these retiring features.
The Product Recognition APIs let you analyze photos of shelves in a retail store. You can detect the presence of products and get their bounding box coordinates. Use it in combination with model customization to train a model to identify your specific products. You can also compare Product Recognition results to your store's planogram document.
Try out the capabilities of Product Recognition quickly and easily in your browser using Vision Studio.
Note
The brands shown in the images are not affiliated with Microsoft and do not indicate any form of endorsement of Microsoft or Microsoft products by the brand owners, or an endorsement of the brand owners or their products by Microsoft.
Important
You can train a custom model for product recognition using either the Custom Vision service or the Image Analysis 4.0 Product Recognition APIs. The following table compares the two services.
| Areas | Products on Shelves – Custom Vision | Product Recognition – Image Analysis API/Customization | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Features | Custom product understanding | Image stitching & rectification, Pretrained product understanding, Custom product understanding, Planogram matching |
||||||||||||||||||||||||||||
| Base model | CNN | Florence transformer model | ||||||||||||||||||||||||||||
| Labeling | Customvision.ai | AML Studio | ||||||||||||||||||||||||||||
| Web Portal | Customvision.ai | Vision Studio | ||||||||||||||||||||||||||||
| Libraries | REST, SDK | REST, Python Sample | ||||||||||||||||||||||||||||
| Minimum training data needed | 15 images per category | 2-5 images per category | ||||||||||||||||||||||||||||
| Training data storage | Uploaded to service | Customer’s blob storage account | ||||||||||||||||||||||||||||
| Model hosting | Cloud and edge | Cloud hosting only, edge container hosting to come | ||||||||||||||||||||||||||||
| AI quality |
|
|
||||||||||||||||||||||||||||
| Pricing | Custom Vision pricing | Image Analysis pricing |
Product Recognition features
Shelf image composition
The stitching and rectification APIs let you modify images to improve the accuracy of the Product Understanding results. You can use these APIs to:
- Stitch together multiple images of a shelf to create a single image.
- Rectify an image to remove perspective distortion.
Shelf product recognition (pretrained model)
The Product Understanding API lets you analyze a shelf image using the out-of-box pretrained model. This operation detects products and gaps in the shelf image and returns the bounding box coordinates of each product and gap, along with a confidence score for each.
The following JSON response illustrates what the Product Understanding API returns.
{
"imageMetadata": {
"width": 2000,
"height": 1500
},
"products": [
{
"id": "string",
"boundingBox": {
"x": 1234,
"y": 1234,
"w": 12,
"h": 12
},
"classifications": [
{
"confidence": 0.9,
"label": "string"
}
]
}
],
"gaps": [
{
"id": "string",
"boundingBox": {
"x": 1234,
"y": 1234,
"w": 123,
"h": 123
},
"classifications": [
{
"confidence": 0.8,
"label": "string"
}
]
}
]
}
Shelf product recognition (customized model)
The Product Understanding API can also be used with a custom trained model to detect your specific products. This operation returns the bounding box coordinates of each product and gap, along with the label of each product.
The following JSON response illustrates what the Product Understanding API returns when used with a custom model.
"detectedProducts": {
"imageMetadata": {
"width": 21,
"height": 25
},
"products": [
{
"id": "01",
"boundingBox": {
"x": 123,
"y": 234,
"w": 34,
"h": 45
},
"classifications": [
{
"confidence": 0.8,
"label": "Product1"
}
]
}
],
"gaps": [
{
"id": "02",
"boundingBox": {
"x": 12,
"y": 123,
"w": 1234,
"h": 123
},
"classifications": [
{
"confidence": 0.9,
"label": "Product1"
}
]
}
]
}
Shelf planogram compliance
The Planogram matching API lets you compare the results of the Product Understanding API to a planogram document. This operation matches each detected product and gap to its corresponding position in the planogram document.
It returns a JSON response that accounts for each position in the planogram document, whether it's occupied by a product or gap.
{
"matchedResultsPerPosition": [
{
"positionId": "01",
"detectedObject": {
"id": "01",
"boundingBox": {
"x": 12,
"y": 1234,
"w": 123,
"h": 12345
},
"classifications": [
{
"confidence": 0.9,
"label": "Product1"
}
]
}
}
]
}
Limitations
- Product Recognition is only available in certain Azure regions.
- Shelf images can be up to 20 MB in size. The recommended size is 4 MB.
- We recommend you do stitching and rectification on the shelf images before uploading them for analysis.
- Using a custom model is optional in Product Recognition, but it's required for the planogram matching function.
Next steps
Get started with Product Recognition by trying out the stitching and rectification APIs. Then do basic analysis with the Product Understanding API.