Azure Computer Vision Cognitive Services - how to perform OCR on image captured via getUserMedia camera stream (JavaScript)

asked 2020-08-13T14:49:14.123+00:00
Jason Brown 1 Reputation point

Hello!

Am using the Computer Vision Cognitive Services (JavaScript) to build a web app where the user can use the device camera to take an image and have OCR performed on it.

Previously I used the JavaScript Tesseract library (https://tesseract.projectnaptha.com/) to do this and used the following code:

Open camera stream

var video = document.getElementById('video');
        if (navigator.mediaDevices && navigator.mediaDevices.getUserMedia) {
            const constraints = {
                "video": {
                    "facingMode": { "ideal": "environment" }
                }
            };
            navigator.mediaDevices.getUserMedia(constraints)
                .then(function(stream) {
                    video.srcObject = stream;
                    video.play();
                })
                .catch(function(err) {
                    unsupportedBrowser(); //custom function I wrote to execute if the browser didn't support camera stream
                });
        } else {
            unsupportedBrowser();
        }

Press capture button

function capture() {
    var canvas = document.getElementById('canvas');
    var context = canvas.getContext('2d');
    context.drawImage(video, 0, 0, 640, 480);
    let tesseractSettings = {
        lang: 'eng'
    };
    Tesseract.recognize(context, tesseractSettings).then(function(result) {
        var scannedText = result.text ? result.text.trim() : '';
        closeScanningUi(scannedText); //text from image scanned with Tesseract would be displayed in another UI
    }, false);
}

Essentially, a still from the camera stream would be taken when the user pressed the 'capture' button and then Tesseract would perform the OCR on it.

I want to use the Computer Vision Cognitive Service instead of Tesseract now because it's more accurate and works on a much wider variety of documents etc.

I've had a look at the tutorial here: https://github.com/Azure-Samples/cognitive-services-javascript-computer-vision-tutorial and have successfully gotten this example to work, but it requires the URL of an image to perform the OCR.

Is there a way to get Computer Vision Cognitive Services to perform the OCR on the still that is captured from my camera stream instead of from an image located at a specific URL?

Cheers!

Azure Cognitive Services
Azure Cognitive Services
A group of Azure artificial intelligence services and cognitive APIs that help build intelligent apps.
939 questions
Azure Computer Vision
Azure Computer Vision
An Azure artificial intelligence service that analyzes content in images and video.
180 questions
{count} votes

1 answer

Sort by: Most helpful
  1. answered 2020-10-17T16:19:56.567+00:00
    Nikhil Gupta 1 Reputation point

    Hi anonymous user-9414

    We also tried the same in one of my projects. Where user can capture or upload an image and get the result on the basis of prediction.

    Few things to remember:

    1. Prediction API - you have to Post an AJAX request to url="your_endpoint" send Content-Type="application/octet-stream" and Prediction-key="your_prediction_key" in header and data in the form of octet-stream remember to make processData=false (So without any attempt to modify data by encoding as a query string)
    2. Make sure you convert your image into binary format (convert base64 to raw binary data in javascript itself)

    Below is the values need to send in ajax call. Where I was sending my captured or uploaded image data_uri to function dataURItoBlob(data_uri) to convert base64 to raw binary data.

    {  
        'url': 'Your-Endpoint',  
     'method': 'POST',  
     'timeout': '0',  
     'processData': false,  
     'headers':{  
     'Content-Type': 'application/octet-stream',  
     'Prediction-key': 'Your-Prediction-Key'  
     },  
     'data': dataURItoBlob(data_uri)  
    };  
    
    
     function dataURItoBlob(dataURI) {  
     // convert base64 to raw binary data held in a string  
     // doesn't handle URLEncoded DataURIs  
     var byteString = atob(dataURI.split(',')[1]);  
      
     // separate out the mime component  
     var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0]  
      
     // write the bytes of the string to an ArrayBuffer  
     var ab = new ArrayBuffer(byteString.length);  
     var ia = new Uint8Array(ab);  
     for (var i = 0; i < byteString.length; i++) {  
     ia[i] = byteString.charCodeAt(i);  
     }  
      
     // write the ArrayBuffer to a blob, and you're done  
     var bb = new Blob([ab], {type: mimeString});  
     return bb;  
     }  
      
    

    NOTE:

    1. your_endpoint will get change if you make the iteration.
    2. Security Recommendation =>Either follow the answer in https://learn.microsoft.com/en-us/answers/questions/121143/azure-computer-vision-api.html or call this Prediction API in the backend language so the user can't access your endpoint and Prediction-key as both are sensitive information.
    3. Performance Recommendation => Better to first capture and then hit for Prediction API on another event (As might be possible that the captured image is by mistake - Avoid unrequited Prediction API calls)

    Hope this will help you...

    Regards,
    Nikhil Gupta