How can face detection and tracking be made to appear in the capture stream?

Question

Hi,
I am using the MediaCapture API for a video recording app.

Part of it is face detection using rectangles, but I would like that to appear in the capture stream also, not just the preview stream.

Even though I indicate VideoRecord as the MediaStreamType, the rectangles do not appear in the end-result video, which is my goal.

var definition = new FaceDetectionEffectDefinition
{
SynchronousDetectionEnabled = false,
DetectionMode = FaceDetectionMode.HighPerformance
};

try
{
faceDetectionEffect = (FaceDetectionEffect)await mediaCapture.AddVideoEffectAsync(definition, MediaStreamType.VideoRecord);
}
catch (Exception exception)
{
// Error message
return;
}
faceDetectionEffect.FaceDetected += FaceDetectionEffect_FaceDetected;
faceDetectionEffect.DesiredDetectionInterval = TimeSpan.FromMilliseconds(33);
faceDetectionEffect.Enabled = true;

Any advice on how to show face detection and tracking in the capture stream?

Thanks!

Answer 1

Hi, thanks for the response! Basically, I want face detection and tracking to display in the CaptureElement so that identity hiding (obscuring the user's face with an image that follows his/her face) can be accomplished in the final video.

My code borrows from the Windows universal samples: https://github.com/microsoft/Windows-universal-samples

There is a CaptureElement where the video stream from the camera is rendered. To draw the rectangles of the detected faces, a Canvas is used...

           <CaptureElement Name="PreviewControl"                            
                       Opacity="0.8"
                       Stretch="Fill"
                       IsDoubleTapEnabled="True" />

         <Canvas>
             <Canvas Name="FacesCanvas"
                     RenderTransformOrigin="0.5,0.5" />
         </Canvas>

This is how the FaceDetectionEffectDefinition is added to the MediaCapture object, and the FaceDetectionEffect is added...

private async Task CreateFaceDetectionEffectAsync()
{
    var definition = new FaceDetectionEffectDefinition
    {
        SynchronousDetectionEnabled = false,
        DetectionMode = FaceDetectionMode.HighPerformance
    };
    try
    {
        faceDetectionEffect = (FaceDetectionEffect)await mediaCapture.AddVideoEffectAsync(definition, MediaStreamType.VideoRecord);
    }
    catch (Exception exception)
    {
        //exception.Message.ToString()
        return;
    }
    faceDetectionEffect.FaceDetected += FaceDetectionEffect_FaceDetected;
    faceDetectionEffect.DesiredDetectionInterval = TimeSpan.FromMilliseconds(33);
    faceDetectionEffect.Enabled = true;
}

This is what goes on inside the FaceDetectionEffect event...

private async void FaceDetectionEffect_FaceDetected(FaceDetectionEffect sender, FaceDetectedEventArgs args)
{
    await MainPage.Current.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () => HighlightDetectedFaces(args.ResultFrame.DetectedFaces));
}

private void HighlightDetectedFaces(IReadOnlyList<DetectedFace> faces)
{
    facesCanvas.Children.Clear();
    if (softwareBitmapSource == null)
        return;

        for (int i = 0; i < faces.Count; i++)
        {
             Rectangle faceBoundingBox = ConvertPreviewToUiRectangle(faces[i].FaceBox);                
             faceBoundingBox.Fill = new ImageBrush
             {
                 ImageSource = softwareBitmapSource
             };                
             facesCanvas.Children.Add(faceBoundingBox);
         }            
}

How the MediaCapture object gets initialized and is set as the source of the CaptureElement in the beginning...

(Some code, such as specifying the media stream type for VideoPreview and VideoRecord, if not identical, has been omitted.)

MediaCaptureInitializationSettings settings = new MediaCaptureInitializationSettings { VideoDeviceId = preferredCamera.Id, AudioDeviceId = preferredMicrophone.Id };

await mediaCapture.InitializeAsync(settings);

PreviewControl.Source = mediaCapture;

Hopefully, there is an easier technique for identity hiding, without having to resort to using MediaComposition and overlays.

Answer 2

Yes. The goal is to have them render in the CaptureElement. Is grabbing a frame, detecting a face, and modifying the frame the only way, or is there an easier method?