Monitoring Audio From a Webcam/Capture Device in UWP

Alfonso Crawford 136 Reputation points
2021-01-05T17:48:40.693+00:00

In my UWP app, I need to monitor the audio levels of a webcam/capture device while previewing and recording. According to my research, MediaCapture isn't designed to give access to audio data outside of the recording process: I'm at a loss at which component within the UWP toolset I am meant to utilize or if I need to get/create a plugin of some kind.

Ultimately, I'm only trying to create a simple audio-level bar, such as in OBS Studio and other media software. What I need isn't terribly complex. Playback would also be nice, but it's secondary.

Any guidance you can provide is greatly appreciated.

Universal Windows Platform (UWP)
0 comments No comments
{count} votes

Accepted answer
  1. Yan Gu - MSFT 2,676 Reputation points
    2021-01-06T07:12:09.687+00:00

    Hello,

    Welcome to Microsoft Q&A.

    You could use a MediaFrameReader with MediaCapture to get AudioFrame which provides information of the frame. To read the audio sample data, you will need to get the AudioBuffer object from the AudioMediaFrame object, create an IMemoryBufferReference, and then call the COM method IMemoryBufferByteAccess::GetBuffer to retrieve the data. Calculate decibels of frames by calculating the average amplitude of all the pcm data from frams.

    You could get more information about using a MediaFrameReader with MediaCapture to get AudioFrame objects referring to the document.

    You could check the following code as a sample:

    MediaCapture mediaCapture;  
    MediaFrameReader mediaFrameReader;  
      
    private async void InitializeMediaCapture()  //Call the method when the page is initialized.  
    {  
        mediaCapture = new MediaCapture();  
        await mediaCapture.InitializeAsync(new MediaCaptureInitializationSettings {   
            MediaCategory=MediaCategory.Speech,  
            StreamingCaptureMode=StreamingCaptureMode.Audio  
        });  
      
    }  
      
    private async void Button_Click(object sender, RoutedEventArgs e)  
    {  
        var audioFrameSources = mediaCapture.FrameSources.Where(x => x.Value.Info.MediaStreamType == MediaStreamType.Audio);  
        if(audioFrameSources.Count()==0)  
        {  
            Debug.WriteLine("No audio frame source was found");  
            return;  
        }  
        MediaFrameSource frameSource = audioFrameSources.FirstOrDefault().Value;  
        MediaFrameFormat format = frameSource.CurrentFormat;  
        if(format.Subtype!=MediaEncodingSubtypes.Float)  
        {  
            return;  
        }  
        if(format.AudioEncodingProperties.ChannelCount!=1||format.AudioEncodingProperties.SampleRate!=48000)  
        {  
            //return;  
        }  
        mediaFrameReader = await mediaCapture.CreateFrameReaderAsync(frameSource);  
        mediaFrameReader.AcquisitionMode = MediaFrameReaderAcquisitionMode.Buffered;  
        mediaFrameReader.FrameArrived += MediaFrameReader_FrameArrived;  
        var status = await mediaFrameReader.StartAsync();  
        if(status!=MediaFrameReaderStartStatus.Success)  
        {  
            Debug.WriteLine("The MediaFrameReader couldn't start");  
        }              
    }  
      
    [ComImport]  
    [Guid("5B0D3235-4DBA-4D44-865E-8F1D0E4FD04D")]  
    [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]  
    unsafe interface IMemoryBufferByteAccess  
    {  
        void GetBuffer(out byte* buffer, out uint capacity);  
    }  
    unsafe private void ProcessAudioFrame(AudioMediaFrame audioMediaFrame)  
    {  
      
        using (AudioFrame audioFrame = audioMediaFrame.GetAudioFrame())  
        using (AudioBuffer buffer = audioFrame.LockBuffer(AudioBufferAccessMode.Read))  
        using (IMemoryBufferReference reference = buffer.CreateReference())  
        {  
            byte* dataInBytes;  
            uint capacityInBytes;  
            float* dataInFloat;  
            ((IMemoryBufferByteAccess)reference).GetBuffer(out dataInBytes, out capacityInBytes);  
      
            // The requested format was float  
            dataInFloat = (float*)dataInBytes;  
            var dataInFloats = new float[capacityInBytes / sizeof(float)];  
            for(int i=0;i<capacityInBytes/sizeof(float);i++)  
            {  
                dataInFloats[i] = dataInFloat[i];  
            }  
            double decibels = 0f;  
            foreach(var sample in dataInFloats)  
            {  
                decibels += Math.Abs(sample);  
            }  
            decibels = 20 * Math.Log10(decibels / dataInFloats.Length);  
            Debug.WriteLine("The value is: " + decibels.ToString());  
            //Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () => {  
            //    textBlock.Text += decibels.ToString()+"\n";  
      
            //});  
        }  
    }  
      
    private void MediaFrameReader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)  
    {  
        using(MediaFrameReference reference=sender.TryAcquireLatestFrame())  
        {  
            if(reference!=null)  
            {  
                ProcessAudioFrame(reference.AudioMediaFrame);  
            }  
        }  
    }  
    private async void Button_Click_1(object sender, RoutedEventArgs e)  
    {  
        await mediaFrameReader.StopAsync();  
    }  
    

    Note, the usage of unsafe keyword in above code needs to check the box Allow unsafe code in the Build tab of the Project -> Properties dialog. The above code use microphone to test, you need to change the settings of MediaCapture to suit your Webcam.


    If the response is helpful, please click "Accept Answer" and upvote it.
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.