Receive H264 and HEVC from camera using Media Foundation class

Donthu, Gangadhar 41

Hi,

I would like to develop an application based on Media Foundation to receive H264 and H265 encoded video streams and pass them to cloud. Please help me with a sample application.

Thanks
GD

2 answers

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-01T02:21:31.237+00:00

Hello,

Welcome to Microsoft Q&A!

First you could try to rreate the appropriate media types for H264 and HEVC. You could refer to the Doc: Writing a Custom Media Source.

And then you could capture video from a video camera to a file. Here is a sample you could refer to: MFCaptureToFile Sample

If you want to pass them to cloud in real time, I suggest you could refer to the thread: https://social.msdn.microsoft.com/Forums/en-US/8f67d241-7e72-4509-b5f8-e2ba3d1a33ad/capture-h264aac-stream-via-rtsp-using-media-foundation?forum=mediafoundationdevelopment

In my opinion, it is related to which cloud do you want to pass the video streams to.

Thank you.

Jeanine

If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.
Please sign in to rate this answer.
Donthu, Gangadhar 41 Reputation points

2022-12-01T08:54:05.373+00:00

Hi Jeanine,

I have already started looking in to MFCaptureToFile and able to open camera stream. I have tried modifying the code to read H264 data from camera with below modifications. But it failed to setInputMediaType for the writer with return value "0xc00d36b4". Please suggest how to debug this issue?

if (SUCCEEDED(hr)) { hr = m_pWriter->SetInputMediaType(sink_stream, pType, NULL); }

Note:
I am able to open the stream using PotPlayer and able to capture video as well.
I would like to develop an app to capture the H264 video stream first in a app and then want to extend the app to stream HEVC as well (camera is capable of streaming HEVC video stream as well).
We already have a video pipeline, developed for another camera, which streams JPEGs to the windows app (Visual C++, but not Media foundation based). As the new camera streams HEVC and H264, would like to use Media Foundation, which has support for H264 and HEVC codecs.
If you insist me to use custom media source, please also suggest me an app, which uses DLL/LIB from MPEG1source example.

Code modifications here:

=================

// GUID subtypes[] = {
// MFVideoFormat_NV12, MFVideoFormat_YUY2, MFVideoFormat_UYVY,
// MFVideoFormat_RGB32, MFVideoFormat_RGB24, MFVideoFormat_IYUV
// };

GUID subtypes[] = { MFVideoFormat_H264, MFVideoFormat_NV12, MFVideoFormat_YUY2, MFVideoFormat_UYVY, MFVideoFormat_RGB32, MFVideoFormat_RGB24, MFVideoFormat_IYUV };

Please help!

Thanks
GD

Donthu, Gangadhar 41 Reputation points

2022-12-01T08:55:52.947+00:00

I think I missed out some more changes I made:

// Try to find a suitable output type. if (SUCCEEDED(hr)) { for (DWORD i = 0; ; i++) { hr = pReader->GetNativeMediaType( (DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, i, &pType ); if (FAILED(hr)) { break; }

// hr = TryMediaType(pType);
subtype = { 0 };
hr = pType->GetGUID(MF_MT_SUBTYPE, &subtype);

if (subtype == subtypes[0]) { hr = pReader->SetCurrentMediaType( (DWORD)MF_SOURCE_READER_FIRST_VIDEO_STREAM, NULL, pType ); bUseNativeType = TRUE; break; } SafeRelease(&pType); } }

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-05T05:49:31.077+00:00

@Donthu, Gangadhar

I suggest you could refer to the thread: https://social.msdn.microsoft.com/Forums/en-US/7c29faa4-ee23-42fa-81bb-98e7625b8673/unexpected-error-0xc00d36b4-the-data-specified-for-the-media-type-is-invalid-inconsistent-or-not

0xC00D36B4 MF_E_INVALIDMEDIATYPE The underlying media sink does not support the format, no conversion is possible, or a dynamic format change is not possible. It tells that you are trying to write a feed that requires conversion, and there is no suitable codec available. That is, the problem is with your media type which is not accepted by H.264 codec.

H.264 is natively supported by Media Foundation on Windows. There are differences in what is supported by various versions of Windows (resolution is a key factor.)
I suggest you should refer to the Doc: Format Constraints, to understand the constraint format of the H264 decoder.

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-05T08:37:55.45+00:00

@Donthu, Gangadhar

According to the offical sample:

if (BST_CHECKED == IsDlgButtonChecked(hDlg, IDC_CAPTURE_WMV)) { params.subtype = MFVideoFormat_WMV3; } else { params.subtype = MFVideoFormat_H264; }

When you select mp4, the program captures H264 data from camera by default. When you use the official sample directly, do you get the same error message?

If you get the same error message, could you please provide the video Format (Resolution, Bitrate……). It may be that the video format is not supported by the H264 decoder.

Donthu, Gangadhar 41 Reputation points

2022-12-05T08:52:34.12+00:00

@Jeanine Zhang-MSFT , thanks for your inputs.

My requirement is: Camera sends encoded (H264) stream to the host. So, my reader should read H264. Please suggest what should be the writer (Sink) configuration? I thought the input format for writer (Sink) can be H264 encoded from Writer. Please help me in understanding this.

Input Media format for the Writer (Source), is H264 (because camera sends H264 encoded video stream over UVC). Does the writer (Sink) accept H264 format? In the above code example by me, I tried to set it as H264 format. According to the error, IMFSinkWriter does not support H264, is that true?

By the way, to answer your question, I was able to record mp4 from a camera which streams NV12 format. But my requirement is to capture H264 encoded video stream from camera. Please help.

Thanks
GD

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-06T03:45:30.477+00:00

@Donthu, Gangadhar

According to your description, the Native Type of Camera is MFVideoFormat_H264? Have you get the MFVideoFormat_H264 stream from the Camera? If you want to capture MFVideoFormat_H264 stream from camera directly?

According to the code:

// The list of acceptable types. GUID subtypes[] = { MFVideoFormat_NV12, MFVideoFormat_YUY2, MFVideoFormat_UYVY, MFVideoFormat_RGB32, MFVideoFormat_RGB24, MFVideoFormat_IYUV };

You couldn't add the MFVideoFormat_H264 directly. MFVideoFormat_H264 is an encoded video type
You need a decoder to decode the MFVideoFormat_H264 stream.

Donthu, Gangadhar 41 Reputation points

2022-12-06T04:03:56.617+00:00

@Jeanine Zhang-MSFT , thanks again for your inputs.

Please share an example how to add a decoder to the "Source Reader".

I have added H264 format to the list as below, looks like it's incorrect as you mentioned above. What is the right way of adding a decoder?
GUID subtypes[] = {
MFVideoFormat_H264, MFVideoFormat_NV12, MFVideoFormat_YUY2, MFVideoFormat_UYVY,
MFVideoFormat_RGB32, MFVideoFormat_RGB24, MFVideoFormat_IYUV
};

What should be the SetCurrentMediaType? for the "Source Reader"

Yes, if I open the camera using "PotPlayer windows app", I am able to capture H264 stream from the camera.

Thanks
GD

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-06T06:30:34.757+00:00

@Donthu, Gangadhar

According to the Doc: IMFSourceReader：：SetCurrentMediaType Method

If the native media type is compressed, you can specify a corresponding uncompressed format. The Source Reader will search for a decoder that can decode from the native format to the specified uncompressed format.

In the case of the decoder, create the media type as follows:

1, Call MFCreateMediaType to create a new media type.
2, Set the MF_MT_MAJOR_TYPE attribute to specify audio or video.
3, Set the MF_MT_SUBTYPE attribute to specify the subtype of the decoding format.
4, Call IMFSourceReader::SetCurrentMediaType.

The Source Reader will automatically load the decoder.

For more details I suggest you could refer to the Doc: Setting Output Formats

Donthu, Gangadhar 41 Reputation points

2022-12-07T06:54:56.333+00:00

Thanks @Jeanine Zhang-MSFT ,

By the way is the timestamp received (llTimestamp) from the async callback, SourceReaderCB::OnReadSample, coming from hardware? or it is created on the host machine? The documentation does not detail enough.

Also, without using the "Sink Writer", I would like to save H264 samples received by "Source Reader" into a file, is it possible?

Thanks
GD

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-07T08:17:48.15+00:00

@Donthu, Gangadhar

When the ReadSample method completes asynchronously, the Source Reader calls your IMFSourceReaderCallback::OnReadSample method.

dwStreamIndex, dwStreamFlags, llTimestamp, and pSample: These three parameters are equivalent to the last three parameters in ReadSample.

According to the Doc: IMFSourceReader::ReadSample method (mfreadwrite.h)

[out] pllTimestamp

Receives the time stamp of the sample, or the time of the stream event indicated in pdwStreamFlags. The time is given in 100-nanosecond units.

As far as I'm concerned, the llTimestamp is the out value of the IMFSourceReader：：ReadSample Method.

I suggest you could refer to the Doc: Using the Source Reader in Asynchronous Mode.

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-07T08:19:11.293+00:00

The Second question:

>Also, without using the "Sink Writer", I would like to save H264 samples received by "Source Reader" into a file, is it possible?

In my opinion, it is impossible. I suggest you could refer to the Doc: Sink Writer

Source Reader and Sink Writer are independent components. We use Source Reader to let the application obtains or generates the input data.

And then the application uses the sink writer to encode and audio/video file.

Donthu, Gangadhar 41 Reputation points

2022-12-07T08:58:06.557+00:00

Thanks for the details @JimmyYang-MSFT ,

OnSample callback gives IMFSample* pointer. How do I extract the H264 data from the IMFSample pointer, which was actually pushed by the camera?

Thanks
GD

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-07T09:10:35.24+00:00

@Donthu, Gangadhar

pSample:A pointer to the IMFSample interface of a media sample. A media sample contains zero or more buffers. Each buffer manages a block of memory and is represented by the IMFMediaBuffer interface. IMFMediaBuffer interface represents a block of memory that contains media data. Use this interface to access the data in the buffer.
Sign in to comment
Donthu, Gangadhar 41 Reputation points

2022-12-08T02:13:21.383+00:00

Hi @Jeanine Zhang-MSFT , thanks for your inputs.

How do I send XU commands to camera, like changing the resolution, White balance, 3A settings etc. Please share some example for the same.

Thanks
GD
Please sign in to rate this answer.
Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-08T06:05:59.33+00:00

@Donthu, Gangadhar

According to the issue, it seems to be related to Windows Drivers. I suggest you could post a new thread with "windows-hardware-wdk"tag.

And have you got any updates or any other doubts about the above issue?

Donthu, Gangadhar 41 Reputation points

2022-12-08T06:11:24.503+00:00

@Jeanine Zhang-MSFT ,

Does the Media Foundation class provide any controls to camera? Like 3A controls? Do I need to go through windows-hardware-wdk if I have to develop a camera application for windows? Please suggest.

Thanks
GD

Donthu, Gangadhar 41 Reputation points

2022-12-08T06:23:29.93+00:00

For example, here is a link to manual controls for camera: https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/capture-device-controls-for-video-capture

But the examples are in C#.

Thanks
GD

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-08T07:19:14.903+00:00

@Donthu, Gangadhar

Microsoft Media Foundation enables the development of applications and components for using digital media on Windows Vista and later. If you want to extended camera controls, in my opinion, it is related to Windows Drivers.

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-08T07:20:09.783+00:00

@Donthu, Gangadhar

The sample you provide is a UWP program, and the program is used to enable enhanced video capture scenarios.If you want to change the camera settings on a video capture device, you could follow the Doc: Configure the Video Quality

The WDM Video Capture filter controls analog capture devices that use Windows Driver Model (WDM) drivers.

Donthu, Gangadhar 41 Reputation points

2022-12-08T08:19:01.827+00:00

@Jeanine Zhang-MSFT ,

For writing a camera application, should I use DirectShow (like you mentioned above)? or Media Foundation classes? Or both of them?

I would like to stream H264/H265 from camera/capture device.
Should be able to program manual controls.

Please provide your recommendation.

Thanks
GD

Jeanine Zhang-MSFT 9,181 Reputation points Microsoft Vendor

2022-12-09T07:15:18.06+00:00

@Donthu, Gangadhar

Both DirectShow and Media Foundation can be used to capture video. As far as I'm concerned, you could choose whichever one you like depending on your needs.

Additional instructions: Microsoft Media Foundation was introduced in Windows Vista as the replacement for DirectShow. DirectShow is relatively old.
Sign in to comment