question

MichaelChourdakis-9912 avatar image
0 Votes"
MichaelChourdakis-9912 asked JunjieZhu-MSFT commented

Extending Media Foundation Encoder to support 10-bit video encoding

The HEVC Media Foundation Encoder in Windows will only encode 8-bit video. My GFX NVidia card also supports 10-bit HDR and alpha-mode video encoding, so I decided to create my own IMFTransform to use the NVidia SDK.

I 've registered my DLL using MFTRegister with a non existing input type to fool the Sink Writer to pick my encoder when calling SetInputMediaType instead of the predefined Microsoft's transform. This works OK.

 int wi = 1920;
 int he = 1080;
 int fps = 30;
 int br = 4000;
 auto fmt = MFVideoFormat_H264;
 bool Our = 1;
 const wchar_t* fil = L"r:\\1.mp4";
 std::vector<DWORD> frame;
 frame.resize(wi * he);
    
 // Test
 CComPtr<IMFSinkWriter> wr;
 DeleteFile(fil);
    
 CComPtr<IMFAttributes> attrs;
 MFCreateAttributes(&attrs, 0);
 auto hr = MFCreateSinkWriterFromURL(fil, 0, attrs, &wr);
 DWORD str = (DWORD)-1;
 CComPtr<IMFMediaType> mt2;
 MFCreateMediaType(&mt2);
 mt2->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
 mt2->SetGUID(MF_MT_SUBTYPE, fmt);
 MFSetAttributeRatio(mt2, MF_MT_FRAME_RATE, fps, 1);
 hr = MFSetAttributeSize(mt2, MF_MT_FRAME_SIZE,wi, he);
 MFSetAttributeRatio(mt2, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
 mt2->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);
 mt2->SetUINT32(MF_MT_VIDEO_NOMINAL_RANGE, MFNominalRange_Normal);
 mt2->SetUINT32(MF_MT_AVG_BITRATE, br*1000);
 hr = wr->AddStream(mt2, &str);
 CComPtr<IMFMediaType> mt1;
 MFCreateMediaType(&mt1);
 mt1->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
 mt1->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_ARGB32);
 hr = MFSetAttributeSize(mt1, MF_MT_FRAME_SIZE, wi, he);
 // Force our selection
 if (Our)
 {
     mt1->SetGUID(MF_MT_SUBTYPE, MyFakeFmt);
     hr = wr->SetInputMediaType(str, mt1, 0);
 }
 mt1->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_ARGB32);
 hr = wr->SetInputMediaType(str, mt1, 0);
 hr = wr->BeginWriting();
 for(int i = 0 ; i < 15 ; i++)
 {
     auto i2 = i % 5;
     if (i2 == 0)            Frm(frame, wi, he, 0xFFFFFFFF);
     if (i2 == 1 || i2 == 4)         Frm(frame, wi, he, 0xFF0000FF); // some colors
     if (i2 == 2)            Frm(frame, wi, he, 0xFFFF00FF); // 
     if (i2 == 3)            Frm(frame, wi, he, 0xFF00FFFF); //
    
     CComPtr<IMFSample> s;
     MFCreateSample(&s);
     int secs = 1;
    
     hr = s->SetSampleDuration(10 * 1000 * 1000 * secs);
     hr = s->SetSampleTime(10 * 1000 * 1000 * i);
    
     CComPtr<IMFMediaBuffer> b;
     MFCreateMemoryBuffer((DWORD)(frame.size() * 4), &b);        
     b->SetCurrentLength((DWORD)(frame.size() * 4));
     BYTE* by = 0;
     DWORD ml = 0, cl = 0;
     b->Lock(&by, &ml, &cl);
     memcpy(by, frame.data(), frame.size() * 4);
     b->Unlock();
     hr = s->AddBuffer(b);
     b = 0;
     hr = wr->WriteSample(str, s);
 }
    
 hr = wr->Finalize();
 wr = 0;


The problems start with the call to Finalize to end the writing. At that point, everything seems to work normally. Note that I have tested the NVidia IMFTransform I 've created with input frames and it encodes and outputs them correctly as raw data.

When I call Finalize and the type is MFVideoFormat_H264 , the call succeeds. However the generated mp4 plays weirdly:

199244-image.png


When the output is MFVideoFormat_HEVC, then Finalize fails with `0xc00d4a45 : Sink could not create valid output file because required headers were not provided to the sink.'.

I 've also tried to convert the raw .h264 file I 'm saving with ffmpeg to mp4, and this works. The mp4 generated plays correctly.

Adding a MF_MT_MPEG_SEQUENCE_HEADER didn't help (besides, I think this is only needed for H.264)

const char* bl4 = "\x00\x00\x00\x01\x67\x42\xC0\x28\x95\xB0\x1E\x00\x89\xF9\x70\x16\xC8\x00\x00\x03\x00\x08\x00\x00\x03\x01\xE0\x6D\x04\x42\x37\x00\x00\x00\x01\x68\xCA\x8F\x20";
mt2->SetBlob(MF_MT_MPEG_SEQUENCE_HEADER, (UINT8*)bl4, 39);

What do you make of all that?
:)


windows-api
image.png (13.8 KiB)
· 6
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

It's been a few days, is there any progress on your question?

0 Votes 0 ·

Yes, now the encoder creates the mp4 but it's still 8 bit although media info contains 10-bit samples. It somehow internally converts then to 8 and back to 10.

0 Votes 0 ·
JunjieZhu-MSFT avatar image JunjieZhu-MSFT MichaelChourdakis-9912 ·

So can you use the NVidia SDK normally now?

0 Votes 0 ·
Show more comments

0 Answers