question

JustinGriep-9858 avatar image
0 Votes"
JustinGriep-9858 asked JustinGriep-9858 commented

Azure Media Services V3 and Thumbnails

I have been trying to determine how to process and support thumbnail images for encoded videos and the answers are very hard to find for Azure Media Services V3, many links to docs on forums are also throwing 404s.

Part of this issue is a new Roku store policy that requires them to be included in HLS or DASH streams for new release and they would be nice for other platforms as well.

After all of this I am left with a number of questions:

  • Does Dyanmic Packaging have thumbnail support? If so were can we learn how to work with it?

  • Can I process and provide thumbnails for existing videos in our catalog without re-encoding the entire video?

  • Is their any documentation for the thumbnails or the schema for transforms for V3?

  • It looks like the JpgImage in the transform schema will at least generate the image files, is this correct?

azure-media-services
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

JohndeuMSFT avatar image
1 Vote"
JohndeuMSFT answered JustinGriep-9858 commented

Hi Justin,

A few quick answers.

  1. Dynamic packaging does not generate thumbnails on the fly at all. The default presets for Content Aware, Adaptive Bitrate, etc, do generate a "best" thumbnail, but if you want to control the thumbnail settings you do need to add that to a custom encoding job. I have some Javascript/TS samples here that show how to do that:
    https://github.com/Azure-Samples/media-services-v3-node-tutorials/blob/b0ad91bb7f0b466fb6ec1237301ff7c6e130c49b/VideoEncoding/Encoding_H264/index.ts#L115

  2. You can send any existing Asset in AMS back through a "thumbnail" specific transform. See example above on how to create a thumbnail in a preset. Just create a preset that does nothing other than generate a single JPG, or a Sprite with VTT file as in this sample. Just remove the video and audio tracks from that Transform - https://github.com/Azure-Samples/media-services-v3-node-tutorials/blob/main/VideoEncoding/Encoding_Sprite_Thumbnail/index.ts

  3. All the schema for this is actually in the REST API. You should look at the Transform entity details in the REST API for all the encoding and thumbnail settings. Search for PNGFormat, JPGFormat, JPGImage, PNGImage, - https://docs.microsoft.com/en-us/rest/api/media/transforms/create-or-update

  4. Correct the JPG or PNGImage objects in the REST API have all the settings you would need.


Hope this helps!


· 7
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Ohh, and if you can find the links that are 404 - let me know! I can try to figure out why.

0 Votes 0 ·

If you search for Azure Media Services V3, they are all over the various forum posts, StackOverlow, the old MS forum, etc. So I don't know what you can do about that.

0 Votes 0 ·
JohndeuMSFT avatar image JohndeuMSFT JustinGriep-9858 ·

Yea, it is tough to fix the older posts on the Stack and previous MSDN forum. Bummer, it is just an issue with moving forward with docs and samples.
If you find any big issues let us know.

As to the Roku store issue - I think that they had a proprietary image format track that was not actually in the HLS or DASH specification in the past. If you have the link to that I can take a look again, but recently there was an update to the official HLS spec to support image tracks - that should be what ROKU adopts rather than pushing their proprietary thing.

I was tracking this issue in the CTA WAVE working group - https://github.com/cta-wave/dash-hls/issues/37 which also mentioned the same

"There is the Image media playlists HLS extension proposal from Roku/Disney/Warner here https://developer.roku.com/docs/developer-program/media-playback/trick-mode/hls-and-dash.md but its relevance/adoption is limited by the fact that it's not part of the RFC. Same logic here: even if not supported by Apple players which don't need it as they can leverage iframe-only tracks, it would be super useful for the rest of the ecosystem to get this officially part of the HLS RFC."

On 2/23, Apple announced upcoming iOS support for M-JPEG (ISO/IEC 23008-12 "High efficiency coding and media delivery in heterogeneous environments —Part 12: Image File Format") encapsulated in fMP4 (CMAF?) containers.

0 Votes 0 ·
Show more comments

Hi John,

Thank you so much for the answers, this gets me most of the way there.

We are using .NET, so JS/TS isn't preferred, but we are using the examples I found here which should cover your examples:
https://docs.microsoft.com/en-us/azure/media-services/latest/samples-overview?tabs=node

On my first question, I am not assuming that Dynamic Packaging would generate the thumbnails, but to get it to work does require the inclusion of the thumbnails in the manifest file based on the specifications for HLS and DASH.

Here is the DASH spec:
https://dashif.org/docs/DASH-IF-IOP-v4.3.pdf

And the HLS spec:
https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis-04

So my question is really, does Dynamic Packaging support generating these thumbnail portions of the manifest? If so, how do we configure that? If not, how can we work around this with Media Services?

Thank you again,

Justin



0 Votes 0 ·
JohndeuMSFT avatar image JohndeuMSFT JustinGriep-9858 ·

For live we only generate IFrame tracks with IDR frames for trick-play mode. But that requires a live stream going through out encoding live events.

For VOD, we do not yet have any packager support for the HLS or DASH style image tracks. As pointed out above, there are some updates to the HLS spec recently that we are looking into and might add that to our roadmap in the future. I'll inform the team of your interest here and point to the latest updates from HLS and DASH orgs.

0 Votes 0 ·