Viseme rendering service

Question

Viseme rendering service

DWinkler 1

The documentation of the TTS viseme callbacks states
"With the help of a 2D or 3D rendering engine, you can use the viseme output to control the animation of your avatar" 2356748

I see that you already provide such functionality for bing, e.g.
https://www.bing.com/search?q=Translate%20house%20to%20Italian

but I couldn't find any public service with an API for the generation of visualizations.

Are you planning to introduce this functionality?

romungi-MSFT 32,931 Reputation points Microsoft Employee

2021-09-29T11:27:07.133+00:00

@DWinkler Do you mean the REST API version of the feature? If Yes, this is not supported yet by the service and there is no visibility of rolling out the same through REST API.
I believe the bing service is using the JS SDK to enable the feature with bing search.
DWinkler 1 Reputation point

2021-09-29T12:20:01.01+00:00

Thanks a lot for your feedback, I am currently using the .net SDK.
I do receive the events, events but the Animation property is empty
.NET SDK JS SDK

Do you think this is a JS only thing, or is there anything else i have to enable?
romungi-MSFT 32,931 Reputation points Microsoft Employee

2021-09-30T05:10:37.317+00:00

@DWinkler I think the animation property is only used to pass the format of file used. The actual animation from viseme events is based on viseme ID and the mapping between viseme events and phonemes. The time based output should help map the viseme ids to SVGs. I could not find a working sample that can help run the animation but this mapping can help build custom animations based on SVGs.

Will certainly lookout for any samples published by speech service team that can be used to build the animation.
DWinkler 1 Reputation point

2021-09-30T05:38:05.287+00:00

Thanks a lot for your feedback.

As you mention, for a start it would be quite easy to use the provided viseme IDs to show the images of a chart like here.
But the animations that were shown in the video look very nice and smooth, and somehow seem to be used also by bing.

If you could find any pointers (service API, github) on how you do that at MS it would be nice if you could let me know.

Kind regards, D

romungi-MSFT 32,931 Reputation points Microsoft Employee

2021-09-29T11:27:07.133+00:00

@DWinkler Do you mean the REST API version of the feature? If Yes, this is not supported yet by the service and there is no visibility of rolling out the same through REST API.
I believe the bing service is using the JS SDK to enable the feature with bing search.
DWinkler 1 Reputation point

2021-09-29T12:20:01.01+00:00

Thanks a lot for your feedback, I am currently using the .net SDK.
I do receive the events, events but the Animation property is empty
.NET SDK JS SDK

Do you think this is a JS only thing, or is there anything else i have to enable?
romungi-MSFT 32,931 Reputation points Microsoft Employee

2021-09-30T05:10:37.317+00:00

@DWinkler I think the animation property is only used to pass the format of file used. The actual animation from viseme events is based on viseme ID and the mapping between viseme events and phonemes. The time based output should help map the viseme ids to SVGs. I could not find a working sample that can help run the animation but this mapping can help build custom animations based on SVGs.

Will certainly lookout for any samples published by speech service team that can be used to build the animation.
DWinkler 1 Reputation point

2021-09-30T05:38:05.287+00:00

Thanks a lot for your feedback.

As you mention, for a start it would be quite easy to use the provided viseme IDs to show the images of a chart like here.
But the animations that were shown in the video look very nice and smooth, and somehow seem to be used also by bing.

If you could find any pointers (service API, github) on how you do that at MS it would be nice if you could let me know.

Kind regards, D

Viseme rendering service

Activity