How to get blendshapes from SSML

Question

How to get blendshapes from SSML

Dexin Huang 5

I am using the following code, however, I cannot retrieve the JSON string that is supposed to be able to drive blendshapes.


private System.Collections.IEnumerator SpeakAsync(string response){    var startTime = DateTime.Now;    var speechSynthesisVoiceName  = "en-US"+"-"+voice+"Neural";

var ssml = @$"<speak version='1.0' xml:lang='en-US' xmlns='http://www.w3.org/2001/10/synthesis' xmlns:mstts='http://www.w3.org/2001/mstts'>    <voice name='{speechSynthesisVoiceName}'>        <mstts:viseme type='FacialExpression'/>        {response}    </voice></speak>";


synthesizer.VisemeReceived += (s, e) =>{    Debug.Log($"Viseme event received. Audio offset: " +                      $"{e.AudioOffset / 10000}ms, viseme id: {e.VisemeId}.{e.Animation}");    // `Animation` is an xml string for SVG or a json string for blend shapes    var animation = e.Animation;    blendShapeDictionary = JsonUtility.FromJson<Dictionary<string, float>>(animation);

romungi-MSFT 48,766 Reputation points Microsoft Employee

2023-01-24T05:34:44.61+00:00

Dexin Huang I have tried the below SSML with the guidance to get the blended shapes with python SDK(1.24) and it returns the event details for animation as expected. In the above snippet I think you are printing the animation details in the log too, are you seeing the info printed in this log apart from the XML string that you are loading in the dictionary?

Also, could you try to check if the .net SDK is also the latest version?
Dexin Huang 5 Reputation points

2023-01-24T16:03:33.9166667+00:00

Hi, unfortunately, I do not see that. I have the "Viseme event received" logs, but not the blendshape logs. In Unity, it is .NET Standard 2.1. The debug.log for animation is executing, but it is empty. Thanks!
romungi-MSFT 48,766 Reputation points Microsoft Employee

2023-01-24T16:57:50.1166667+00:00

Could you add an issue to the SDK repo on this page? The SDK team should confirm is this is supported. I don't see an issue reported already for this or details in release notes of SDK for unity.

I will try to check internally if this is supported with unity. Thanks!!

1 answer

Your answer

romungi-MSFT 48,766 Reputation points Microsoft Employee

2023-01-24T05:34:44.61+00:00

Dexin Huang I have tried the below SSML with the guidance to get the blended shapes with python SDK(1.24) and it returns the event details for animation as expected. In the above snippet I think you are printing the animation details in the log too, are you seeing the info printed in this log apart from the XML string that you are loading in the dictionary?

Also, could you try to check if the .net SDK is also the latest version?
Dexin Huang 5 Reputation points

2023-01-24T16:03:33.9166667+00:00

Hi, unfortunately, I do not see that. I have the "Viseme event received" logs, but not the blendshape logs. In Unity, it is .NET Standard 2.1. The debug.log for animation is executing, but it is empty. Thanks!
romungi-MSFT 48,766 Reputation points Microsoft Employee

2023-01-24T16:57:50.1166667+00:00

Could you add an issue to the SDK repo on this page? The SDK team should confirm is this is supported. I don't see an issue reported already for this or details in release notes of SDK for unity.

I will try to check internally if this is supported with unity. Thanks!!

Answer 1

romungi-MSFT 48,766 Microsoft Employee

Adding reference to the SDK issue as answer to help other members of the community. Thanks!!

Share via

How to get blendshapes from SSML

1 answer

Your answer