Inquiry: Speech SDK Audio-Text Synchronization Challenge

Question

Issue Description

I am developing an Android application using Microsoft's Cognitive Services Speech SDK, specifically the speech translation feature with text-to-speech synthesis. I've encountered a significant challenge with synchronizing the synthesized audio streams with their corresponding translated text.

When using the TranslationRecognizer with synthesizing enabled, the synthesizing event provides audio data via TranslationSynthesisEventArgs, but there appears to be no mechanism to determine which translated text segment corresponds to each audio chunk.

Current Implementation

Here's a snippet from my current implementation:

java
Copy
translationRecognizer.synthesizing.addEventListener { s, e ->
    val audio = e.result.audio
    if (e.result.reason == ResultReason.SynthesizingAudio) {
        // I have no reliable way to determine which text this audio corresponds to
        // Currently using an incremental index as a workaround
        int index = voiceIndex++;
        qualcommPlayer.play(audioFilePath, translatedList.getOrNull(index), index);
    }
}

Technical Details

After examining the TranslationSynthesisEventArgs class definition:

java
Copy
public final class TranslationSynthesisEventArgs extends SessionEventArgs {
    // Methods and properties
    public final TranslationSynthesisResult getResult() {
        return result;
    }
    // Other implementation details...
}

I found that it provides no method to directly associate the audio with its source text. The sessionId property is available but doesn't appear to correlate consistently between recognition and synthesis events.

Questions

1. Is there an existing method within the Speech SDK to reliably link synthesized audio with its corresponding text?
2. Does Microsoft plan to enhance the SDK to include text-audio correlation in future releases?
3. Can you recommend an official solution for applications that require precise alignment between synthesized audio and source text?
4. Are there any undocumented properties or methods in TranslationSynthesisEventArgs or TranslationSynthesisResult that could help solve this issue?
5. Would it be possible to add a feature request to include text/segment identification in the synthesis results?

Potential Solutions

I've considered several workarounds, including:

• Queue-based tracking of texts and audio segments
• Session ID mapping (though this appears unreliable)
• Batch processing with synchronous waiting
• Custom timing solutions

However, a native SDK solution would greatly improve reliability and reduce implementation complexity.

Thank you for your assistance in addressing this technical challenge.

Contact Information

Name: [Your Name] Email: [Your Email] Company: [Your Company] Application: [Brief description of your application] SDK Version: [Version of Speech SDK you're using]

Issue Description

I am developing an Android application using Microsoft's Cognitive Services Speech SDK, specifically the speech translation feature with text-to-speech synthesis. I've encountered a significant challenge with synchronizing the synthesized audio streams with their corresponding translated text.

When using the TranslationRecognizer with synthesizing enabled, the synthesizing event provides audio data via TranslationSynthesisEventArgs, but there appears to be no mechanism to determine which translated text segment corresponds to each audio chunk.

Current Implementation

Here's a snippet from my current implementation:

java
Copy
translationRecognizer.synthesizing.addEventListener { s, e ->
    val audio = e.result.audio
    if (e.result.reason == ResultReason.SynthesizingAudio) {
        // I have no reliable way to determine which text this audio corresponds to
        // Currently using an incremental index as a workaround
        int index = voiceIndex++;
        qualcommPlayer.play(audioFilePath, translatedList.getOrNull(index), index);
    }
}

Technical Details

After examining the TranslationSynthesisEventArgs class definition:

java
Copy
public final class TranslationSynthesisEventArgs extends SessionEventArgs {
    // Methods and properties
    public final TranslationSynthesisResult getResult() {
        return result;
    }
    // Other implementation details...
}

I found that it provides no method to directly associate the audio with its source text. The sessionId property is available but doesn't appear to correlate consistently between recognition and synthesis events.

Questions

1. Is there an existing method within the Speech SDK to reliably link synthesized audio with its corresponding text?
2. Does Microsoft plan to enhance the SDK to include text-audio correlation in future releases?
3. Can you recommend an official solution for applications that require precise alignment between synthesized audio and source text?
4. Are there any undocumented properties or methods in TranslationSynthesisEventArgs or TranslationSynthesisResult that could help solve this issue?
5. Would it be possible to add a feature request to include text/segment identification in the synthesis results?

Potential Solutions

I've considered several workarounds, including:

• Queue-based tracking of texts and audio segments
• Session ID mapping (though this appears unreliable)
• Batch processing with synchronous waiting
• Custom timing solutions

However, a native SDK solution would greatly improve reliability and reduce implementation complexity.

Thank you for your assistance in addressing this technical challenge.

Contact Information

Name: will
Email: ******@gmail.com
SDK Version: 1.43.0