No audio when using SpeechSDK in pcf control (canvas app)

Question

I made a pcf control which uses the speechsdk to synthesize text to speech. This is working when I run "npm start watch" to test this. When publishing this pcf control and use it in a canvas powerapp I cannot hear the synthisized text.

What can be wrong?

    private onStartButtonClick(e: Event) {
        this.displayTempResult.value = this.messageToSay || "";

        const player = new SpeechSDK.SpeakerAudioDestination();

        if (player) {
            console.log('Player is initialized');
        } else {
            console.error('Player is NOT initialized');
        }

        player.onAudioStart = () => {

            console.log('playback started');

        };

        player.onAudioEnd = () => {

            console.log('playback finished');

        };

        console.log('Before AudioConfig');
        const audioConfig = SpeechSDK.AudioConfig.fromSpeakerOutput(player);

        console.log('Before SpeechConfig');
        const speechConfig = SpeechSDK.SpeechConfig.fromSubscription(this._context.parameters.SubscriptionKey.raw || "", this._context.parameters.AzureRegion.raw || "westeurope");

        speechConfig.speechSynthesisLanguage = "nl-nl";
        speechConfig.speechSynthesisVoiceName = "de-DE-FlorianMultilingualNeural";
        speechConfig.speechSynthesisOutputFormat = SpeechSDK.SpeechSynthesisOutputFormat.Audio16Khz32KBitRateMonoMp3;

        console.log('Before new SpeechSynthesizer');
        const ss = new SpeechSDK.SpeechSynthesizer(speechConfig, audioConfig);

        ss.synthesizing = (s, e) => {

            console.log('Synthesizing...');
            console.log(e);

            // Handle synthesizing event
        };

        ss.synthesisStarted = (s, e) => {

            console.log('Synthesis started');
            console.log(e);

            // Handle synthesis started event
        };

        ss.synthesisCompleted = (s, e) => {

            console.log('Synthesis completed');
            console.log(e);

            // Handle synthesis completed event
        };

        ss.SynthesisCanceled = (s, e) => {

            const cancellationDetails = SpeechSDK.CancellationDetails.fromResult(e.result);

            let str = "(cancel) Reason: " + SpeechSDK.CancellationReason[cancellationDetails.reason];

            if (cancellationDetails.reason === SpeechSDK.CancellationReason.Error) {

                str += ": " + e.result.errorDetails;

            }

            console.log('Synthesis canceled');
            console.log(e);

            // Handle synthesis canceled event
        };

        console.log('Before SpeakTextAsync');

        if (ss) {
            console.log('Synthesizer is initialized and want to say: ' + this.messageToSay);
            ss.speakTextAsync(
                this.messageToSay,
                result => {
                    if (result.reason === SpeechSDK.ResultReason.SynthesizingAudioCompleted) {
                        console.log('Result --> Synthesis finished.');
                    } else {
                        console.error('Result --> Speech synthesis canceled, ' + result.errorDetails);
                    }
                },
                error => {
                    console.error(error);
                }
            );
        } else {
            console.error('Synthesizer is NOT initialized');
        }
    }

Answer 1

Hello, thanks for reaching out to us, could you please check below steps to see if that helps?

Please Verify Audio Output Environment

• Canvas App Environment: Ensure that the Canvas app environment supports audio playback and that there are no restrictions or limitations on audio output.
• Speaker/Headphones: Make sure that your device's audio output (speakers or headphones) is configured correctly and working for other applications.

Also Check Permissions and Network Restrictions

• Ensure that there are no network restrictions or firewalls blocking outbound traffic from the Canvas app that might affect the SpeechSDK's ability to fetch or play audio.

Some Debugging Steps FYI

• Console Logs: Use browser developer tools (F12 in most browsers) to inspect console logs from your PowerApps control. Ensure that there are no errors related to audio initialization or playback.
• Player Initialization: Confirm that the player object from SpeechSDK.SpeakerAudioDestination() is correctly initialized and that there are no errors logged during its creation.

Environment and Contextual Considerations

• PCF Control Lifecycle: Understand the lifecycle of your PCF control within the Canvas app. Ensure that audio synthesis is triggered at an appropriate lifecycle stage where audio playback is supported.
• Asynchronous Operations: Ensure that asynchronous operations like audio synthesis and playback are handled correctly within the PCF control without relying on blocking operations that may not be supported in the Canvas app context.

I hope this helps, please let us know if you need more information.

Regards,

Yutong

-Please kindly accept the answer if you feel helpful to support the community, thanks.