OpenSSL Issue When Running Azure Speech (TTS) in GKE Container

Alan Liu 0

Hey folks, I'm attempting to run an API call to Azure Speech (TTS), however, I seem to be having connection issues with OpenSSL as there seem to be cert issues when I attempt to make the connection. I've followed all of the instructions in the "Installing Speech SDK" page and the "Configuring Linux for Speech" page with more details below.

The relevant snippet of logs included here:

[508749]: 5933081ms SPX_TRACE_SCOPE_EXIT:  uws_web_socket.cpp:149 Open
[508749]: 5933081ms SPX_TRACE_INFO:  usp_connection.cpp:762 Create requestId  for messageType 0
[508749]: 5933155ms SPX_TRACE_ERROR: AZ_LOG_ERROR:  tlsio_openssl.c:691 error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
[508749]: 5933155ms SPX_TRACE_ERROR: AZ_LOG_ERROR:  tlsio_openssl.c:2441 FORCE-Closing tlsio instance.
[508749]: 5933155ms SPX_TRACE_SCOPE_ENTER:  uws_web_socket.cpp:247 OnWebSocketOpened
[508749]: 5933155ms SPX_TRACE_ERROR:  web_socket.cpp:907 WS open operation failed with result=1(WS_OPEN_ERROR_UNDERLYING_IO_OPEN_FAILED), code=2550[0x000009f6], time=2024-02-27T02:48:42.3471745Z

Setup script:

# OpenSSL 1.1.1 since 3.0 isn't supported
wget -O - https://www.openssl.org/source/openssl-1.1.1u.tar.gz | tar zxf -
cd openssl-1.1.1u
./config --prefix=/usr/local
make -j $(nproc)
make install_sw install_ssldirs
ldconfig -v 
openssl version -d # Confirmed that /etc/ssl/certs contains a whole list of certs (~600 pem files), there's a ca_certs.crt in that file as well
export SSL_CERT_DIR=/etc/ssl/certs

# Additional package upgrades
apt-get update
apt-get install build-essential libssl-dev ca-certificates libasound2 wget

Since this may be OS Specific:

# cat /etc/issue
Debian GNU/Linux 12

This is running in a container in GKE in an autopilot cluster (with their Debian based custom OS)

What I've tried:

Setting SSL_CERT_FILE to ca_certs.crt in the respective directory
Verifying that the cert dir actually contains the list of certificates
Disabling CRL (which doesn't seem to affect this step)
Confirming that I can access microsoft speech endpoints from the container
Confirming the same commit works as expected outside the container on an Ubuntu VM

Would appreciate any help to debug, thanks!

navba-MSFT 23,895 Reputation points Microsoft Employee

2024-02-27T04:46:54.8433333+00:00
@Alan Liu Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

The error message talks about the SSL socket connection failure while verifying the SSL certificate.

Plan 1:

If you have any proxy / firewall within your network, try to test your application from a different network and check if that helps.

Plan 2:

The Speech SDK depends on the following Linux system libraries:

The shared libraries of the GNU C library, including the POSIX Threads Programming library, libpthreads.

The OpenSSL library (libssl) version 1.x and certificates (ca-certificates).

The shared library for ALSA applications (libasound).

Also you need to set LD_LIBRARY_PATH with the same scope as SSL_CERT_DIR (/etc/ssl/certs) to add /usr/local/lib to the library path:

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

Plan 3:

For Debian, To use the Speech SDK in Alpine Linux, create a Debian chroot environment as documented in the Alpine Linux Wiki on running glibc programs. Then follow the Debian instructions here:

sudo apt-get update sudo apt-get install build-essential libssl-dev ca-certificates libasound2 wget

Plan 4:

Can you also try with the below ?

export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

Plan 5:

Try setting the below in your app for speech config and check if that helps:

speech_config.set_property_by_name("OPENSSL_DISABLE_CRL_CHECK", "true") speech_config.set_property_by_name("OPENSSL_SINGLE_TRUSTED_CERT", "false")

Plan 6:

If none of the above plan helps, raise an issue with Speech SDK repo it would help the SDK team to advise what could be wrong in your case. Here is a list of previous issues that you could refer and check for a possible resolution.

Hope this helps.
Alan Liu 0 Reputation points

2024-02-27T17:21:15.78+00:00
Will be cross-posting into the speech sdk repo.

No egress firewalls set up. Already tested that it's able to access speech sdk endpoints from container.

ldconfig -d already shows /usr/local/lib

That line is already in the setup script above.

It's already been tried in the section above

No effect (also in already tried)
navba-MSFT 23,895 Reputation points Microsoft Employee

2024-02-28T05:06:21.76+00:00

@Alan Liu Thanks for the update. Please raise an issue with Speech SDK repo it would help the SDK team to advise what could be wrong in your case. Here is a list of previous issues that you could refer and check for a possible resolution.

Once you get a resolution on this issue, please post it in this thread, so that other community audience can benefit from this.

Share via

OpenSSL Issue When Running Azure Speech (TTS) in GKE Container

Your answer