problem in running speech container

Question

I have access to Microsoft speech containers, and I have already deployed in OCI under OKE in Oracle using the API keys given.

We are getting some API Key Errors on running the pods inside OKE@OCI

Here are the detailed logs:

[opc@bastionhost ~]$ kubectl logs flitc-azure-stt-420-amd64-enus-deployment-5c57f6d88c-9959w
/home/opc/lib/oracle-cli/lib64/python3.6/site-packages/oci/_vendor/httpsig_cffi/sign.py:10: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
  from cryptography.hazmat.backends import default_backend  # noqa: F401
========================= ENVIRONMENT VARS ===========================================
EULA:              accept
BILLING:           https://eastus.api.cognitive.microsoft.com/sts/v1.0/issuetoken
APIKEY:            ***
HTTP_PROXY:
HTTP_PROXY_CREDS:
CUSTOM ARGS:
========================= ARGUMENT VARS ==============================================
/host /onprem_start
Starting dotnet
max number of threads 32767, 1000
min number of threads 16, 16
New min number of threads 20, 16


EULA Notice: Copyright © Microsoft Corporation 2020. This Cognitive Services Container image is made available to you under the terms [https://go.microsoft.com/fwlink/?linkid=2018657] governing your subscription to Microsoft Azure Services (including the Online Services Terms [https://go.microsoft.com/fwlink/?linkid=2018760]). If you do not have a valid Azure subscription, then you may not use this container.


Using API key for authentication.
Logging to console.
Submitting metering to 'https://eastus.api.cognitive.microsoft.com/sts/v1.0/issuetoken'.
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
      Storing keys in a directory '/nonexistent/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured. Key {44304c38-86ab-446c-a5d7-a2dfc554f773} may be persisted to storage in unencrypted form.
warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Overriding address(es) 'http://+:80'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
Hosting environment: Production
Content root path: /host
Now listening on: http://0.0.0.0:5000
Application started. Press Ctrl+C to shut down.
Child pid=123 created
/onprem_start
Starting /opt/bin/run-decoder
==========================================
Running an instance of gRPC based Richland
==========================================

=======================
Checking ENV variables:
=======================

DECODER_COUNT_AUTO is unset
DECODER_COUNT is unset
DECODER_MAX_COUNT is unset
ENGINE_CONFIG_OVERRIDE is set to '/usr/local/models/properties.json'
MODELSPATH is set to '/usr/local/models'
MODELSPATH will be set to: '/usr/local/models'
SSL_ENABLE is unset
SSL_SERVER_CERT_PATH is unset
SSL_SERVER_PRIVATE_KEY_PATH is unset
SSL_ROOT_CERT_PATH is unset
SSL_FORCE_CLIENT_AUTH is unset
STATSD_HOST is unset
STATSD_PORT is unset (will use default port 8125)
STATSD_METRIC_PREFIX is unset
COGS_ENABLE is set
PREFAULT_MODELS is set
SERVICE_HOSTNAME is set to: 'flitc-azure-stt-420-amd64-enus-deployment-5c57f6d88c-9959w'
cat: opt/bin/version.txt: No such file or directory
GRPC_BUILD_VERSION is set to
====================
Starting gRPC server
====================

2023-10-10 11:36:17.407321 srbackend 163 163 info
                  _      _     _                 _
                 (_)    | |   | |               | |
             _ __ _  ___| |__ | | __ _ _ __   __| |
            | '__| |/ __| '_ \| |/ _` | '_ \ / _` |
            | |  | | (__| | | | | (_| | | | | (_| |
            |_|  |_|\___|_| |_|_|\__,_|_| |_|\__,_|

2023-10-10 11:36:17.407375 srbackend 163 163 info log_level set to 2 ('info')
2023-10-10 11:36:17.407376 srbackend 163 163 info Version: 00000000 2023-05-30 14:33:05
10-10-2023 11:36:17.407 southpool 163 163 info initializing service host starter
10-10-2023 11:36:17.407 southpool 163 163 info loading server properties
10-10-2023 11:36:17.407 southpool 163 163 info model prefaulting status: false
10-10-2023 11:36:17.407 southpool 163 163 info engine_config path overridden by environment variable ENGINE_CONFIG_OVERRIDE: /usr/local/models/properties.json
10-10-2023 11:36:17.407 southpool 163 163 info engine_config path: /usr/local/models/properties.json
2023-10-10 11:36:17.407426 srbackend 163 163 info DefaultModelPath (evaluated MODELSPATH): /usr/local/models/
2023-10-10 11:36:17.407442 srbackend 163 163 info Using /usr/local/models/builtin-entities as built in entity path
2023-10-10 11:36:17.407444 srbackend 163 163 info Could not find IN_mainRecognizerINI
2023-10-10 11:36:17.407445 srbackend 163 163 info Defaulting to richland ini here: %MODELSPATH%richland.ini
2023-10-10 11:36:17.407461 srbackend 163 163 info richland config file: /usr/local/models/richland.ini exists and will be used
2023-10-10 11:36:17.408219 srbackend 163 163 info IniParser::Read(/usr/local/models/richland.ini)
2023-10-10 11:36:17.408598 srbackend 163 163 info [uint] beam-size:5000
2023-10-10 11:36:17.408619 srbackend 163 163 info [flt ] beam-threshold:170.0
2023-10-10 11:36:17.408642 srbackend 163 163 info [str ] confidence:/usr/local/models/ccb1033.bin
2023-10-10 11:36:17.408659 srbackend 163 163 info [uint] decoder-silence-threshold:1500
2023-10-10 11:36:17.408832 srbackend 163 163 info [str ] dnn-spec:onnxHalide(/usr/local/models/model.onnx.bin.prod,28,160,input,likelihood,2.5.4),outputDelayFrameCount(0),frameCopyCount(1),resetOnSegmentation(1),allValidInUtt(1)
2023-10-10 11:36:17.408849 srbackend 163 163 info [uint] dynbeam-growth-mode:2
2023-10-10 11:36:17.408863 srbackend 163 163 info [flt ] dynbeam-growth-rate:1.044
2023-10-10 11:36:17.408918 srbackend 163 163 info [str ] fe-spec:audio(/usr/local/models/c1033.fe,8kHz16kHzLFB80x2EnergyMLPVADRuntime,/usr/local/models/smartvadthresh.tsv)
2023-10-10 11:36:17.408935 srbackend 163 163 info [str ] hcb:/usr/local/models/hcb1033.bin
2023-10-10 11:36:17.408950 srbackend 163 163 info [str ] hcb-clm:/usr/local/models/hcb1033_clm.bin
2023-10-10 11:36:17.408973 srbackend 163 163 info [str ] hcb-expanded-class:/usr/local/models/hcb1033_expanded_class.bin
2023-10-10 11:36:17.409007 srbackend 163 163 info [str ] hcb-word-boundary:/usr/local/models/hcb1033_word_boundary.bin
2023-10-10 11:36:17.409017 srbackend 163 163 info [str ] ini-path:/usr/local/models/richland.ini
2023-10-10 11:36:17.409031 srbackend 163 163 info [str ] lexicon:/usr/local/models/lsr1033.lxa
2023-10-10 11:36:17.409046 srbackend 163 163 info [str ] list:list011033
2023-10-10 11:36:17.409056 srbackend 163 163 info [flt ] lm-weight:13.25
2023-10-10 11:36:17.409063 srbackend 163 163 info [uint] locale-id:1033
2023-10-10 11:36:17.409075 srbackend 163 163 info [str ] lts:/usr/local/models/che1033.lxa
2023-10-10 11:36:17.409093 srbackend 163 163 info [str ] lts-spell:/usr/local/models/ltsspell1033
2023-10-10 11:36:17.409106 srbackend 163 163 info [str ] mmf:mmf011033.cdbn
2023-10-10 11:36:17.409116 srbackend 163 163 info [uint] nbest-beam-size:100
2023-10-10 11:36:17.409130 srbackend 163 163 info [str ] phn:/usr/local/models/L1033.phn
2023-10-10 11:36:17.409150 srbackend 163 163 info [str ] senone-to-phone-map:/usr/local/models/senone2phone1033
2023-10-10 11:36:17.409162 srbackend 163 163 info [flt ] silence-cost:0.8432
2023-10-10 11:36:17.409173 srbackend 163 163 info [str ] tn:/usr/local/models/tn1033.bin
2023-10-10 11:36:17.409205 srbackend 163 163 info IN_DNNSpec: onnxHalide(/usr/local/models/model.onnx.bin.prod,28,160,input,likelihood,2.5.4),outputDelayFrameCount(0),frameCopyCount(1),resetOnSegmentation(1),allValidInUtt(1)
2023-10-10 11:36:17.409300 srbackend 163 163 info IN_ModelMap[baseModel]: type=Static, path='/usr/local/models/graphV2', interpolated_base_path='/usr/local/models/onlineInterpolation', hclg='dyn_comp(prune.1e-06.minhclg.hclg,prune.1e-06.minhclg.lms)', base_hclg='', interpolated_base_hclg='expanded_class_hclg_interpolated_lm_base(NUL,NUL,clm.minhclg.hclg,clm.minhclg.lms,base.lms,class_map.txt)', interpolated_default_hclg='', weight='0.0', default='true'
2023-10-10 11:36:17.409352 srbackend 163 163 info IN_ModelMap[topLevelClm]: type=BaseClm, path='/usr/local/models/dgi', interpolated_base_path='', hclg='expanded_class_hclg()', base_hclg='expanded_class_hclg_base(NUL,NUL,clm.minhclg.hclg,clm.minhclg.lms,class_map.txt)', interpolated_base_hclg='', interpolated_default_hclg='', weight='0.0', default='false'
10-10-2023 11:36:17.409 southpool 163 163 info About to start gRPC server @ 0.0.0.0:50051
10-10-2023 11:36:17.409 southpool 163 163 info gRPC server decoder pool MIN count: 1
10-10-2023 11:36:17.409 southpool 163 163 info gRPC server decoder pool MAX count: 10
10-10-2023 11:36:17.409 southpool 163 163 info gRPC IPv4 exclusive mode
10-10-2023 11:36:17.409 southpool 163 163 info Memory trimming (malloc_trim) enabled.
10-10-2023 11:36:17.409 southpool 163 163 info service host starter initialized
10-10-2023 11:36:17.410 southpool 163 163 info cpuinfo: {"TLB size":"1024 4K pages","address sizes":"40 bits physical, 48 bits virtual","bogomips":"4491.56","bugs":"sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass retbleed smt_rsb","cache size":"512 KB","cache_alignment":"64","clflush size":"64","cpu MHz":"2245.780","cpu cores":"8","cpu family":"23","cpuid level":"16","flags":"fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 nt_good clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities","fpu":"yes","fpu_exception":"yes","microcode":"0x1000065","model":"49","model name":"AMD EPYC 7742 64-Core Processor","physical id":"0","processors count":"16","siblings":"16","stepping":"0","vendor_id":"AuthenticAMD","wp":"yes"}
2023-10-10 11:36:17.411507 srbackend 163 163 info Loaded CoreCLR from /opt/bin/libcoreclr.so
2023-10-10 11:36:17.435702 srbackend 163 163 info CoreCLR started


EULA Notice: Copyright © Microsoft Corporation 2020. This Cognitive Services Container image is made available to you under the terms [https://go.microsoft.com/fwlink/?linkid=2018657] governing your subscription to Microsoft Azure Services (including the Online Services Terms [https://go.microsoft.com/fwlink/?linkid=2018760]). If you do not have a valid Azure subscription, then you may not use this container.


Using API key for authentication.
Logging to console.
Submitting metering to 'https://eastus.api.cognitive.microsoft.com/sts/v1.0/issuetoken'.
10-10-2023 11:36:17.615 southpool 163 163 info creating cached engine instance
2023-10-10 11:36:17.615978 srbackend 163 163 info Processing onnxHalide spec
File: /usr/local/models/model.onnx.bin.prod, length: 84305509
File: Start API connection
Child pid=159 created
/dpp /onprem_start
Starting dotnet
[Informational] dotnet starting...
File: Timed out, retry #1
File: Start API connection
Child pid=178 created
/onprem_start
/rescoring /onprem_start
Starting dotnet
[Informational] dotnet starting...


EULA Notice: Copyright © Microsoft Corporation 2020. This Cognitive Services Container image is made available to you under the terms [https://go.microsoft.com/fwlink/?linkid=2018657] governing your subscription to Microsoft Azure Services (including the Online Services Terms [https://go.microsoft.com/fwlink/?linkid=2018760]). If you do not have a valid Azure subscription, then you may not use this container.


Logging to console.
Submitting metering to 'https://eastus.api.cognitive.microsoft.com/sts/v1.0/issuetoken'.
Child pid=207 created
/onprem_start
/mts /onprem_start
Starting ./MediaTransformService
[libprotobuf ERROR /tmp/dependencies/grpc/third_party/protobuf/src/google/protobuf/descriptor_database.cc:641] File already exists in database: google/protobuf/compiler/plugin.proto
[libprotobuf FATAL /tmp/dependencies/grpc/third_party/protobuf/src/google/protobuf/descriptor.cc:1936] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
File: Timed out, retry #2
238: 2023-10-10 11:36:21.068 [I] [main] setting up unrecoverable error handlers
238: 2023-10-10 11:36:21.068 [I] [main] setting up external library logging
238: 2023-10-10 11:36:21.068 [I] [main] initializing prometheus client
238: 2023-10-10 11:36:21.068 [I] [main] initializing grpc server
238: 2023-10-10 11:36:21.074 [I] [main] setting up asynchronous signal processing
238: 2023-10-10 11:36:21.074 [I] [main] setting up application monitoring
238: 2023-10-10 11:36:21.074 [I] [main] entering I/O loop
File: Start API connection
Child pid=238 created
/onprem_start
/diarizer/app /onprem_start
Starting dotnet


EULA Notice: Copyright © Microsoft Corporation 2020. This Cognitive Services Container image is made available to you under the terms [https://go.microsoft.com/fwlink/?linkid=2018657] governing your subscription to Microsoft Azure Services (including the Online Services Terms [https://go.microsoft.com/fwlink/?linkid=2018760]). If you do not have a valid Azure subscription, then you may not use this container.


Logging to console.
Submitting metering to 'https://eastus.api.cognitive.microsoft.com/sts/v1.0/issuetoken'.
[2023-10-10T11:36:21.8337512Z],[Information]: [AddGrpcClientFactory] inputs: southpawEnabled False, providenceEnabled False, providenceUri     {"memberName":"AddGrpcClientFactory","sourceFilePath":"D:\dbs\el\ihdt\PrincetonFE\private\PrincetonMeeting\CTS.Grpc\ServiceCollectionExtensions.cs","sourceLineNumber":"97"}
warn: DefaultBypassScrubberFactory[0]
      ECS DataCollectionMode allowed endpoints config was read on service startup. There are zero endpoints in the allowed endpoints collection.
info: Microsoft.CloudAI.Containers.Metering.CloudMeterClient[0]
      Reporting meter usage of 0 units at 638325345820264245.
[2023-10-10T11:36:22.1073754Z],[Information]: GRPC ChannelOptions: grpc.keepalive_permit_without_calls:1,grpc.http2.min_ping_interval_without_data_ms:100  {"memberName":".ctor","sourceFilePath":"D:\dbs\el\ihdt\PrincetonFE\private\CTSDiarizerService\CTSDiarizer.Server\DiarizerServer.cs","sourceLineNumber":"67"}
Child pid=273 created
/onprem_start
/dgs /onprem_start
Starting dotnet
[Informational] max number of threads 32767, 1000
[Informational] min number of threads 16, 16
[Informational] New min number of threads 20, 16
[Informational] dotnet starting...
===> Logging context is console
===> Environment name is Local
Child pid=317 created
/onprem_start
File: Timed out, retry #3
Failed with message: File: Validate API key, billing endpoint and check container connection to cognitive sevices., exception: System.Exception: File: Validate API key, billing endpoint and check container connection to cognitive sevices.
   at Microsoft.CloudAI.Containers.NativeWrapper.CognitiveServicesWrapper.LoadData(String filename, IntPtr data, UInt64 length) in /build/CognitiveServicesWrapper/CognitiveServicesWrapper.cs:line 146
/opt/bin/run-decoder: line 78:   163 Segmentation fault      (core dumped) /opt/bin/unidec_grpc -- -host_string=0.0.0.0:50051 -cogs="$COGS_ENABLE" -prefault_models="$PREFAULT_MODELS"
Method: LoadFiles Started loading universal ngram list
Method: LoadFiles Completed loading universal ngram list
Hosting environment: Production
Content root path: /dgs
Now listening on: http://[::]:50055
Application started. Press Ctrl+C to shut down.
Child pid=159 no found
Child pid=178 no found
Child pid=207 no found
Child pid=238 no found
Child pid=273 no found
Child pid=317 no found
./run-host: line 9:   123 Killed                  "$@"  (wd: /host)
./run-host: line 9:   238 Killed                  "$@"  (wd: /mts)
./run-host: line 9:   178 Killed                  "$@"  (wd: /dpp)
./run-host: line 9:   207 Killed                  "$@"  (wd: /rescoring)
./run-host: line 9:   273 Killed                  "$@"  (wd: /diarizer/app)
./run-host: line 9:   317 Killed                  "$@"  (wd: /dgs)
[opc@bastionhost ~]$ kubectl get pod

Accepted Answer

@Mohammed Salama

Thanks for reaching out to us, based on the logs you provided, it seems like there is an issue with the API key. The error message "Failed with message: File: Validate API key, billing endpoint and check container connection to cognitive services" indicates that the API key is not valid or there is an issue with the billing endpoint.

Please make sure that you have provided the correct API key and billing endpoint. You can also check if the container is able to connect to the cognitive services by testing the connection from inside the container.

To test the network connectivity from inside a container, you can use a few different methods. One common way to check connectivity is by using command-line tools like ping, curl, or wget. Here's how you can test the connection to an endpoint URL from inside a container:

Ping:

You can use the ping command to check if a remote server is reachable. For example, to test connectivity to a website like www.google.com, you can run:

YAMLAI ConvertCopy

Copy code
ping www.google.com

Curl:

If you want to test HTTP/HTTPS connectivity, you can use the curl command. For example, to check if you can reach a specific URL, run:

YAMLAI ConvertCopy

arduinoCopy code
curl http://example.com

or for HTTPS:

YAMLAI ConvertCopy

arduinoCopy code
curl https://example.com

Wget:

Similar to curl, you can use the wget command to download a file from a URL. For example:

YAMLAI ConvertCopy

arduinoCopy code
wget http://example.com/somefile.txt

These commands will help you determine whether the container can reach the external endpoint and receive a response. If you're testing connectivity to a specific Azure Cognitive Services endpoint, replace the placeholder URL with the actual endpoint URL you are trying to access.

For your specific case, where you're encountering API key errors, you can test the connection to the Azure Cognitive Services billing endpoint by using curl or wget with the billing endpoint URL provided in your error logs. For example:

YAMLAI ConvertCopy

bashCopy code
curl https://eastus.api.cognitive.microsoft.com/sts/v1.0/issuetoken

If you receive a response, it means that the container can access the Azure Cognitive Services billing endpoint. If you encounter errors or cannot connect, there might be network or configuration issues that need to be addressed.

Keep in mind that some organizations may have network restrictions or firewalls in place that can affect outbound network traffic from containers. If you suspect that network policies are causing issues, consult with your network or infrastructure team to ensure proper access.

Testing connectivity using these commands should help you identify any issues related to network access from within the container.

I hope this helps.

Regards,

Yutong

-Please kindly accept the answer if you feel helpful to support the community, thanks a lot.

Answer

Thanks @YutongTie-MSFT , the problem was from oe=racle OCI itself, we contacted them and its solved.

Share via

problem in running speech container

1 additional answer