Docker container fails to run with a model trained structured text dataset

Sunmin Cho 0 Reputation points
2023-02-08T11:02:15.15+00:00

I'm testing and using a model trained with a specific dataset using a custom-speech-to-text container.

Below is the command line I used. I only change the modelId for this command.

docker run --name stt --rm -it -p 5500:5000 \
-v `pwd`/stt_model:/usr/local/models \
mcr.microsoft.com/azure-cognitive-services/speechservices/custom-speech-to-text:3.10.0-amd64 \
Punct Wfstitn \
displaylocale=ko-kr \
Eula=accept \
ModelId=*** \
Billing=*** \
ApiKey=***

When the modeId is trained on a plain text dataset, the container works.

However, when the modelId is trained on a structured text dataset, the container stops and fails to run with an error like below.

The number of list and item per list in the structured text dataset is less than 10 and 4000.

...
2023-02-08 09:45:14.172030 srbackend 255 386 info IUnidecSearchGraphCombo::Create(dyn_class_hclg(NUL,NUL))
2023-02-08 09:45:14.172249 srbackend 255 386 info IUnidecSearchGraphCombo::Create(dyn_class_hclg(NUL,NUL))
2023-02-08 09:45:14.172281 srbackend 255 386 info IUnidecSearchGraphCombo::Create(dyn_class_hclg(NUL,NUL))
Hosting environment: Production
Content root path: /rescoring
Now listening on: http://0.0.0.0:50053
Application started. Press Ctrl+C to shut down.

2023-02-08 06:30:13.825006 srbackend 263 388 info IUnidecSearchGraphCombo::Create(/usr/local/models/onlineInterpolation/hclg_spec.txt)
2023-02-08 06:30:13.834439 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS0.determinize.fsm.minhclg.hclg,CLASS0.determinize.fsm.minhclg.lms))
2023-02-08 06:30:13.867399 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS0.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:13.870908 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS10.determinize.fsm.minhclg.hclg,CLASS10.determinize.fsm.minhclg.lms))
2023-02-08 06:30:13.902752 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS10.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:13.905000 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS11.determinize.fsm.minhclg.hclg,CLASS11.determinize.fsm.minhclg.lms))
2023-02-08 06:30:13.932384 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS11.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:13.934612 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS12.determinize.fsm.minhclg.hclg,CLASS12.determinize.fsm.minhclg.lms))
2023-02-08 06:30:13.949406 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS12.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:13.951517 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS1.determinize.fsm.minhclg.hclg,CLASS1.determinize.fsm.minhclg.lms))
2023-02-08 06:30:14.019771 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS1.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:14.019848 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS2.determinize.fsm.minhclg.hclg,CLASS2.determinize.fsm.minhclg.lms))
2023-02-08 06:30:14.109388 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS2.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:14.111659 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS3.determinize.fsm.minhclg.hclg,CLASS3.determinize.fsm.minhclg.lms))
2023-02-08 06:30:14.133489 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS3.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:14.133780 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS4.determinize.fsm.minhclg.hclg,CLASS4.determinize.fsm.minhclg.lms))
2023-02-08 06:30:14.153797 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS4.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:14.154448 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS5.determinize.fsm.minhclg.hclg,CLASS5.determinize.fsm.minhclg.lms))
2023-02-08 06:30:14.171484 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS5.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:14.173269 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS6.determinize.fsm.minhclg.hclg,CLASS6.determinize.fsm.minhclg.lms))
2023-02-08 06:30:14.213133 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS6.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:14.215834 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS7.determinize.fsm.minhclg.hclg,CLASS7.determinize.fsm.minhclg.lms))
2023-02-08 06:30:14.462716 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS7.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:14.467582 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS8.determinize.fsm.minhclg.hclg,CLASS8.determinize.fsm.minhclg.lms))
2023-02-08 06:30:14.611373 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS8.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:14.628311 srbackend 263 388 info IUnidecSearchGraphCombo::Create(persisted_dyn_class_hclg(CLASS9.determinize.fsm.minhclg.hclg,CLASS9.determinize.fsm.minhclg.lms))
2023-02-08 06:30:14.732771 srbackend 263 388 info CPersistedLMLex loaded lms: CLASS9.determinize.fsm.minhclg.lms, silCost 0.8432, lmWeight 13.25
2023-02-08 06:30:14.738796 srbackend 263 388 info CPersistedLMLex loaded lms: domain.lms, silCost 0.0, lmWeight 0.0
2023-02-08 06:30:14.742387 srbackend 263 388 info Overwrite class expansion '<#CLASS7>' in base with domain class 0x5637db54b290
2023-02-08 06:30:14.747837 srbackend 263 388 info Overwrite class expansion '<#CLASS6>' in base with domain class 0x5637db566a60
2023-02-08 06:30:14.759797 srbackend 263 388 info Overwrite class expansion '<#CLASS5>' in base with domain class 0x5637db564b20
2023-02-08 06:30:14.781569 srbackend 263 388 info Overwrite class expansion '<#CLASS4>' in base with domain class 0x5637db560a10
2023-02-08 06:30:14.798813 srbackend 263 388 info Overwrite class expansion '<#CLASS2>' in base with domain class 0x5637db50c900
2023-02-08 06:30:14.804453 srbackend 263 388 info Overwrite class expansion '<#CLASS1>' in base with domain class 0x5637db5476a0
2023-02-08 06:30:14.815460 srbackend 263 388 info Overwrite class expansion '<#CLASS12>' in base with domain class 0x7f193c002f00
2023-02-08 06:30:14.827781 srbackend 263 388 info Overwrite class expansion '<#CLASS9>' in base with domain class 0x5637db53dba0
2023-02-08 06:30:14.836232 srbackend 263 388 info Overwrite class expansion '<#CLASS8>' in base with domain class 0x5637db541020
2023-02-08 06:30:14.842995 srbackend 263 388 info Overwrite class expansion '<#CLASS11>' in base with domain class 0x5637db509d50
2023-02-08 06:30:14.845190 srbackend 263 388 info Overwrite class expansion '<#CLASS3>' in base with domain class 0x5637db513340
2023-02-08 06:30:14.853730 srbackend 263 388 info Overwrite class expansion '<#CLASS10>' in base with domain class 0x5637db504a10
2023-02-08 06:30:14.854983 srbackend 263 388 info Overwrite class expansion '<#CLASS0>' in base with domain class 0x5637db5049a0
2023-02-08 06:30:14.893605 srbackend 263 388 info Domain domain.lms, weight 0.300000, #domain word 22, accumulated OOV word 7
2023-02-08 06:30:15.078926 srbackend 263 388 info class word <#CLASS0ng> has no name matched class senone
2023-02-08 06:30:15.080842 srbackend 263 388 info Pronunciation Source Stats: DLP:0 PLS:0 VENDOR_LEX:0 TN:0 LTS_SYM:0 LTS_NO_SYM:0 JA_JP_MECAB:0 JA_JP_KATAKANA:0 NOT_SET:0
2023-02-08 06:30:15.127940 srbackend 263 388 info LMWeight=13.25 in FSTB 
2023-02-08 06:30:15.128714 srbackend 263 388 info Set OOV begin word ID 1497423
2023-02-08 06:30:15.129902 srbackend 263 388 error Invalid WFST 'min(Lex)*ClassLMWithSilence(Persist())': min symbol 0, max symbol 2, state count 1, transitions count 0
2023-02-08 06:30:15.132522 srbackend 263 388 error rassert at line 1334 of /src/private/dev/unidec/src/unidec/CWFSTMinimizedSearchGraph.h: !first
2023-02-08 06:30:15.134064 srbackend 263 388 error Dead state 0 with 0 transitions in minimization, transposed=F
/opt/bin/run-decoder: line 78:   263 Aborted                 /opt/bin/unidec_grpc -- -host_string=0.0.0.0:50051 -cogs="$COGS_ENABLE" -prefault_models="$PREFAULT_MODELS"
Child pid=258 no found
Child pid=276 no found
Child pid=292 no found
Child pid=311 no found
Child pid=315 no found
Child pid=326 no found
Child pid=355 no found
./run-host: line 9:   242 Killed                  "$@"  (wd: /host)
./run-host: line 9:   311 Killed                  "$@"  (wd: /mts)
./run-host: line 9:   315 Killed                  "$@"  (wd: /diarizer)
./run-host: line 9:   276 Killed                  "$@"  (wd: /dpp)
./run-host: line 9:   292 Killed                  "$@"  (wd: /rescoring)
./run-host: line 9:   326 Killed                  "$@"  (wd: /textclassifier/app)
./run-host: line 9:   355 Killed                  "$@"  (wd: /dgs)

Any thought on how to fix this would be appreciated.

Not Monitored
Not Monitored
Tag not monitored by Microsoft.
35,924 questions
{count} votes