Is decoding RnnT with a language model supported?

[RuABraun]

I saw the script for compiling an LG, so I tried using that but got WER of 90% (greedy search gets 15) with fast_beam_search decoding (I forgot why but I got the impression that is what one should use for decoding with an LM? happy to be corrected).

Glancing around again at the documentation and the RESULT files I notice they all seem to be for decoding without an LM, which makes me wonder if maybe that's not supported yet?

Some more information about what I did:

1. Ran prepare_lang.py on lang_lm/tmp which has a lexicon.txt (contains only those words from arpa whose prons are in the lang/tokens.txt used for training) to create a words.txt, copy to lang_lm/
2. Used words.txt and the bpe.model used for training to create the BPE lang files in lang_lm/ with prepare_lang_bpe.py
3. Ran kaldilm using lang_lm/words.txt and an arpa model
4. Ran compile_lg.py

Then, after modifying decode.py to load an LG instead of a trivial graph, I run

python decode.py --decoding-method fast_beam_search --enable-spec-aug False --manifest-dir cuts --avg 1 --epoch 8 --exp-dir exp/ --bpe-model lang_lm/bpe.model --decode-graph lang_lm/LG.pt

Could I be messing something up with my approach?

edit: My model and scripts are based on librespeech/pruned_transducer_stateless2, I just noticed pruned_transducer_stateless has additional options for LG stuff from the issue #403

edit2: Using decoding_graph.scores *= ngram_lm_scale seems to have fixed the WER degradation, but I'm not getting much of an improvement (tried 0.1, 0.01). Is it to be expected that including an LM will not make a big difference for RNNT models?

[csukuangfj]

Please try the latest master.

fast_beam_search with LG decoding is updated in #420 and it has been merged into the master.

Sorry for the poor documentation about LG decoding. I am updating the documentation about it.

[RuABraun]

Thank you, no worries, I understand you guys are busy with trying stuff out! Will give it a go 👍

[RuABraun]

Works pretty well for me thank you @csukuangfj !

Do you have an idea why such a small LM scale is needed?

There has been some work on subtracting the internal LM scores from the transducer (for example) do you think this will be necessary if one wants to use an external LM with a more normal weight (~1.0)?

[danpovey]

My suspicion is that it will not solve the issue very definitively, but we could try. I have more hope for a new type of loss function, more MMI-like, without the sum-to-one normalization that RNN-T has.

[RuABraun]

Interesting. On the topic of loss functions, might you know of a resource to help understand the mutual_information_recursion function? I'm struggling to understand the connection of mutual information to aligning two sequences (also because px and py are not actually 1D sequences), in particular how one can apply mutual information (which works on prob distributions) to lists of probability distributions (the output of our model across several timesteps)

[danpovey]

The justification is quite complicated and I originally wrote the code as part of a specific way of computing sequence mutual information via an implicit alignment, that I didn't end up using. We just never got around to changing the name.. it's not really clear what else it should be called.