[SRILM User List] -posterior-decode vs -viterbi-decode

[Andreas Stolcke]

On 1/25/2010 9:56 AM, Ilana Heintz wrote:
> Hello Andreas et al,
>
> I have installed version 1.5.10 successfully. (I ran make test, for 
> which disambig, make-ngram-pfsg, merge-batch-counts, 
> nbest-optimize-bleu, and ngram-server have differing output, 
> everything else Ok).
>
> But I am still having trouble with lattice-tool.  It's true that 
> -posterior-decode now takes into account the new LM before decoding, 
> so I'm getting different answers with different order LMs. But, now I 
> can't seem to get -viterbi-decode to work with an LM at all...
>
> lattice-tool -read-htk -in-lattice $lattice -posterior-decode
> --- gives 1-best output
> lattice-tool -read-htk -in-lattice $lattice -posterior-decode -lm $lm
> --- gives differing 1-best output
> lattice-tool -read-htk -in-lattice $lattice -viterbi-decode
> --- gives 1-best output
> lattice-tool -read-htk -in-lattice $lattice -viterbi-decode -lm $lm
> --- gives _no_ output
>
> Can you replicate this?  Am I missing an option or flag?
>
Ilana,

the problem is caused by the presence of explicit start and 
end-of-sentence tags in your lattice (<s> and </s>).  These are already 
assumed to be attached to the initial / final nodes of the lattice, so 
you effectively have them duplicated.  As a result the LM rescoring 
fails because the bigram <s> <s> has probability zero.

However, I realize this might be common situation, and should be handled 
sensibly. The solution (for now at least) is that the HTK lattice 
reading code will map <s> and </s> to null labels.   This is now in the 
new beta version that you can download.
Alternatively, you could edit your lattices and replace these tags with 
!NULL everywhere, and use the version you already have.

Andreas