lattice-tool to rescore a lattice

[Stephane Huet]

Hi,

By rescoring lattices in HTK format with a 4-gram LM with the following 
options:
   lattice-tool -in-lattice <IN> -read-htk -out-lattice <OUT> -write-htk 
-order 4 -lm <LM>,
I noticed that the LM scores written in the output lattice were 
sometimes different from what was expected.

In the lattice-tool manpages (I use SRILM 1.5.0), I read that the 
algorithm by default for lattice-expansion is "General LM expansion" and 
expands the lattice "without use of backoff transitions". Does this mean 
that during LM expansion, no backoff is taken into account in the LM 
probabilities?
However, by investigating in the source code, I noticed that the 
following line of Lattice::expandAddTransition function in LatticeExpand.cc:
   transProb += lm.contextBOW(usedContext, usedLength);
includes backoff transition.

To get the expected linguistic scores when expanding the lattice by the 
LM, I put the previous line in comment and take into account the LM 
back-off by modifying the  lm.contextID(nextWord, usedContext, 
usedLength2) call in Lattice::expandNodeToLM function of 
LatticeExpand.cc. Indeed, I noticed that when lm.contextID returns the 
LM order instead of what he originally did, the context of the 
conditional probability is not anymore truncated and the LM scores of 
the output lattices are coherent with what I expected.

There may be options that I didn't understand to rescore lattices with a 
LM but I find strange the LM scores processed by lattice-tool. I can 
send the files I modified if you want.

Regards,

Stéphane