Rescore HTK lattice

[Andreas Stolcke]

Sopheap SENG wrote:
> Hello,
>
> I need HTK lattice in my experiments but the sphinx3 decoder I used, 
> could not generate HTK lattice. So I have to convert sphinx lattice to 
> HTK lattice.
>
> My problem is : the lattice generated by sphinx3 decoder provides only 
> the acoustic score of word transitions, I did not find the option to 
> obtain the lmscore in sphninx lattice.
>
> In order to obtain HTK lattice with lmscore, first I converted sphinx 
> lattice to HTK SLF lattice format (I added l=0 as lmscore, the 
> acoustic score is  kept as it is)
>
> Then I used lattice tool (Srilm V 1.5.2) to rescore the lattice by 
> giving a LM :
>
> > lattice-tool  -in-lattice in.slf  -read-htk -lm LM.BO 
> <http://lm.bo/> -htk-lmscale 9.5 -htk-wdpenalty 0.7 -htk-logbase 
> 1.0003 -out-lattice out.slf  -write-htk
>
> (the lmscale, wdpenalty and logbase are the values that I used during 
> lattice generation with sphninx3, the LM is the same as in sphinx3)
>
> I obtained in the output a lattice with acoustic score and new 
> lmscore. What I observed is that the acoustic score in the output 
> lattice is recalculated using the logbase.
My first thought is that you have to make sure the logbase specified in 
the header of your converted sphinx lattices needs to reflect the based 
used by the actual scores.
This should be obvious, but maybe not.
>
> In order to verify that the output lattice in HTK format is equivalent 
> to the orginal sphinx lattice once, I generated 200-Best lists from 
> these two lattices.
>
> - for sphinx lattice I used sphinx3_astar to generate N-best
> - for the rescore HTK lattice, i used lattice-tool :
>
>     >lattice-tool -in-lattice out.slf  -read-htk -lm lm.BO 
> -htk-lmscale 9.5 -htk-wdpenalty 0.7 -htk-logbase 1.0003  -nbest-decode 
> 200 -out-nbest-dir OUT/
Do you have a way of generating the total (combined acoustic and lm) 
scores of the sphinx system?  then try comparing them to the 
lattice-tool output and make sure they are the same (or nearly, up to 
numerical issues).  if not then repeat the comparison for all component 
scores, but setting the weights of all by one score (including 
wedpenalty) to zero. that way you should be able to pinpoint the source 
of any discrepancy.

Note that wdpenalty is also sensitive to logbase.

Andreas