[SRILM User List] Lattice decoding problems

[Andreas Stolcke]

Stephan Gouws wrote:
> Hi,
>
> I am attempting to decode the most likely sentence, given a confusion
> network of possible words. I wrote my own script to encode this
> confusion net ("sausage") in the PFSG format, and it seems to work
> fine. However, on roughly one out of 2 cases, I get an error in the
> decoding process.
>
> I first compute the confusion net probabilities based on my own
> heuristics (standard, normalised probability scores), which are then
> converted via the formula 10000,5*log(p) given in the documentation
> into log-probs. Thereafter I rescore using a language model run in
> server mode, by passing the pfsg via stdin, as follows:
>   
Lattice-tool has the -read-mesh option which allows it to read CNs directly.
> "lattice-tool -debug 9 -in-lattice - -posterior-decode -zeroprob-word
> blemish -use-server 12345 at 127.0.0.1"
>
> However, one out of two times it breaks and I am not sure why. Here is
> sample output with debug turned on as above:
>
> server 12345 at 127.0.0.1: probserver ready
> Lattice::readPFSGs: reading in nested PFSGs...
> Lattice::readPFSG: reading in PFSG....
> Lattice::expandToLM: starting expansion to general LM (maxNodes = 0) ...
> In Lattice::removeNode: remove node 0
> In Lattice::removeNode: remove node 1
> In Lattice::removeNode: remove node 2
> In Lattice::removeNode: remove node 3
> In Lattice::removeNode: remove node 4
> In Lattice::removeNode: remove node 5
> In Lattice::removeNode: remove node 6
> In Lattice::removeNode: remove node 7
> In Lattice::removeNode: remove node 8
> In Lattice::removeNode: remove node 9
> In Lattice::removeNode: remove node 10
> In Lattice::removeNode: remove node 11
> In Lattice::removeNode: remove node 12
> In Lattice::removeNode: remove node 18
> In Lattice::removeNode: remove node 17
> In Lattice::removeNode: remove node 16
> In Lattice::removeNode: remove node 15
> In Lattice::removeNode: remove node 14
> In Lattice::removeNode: remove node 13
> In Lattice::removeNode: remove node 19
> In Lattice::removeNode: remove node 20
> Lattice::computeForwardBackward: processing (posterior scale = 8)
> Lattice::computeForwardBackward: unnormalized posterior = -6.83499
> max-min path posterior = -7.24399
>   
Up to here there are no error messages, only debugging and informational 
messages.
> And sometimes I get something like the following, which I don't
> totally understand:
>
> "WordMesh::normalizeDeletes: word posteriors exceed total: 1.695"
>   
This is from a routine that performs a sanity check on the CNs ot make 
sure the sum of all words plus the deletion (null word) posterior add up 
to 1.
It means that the word posteriors exceed 1, which shouldn't happen.

To find out why, you should find a (preferably small) test case, and 
write out the CNs that is created (write-mesh), both without and with LM 
rescoring.

Andreas
> I suspect that there might be some error in how I compute the
> log-probs, or after rescoring with the LM. However I am not sure what.
> Any ideas?
>
> Thanks
> Stephan
> _______________________________________________
> SRILM-User site list
> SRILM-User at speech.sri.com
> http://www.speech.sri.com/mailman/listinfo/srilm-user
>