[SRILM User List] Lattice decoding problems

[Stephan Gouws]

Hi,

I am attempting to decode the most likely sentence, given a confusion
network of possible words. I wrote my own script to encode this
confusion net ("sausage") in the PFSG format, and it seems to work
fine. However, on roughly one out of 2 cases, I get an error in the
decoding process.

I first compute the confusion net probabilities based on my own
heuristics (standard, normalised probability scores), which are then
converted via the formula 10000,5*log(p) given in the documentation
into log-probs. Thereafter I rescore using a language model run in
server mode, by passing the pfsg via stdin, as follows:

"lattice-tool -debug 9 -in-lattice - -posterior-decode -zeroprob-word
blemish -use-server 12345 at 127.0.0.1"

However, one out of two times it breaks and I am not sure why. Here is
sample output with debug turned on as above:

server 12345 at 127.0.0.1: probserver ready
Lattice::readPFSGs: reading in nested PFSGs...
Lattice::readPFSG: reading in PFSG....
Lattice::expandToLM: starting expansion to general LM (maxNodes = 0) ...
In Lattice::removeNode: remove node 0
In Lattice::removeNode: remove node 1
In Lattice::removeNode: remove node 2
In Lattice::removeNode: remove node 3
In Lattice::removeNode: remove node 4
In Lattice::removeNode: remove node 5
In Lattice::removeNode: remove node 6
In Lattice::removeNode: remove node 7
In Lattice::removeNode: remove node 8
In Lattice::removeNode: remove node 9
In Lattice::removeNode: remove node 10
In Lattice::removeNode: remove node 11
In Lattice::removeNode: remove node 12
In Lattice::removeNode: remove node 18
In Lattice::removeNode: remove node 17
In Lattice::removeNode: remove node 16
In Lattice::removeNode: remove node 15
In Lattice::removeNode: remove node 14
In Lattice::removeNode: remove node 13
In Lattice::removeNode: remove node 19
In Lattice::removeNode: remove node 20
Lattice::computeForwardBackward: processing (posterior scale = 8)
Lattice::computeForwardBackward: unnormalized posterior = -6.83499
max-min path posterior = -7.24399

And sometimes I get something like the following, which I don't
totally understand:

"WordMesh::normalizeDeletes: word posteriors exceed total: 1.695"

I suspect that there might be some error in how I compute the
log-probs, or after rescoring with the LM. However I am not sure what.
Any ideas?

Thanks
Stephan