Kullback Leibler

[Andreas Stolcke]

In message <1594.145.116.14.61.1181339193.squirrel at webmail.science.uva.nl>you w
rote:
> Dear Andreas Stolcke,
> 
> just a very small question for curiosity. Is there already some tool
> included in the SRILM toolkit to calculate the Kullback Leibler divergence
> between two LMs?

No. it would be cool to have such a tool.

For certain models (e.g., ngram models)
you could come up with an exact computation, similar to what the pruning
algorithm uses.

For the general case you could sample from one of the LMs and 
compute an empirical cross-entropy.

Feel free to implement something ...

Andreas