[SRILM User List] distance between two language models

[Andreas Stolcke]

On 3/14/2012 6:45 AM, L. Amber Wilcox-O'Hearn wrote:
> On Tue, Mar 13, 2012 at 8:34 PM, John Dowding<john at dowding.net>  wrote:
>> I have an application where I need to create LMs for a large number of
>> categories of text (thousands).
>>
>> I’ld like to be able to combine the LMs in cases where two (or more)
>> categories are sufficiently similar.
>>
>> Does SRILM provide a way to compute the distance between two LMs?   Is there
>> another approach I should consider?
> I would use KL divergence, or a related measure.
Exactly, but computing the KL divergence between two ngram models 
exactly would require some work.
You'd have to iterate over all ngrams occurring in either model 
(including the those handled by backoff) and sum up p1(w,h) log p2(w|h).

Of course an empirical estimate of KL divergence is easy:  to estimate 
cross-entropy you just run ngram -ppl on a sample of the source for 
model 2, computing probabilities using model 1.

Andreas
>