Optimizing Weights in Log-Linear Interpolation

[Andreas Stolcke]

Sibel Yaman wrote:
>
> Hello,
> I was wondering how I can train the weights in log-linear 
> interpolation of several language models (as in Klakow's paper).
>
There isn't such a tool in SRILM, sorry. You would have to implement a 
gradient descent type optimization, but it's going to be slow due to the 
normalization term.

Optimizing the linear interpolation weight and then using that in the 
log-linear model might give decent results in many cases.
I'd be interested to hear from others on this list what they do.

Andreas

> I have successfully used "compute-best-mix" script to use linear 
> interpolation weights but do not see how to modify the process to 
> optimize log-linear interpolation weights so that the perplexity is 
> minimized on a cross-validation set.
>
> Thank you,
> Sibel Yaman
> *From:* Andreas Stolcke <stolcke at ADDRESS HIDDEN>
> *Date:* Mon, 18 Jul 2005 07:28:40 PDT
> In message <32809.213.58.88.69.1081673875.squirrel at ADDRESS 
> HIDDEN>you wrote:
> >
> > Hi!
> >
> > Does anyone know a program or toolkit allowing to do log-linear
> > interpolation of different language models? since SRILM only permit 
> to do
> > linear interpolation.
> > Thanks for your help,
> >
> > Ciro Martins
>
> Ciro,
>
> sorry for the late response ;-)
>
> There is now, in the current development version of SRILM, an
> implementation of log-linear interpolation. The class name is
> LoglinearMix, and the ngram -loglinear-mix option triggers its use.
> Note that log-linear interpolation is much slower to evaluate than
> linear interpolation, due to the need to normalize the combined LM.
> This is done somewhat efficiently in SRILM by caching the normalizers
> for previously seen contexts.
>
> You might also want to try using log-linear combination of LM scores
> without normalization. This can be done in the nbest or lattice
> rescoring framework implemented by the toolkit, simply by computing
> scores from multiple LMs.
>
> The latest version of the toolkit can by downloaded in the usual way
> by choosing the "1.4.5 (Beta)" version in the web form.
>
> --Andreas
>
> Click _here_ <file:///projects/srilm/> to go to the SRILM home page.
>