[SRILM User List] Interpolating LMs with different smoothing

[Andreas Stolcke]

This is as expected.  You have two estimators (of conditional word 
probabilities, i.e., LMs), each with random deviations from the true 
probabilities.  By averaging their predictions you reduce the deviation 
from the truth (assuming the deviations are randomly distributed).

For this reason you can almost always get a win out of interpolating 
models that are approximately on par in their individual performance.  
Other examples are

- random forest models
- sets of neural LMs initialized with different initial random weights
- log-linear combination of forward and backward running LMs
- sets of LMs trained on random samples from the same training set

These techniques all reduce the "variance" part of the modeling error 
<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FBias%25E2%2580%2593variance_tradeoff&data=01%7C01%7Csrilm-user%40speech.sri.com%7Cbf1d15c6a918412c407c08d5ecd93a1b%7C40779d3379c44626b8bf140c4d5e9075%7C1&sdata=QOmL3M1bKDrphq%2FbwLkFVSVkqCxvLFOg3ibgx5FaAHw%3D&reserved=0>.  Other 
techniques (like interpolating models trained on different genres) do a 
similar thing for the "bias"  part of the error.

Andreas

On 7/17/2018 9:22 PM, Fed Ang wrote:
> Hi,
>
> I don't know if it has been asked before, but does it make sense to 
> interpolate on the basis of smoothing instead of domain/genre?  What 
> should be the assumptions in considering this when the resulting 
> perplexity is lower than any of the two separately?
>
> Let's say: 5-gram Katz yields 100, and 5-gram Modified KN yields 90
> Then best-mix of the two yields 87
>
> On a theoretical perspective, is it sound to simply trust that the 
> interpolated LM is better/generalizable to different smoothing 
> combinations?
>
> -Fred
>
>
> _______________________________________________
> SRILM-User site list
> SRILM-User at speech.sri.com
> http://mailman.speech.sri.com/cgi-bin/mailman/listinfo/srilm-user


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.speech.sri.com/pipermail/srilm-user/attachments/20180718/dd412370/attachment.html>