make-big-lm out of memory

[Andreas Stolcke]

Alexy Khrabrov wrote:
> I've got a couple corpora where make-big-lm is killed after a couple 
> days of swapping.  I've originally started it after merge-batch-counts 
> generated the total counts fine, and the kncounts file is produced OK 
> as a part of the make-big-lm run.  I've used a -max-per-file option, 
> but once the kncounts is done, I understand it no longer applies?  
> Basically trying to rerun the same command line again reuses the 
> kncounts and the thing gets killed again.  Which strategies do we have 
> to overcome this?
Increase the count cutoffs or get a machine with more memory.

This is a FAQ.  check 
http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html .

Andreas

>
> Cheers,
> Alexy