[SRILM User List] unexplained failure in ngram-count

[Andreas Stolcke]

Andrew Philpot wrote:
>
> ngram-count failed on a largish file (but not the largest I was
> considering applying it to).  The file contained 325 million short
> sentences, and was 10 GB in total size.
>
> I am running on a 64-bit machine with 16GB physical memory (2 2.3 Ghz
> CPU x 4 cores) running RH EL5.
>
> The precise invocation was:
>
> ./bin/i686-m64/ngram-count -text all_lm5 -no-sos -no-eos -unk 
> -write-vocab vocab5.txt -write counts5.txt -order 5 -lm lm5.txt
>
> The precise error message was:
>
> ngram-count: /home/eh-01/philpot/srilm/include/LHash.cc:138: void 
> LHash<KeyT, DataT>::alloc(unsigned int) [with KeyT = unsigned int, 
> DataT = Trie<unsigned int, long unsigned int>]: Assertion `body != 0' 
> failed.
>
> Is this likely to be a problem with my data, with my machine's memory,
> with the code, or what?
Please check the FAQ item on large data and memory issues.

http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html

Andreas



>
> Andrew
>
> _______________________________________________
> SRILM-User site list
> SRILM-User at speech.sri.com
> http://www.speech.sri.com/mailman/listinfo/srilm-user