[SRILM User List] ngram crashing

[Andreas Stolcke]

On 3/18/2014 8:20 AM, Oldrich Kruza wrote:
>
> Hello everybody.
>
> I'm trying to reduce the vocabulary of a huge (64GB) trigram language 
> model.
>
> I ran the script change-lm-vocab, and the ngram process died with this 
> error message:
>
> include/LHash.cc:141: void LHash<KeyT, DataT>::alloc(unsigned int) 
> [with KeyT = unsigned int, DataT = Trie<unsigned int, BOnode>]: 
> Assertion `body != 0' failed.
>
> I'm positive this is not due to insufficient memory.
>
This is the error message when SRILM fails to allocate more memory. The 
reasons could be

- you are using a 32bit binary and running up against the 4GB limit of 
the architecture
- you have a memory resource limit in force (set by you or your 
sysadmin) - check the ulimit or limit (csh) command
- your system is actually, really out of memory (which also depends on 
what other users are doing)

By running top or some similar tool concurrently you can see how big 
your ngram process actually grows before crashing, and this can give you 
additional clues.

Andreas