[SRILM User List] Assertion failed: (!Map_noKeyP(key)) in LHash.cc -Error when using segment

[Andreas Stolcke]

Eeva,

I think this bug was fixed in 2015, but is not in the beta version on 
the download server.  I need to work with people at SRI to get it 
updated.  Sorry about that.

Below  is the patch to lm/src/segment.cc.

Andreas

***************
*** 74,80 ****
   }

   inline Boolean
! Map_noKeyP(SegmentState &state)
   {
       return state == NOSTATE;
   }
--- 74,80 ----
   }

   inline Boolean
! Map_noKeyP(const SegmentState &state)
   {
       return state == NOSTATE;
   }




On 11/4/2016 7:54 AM, Eeva Nikkari wrote:
> Hi,
>
> I installed the SRILM 1.7.1 version in a linux environment and I get 
> the same error with the segment function.
>
> However, I managed to get results on both mac and linux using the 
> hidden-ngram function and hidden vocabulary with words <s> and </s>
>
> $ hidden-ngram -lm minicorpus.lm -text minicorpus.txt -hidden-vocab 
> hidden_vocab -continuous
>
> dogs chase cats </s> dogs bark </s> cats meow </s> dogs chase birds 
> </s> cats chase birds </s> dogs chase the cats </s> the birds chirp
>
>
> Thanks,
>
> Eeva
>
>
> 2016-11-03 14:39 GMT+02:00 Eeva Nikkari <eevanikkari at gmail.com 
> <mailto:eevanikkari at gmail.com>>:
>
>     Thank you for the quick answer,
>
>     I was using version 1.7.1, but updated now to 1.7.2-beta. The
>     problem persists.
>
>     $ segment -version
>     SRILM release 1.7.2-beta (with third-party contributions)
>
>     Program version @(#)$Id: segment.cc,v 1.21 2014-08-29 21:35:48
>     frandsen Exp $
>
>     Support for compressed files is included.
>     Using libLBFGS.
>
>     This software is subject to the SRILM Community Research License
>     Version
>     1.0 (the "License"); you may not use this software except in
>     compliance
>     with the License.  A copy of the License is included in the SRILM root
>     directory in the "License" file.  Software distributed under the
>     License
>     is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND,
>     either
>     express or implied.  See the License for the specific language
>     governing
>     rights and limitations under the License.
>
>     This software is Copyright (c) 1995-2014 SRI International.  All
>     rights
>     reserved.
>
>     Portions of this software are
>     Copyright (c) 2002-2005 Jeff Bilmes
>     Copyright (c) 2009-2013 Tanel Alumae
>     Copyright (c) 2012-2013 Microsoft Corp.
>
>     SRILM also includes open-source software as listed in the
>     ACKNOWLEDGEMENTS file in the SRILM root directory.
>
>     If this software was obtained under a commercial license agreement
>     with
>     SRI then the provisions therein govern the use of the software and the
>     above notice does not apply.
>
>
>
>     I still get the same error
>
>     $ segment -order 2 -lm minicorpus.lm -text minicorpus.txt
>     -continuous -debug 5
>     reading 10 1-grams
>     reading 18 2-grams
>     warning: p(w1) < p(<s> w1))
>     0: p(NOS) = 0, P(S) = 0.148148
>     1: p(NOS) = 0.111111, P(S) = 0
>     2: p(NOS) = 0.0277778, P(S) = 6.10653e-10
>     3: p(NOS) = 3.66393e-10, P(S) = 0.00793651
>     4: p(NOS) = 0.00198413, P(S) = 0
>     5: p(NOS) = 0, P(S) = 0.000566893
>     6: p(NOS) = 0.000141723, P(S) = 0
>     7: p(NOS) = 0, P(S) = 8.09848e-05
>     8: p(NOS) = 6.07386e-05, P(S) = 0
>     9: p(NOS) = 3.03693e-05, P(S) = 0
>     10: p(NOS) = 0, P(S) = 5.78463e-06
>     11: p(NOS) = 1.44616e-06, P(S) = 0
>     12: p(NOS) = 7.23079e-07, P(S) = 0
>     13: p(NOS) = 0, P(S) = 2.75459e-07
>     14: p(NOS) = 2.06594e-07, P(S) = 0
>     15: p(NOS) = 5.16485e-08, P(S) = 5.67708e-16
>     16: p(NOS) = 2.58243e-08, P(S) = 1.70313e-16
>     17: p(NOS) = 1.70313e-16, P(S) = 1.84459e-09
>     18: p(NOS) = 9.22294e-10, P(S) = 0
>     19: p(NOS) = 3.07431e-10, P(S) = 0
>     Assertion failed: (!Map_noKeyP(key)), function locate, file
>     /Users/eevanikkari/src/srilm//include/LHash.cc, line 275.
>     Abort trap: 6
>
>
>
>     Before updating the version was
>
>
>     $ segment -version
>     SRILM release 1.7.1 (with third-party contributions)
>
>     Program version @(#)$Id: segment.cc,v 1.20 2012/05/17 06:46:46
>     stolcke Exp $
>
>     Support for compressed files is included.
>     Using libLBFGS.
>
>     ...
>
>
>
>     Thank you,
>     Eeva
>
>
>     2016-11-02 20:42 GMT+02:00 Andreas Stolcke
>     <stolcke at icsi.berkeley.edu <mailto:stolcke at icsi.berkeley.edu>>:
>
>         On 11/2/2016 6:22 AM, Eeva Nikkari wrote:
>>         I get the
>>
>>         Assertion failed: (!Map_noKeyP(key)), function locate, file
>>         ../../include/LHash.cc, line 275.
>>
>>         Abort trap: 6
>>
>>         -error every time I use the segment function. I've tried with
>>         different texts and language models (different orders,
>>         smoothing and corpora). Is my model missing something? The
>>         man page says to use "standard backoff N-gram model in ARPA
>>         ngram-format(5)
>>         <http://www.speech.sri.com/projects/srilm/manpages/ngram-format.5.html>,
>>         modeling segmentation using the boundary tags <s> and </s>",
>>         which to my understanding minicorpus.lm is. I use macOS
>>         Sierra Version 10.12.1
>>
>         Eeva,
>
>         I cannot reproduce your error.  It could be a bug that's fixed
>         in the latest version. What's the output when you run "segment
>         -version"?
>
>         Andreas
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.speech.sri.com/pipermail/srilm-user/attachments/20161104/6271f543/attachment.html>