[SRILM User List] Interpreting ngram -ppl output in case of backoff

[Sander Maijers]

On 05-06-13 07:58, Andreas Stolcke wrote:
> On 6/1/2013 6:00 AM, Sander Maijers wrote:
>> Andreas, you wrote:
>> > If the word is 'a'  and the last two words 'b' and 'c' (in that 
>> order), and you  have a bigram hit (output says '[2gram]' ), you'd 
>> have to look up the bigram log probability for 'a c' and add to that 
>> the backoff weight for 'b c'.
>> I interpret this as P(a | b c). If that is correct, then shouldn't I 
>> actually look up the line in the ARPA LM with "c a" (reverse)?
> You are correct, the backoff bigram would be  'c a' .
>> Can you further comment to clear up my remaining confusion please ...
>>
>> 1. The word "Albert" is not in the word list/vocabulary. Nor are 
>> there any N-grams with "Albert". This confuses me. I cannot trace the 
>> appropriate N-gram that lead to the logprob that was reported by 
>> 'ngram'. It seems that I cannot directly see in the 'ngram' output 
>> if/when there had been any backing off during the LM lookups for this 
>> sentence. I assume that the word Albert was actually replaced with 
>> [unk], but in the ngram output, such is not displayed. There also 0 
>> OOVs reported, which strikes me as odd. All in all I believe that the 
>> following N-grams were looked up:
>>
>> -1.358477    <s> dat    -0.6622628
>> -1.334724    dat [unk]    -0.0686222
>> -0.6776069    dat [unk] [unk]
>>
>> for the 'n gram' output
>>
>> dat Albert Heijn het doet zou niet de aanleiding zijn
>>    p( dat | <s> ) = 0.0438046 [ -1.35848 ]
>>    p( Albert | dat ...) = 0.0100695 [ -1.99699 ]
>>    p( Heijn | Albert ...) = 0.210084 [ -0.677607 ]
>>
>> How did 'ngram' come to the -1.99699 logprob?
>
> I'm not sure how exactly you are invoking ngram.   But if your ppl 
> output shows the original words and not the [unk]  word that means 
> your OOV words were NOT mapped to [unk].
> Also, I'm confused by that had that your -debug 2 output doesn't 
> indicate what order of ngram was hit.
>
> For reference, if you invoke ngram on the 3gram model that is packaged 
> with SRILM:
>
>     % ngram -unk -map-unk '@reject@' -lm 
> $SRILM/lm/test/tests/ngram-count-gt/swbd.3bo.gz -debug 2 -ppl -
>
> and feed two sentences
>
>     hi there
>     hix there
>
> the output should look like this:
>
> hi there
>         p( hi | <s> )   = [2gram] 0.000625719 [ -3.20362 ]
>         p( there | hi ...)      = [3gram] 0.0131176 [ -1.88215 ]
>         p( </s> | there ...)    = [2gram] 0.137458 [ -0.86183 ]
> 1 sentences, 2 words, 0 OOVs
> 0 zeroprobs, logprob= -5.9476 ppl= 96.0578 ppl1= 941.453
>
> hix there
>         p( @reject@ | <s> )     = [2gram] 0.00733392 [ -2.13466 ]
>         p( there | @reject@ ...)        = [3gram] 0.00498615 [ -2.30223 ]
>         p( </s> | there ...)    = [2gram] 0.0603144 [ -1.21958 ]
> 1 sentences, 2 words, 0 OOVs
> 0 zeroprobs, logprob= -5.65648 ppl= 76.8232 ppl1= 673.347
>
> Notice the [2gram] [3gram] indicators, and notice how the OOV "hix" is 
> mapped to the unknown word token "@reject@" (that's the label that 
> this particular model uses).
>
> So you must be invoking ngram in a difference way and that may have 
> something to do with the problems you have interpreting the output.
>
> Andreas
Yes, there is confusion here. I did not specify the complete command 
line that I used to invoke 'ngram', only the options that I believed 
were relevant. Actually, I used the "-use-server" parameter (for the 
ngram client invocation) ... I haven't read that the debug output is 
more limited in this sense if you use the 'ngram' server. And I had 
specified the same debug level with both the "'ngram' -server-port ..." 
and "ngram -use-server ..." invocations) .

Still, there are no logprobs of "-1.99699" in my LM, so I do not know 
where that comes from. (I tried to look for the values in the ARPA LM 
with fewer decimals: the 'ngram' output logprobs and the ARPA logprobs 
are not exactly the same in my output). There simply must be OOVs in 
that "Albert Heijn" sentence but they weren't detected, it seems? Could 
this be problem that has to do with the 'ngram' server mode?

If you are interested in it, I can give you the relevant ARPA LM and the 
command lines used for a test case.

Sander