[SRILM User List] Interpreting ngram -ppl output in case of backoff

[Andreas Stolcke]

On 6/1/2013 6:00 AM, Sander Maijers wrote:
> Andreas, you wrote:
> > If the word is 'a'  and the last two words 'b' and 'c' (in that 
> order), and you  have a bigram hit (output says '[2gram]' ), you'd 
> have to look up the bigram log probability for 'a c' and add to that 
> the backoff weight for 'b c'.
> I interpret this as P(a | b c). If that is correct, then shouldn't I 
> actually look up the line in the ARPA LM with "c a" (reverse)?
You are correct, the backoff bigram would be  'c a' .


>
> Can you further comment to clear up my remaining confusion please ...
>
> 1. The word "Albert" is not in the word list/vocabulary. Nor are there 
> any N-grams with "Albert". This confuses me. I cannot trace the 
> appropriate N-gram that lead to the logprob that was reported by 
> 'ngram'. It seems that I cannot directly see in the 'ngram' output 
> if/when there had been any backing off during the LM lookups for this 
> sentence. I assume that the word Albert was actually replaced with 
> [unk], but in the ngram output, such is not displayed. There also 0 
> OOVs reported, which strikes me as odd. All in all I believe that the 
> following N-grams were looked up:
>
> -1.358477    <s> dat    -0.6622628
> -1.334724    dat [unk]    -0.0686222
> -0.6776069    dat [unk] [unk]
>
> for the 'n gram' output
>
> dat Albert Heijn het doet zou niet de aanleiding zijn
>    p( dat | <s> ) = 0.0438046 [ -1.35848 ]
>    p( Albert | dat ...) = 0.0100695 [ -1.99699 ]
>    p( Heijn | Albert ...) = 0.210084 [ -0.677607 ]
>
> How did 'ngram' come to the -1.99699 logprob?

I'm not sure how exactly you are invoking ngram.   But if your ppl 
output shows the original words and not the [unk]  word that means your 
OOV words were NOT mapped to [unk].
Also, I'm confused by that had that your -debug 2 output doesn't 
indicate what order of ngram was hit.

For reference, if you invoke ngram on the 3gram model that is packaged 
with SRILM:

     % ngram -unk -map-unk '@reject@' -lm 
$SRILM/lm/test/tests/ngram-count-gt/swbd.3bo.gz -debug 2 -ppl -

and feed two sentences

     hi there
     hix there

the output should look like this:

hi there
         p( hi | <s> )   = [2gram] 0.000625719 [ -3.20362 ]
         p( there | hi ...)      = [3gram] 0.0131176 [ -1.88215 ]
         p( </s> | there ...)    = [2gram] 0.137458 [ -0.86183 ]
1 sentences, 2 words, 0 OOVs
0 zeroprobs, logprob= -5.9476 ppl= 96.0578 ppl1= 941.453

hix there
         p( @reject@ | <s> )     = [2gram] 0.00733392 [ -2.13466 ]
         p( there | @reject@ ...)        = [3gram] 0.00498615 [ -2.30223 ]
         p( </s> | there ...)    = [2gram] 0.0603144 [ -1.21958 ]
1 sentences, 2 words, 0 OOVs
0 zeroprobs, logprob= -5.65648 ppl= 76.8232 ppl1= 673.347

Notice the [2gram] [3gram] indicators, and notice how the OOV "hix" is 
mapped to the unknown word token "@reject@" (that's the label that this 
particular model uses).

So you must be invoking ngram in a difference way and that may have 
something to do with the problems you have interpreting the output.

Andreas