[SRILM User List] class perplexity -debug 2 output

[Andreas Stolcke]

Someone else asked this question recently:

http://www-speech.sri.com/pipermail/srilm-user/2011q1/000979.html

FYI, you could have found this page by web-searching for "class ngram
ppl output" .

Andreas


Fabian - wrote:
> Hello,
>
> I used the "-debug 2" during the perplexity computation of a word
> language model and a class language model, and don't understand the
> output.
>
> I start with:
> ngram -lm lm.3g -order 3 -mix-lm lm300 -lambda 0.518735 -bayes 0
> -classes classes300 -ppl dev.text -debug 2
> lm.3g: word lm
> lm300: class lm
>
> and get:
> Ëý »¹ÔÚ ¶Á ! ýµ ...
> p( Ëý | <s> ) = [OOV][2gram][2gram][OOV] 0.00193608 [ -2.71308 ]
> p( »¹ÔÚ | Ëý ...) = [OOV][1gram][OOV][2gram][1gram][OOV][1gram][OOV]
> 7.96669e-05 [ -4.09872 ]
> p( ¶Á | »¹ÔÚ ...) = [OOV][1gram][OOV][2gram][1gram][OOV][1gram][OOV]
> 0.00486257 [ -2.31313 ]
> p( ËýµÄ | ¶Á ...) = [OOV][1gram][OOV][2gram][1gram][OOV][1gram][OOV]
> 0.000237892 [ -3.62362 ]
> p( ¶Ô | ËýµÄ ...) = [OOV][1gram][OOV][2gra! m][2gram][OOV][1gram][OOV]
> 0.00242938 [ -2.6145 ]
> ...
> are the words covered by 1-grams/2-grams or OOV?
>
> Thank you!!!
>
> -Fabian
> ------------------------------------------------------------------------
>
> _______________________________________________
> SRILM-User site list
> SRILM-User at speech.sri.com
> http://www.speech.sri.com/mailman/listinfo/srilm-user