[SRILM User List] compute perplexity

[Stefy D.]

Dear all,

I have some questions regarding perplexity...I am very thankful for your time/ answers.

Settings:
- one language model LM_A estimated using training corpus A 
- one language model LM_B estimated using training corpus B (B = corpus_A + corpus_X)

My intention is to prove that model B is better than model A so I though I should show that the perplexity decreased (which can be seen from the ppl files).

Commands used to estimate ppl:
$NGRAM_FILE -order 3  -lm $WORKING_DIR"lm_A/lmodel.lm" -ppl $WORKING_DIR"test.lowercased."$TARGET >  $WORKING_DIR"ppl_A.ppl"


$NGRAM_FILE -order 3  -lm $WORKING_DIR"lm_B/lmodel.lm" -ppl $WORKING_DIR"test.lowercased."$TARGET >  $WORKING_DIR"ppl_B.ppl"


This contents of the two ppl files is (A then B):
1000 sentences, 21450 words, 0 OOVs
0 zeroprobs, logprob= -57849.4 ppl= 377.407 ppl1= 497.67
-------------------------------------------------------------------------------------------
1000 sentences, 21450 words, 0 OOVs
0 zeroprobs, logprob= -55535.3 ppl= 297.67 ppl1= 388.204

Questions:
1. Why do I get 0 OOVs? I checked using the compute-oov-rate script how many OOV there are in the test data compared to the training and it gave me the result "OOV tokens: 393 / 21450 (1.83%) excluding fragments: 390 / 21442 (1.82%)".

2. I read on the srilm-faq that "Note that perplexity comparisons are only ever meaningful if the vocabularies of all LMs are the same." Since I want to compare perplexities of two LM I am wondering if I did the right thing with my settings and commands used. The two LM were estimated on different training corpora so the vocabularies are not identical, right? Please tell me what am I doing wrong.

3. If those two perplexities were computed correctly, then could you please tell me if their difference means that the LM model has been really improved and if there is a measure that says if this improvement is significantly? 

Thank you very much for your time.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.speech.sri.com/pipermail/srilm-user/attachments/20140318/91307a95/attachment.html>