<div dir="ltr"><span style="font-size:12.8px">Hello David.</span><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Thank you very much for answering. I am not sure if you received my reply as the yahoo servers have problems right now so i switched to gmail (sorry if you received already the email).</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">I used Wikipedia parallel corpus en-es for training the two lms (<a href="http://opus.lingfil.uu.se/Wikipedia.php" target="_blank">http://opus.lingfil.uu.se/<wbr>Wikipedia.php</a>, 1.8M sentence pairs). I used the -debug 2 as you said and below are the results. Could you please help me understand why the perplexity numbers are so high for the EN sentences since they are well formed? For testing spanish i used machine translated output so i was expecting big numbers for ppl. Thank you!</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>Sixty-six parent coordinators were laid off," the draft complaint says, "and not merely excessed.</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">  </span>p( Sixty-six | <s> ) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">  </span>= [1gram] 2.16995e-09 [ -8.66355 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( parent | Sixty-six ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">  </span>= [1gram] 1.0949e-05 [ -4.96063 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>p( coordinators | parent ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">       </span>= [1gram] 3.37871e-07 [ -6.47125 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( were | coordinators ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>= [1gram] 0.00120231 [ -2.91998 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>p( laid | were ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>= [2gram] 0.000696035 [ -3.15737 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( off," | laid ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">   </span>= [1gram] 2.33407e-08 [ -7.63189 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( the | off," ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">    </span>= [2gram] 0.0469306 [ -1.32854 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">       </span>p( draft | the ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>= [2gram] 7.67904e-05 [ -4.11469 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( complaint | draft ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">   </span>= [1gram] 8.13141e-07 [ -6.08983 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( says, | complaint ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">   </span>= [1gram] 1.17395e-05 [ -4.93035 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( "and | says, ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">   </span>= [2gram] 0.00147669 [ -2.83071 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>p( not | "and ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>= [1gram] 0.000275198 [ -3.56035 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( merely | not ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">        </span>= [2gram] 0.00173666 [ -2.76029 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>p( <unk> | merely ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">        </span>= [1gram] 0.0796503 [ -1.09881 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">       </span>p( </s> | <unk> ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">    </span>= [1gram] 0.0258359 [ -1.58778 ]</div><div>1 sentences, 14 words, 0 OOVs</div><div>0 zeroprobs, logprob= -62.106 ppl= 13816.6 ppl1= 27298.9</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>Mexico's Enrique Pena Nieto faces tough start</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">  </span>p( Mexico's | <s> ) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">       </span>= [2gram] 1.31547e-06 [ -5.88092 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( Enrique | Mexico's ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>= [1gram] 1.34348e-05 [ -4.87177 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( Pena | Enrique ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>= [1gram] 1.83116e-06 [ -5.73727 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( Nieto | Pena ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">        </span>= [1gram] 1.6622e-06 [ -5.77932 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>p( faces | Nieto ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">       </span>= [1gram] 1.61354e-05 [ -4.79222 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( tough | faces ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">       </span>= [1gram] 2.80928e-06 [ -5.5514 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>p( start | tough ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">       </span>= [1gram] 2.90611e-05 [ -4.53669 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( </s> | start ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">  </span>= [1gram] 0.00941231 [ -2.0263 ]</div><div>1 sentences, 7 words, 0 OOVs</div><div>0 zeroprobs, logprob= -39.1759 ppl= 78883.7 ppl1= 394964</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>The NATO mission officially ended Oct. 31.</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>p( The | <s> ) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">        </span>= [2gram] 0.143584 [ -0.842893 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">       </span>p( NATO | The ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">  </span>= [3gram] 5.55208e-06 [ -5.25554 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( mission | NATO ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>= [1gram] 3.10877e-05 [ -4.50741 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( officially | mission ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">        </span>= [1gram] 2.81221e-05 [ -4.55095 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( ended | officially ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">  </span>= [2gram] 0.00976927 [ -2.01014 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>p( Oct. | ended ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">        </span>= [1gram] 2.4073e-07 [ -6.61847 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">      </span>p( 31. | Oct. ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">  </span>= [1gram] 3.60453e-06 [ -5.44315 ]</div><div><span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">     </span>p( </s> | 31. ...) <span class="gmail-m_2220653380549948595gmail-Apple-tab-span" style="white-space:pre-wrap">    </span>= [2gram] 0.907671 [ -0.0420717 ]</div><div>1 sentences, 7 words, 0 OOVs</div><div>0 zeroprobs, logprob= -29.2706 ppl= 4558.57 ppl1= 15188.6</div></div></div>