<div dir="ltr">Thank you Andreas! This approach is getting me the probabilities really quickly (within 0.5 seconds including steps of pre and post processing in a Python wrapper on a single core). It was very satisifying to see `np.sum(distribution)` returning values like `0.99999994929300007`.<div>Thank you for your help <span style="font-size:12.8px;white-space:nowrap">Dávid</span>! I'd love to have a look at your code. Here is my Github handle - martiansideofthemoon</div><div><br></div><div>With Regards,</div><div>Kalpesh Krishna</div><div><a href="http://martiansideofthemoon.github.io/" target="_blank" style="font-size:12.8px">http://martiansideofthemoon.<wbr>github.io/</a><br><br><br><br></div><img width="0" height="0" class="mailtrack-img" src="https://mailtrack.io/trace/mail/159a401e4ebc563d9dda437d81d34d0d88939c35.png?u=981539"></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 13, 2017 at 10:41 PM, Andreas Stolcke <span dir="ltr"><<a href="mailto:stolcke@icsi.berkeley.edu" target="_blank">stolcke@icsi.berkeley.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="m_8521084142019788319moz-cite-prefix"><br>
A brute force solution to this (if you don't want to modify any
code) is to generate an N-gram count file of the form<br>
<br>
apple banana banana carrot apple 1<br>
apple banana banana carrot banana 1<br>
apple banana banana carrot carrot 1<br>
<br>
and pass it to <br>
<br>
ngram -lm LM -order 5 -counts COUNTS -debug 2 <br>
<br>
If you want to make a minimal code change to enumerate all
conditional probabilities for any context encountered, you could
do so in LM::wordProbSum() and have it dump out the word tokens
and their log probabilities. Then process some text with ngram
-debug 3.<span class="HOEnZb"><font color="#888888"><br>
<br>
Andreas</font></span><div><div class="h5"><br>
<br>
<br>
<br>
On 3/12/2017 12:12 AM, Dávid Nemeskey wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Kalpesh,<br>
<br>
</div>
well, there's LM::<span class="m_8521084142019788319gmail-pl-en">wordProb</span>(VocabIndex
word, <span class="m_8521084142019788319gmail-pl-k">const</span> VocabIndex
*context<span class="m_8521084142019788319gmail-pl-k"></span><span class="m_8521084142019788319gmail-pl-k"></span>) in lm/src/LM.cc (and in
lm/src/NgramLM.cc, if you are using an ngram model). You could
simply call it on every word in the vocabulary. However, be
warned that this will be very slow for any reasonable
vocabulary size (say 10k and up). This function is also what
generateWord() calls, that is why the latter is so slow.<br>
<br>
If you just wanted the top n most probable words, the
situation would be a bit different. Then wordProb() wouldn't
be the optimal solution because the trie built by ngram is
reversed (meaning you have to go back from the word to the
root, and not the other way around), and you had to query all
words to get the most probably one. So when I wanted to do
this, I built another trie (from the root up to the word),
which made it much faster, though I am not sure it was 100%
correct in the face of negative backoff weights. But it
wouldn't help in your case, I guess.<br>
<br>
</div>
<div>Best,<br>
</div>
<div>Dávid<br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Mar 11, 2017 at 8:32 PM,
Kalpesh Krishna <span dir="ltr"><<a href="mailto:kalpeshk2011@gmail.com" target="_blank">kalpeshk2011@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hello,
<div>I have a context of words and I've built an N-gram
language model using ./ngram-count. I wish to generate a
probability distribution (over the entire vocabulary of
words) of the next word. I can't seem to be able to find
a good way to do this with ./ngram.</div>
<div>What's the best way to do this?</div>
<div>For example, if my vocabulary has words "apple,
banana, carrot", and my context is "apple banana banana
carrot", I want a distribution like - {"apple": 0.25,
"banana": 0.5, "carrot": 0.25}.</div>
<div><br>
</div>
<div>Thank you,</div>
<div>Kalpesh Krishna</div>
<div><a href="http://martiansideofthemoon.github.io/" target="_blank">http://martiansideofthemoon.gi<wbr>thub.io/</a><br>
</div>
<img class="m_8521084142019788319m_8658768150482666164mailtrack-img" height="0" width="0"></div>
<br>
______________________________<wbr>_________________<br>
SRILM-User site list<br>
<a href="mailto:SRILM-User@speech.sri.com" target="_blank">SRILM-User@speech.sri.com</a><br>
<a href="http://mailman.speech.sri.com/cgi-bin/mailman/listinfo/srilm-user" rel="noreferrer" target="_blank">http://mailman.speech.sri.com/<wbr>cgi-bin/mailman/listinfo/srilm<wbr>-user</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="m_8521084142019788319mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
SRILM-User site list
<a class="m_8521084142019788319moz-txt-link-abbreviated" href="mailto:SRILM-User@speech.sri.com" target="_blank">SRILM-User@speech.sri.com</a>
<a class="m_8521084142019788319moz-txt-link-freetext" href="http://mailman.speech.sri.com/cgi-bin/mailman/listinfo/srilm-user" target="_blank">http://mailman.speech.sri.com/<wbr>cgi-bin/mailman/listinfo/<wbr>srilm-user</a></pre>
</blockquote>
<p><br>
</p>
</div></div></div>
<br>______________________________<wbr>_________________<br>
SRILM-User site list<br>
<a href="mailto:SRILM-User@speech.sri.com">SRILM-User@speech.sri.com</a><br>
<a href="http://mailman.speech.sri.com/cgi-bin/mailman/listinfo/srilm-user" rel="noreferrer" target="_blank">http://mailman.speech.sri.com/<wbr>cgi-bin/mailman/listinfo/<wbr>srilm-user</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Kalpesh Krishna,</div><div>Junior Undergraduate,</div><div dir="ltr"><div>Electrical Engineering,</div><div>IIT Bombay</div></div></div></div></div></div></div></div></div>
</div>