[SRILM User List] Does '-omit' work?
stolcke at icsi.berkeley.edu
Tue Jan 31 01:12:18 PST 2012
On 1/29/2012 7:45 AM, Dmytro Prylipko wrote:
> Dear Andreas,
> I found that using -omit and -observed options does not influence on
> the calculation of perplexity.
> I trained an skip-LM for filled pauses as you advised me (generated
> n-grams, where FPs were skipped from context).
> But when I apply it to the test data it does not matter which
> combination of options do I use for the hidden-vocabulary:
> <FP> -omit -observed
> <FP> -omit
> <FP> -observed
> or just
This is a bug, more in the documentation than in the code.
The hidden event "options" (-omit, -observed, etc) are only processed
when they appear in the -lm file following the ngram parameters.
When processing the -hidden-vocab file, on the other hand, only the
names of the hidden events are recorded (like -vocab).
This should be fixed. But for now, simply append your hidden-event file
to the contents of the -lm file .
Sorry for the confusion in the man page. It kind of says this but in a
very confusing way, and I agree that the -hidden-vocab file should also
interpret the full hidden event specifications.
More information about the SRILM-User