|
Parsing Engine | |||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectdanbikel.parser.Model
public class Model
This class computes the probability of generating an output element of this
parser, where an output element might be, for example, a word, a part of
speech tag, a nonterminal label or a subcat frame. It derives counts from
top-level TrainerEvent
objects, storing these derived counts in
its internal data structures. The derived counts are necessary for the
smoothing of the top-level probabilities used by the parser, and the
particular structure of those levels of smoothing (or, less accurately,
back-off) are specified by the ProbabilityStructure
argument to
the constructor
and to the estimateLogProb(int,TrainerEvent)
method.
ProbabilityStructure
,
Serialized FormField Summary | |
---|---|
protected HashMap[] |
backOffMap
A set of numLevels - 1 maps, where map
i is a map from back-off level i transitions to
i + 1 transitions. |
protected ProbabilityCache[] |
cache
A cache of probability estimates at the various back-off levels of this model, used when precomputeProbs is false . |
protected int[] |
cacheAccesses
Records the number of cache accesses for each back-off level of this model. |
protected int[] |
cacheHits
Records the number of cache hits for each back-off level of this mdoel. |
protected FlexibleMap |
canonicalEvents
A reflexive map of canonical Event objects to save memory
in the various tables of this model that store such Event
objects. |
protected CountsTrio[] |
counts
The derived event counts used to estimate probabilities of this model. |
protected boolean |
createHistBackOffMap
Indicates whether the histBackOffMap should be created when
precomputing probabilities. |
protected static boolean |
deleteCountsWhenPrecomputingProbs
Indicates whether to set counts to null just before
writing this model object to an ObjectOutputStream . |
protected boolean |
dontAddNewParams
The boolean value of the Settings.dontAddNewParams setting. |
protected boolean |
doPruning
The value of this data member determines whether this model will be pruned when probabilities are precomputed. |
protected static boolean |
globalDoPruning
Caches the value of Settings.modelDoPruning . |
protected HashMap[] |
histBackOffMap
A set of numLevels - 1 maps, where map
i is a map from back-off level i histories to
i + 1 histories. |
protected HashSet[] |
historiesToPrune
A set of sets used to collect histories that are to be pruned. |
protected double[] |
lambdaFudge
A cached copy of the smoothing factors of the ProbabilityStructure
used by this model. |
protected double[] |
lambdaFudgeTerm
A cached copy of the smoothing terms of the ProbabilityStructure
used by this model. |
protected double[] |
lambdaPenalty
A cached copy of the smoothing penalty factors contained in the ProbabilityStructure used by this model. |
protected double[] |
logOneMinusLambdaPenalty
The values of lambdaPenalty but modified such thatlogOneMinusLambdaPenalty[i] = Math.log(1 - lambdaPenalty[i])
for all i: 0 ≤ i < lambdaPenalty.size . |
protected int |
numLevels
A cached copy of the number of back-off levels in the ProbabilityStructure used by this model. |
protected HashMapDouble[] |
precomputedLambdas
Precomputed lambdas for each back-off level of this model. |
protected int |
precomputedNPBProbCalls
Records the number of times estimateLogProbUsingPrecomputed(Transition,int) is invoked
requesting a probability for an event whose history context has
a base NP (NPB) the parent nonterminal. |
protected int[] |
precomputedNPBProbHits
Records the number of “hits” to the caches of precomputed probability estimates at the various back-off levels when the caller requests a probability for a context that has a base NP (NPB) as the parent nonterminal. |
protected int |
precomputedProbCalls
Records the number of times estimateLogProbUsingPrecomputed(Transition,int) is invoked. |
protected int[] |
precomputedProbHits
Records the number of “hits” to the caches of precomputed probability estimates at the various back-off levels, to determine the amount each back-off level is used while decoding. |
protected HashMapDouble[] |
precomputedProbs
Precomputed probabilities for each back-off level of this model. |
protected static boolean |
precomputeProbs
The boolean value of Settings.precomputeProbs , cached here
for convenience. |
protected static boolean |
printPrunedEvents
Indicates whether the method pruneHistoriesAndTransitions() will
output pruned events to a special pruned event log file. |
protected static boolean |
printUnprunedEvents
Indicates whether the method pruneHistoriesAndTransitions() will
output events that were not pruned to a special pruned event log file. |
protected static double |
pruningThreshold
Caches the double value of Settings.modelPruningThreshold . |
protected static boolean |
saveBackOffMap
If true, indicates that backOffMap should not be set to
null after probabilities have been precomputed, which means that it will
be saved with this Model instance (for debugging purposes);
otherwise, backOffMap is set to null just after
precomputation of probabilities. |
protected static boolean |
saveHistBackOffMap
Indicates whether the histBackOffMap should be created when
precomputing probabilities and saved with this Model
for debugging purposes. |
protected boolean |
saveSmoothingParams
The boolean value of the Settings.saveSmoothingParams setting. |
protected String |
shortStructureClassName
The value of structureClassName but without the package
qualification. |
protected CountsTable[] |
smoothingParams
The smoothing parameters for the history contexts ( Event instances)
at the back-off levels of this model. |
protected String |
smoothingParamsFile
The value of the smoothing parameters file for this model, as given by ProbabilityStructure.smoothingParametersFile() . |
protected ProbabilityStructure |
structure
The probability structure for this model to use. |
protected String |
structureClassName
A cached copy of the name of the concrete type of the ProbabilityStructure instance used by this model. |
protected ProbabilityCache |
topLevelCache
A currently-unused cache of probabilities of TrainerEvent
objects. |
protected HashSet[] |
transitionsToPrune
A set of sets used to collect transitions that are to be pruned. |
protected static boolean |
useCache
A constant that indicates whether this Model should perform
probability caching. |
protected boolean |
useSmoothingParams
The boolean value of the Settings.useSmoothingParams setting. |
protected boolean |
verbose
Indicates whether to report to stderr what this class is doing. |
protected static boolean |
warnSmoothingHasHistoryNotInTraining
The value of this constant determines whether estimateProb(ProbabilityStructure,TrainerEvent) emits a warning when it
encounters a history for which there is a saved smoothing parameter but
was not an observed history as far as the current model is concerned. |
Constructor Summary | |
---|---|
Model(ProbabilityStructure structure)
Constructs a new object for deriving all counts using the specified probability structure. |
Method Summary | |
---|---|
void |
beQuiet()
Causes this class not to output anything to System.err
during the invocation of its methods, such as
deriveCounts(CountsTable,Filter,double,FlexibleMap) . |
void |
beVerbose()
Causes this class to be verbose in its output to System.err
during the invocation of its methods, such as
deriveCounts(CountsTable,Filter,double,FlexibleMap) . |
void |
canonicalize()
Since events are typically read-only, this method will allow for canonicalization (or "unique-ifying") of the information contained in the events contained in this object. |
void |
canonicalize(FlexibleMap map)
Since events are typically read-only, this method will allow for canonicalization (or "unique-ifying") of the information contained in the events contained in this object using the specified map. |
protected static Event |
canonicalizeEvent(Event event,
FlexibleMap canonical)
This method first canonicalizes the information in the specified event (a Sexp or a Subcat and a Sexp), then it returns a canonical version of the event itself, copying it into the map if necessary. |
protected void |
cleanup()
A method invoked after probabilities have been precomputed by precomputeProbs() to clean up (that is, remove) objects from the various
counts tables that are no longer needed, as determined by ProbabilityStructure.removeHistory(int,Event) and ProbabilityStructure.removeTransition(int,Transition) . |
protected void |
computeHistoriesAndTransitionsToPrune()
Schedule for pruning every history and transition whose MLE is equal to that of back-off level's transition. |
void |
deriveCounts(CountsTable trainerCounts,
Filter filter,
double threshold,
FlexibleMap canonical)
Derives all counts from the specified counts table, using the probability structure specified in the constructor. |
void |
deriveCounts(CountsTable trainerCounts,
Filter filter,
double threshold,
FlexibleMap canonical,
boolean deriveOtherModelCounts)
Derives all counts from the specified counts table, using the probability structure specified in the constructor. |
protected void |
deriveDiversityCounts()
Deprecated. This method used to be called by deriveCounts(CountsTable,Filter,double,FlexibleMap,boolean) , but
diversity counts are now derived directly by that method. |
protected void |
deriveHistories(CountsTable trainerCounts,
Filter filter,
FlexibleMap canonical)
Deprecated. This method used to be called by deriveCounts(CountsTable,Filter,double,FlexibleMap,boolean) , but
histories are now derived directly by that method. |
double |
estimateLogProb(int id,
TrainerEvent event)
Estimates the log-probability of a conditional event. |
protected double |
estimateLogProbUsingPrecomputed(ProbabilityStructure structure,
TrainerEvent event)
Estimates the log prob using precomputed probabilities and smoothing values (lambdas). |
protected double |
estimateLogProbUsingPrecomputed(Transition transition,
int atLevel)
Estimates the log prob of the specified transition using precomputed probabilities and lambdas and histBackOffMap (debugging method). |
double |
estimateProb(int id,
TrainerEvent event)
Estimates the probability of a conditional event. |
protected double |
estimateProb(ProbabilityStructure probStructure,
TrainerEvent event)
Returns the smoothed probability estimate of a transition contained in the specified TrainerEvent object. |
protected double |
estimateProbOld(ProbabilityStructure structure,
TrainerEvent event,
int level,
double prevHistCount)
|
String |
getCacheStats()
Returns a human-readable string containing all the precomputed or non-precomputed probability cache statistics for the life of this Model object. |
protected static Transition |
getCanonical(Transition trans,
FlexibleMap canonical)
This method assumes trans already contains a canonical history and a canonical future. |
Model |
getModel(int idx)
Returns this model object. |
ProbabilityStructure |
getProbStructure()
Returns the type of ProbabilityStructure object used
during the invocation of
deriveCounts(CountsTable,Filter,double,FlexibleMap) . |
protected Transition[] |
getTransitions(Transition zeroLevelTrans,
Transition[] trans)
Inserts the Transition objects representing conditional events for
all back-off levels of this model into the specified array, with
trans[0] = zeroLevelTrans . |
protected void |
initializeSmoothingParams()
Sets up the smoothing parameter arrays and maps. |
int |
numModels()
Returns 1, as this object does not contain any other, internal Model instances. |
void |
precomputeProbs()
Precomputes all probabilities and smoothing values for events seen during all previous invocations of deriveCounts(CountsTable,Filter,double,FlexibleMap,boolean) . |
protected void |
precomputeProbs(CountsTable trainerCounts,
Filter filter)
Deprecated. This method has been superseded by precomputeProbs() . |
protected void |
precomputeProbs(MapToPrimitive.Entry transEntry,
double[] lambdas,
double[] estimates,
Transition[] transitions,
Event[] histories,
int lastLevel)
Precomputes the probabilities and smoothing values for the Transition object contained as a key within the specified
map entry, where the value is the count of the transition. |
protected void |
precomputeProbs(TrainerEvent event,
Transition[] transitions,
Event[] histories)
Deprecated. This method is called by precomputeProbs(CountsTable,Filter) , which is also deprecated. |
protected void |
pruneHistoriesAndTransitions()
Analyzes the distributions of this model in order to prune history and transition (i.e., conditional) events from the various counts tables. |
protected void |
pruneHistoriesAndTransitionsOld()
Prune every history and transition with a back-off level less than the last level in which the last level history has a diversity of 1 (meaning that the probability is 1, so no need to store a history and transition). |
protected void |
readSmoothingParams()
Reads all necessary smoothing parameters from smoothingParamsFile
instead of deriving values for smoothing parameters. |
protected void |
readSmoothingParams(boolean verboseOutput)
Reads all necessary smoothing parameters from smoothingParamsFile
instead of deriving values for smoothing parameters. |
protected void |
savePrecomputeData(CountsTable trainerCounts,
Filter filter)
Saves the back-off chain for each event derived from each TrainerEvent in the key set of the specified counts table. |
void |
setCanonicalEvents(FlexibleMap canonical)
Sets the canonicalEvents member of this object. |
void |
share(int backOffLevel,
Model otherModel,
int otherModelBackOffLevel)
Indicates to use counts or precomputed probabilities from the specified back-off level of this model when estimating probabilities for the specified back-off level of another model. |
protected void |
storePrecomputedProbs(double[] lambdas,
double[] estimates,
Transition[] transitions,
Event[] histories,
int lastLevel)
Stores the specified smoothing values (lambdas) and smoothed probability estimates in the precomputedProbs and smoothingParams
map arrays. |
protected void |
writeSmoothingParams()
Writes the smoothing parameters of this model to the file named by smoothingParamsFile . |
protected void |
writeSmoothingParams(boolean verboseOutput)
Writes the smoothing parameters of this model to the file named by smoothingParamsFile . |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
protected static final boolean warnSmoothingHasHistoryNotInTraining
estimateProb(ProbabilityStructure,TrainerEvent)
emits a warning when it
encounters a history for which there is a saved smoothing parameter but
was not an observed history as far as the current model is concerned.
When using smoothing parameters from another training run, it is typical
to be operating with a model trained from the same data. In such a case,
the set of history contexts observed during the training run that produced
the smoothing parameters would be identical to the set of history contexts
encountered when training again on that same data. However, there are
circumstances when a history context observed in the smoothing training
run would not be observed in the subsequent training run, such as when
performing EM there is, for example, a long sentence with lots of
structure whose total inside probability mass is less than Double.MIN_VALUE
. In such a case, the EMDecoder
will issue an
underflow warning and not emit any expected events for that sentence. If
a history context was only observed in the one or more sentences that have
underflow problems in a particular EM iteration, then it will effectively
not be observed in that iteration, and therefore not in any subsequent
iteration.
protected static final boolean deleteCountsWhenPrecomputingProbs
counts
to null
just before
writing this model object to an ObjectOutputStream
. Normally,
this boolean should be true, but setting it to false can
be useful for debugging purposes.
AnalyzeDisns
,
Constant Field Valuesprotected static boolean precomputeProbs
Settings.precomputeProbs
, cached here
for convenience.
protected static final boolean useCache
Model
should perform
probability caching. This constant is usually true
,
but may redefined as false
for debugging purposes
(recompilation is necessary after redefining this constant).
protected static final boolean saveBackOffMap
backOffMap
should not be set to
null after probabilities have been precomputed, which means that it will
be saved with this Model
instance (for debugging purposes);
otherwise, backOffMap
is set to null
just after
precomputation of probabilities. Normally, the value of this boolean
should be false. The value of this boolean is only consulted
when Settings.precomputeProbs
is true.
protected static final boolean saveHistBackOffMap
histBackOffMap
should be created when
precomputing probabilities and saved with this Model
for debugging purposes. Normally, the value of this boolean should be
false. The value of this boolean is only consulted when
Settings.precomputeProbs
is true.
AnalyzeDisns
,
Constant Field Valuesprotected static boolean globalDoPruning
Settings.modelDoPruning
.
protected static double pruningThreshold
Settings.modelPruningThreshold
.
protected static final boolean printPrunedEvents
pruneHistoriesAndTransitions()
will
output pruned events to a special pruned event log file.
protected static final boolean printUnprunedEvents
pruneHistoriesAndTransitions()
will
output events that were not pruned to a special pruned event log file.
protected ProbabilityStructure structure
protected String structureClassName
ProbabilityStructure
instance used by this model.
protected String shortStructureClassName
structureClassName
but without the package
qualification.
protected int numLevels
ProbabilityStructure
used by this model.
ProbabilityStructure.numLevels()
protected double[] lambdaFudge
ProbabilityStructure
used by this model. This array is of size numLevels
.
ProbabilityStructure.lambdaFudge(int)
protected double[] lambdaFudgeTerm
ProbabilityStructure
used by this model. This array is of size numLevels
.
ProbabilityStructure.lambdaFudgeTerm(int)
protected double[] lambdaPenalty
ProbabilityStructure
used by this model. This array is
of size equal to numLevels
.
ProbabilityStructure.lambdaPenalty(int)
protected double[] logOneMinusLambdaPenalty
lambdaPenalty
but modified such thatlogOneMinusLambdaPenalty[i] = Math.log(1 - lambdaPenalty[i])
i: 0 ≤ i < lambdaPenalty.size
.
protected CountsTrio[] counts
protected boolean verbose
protected HashMapDouble[] precomputedProbs
HashMapDouble
maps in this array are Transition
objects.
protected HashMapDouble[] precomputedLambdas
HashMapDouble
maps in this array are Event
instances.
For the modified Witten-Bell smoothing method used by this class, the
values of the maps of this array are actually the log of one minus the
lambda of a particular event at a particular back-off level, for ease of
computing a smoothed estimate. That is, if event
is some
history context whose associated smoothing value is
λi, then
precomputedLambdas[i].get(event)
will be equal to
ln(1 − λi), where ln is the
natural log function that is implemented by Math.log
.
protected transient int[] precomputedProbHits
protected transient int precomputedProbCalls
estimateLogProbUsingPrecomputed(Transition,int)
is invoked.
protected transient int[] precomputedNPBProbHits
protected transient int precomputedNPBProbCalls
estimateLogProbUsingPrecomputed(Transition,int)
is invoked
requesting a probability for an event whose history context has
a base NP (NPB) the parent nonterminal. This allows a comparison
between NPB method invocations and overall method invocations.
protected HashMap[] backOffMap
numLevels
- 1
maps, where map
i is a map from back-off level i transitions to
i + 1 transitions. These maps are only used temporarily
when precomputing probs (and are necessary for incremental training).
savePrecomputeData(CountsTable,Filter)
,
saveBackOffMap
protected HashMap[] histBackOffMap
numLevels
- 1
maps, where map
i is a map from back-off level i histories to
i + 1 histories. These maps are not necessary
for precomputing probabilities, but can be useful when debugging.
saveHistBackOffMap
,
savePrecomputeData(CountsTable,Filter)
protected transient ProbabilityCache topLevelCache
TrainerEvent
objects.
protected transient ProbabilityCache[] cache
precomputeProbs
is false
.
protected transient int[] cacheHits
protected transient int[] cacheAccesses
protected transient FlexibleMap canonicalEvents
Event
objects to save memory
in the various tables of this model that store such Event
objects.
protected transient String smoothingParamsFile
ProbabilityStructure.smoothingParametersFile()
.
useSmoothingParams
,
dontAddNewParams
,
ProbabilityStructure.smoothingParametersFile()
protected transient boolean saveSmoothingParams
Settings.saveSmoothingParams
setting.
protected transient boolean dontAddNewParams
Settings.dontAddNewParams
setting.
protected transient boolean useSmoothingParams
Settings.useSmoothingParams
setting.
protected transient CountsTable[] smoothingParams
Event
instances)
at the back-off levels of this model.
protected transient HashSet[] transitionsToPrune
doPruning
,
pruneHistoriesAndTransitions()
protected transient HashSet[] historiesToPrune
doPruning
,
pruneHistoriesAndTransitions()
protected transient boolean createHistBackOffMap
histBackOffMap
should be created when
precomputing probabilities. If either doPruning
or saveHistBackOffMap
is true, then this data member will be set
to true as well. The value of this boolean is set automatically
in the constructor, and is only consulted when Settings.precomputeProbs
is true.
AnalyzeDisns
,
doPruning
protected transient boolean doPruning
globalDoPruning
is true or if
the ProbabilityStructure.doPruning()
method invoked on this
model’s probability structure object returns
true.
pruneHistoriesAndTransitions()
,
precomputeProbs()
Constructor Detail |
---|
public Model(ProbabilityStructure structure)
structure
- the probability structure to use when deriving countsMethod Detail |
---|
public void setCanonicalEvents(FlexibleMap canonical)
canonicalEvents
member of this object.
canonical
- the reflexive map of canonical Event
objectsModelCollection.internalReadObject(java.io.ObjectInputStream)
public void deriveCounts(CountsTable trainerCounts, Filter filter, double threshold, FlexibleMap canonical)
trainerCounts
- a map from TrainerEvent
objects to
their counts (as double
s) from which to derive countsfilter
- used to filter out TrainerEvent
objects
whose derived counts should not be derived for this modelthreshold
- a (currently unused) count cut-off thresholdcanonical
- a reflexive map used to canonicalize objects
created when deriving countspublic void deriveCounts(CountsTable trainerCounts, Filter filter, double threshold, FlexibleMap canonical, boolean deriveOtherModelCounts)
trainerCounts
- a map from TrainerEvent
objects to
their counts (as double
s) from which to derive countsfilter
- used to filter out TrainerEvent
objects
whose derived counts should not be derived for this modelthreshold
- a (currently unused) count cut-off thresholdcanonical
- a reflexive map used to canonicalize objects
created when deriving countsderiveOtherModelCounts
- an unused parameter, as this class
does not contain other, internal Model
instancesprotected void cleanup()
precomputeProbs()
to clean up (that is, remove) objects from the various
counts tables that are no longer needed, as determined by ProbabilityStructure.removeHistory(int,Event)
and ProbabilityStructure.removeTransition(int,Transition)
.
If precomputeProbs
is true
, then this method will
remove entries from the maps of the precomputedProbs
array. If
precomputeProbs
is false
or if deleteCountsWhenPrecomputingProbs
is false
, then this method
will remove entries from the maps in the counts
array.
ProbabilityStructure.removeHistory(int,Event)
,
ProbabilityStructure.removeTransition(int,Transition)
protected static final Event canonicalizeEvent(Event event, FlexibleMap canonical)
protected static final Transition getCanonical(Transition trans, FlexibleMap canonical)
public double estimateLogProb(int id, TrainerEvent event)
id
- the id of the caller (typically a
Switchboard
client ID)event
- the maximal-context event containing both the history
(conditioning context) and future of the conditional event
whose probability is to be estimated
public double estimateProb(int id, TrainerEvent event)
id
- the id of the caller (typically a
Switchboard
client ID)event
- the maximal-context event containing both the history
(conditioning context) and future of the conditional event
whose probability is to be estimated
protected double estimateLogProbUsingPrecomputed(ProbabilityStructure structure, TrainerEvent event)
estimateLogProb(int,TrainerEvent)
if precomputeProbs
is
true
.
protected double estimateLogProbUsingPrecomputed(Transition transition, int atLevel)
histBackOffMap
(debugging method).
N.B.: The history contained within the specified transition
must have been observed during training (but not necessarily with
the particular future contained in the specified transition).
transition
- the transition for which to get a smoothed
log-probability estimateatLevel
- the back-off level of the specified transitionhistBackOffMap
,
createHistBackOffMap
protected double estimateProb(ProbabilityStructure probStructure, TrainerEvent event)
TrainerEvent
object.
probStructure
- a ProbabilityStructure
object that is
either structure
or a copy of it, used for
temporary storage during the computation performed by
this methodevent
- the TrainerEvent
containing a transition
from a history to a future whose smoothed probability
is to be computed
TrainerEvent
objectprotected double estimateProbOld(ProbabilityStructure structure, TrainerEvent event, int level, double prevHistCount)
protected void deriveDiversityCounts()
deriveCounts(CountsTable,Filter,double,FlexibleMap,boolean)
, but
diversity counts are now derived directly by that method.
deriveCounts(CountsTable,Filter,double,FlexibleMap)
, for each
type of transition observed, this method derives the number of
unique transitions from the history context to the possible
futures. This number of unique transitions, called the
diversity of a random variable, is used in a modified
version of Witten-Bell smoothing.
protected void deriveHistories(CountsTable trainerCounts, Filter filter, FlexibleMap canonical)
deriveCounts(CountsTable,Filter,double,FlexibleMap,boolean)
, but
histories are now derived directly by that method.
Model
object's probability structure.
trainerCounts
- a map from TrainerEvent
objects to
their counts (as double
s) from which to derive countsfilter
- used to filter out TrainerEvent
objects
whose derived counts should not be derived for this modelcanonical
- a reflexive map used to canonicalize objects
created when deriving countsprotected Transition[] getTransitions(Transition zeroLevelTrans, Transition[] trans)
Transition
objects representing conditional events for
all back-off levels of this model into the specified array, with
trans[0] = zeroLevelTrans
. Higher-numbered back-off level
events (i.e., events with increasingly coarser history contexts) are gotten
from the specified zeroeth-level Transition
by use of the
backOffMap
.
zeroLevelTrans
- the Transition
object representing
a conditional event with a maximal-context historytrans
- the array in which to insert Transition
objects
for all levels of back-off of this model
Transition
array, having been modified
to include Transition
objects for all levels of back-off
of this modelprotected void pruneHistoriesAndTransitionsOld()
protected void computeHistoriesAndTransitionsToPrune()
protected void pruneHistoriesAndTransitions()
structureClassName
+
".prune-log"
, if printPrunedEvents
or
printUnprunedEvents
, respectively, are true
. This
allows further analysis of the model.
AnalyzeDisns
,
printPrunedEvents
,
printUnprunedEvents
protected void initializeSmoothingParams()
smoothingParams
protected void savePrecomputeData(CountsTable trainerCounts, Filter filter)
TrainerEvent
in the key set of the specified counts table.
This method is called by deriveCounts(CountsTable,Filter,double,FlexibleMap,boolean)
when
Settings.precomputeProbs
is true
.
trainerCounts
- a counts table containing some or all of the
TrainerEvent
objects collected during trainingfilter
- a filter specifying which TrainerEvent
objects to ignore in the key set of the specified counts tablederiveCounts(CountsTable,Filter,double,FlexibleMap,boolean)
,
backOffMap
public void precomputeProbs()
deriveCounts(CountsTable,Filter,double,FlexibleMap,boolean)
.
precomputeProbs(MapToPrimitive.Entry, ...)
,
storePrecomputedProbs
protected void precomputeProbs(MapToPrimitive.Entry transEntry, double[] lambdas, double[] estimates, Transition[] transitions, Event[] histories, int lastLevel)
Transition
object contained as a key within the specified
map entry, where the value is the count of the transition.
transEntry
- a map entry mapping a Transition
object to its count (a double
)lambdas
- an array in which to store the smoothing value for
each of the back-off levelsestimates
- an array in which to store the maximum-likelihood
estimate at each of the back-off levelstransitions
- an array in which to store the Transition
instance for each of the back-off levelshistories
- an array in which to store the history, an
Event
instance, for each of the back-off levelslastLevel
- the last back-off level (the value equal to
numLevels
- 1
)precomputeProbs()
protected void storePrecomputedProbs(double[] lambdas, double[] estimates, Transition[] transitions, Event[] histories, int lastLevel)
precomputedProbs
and smoothingParams
map arrays.
lambdas
- an array containing the smoothing value for each of the
back-off levelsestimates
- an array containing the maximum-likelihood estimate at
each of the back-off levelstransitions
- an array containing the Transition
instance for each of the back-off levelshistories
- an array in which to store the history, an
Event
instance, for each of the back-off levelslastLevel
- the last back-off level (the value equal to
numLevels
- 1
)precomputeProbs()
protected void precomputeProbs(CountsTable trainerCounts, Filter filter)
precomputeProbs()
.
TrainerEvent
instances and their
counts contained in the specified counts table.
trainerCounts
- a map of TrainerEvent
instances
to their observed countsfilter
- a filter indicating which of the TrainerEvent
objects in the specified counts table should be ignored by this method
as it iterates over all entires in the counts tableprotected void precomputeProbs(TrainerEvent event, Transition[] transitions, Event[] histories)
precomputeProbs(CountsTable,Filter)
, which is also deprecated.
event
- the TrainerEvent
object from which probabilities
are to be precomputedtransitions
- temporary storage to be used during an invocation
of this methodhistories
- temporary storage to be used during an invocation
of this methodprotected void readSmoothingParams()
smoothingParamsFile
instead of deriving values for smoothing parameters. Verbose output is
produced.
useSmoothingParams
,
dontAddNewParams
,
readSmoothingParams(boolean)
protected void readSmoothingParams(boolean verboseOutput)
smoothingParamsFile
instead of deriving values for smoothing parameters. Verbose output is
produced if the specified argument is true
.
verboseOutput
- indicates whether or not this method should output
verbose messages to System.err
.useSmoothingParams
,
dontAddNewParams
protected void writeSmoothingParams()
smoothingParamsFile
. Verbose output to System.err
is produced.
protected void writeSmoothingParams(boolean verboseOutput)
smoothingParamsFile
.
verboseOutput
- indicates whether to output verbose messages to
System.err
public void share(int backOffLevel, Model otherModel, int otherModelBackOffLevel)
Model
.
backOffLevel
- the back-off level of this model that is to be
shared by another modelotherModel
- the other model that will share a particular
back-off level with this mdoel (that is, use the counts or precomputed
probabilities from this model)otherModelBackOffLevel
- the back-off level of the other model
that is to be made the same as the specified back-off level
of this model (that is, use the counts or precomputed probabilities
from this model)public void canonicalize()
public void canonicalize(FlexibleMap map)
map
- a map of canonical information structures of the
Event
objects contained in this object; this parameter
allows multiple Model
objects to have their
events structures canonicalized with respect to each otherpublic ProbabilityStructure getProbStructure()
ProbabilityStructure
object used
during the invocation of
deriveCounts(CountsTable,Filter,double,FlexibleMap)
.
A copy of this object should be created and stored for each
parsing client thread, for use when the clients need to call the
probability-computation methods of this class. This scheme
allows the reusable data members inside the
ProbabilityStructure
objects to be used by multiple
clients without any concurrency problems, thereby maintaining
their efficiency and thread-safety.
public int numModels()
Model
instances.
public Model getModel(int idx)
idx
- an unused parameter, as this object does not contain any other,
internal Model
instances.
public void beVerbose()
System.err
during the invocation of its methods, such as
deriveCounts(CountsTable,Filter,double,FlexibleMap)
.
public void beQuiet()
System.err
during the invocation of its methods, such as
deriveCounts(CountsTable,Filter,double,FlexibleMap)
.
public String getCacheStats()
Model
object.
Model
object.
|
Parsing Engine | |||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |