/h`SrSSKrSSKrSSKrSSKrSSKJrJr SSKJ r J r SSK J r SSK Jr \"S5r"SS \ 5r"S S \S 9r"S S\5r"SS\5r"SS\5r"SS\5r"SS\5r"SS\5r"SS\5r"SS\5r"SS\5r"SS \5r"S!S"\5r"S#S$\5r"S%S&\5rS'r S(r!"S)S*\ 5r""S+S,\#\S 9r$"S-S.\$5r%"S/S0\$5r&\RN"S1S25r(S3r)S4r*"S5S65r+"S7S8\+5r,S9r-S:r.S;r/SDS=jr0S>r1\2S?:Xa\0"S>> from nltk.tokenize import word_tokenize >>> from nltk.probability import FreqDist >>> sent = 'This is an example sentence' >>> fdist = FreqDist() >>> for word in word_tokenize(sent): ... fdist[word.lower()] += 1 An equivalent way to do this is with the initializer: >>> fdist = FreqDist(word.lower() for word in word_tokenize(sent)) c>[R"X5 SUlg)a Construct a new frequency distribution. If ``samples`` is given, then the frequency distribution will be initialized with the count of each object in ``samples``; otherwise, it will be initialized to be empty. In particular, ``FreqDist()`` returns an empty frequency distribution; and ``FreqDist(samples)`` first creates an empty frequency distribution, and then calls ``update`` with the list ``samples``. :param samples: The samples to initialize the frequency distribution with. :type samples: Sequence N)r__init___Nselfsampless H/var/www/auris/envauris/lib/python3.13/site-packages/nltk/probability.pyr FreqDist.__init__Vs 'cpURc[UR55UlUR$)z Return the total number of sample outcomes that have been recorded by this FreqDist. For the number of unique sample values (or bins) with counts greater than zero, use ``FreqDist.B()``. :rtype: int )rsumvaluesrs rN FreqDist.Nks( 77?$++-(DGwwrc2>SUl[TU] X5 g)z? Override ``Counter.__setitem__()`` to invalidate the cached N N)rsuper __setitem__rkeyval __class__s rrFreqDist.__setitem__ys C%rc2>SUl[TU] U5 g)z? Override ``Counter.__delitem__()`` to invalidate the cached N N)rr __delitem__)rrr!s rr$FreqDist.__delitem__s C rc4>SUl[TU]"U0UD6 g)z: Override ``Counter.update()`` to invalidate the cached N N)rrupdate)rargskwargsr!s rr'FreqDist.updates ''rc2>SUl[TU] X5 g)z> Override ``Counter.setdefault()`` to invalidate the cached N N)rr setdefaultrs rr,FreqDist.setdefaults 3$rc[U5$)z Return the total number of sample values (or "bins") that have counts greater than zero. For the total number of sample outcomes recorded, use ``FreqDist.N()``. (FreqDist.B() is the same as len(FreqDist).) :rtype: int lenrs rB FreqDist.Bs4yrcFUVs/sHoUS:XdM UPM sn$s snf)zM Return a list of all samples that occur once (hapax legomena) :rtype: list )ritems rhapaxesFreqDist.hapaxess$ "&9dq999s c*URU5U$N)r_Nr)rrbinss rNr FreqDist.Nrsyyq!!rc[[5nUR5HnX#==S- ss'M UbXR5- OSUS'U$)a_ Return the dictionary mapping r to Nr, the number of samples with frequency r, where Nr > 0. :type bins: int :param bins: The number of possible sample outcomes. ``bins`` is used to calculate Nr(0). In particular, Nr(0) is ``bins-self.B()``. If ``bins`` is not specified, it defaults to ``self.B()`` (so Nr(0) will be 0). :rtype: int r4r)rintrr1)rr=_r_Nrcounts rr; FreqDist.r_NrsIC [[]E LA L#'+&64&&(?Aa rc#8# SnUHnX U- nUv M g7f)z Return the cumulative frequencies of the specified samples. If no samples are specified, all counts are returned, starting with the largest. :param samples: the samples whose frequencies should be returned. :type samples: any :rtype: list(float) Nr5)rrcfsamples r_cumulative_frequencies FreqDist._cumulative_frequenciess'F v, BHc>UR5nUS:XagXU- $)a Return the frequency of a given sample. The frequency of a sample is defined as the count of that sample divided by the total number of sample outcomes that have been recorded by this FreqDist. The count of a sample is defined as the number of times that sample outcome was recorded by this FreqDist. Frequencies are always real numbers in the range [0, 1]. :param sample: the sample whose frequency should be returned. :type sample: any :rtype: float rr)rrHns rfreq FreqDist.freqs& FFH 6|arcd[U5S:Xa [S5eURS5SS$)a Return the sample with the greatest number of outcomes in this frequency distribution. If two or more samples have the same number of outcomes, return one of them; which sample is returned is undefined. If no outcomes have occurred in this frequency distribution, return None. :return: The sample with the maximum number of outcomes in this frequency distribution. :rtype: any or None rz?A FreqDist must have at least one sample before max is defined.r4)r0 ValueError most_commonrs rmax FreqDist.maxs< t9>Q "1%a((rF)title cumulativepercentsshowcSSKJn [ U5S:Xa [ U5/nUR "U6V V s/sHupU PM n n n U(a[ URU 55n Sn OU Vs/sHoUPM n nSn U(a+U Vs/sHoUR5- S-PM n nU S- n OU S- n UR5nURS S S 9 S U;aS US 'U(aURU5 UR"U 40UD6 UR[[ U 555 URU Vs/sHn[!U5PM snSS9 UR#S5 UR%U 5 U(aUR'5 U$![an[S5UeSnAff=fs sn n fs snfs snfs snf)a Plot samples from the frequency distribution displaying the most frequent sample first. If an integer parameter is supplied, stop after this many samples have been plotted. For a cumulative plot, specify cumulative=True. Additional ``**kwargs`` are passed to matplotlib's plot function. (Requires Matplotlib to be installed.) :param title: The title for the graph. :type title: str :param cumulative: Whether the plot is cumulative. (default = False) :type cumulative: bool :param percents: Whether the plot uses percents instead of counts. (default = False) :type percents: bool :param show: Whether to show the plot, or only return the ax. :type show: bool rNQThe plot function requires matplotlib to be installed.See https://matplotlib.org/ Cumulative rVdPercentsCountsTsilvercolor linewidthZrotationSamples)matplotlib.pyplotpyplot ImportErrorrRr0rSlistrIrgcagrid set_titleplot set_xticksrangeset_xticklabelsstr set_xlabel set_ylabelrZ)rrWrXrYrZr(r)plter6_rfreqsylabelrHfaxss rrq FreqDist.plots(  + t9>I;D'+'7'7'>?'>GD4'>? 55g>?E"F078f&\E8F 167A\C'E7 j F h F WWY H% f $"#F;   LL     eCL)* G4GqCFG4rB i  f  HHJ Q .  @ 985s.FF3.F9 F>?G F0 F++F0c h[U5S:Xa [U5/n[USUR"U6VVs/sHup4UPM snn5n[USS5nU(a[UR U55nOUVs/sHoUPM nn[ SU55n [ U [ SU555n [ [U55Hn [SXU 4-SS 9 M [5 [ [U55Hn [S XU 4-SS 9 M [5 g s snnfs snf) a Tabulate the given samples from the frequency distribution (cumulative), displaying the most frequent sample first. If an integer parameter is supplied, stop after this many samples have been plotted. :param samples: The samples to plot (default is all samples) :type samples: list :param cumulative: A flag to specify whether the freqs are cumulative (default = False) :type title: bool rrrXFc3:# UHn[U5v M g7fr:r/.0rs r $FreqDist.tabulate..Ps1AC1#KKsc3># UHn[SU-5v M g7fz%dNr/rr}s rrrQsI;D ID4D4Dd4KL4K4KL   e< 55g>?E078f&\E8111E3D/c$URU5$)zA Create a copy of this frequency distribution. :rtype: FreqDist r!rs rcopy FreqDist.copyZs ~~d##rc@>UR[TU] U55$)zk Add counts from two counters. >>> FreqDist('abbb') + FreqDist('bcc') FreqDist({'b': 4, 'c': 2, 'a': 1}) )r!r__add__rotherr!s rrFreqDist.__add__d~~egoe455rc@>UR[TU] U55$)z Subtract count, but keep only results with positive counts. >>> FreqDist('abbbc') - FreqDist('bccd') FreqDist({'b': 2, 'a': 1}) )r!r__sub__rs rrFreqDist.__sub__nrrc@>UR[TU] U55$)z Union is the maximum of value in either of the input counters. >>> FreqDist('abbb') | FreqDist('bcc') FreqDist({'b': 3, 'c': 2, 'a': 1}) )r!r__or__rs rrFreqDist.__or__xs~~egnU344rc@>UR[TU] U55$)zr Intersection is the minimum of corresponding counts. >>> FreqDist('abbb') & FreqDist('bcc') FreqDist({'b': 1}) )r!r__and__rs rrFreqDist.__and__rrc^^[T[5(d [STT5 [T5R T5=(a [ UU4SjT55$)a Returns True if this frequency distribution is a subset of the other and for no key the value exceeds the value of the same key from the other frequency distribution. The <= operator forms partial order and satisfying the axioms reflexivity, antisymmetry and transitivity. >>> FreqDist('a') <= FreqDist('a') True >>> a = FreqDist('abc') >>> b = FreqDist('aabc') >>> (a <= b, b <= a) (True, False) >>> FreqDist('a') <= FreqDist('abcd') True >>> FreqDist('abc') <= FreqDist('xyz') False >>> FreqDist('xyz') <= FreqDist('abc') False >>> c = FreqDist('a') >>> d = FreqDist('aa') >>> e = FreqDist('aaa') >>> c <= d and d <= e and c <= e True <=c3:># UHnTUTU:*v M g7fr:r5rrrrs rr"FreqDist.__le__..s!1 /3DIs #t) isinstancer rsetissubsetallrrs``r__le__FreqDist.__le__sN6%** #D$ 64y!!%( S1 /31 .  rc^^[T[5(d [STT5 [T5R T5=(a [ UU4SjT55$)N>=c3:># UHnTUTU:v M g7fr:r5rs rr"FreqDist.__ge__..s!3 /4DIs #ur)rr rr issupersetrrs``r__ge__FreqDist.__ge__sL%** #D$ 64y##E* s3 /43 0  rc"X:*=(a X:X+$r:r5rs rFreqDist.!Dt3D!Drc"X:=(a X:X+$r:r5rs rrrrrc"UR5$)B Return a string representation of this FreqDist. :rtype: string )pformatrs r__repr__FreqDist.__repr__s ||~rc2[URUS9US9 g)z Print a string representation of this FreqDist to 'stream' :param maxlen: The maximum number of items to print :type maxlen: int :param stream: The stream to print to. stdout by default )maxlen)fileN)rr)rrstreams rpprintFreqDist.pprints dll&l)7rcURU5Vs/sHnSR"U6PM nn[U5U:aURS5 SRSR U55$s snf)z Return a string representation of this FreqDist. :param maxlen: The maximum number of items to display :type maxlen: int :rtype: string z {!r}: {!r}z...zFreqDist({{{0}}})z, )rSformatr0appendjoin)rrr6itemss rrFreqDist.pformatsh9=8H8H8PQ8P$$d+8PQ t9v  LL "))$))E*:;;RsA/c>S[U5UR54-$)rz*)r0rrs r__str__FreqDist.__str__s SSSrc#d# URUR55H upUv M g7f)zP Return an iterator which yields tokens ordered by frequency. :rtype: iterator N)rSr1)rtokenrzs r__iter__FreqDist.__iter__s) ((2HEK3s.0)rr:) N)r)$__name__ __module__ __qualname____firstlineno____doc__r rrr$r'r,r1r7r>r;rIrOrTrqrrrrrrrr__lt____gt__rrrrr__static_attributes__ __classcell__rs@rr r 9s8* &!(% :"* " ()&%%e>@!F$6656 B EF DF8 <Trr cp\rSrSrSrSr\S5r\S5rSr \S5r \S5r S r S r S rg ) ProbDistIiac A probability distribution for the outcomes of an experiment. A probability distribution specifies how likely it is that an experiment will have any given outcome. For example, a probability distribution could be used to predict the probability that a token in a document will have a given type. Formally, a probability distribution can be defined as a function mapping from samples to nonnegative real numbers, such that the sum of every number in the function's range is 1.0. A ``ProbDist`` is often used to model the probability distribution of the experiment used to generate a frequency distribution. Tcg)z> Classes inheriting from ProbDistI should implement __init__. Nr5rs rr ProbDistI.__init__rcg)z Return the probability for a given sample. Probabilities are always real numbers in the range [0, 1]. :param sample: The sample whose probability should be returned. :type sample: any :rtype: float Nr5rrHs rprobProbDistI.probrrcjURU5nUS:wa[R"US5$[$)z Return the base 2 logarithm of the probability for a given sample. :param sample: The sample whose probability should be returned. :type sample: any :rtype: float rre)rmathlog_NINF)rrHps rlogprobProbDistI.logprobs- IIf !"atxx1~2U2rcg)z Return the sample with the greatest probability. If two or more samples have the same probability, return one of them; which sample is returned is undefined. :rtype: any Nr5rs rrT ProbDistI.maxrrcg)z Return a list of all samples that have nonzero probabilities. Use ``prob`` to find the probability of each sample. :rtype: list Nr5rs rrProbDistI.samples(rrcg)zQ Return the ratio by which counts are discounted on average: c*/c :rtype: float rFr5rs rdiscountProbDistI.discount2s rcf[R"5nUnUR5H nXRU5-nUS::dMUs $ US:aW$UR(a![R "SU<SX!- <S35 [R "[UR555$)z Return a randomly selected sample from this probability distribution. The probability of returning each sample ``samp`` is equal to ``self.prob(samp)``. rg-C6?zProbability distribution z sums to z.; generate() is returning an arbitrary sample.)randomrr SUM_TO_ONEwarningswarnchoicerm)rrp_initrHs rgenerateProbDistI.generate<s MMOllnF 6" "AAv % v:M ?? MM8)r0r rs rrUniformProbDist.__repr__vs2S5IIIr)r r r N) rrrrrr rrTrrrr5rrrrTs! .$> JrrcF\rSrSrSrSr\S5rSrSr Sr Sr S r g ) RandomProbDistizz Generates a random probability distribution whereby each sample will be between 0 and 1 with equal probability (uniform random distribution. Also called a continuous uniform distribution). c[U5S:Xa [S5eURU5Ul[ URR 55Ulg)Nrz9A probability distribution must have at least one sample.)r0rRunirand_probsrmkeysr rs rr RandomProbDist.__init__sI w<1 P ll7+ T[[--/0 rcn[U5n[[U55Vs/sHn[R"5PM nn[ U5n[ U5H up%XT- X2'M [ U5nUS:waUS==US- -ss'[ U5VVs0sH up&XcU_M snn$s snfs snnf)z The key function that creates a randomized initial distribution that still sums to 1. Set as a dictionary of prob values so that it can still be passed to MutableProbDist and called with identical syntax to UniformProbDist r4)rrsr0rr enumerate)clsrrrandrowtotalxrs rrRandomProbDist.unirandsg,,1#g,,?@,?q6==?,?@G g&DAGJ'G  A: BK519 $K*3G*<=*<$!1: *<==A>s B,B1c[US5(d2[SURR555SUlUR$)N_maxc3,# UH upX!4v M g7fr:r5rvrs rr%RandomProbDist.max..sE1DvQF1Dr4)hasattrrTrrr(rs rrTRandomProbDist.maxs=tV$$E1B1B1DEEaHDIyyrc:URRUS5$r)rgetrs rrRandomProbDist.probs{{vq))rcUR$r:rrs rrRandomProbDist.samplesrrc2S[UR5-$)Nz')r0rrs rrRandomProbDist.__repr__s83t{{;KKKr)r(rr N) rrrrrr classmethodrrTrrrrr5rrrrzs5 1>>* *Lrrc@\rSrSrSrS SjrSrSrSrSr S r S r g) DictionaryProbDistiz A probability distribution whose probabilities are directly specified by a given dictionary. The given dictionary maps samples to probabilities. Nc UbUR5O0UlX lU(Ga\[U5S:Xa [ S5eU(a[ [ URR555nU[::a;[R"S[U5- S5nUHnXPRU'M gURR5HupgURU==U-ss'M g[URR55nUS:Xa&S[U5- nUHnXpRU'M gSU- nURR5HupgURU==U-ss'M gg)at Construct a new probability distribution from the given dictionary, which maps values to probabilities (or to log probabilities, if ``log`` is true). If ``normalize`` is true, then the probability values are scaled by a constant factor such that they sum to 1. If called without arguments, the resulting probability distribution assigns zero probability to all values. NrzOA DictionaryProbDist must have at least one sample before it can be normalized.rre) r _prob_dict_logr0rRsum_logsrmrrrrrr) r prob_dictr normalize value_sumlogpr%r norm_factors rr DictionaryProbDist.__init__sI/8.C)..*  9~" 5$T$//*@*@*B%CD %88C#i.$8!c)n,A&-.*'#& /K $ 5 5 7*k9*!8/ rcUR(a#XR;aSURU-$S$URRUS5$)Nrer)r<r;r1rs rrDictionaryProbDist.probsC 995;5N101 UTU U??&&vq1 1rcUR(a URRU[5$XR;a[$URUS:Xa[$[R "URUS5$)Nrre)r<r;r1rrrrs rrDictionaryProbDist.logprobs_ 99??&&vu5 5__, (A- xx 7;;rc[US5(d2[SURR555SUlUR$)Nr(c3,# UH upX!4v M g7fr:r5r*s rr)DictionaryProbDist.max..sI1HvQF1Hr-r4)r.rTr;rr(rs rrTDictionaryProbDist.maxs=tV$$I1F1F1HII!LDIyyrc6URR5$r:)r;rrs rrDictionaryProbDist.sampless##%%rc2S[UR5-$)Nz)r0r;rs rrDictionaryProbDist.__repr__s+c$//.BBBr)r<r(r;)NFF) rrrrrr rrrTrrrr5rrr9r9s' (:T2 < &Crr9c@\rSrSrSrS SjrSrSrSrSr S r S r g) MLEProbDistia  The maximum likelihood estimate for the probability distribution of the experiment used to generate a frequency distribution. The "maximum likelihood estimate" approximates the probability of each sample as the frequency of that sample in the frequency distribution. NcXlg)z Use the maximum likelihood estimate to create a probability distribution for the experiment used to generate ``freqdist``. :type freqdist: FreqDist :param freqdist: The frequency distribution that the probability estimates should be based on. N _freqdistrfreqdistr=s rr MLEProbDist.__init__s "rcUR$ze Return the frequency distribution that this probability distribution is based on. :rtype: FreqDist rSrs rrVMLEProbDist.freqdist~~rc8URRU5$r:)rTrOrs rrMLEProbDist.probs~~""6**rc6URR5$r:rTrTrs rrTMLEProbDist.max~~!!##rc6URR5$r:rTrrs rrMLEProbDist.samples~~""$$rc<SURR5-$)D :rtype: str :return: A string representation of this ``ProbDist``. z!rTrrs rrMLEProbDist.__repr__"s 3T^^5E5E5GGGrrSr:) rrrrrr rVrrTrrrr5rrrQrQs& "+$%HrrQcJ\rSrSrSrSrS SjrSrSrSr S r S r S r S r g)LidstoneProbDisti*a The Lidstone estimate for the probability distribution of the experiment used to generate a frequency distribution. The "Lidstone estimate" is parameterized by a real number *gamma*, which typically ranges from 0 to 1. The Lidstone estimate approximates the probability of a sample with count *c* from an experiment with *N* outcomes and *B* bins as ``c+gamma)/(N+B*gamma)``. This is equivalent to adding *gamma* to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. FNcLUS:XdUc>UR5S:Xa*URRSSn[SU-S-5eUbWX1R 5:aDURRSSn[SU-SU--S-S UR 5--5eXl[ U5UlUR R5UlUcUR 5nX0l URX2--Ul URS :XaSUlS Ul gg) a7 Use the Lidstone estimate to create a probability distribution for the experiment used to generate ``freqdist``. :type freqdist: FreqDist :param freqdist: The frequency distribution that the probability estimates should be based on. :type gamma: float :param gamma: A real number used to parameterize the estimate. The Lidstone estimate is equivalent to adding *gamma* to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. :type bins: int :param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If ``bins`` is not specified, it defaults to ``freqdist.B()``. rNizA %s probability distribution zmust have at least one bin.z) The number of bins in a %s distribution z&(%d) must be greater than or equal to z(the number of bins in the FreqDist used zto create it (%d).rFr4) rr!rrRr1rTfloat_gammar_bins_divisor)rrVgammar=names rr LidstoneProbDist.__init__9s$* AI4>**3B/D047:WW   4**,#6>>**3B/Drhrs rrLidstoneProbDist.__repr__s 8$..:J:J:LLLr)rrorprTrnr:)rrrrrrr rVrrTrrrrr5rrrkrk*s3 J0d1$ %#Mrrkc(\rSrSrSrSSjrSrSrg)LaplaceProbDistia The Laplace estimate for the probability distribution of the experiment used to generate a frequency distribution. The "Laplace estimate" approximates the probability of a sample with count *c* from an experiment with *N* outcomes and *B* bins as *(c+1)/(N+B)*. This is equivalent to adding one to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. Nc2[RXSU5 g)a. Use the Laplace estimate to create a probability distribution for the experiment used to generate ``freqdist``. :type freqdist: FreqDist :param freqdist: The frequency distribution that the probability estimates should be based on. :type bins: int :param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If ``bins`` is not specified, it defaults to ``freqdist.B()``. r4Nrkr rUs rr LaplaceProbDist.__init__s !!$!T:rc<SURR5-$)rgz%rhrs rrLaplaceProbDist.__repr__s 79I9I9KKKrr5r:rrrrrr rrr5rrrrs;"Lrrc(\rSrSrSrSSjrSrSrg) ELEProbDistia The expected likelihood estimate for the probability distribution of the experiment used to generate a frequency distribution. The "expected likelihood estimate" approximates the probability of a sample with count *c* from an experiment with *N* outcomes and *B* bins as *(c+0.5)/(N+B/2)*. This is equivalent to adding 0.5 to the count for each bin, and taking the maximum likelihood estimate of the resulting frequency distribution. Nc2[RXSU5 g)a: Use the expected likelihood estimate to create a probability distribution for the experiment used to generate ``freqdist``. :type freqdist: FreqDist :param freqdist: The frequency distribution that the probability estimates should be based on. :type bins: int :param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If ``bins`` is not specified, it defaults to ``freqdist.B()``. ?NrrUs rr ELEProbDist.__init__s !!$#trhrs rrELEProbDist.__repr__s 3T^^5E5E5GGGrr5r:rr5rrrrs="Hrrc\\rSrSrSrSrSSjrSrSrSr S r S r S r S r S rSrSrg)HeldoutProbDistia The heldout estimate for the probability distribution of the experiment used to generate two frequency distributions. These two frequency distributions are called the "heldout frequency distribution" and the "base frequency distribution." The "heldout estimate" uses uses the "heldout frequency distribution" to predict the probability of each sample, given its frequency in the "base frequency distribution". In particular, the heldout estimate approximates the probability for a sample that occurs *r* times in the base distribution as the average frequency in the heldout distribution of all samples that occur *r* times in the base distribution. This average frequency is *Tr[r]/(Nr[r].N)*, where: - *Tr[r]* is the total count in the heldout distribution for all samples that occur *r* times in the base distribution. - *Nr[r]* is the number of samples that occur *r* times in the base distribution. - *N* is the number of outcomes recorded by the heldout frequency distribution. In order to increase the efficiency of the ``prob`` member function, *Tr[r]/(Nr[r].N)* is precomputed for each value of *r* when the ``HeldoutProbDist`` is created. :type _estimate: list(float) :ivar _estimate: A list mapping from *r*, the number of times that a sample occurs in the base distribution, to the probability estimate for that sample. ``_estimate[r]`` is calculated by finding the average frequency in the heldout distribution of all samples that occur *r* times in the base distribution. In particular, ``_estimate[r]`` = *Tr[r]/(Nr[r].N)*. :type _max_r: int :ivar _max_r: The maximum number of times that any sample occurs in the base distribution. ``_max_r`` is used to decide how large ``_estimate`` must be. FNc8XlX lXR5UlUR 5nUR U5n[ URS-5Vs/sHoeUPM nnUR5nURXGU5Ul gs snf)aq Use the heldout estimate to create a probability distribution for the experiment used to generate ``base_fdist`` and ``heldout_fdist``. :type base_fdist: FreqDist :param base_fdist: The base frequency distribution. :type heldout_fdist: FreqDist :param heldout_fdist: The heldout frequency distribution. :type bins: int :param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If ``bins`` is not specified, it defaults to ``freqdist.B()``. r4N) _base_fdist_heldout_fdistrT_max_r _calculate_Trr;rsr_calculate_estimate _estimate) r base_fdist heldout_fdistr=Trr;r<r>rs rr HeldoutProbDist.__init__s$&+!!12    !t$$T[[1_5 65!1g5 6 OO 11"!< 7sBcS/URS--nURH+nURUnX==URU- ss'M- U$)z Return the list *Tr*, where *Tr[r]* is the total count in ``heldout_fdist`` for all samples that occur *r* times in ``base_fdist``. :rtype: list(float) rFr4)rrr)rrrHr<s rrHeldoutProbDist._calculate_Tr%sVUdkkAo &))F  (A ET((0 0E* rc/n[URS-5H9nX%S:XaURS5 MURXX%U-- 5 M; U$)a Return the list *estimate*, where *estimate[r]* is the probability estimate for any sample that occurs *r* times in the base frequency distribution. In particular, *estimate[r]* is *Tr[r]/(N[r].N)*. In the special case that *N[r]=0*, *estimate[r]* will never be used; so we define *estimate[r]=None* for those cases. :rtype: list(float) :type Tr: list(float) :param Tr: the list *Tr*, where *Tr[r]* is the total count in the heldout distribution for all samples that occur *r* times in base distribution. :type Nr: list(float) :param Nr: The list *Nr*, where *Nr[r]* is the number of samples that occur *r* times in the base distribution. :type N: int :param N: The total number of outcomes recorded by the heldout frequency distribution. r4rN)rsrr)rrr>restimater<s rr#HeldoutProbDist._calculate_estimate3sU(t{{Q'Auz% 34 ( rcUR$)zj Return the base frequency distribution that this probability distribution is based on. :rtype: FreqDist )rrs rrHeldoutProbDist.base_fdistOsrcUR$)zm Return the heldout frequency distribution that this probability distribution is based on. :rtype: FreqDist )rrs rrHeldoutProbDist.heldout_fdistXs"""rc6URR5$r:)rrrs rrHeldoutProbDist.samplesas$$&&rc>URUnURU$r:)rr)rrHr<s rrHeldoutProbDist.probds!   V $~~a  rc6URR5$r:)rrTrs rrTHeldoutProbDist.maxis##%%rc[5er:NotImplementedErrorrs rrHeldoutProbDist.discounto !##rcrSnXRR5URR54-$)rgz6)rrr)rrs rrHeldoutProbDist.__repr__rs6 E$$&&($*=*=*?*?*ABBBr)rrrrr:)rrrrrrr rrrrrrrTrrrr5rrrrsC'RJ =D 8 #'! & $Crrc@\rSrSrSrSrSrSrSrSr Sr S r S r g ) CrossValidationProbDisti{a) The cross-validation estimate for the probability distribution of the experiment used to generate a set of frequency distribution. The "cross-validation estimate" for the probability of a sample is found by averaging the held-out estimates for the sample in each pair of frequency distributions. FcXl/UlUH9nUH0nX4LdM [X4U5nURRU5 M2 M; g)a5 Use the cross-validation estimate to create a probability distribution for the experiment used to generate ``freqdists``. :type freqdists: list(FreqDist) :param freqdists: A list of the frequency distributions generated by the experiment. :type bins: int :param bins: The number of sample values that can be generated by the experiment that is described by the probability distribution. This value must be correctly set for the probabilities of the sample values to sum to one. If ``bins`` is not specified, it defaults to ``freqdist.B()``. N) _freqdists_heldout_probdistsrr)r freqdistsr=fdist1fdist2probdists rr CrossValidationProbDist.__init__sM $#%F#'.vtDH++228<$ rcUR$)zh Return the list of frequency distributions that this ``ProbDist`` is based on. :rtype: list(FreqDist) )rrs rr!CrossValidationProbDist.freqdistss rcN[[SUR5/55$)Nc38# UHn[U5v M g7fr:)rm)rfds rr2CrossValidationProbDist.samples..s;?RR?rK)rrrrs rrCrossValidationProbDist.sampless3;4??;R@AArcSnURHnX#RU5- nM U[UR5- $)NrF)rrr0)rrHrheldout_probdists rrCrossValidationProbDist.probsC $ 7 7  ))&1 1D!8c$11222rc[5er:rrs rr CrossValidationProbDist.discountrrc2S[UR5-$)rz!)r0rrs rr CrossValidationProbDist.__repr__s 3S5IIIr)rrN) rrrrrrr rrrrrrr5rrrr{s.J=6B3$JrrcF\rSrSrSrS SjrSrSrSrSr S r S r S r g) WittenBellProbDistiao The Witten-Bell estimate of a probability distribution. This distribution allocates uniform probability mass to as yet unseen events by using the number of events that have only been seen once. The probability mass reserved for unseen events is equal to *T / (N + T)* where *T* is the number of observed event types and *N* is the total number of observed events. This equates to the maximum likelihood estimate of a new type event occurring. The remaining probability mass is discounted such that all probability estimates sum to one, yielding: - *p = T / Z (N + T)*, if count = 0 - *p = c / (N + T)*, otherwise Nc Ub+X!R5:dSUR5-5eUcUR5nXlURR5UlX RR5- UlURR 5UlUR S:XaSUR- UlgURURUR UR--- Ulg)ay Creates a distribution of Witten-Bell probability estimates. This distribution allocates uniform probability mass to as yet unseen events by using the number of events that have only been seen once. The probability mass reserved for unseen events is equal to *T / (N + T)* where *T* is the number of observed event types and *N* is the total number of observed events. This equates to the maximum likelihood estimate of a new type event occurring. The remaining probability mass is discounted such that all probability estimates sum to one, yielding: - *p = T / Z (N + T)*, if count = 0 - *p = c / (N + T)*, otherwise The parameters *T* and *N* are taken from the ``freqdist`` parameter (the ``B()`` and ``N()`` values). The normalizing factor *Z* is calculated using these values along with the ``bins`` parameter. :param freqdist: The frequency counts upon which to base the estimation. :type freqdist: FreqDist :param bins: The number of possible event types. This must be at least as large as the number of bins in the ``freqdist``. If None, then it's assumed to be equal to that of the ``freqdist`` :type bins: int Nz4bins parameter must not be less than %d=freqdist.B()rr)r1rT_T_Zrr_P0rUs rr WittenBellProbDist.__init__s6|tzz|3 BXZZ\ Q 3 <::"?@DHrczURUnUS:waX RUR-- $UR$r)rTrrrrws rrWittenBellProbDist.probs5 NN6 "*+q&qGGdgg%&>dhh>rc6URR5$r:r_rs rrTWittenBellProbDist.maxrarc6URR5$r:rcrs rrWittenBellProbDist.samplesrercUR$r:rSrs rrVWittenBellProbDist.freqdist ~~rc[5er:rrs rrWittenBellProbDist.discount rrc<SURR5-$)rz(rhrs rrWittenBellProbDist.__repr__ s :DNN 0 )rzipsortedr)rnonzeros rrBSimpleGoodTuringProbDist._r_Nrs4%%'r6MF7==?+,,rcU(aU(dg/n[[U55HRnUS:aXS- OSnU[U5S- :Xa SX-U- OXS-nSX$-Xe- - nURU5 MT UVs/sHn[R"U5PM nnUVs/sHn[R"U5PM n nS=p[ U5[U5- n [ U 5[U 5- n [ X5HupXU - X- -- n XU - S-- n M U S:waX- OSUlURS:a[R"S5 XRU -- Ul gs snfs snf) z Use simple linear regression to tune parameters self._slope and self._intercept in the log-log space based on count and Nr(count) (Work in log space to avoid floating point underflow.) Nrr4reg@rFr zSimpleGoodTuring did not find a proper best fit line for smoothing probabilities of occurrences. The probability estimates are likely to be unreliable.) rsr0rrrrr_sloperr _intercept)rr<rzrjrkzr_log_rlog_zrxy_covx_varx_meany_meanr%ys rr&SimpleGoodTuringProbDist.find_best_fits`  s1vAEa%qA !SVaZAD1 Q1uXA+'C IIcN  '((a!a(')*r!$((1+r*Uc%j(Vs6{*&DA 6zaj1 1F &jQ& &E'). fn ;;"  MM  !;;#77#)*s E1& E6c [U5Hup4[U5US-:XdXS-US-:waX@l gURnUS-U"US-5-U"U5- nUS-X#S--X#- n[R "UR XBUX#S-55n[Xv- 5SU-::dMX@l g g)zT Calculate the r frontier where we must switch from Nr to Sr when estimating E[Nr]. r4g\(\?N)r!r0 _switch_at smoothedNrrsqrt _varianceabs) rr<rrr_Sr smooth_r_starunsmooth_r_starstds rr SimpleGoodTuringProbDist._switchs q\EA1vQ!E(b1f"4"$B!Vr"q&z1BrF:M!AvE2RU:O))DNN2!ubQi@AC?23tczA"$"rct[U5n[U5n[U5nUS-S-X2S-- -SX2- --$)Nrre)rm)rr<rnr_1s rr"SimpleGoodTuringProbDist._variancesA !H 2YT{CA~A.# /BBrcSn[X5HupEX5URU5-- nM U(aSURS5- U- Ulgg)aI It is necessary to renormalize all the probability estimates to ensure a proper probability distribution results. This can be done by keeping the estimate of the probability mass for unseen items as N(1)/N and renormalizing all the estimates for previously seen items (as Gale and Sampson (1995) propose). (See M&S P.213, 1999) rFr4rN)r _prob_measure _renormal)rr<rprob_covrnr_s rr%SimpleGoodTuringProbDist._renormalizesU1zGB d0044 4H" $"4"4Q"778CDN rc[R"URUR[R"U5--5$)zk Return the number of samples with count r. :param r: The amount of frequency. :type r: int :rtype: float )rexprrr)rr<s rr#SimpleGoodTuringProbDist.smoothedNrs-xx$++ *CCDDrcURUnURU5nUS:XaXURURR5:XaSnU$X0RURR5- - nU$X0R-nU$)zf Return the sample's probability. :param sample: sample of the event :type sample: str :rtype: float rrF)rTrror1r)rrHrCrs rrSimpleGoodTuringProbDist.probsv&   u % A:zzT^^--// dnn&6&6&889NN"Arc,US:XaURR5S:XagUS:XaTURR5S:wa6URRS5URR5- $URU:a:URRUS-5nURRU5nO%UR US-5nUR U5nUS-U-U- nX@RR5- $)Nrrr4)rTrr>rr)rrCEr_1Err_stars rr&SimpleGoodTuringProbDist._prob_measures A:$..**,1 aZDNN,,.!3>>$$Q'$..*:*:*<< < ??U ">>$$UQY/D""5)B??519-D'B!)t#b(((***rcSn[S[UR55H3nXRUURU5-UR- - nM5 [ SU5 g)NrFrzProbability Sum:)rsr0_Nrrrr)rprob_sumrs rcheckSimpleGoodTuringProbDist.checksTq#dhh-(A  d&8&8&;;dnnL LH)  (+rcZURS5URR5- $)zl This function returns the total mass of probability transfers from the seen samples to the unseen samples. r4)rrTrrs rr!SimpleGoodTuringProbDist.discounts% q!DNN$4$4$666rc6URR5$r:r_rs rrTSimpleGoodTuringProbDist.maxrarc6URR5$r:rcrs rr SimpleGoodTuringProbDist.samples rercUR$r:rSrs rrV!SimpleGoodTuringProbDist.freqdist#rrc<SURR5-$)rz.rhrs rr!SimpleGoodTuringProbDist.__repr__&s @$..BRBRBTTTr)rorTrrrrr:)rrrrrrr rrBrrrrrrrr#rrTrrVrrr5rrrrPsc&J!, -'8R(C DE &+ ,7$%UrrcD\rSrSrSrS SjrSrSrSrSr S Sjr S r g ) MutableProbDisti/z An mutable probdist where the probabilities may be easily modified. This simply copies an existing probdist, storing the probability values in a mutable dictionary and providing an update method. cX l[[U55Vs0sHoBUU_M snUl[R"SS/5[U5-Ul[[U55HLnU(a"UR X$5UR U'M,URX$5UR U'MN X0lgs snf)a Creates the mutable probdist based on the given prob_dist and using the list of samples given. These values are stored as log probabilities if the store_logs flag is set. :param prob_dist: the distribution from which to garner the probabilities :type prob_dist: ProbDist :param samples: the complete set of samples :type samples: sequence of any :param store_logs: whether to store the probabilities as logarithms :type store_logs: bool drFN) r rsr0 _sample_dictarray_datarr_logs)r prob_distr store_logsrs rr MutableProbDist.__init__6s 49#g,4GH4GqQZ]4GH[[se,s7|; s7|$A ) 1 1'* = 1 )wz : 1 %  IsCc\[SURR555S$)Nc3,# UH upX!4v M g7fr:r5r*s rr&MutableProbDist.max..PsB(AfqA6(Ar-r4)rTr3rrs rrTMutableProbDist.maxNs'B(9(9(?(?(ABB1EErcUR$r:rrs rrMutableProbDist.samplesRs }}rcURRU5nUcgUR(aSURU-$URU$)NrFre)r3r1r6r5rrHrs rrMutableProbDist.probVsF    ! !& ) 9'+zzqTZZ]#Dtzz!}DrcURRU5nUc [S5$UR(aURU$[ R "URUS5$)Nz-infre)r3r1rmr6r5rrrAs rrMutableProbDist.logprob]sS    ! !& ) 9= $ tzz!}JA0JJrcURRU5nUceUR(a.U(aUO[R"US5UR U'gU(aSU-OUUR U'g)a Update the probability for the given sample. This may cause the object to stop being the valid probability distribution - the user must ensure that they update the sample probabilities such that all samples have probabilities between 0 and 1 and that all probabilities sum to one. :param sample: the sample for which to update the probability :type sample: any :param prob: the new probability :type prob: float :param log: is the probability already logged :type log: bool Nre)r3r1r6rrr5)rrHrrrs rr'MutableProbDist.updateds\    ! !& )}} ::$'DTXXdA->DJJqM+.A$KDDJJqMr)r5r6r3r N)T) rrrrrr rTrrrr'rr5rrr0r0/s(  0FEK9rr0cF\rSrSrSrS SjrSrSrSrSr S r S r S r g) KneserNeyProbDistiaf Kneser-Ney estimate of a probability distribution. This is a version of back-off that counts how likely an n-gram is provided the n-1-gram had been seen in training. Extends the ProbDistI interface, requires a trigram FreqDist instance to train on. Optionally, a different from default discount value can be specified. The default discount is set to 0.75. NcU(dUR5UlOX lX0l0Ul[ [ 5UlXl[ [5Ul [ [5Ul [ [5Ul UHiupEnUR XE4==XXV4- ss'URXE4==S- ss'URU==S- ss'URXV4==S- ss'Mk g)aM :param freqdist: The trigram frequency distribution upon which to base the estimation :type freqdist: FreqDist :param bins: Included for compatibility with nltk.tag.hmm :type bins: int or float :param discount: The discount applied when retrieving counts of trigrams :type discount: float (preferred, but can be set to int) r4N) r1ro_D_cacherrA_bigrams _trigramsrm_wordtypes_after_trigrams_contain_wordtypes_before)rrVr=rw0w1w2s rr KneserNeyProbDist.__init__s!DJJ $C( !!,E 2!,U!3!,U!3"JBB MM2( #xR '= = #  ! !2( +q 0 +  " "2 &! + &  " "B8 , 1 , #rc4[U5S:wa [S5e[U5nUup#nXR;aURU$XR;a2URUUR 5- UR X#4- nOX#4UR ;amX44UR;a\URX#4nURX44nX`R 5-UR X#4- nXpRUU- - n X-nOSnXPRU'U$)Nz$Expected an iterable with 3 members.rF) r0rRtuplerKrMrrLrPrNrO) rtrigramrQrRrSraftrbfr leftover_probbetas rrKneserNeyProbDist.probs w<1 CD D.  kk !;;w' '..(w/$--/AT]]HF T]]*x4;Q;Q/Q,,bX6,,bX6"& !74==";R R 44R84?@$+#'KK KrcUR$)zY Return the value by which counts are discounted. By default set to 0.75. :rtype: float rJrs rrKneserNeyProbDist.discounts wwrcXlg)z Set the value by which counts are discounted to the value of discount. :param discount: the new value to discount counts by :type discount: float (preferred, but int possible) :rtype: None Nr_)rrs r set_discountKneserNeyProbDist.set_discounts rc6URR5$r:)rMrrs rrKneserNeyProbDist.samplesrerc6URR5$r:)rMrTrs rrTKneserNeyProbDist.maxrarc>SURR5S3$)z> Return a string representation of this ProbDist :rtype: str z.>.@-AKKr)rJrLrorKrMrOrNrP)Ng?) rrrrrr rrrbrrTrrr5rrrHrHs-!2F"H%$LrrHc^^[T[5(a[T[5(d [S5e[UU4SjT55$)Nzexpected a ProbDist.c3># UH=nTRU5[R"TRU5S5-v M? g7freN)rrr)rr actual_pdist test_pdists rr!log_likelihood..s8HT1 !txx (:A>> sAA)rrrRr)rnrms``rlog_likelihoodrpsB j) , ,J|Y4W4W/00 HT rc^^U4SjTR55n[SU55*$)Nc3F># UHnTRU5v M g7fr:)r)rrpdists rrentropy.. s 4OqUZZ]]Os!c3T# UHo[R"US5-v M g7frl)rr)rrs rrrt!s2EqDHHQN"Es&()rr)rsprobss` rentropyrws' 4EMMO 4E 2E2 2 22rc\rSrSrSrSSjrSrSrSrSSS S SS S .S jr S r S r Sr Sr SrSrSrSrSrSr\rSrSrg)ConditionalFreqDisti)a& A collection of frequency distributions for a single experiment run under different conditions. Conditional frequency distributions are used to record the number of times each sample occurred, given the condition under which the experiment was run. For example, a conditional frequency distribution could be used to record the frequency of each word (type) in a document, given its length. Formally, a conditional frequency distribution can be defined as a function that maps from each condition to the FreqDist for the experiment under that condition. Conditional frequency distributions are typically constructed by repeatedly running an experiment under a variety of conditions, and incrementing the sample outcome counts for the appropriate conditions. For example, the following code will produce a conditional frequency distribution that encodes how often each word type occurs, given the length of that word type: >>> from nltk.probability import ConditionalFreqDist >>> from nltk.tokenize import word_tokenize >>> sent = "the the the dog dog some other words that we do not care about" >>> cfdist = ConditionalFreqDist() >>> for word in word_tokenize(sent): ... condition = len(word) ... cfdist[condition][word] += 1 An equivalent way to do this is with the initializer: >>> cfdist = ConditionalFreqDist((len(word), word) for word in word_tokenize(sent)) The frequency distribution for each condition is accessed using the indexing operator: >>> cfdist[3] FreqDist({'the': 3, 'dog': 2, 'not': 1}) >>> cfdist[3].freq('the') 0.5 >>> cfdist[3]['dog'] 2 When the indexing operator is used to access the frequency distribution for a condition that has not been accessed before, ``ConditionalFreqDist`` creates a new empty FreqDist for that condition. Nc~[R"U[5 U(aUHup#XU==S- ss'M gg)a" Construct a new empty conditional frequency distribution. In particular, the count for every sample, under every condition, is zero. :param cond_samples: The samples to initialize the conditional frequency distribution with :type cond_samples: Sequence of (condition, sample) tuples r4N)rr r )r cond_samplescondrHs rr ConditionalFreqDist.__init__Ys: T8,  ,  6"a'"!- rcZ^U4SjTR55nTRSSSU4$)Nc30># UH oTU4v M g7fr:r5)rr|rs rr1ConditionalFreqDist.__reduce__..jsE3D44:&3Dsr5) conditionsr!)rkv_pairss` r __reduce__ConditionalFreqDist.__reduce__is)E4??3DED$99rc4[UR55$)a Return a list of the conditions that have been accessed for this ``ConditionalFreqDist``. Use the indexing operator to access the frequency distribution for a given condition. Note that the frequency distributions for some conditions may contain zero sample outcomes. :rtype: list rmrrs rrConditionalFreqDist.conditionsmsDIIK  rcB[SUR555$)zr Return the total number of sample outcomes that have been recorded by this ``ConditionalFreqDist``. :rtype: int c3@# UHoR5v M g7fr:rM)rfdists rr(ConditionalFreqDist.N..s8-7799-s)rrrs rrConditionalFreqDist.Nys8$++-888rrVF)rrWrXrYrrZcSSKJn U(dUR 5nOUV s/sH oU;dM U PM nn U(d)[ UV V s1sHoU HoiM M sn n 5nSU;aSUS'U R 5n U(Ga|/nUH~nU(a[XRU55nOUVs/sH nXUPM nnU(a(UVs/sHnUXR5- S-PM nnURU5 M U(aSnSnOS nS nU(aUS - nOUS - nSnUH&nUUUS 'US- nU R"U/UQ70UD6 M( U RUS9 U RSSS9 U R[[!U555 U R#UVs/sHn[%U5PM snSS9 U(aU R'U5 U R)S5 U R+U5 U(aU R-5 U $![an [S5U eSn A ff=fs sn fs sn n fs snfs snfs snf)a Plot the given samples from the conditional frequency distribution. For a cumulative plot, specify cumulative=True. Additional ``*args`` and ``**kwargs`` are passed to matplotlib's plot function. (Requires Matplotlib to be installed.) :param samples: The samples to plot :type samples: list :param title: The title for the graph :type title: str :param cumulative: Whether the plot is cumulative. (default = False) :type cumulative: bool :param percents: Whether the plot uses percents instead of counts. (default = False) :type percents: bool :param conditions: The conditions to plot (default is all) :type conditions: list :param show: Whether to show the plot, or only return the ax. :type show: bool rNr\rdrer^r]z lower rightrVz upper rightr_r`labelr4)locTrarbrfrgri)rjrkrlrRrrrnrmrIrrrqlegendrorrrsr0rtrurprvrwrZ)rrrWrXrYrrZr(r)rxryrxr+r~r{ conditionrOrHr}r| legend_locrrs rrqConditionalFreqDist.plots5<  +*J%/=Z9!ZJ=EA!WaWaEFG f $"#F;  WWY E'  G G PQDBIJ'DOF3'DJCGH4aA 1 1 33c94DH T"(&* * *$("A",Q-wQ.t.v. II*I % GGDG ) MM%G - .   81A82  F U# MM) $ MM& !  HHJ u .  >EKI0 9s9H! H?H? I ;I "I3I! H<+ H77H<c [USS5n[US[UR555n[US[UVVs1sHoUU;dM XHofiM M snn55n[SU55n[ 5n UHanU(a[ XR U55X'OUV s/sH oUU PM sn X'[U[SX555nMc [SU55n [SU -SS 9 UHn [S X4-SS 9 M [5 UH4n[S X4-SS 9 XHn [S X4-SS 9 M [5 M6 g s snnfs sn f) a> Tabulate the given samples from the conditional frequency distribution. :param samples: The samples to plot :type samples: list :param conditions: The conditions to plot (default is all) :type conditions: list :param cumulative: A flag to specify whether the freqs are cumulative (default = False) :type title: bool rXFrrc3># UHn[SU-5v M g7f%sNr/rs rr/ConditionalFreqDist.tabulate..s37aCqMM7rc3># UHn[SU-5v M g7frr/rs rrrs"C(Q3tax==(rc3># UHn[SU-5v M g7frr/)rrxs rrrs?JqS]]JrrrrrN)rrrrTdictrmrIr)rr(r)rXrrxr+rrr{rHcondition_sizerr}s rrConditionalFreqDist.tabulatesb  e<  fT__=N6OP    zHz!$YA1AAzH I  3733A ? ? HI:AB'GFO'Bs"C%("CCDE ?J?? cN",A %5*$# . A %>--3 7Xeuj(c2 G #ICs E+E+8E1c[U[5(d[$UR5nUR 5HnX#==X- ss'M U$)z+ Add counts from two ConditionalFreqDists. rryNotImplementedrrrrresultr|s rrConditionalFreqDist.__add__I%!455! !$$&D LEK 'L' rc[U[5(d[$UR5nUR 5HnX#==X-ss'X#(aMX# M U$)z= Subtract count, but keep only results with positive counts. rrs rrConditionalFreqDist.__sub__sW%!455! !$$&D LEK 'L<<L' rc[U[5(d[$UR5nUR 5HnX#==X-ss'M U$)z@ Union is the maximum of value in either of the input counters. rrs rrConditionalFreqDist.__or__ rrc[U[5(d[$[5nUR5HnXX-nU(dMXBU'M U$)z6 Intersection is the minimum of corresponding counts. )rryrr)rrrr| newfreqdists rrConditionalFreqDist.__and__+sR%!455! !$&OO%D*u{2K{*t & rc ^^[T[5(d [STT5 [TR 55R TR 55=(a$ [ UU4SjTR 555$)Nrc3:># UHnTUTU:*v M g7fr:r5)rrxrrs rr-ConditionalFreqDist.__le__..<s#K ):ADGuQx ):r)rryrrrrrrs``rrConditionalFreqDist.__le__9sh%!455 #D$ 64??$%..u/?/?/AB sK )-):K H  rcb[U[5(d [SX5 X:*=(a X:g$)N<rryrrs rrConditionalFreqDist.__lt__@s)%!455 #C 5}..rcL[U[5(d [SX5 X:*$)Nrrrs rrConditionalFreqDist.__ge__Es"%!455 #D$ 6}rcL[U[5(d [SX5 X:$)N>rrs rrConditionalFreqDist.__gt__Js"%!455 #C 5|rcSSKJn U"U5$)Nr)deepcopy)rr)rrs rrConditionalFreqDist.deepcopyOs!~rcS[U5-$)zN Return a string representation of this ``ConditionalFreqDist``. :rtype: str z(r/rs rrConditionalFreqDist.__repr__Vs :CIEErr5r:)rrrrrr rrrrqrrrrrrrrrrrrrr5rrryry)sw-^( : !9 Zx&T     /    DFrryc4\rSrSrSr\S5rSrSrSr g)ConditionalProbDistIi_aC A collection of probability distributions for a single experiment run under different conditions. Conditional probability distributions are used to estimate the likelihood of each sample, given the condition under which the experiment was run. For example, a conditional probability distribution could be used to estimate the probability of each word type in a document, given the length of the word type. Formally, a conditional probability distribution can be defined as a function that maps from each condition to the ``ProbDist`` for the experiment under that condition. cg)zI Classes inheriting from ConditionalProbDistI should implement __init__. Nr5rs rr ConditionalProbDistI.__init__mrrc4[UR55$)z Return a list of the conditions that are represented by this ``ConditionalProbDist``. Use the indexing operator to access the probability distribution for a given condition. :rtype: list rrs rrConditionalProbDistI.conditionsssDIIK  rcHS[U5R[U54-$)zN Return a string representation of this ``ConditionalProbDist``. :rtype: str z<%s with %d conditions>)typerr0rs rrConditionalProbDistI.__repr__}s" )DJ,?,?T+KKKrr5N) rrrrrrr rrrr5rrrr_s&   !Lrrc$\rSrSrSrSrSrSrg)ConditionalProbDistiaZ A conditional probability distribution modeling the experiments that were used to generate a conditional frequency distribution. A ConditionalProbDist is constructed from a ``ConditionalFreqDist`` and a ``ProbDist`` factory: - The ``ConditionalFreqDist`` specifies the frequency distribution for each condition. - The ``ProbDist`` factory is a function that takes a condition's frequency distribution, and returns its probability distribution. A ``ProbDist`` class's name (such as ``MLEProbDist`` or ``HeldoutProbDist``) can be used to specify that class's constructor. The first argument to the ``ProbDist`` factory is the frequency distribution that it should model; and the remaining arguments are specified by the ``factory_args`` parameter to the ``ConditionalProbDist`` constructor. For example, the following code constructs a ``ConditionalProbDist``, where the probability distribution for each condition is an ``ELEProbDist`` with 10 bins: >>> from nltk.corpus import brown >>> from nltk.probability import ConditionalFreqDist >>> from nltk.probability import ConditionalProbDist, ELEProbDist >>> cfdist = ConditionalFreqDist(brown.tagged_words()[:5000]) >>> cpdist = ConditionalProbDist(cfdist, ELEProbDist, 10) >>> cpdist['passed'].max() 'VBD' >>> cpdist['passed'].prob('VBD') #doctest: +ELLIPSIS 0.423... cZX lX0lX@lUHnU"X/UQ70UD6X'M g)a Construct a new conditional probability distribution, based on the given conditional frequency distribution and ``ProbDist`` factory. :type cfdist: ConditionalFreqDist :param cfdist: The ``ConditionalFreqDist`` specifying the frequency distribution for each condition. :type probdist_factory: class or function :param probdist_factory: The function or class that maps a condition's frequency distribution to its probability distribution. The function is called with the frequency distribution as its first argument, ``factory_args`` as its remaining arguments, and ``factory_kw_args`` as keyword arguments. :type factory_args: (any) :param factory_args: Extra arguments for ``probdist_factory``. These arguments are usually used to specify extra properties for the probability distributions of individual conditions, such as the number of bins they contain. :type factory_kw_args: (any) :param factory_kw_args: Extra keyword arguments for ``probdist_factory``. N)_probdist_factory _factory_args_factory_kw_args)rcfdistprobdist_factory factory_argsfactory_kw_argsrs rr ConditionalProbDist.__init__sB0"2) /I.!$04CDO rcrUR"[5/URQ70URD6X'X$r:)rr rrrrs r __missing__ConditionalProbDist.__missing__s?** J ++ /3/D/D  yr)rrrNrrrrrr rrr5rrrrsBBrrc$\rSrSrSrSrSrSrg)DictionaryConditionalProbDistiz{ An alternative ConditionalProbDist that simply wraps a dictionary of ProbDists rather than creating these from FreqDists. c&URU5 g)z :param probdist_dict: a dictionary containing the probdists indexed by the conditions :type probdist_dict: dict any -> probdist N)r')r probdist_dicts rr &DictionaryConditionalProbDist.__init__s M"rc"[5X'X$r:)r9rs rr)DictionaryConditionalProbDist.__missing__s&( yrr5Nrr5rrrrs #rrgKH9recX[-:aU$X[-:aU$[X5nU[R"SX- -SX- --S5-$)z Given two numbers ``logx`` = *log(x)* and ``logy`` = *log(y)*, return *log(x+y)*. Conceptually, this is the same as returning ``log(2**(logx)+2**(logy))``, but the actual implementation avoids overflow errors that could result from direct computation. re)_ADD_LOGS_MAX_DIFFminrr)logxlogybases radd_logsrsX '''  ''' t?D $((1-dk0BBAF FFrcZ[U5S:wa[[USSUS5$[$)Nrr4)r0rrr)logss rr=r=s*25d)q.6(DHd1g .KeKrc6\rSrSrSrSrSrSrSrSr Sr g ) ProbabilisticMixIni a A mix-in class to associate probabilities with other classes (trees, rules, etc.). To use the ``ProbabilisticMixIn`` class, define a new class that derives from an existing class and from ProbabilisticMixIn. You will need to define a new constructor for the new class, which explicitly calls the constructors of both its parent classes. For example: >>> from nltk.probability import ProbabilisticMixIn >>> class A: ... def __init__(self, x, y): self.data = (x,y) ... >>> class ProbabilisticA(A, ProbabilisticMixIn): ... def __init__(self, x, y, **prob_kwarg): ... A.__init__(self, x, y) ... ProbabilisticMixIn.__init__(self, **prob_kwarg) See the documentation for the ProbabilisticMixIn ``constructor<__init__>`` for information about the arguments it expects. You should generally also redefine the string representation methods, the comparison methods, and the hashing method. c SU;a*SU;a [S5e[RXS5 gSU;a[RXS5 gS=UlUlg)aY Initialize this object's probability. This initializer should be called by subclass constructors. ``prob`` should generally be the first argument for those constructors. :param prob: The probability associated with the object. :type prob: float :param logprob: The log of the probability associated with the object. :type logprob: float rrz.Must specify either prob or logprob (not both)N) TypeErrorrset_prob set_logprob_ProbabilisticMixIn__prob_ProbabilisticMixIn__logprob)rr)s rr ProbabilisticMixIn.__init__ s[ V F" STT"++D.A &  * *4 1B C+/ /DK$.rcXlSUlg)zr Set the probability associated with this object to ``prob``. :param prob: The new probability :type prob: float Nrrrrs rrProbabilisticMixIn.set_prob3 s rcXlSUlg)z Set the log probability associated with this object to ``logprob``. I.e., set the probability associated with this object to ``2**(logprob)``. :param logprob: The new log probability :type logprob: float N)rr)rrs rrProbabilisticMixIn.set_logprob= s! rcxURc"URcgSUR-UlUR$)zD Return the probability associated with this object. :rtype: float Nrerrs rrProbabilisticMixIn.probI s5 ;; ~~%/DK{{rcURc4URcg[R"URS5UlUR$)z_ Return ``log(p)``, where ``p`` is the probability associated with this object. :rtype: float Nre)rrrrrs rrProbabilisticMixIn.logprobU s; >> !{{"!XXdkk15DN~~r) __logprob__probN) rrrrrr rrrrrr5rrrr s 20,   rrc \rSrSrSrSrSrg)ImmutableProbabilisticMixInic cF[SURR-5eNz%s is immutablerRr!rrs rr$ImmutableProbabilisticMixIn.set_probd *T^^-D-DDEErcF[SURR-5errrs rr'ImmutableProbabilisticMixIn.set_logprobg rrr5N)rrrrrrrr5rrrrc sFFrrc$X;aXnX U$UnU$r:r5)r)rdefaultargs rrrn s& }k K J Jrc[5n[U5HGn[R"SSU-S-5[R"SUS-5-nX$==S- ss'MI U$)z Create a new frequency distribution, with random samples. The samples are numbers from 1 to ``numsamples``, and are generated by summing two numbers, each of which has a uniform distribution. r4rer)r rsrrandint) numsamples numoutcomesrr%rs r_create_rand_fdistr | s` JE ;  NN1q:~!3 4v~~ zQ8   A   Lrc[5n[SSU-S-S-5H+n[SUS-S-5HnXU-==S- ss'M M- [U5$)zd Return the true probability distribution for the experiment ``_create_rand_fdist(numsamples, x)``. r4rer)r rsrQ)r rr%rs r_create_sum_pdistr sa JE 1q:~!+a/ 0q*/A-.A a%LA L/1 u rc ^[X5n[X5n[X5n[U5[USU5[X#U5[X2U5[ X#U/U5[ U5[ US5[ U5/n/n[SUS-5HOnUR[XrRU5/UVs/sHoRU5PM sn-55 MQ [SXU4-5 [S[U5S--5 SS[U5S- --S -n [U [S US S 55-5 [S [U5S--5 SS[U5S- --S-n UHn [X-5 M [[U65n U SS V s/sHn [!U 5PM n n [S [U5S--5 SS[U5--S-n [U [U 5-5 [S[U5S--5 [SU-5S:a*[SU-5 [SU-5 [SU-5 [5 [S5 UHYm[#U4Sj[S555n [SR%TR&R(S SSU -S S55 M[ [5 g s snfs sn f)a A demonstration of frequency distributions and probability distributions. This demonstration creates three frequency distributions with, and uses them to sample a random process with ``numsamples`` samples. Each frequency distribution is sampled ``numoutcomes`` times. These three frequency distributions are then used to build six probability distributions. Finally, the probability estimates of these distributions are compared to the actual probability of each sample. :type numsamples: int :param numsamples: The number of samples to use in each demo frequency distributions. :type numoutcomes: int :param numoutcomes: The total number of outcomes for each demo frequency distribution. These outcomes are divided into ``numsamples`` bins. :rtype: None rr4z=%d samples (1-%d); %d outcomes were sampled for each FreqDistz =========rez FreqDist z%8s z | Actualc3># UHn[U5SSv M g7f)r4 N)repr)rrss rrdemo.. sF+DK!,+rNr z ---------z %3d %8.6f z%8.6f z| %8.6fzTotal rFz fdist1: %sz fdist2: %sz fdist3: %sz Generating:c3D># UHnTR5v M g7fr:)r)rrrss rrr s?;a));s z {:>20} {}7)r rQrkrrrrrsrrWrOrrr0rmrrr rr!r)r r rrfdist3pdistsvalsrNrs FORMATSTRr zvalssumsrs ` rdemor! s, 8F  8F  8F Fj1 3 3 8*E ( +*% F D 1j1n % E1kk!n-F0SF5AF0SSTU& G ; / 0 'S[1_ %&!Fc&kAo$>>LI )eF&"+FF FG 'S[1_ %&S[1_!== II io d E %ab * CH D * 'S[1_ %&8s6{33i?I )eDk !" 'S[1_ %& 4&=B nv%& nv%& nv%& G -?5;?? k  !9!9#2!>sPR@STU GE1T" +s 6K"Kc BSSKJn URRS5n[ U5n[ U5n[ SRSSS55 S[UR5S S S 95nUH%n[ S XRUURU54-5 M' g) Nr)corpuszausten-emma.txtz{:>18} {:>8} {:>14}word frequencySimpleGoodTuringc3*# UH upUv M g7fr:r5)rrvalues rrgt_demo.. sX Xsc US$)Nr4r5)r6s rrgt_demo.. s$q'rT)rreversez%18s %8d %14e) nltkr# gutenbergwordsr rrrrrr)r# emma_wordsrsgtfd_keys_sortedrs rgt_demor3 s!!''(9:J * B "2 &C ' ' =O PQ$RXXZ5ISWXN ##w !>>?r__main__rr)ryrrrrr9rr rrrrrkrQr0rHrrrrrrpr=rw)ri)4rr4rrrabcrr collectionsrr functoolsrnltk.internalsrrmrr rrrr9rQrkrrrrrrr0rHrprwryrrrrrrrr=rrrr rr!r3r__all__r5rrr;s2 ',2 hlwlh d3'd3N#Ji#JL1LY1LhKCKC\+H)+H\aMyaMH!L&!LH"H""HJaCiaCHCJiCJLQOQO^\Uy\U~I9iI9`oL oLn3sF+sFl $L47$LNG.GT$80XXeQ' GL]]@F"4F  K \ @ zBKDM I r