o rZŽhšã@s¬dZddlmZmZddlmZmZmZGdd„deƒZGdd„deƒZ Gdd „d e ƒZ Gd d „d eƒZ Gd d „d eƒZ Gdd„de ƒZ Gdd„de ƒZGdd„de ƒZdS)zLanguage Modelsé)Ú LanguageModelÚ Smoothing)ÚAbsoluteDiscountingÚ KneserNeyÚ WittenBellc@seZdZdZddd„ZdS)ÚMLEzbClass for providing MLE ngram model scores. Inherits initialization from BaseNgramModel. NcCs| |¡ |¡S)zÃReturns the MLE score for a word given a context. Args: - word is expected to be a string - context is expected to be something reasonably convertible to a tuple )Úcontext_countsÚfreq)ÚselfÚwordÚcontext©r ú=/var/www/auris/lib/python3.10/site-packages/nltk/lm/models.pyÚunmasked_scoreszMLE.unmasked_score©N)Ú__name__Ú __module__Ú __qualname__Ú__doc__rr r r rrsrcó*eZdZdZ‡fdd„Zddd„Z‡ZS)ÚLidstonez«Provides Lidstone-smoothed scores. In addition to initialization arguments from BaseNgramModel also requires a number by which to increase the counts, gamma. cótƒj|i|¤Ž||_dSr)ÚsuperÚ__init__Úgamma)r rÚargsÚkwargs©Ú __class__r rr%ó zLidstone.__init__NcCs8| |¡}||}| ¡}||j|t|jƒ|jS)ztAdd-one smoothing: Lidstone or Laplace. To see what kind, look at `gamma` attribute on the class. )rÚNrÚlenÚvocab©r r r ÚcountsZ word_countZ norm_countr r rr)s zLidstone.unmasked_scorer©rrrrrrÚ __classcell__r r rrrs rcó eZdZdZ‡fdd„Z‡ZS)ÚLaplacezwImplements Laplace (add one) smoothing. Initialization identical to BaseNgramModel because gamma is always 1. cstƒjdg|¢Ri|¤ŽdS)Né)rr)r rrrr rr;szLaplace.__init__©rrrrrr&r r rrr(5sr(cs,eZdZdZd‡fdd„ Zd dd„Z‡ZS) Ú StupidBackoffa8Provides StupidBackoff scores. In addition to initialization arguments from BaseNgramModel also requires a parameter alpha with which we scale the lower order probabilities. Note that this is not a true probability distribution as scores for ngrams of the same order do not sum up to unity. çš™™™™™Ù?crr)rrÚalpha)r r-rrrr rrHrzStupidBackoff.__init__NcCsV|s |jj |¡S| |¡}||}| ¡}|dkr||S|j| ||dd…¡S)Nrr))r$Zunigramsr rr r-rr#r r rrLs zStupidBackoff.unmasked_score)r,rr%r r rrr+?sr+cr)ÚInterpolatedLanguageModelz¡Logic common to all interpolated language models. The idea to abstract this comes from Chen & Goodman 1995. Do not instantiate this class directly! c s<| di¡}tƒj|fi|¤Ž||j|jfi|¤Ž|_dS)NÚparams)Úpoprrr"r$Ú estimator)r Z smoothing_clsÚorderrr/rr rr`s z"InterpolatedLanguageModel.__init__NcCsR|s|j |¡S|j|sd\}}n |j ||¡\}}||| ||dd…¡S)N)rr)r))r1Z unigram_scorer$Z alpha_gammar)r r r r-rr r rres    z(InterpolatedLanguageModel.unmasked_scorerr%r r rrr.Ys r.cr')ÚWittenBellInterpolatedz.Interpolated version of Witten-Bell smoothing.c stƒjt|fi|¤ŽdSr)rrr)r r2rrr rrvszWittenBellInterpolated.__init__r*r r rrr3ssr3có"eZdZdZd‡fdd„ Z‡ZS)ÚAbsoluteDiscountingInterpolatedz9Interpolated version of smoothing with absolute discount.çè?c s"tƒjt|fdd|ii|¤ŽdS)Nr/Údiscount)rrr©r r2r7rrr rr}sÿÿ ÿz(AbsoluteDiscountingInterpolated.__init__)r6r*r r rrr5zór5cr4)ÚKneserNeyInterpolatedz-Interpolated version of Kneser-Ney smoothing.çš™™™™™¹?c sHd|kr dkstdƒ‚tdƒ‚tƒjt|fd||dœi|¤ŽdS)Nrr)zCDiscount must be between 0 and 1 for probabilities to sum to unity.r/)r7r2)Ú ValueErrorrrrr8rr rr†sÿÿÿÿÿ ÿzKneserNeyInterpolated.__init__)r;r*r r rrr:ƒr9r:N)rZ nltk.lm.apirrZnltk.lm.smoothingrrrrrr(r+r.r3r5r:r r r rÚs