o rZh @sdZddlZddlZddlmZmZddlmZddlmZddl m Z ddl m Z ddl mZGd d d ed Zd d ZddZdddZGddded ZdS)zLanguage Model Interface.N)ABCMetaabstractmethod)bisect) accumulate) NgramCounter) log_base2) Vocabularyc@s0eZdZdZddZeddZeddZdS) SmoothingzNgram Smoothing Interface Implements Chen & Goodman 1995's idea that all smoothing algorithms have certain features in common. This should ideally allow smoothing algorithms to work both with Backoff and Interpolation. cCs||_||_dS)z :param vocabulary: The Ngram vocabulary object. :type vocabulary: nltk.lm.vocab.Vocabulary :param counter: The counts of the vocabulary items. :type counter: nltk.lm.counter.NgramCounter N)vocabcounts)self vocabularycounterr:/var/www/auris/lib/python3.10/site-packages/nltk/lm/api.py__init__s zSmoothing.__init__cCtNNotImplementedError)r wordrrr unigram_score&zSmoothing.unigram_scorecCrrrr rcontextrrr alpha_gamma*rzSmoothing.alpha_gammaN)__name__ __module__ __qualname____doc__rrrrrrrrr s r ) metaclasscCst|t|S)z0Return average (aka mean) for sequence of items.)sumlen)itemsrrr_mean/sr$cCst|tjr|St|Sr) isinstancerandomRandom)Zseed_or_generatorrrr_random_generator4s  r(cCsR|stdt|t|krtdtt|}|d}|}|t|||S)z`Like random.choice, but with weights. Heavily inspired by python 3.6 `random.choices`. z"Can't choose from empty populationz3The number of weights does not match the population) ValueErrorr"listrr&r) populationweightsrandom_generator cum_weightstotal thresholdrrr_weighted_choice:s r2c@sheZdZdZdddZdddZdddZedd d Zdd d Z d dZ ddZ ddZ dddZ dS) LanguageModelzKABC for Language Models. Cannot be directly instantiated itself. NcCs`||_|rt|tstjd|jjddd|durtn||_|dur+t|_ dS||_ dS)apCreates new LanguageModel. :param vocabulary: If provided, this vocabulary will be used instead of creating a new one when training. :type vocabulary: `nltk.lm.Vocabulary` or None :param counter: If provided, use this object to count ngrams. :type counter: `nltk.lm.NgramCounter` or None :param ngrams_fn: If given, defines how sentences in training text are turned to ngram sequences. :type ngrams_fn: function or None :param pad_fn: If given, defines how sentences in training text are padded. :type pad_fn: function or None z$The `vocabulary` argument passed to z- must be an instance of `nltk.lm.Vocabulary`.) stacklevelN) orderr%rwarningswarn __class__rr rr )r r6r rrrrrPszLanguageModel.__init__cs@js|dur tdj|jfdd|DdS)zeTrains the model on a text. :param text: Training text as a sequence of sentences. Nz:Cannot fit without a vocabulary or text to create it from.c3s|] }j|VqdSr)r lookup).0sentr rr tz$LanguageModel.fit..)r r*updater )r textZvocabulary_textrr=rfiths zLanguageModel.fitcCs&||j||r|j|SdS)zMasks out of vocab (OOV) words and computes their model score. For model-specific logic of calculating scores, see the `unmasked_score` method. N)unmasked_scorer r:rrrrscorevs zLanguageModel.scorecCr)aScore a word given some optional context. Concrete models are expected to provide an implementation. Note that this method does not mask its arguments with the OOV label. Use the `score` method for that. :param str word: Word for which we want the score :param tuple(str) context: Context the word is in. If `None`, compute unigram score. :param context: tuple(str) or None :rtype: float rrrrrrCszLanguageModel.unmasked_scorecCst|||S)zEvaluate the log score of this word in this context. The arguments are the same as for `score` and `unmasked_score`. )rrDrrrrlogscorezLanguageModel.logscorecCs"|r |jt|d|S|jjS)zHelper method for retrieving counts for a given context. Assumes context has been checked and oov words in it masked. :type context: tuple(str) or None )r r"Zunigrams)r rrrrcontext_countsszLanguageModel.context_countscsdtfdd|DS)a?Calculate cross-entropy of model for given evaluation text. This implementation is based on the Shannon-McMillan-Breiman theorem, as used and referenced by Dan Jurafsky and Jordan Boyd-Graber. :param Iterable(tuple(str)) text_ngrams: A sequence of ngram tuples. :rtype: float r)cs$g|]}|d|ddqS)r)N)rE)r;Zngramr=rr s$z)LanguageModel.entropy..)r$r Z text_ngramsrr=rentropys zLanguageModel.entropycCstd||S)zCalculates the perplexity of the given text. This is simply 2 ** cross-entropy for the text, so the arguments are the same. g@)powrKrJrrr perplexityrFzLanguageModel.perplexityrGcs|durgnt|}t|}|dkrat|jkr#|j ddn|j}rM|sMtdkr>ddngj}rM|r2t|}t|t fdd|D|Sg}t |D]}| j d|||dqg|S)aGenerate words from the model. :param int num_words: How many words to generate. By default 1. :param text_seed: Generation can be conditioned on preceding context. :param random_seed: A random seed or an instance of `random.Random`. If provided, makes the random sampling part of generation reproducible. :return: One (str) word or a list of words generated from model. Examples: >>> from nltk.lm import MLE >>> lm = MLE(2) >>> lm.fit([[("a", "b"), ("b", "c")]], vocabulary_text=['a', 'b', 'c']) >>> lm.fit([[("a",), ("b",), ("c",)]]) >>> lm.generate(random_seed=3) 'a' >>> lm.generate(text_seed=['a']) 'b' NrGc3s|] }|VqdSr)rD)r;wrr rrr>r?z)LanguageModel.generate..) num_words text_seed random_seed) r+r(r"r6rHr r:sortedr2tuplerangeappendgenerate)r rPrQrRr.Zsamples generated_rrOrrWs8 zLanguageModel.generate)NNr)rGNN)rrrrrrBrDrrCrErHrKrMrWrrrrr3Is      r3r)rr&r7abcrrr itertoolsrZnltk.lm.counterrZ nltk.lm.utilrZnltk.lm.vocabularyrr r$r(r2r3rrrrs